Search reindex doesn't work

[vintager]

Tried with both Tools → Reindex search backend and

sudo docker compose run frontend run_command search_initialize
sudo docker compose run frontend run_command search_reindex

2.5 GB RAM used, 5.5 GB RAM remain free. 4 CPU cores, with the CPU load near zero the entire time.

RabbitMQ logs show no relevant errors, neither do the app container logs. The “search” queue remains idle most of the time:

The reindexing happens at an ultra-slow pace, eventually petering out with the “search” queue becoming empty. At the same time, the number of indexed document pages is vanishingly small, i.e., I’m experiencing the same problem as the guy from

https://forum.mayan-edms.com/t/search-not-working-anymore/

This appears to be a related problem:

https://forum.mayan-edms.com/t/documents-dont-get-send-to-tesseract/

PS RabbitMQ logs do show errors occasionally, not sure if related:

2023-05-14 11:05:56.376991+00:00 [warning] <0.2874.0> closing AMQP connection <0.2874.0> (172.30.0.5:55202 -> 172.30.0.4:5672, vhost: 'mayan', user: 'mayan'):
2023-05-14 11:05:56.376991+00:00 [warning] <0.2874.0> client unexpectedly closed TCP connection
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>     supervisor: {<0.2878.0>,rabbit_channel_sup}
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>     errorContext: shutdown_error
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>     reason: noproc
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>     offender: [{pid,<0.2881.0>},
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                {id,channel},
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                {mfargs,
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                    {rabbit_channel,start_link,
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                        [1,<0.2874.0>,<0.2879.0>,<0.2874.0>,
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                         <<"172.30.0.5:55202 -> 172.30.0.4:5672">>,
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                         rabbit_framing_amqp_0_9_1,
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                         {user,<<"mayan">>,
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                             [administrator],
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                             [{rabbit_auth_backend_internal,none}]},
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                         <<"mayan">>,
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                         [{<<"consumer_cancel_notify">>,bool,true},
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                          {<<"connection.blocked">>,bool,true},
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                          {<<"authentication_failure_close">>,bool,true}],
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                         <0.2875.0>,<0.2880.0>]}},
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                {restart_type,intrinsic},
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                {shutdown,70000},
2023-05-14 11:05:56.377652+00:00 [error] <0.2878.0>                {child_type,worker}]

The same warning pops up quite often:

2023-05-14 11:09:02.178962+00:00 [warning] <0.3372.0> closing AMQP connection <0.3372.0> (172.30.0.5:58246 -> 172.30.0.4:5672, vhost: 'mayan', user: 'mayan'):
2023-05-14 11:09:02.178962+00:00 [warning] <0.3372.0> client unexpectedly closed TCP connection

[tomkolp]

I have the same problem from the very beginning when I started using mayan edms.
The logs keep showing:

client unexpectedly closed TCP connection

At first I ignored it because everything seemed to be working properly.
When the number of documents added increased significantly, a search problem appeared. Newly added documents are never indexed in the search engine. Only docker compose down/up triggers re-indexing and the search works correctly. After some time, the problem returns.

mayan-rabbitmq-1    | 2023-11-08 07:31:17.861518+00:00 [warning] <0.15041.0> closing AMQP connection <0.15041.0> (172.28.0.4:52096 -> 172.28.0.3:5672, vhost: 'mayan', user: 'user'):
mayan-rabbitmq-1    | 2023-11-08 07:31:17.861518+00:00 [warning] <0.15041.0> client unexpectedly closed TCP connection
mayan-rabbitmq-1    | 2023-11-08 07:31:17.862591+00:00 [warning] <0.15030.0> closing AMQP connection <0.15030.0> (172.28.0.4:52086 -> 172.28.0.3:5672, vhost: 'mayan', user: 'user'):
mayan-rabbitmq-1    | 2023-11-08 07:31:17.862591+00:00 [warning] <0.15030.0> client unexpectedly closed TCP connection
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>     supervisor: {<0.15034.0>,rabbit_channel_sup}
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>     errorContext: shutdown_error
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>     reason: noproc
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>     offender: [{pid,<0.15037.0>},
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                {id,channel},
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                {mfargs,
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                    {rabbit_channel,start_link,
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                        [1,<0.15030.0>,<0.15035.0>,<0.15030.0>,
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                         <<"172.28.0.4:52086 -> 172.28.0.3:5672">>,
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                         rabbit_framing_amqp_0_9_1,
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                         {user,<<"user">>,
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                             [administrator],
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                             [{rabbit_auth_backend_internal,
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                                  #Fun<rabbit_auth_backend_internal.3.131409679>}]},
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                         <<"mayan">>,
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                         [{<<"consumer_cancel_notify">>,bool,true},
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                          {<<"connection.blocked">>,bool,true},
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                          {<<"authentication_failure_close">>,bool,true}],
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                         <0.15031.0>,<0.15036.0>]}},
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                {restart_type,intrinsic},
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                {shutdown,70000},
mayan-rabbitmq-1    | 2023-11-08 07:31:17.864126+00:00 [error] <0.15034.0>                {child_type,worker}]
mayan-rabbitmq-1    | 2023-11-08 07:31:17.869492+00:00 [warning] <0.15054.0> closing AMQP connection <0.15054.0> (172.28.0.4:52102 -> 172.28.0.3:5672, vhost: 'mayan', user: 'user'):
mayan-rabbitmq-1    | 2023-11-08 07:31:17.869492+00:00 [warning] <0.15054.0> client unexpectedly closed TCP connection

mayan-rabbitmq-1    | 2023-11-08 07:31:15.945492+00:00 [info] <0.14998.0> accepting AMQP connection <0.14998.0> (172.28.0.4:52076 -> 172.28.0.3:5672)
mayan-rabbitmq-1    | 2023-11-08 07:31:15.947166+00:00 [info] <0.14998.0> connection <0.14998.0> (172.28.0.4:52076 -> 172.28.0.3:5672): user 'user' authenticated and granted access to vhost 'mayan'
mayan-rabbitmq-1    | 2023-11-08 07:31:15.951042+00:00 [info] <0.15010.0> accepting AMQP connection <0.15010.0> (172.28.0.4:52080 -> 172.28.0.3:5672)
mayan-rabbitmq-1    | 2023-11-08 07:31:15.952960+00:00 [info] <0.15010.0> connection <0.15010.0> (172.28.0.4:52080 -> 172.28.0.3:5672): user 'user' authenticated and granted access to vhost 'mayan'
mayan-rabbitmq-1    | 2023-11-08 07:31:16.243389+00:00 [warning] <0.14998.0> closing AMQP connection <0.14998.0> (172.28.0.4:52076 -> 172.28.0.3:5672, vhost: 'mayan', user: 'user'):
mayan-rabbitmq-1    | 2023-11-08 07:31:16.243389+00:00 [warning] <0.14998.0> client unexpectedly closed TCP connection
mayan-rabbitmq-1    | 2023-11-08 07:31:16.541911+00:00 [warning] <0.15010.0> closing AMQP connection <0.15010.0> (172.28.0.4:52080 -> 172.28.0.3:5672, vhost: 'mayan', user: 'user'):
mayan-rabbitmq-1    | 2023-11-08 07:31:16.541911+00:00 [warning] <0.15010.0> client unexpectedly closed TCP connection

mayan-rabbitmq-1    | 2023-11-08 07:53:33.749015+00:00 [warning] <0.971.0> Consumer 25143 on channel 1 has timed out waiting for delivery acknowledgement. Timeout used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
mayan-rabbitmq-1    | 2023-11-08 07:53:33.805435+00:00 [error] <0.971.0> Channel error on connection <0.962.0> (172.28.0.4:43030 -> 172.28.0.3:5672, vhost: 'mayan', user: 'user'), channel 1:
mayan-rabbitmq-1    | 2023-11-08 07:53:33.805435+00:00 [error] <0.971.0> operation none caused a channel exception precondition_failed: delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
mayan-app-1         | [2023-11-08 07:53:33,807: CRITICAL/MainProcess] Unrecoverable error: PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')
mayan-app-1         | Traceback (most recent call last):
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/celery/worker/worker.py", line 202, in start
mayan-app-1         |     self.blueprint.start(self)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/celery/bootsteps.py", line 116, in start
mayan-app-1         |     step.start(parent)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/celery/bootsteps.py", line 365, in start
mayan-app-1         |     return self.obj.start()
mayan-app-1         |            ^^^^^^^^^^^^^^^^
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 336, in start
mayan-app-1         |     blueprint.start(self)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/celery/bootsteps.py", line 116, in start
mayan-app-1         |     step.start(parent)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 726, in start
mayan-app-1         |     c.loop(*c.loop_args())
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/celery/worker/loops.py", line 97, in asynloop
mayan-app-1         |     next(loop)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
mayan-app-1         |     cb(*cbargs)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/kombu/transport/base.py", line 248, in on_readable
mayan-app-1         |     reader(loop)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/kombu/transport/base.py", line 230, in _read
mayan-app-1         |     drain_events(timeout=0)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/amqp/connection.py", line 525, in drain_events
mayan-app-1         |     while not self.blocking_read(timeout):
mayan-app-1         |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/amqp/connection.py", line 531, in blocking_read
mayan-app-1         |     return self.on_inbound_frame(frame)
mayan-app-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/amqp/method_framing.py", line 53, in on_frame
mayan-app-1         |     callback(channel, method_sig, buf, None)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/amqp/connection.py", line 537, in on_inbound_method
mayan-app-1         |     return self.channels[channel_id].dispatch_method(
mayan-app-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
mayan-app-1         |     listener(*args)
mayan-app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/amqp/channel.py", line 293, in _on_close
mayan-app-1         |     raise error_for_code(
mayan-app-1         | amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

https://ss-yz.sentry.io/share/issue/f8043024ed3d4059b10fc48579443b81/

EDIT:
I did the test on clean Ubuntu 22.04.3LTS and ran mayan edms via docker compose. I didn’t change anything in the env configuration. Without SSL on port 80. I added 5 files, after some time I see a repeating message in the console:
client unexpectedly closed TCP connection
According to sentry, the problem first appeared in version 4.5.7, but this may be a coincidence.

[joerg.sichermann]

Hello everyone,

I wrote https://forum.mayan-edms.com/t/search-not-working-anymore/ and switched my direct install from Mayan to docker (version image: 4.5s) over the weekend. I left the configuration largely untouched. And I have the same problem now as described here.

The following helped me out:
I reduced SEARCH_INDEXING_CHUNK_SIZE from default 25 to 5 and did

docker compose stop
docker compose start

The index is currently being built and it seems to keep running. As I already wrote in https://forum.mayan-edms.com/t/search-not-working-anymore/, I suspect that a large SEARCH_INDEXING_CHUNK_SIZE leads to strange problems during the initial setup of the index. SEARCH_INDEXING_CHUNK_SIZE can be increased again after the index is built.

I hope this helps

BR, Joerg

[Ralf]

Hello everyone,

where can i change that SEARCH_INDEXING_CHUNGK_SIZE parameter?

if have a “config_backup.yml” in my media folder which contains that parameter, but the “config.yml” in the same folder doesn’t contain it.

Thanks,
Ralf

[joerg.sichermann]

Hi Ralf,

you find the parameter SEARCH_INDEXING_CHUNGK_SIZE in the settings inside Mayan GUI. There is no need to edit some files directly. You can just login and change the settings.

My Mayan is in German. System → Einrichtung-> Einstellungen → Suche

Good luck and Best regards
Jörgs

[mknichala]

Hi,

I have the same errors at my log and change the parameter. But my search do not work and i get the following error in the gui:
“Search backend error. Verify that the search service is available and that the search syntax is valid for the active search backend; Error -3 while decompressing data: incorrect header check”

Best Regards
Marian

[joerg.sichermann]

Hi mknichala,

I assume that you are using whoosh as your search backend. have you tried deleting the whoosh search folder on the server and reinitialising the search with search_initialize? For me it would be like this

rm -R /home/mayan/media/whoosh
MAYAN_MEDIA_ROOT="/home/mayan/media/" /opt/mayan-edms/bin/mayan-edms.py search_initialise

BR joerg

[mknichala]

Hi Joerg,

thanks for the reply. I use a docker environment. I rename the whoosh folder and lunch /opt/mayan-edms/bin/mayan-edms.py search_initialise at the mayan_app_1 container. the folder is created but empty. for me it looks like nothing rebuild, because all process are low. I also tried

docker-compose run frontend run_command search_initialize
docker-compose run frontend run_command search_reindex

from vintager but it looks nothing happen after it for me too.
So i moved back my whoosh folder. Now the errors at search from GUI are gone, but he found nothing.

Best Regards
Marian

[joerg.sichermann]

Hello Marian,

Your description leads me to believe that the worker for the woosh queue is not working or that the queue is not being created correctly.

You can now first check whether the queue is full. If it is empty, no index can be created. If the queue is full but not being processed, the process does not seem to be working.

In your case I would backup the database, backup the document files and then check out a new Docker environment and feed it with the new data. An article in the forum on switching from direct deploy installation to Docker installation is suitable as a guide. You will certainly find it via the search

The instructions work in your case because you are switching from a docker installation to a docker installation. That works in exactly the same way.

I apologize for suggesting such drastic measures. However, Mayan edms consists of so many components that a fresh start is usually implemented much faster than repairing the current system. At least that’s my experience from the last few years. It is also a good exercise to reconstruct the data from the backup in the event of a system failure.

In addition to the documents and the database, please also consider the mayan configuration file. Using the original installation file as origin and then only copying necessary changes from the old file has proven to be a good idea.

Kind regards, Jörg

[mknichala]

Hi Jörg,

thank you very much for long answer. I found some time to try your recommends. I think you means this thread:

Direct deployment to Docker Compose migration

Thanks to help from @DocCyblade the following procedure was created. Please try this procedure on a backup copy of your installations (or even better a VM snapshot) and report your results (good or bad) so we can finalize this and add it to the documentation. Update the direct deployment to the 4.4 series. Follow the instructions located in: https://docs.mayan-edms.com/parts/upgrades.html Purge all file caches. Option 1: Via the user interface. System → Tools → File caches [image] Refresh…

I deleted all my cashes and the woosh folder. After a new deploy of the docker container he was now created automatically and some files were in. I can now upload new files and the search works fine for this, but unfortunately my old files not get in the index again also

docker-compose run frontend run_command search_initialize
docker-compose run frontend run_command search_reindex

still not help. I see still a error at the log of the app container if i run search_reindex

ERROR/ForkPoolWorker-7] Task mayan.apps.dynamic_search.tasks.task_reindex_backend[3f4192b3-4bbf-406a-bad8-4d3e7a13f708] raised unexpected: TypeError("'>' not supported between instances of 'str' and 'int'")
Traceback (most recent call last):
  File "/opt/mayan-edms/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/mayan/apps/dynamic_search/tasks.py", line 176, in task_reindex_backend
    for id_list in search_model.get_id_groups():
  File "/opt/mayan-edms/lib/python3.11/site-packages/mayan/apps/common/utils.py", line 272, in group_iterator
    if group_size > 1:
       ^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'str' and 'int'

what can i do?

Best Regards
Marian

[mknichala]

Hi,

I make a upgrade to 4.7 today. But still the same error. Also change my search backend to elasticsearch was not helpful, still the same error message.
All new documents that i upload are still indexed but if i try to reindex (cli or gui) all indexes are gone and i see the error from my last post.

Best Regards
Marian