Unable to reindex search

Using Mayan EDMS Bugs or errors
1

I have migrated to the latest version of mayan edms however I am unable to reindex my search backend.

The search status shows that indexing has stopped, it doesnt continue after sometime.

I see a lot of errors thrown by rabbitmq. How would I fix my search?

Below is the link for the rabbitmq log

https://drive.google.com/file/d/12LZZq2x4IzA3gHntpReT41qPOAt4c29z/view?usp=sharing

I have deleted the woosh folder in the mayan_app volume and now I get an error after intializing search and trying to reindex. Here is the error.

Task mayan.apps.dynamic_search.tasks.task_reindex_backend[7a554ce9-eed1-4682-9237-7f0df2a54d81] raised unexpected: PermissionError(13, 'Permission denied')
Traceback (most recent call last):
  File "/opt/mayan-edms/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/tasks.py", line 157, in task_reindex_backend
    search_backend.reset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/whoosh.py", line 483, in reset
    self.tear_down(search_model=search_model)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/whoosh.py", line 494, in tear_down
    self._clear_search_model_index(search_model=search_model)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/whoosh.py", line 210, in _clear_search_model_index
    self._get_storage().create_index(
  File "/opt/mayan-edms/lib/python3.9/site-packages/whoosh/filedb/filestore.py", line 150, in create_index
    return indexclass.create(self, schema, indexname)
  File "/opt/mayan-edms/lib/python3.9/site-packages/whoosh/index.py", line 425, in create
    TOC.create(storage, schema, indexname)
  File "/opt/mayan-edms/lib/python3.9/site-packages/whoosh/index.py", line 607, in create
    storage.delete_file(filename)
  File "/opt/mayan-edms/lib/python3.9/site-packages/whoosh/filedb/filestore.py", line 544, in delete_file
    os.remove(self._fpath(name))
PermissionError: [Errno 13] Permission denied: '/var/lib/mayan/whoosh/_cabinets.cabinetsearchresult_0.toc'```
2 
`PermissionError: [Errno 13] Permission denied: '/var/lib/mayan/whoosh/_cabinets.cabinetsearchresult_0.toc'````

The mayan user inside the Docker container does not have filesystem permissions to create/update the Whoosh search index.

This does not happen under normal circumstances. At some point the /var/lib/mayan folder was modified by a user other than the mayan user, which caused the ownership or the access mode of the folder or a subfolder to be changed.

Every time the Docker image starts it resets the ownership of the volume. There is something on your installation either disabling or undoing that ownership change.

A quick fix would be to open a shell inside a running Docker container and change the ownership.

docker exec -ti <container name> /bin/bash

chown mayan:mayan -R /var/lib/mayan

/opt/mayan-edms/bin/mayan-edms.py search_initialize

This would only be a temporary fix since this chown command is executed every time the container starts, and there is something else on your installation changing back the ownership.

3

Thank you for the help!

I have tried changing the ownership of /var/lib/mayan. I was able to start the search reindex process, but it has stopped again after some time.

I dont think there is an issue with the ownshership anymore, but i found these write lock files

-rwxr-xr-x 1 mayan mayan 0 Feb 12 06:57 documents.documentsearchresult_WRITELOCK

-rwxr-xr-x 1 mayan mayan 0 Feb 12 07:00 documents.documentversionsearchresult_WRITELOCK

and here’s a new error logged by the mayan container

[2023-02-12 07:33:03,148: CRITICAL/MainProcess] Unrecoverable error: PreconditionFailed(406, ‘PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more’, (0, 0), ‘’)
Traceback (most recent call last):
File “/opt/mayan-edms/lib/python3.9/site-packages/celery/worker/worker.py”, line 203, in start
self.blueprint.start(self)
File “/opt/mayan-edms/lib/python3.9/site-packages/celery/bootsteps.py”, line 116, in start
step.start(parent)
File “/opt/mayan-edms/lib/python3.9/site-packages/celery/bootsteps.py”, line 365, in start
return self.obj.start()
File “/opt/mayan-edms/lib/python3.9/site-packages/celery/worker/consumer/consumer.py”, line 332, in start
blueprint.start(self)
File “/opt/mayan-edms/lib/python3.9/site-packages/celery/bootsteps.py”, line 116, in start
step.start(parent)
File “/opt/mayan-edms/lib/python3.9/site-packages/celery/worker/consumer/consumer.py”, line 628, in start
c.loop(*c.loop_args())
File “/opt/mayan-edms/lib/python3.9/site-packages/celery/worker/loops.py”, line 97, in asynloop
next(loop)
File “/opt/mayan-edms/lib/python3.9/site-packages/kombu/asynchronous/hub.py”, line 362, in create_loop
cb(*cbargs)
File “/opt/mayan-edms/lib/python3.9/site-packages/kombu/transport/base.py”, line 235, in on_readable
reader(loop)
File “/opt/mayan-edms/lib/python3.9/site-packages/kombu/transport/base.py”, line 217, in _read
drain_events(timeout=0)
File “/opt/mayan-edms/lib/python3.9/site-packages/amqp/connection.py”, line 525, in drain_events
while not self.blocking_read(timeout):
File “/opt/mayan-edms/lib/python3.9/site-packages/amqp/connection.py”, line 531, in blocking_read
return self.on_inbound_frame(frame)
File “/opt/mayan-edms/lib/python3.9/site-packages/amqp/method_framing.py”, line 53, in on_frame
callback(channel, method_sig, buf, None)
File “/opt/mayan-edms/lib/python3.9/site-packages/amqp/connection.py”, line 537, in on_inbound_method
return self.channels[channel_id].dispatch_method(
File “/opt/mayan-edms/lib/python3.9/site-packages/amqp/abstract_channel.py”, line 156, in dispatch_method
listener(*args)
File “/opt/mayan-edms/lib/python3.9/site-packages/amqp/channel.py”, line 293, in _on_close
raise error_for_code(
amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
2023-02-12 07:33:06,035 INFO exited: mayan-edms-worker_b (exit status 0; expected)

4

I dont think there is an issue with the ownshership anymore, but i found these write lock files

Those are normal, they are part of the Whoosh search library. Locks are objects (files, database entries, etc) used to avoid multiple code from corrupting a resource by modifying it at the same time.

PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

Never seen this error before. Looking online seems to be that the worker is having problems communicating with the RabbitMQ.

Double check your networking configuration and remove any custom changes.

Could also be insufficient resources causing the worker to be killed off after reserving the task and never acknowledging it back.

https://www.rabbitmq.com/consumers.html#acknowledgement-timeout

5

Does rabbitmq store anything on disk? Would it be alright if I delete the volume and create it again?

6

Yes it does, all pending background tasks. If you delete the volume all pending Mayan background tasks will be deleted.

As for settings, after deleting the volume, RabbitMQ does seem to recover and rebuilds everything it needs to startup again, minus any old task data.

You can also try restarting the entire stack first before deleting the RabbitMQ volume.

docker compose down

docker compose up -d

This will stop and deletes all running containers. For many cases is enough to kick start an installation with issues.

7

Unrecoverable error: "PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')"

I think I managed to increase the timeout by adding docker compose to the configuration:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbit consumer_timeout 36000000"

I temporarily set the limit to 10 hours for testing, and after 30 minutes of startup I no longer receive the timeout error.

8

I can’t edit the post, so I’m adding another answer.

Adding an entry to docker compose:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbit consumer_timeout 36000000"
It solved all my problems. Search now works properly for new files. I no longer have to restart containers to resume search engine indexing. Indexing is no longer interrupted.

2 Likes
9

Testing this out on some large Kubernetes installations. If it works out, well make it an official recommendation.

Thanks for the research!

1 Like
10

Dear Roberto,

did you perform the testst as stated above? I also wrote Search reindex doesn't work - #3 by joerg.sichermann and I am currently testing the indexing on my system.

BR, Joerg