[SOLVED] I lost previews and thumbnail after 3.3.9 upgrade.

[sal-e]

Hi everyone,

I have been using Mayan EDMS for more then year without any issues, but since 3.3.9 upgrade I start having many image issues. First I lost my hi-res previews, next I lost some of my thumbnail and finally I don't have any images any more.

Currently, I am running 3.3.11 Docker install. My backend is Docker: postgres:9.6-alpine and redis:5.0-alpine. In my log see this error and traceback:

Code: Select all

mayan.apps.common.middleware.error_logging <241> [ERROR] "process_exception() line 18 [31;1mException caught by request middleware; <WSGIRequest: GET '/api/documents/646/versions/663/pages/9324/image/?width=&height=&zoom=100&rotation=0&_hash=92bcac02c4a72586e21044c0b244b052f5747c7d2c25e6086ca89ca64098e3f3'>, The operation timed out.[0m"

Traceback (most recent call last):
File "/opt/mayan-edms/lib/python3.7/site-packages/celery/backends/asynchronous.py", line 255, in _wait_for_pending
on_interval=on_interval):
File "/opt/mayan-edms/lib/python3.7/site-packages/celery/backends/asynchronous.py", line 54, in drain_events_until
raise socket.timeout()
socket.timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/mayan-edms/lib/python3.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/mayan-edms/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/opt/mayan-edms/lib/python3.7/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/opt/mayan-edms/lib/python3.7/site-packages/rest_framework/views.py", line 494, in dispatch
response = self.handle_exception(exc)
File "/opt/mayan-edms/lib/python3.7/site-packages/rest_framework/views.py", line 454, in handle_exception
self.raise_uncaught_exception(exc)
File "/opt/mayan-edms/lib/python3.7/site-packages/rest_framework/views.py", line 491, in dispatch
response = handler(request, *args, **kwargs)
File "/opt/mayan-edms/lib/python3.7/site-packages/rest_framework/generics.py", line 210, in get
return self.retrieve(request, *args, **kwargs)
File "/opt/mayan-edms/lib/python3.7/site-packages/django/views/decorators/cache.py", line 43, in _cache_controlled
response = viewfunc(request, *args, **kw)
File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/api_views.py", line 221, in retrieve
cache_filename = task.get(timeout=DOCUMENT_IMAGE_TASK_TIMEOUT)
File "/opt/mayan-edms/lib/python3.7/site-packages/celery/result.py", line 226, in get
on_message=on_message,
File "/opt/mayan-edms/lib/python3.7/site-packages/celery/backends/asynchronous.py", line 188, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "/opt/mayan-edms/lib/python3.7/site-packages/celery/backends/asynchronous.py", line 259, in _wait_for_pending
raise TimeoutError('The operation timed out.')
celery.exceptions.TimeoutError: The operation timed out.

But I have no idea what is wrong and where to start. Any help would be appreciated.

Thank you.

[rssfed23]

That looks to me as if it can't talk to redis properly. It would also explain your lack of images; as the task to generate them potentially isn't getting sent correctly via the redis broker.

What version were you on before 3.3.9? - Around the 3.3 timeframe we added a default password to the redis setup after user requests to make it more secure. If you review the latest docker deployment instructions or the docker compose file and look for any major differences between yours and the examples (especially around redis).

We had another forum user recently with a similar experience coming from 3.2 and that was the underlying cause. Of course; if you're coming over from 3.3.8 to .9 then it's potentially something entirely different and that shouldn't happen.

Just FYI: 3.3.12 should be out within a couple hours. Nothing in it that will impact your issue I think as it's a hotfix release, but worth mentioning.

[sal-e]

I have been tracking the release tightly, Usually installing the updates as soon as they are released. So I'm sure that before 3.3.9 I was running 3.3.8.

I remember when 3.3 was released I lost my images because I didn't have radis backend. I had to install the redis, purge my cache and everything start working as expected. I think it was 3.3.8 when I noticed change in installation manual and the addition of redis password. Using telnet on my docker host I managed to connect to my redis backend and I can authenticate and list my DBs, but they both are empty. I tried to verify redis connection from inside the mayan-edms docker container, but I could not find telnet or something similar to redis-cli.

I just notice something else. When I restart postgres container, before I start mayan-edms container, I see following error:

Code: Select all

STATEMENT: SELECT (date_trunc('month', document__date_added)) AS "d", COUNT("documents_documentversion"."id") AS "agg" FROM "documents_documentversion" INNER JOIN "documents_document" ON ("documents_documentversion"."document_id" = "documents_document"."id") WHERE "documents_document"."date_added" BETWEEN '2020-01-01T00:00:00+00:00'::timestamptz AND '2020-02-29T23:59:59.999999+00:00'::timestamptz GROUP BY (date_trunc('month', document__date_added))
ERROR: column "document__date_added" does not exist at character 29

[sal-e]

This was strange! This morning I upgraded to 3.3.12 and it still was not working. I went to work and when I came back everything is working again. Are there any maintenance tasks that might have ran and fix the issue?

[rssfed23]

It's entirely possible when upgrading to 3.3.12 that the migration process (that upgrades the DB to the latest schema) added in the column for you. We'd have no way to know without actually testing and checking it, but the second error you posted was a missing column in the database so potentially this could have fixed it. Not 100% sure to be fair. Keep an eye on it over the next couple days so we can ensure the issue doesn't return, as if there was a missing column i'd really want to know why it was missing to begin with (especially if it was part of an upgrade that didn't go through correctly before).

But for now, I'm glad it's working

[sal-e]

Hi,

Yes, It did happen again. But this time, I think, I managed to narrow down the problem. I believe, it is cache invalidation issue. Here is my setup:
1) I'm running docker on my URAID server. I have developed 'UNRAID app' for Mayan-EDMS and I would like to contribute it back to everyone.[2] It is just XML file telling UNRAID how to download configure official Mayan EDMS docker image using WebGUI and custom icon. (it needs to be updated for 3.3.x releases)
2) I'm also running official redis and postgress docker images. I'm following instructions found here: https://docs.mayan-edms.com/chapters/do ... er-install
3) I'm also running backup program that once a week will shutdown all docker containers, backup all application data and restart the docker containers.

Here is my problem, based on 'Simple Docker installation" the radis server is configured with persistent storage, but also also has --save "" config option. According to redis doc: 'It is also possible to remove all the previously configured save points by adding a save directive with a single empty string argument like in the following example: save ""' [1] As result every time my backup app restarts the redis docker container, it looses its DB. But mayan-edms still shows that some of the images are cached. (System > Tools > File caches: document_images > 0MB. I would have expected that if the image is missing in redis to invalidate the cache and rebuild it, but instead I get spinning wheel and eventually [X] for the document images.

I have found a work around:
1. Restart the mayan-edms container and before doing anything else go to File caches and purge cache. I might have to do it several times, but once document_images = 0MB I can move to step #2
2. Restart my redis container
3. Restart the mayan-edms container again

After that my images cache start rebuilding as expected and all documents are displayed correctly.

Possible solution for me is to make sure redis DBs are saved and survive restart. But this potentially could lead to other problem

Any ideas are appreciated, Thanks.


PS. I there any script or other method to validate my postgres DB. I might have messed up my DB upgrade during the new release upgrades.

[1] https://raw.githubusercontent.com/antir ... redis.conf
[2] https://github.com/SAL-e/docker-templates

[rssfed23]

We don't use redis to cache the images - they're stored inside the cache directory within your main Mayan Media directory.

By my understanding, the cache references are all stored in the postgres DB. You can run redis without persistence relatively safely.

We use redis more as a message broker and worker queue result store and not as a caching system. That's why we launch it without persistence fully enabled and set it to overwrite old entries when it starts running low in memory. The initial error you posted is to do with the database and not redis.

The image is never stored in redis. What you're experiencing is almost similar symptoms to what we have at https://gitlab.com/mayan-edms/mayan-edms/issues/735
There are some situations when Mayan fails to generate a cache image. Sometimes it's a timeout, or it tries to access a cache file that's not there. I think the latter is what you're experiencing. When it fails to read the cached image what we've observed is rather than trying to regenerate the cached version again it throws an error and never retries until you manually clear the cache out. This feels similar to what you're experiencing here on a single node setup (which I've also seen during some specific situations).

There are improvements coming in v4 to how caching is handled to ensure if a cache image isn't there (even if the db thinks it is) then that cache image is regenerated.

[itsthejb]

1. Restart the mayan-edms container and before doing anything else go to File caches and purge cache. I might have to do it several times, but once document_images = 0MB I can move to step #2

Is purge cache a Mayan feature? If so, where is it? I can't find it... Or do you mean delete/move/rename the cache directory on the server?

Thanks

[rssfed23]

Is purge cache a Mayan feature? If so, where is it? I can't find it... Or do you mean delete/move/rename the cache directory on the server?

You go to to System > Tools then "File caches" then click purge cache on the right.

[itsthejb]

Hmmmm I don't have File caches in there at all. Why would that be? I'm using the latest docker version