Page 1 of 1

Error 111 connecting to 172.17.0.1:6379.

Posted: Tue Jan 14, 2020 11:11 am
by Costo
Hi,
Thank you for the good conception and your job. I like Mayan.
I upgraded from 3.2.7 to 3.3.7. I met some strange things. I am on the observation.
My system: postgresql 9.5 + redis 5.0 alpine + Mayan edms 3.3.7 on docker

First I get this type of errors
----------------
mayan.apps.common.middleware.error_logging <5769> [ERROR] "process_exception() line 18 Exception caught by request middleware; <WSGIRequest: GET '/api/documents/12350/versions/12350/pages/15318/image/?width=&height=&zoom=&rotation=&_hash=92 bcac02c4a72586e21044c0b244b052f5747c7d2c25e6086ca89ca64098e3f3'>, Error 111 connecting to 172.17.0.1:6379. Connection refused."
Traceback (most recent call last):
File "/opt/mayan-edms/lib/python3.7/site-packages/redis/connection.py", line 539, in connect sock = self._connect()
File "/opt/mayan-edms/lib/python3.7/site-packages/redis/connection.py", line 596, in _connect raise err
File "/opt/mayan-edms/lib/python3.7/site-packages/redis/connection.py", line 584, in _connect sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused
---------------------------
It happens when I go on preview. On the small icons I can see preview no problem(not cross) and sometime I get the preview after many try.

Second, I see the 3 procceses with the sign ERROR, They take always 70% of my 4 CPU.
-------------------
/opt/mayan-edms/python3 /opt/mayan-edms/bin/celery -A mayan -Ofair -l ERROR -Q defaulr, checkouts_periodic, indexing……

On the logs
[2020-01-14 08:11:03 +0000] [5584] [INFO] Worker exiting (pid: 5584)
[2020-01-14 08:12:39 +0000] [290] [CRITICAL] WORKER TIMEOUT (pid:5583)
[2020-01-14 08:12:40 +0000] [5699] [INFO] Booting worker with pid: 5699
[2020-01-14 08:13:03 +0000] [290] [CRITICAL] WORKER TIMEOUT (pid:5584)
[2020-01-14 08:13:03 +0000] [5708] [INFO] Booting worker with pid: 5708
[2020-01-14 09:07:25 +0000] [5699] [INFO] Autorestarting worker after current request.
-------------------------
I see also the errors like :
mayan.apps.ocr.managers <5606> [ERROR] "process_document_version() line 96 OCR error for document version: 13326; The operation timed out."

What could be suggestions where to look to identify the problem. The EDMS is very slow and heavy.
Thank you in advance

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Tue Jan 14, 2020 11:33 am
by rssfed23
Thank you for your kind words!

That error is happening because the jobs that do things like thumbnail generation (and almost everything else in Mayan) aren't able to connect to Redis, which they use to communicate between one another and send tasks to different processes.

You're already running a separate redis container, correct? - it used to be that redis was bundled with Mayan but now it has to be deployed as a separate container entirely (as per the docs you have to launch a mayan container and a separate redis one now)

Have you had a look at the release notes for the upgrade paths from 3.2.7 to 3.3.7? - If you start at https://docs.mayan-edms.com/releases/3.2.8.html and work your way up each version at a time and have a read of the significant changes/upgrade steps you may notice something that wasn't done. There were some configuration values that changed format and might be causing the problem connecting.

Can you paste us your docker-compose environment section please (or all the -e values you pass into docker run)?

Also, as long as you have your data stored in mounted volumes (such as /docker-volumes/mayan-edms/media) you are able to remove the existing container and deploy again using an updated version of the docker-compose template file. That might help mitigate any environment variable changes that have happened over the various releases but didn't get applied.

But I would start with checking your redis container - it's either not running or mayan is trying to talk to the wrong redis container/one that doesn't exist as a connection refused usually means the service isn't there at all or hasn't started up rather than an error while running.

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Tue Jan 14, 2020 4:14 pm
by Costo
Thank for your quick reply.

Correct, I have ran 3 applications via docker:
Postgresql 9.5 (not 9.6 alpina, I tried to upgrade via pg_dumpt but I couldn't pass the test due to instability EDMS, I will try later).
Redis 5.0 alpina: I take the script from exemple
Mayanedms.3.3.7 :
docker run -d --name mayan-edms --restart=always -p 85:8000 -e MAYAN_DATABASE_ENGINE=django.db.backends.postgresql -e MAYAN_DATABASE_HOST=172.17.0.1 -e MAYAN_DATABASE_NAME=mayan -e MAYAN_DATABASE_PASSWORD=xxxx -e MAYAN_DATABASE_USER=mayan -e MAYAN_CELERY_BROKER_URL="redis://:yyyy@172.17.0.1:6379/0" -e MAYAN_CELERY_RESULT_BACKEND="redis://:yyyy@172.17.0.1:6379/1" -v /docker-volumes/mayan-edms/media:/var/lib/mayan -e MAYAN_APT_INSTALLS="mc tesseract-ocr-lit nano" -v /home/222:/srv/WatchFolder mayanedms/mayanedms:3.3.7

Sometimes it works well sometimes slow or doesn't load.
I tried all except one thing: I didn't reboot my linux. Its my fault. It turn 126days. After the reboot it works like a charm. To come to this simple decision that normally you do many times… sorry for your time.

When I pass in dictionary :
MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'xxxx','USER':'mayan','HOST':'172.17.0.1'}}"
I get:
-bash: !','USER': event not found
That is why I take in full string to pass.

I have some pdf files(especially on this test period) with the cross sign. I find it on dock-volume/mayan-edms/media folder, but not on the /var/lib/mayan/document_storage/ in container. I can download via EDMS but Pillow can't count the pages, OCR read.
Is it the possibility to sync? In which way the document from volume media folder appears in /var/lib/mayan/document_storage/? If I restart the container it not helps.

Thank you in advance

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Tue Jan 14, 2020 7:00 pm
by rssfed23
Something must be going a bit weird in your environment somewhere.

Postgresql 9.6 has been supported for a long time (I use postgresql 11 myself for Mayan).

If you're on 3.3.7 like you suggest then the MAYAN_DATABASES line should be something like:

Code: Select all

MAYAN_DATABASES: "{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'dbmayan','PASSWORD':'mayandvpassword','USER':'mayandbuser','HOST':'postgres'}}"
The fact you're getting the "BASH: !" error tells me there's a typo somewhere in your environment line.

Also, I wouldn't recommend hard-coding in the mayan database host (172.17.0.1). If you're using docker-compose then you can just use the name of the postgres container and as long as both containers are in the same docker network ("mayan-bridge" in our documentation) it should work.

I think the reboot fixed your initial process as Redis wasn't running (which would be why you got red X's and things were not working correctly).

Thumbnails can get a red X when the thumbnail hasn't generated correctly and mayan "gives up" trying to generate it. You can remove the container and start it again along with clearing the file cache (Tools > Caches > clear cache) and once both those tasks are done Mayan should try to generate the thumbnail once again.

I also always recommend people use docker-compose where possible as it makes the ongoing admin work of managing Mayan containers significantly easier - https://gitlab.com/mayan-edms/mayan-edm ... ompose.yml is our example file.
It wouldn't be a huge task to migrate your DB to at least PG 9.6 (which is our minimum supported version) and then re-deploy using docker compose instead.

It would also be helpful to have the docker logs while the thumbnail generation is failing as that can help us identify why it might not be generating thumbnails.

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Tue Jan 14, 2020 10:23 pm
by Costo
Thank you for good advice.
I will go deeper and learn about docker.

For the thumbnail generation:
I get for some documents the red X appeared on this period of test( maybe few early). I have 10k pages. I took one document. This document is in the docker_volume/... / media/document_storage and in the container /var/lib/mayan/document_storage/
I try to count the pages and I get the error below:

mayan.apps.converter.backends.python <497> [ERROR] "get_page_count() line 181 Exception determining page count using Pillow; cannot identify image file <File: /var/lib/mayan/document_storage/eb83fed5-b016-4cb7-aaa6-da610190b567>"
[2020-01-14 21:59:42,204: ERROR/ForkPoolWorker-4] Exception determining page count using Pillow; cannot identify image file <File: /var/lib/mayan/document_storage/eb83fed5-b016-4cb7-aaa6-da610190b567>

That it means, Pillow cant find this in some list. Physically it exists.

Thank you for your help

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Tue Jan 14, 2020 11:25 pm
by rssfed23
What file format is the document?

Pillow is able to find the file okay based on that error message. The error message means it can't process it after it's found it.

Pillow can handle things like jpg/pdf/txt. That error message suggests to me it's a file/document format pillow can't handle (you're likely to not get thumbnails on xlsx/docx/html/xml and a few other file formats).

You can also try uploading it yourself at https://mayan.app. Ignore the popup and use the username "upload" and password "uploadtest" and try uploading your document there to the "upload test" document type. If you still get a red X on that test server we can be 100% sure it's a format pillow doesn't like (then remove the document and empty the trash after you've finished).

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Thu Jan 16, 2020 12:00 am
by Costo
Apologise but I can't test via your system this kind of documents. I have done by myself. it is pdf.
On the real machine:
I took from docker-volume/.../document-storage. I converted the same document via pdftoppm and read with pillow. It works.
On the docker:
I repeated the same but I was stopped. After pdftoppm command:
I/O Error: Couldn't open file 'a743b02e-e542-4274-b6ae-e8d6a7998d3f': No such file or directory
I tried with another one and I got the ppm files.

In which order to proceed to get the same document-storage in the edms container? I cleaned the cache memory and I restarted the edms container. I verified if the cache was 0k. But somewhere it stay in the layer.
Could I remove the old edms images(3.1.1...3.2.8) safely , no impact on the newest version or on the volume?

Thank you in advance

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Thu Jan 16, 2020 12:59 am
by rssfed23
That's understandable if there's confidential data in there.

Yes you can remove old versions of the mayan image.

As long as you have mounted storage into the container then there's no "volume" to delete. Only if you're using a docker volume would it be dangerous to run a "docker rm volume".

As you say; mayan itself is handling other pdf files okay which means it has access to the storage.


How is the storage being mounted into the container? Is this a multi-node setup or single node?
If you're using a -v or mounted linux folder, is the document_storage sitting on nfs or samba or similar?

When you go into the container and into /var/lib/mayan/document_storage/ (inside the container) can you physically see the file there (ls -la a743b02e-e542-4274-b6ae-e8d6a7998d3f)?
Can you see the file from the host side (outside of the container)?

Although the preview doesn't work, when you go to all documents and find the file can you browse the pages within it (I'm wondering if it's just the preview that's not working but the pages themselves are okay as there's an issue under investigation that sounds similar)

How big is the document? How many pages does it have? It's possible mayan is timing out generating the thumbnail if it's a large file (the timeout is 20 or 30 seconds by default). Because of that with some larger files we may never get a preview image. That's why I ask if you can view pages inside the document view to see if it's just the thumbnail broken)

https://gitlab.com/mayan-edms/mayan-edms/issues/723
https://gitlab.com/mayan-edms/mayan-edms/issues/735

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Thu Jan 16, 2020 11:00 pm
by Costo
I decided to go from zero.
I left only mayan-volumes/... no any container up on disk and start:
Postgres.9.5
Redis 5 alpine
Mayan edms 3.3.7
All work very stable except for some documents(the same) with ‘x’ red.

About the disk : docker-volumes/ with all folders in is on the linux /sda1 (unix/linux disk).
It is mounted with -v in the container.

Yes, as I described, I can see the file physically in /var/lib/mayan/document_storage/ (inside the container).
root@8ce569695922:/var/lib/mayan/document_storage# ls -la a743b02e-e542-4274-b6ae-e8d6a7998d3f
-rw-r--r-- 1 mayan mayan 304407 Jan 14 11:38 a743b02e-e542-4274-b6ae-e8d6a7998d3f

I can see the same file on the mounted disk in on /sda1 docker-volumes/mayan-edms/media/document_storage/…
a@EDMS:/docker-volumes/mayan-edms/media/document_storage$ ls -la a743b02e-e542-4274-b6ae-e8d6a7998d3f
-rw-r--r-- 1 bretalita bretalita 304407 Jan 14 13:38 a743b02e-e542-4274-b6ae-e8d6a7998d3f

I said that I see the mirror of mounted disk in container.

The preview doesn't work for the thumbnail nor the preview page “No pages to display”. But I can see all information from a metadata, properties. I can download it via web mayan edms application. The file isn't corrupted. The document has only 3 pages and 297 kB
I tested manually to pass the command pdftoppm and Pillow on the same file in the container and in the mayan-volumes on disk. The both work.

I repeated several times : delete cache, restart mayan-edms container. No changes.
On the disk in docker-volumes/mayan-edms/media/ I see the /documents-cache about 4497408 of size. Should I clean it manually or it must be?

Interesting thing, that I can manually convert and read in maybe the same way as Mayan does.
Maybe it comes from some parameters written is some critical conditions in PostgreSQL.

I have about ~50 files maybe a little bit more with ‘x’ red from 10k. I encoded the file names for easy find and it is no problem since I can download it.
Thank you in advance

Re: Error 111 connecting to 172.17.0.1:6379.

Posted: Fri Jan 17, 2020 2:23 am
by rssfed23
Thanks for providing those details. This does seem to be a challenging one.

If this is only a single node running everything, then it's not the known issue I thought it was.

The *only* thing I can think of is the old postgres version, but that doesn't tie in with it only happening on around 50 out of 10k documents so it's unlikely (although it is an old version we no longer test for).

To clear the cache you go to System > tools > File caches and you'll see the document cache there you can click purge cache to manually empty the cache. There is a chance the thumbnail will try regenerating again then. But if it's the same file that won't generate even after deleting it and trying again when other files are, it has to be something specific about that file that the converter doesn't like. Figuring out what that is is the hard part though.