Mayan keeps crashing

Using Mayan EDMS Bugs or errors
1

Our live Mayan EDMS instance keeps crashing regularly sometime even in the middle of the night wothout any user action. It consumes about eight to ten GB of RAM even in idle. Is this normal?

Also a test instance lost 1.5 to 2 GB of RAM out of nowhere last midnight. We have no clue why it is like this.

grafik
grafik1040×321 13.9 KB

The RAM also keeps jumping from 5 GB to 6.5 GB and back in one or two seconds and back again repeatedly.

Both instances run on a Proxmox LXC Container.

Our live system has about 10,000 Documents. But a goot EDMS should work fine with that many files.

I don’t assume this is the intended behaviour of Mayan EDMS. But I don’t know what we are doing wrong.

2

try getting the logs

3

After a reboot of the docker container it is back up at twelve to fourteen GB of RAM.

I saved the startup output, the docker-compose.yml and the XXX-json.log in a zip file and you can download it here: https://cloud.gtr.de/index.php/s/YyrQfws8gNYqJJp/download

I am afraid that the reboot has overwritten the logfile from tonight.

4

I have run a new test with debug log level in the celery-beat docker-compose section. Here are the files: https://cloud.gtr.de/index.php/s/Dt4Cbq8Wc2AHdHQ/download

5

Going through your app logs looks like Mayan EDMS is working well, what’s crashing is RabbitMQ.

Also your database is badly configured, without a proper database connection nothing will work.

{"log":"django.db.utils.OperationalError: connection is bad: Name or service not known\n","stream":"stdout","time":"2024-04-25T13:04:13.59162113Z"}

Mayan EDMS is very advanced and flexible with many features. All that comes at a cost of more resources. Recommended RAM is 16GB which is quite modest for all that it does and on the low end compared to other enterprise software.

https://docs.mayan-edms.com/chapters/requirements.html#requirements

Our live system has about 10,000 Documents. But a goot EDMS should work fine with that many files.

A few sayings come to mind:

“Don’t look a gift horse in the mouth.”

“Don’t bite the hand that feeds you.”

“Appreciate the generosity, for in the warmth of a free bed lodging, the color of the sheets matters little.”

Their support is very affordable. For 10,000 documents you would pay $599 per month. For comparison we pay per month 1,400 for a CMS, $5,000 for a CRM and $2,400 for MinIO. Even combined they offer a fraction of the feature of Mayan EDMS.

6

Thanks for checking the log files.

The database is configured in the way it is done by the original docker-compose file supplied with the mayan setup procedure.
So no manual changes have been applied.
I don’t know where we could have messed something up with the database configuration.
Maybe the database connection needs to be adjusted to accept more connections at a time.

Most of the time the database seems to be available but then out of a sudden, the connection drops.
We can not see why the connection is dropped or why the other containers can not find the database sporadically.

7

Our live Mayan EDMS instance keeps crashing regularly sometime even in the middle of the night wothout any user action.

vs.

Most of the time the database seems to be available but then out of a sudden, the connection drops.
We can not see why the connection is dropped or why the other containers can not find the database sporadically.

So, you knew the database was the cause of the crash but did not mention this? Kinda important. Should be clearer about those details instead of jumping right to criticize the app.

The database is configured in the way it is done by the original docker-compose file supplied with the mayan setup procedure.

The documentation does not mention support for LXC, which means you are running Docker inside an LXC. Nested containers. So this is not a “by-the-book” installation.

Containers are not like virtual machines. From the system’s point of view, the container is a single process and a cgroup. When resources are low, the operating system will kill the process, which will kill the entire container. You made the problem worse by running everything in a single container, which itself is inside another single container. Probably two layers of AppArmor fighting each other too. You are running the DB, RabbitMQ, Mayan, OCR as a single process and a single resource cgroup. That’s a Frankenstein installation. A testament to the engineering of Mayan EDMS that it can even run in such a poor and tortured scenario! This explains the memory chart and the crash behavior perfectly! Your installation is fighting two layers of memory management as a single process.

Also the error message Name or service not known has nothing to do with connections. Might have to do with the database crashing and the app trying to access it while it boots up again, or it is a configuration error. Did added something else to try to mitigate access to the database? It is a miracle (and to Mayan EDMS credit too) that your database has not gotten corrupted with such setup.

Don’t get creative and follow the tried and test documentation, it has been proved to work thousands of times. If you don’t follow the documentation then don’t blame the app.

8

I tested Mayan in a Virtual Box VM. The Host is a Debian 12 System with 6 Cores and 10 GB RAM.

It runs with less RAM but the errors are the same:

rabbitmq-1    | 2024-05-02 11:51:07.987820+00:00 [warning] <0.1396.0> closing AMQP connection <0.1396.0> (172.18.0.5:37456 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:51:07.987820+00:00 [warning] <0.1396.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:51:08.488187+00:00 [info] <0.1489.0> accepting AMQP connection <0.1489.0> (172.18.0.5:44880 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:51:08.492396+00:00 [info] <0.1489.0> connection <0.1489.0> (172.18.0.5:44880 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:51:08.948108+00:00 [warning] <0.1384.0> closing AMQP connection <0.1384.0> (172.18.0.5:37446 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:51:08.948108+00:00 [warning] <0.1384.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:51:09.259250+00:00 [info] <0.1505.0> accepting AMQP connection <0.1505.0> (172.18.0.5:44890 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:51:09.266589+00:00 [info] <0.1505.0> connection <0.1505.0> (172.18.0.5:44890 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:51:13.587839+00:00 [warning] <0.1371.0> closing AMQP connection <0.1371.0> (172.18.0.5:37430 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:51:13.587839+00:00 [warning] <0.1371.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:51:13.889393+00:00 [info] <0.1521.0> accepting AMQP connection <0.1521.0> (172.18.0.5:58078 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:51:13.892463+00:00 [info] <0.1521.0> connection <0.1521.0> (172.18.0.5:58078 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:52:00.282244+00:00 [warning] <0.1336.0> closing AMQP connection <0.1336.0> (172.18.0.5:37416 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:52:00.282244+00:00 [warning] <0.1336.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:52:00.566412+00:00 [info] <0.1538.0> accepting AMQP connection <0.1538.0> (172.18.0.5:53720 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:52:00.572607+00:00 [info] <0.1538.0> connection <0.1538.0> (172.18.0.5:53720 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:52:08.404683+00:00 [warning] <0.1461.0> closing AMQP connection <0.1461.0> (172.18.0.5:44866 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:52:08.404683+00:00 [warning] <0.1461.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:52:09.482248+00:00 [info] <0.1554.0> accepting AMQP connection <0.1554.0> (172.18.0.5:35426 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:52:09.488248+00:00 [info] <0.1554.0> connection <0.1554.0> (172.18.0.5:35426 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:52:09.973130+00:00 [warning] <0.1470.0> closing AMQP connection <0.1470.0> (172.18.0.5:44874 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:52:09.973130+00:00 [warning] <0.1470.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:52:10.619456+00:00 [info] <0.1570.0> accepting AMQP connection <0.1570.0> (172.18.0.5:35432 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:52:10.622097+00:00 [info] <0.1570.0> connection <0.1570.0> (172.18.0.5:35432 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:52:12.681838+00:00 [warning] <0.1489.0> closing AMQP connection <0.1489.0> (172.18.0.5:44880 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:52:12.681838+00:00 [warning] <0.1489.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:52:14.187207+00:00 [info] <0.1586.0> accepting AMQP connection <0.1586.0> (172.18.0.5:34504 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:52:14.191834+00:00 [info] <0.1586.0> connection <0.1586.0> (172.18.0.5:34504 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:52:34.373484+00:00 [warning] <0.1505.0> closing AMQP connection <0.1505.0> (172.18.0.5:44890 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:52:34.373484+00:00 [warning] <0.1505.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:52:34.924519+00:00 [info] <0.1602.0> accepting AMQP connection <0.1602.0> (172.18.0.5:45916 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:52:34.927254+00:00 [info] <0.1602.0> connection <0.1602.0> (172.18.0.5:45916 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:53:18.056298+00:00 [warning] <0.1521.0> closing AMQP connection <0.1521.0> (172.18.0.5:58078 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:53:18.056298+00:00 [warning] <0.1521.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:54:02.789250+00:00 [info] <0.1625.0> accepting AMQP connection <0.1625.0> (172.18.0.5:39618 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:54:02.793583+00:00 [info] <0.1625.0> connection <0.1625.0> (172.18.0.5:39618 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:54:08.768810+00:00 [warning] <0.1570.0> closing AMQP connection <0.1570.0> (172.18.0.5:35432 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:54:08.768810+00:00 [warning] <0.1570.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:54:09.077962+00:00 [info] <0.1641.0> accepting AMQP connection <0.1641.0> (172.18.0.5:39634 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:54:09.081924+00:00 [info] <0.1641.0> connection <0.1641.0> (172.18.0.5:39634 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:54:09.752137+00:00 [warning] <0.1538.0> closing AMQP connection <0.1538.0> (172.18.0.5:53720 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:54:09.752137+00:00 [warning] <0.1538.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:54:10.058876+00:00 [info] <0.1657.0> accepting AMQP connection <0.1657.0> (172.18.0.5:39646 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:54:10.064280+00:00 [info] <0.1657.0> connection <0.1657.0> (172.18.0.5:39646 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:54:10.343746+00:00 [warning] <0.1586.0> closing AMQP connection <0.1586.0> (172.18.0.5:34504 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:54:10.343746+00:00 [warning] <0.1586.0> client unexpectedly closed TCP connection
rabbitmq-1    | 2024-05-02 11:54:10.688279+00:00 [info] <0.1673.0> accepting AMQP connection <0.1673.0> (172.18.0.5:39656 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 11:54:10.692997+00:00 [info] <0.1673.0> connection <0.1673.0> (172.18.0.5:39656 -> 172.18.0.4:5672): user 'mayan' authenticated and granted access to vhost 'mayan'
rabbitmq-1    | 2024-05-02 11:54:11.404196+00:00 [warning] <0.1554.0> closing AMQP connection <0.1554.0> (172.18.0.5:35426 -> 172.18.0.4:5672, vhost: 'mayan', user: 'mayan'):
rabbitmq-1    | 2024-05-02 11:54:11.404196+00:00 [warning] <0.1554.0> client unexpectedly closed TCP connection

Those errors only occur whilst the CPU is busy. For example when the system does OCR.

I used your documentation: https://docs.mayan-edms.com/chapters/docker/install_docker_compose.html#docker-compose-install

No further changes in any files where made.

Both IPs 172.18.0.5 and 172.18.0.3 are answering to pings

9

After further testing there appears also a broken pipe error.

app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:42,430: ERROR/ForkPoolWorker-7] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:42,464: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
rabbitmq-1    | 2024-05-02 13:07:43.819991+00:00 [warning] <0.3325.0> closing AMQP connection <0.3325.0> (172.18.0.3:39906 -> 172.18.0.4:5672, vhost: 'mayan_rabbitmq_vhost', user: 'mayan_rabbitmq_user'):
rabbitmq-1    | 2024-05-02 13:07:43.819991+00:00 [warning] <0.3325.0> client unexpectedly closed TCP connection
app-1         | [2024-05-02 13:07:44,359: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:45,372: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:45,519: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:45,619: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:45,752: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
rabbitmq-1    | 2024-05-02 13:07:46.851853+00:00 [warning] <0.3373.0> closing AMQP connection <0.3373.0> (172.18.0.3:60462 -> 172.18.0.4:5672, vhost: 'mayan_rabbitmq_vhost', user: 'mayan_rabbitmq_user'):
rabbitmq-1    | 2024-05-02 13:07:46.851853+00:00 [warning] <0.3373.0> client unexpectedly closed TCP connection
app-1         | [2024-05-02 13:07:47,038: ERROR/ForkPoolWorker-7] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:47,453: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
rabbitmq-1    | 2024-05-02 13:07:47.993923+00:00 [info] <0.3462.0> accepting AMQP connection <0.3462.0> (172.18.0.3:56540 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 13:07:47.996885+00:00 [info] <0.3462.0> connection <0.3462.0> (172.18.0.3:56540 -> 172.18.0.4:5672): user 'mayan_rabbitmq_user' authenticated and granted access to vhost 'mayan_rabbitmq_vhost'
rabbitmq-1    | 2024-05-02 13:07:48.022912+00:00 [info] <0.3474.0> accepting AMQP connection <0.3474.0> (172.18.0.3:56550 -> 172.18.0.4:5672)
rabbitmq-1    | 2024-05-02 13:07:48.030323+00:00 [info] <0.3474.0> connection <0.3474.0> (172.18.0.3:56550 -> 172.18.0.4:5672): user 'mayan_rabbitmq_user' authenticated and granted access to vhost 'mayan_rabbitmq_vhost'
rabbitmq-1    | 2024-05-02 13:07:48.158938+00:00 [warning] <0.3357.0> closing AMQP connection <0.3357.0> (172.18.0.3:39926 -> 172.18.0.4:5672, vhost: 'mayan_rabbitmq_vhost', user: 'mayan_rabbitmq_user'):
rabbitmq-1    | 2024-05-02 13:07:48.158938+00:00 [warning] <0.3357.0> client unexpectedly closed TCP connection
app-1         | [2024-05-02 13:07:48,394: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:48,670: ERROR/ForkPoolWorker-9] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:48,720: ERROR/ForkPoolWorker-7] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:48,764: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:49,041: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)
app-1         | BrokenPipeError: [Errno 32] Broken pipe
app-1         | [2024-05-02 13:07:50,400: ERROR/ForkPoolWorker-8] Error sending data
app-1         | Traceback (most recent call last):
app-1         |   File "/opt/mayan-edms/lib/python3.11/site-packages/gnupg.py", line 146, in _copy_data
app-1         |     outstream.write(data)