Upgrade from 4.6.4 to 4.7 failing during perform_upgrade()

jeli0001 · May 19, 2024, 5:39pm

Hello everyone,

Just tried to upgrade from 4.6.4 to 4.7 as per the instructions for a major upgrade in the Mayan documentation. As with previous major upgrades the PostgreSQL data base could be dumped by the outgoing 13.14 server and reimported by the new PostgreSQL 14.11 server.

When the complete stack is started, the following things happen according to the docker log:

mayan starts with entrypoint.sh, connects to postgresql, rabbitmq and redis, runs update_uid_gid, os_package_installs, pip_installs,
When perform_upgrade() is called, a series of exceptions is thrown. Reading the logs backward, the following happens:

mayan.apps.common.exceptions.BaseCommonException: Error during signal_pre_upgrade signal.
django.db.utils.OperationalError: connection failed: FATAL: password authentication failed for user "mayan"
psycopg.OperationalError: connection failed: FATAL: password authentication failed for user "mayan"

The whole procedure then restarts at 1. and loops infinitely until the whole stack is stopped manually. Looking at the outputs of docker ps, the app container seems to crash after ~20 secs and then restarts.

Data base user and password are set correctly in the .env file.

Just wondering if anyone else encountered this problem and has an idea how to solve it.

Here’s a complete output of the exceptions logged by the app container:

mayan: perform_upgrade()
Traceback (most recent call last):
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/psycopg/connection.py", line 745, in connect
    raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: FATAL:  password authentication failed for user "mayan"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/mayan-edms/lib/python3.11/site-packages/mayan/apps/common/management/base.py", line 106, in do_perform_upgrade
    signal_pre_upgrade.send(sender=self)
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
    return [
           ^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/mayan/apps/common/handlers.py", line 13, in handler_pre_upgrade
    management.call_command(
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/migrations/loader.py", line 235, in build_graph
    self.applied_migrations = recorder.applied_migrations()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/migrations/recorder.py", line 81, in applied_migrations
    if self.has_table():
       ^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/migrations/recorder.py", line 57, in has_table
    with self.connection.cursor() as cursor:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
           ^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
    with self.wrap_database_errors:
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/psycopg/connection.py", line 745, in connect
    raise last_ex.with_traceback(None)
django.db.utils.OperationalError: connection failed: FATAL:  password authentication failed for user "mayan"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/mayan-edms/bin/mayan-edms.py", line 22, in <module>
    main()
  File "/opt/mayan-edms/bin/mayan-edms.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/mayan-edms/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/mayan-edms/lib/python3.11/site-packages/mayan/apps/common/management/commands/common_perform_upgrade.py", line 18, in handle
    instance.do_perform_upgrade(
  File "/opt/mayan-edms/lib/python3.11/site-packages/mayan/apps/common/management/base.py", line 108, in do_perform_upgrade
    raise BaseCommonException(
mayan.apps.common.exceptions.BaseCommonException: Error during signal_pre_upgrade signal.
mayan: starting entrypoint.sh

mattwebboz · May 21, 2024, 10:13pm

Hi,
I had similar logs when I upgraded (password authentication failed for user “mayan”). My usernames and passwords were also correct. Though, my logs also contained the phase that the user/password “does not have a valid SCRAM secret”.

I fixed it by logging into the Postgres container and changing the pg_hba.conf file to allow MD5 authentication:

Log into Postgres container: docker exec -it mayan-postgresql-1 /bin/bash
Open the pg_hba.conf file: vi /var/lib/postgresql/data/pg_hba.conf
Look for the host line and change it to this: host all all all trust
Restart services (or reboot)

For production/commercial environments, I would recommend you investigate this “fix” before applying it, as changing this setting may lower the security of the infrastructure. I run my services in a homelab and not a commercial environment, but I thought I’d post this if nothing else if it helps track down the issue.

Cheers.

jeli0001 · May 22, 2024, 4:18pm

This perfectly solves the problem. The culprit is indeed the pg_hba.conf file in the mayan-postgresql container. The new postgresql 14.11 image ships with a default line host all all all scram-sha-256 while the setting in my outgoing postgres 13.14 server was host all all all md5. After changing scram-sha-256 back to md5 in the new container, the Mayan stack starts without error.

Thanks a lot for the suggestion.

Lessons learned:
bart_postgres

mattwebboz · May 23, 2024, 11:32am

No worries - glad it helped. And thanks for pointing out the setting to “MD5” and not “all”. I’ll go in an update mine.

DrRSatzteil · May 28, 2024, 10:09pm

Found this in the Postgres documentation:

“To upgrade an existing installation from md5 to scram-sha-256 , after having ensured that all client libraries in use are new enough to support SCRAM, set password_encryption = 'scram-sha-256' in postgresql.conf , make all users set new passwords, and change the authentication method specifications in pg_hba.conf to scram-sha-256 .”

So this should be the correct approach to really fix this issue. Will try that in the next couple of days. Thanks for the quick fix for now!

I updated my installation today without issues:

Shutdown the entire container stack
Spin up just the postgresql container
Change postgresql.conf file (password_encryption setting)
Logged in psql and changed mayan password
Changed pg_hba.conf (auth method to scram-sha-256)
Change password in mayan .env file
Spin up the rest of the container stack

Done!

vintager · June 19, 2024, 6:03pm

No no no folks, don’t change the container. Change your database dump script

You’re correct, of course, the main app can’t connect to the DB because it expects the password to be SCRAM-SHA-256-encoded.

Do a fresh install, set your DB username and password as usual.
Start all the containers, wait for the main app to set up the DB.
Do a database dump.
Open the .sql file and copy the following line:

ALTER ROLE mayan_db_user WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN REPLICATION BYPASSRLS PASSWORD ‘SCRAM-SHA-256 mayan_db_password_hash’;

Replace the corresponding line in your backup script with it.
Delete everything, do another fresh install.
Start the DB container.
Import the sql script.
Copy over the media/ folder.
Start the rest of the containers.

mattwebboz · June 25, 2024, 8:28pm

This is great! Thanks so much!!