S3 Object Storage

afreeman28 · Post by **afreeman28** » Fri Jan 10, 2020 3:53 am

Hi All,

I'm looking for some assistance with utilizing object storage instead of the filesystem storage. I followed the guide outlined here: https://docs.mayan-edms.com/chapters/object_storage.htm, but after I do so my docker container will not start.

I am using the 3.3.7 docker container on Ubuntu 18.04.1 LTS. Below is some the logs from my docker container upon startup. Any Ideas?

Code: Select all

Traceback (most recent call last):,
  File "/opt/mayan-edms/bin/mayan-edms.py", line 10, in <module>,
    execute_from_command_line(sys.argv),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line,
    utility.execute(),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/core/management/__init__.py", line 338, in execute,
    django.setup(),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/__init__.py", line 27, in setup,
    apps.populate(settings.INSTALLED_APPS),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/apps/registry.py", line 108, in populate,
    app_config.import_models(),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/apps/config.py", line 202, in import_models,
    self.models_module = import_module(models_module_name),
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module,
    return _bootstrap._gcd_import(name[level:], package, level),
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import,
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load,
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked,
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked,
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module,
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/cabinets/models.py", line 13, in <module>,
    from mayan.apps.documents.models import Document,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/__init__.py", line 2, in <module>,
    from .document_page_models import *  # NOQA,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_page_models.py", line 29, in <module>,
    from .document_version_models import DocumentVersion,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_version_models.py", line 28, in <module>,
    from ..storages import storage_documentversion,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/storages.py", line 13, in <module>,
    )(**setting_storage_backend_arguments.value),
TypeError: type object argument after ** must be a mapping, not str,
mayan: starting entrypoint.sh,
mayan: update_uid_gid(),
mayan: os_package_installs(),
mayan: pip_installs(),
The directory '/home/mayan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,
The directory '/home/mayan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,
Requirement already satisfied: django-storages in /opt/mayan-edms/lib/python3.7/site-packages (1.8),
Requirement already satisfied: boto3 in /opt/mayan-edms/lib/python3.7/site-packages (1.10.50),
Requirement already satisfied: Django>=1.11 in /opt/mayan-edms/lib/python3.7/site-packages (from django-storages) (1.11.27),
Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /opt/mayan-edms/lib/python3.7/site-packages (from boto3) (0.2.1),
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/mayan-edms/lib/python3.7/site-packages (from boto3) (0.9.4),
Requirement already satisfied: botocore<1.14.0,>=1.13.50 in /opt/mayan-edms/lib/python3.7/site-packages (from boto3) (1.13.50),
Requirement already satisfied: pytz in /opt/mayan-edms/lib/python3.7/site-packages (from Django>=1.11->django-storages) (2019.1),
Requirement already satisfied: docutils<0.16,>=0.10 in /opt/mayan-edms/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.50->boto3) (0.15.2),
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /opt/mayan-edms/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.50->boto3) (2.8.0),
Requirement already satisfied: urllib3<1.26,>=1.20; python_version >= "3.4" in /opt/mayan-edms/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.50->boto3) (1.24.3),
Requirement already satisfied: six>=1.5 in /opt/mayan-edms/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.14.0,>=1.13.50->boto3) (1.13.0),
mayan: performupgrade(),
Traceback (most recent call last):,
  File "/opt/mayan-edms/bin/mayan-edms.py", line 10, in <module>,
    execute_from_command_line(sys.argv),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line,
    utility.execute(),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/core/management/__init__.py", line 338, in execute,
    django.setup(),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/__init__.py", line 27, in setup,
    apps.populate(settings.INSTALLED_APPS),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/apps/registry.py", line 108, in populate,
    app_config.import_models(),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/apps/config.py", line 202, in import_models,
    self.models_module = import_module(models_module_name),
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module,
    return _bootstrap._gcd_import(name[level:], package, level),
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import,
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load,
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked,
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked,
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module,
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/cabinets/models.py", line 13, in <module>,
    from mayan.apps.documents.models import Document,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/__init__.py", line 2, in <module>,
    from .document_page_models import *  # NOQA,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_page_models.py", line 29, in <module>,
    from .document_version_models import DocumentVersion,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_version_models.py", line 28, in <module>,
    from ..storages import storage_documentversion,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/storages.py", line 13, in <module>,
    )(**setting_storage_backend_arguments.value),
TypeError: type object argument after ** must be a mapping, not str,
mayan: starting entrypoint.sh,
mayan: update_uid_gid(),
mayan: os_package_installs(),
mayan: pip_installs(),
The directory '/home/mayan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,
The directory '/home/mayan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,
Requirement already satisfied: django-storages in /opt/mayan-edms/lib/python3.7/site-packages (1.8),
Requirement already satisfied: boto3 in /opt/mayan-edms/lib/python3.7/site-packages (1.10.50),
Requirement already satisfied: Django>=1.11 in /opt/mayan-edms/lib/python3.7/site-packages (from django-storages) (1.11.27),
Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /opt/mayan-edms/lib/python3.7/site-packages (from boto3) (0.2.1),
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/mayan-edms/lib/python3.7/site-packages (from boto3) (0.9.4),
Requirement already satisfied: botocore<1.14.0,>=1.13.50 in /opt/mayan-edms/lib/python3.7/site-packages (from boto3) (1.13.50),
Requirement already satisfied: pytz in /opt/mayan-edms/lib/python3.7/site-packages (from Django>=1.11->django-storages) (2019.1),
Requirement already satisfied: docutils<0.16,>=0.10 in /opt/mayan-edms/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.50->boto3) (0.15.2),
Requirement already satisfied: urllib3<1.26,>=1.20; python_version >= "3.4" in /opt/mayan-edms/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.50->boto3) (1.24.3),
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /opt/mayan-edms/lib/python3.7/site-packages (from botocore<1.14.0,>=1.13.50->boto3) (2.8.0),
Requirement already satisfied: six>=1.5 in /opt/mayan-edms/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.14.0,>=1.13.50->boto3) (1.13.0),
mayan: performupgrade(),

rssfed23 · Post by **rssfed23** » Fri Jan 10, 2020 9:49 am

Hi Afreeman28.

That error looks identical to the one experienced at https://gitlab.com/mayan-edms/mayan-edms/issues/690

It's caused by an incorrect value in config.yml.

How have you configured S3 storage? Can you provide your config file please? (remove the access/secret key before posting) - This may be a case of quotes not being in the right place in the new config file format.

afreeman28 · Post by **afreeman28** » Fri Jan 10, 2020 5:41 pm

I'm not sure where the .yaml file is located in the docker container. As the original documentation instructed, I just made the changes through the Web GUI.

: Screen Shot 2020-01-10 at 11.36.20 AM.png (38.39 KiB) Viewed 1098 times

The attachment Screen Shot 2020-01-10 at 11.36.20 AM.png is no longer available

rssfed23 · Post by **rssfed23** » Fri Jan 10, 2020 6:48 pm

The config file will be stored wherever you mounted the MAYAN_MEDIA directory to. In the documentation that's /docker-volumes/mayan-edms/media/config.yml. When you change something in the gui it just updates that file (which is why Mayan needs to be restarted afterwards).
It can also be set as an environment variable as per viewtopic.php?f=9&t=478&p=1076

I'm assuming you also set the DOCUMENTS_STORAGE_BACKEND_ARGUMENTS argument so Mayan knows the access key/secret key etc.

The documentation page https://docs.mayan-edms.com/topics/stor ... ct-storage isn't referred to anymore as the instructions are missing a couple of options and users had lots of problems.

The BACKEND_ARGUMENTS section in the documentation suggests:

Code: Select all

'{access_key: <your S3 access key>, secret_key: <your S3 secret key>, bucket_name: <S3 bucket name>}'

You may also need to add in "region_name: eu-west-1" (or whatever region you want) also.

The error you're getting does indicate a missing value somewhere or that a value is wrong (the error message means "this field must map to a known value rather than being an unknown string).

Someone made a post about this over at viewtopic.php?f=9&t=1174&p=2188&hilit=aws+s3#p2188 although that's for s3-compatible services like minion rather than AWS S3. But I'm fairly sure region name needs to be there.
https://django-storages.readthedocs.io/ ... on-S3.html has the upstream documentation which may also be some assistance

afreeman28 · Post by **afreeman28** » Fri Jan 10, 2020 7:20 pm

Thanks for the info. Yes I have the BACKEND_Arguments set as well, for some reason the screenshot didnt upload correctly in the forum. Here is the contents of my config.yaml file. I tried adding in the AWS location as well, but still no luck. Not sure what formatting I might be missing.

Code: Select all

ALLOWED_HOSTS:
- '*'
APPEARANCE_MAXIMUM_TITLE_LENGTH: 120
APPEARANCE_MESSAGE_POSITION: top-right
APPEND_SLASH: true
AUTHENTICATION_LOGIN_METHOD: username
AUTHENTICATION_MAXIMUM_SESSION_LENGTH: 2592000
AUTH_PASSWORD_VALIDATORS:
- NAME: django.contrib.auth.password_validation.UserAttributeSimilarityValidator
- NAME: django.contrib.auth.password_validation.MinimumLengthValidator
- NAME: django.contrib.auth.password_validation.CommonPasswordValidator
- NAME: django.contrib.auth.password_validation.NumericPasswordValidator
AUTOADMIN_EMAIL: autoadmin@example.com
AUTOADMIN_PASSWORD: null
AUTOADMIN_USERNAME: admin
CELERY_BROKER_URL: redis://:mayanredispassword@172.17.0.1:6379/0
CELERY_RESULT_BACKEND: redis://:mayanredispassword@ 172.17.0.1:6379/1
COMMON_AUTO_LOGGING: true
COMMON_DB_SYNC_TASK_DELAY: 2
COMMON_DISABLED_APPS: []
COMMON_EXTRA_APPS: []
COMMON_HOME_VIEW: common:home
COMMON_PAGINATE_BY: 40
COMMON_PRODUCTION_ERROR_LOGGING: false
COMMON_PRODUCTION_ERROR_LOG_PATH: /var/lib/mayan/error.log
COMMON_PROJECT_TITLE: Mayan EDMS
COMMON_PROJECT_URL: https://www.mayan-edms.com
COMMON_SHARED_STORAGE: django.core.files.storage.FileSystemStorage
COMMON_SHARED_STORAGE_ARGUMENTS:
  location: /var/lib/mayan/shared_files
CONVERTER_GRAPHICS_BACKEND: mayan.apps.converter.backends.python.Python
CONVERTER_GRAPHICS_BACKEND_ARGUMENTS:
  libreoffice_path: /usr/bin/libreoffice
  pdfinfo_path: /usr/bin/pdfinfo
  pdftoppm_dpi: 300
  pdftoppm_format: jpeg
  pdftoppm_path: /usr/bin/pdftoppm
  pillow_format: JPEG
DATABASES:
  default:
    ENGINE: django.db.backends.postgresql
    HOST: 172.17.0.1
    NAME: mayan
    PASSWORD: mayanuserpass
    USER: mayan
DATA_UPLOAD_MAX_MEMORY_SIZE: 2621440
DEFAULT_FROM_EMAIL: webmaster@localhost
DISALLOWED_USER_AGENTS: []
DOCUMENTS_CACHE_MAXIMUM_SIZE: 524288000
DOCUMENTS_CACHE_STORAGE_BACKEND: django.core.files.storage.FileSystemStorage
DOCUMENTS_CACHE_STORAGE_BACKEND_ARGUMENTS:
  location: /var/lib/mayan/document_cache
DOCUMENTS_DISABLE_BASE_IMAGE_CACHE: false
DOCUMENTS_DISABLE_TRANSFORMED_IMAGE_CACHE: false
DOCUMENTS_DISPLAY_HEIGHT: ''
DOCUMENTS_DISPLAY_WIDTH: '3600'
DOCUMENTS_FAVORITE_COUNT: 400
DOCUMENTS_FIX_ORIENTATION: false
DOCUMENTS_HASH_BLOCK_SIZE: 65535
DOCUMENTS_LANGUAGE: eng
DOCUMENTS_LANGUAGE_CODES:
- ilo
- run
- uig
- hin
- pan
- pnb
- wuu
- msa
- kxd
- ind
- zsm
- jax
- meo
- kvr
- xmm
- min
- mui
- zmi
- max
- mfa
- cjy
- nan
- pus
- pbu
- pbt
- wne
- hsn
- hak
- ful
- fuc
- fuf
- ffm
- fue
- fuh
- fuq
- fuv
- fub
- fui
- nep
- npi
- dty
- sin
- khm
- kxm
- ell
- grc
- cpg
- gmy
- pnt
- tsd
- yej
- nya
- mnp
- dhd
- cdo
- hil
- bcc
- bgn
- bgp
- cmn
- kok
- spa
- eng
- ara
- por
- ben
- rus
- jpn
- deu
- jav
- tel
- vie
- kor
- fra
- mar
- tam
- urd
- tur
- ita
- yue
- tha
- guj
- fas
- pol
- kan
- mal
- sun
- hau
- ory
- mya
- ukr
- bho
- tgl
- yor
- mai
- uzb
- snd
- amh
- ron
- orm
- ibo
- aze
- awa
- gan
- ceb
- nld
- kur
- hbs
- mlg
- skr
- ctg
- zha
- tuk
- asm
- mad
- som
- mwr
- mag
- bgc
- hun
- hne
- dcc
- aka
- kaz
- syl
- zul
- ces
- kin
- hat
- que
- swe
- hmn
- sna
- mos
- xho
- bel
DOCUMENTS_PAGE_IMAGE_CACHE_TIME: '31556926'
DOCUMENTS_PREVIEW_HEIGHT: ''
DOCUMENTS_PREVIEW_WIDTH: '800'
DOCUMENTS_PRINT_HEIGHT: ''
DOCUMENTS_PRINT_WIDTH: '3600'
DOCUMENTS_RECENT_ACCESS_COUNT: 400
DOCUMENTS_RECENT_ADDED_COUNT: 400
DOCUMENTS_ROTATION_STEP: 90
DOCUMENTS_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage
DOCUMENTS_STORAGE_BACKEND_ARGUMENTS: '{access_key: xxxxxxxxxxxxxxxxxxxxxxxxx, secret_key:
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, bucket_name: xxxxxxxxxxxxxxxxxxx,
  AWS_LOCATION: us-east-2}'
DOCUMENTS_THUMBNAIL_HEIGHT: ''
DOCUMENTS_THUMBNAIL_WIDTH: '800'
DOCUMENTS_ZOOM_MAX_LEVEL: 300
DOCUMENTS_ZOOM_MIN_LEVEL: 25
DOCUMENTS_ZOOM_PERCENT_STEP: 25
DOCUMENT_PARSING_AUTO_PARSING: true
DOCUMENT_PARSING_PDFTOTEXT_PATH: /usr/bin/pdftotext
EMAIL_BACKEND: django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST: localhost
EMAIL_HOST_PASSWORD: ''
EMAIL_HOST_USER: ''
EMAIL_PORT: 25
EMAIL_TIMEOUT: null
EMAIL_USE_SSL: false
EMAIL_USE_TLS: false
FILE_METADATA_AUTO_PROCESS: true
FILE_METADATA_DRIVERS_ARGUMENTS:
  exif_driver:
    exiftool_path: /usr/bin/exiftool
FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440
INTERNAL_IPS:
- 127.0.0.1
LANGUAGES:
- - ar
  - Arabic
- - bg
  - Bulgarian
- - bs
  - Bosnian
- - cs
  - Czech
- - da
  - Danish
- - de
  - German
- - el
  - Greek
- - en
  - English
- - es
  - Spanish
- - fa
  - Persian
- - fr
  - French
- - hu
  - Hungarian
- - id
  - Indonesian
- - it
  - Italian
- - lv
  - Latvian
- - nl
  - Dutch
- - pl
  - Polish
- - pt
  - Portuguese
- - pt-br
  - Portuguese (Brazil)
- - ro
  - Romanian
- - ru
  - Russian
- - sl
  - Slovenian
- - tr
  - Turkish
- - vi
  - Vietnamese
- - zh
  - Chinese
LANGUAGE_CODE: en-us
LOCK_MANAGER_BACKEND: mayan.apps.lock_manager.backends.file_lock.FileLock
LOCK_MANAGER_BACKEND_ARGUMENTS: {}
LOCK_MANAGER_DEFAULT_LOCK_TIMEOUT: 30
LOGIN_REDIRECT_URL: common:home
LOGIN_URL: authentication:login_view
LOGOUT_REDIRECT_URL: authentication:login_view
MAILER_DOCUMENT_BODY_TEMPLATE: "Attached to this email is the document: {{ document\
  \ }}\n\n --------\n This email has been sent from %(project_title)s (%(project_website)s)"
MAILER_DOCUMENT_SUBJECT_TEMPLATE: 'Document: {{ document }}'
MAILER_LINK_BODY_TEMPLATE: "To access this document click on the following link: {{\
  \ link }}\n\n--------\n This email has been sent from %(project_title)s (%(project_website)s)"
MAILER_LINK_SUBJECT_TEMPLATE: 'Link for document: {{ document }}'
MAYAN_CELERY_CLASS: celery.Celery
METADATA_AVAILABLE_PARSERS:
- mayan.apps.metadata.parsers.DateAndTimeParser
- mayan.apps.metadata.parsers.DateParser
- mayan.apps.metadata.parsers.TimeParser
METADATA_AVAILABLE_VALIDATORS:
- mayan.apps.metadata.validators.DateAndTimeValidator
- mayan.apps.metadata.validators.DateValidator
- mayan.apps.metadata.validators.TimeValidator
MIRRORING_DOCUMENT_CACHE_LOOKUP_TIMEOUT: 10
MIRRORING_NODE_CACHE_LOOKUP_TIMEOUT: 10
OCR_AUTO_OCR: true
OCR_BACKEND: mayan.apps.ocr.backends.tesseract.Tesseract
OCR_BACKEND_ARGUMENTS: {}
SEARCH_DISABLE_SIMPLE_SEARCH: false
SIGNATURES_GPG_HOME: /var/lib/mayan/gpg_home
SIGNATURES_GPG_PATH: /usr/bin/gpg1
SIGNATURES_KEYSERVER: pool.sks-keyservers.net
SIGNATURES_STORAGE_BACKEND: django.core.files.storage.FileSystemStorage
SIGNATURES_STORAGE_BACKEND_ARGUMENTS:
  location: /var/lib/mayan/document_signatures
SMART_SETTINGS_NAMESPACES:
  appearance:
    version: '0001'
  authentication:
    version: '0001'
  autoadmin:
    version: '0001'
  celery:
    version: '0001'
  common:
    version: '0002'
  converter:
    version: '0002'
  django:
    version: '0001'
  django_gpg:
    version: '0001'
  document_parsing:
    version: '0001'
  document_states:
    version: '0001'
  documents:
    version: '0002'
  file_metadata:
    version: '0002'
  lock_manager:
    version: '0001'
  mailer:
    version: '0001'
  mayan:
    version: '0001'
  metadata:
    version: '0001'
  mirroring:
    version: '0001'
  ocr:
    version: '0002'
  search:
    version: '0001'
  signatures:
    version: '0002'
  sources:
    version: '0002'
  storage:
    version: '0001'
SOURCES_SCANIMAGE_PATH: /usr/bin/scanimage
SOURCES_STAGING_FILE_CACHE_STORAGE_BACKEND: django.core.files.storage.FileSystemStorage
SOURCES_STAGING_FILE_CACHE_STORAGE_BACKEND_ARGUMENTS:
  location: /var/lib/mayan/staging_file_cache
STATICFILES_STORAGE: whitenoise.storage.CompressedManifestStaticFilesStorage
STATIC_URL: /static/
STORAGE_TEMPORARY_DIRECTORY: /tmp
TIME_ZONE: UTC
WORKFLOWS_IMAGE_CACHE_MAXIMUM_SIZE: 52428800
WORKFLOWS_IMAGE_CACHE_STORAGE_BACKEND: django.core.files.storage.FileSystemStorage
WORKFLOWS_IMAGE_CACHE_STORAGE_BACKEND_ARGUMENTS:
  location: /var/lib/mayan/workflows
WORKFLOWS_IMAGE_CACHE_TIME: '31556926'
WSGI_APPLICATION: mayan.wsgi.application

Any other ideas?

rssfed23 · Post by **rssfed23** » Fri Jan 10, 2020 7:27 pm

AWS_LOCATION should be "region_name" (lower case).

I don't know why, but the django-storage variables are different inside Mayan. We don't have a "translation guide" to what you put into mayan gets turned to in the Django-storages side. That may explain the error as mayan doesn't know what AWS_LOCATION: is but it does know what region_name: is.

Can you try with everything the same but using region_name: ?

afreeman28 · Post by **afreeman28** » Fri Jan 10, 2020 7:35 pm

Getting this error now:

Code: Select all

Error loading configuration file: /var/lib/mayan/config.yml; mapping values are not allowed here

  in "/var/lib/mayan/config.yml", line 207, column 83

line 207, column 83 is the colon after "secret_key"

rssfed23 · Post by **rssfed23** » Fri Jan 10, 2020 7:54 pm

Can you make all of the values on one line if they're not already (the extra spaces may be confusing things). I'm also doubting myself that the region_name field is actually needed so you can try removing that.

Those are the only two suggestions I've got. I'll try playing around with s3 this evening see if I can get something working and report back.

rssfed23 · Post by **rssfed23** » Fri Jan 10, 2020 8:46 pm

GOT IT

D:

That was painful, but there's a new way to define variables in 3.x.

In your docker-compose file, you need something like (including databases so you can see where it fits inside compose)

Code: Select all

      MAYAN_DATABASES: "{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayandbpass','USER':'mayan','HOST':'postgresql'}}"
      MAYAN_PIP_INSTALLS: 'django-storages boto3'
      MAYAN_DOCUMENTS_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage
      MAYAN_DOCUMENTS_STORAGE_BACKEND_ARGUMENTS: "{'access_key':'mayanKey','secret_key':'mayanSecret','bucket_name':'mayan','default_acl':'private'}"

I have no idea how that translates to the config.yml file so just set it as docker environment variables instead. Theres no more single quoting and each variable+answer has to be located inside single quotes instead. There's also : between them with no spaces.
Basically copy that whole block into an environment variable and change the key/bucket name etc. No need for the location value (although you can add that I think if you want the client to use a dedicated access point location rather than the usual s3.amazonaws... address)

You DO need the default_acl value though otherwise if you have public access to your bucket blocked amazon will give you permission denied errors for every PUT operation

afreeman28 · Post by **afreeman28** » Fri Jan 10, 2020 11:04 pm

YOU DA MAN.

Up and running again:

Code: Select all

docker run \
-d \
--name mayan-edms \
--restart=always \
-p 80:8000 \
-e MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'172.17.0.1'}}" \
-e MAYAN_CELERY_BROKER_URL="redis://:mayanredispassword@172.17.0.1:6379/0" \
-e MAYAN_CELERY_RESULT_BACKEND="redis://:mayanredispassword@172.17.0.1:6379/1" \
-e MAYAN_PIP_INSTALLS='django-storages boto3' \
-e MAYAN_DOCUMENTS_STORAGE_BACKEND=storages.backends.s3boto3.S3Boto3Storage \
-e MAYAN_DOCUMENTS_STORAGE_BACKEND_ARGUMENTS="{'access_key':'xxxxxxxxxxxxxxxxxxxx','secret_key':'xxxxxxxxxxxxxxxxxxxx','bucket_name':'edms-storage.xxxxx.com','default_acl':'private'}" \
-v /docker-volumes/mayan-edms/media:/var/lib/mayan \
mayanedms/mayanedms:3.3.7

Mayan EDMS community forum

S3 Object Storage

S3 Object Storage

Re: S3 Object Storage

Re: S3 Object Storage

Re: S3 Object Storage

Re: S3 Object Storage

Re: S3 Object Storage

Re: S3 Object Storage

Re: S3 Object Storage

Re: S3 Object Storage

Re: S3 Object Storage