Page 1 of 1

Passthrough encryption setup

Posted: Wed Sep 09, 2020 6:27 am
by drilling005
Hi, I see that passthrough-encryption is now supported, as described in this issue: https://gitlab.com/mayan-edms/mayan-edms/-/issues/143

However I'm not sure how to configure the new EncryptedPassthroughStorage. I have an existing S3Boto3Storage backend that is working but I'd like to introduce at-rest encryption for this. Is anyone able to point me in the right direction? I'd be happy to write up a new documentation page for it once I know the steps to take.

Thanks in advance.

Re: Passthrough encryption setup

Posted: Thu Sep 10, 2020 2:06 am
by drilling005
Okay, after some digging through the git repo I've figured it out. Here is my YAML configuration reference for anyone else interested. Note the all-important "next_storage_backend" and "next_storage_backend_arguments":

Code: Select all

DOCUMENTS_STORAGE_BACKEND: mayan.apps.storage.backends.encryptedstorage.EncryptedPassthroughStorage
DOCUMENTS_STORAGE_BACKEND_ARGUMENTS:
  password: "testpassword"
  next_storage_backend: storages.backends.s3boto3.S3Boto3Storage
  next_storage_backend_arguments:
    access_key: AKIA0000000000000000
    secret_key: trustno1
    bucket_name: my-mayan
    location: subfolder
However I found that both the ZipCompressedPassthroughStorage and EncryptedPassthroughStorage give me this error in the docker container log output:

Code: Select all

mayanedms      | mayan.apps.documents.models.document_version_models <71> [ERROR] "save() line 390 Error creating new document version for document "sample.pdf"; 'BufferedZipFile' object has no attribute 'file'"
mayanedms      | [2020-09-10 01:44:22,002: ERROR/ForkPoolWorker-9] Error creating new document version for document "sample.pdf"; 'BufferedZipFile' object has no attribute 'file'
mayanedms      | mayan.apps.sources.models.base <71> [CRITICAL] "upload_document() line 141 Unexpected exception while trying to create version for new document "sample.pdf" from source "Upload file"; 'BufferedZipFile' object has no attribute 'file'"
mayanedms      | Traceback (most recent call last):
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/sources/models/base.py", line 127, in upload_document
mayanedms      |     file_object=file_object, _user=user,
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_models.py", line 218, in new_version
mayanedms      |     document_version.save(_user=_user)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_version_models.py", line 373, in save
mayanedms      |     self.update_page_count(save=False)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_version_models.py", line 471, in update_page_count
mayanedms      |     file_object=file_object, mime_type=self.mimetype
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/converter/classes.py", line 47, in __init__
mayanedms      |     file_object=file_object, mimetype_only=False
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/mimetype/api.py", line 17, in get_mimetype
mayanedms      |     file_object.seek(0)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/django/core/files/utils.py", line 20, in <lambda>
mayanedms      |     seek = property(lambda self: self.file.seek)
mayanedms      | AttributeError: 'BufferedZipFile' object has no attribute 'file'
mayanedms      | [2020-09-10 01:44:22,003: CRITICAL/ForkPoolWorker-9] Unexpected exception while trying to create version for new document "sample.pdf" from source "Upload file"; 'BufferedZipFile' object has no attribute 'file'
mayanedms      | Traceback (most recent call last):
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/sources/models/base.py", line 127, in upload_document
mayanedms      |     file_object=file_object, _user=user,
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_models.py", line 218, in new_version
mayanedms      |     document_version.save(_user=_user)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_version_models.py", line 373, in save
mayanedms      |     self.update_page_count(save=False)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_version_models.py", line 471, in update_page_count
mayanedms      |     file_object=file_object, mime_type=self.mimetype
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/converter/classes.py", line 47, in __init__
mayanedms      |     file_object=file_object, mimetype_only=False
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/mimetype/api.py", line 17, in get_mimetype
mayanedms      |     file_object.seek(0)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/django/core/files/utils.py", line 20, in <lambda>
mayanedms      |     seek = property(lambda self: self.file.seek)
mayanedms      | AttributeError: 'BufferedZipFile' object has no attribute 'file'
mayanedms      | [2020-09-10 01:44:22,132: ERROR/ForkPoolWorker-9] Task mayan.apps.sources.tasks.task_upload_document[a800f082-0771-46e2-8a54-83c568a38a86] raised unexpected: AttributeError("'BufferedZipFile' object has no attribute 'file'")
mayanedms      | Traceback (most recent call last):
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
mayanedms      |     R = retval = fun(*args, **kwargs)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 648, in __protected_call__
mayanedms      |     return self.run(*args, **kwargs)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/sources/tasks.py", line 195, in task_upload_document
mayanedms      |     querystring=querystring, user=user,
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/sources/models/base.py", line 127, in upload_document
mayanedms      |     file_object=file_object, _user=user,
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_models.py", line 218, in new_version
mayanedms      |     document_version.save(_user=_user)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_version_models.py", line 373, in save
mayanedms      |     self.update_page_count(save=False)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/documents/models/document_version_models.py", line 471, in update_page_count
mayanedms      |     file_object=file_object, mime_type=self.mimetype
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/converter/classes.py", line 47, in __init__
mayanedms      |     file_object=file_object, mimetype_only=False
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/mimetype/api.py", line 17, in get_mimetype
mayanedms      |     file_object.seek(0)
mayanedms      |   File "/opt/mayan-edms/lib/python3.7/site-packages/django/core/files/utils.py", line 20, in <lambda>
mayanedms      |     seek = property(lambda self: self.file.seek)
mayanedms      | AttributeError: 'BufferedZipFile' object has no attribute 'file'
I don't know if this error is specific to my use of S3 as the end data store, or if it's a general problem with the passthrough storage layer.

Re: Passthrough encryption setup

Posted: Thu Sep 10, 2020 6:41 am
by michael
Thanks for the report and the debug information. I've queued this for investigation and will open a ticket to track it. Thanks!

Re: Passthrough encryption setup

Posted: Mon Sep 14, 2020 11:45 am
by drilling005
Thanks for your reply Michael, and confirming it wasn't just a silly error on my part. Is there a Gitlab issue I can watch so I know when this is resolved, or will you/a dev reply in this thread?

Re: Passthrough encryption setup

Posted: Mon Sep 21, 2020 4:05 am
by rosarior
Here is the link to the ticket on GitLab: https://gitlab.com/mayan-edms/mayan-edms/-/issues/876

Thanks for the report!