Is it possible to use ONLY the object storage instead of the local file system

Questions
1

We are setting up a Mayan-EDMS system on Kubernetes using Helm charts. We want to only use the S3 object storage we have running on our network (Minio) but for some reason a volume is still being mounted on /var/lib/mayan.

We have set the documentsFileStorage, documentsFilePageImageCacheStorage and the
documentsVersionPageImageCacheStorage to type: objectExternal which to our knowledge should prevent anything going to the local filesystem.

Is it possible to prevent the volume mount from happening or is this something that Mayan-EDMS defaults to even when using an S3 object storage solution?

Thanks in advance!

kind regards,

Hakiem

2

Yes, it is possible but more work is required. There are other apps that use storage that also need to be configured for block storage via their own environment variables which are not exposed by the Helm Chart:

  • Workflow image cache storage
  • Source cache storage
  • Signature capture cache storage (signature capture)
  • Signature backend (GPG signatures)
  • Converter asset storage
  • Converter asset cache storage

After that set COMMON_DISABLE_LOCAL_STORAGE to False to disable any attempt to create file content. If COMMON_DISABLE_LOCAL_STORAGE is disabled, then the SECRET KEY must also be provided via the environment variable MAYAN_SECRET_KEY. You can get a random valid one using the command common_generate_random_secret_key.

https://docs.mayan-edms.com/releases/4.3.html#storage

Support was added for disabling use of the media folder. This is controlled with the new setting COMMON_DISABLE_LOCAL_STORAGE. Changing this setting to True will disable use of the local media folder. When using this setting, all apps must be also configured via their respective storage backend settings to use alternate persistence methods. This change is important for Kubernetes deployments on providers that do not support multiple read and write volumes such as Google Kubernetes Engine (GKE).

3

Accidentally stumbled on this thread whilst deploying on Amazon K8S platform. Storage will be on S3 as I don’t want to use a distributed filesystem. So if I read it correctly:

set COMMON_DISABLE_LOCAL_STORAGE to True

Then ensure that all the other apps also use the S3 storage: so for example https://github.com/mayan-edms/Mayan-EDMS/blob/master/mayan/apps/signature_captures/settings.py has a setting_signature_capture_cache_storage_backend_arguments that should probably point to the storage solution. If these are not exposed via the helm chart: what would be the best way to override these values from the environment? Or should I fall back to adding YAML or python settings files?

4

we added these params:

  MAYAN_COMMON_DISABLE_LOCAL_STORAGE: True
  MAYAN_CONVERTER_ASSET_CACHE_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage
  MAYAN_CONVERTER_ASSET_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage
  MAYAN_SIGNATURES_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage
  MAYAN_DOCUMENTS_FILE_PAGE_IMAGE_CACHE_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage
  MAYAN_SOURCES_CACHE_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage
  MAYAN_STORAGE_DOWNLOAD_FILE_STORAGE: storages.backends.s3boto3.S3Boto3Storage
  MAYAN_STORAGE_SHARED_STORAGE: storages.backends.s3boto3.S3Boto3Storage
  MAYAN_WORKFLOWS_IMAGE_CACHE_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage

apart from this we added these params to the _secrets.tpl:

MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS: "{'endpoint_url':'https:.....
MAYAN_CONVERTER_ASSET_CACHE_STORAGE_BACKEND_ARGUMENTS:{'endpoint_url':'https:
MAYAN_CONVERTER_ASSET_STORAGE_BACKEND_ARGUMENTS: "{'endpoint_url':'https....
MAYAN_SIGNATURES_STORAGE_BACKEND_ARGUMENTS: "{'endpoint_url':'https:....
MAYAN_SOURCES_CACHE_STORAGE_BACKEND_ARGUMENTS: "{'endpoint_url':'https://....
MAYAN_STORAGE_DOWNLOAD_FILE_STORAGE_ARGUMENTS: "{'endpoint_url':'https:...
MAYAN_STORAGE_SHARED_STORAGE_ARGUMENTS: "{'endpoint_url':'https:/.....
MAYAN_WORKFLOWS_IMAGE_CACHE_STORAGE_BACKEND_ARGUMENTS: "{'endpoint_url':....

it seems to now not use the local file system for storage

2 Likes
5

Is there anyway you can put me through in other to migrate from basic docker-compose to ks8?

6

I think it would be cool to release some updated helm charts for a S3 based storage solution. This way you can run Mayan in a k8s cluster without a distributed filesystem.