I have a homelab/small personal instance and am currently entertaining the idea to move the slower workers to a beefier VM.
What is the best approach, my idea is basically
Small VM: Gunicorn, Worker fast (a,b)
Big VM: worker slow (b,c,d).
I tried to (naively) create two identical instances sharing the media folder and connecting to the same postgres+rabbitMQ. On each machine, I have tweaked the supervisor conf file to only contain the workers I need.
It appears to work… but is this the right approach?
Mayan is a distributed architecture by default so each worker doesn’t know or care about the actual physical location. The important part is making sure all copies are connected to the same the database, Redis, RabbitMQ and the storage. This is actually how we scale it up to handle millions of documents, and how it is deployed by default in Kubernetes.
The second part is to not use the direct deployment, it is deprecated and will be marked obsolete in the next version. Use the Docker Compose file. You can launch only the selected workers by setting the value of COMPOSE_PROFILES to extra_worker_a or extra_worker_b, etc. The value of COMPOSE_PROFILES can be a single profile or a comma separated list of profiles.
Thanks for confirming, I guess that’s why Mayan needs the transactional locking offered by Redis.
The second part is to not use the direct deployment
Heard loud and clear… I have taken a look at the default compose.
It looks like I need to build my own compose—and likely mayan image—to suit my purpose. I have small apps installed, and my postgres is in a separate VM, etc etc.
It looks like I need to build my own compose—and likely mayan image—to suit my purpose. I have small apps installed, and my postgres is in a separate VM, etc etc.
The Makefile has a lot of targets to help you out.
To build packages and Docker images, the Makefile has GitOps targets. For example to build a test Docker image you just do:
make gitlab-ci-builds-docker
For continuous delivery and testing just do:
make gitlab-ci-deployments-staging
This takes care of everything needed test a build (SSH to the VM, install Docker, pull the test image, install and initialize Mayan).
For PostgreSQL in separate a VM just remove postgresql from COMPOSE_PROFILES and it won’t be launched together with Mayan. Same with RabbitMQ and Redis.
Our Docker Compose files leverages profiles (Using profiles with Compose) to allow a single file to work in multiple scenarios and deployment configurations.
