Page 1 of 1

Mayan on Kubernetes - Experiences?

Posted: Thu Jan 02, 2020 8:14 pm
by rssfed23
Hey Mayan users!

One of the items I'm going to be working on soon in the background is Kubernetes deployments for Mayan (documentation, fixes, helm charts, etc). To be clear and set expectations this isn't something on the roadmap but a community contribution that I think would be nice to have.
I've seen a couple of people mention K8s in the past on here and gitlab issues so I'm putting wondering if you're able to share your experiences with us.

If you're in a position to share deployment yaml or any tooling you've used to deploy Mayan to k8s then please do share (if you aren't able to publicly then feel free to log a confidential issue on gitlab)

Hearing your experiences of using Mayan on k8s (what works well, what doesn't work, etc) will greatly speed up the process.

My initial plan is to produce deployment spec yamls and then hopefully helm charts. A longer term view would be a native Mayan K8s Operator but that is definitely way down the line.

I'm aware of the locking issues with NFS and the preparestatic generation (and the fact it runs on every startup) but any other feedback or experiences will be truly welcome to help shape things.

A couple of other initial questions:
- If producing a helm chart, should our chart contain all services required (Mayan, Redis, RabbitMQ, Flower, Postgres, etc) which can be enabled/disable through the values.yaml or should it be only Mayan with documentation on how to use the upstream charts for those other services?
- Are you using operators for postgres, redis & rabbitmq? - This is related to the first question in a way because I wouldn't want to go through the work of creating deployments of those services or documentation for them if it's commonplace that people are using the operator for that technology already

Many thanks!
Rob

Re: Mayan on Kubernetes - Experiences?

Posted: Fri Mar 06, 2020 8:06 pm
by jcollie
I've been running Mayan in my Kubernetes cluster for a couple of weeks now and it's working well so far (although I wish that I had more resources to dedicate to it). This is just a small home/personal deployment so YMMV. Right now I just have everything running in a single pod although I'd like to break the pod up into several pods. One main Mayan/gunicorn pod serving up the web interface and then a number of pods running the workers. Then using autoscalers it should be possible to dynamically add workers as needed.

I've put sanitized versions of my yaml files up here:

https://gist.github.com/jcollie/bb27ae1 ... 1f19b4124e

Based on my experience so far I'd start out by only deploying Mayan and Redis in the Helm chart. Let the user install other things like RabbitMQ or PostgreSQL. Then maybe build up to optionally deploying those other parts.

Re: Mayan on Kubernetes - Experiences?

Posted: Tue Mar 10, 2020 8:55 pm
by rssfed23
Jcollie, I'm no longer working personally on this topic at all, but your contribution there is much appreciated! - It is indeed quite close to what I would have gone for myself, and you're right, one deployment for gunicorn (I did get it working with Passenger also) + one deployment for each worker type (fast/medium/slow) is the most ideal setup.
Ultimately, I'd go as far as to say one deployment for each worker queue would be most ideal for an enterprise grade deployment. We did have such a thing architected out and drawn to fit, but as of yet not implemented.

One of the larger problems to solve is storage. I can see in your provided yaml you're using hostpath storage, which of course wouldn't be suitable for most. The biggest issue I see is a lot of people expect these days for us to support dynamic/StorageClass, but that's impossible to do if a user wants to delete the deployment and then redeploy pointing to an existing setup (well without modifying the helm chart during install time anyway). My initial recommendation to the team was to use manual PV creation with documentation on how to do it, and we can worry about StorageClasses later on.

As far as I'm concerned, the Mayan team should only really be concerned about deploying Mayan. Redis, RabbitMQ (a must I would say for enterprise k8s support) and PostgreSQL should all be provisioned and managed via operators. I did get a POC working with the RedisFailover operator and one of the Postgres operators as well and the whole setup worked really quite well! So thanks for sharing your Yaml, I'm sure someone here will find it very useful, in particular because you're using all the right k8s patterns with ConfigMaps, Ingresses and Secrets! - Great job :)