Page 1 of 1

Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Mon Jan 06, 2020 12:53 pm
by ahmed
First, many thanks to rosarior and rssfed23 who seem very active and support all users.

I read the guides on how to install Mayan EDMS. It was straight forward when I installed the docker image on a linux Ubuntu VM. But I am used to Windows not Linux. So can I run Mayan EDMS on my Windows Machine without Docker (for example by compiling the source code directly on my Windows 10 machine)? I want to customize the source code and run it directly from PyCharm or Visual Studio Code.

Also are there any advantages of deploying Mayan EDMS on Linux as compared to Windows?

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Mon Jan 06, 2020 1:00 pm
by rssfed23
Thanks for your kind words - it's much appreciated :)

At this point in time no, you can't run Mayan EDMS natively on windows.
The only option for windows users is to use docker for windows OR to install into WSL2.

This is because some of the Python packages Mayan depends on to function do not have Windows wheel (precompiled python binary) versions available and can not be compiled on Windows or have no support for windows. There's also missing Linux libraries that have no windows equivalents and cause quirks getting Mayan to run.

I will soon be trying to see if there's any way around this after I've written the MacOS deployment guide today but given the missing dependencies it is very, very unlikely to work successfully.

So for now you have to use docker for windows, a VM or WSL2. You don't need an Ubuntu VM for Docker though the native docker for windows (which manages a small Linux VM for you) works okay.

There's many benefits of using Linux to host critical services, but specifically for Mayan it relates to the dependencies all being available on Linux and performance. Of course there's security as well but that's a whole different topic and takes us into "why Linux over Windows for anything" debate.

Mayan does run on WSL2 though. So you can create a Windows Subsystem for Linux 2 Ubuntu Environment and deploy there. It is not as performant as running it inside of a virtual machine however (WSL2 is notorious for taking a performance hit on disk IO and CPU performance).
I actually do all my Mayan development/test work on a WSL2 environment (but host it in production on a standard Linux VM).

So to summarise the 3 options are (in my preferred order in terms of performance):
Docker for Windows
Ubuntu VM

I will type up a guide for 1 and 3 at some point, but we already have a great docker for windows 8 guide available at viewtopic.php?f=15&t=1536 and most of the same concepts apply (the installing docker section you can follow the install instructions as they fully support windows 10)

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Tue Jan 07, 2020 8:04 am
by ahmed
Thanks a lot.

I will be working with Linux VM for the time being, but I think I will use a spare machine to run it when I am done.
I think it's time to learn more about Linux.

I am also learning Python and Django to be able to customize Mayan EDMS for my needs. Any other things that I need to learn to be able to customize Mayan EDMS?

Thanks again

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Tue Jan 07, 2020 11:44 am
by rssfed23
ahmed wrote: Tue Jan 07, 2020 8:04 am I am also learning Python and Django to be able to customize Mayan EDMS for my needs. Any other things that I need to learn to be able to customize Mayan EDMS?
Mayan EDMS is quite a large project as you may have gathered. It has many different Django apps all working together to form one overall project. Checking out how that all works and fits together in the gitlab repo will be helpful:
Getting a local development version up and running is always a good learning experience as well. You can read through our developers documentation for this: ... index.html

Aside from Python/Django/Virtualenvs it's also handy to learn about concepts relating to Tasks. We use Celery for this: This is how Mayan runs as a distributed app with different workers (so you can have machines dedicated to running OCR) and how we make stuff happen in the backend without locking up the web UI.
Message brokers/queues also play an important role here so basic Redis and RabbitMQ is also handy transferable skills to have. Roberto actually did a great talk about this:

Of course Docker also comes in handy to learn these days. But that's a whole different ball game in its own right!

Having said all that, it really depends what kind of customisations you're looking to do? - The above applies more from a development of Mayan Apps side of things. If it's purely UI/visual changes you want to make then learning Mayans templating engine and how you can override our default templates is all you need (and is a different skill set from the above).

I also ask about what customisations as if there's a requirement we're missing or room for improvement of course we want to know about it :) - It's possible the customisations you need are already on our roadmap.

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Wed Jan 08, 2020 6:10 pm
by rssfed23
Okay, so it turns out if you set up a proper python 3.7 environment all the requirements of Mayan can be installed successfully and the app will start up in development mode.

HOWEVER, the mayan code has a dependency on sh to locate binaries. Sh does not exist in any way on windows and there's no alternative for it. Meaning; we'd have to do some modifications to the mayan code to work around this and they're not small changes.
That's just the first issue I've encountered. There's likely to be many, many others!

I think the best way forward is to use WSL/WSL2. I'm going to look and see if I can get Mayan running on Cygwin, but I don't know many people that would be comfortable using Cygwin in a production environment for critical document storage!

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Mon Feb 03, 2020 9:33 pm
by Weasy
I'm not sure if it would have been better to create a new thread, but because we are talking about a new deployment target here i'll just ask.

Have you ever considered to pack/bundle Mayan with/in Electron and SQLite as backend? I think that would be really nice for single person home use. Maybe you could also use this as a client for a Mayan server installation.

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Mon Feb 03, 2020 11:34 pm
by rssfed23
It’s a nice idea in principle, but Mayan really wouldn’t fit into the electron way of doing things. It’s not a node/JS based app but rather a python/django client and worker model.
Sure; you can get Django apps working inside electron, but that only works if your dependencies are also cross OS compatible/compatible with electron. Many of Mayans aren’t, and that’s not going to change.
I doubt we’ll ever see Mayan EDMS on windows unless an enterprise client specifically asks for it and puts up the $$ to pay for the significant amount of refactoring work required.

Even with only one user we’d likely still get SQLite corruption in many situations as well.

By all means the community is welcome to try and package something up and disable the features with missing electron/windows dependencies and share it here! - but for the core Mayan team home use isn’t a focus point right now.

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Wed Feb 05, 2020 7:47 pm
by Weasy
That's a shame, but understandable. Thanks for the insight.

I'm curious, why do you think there would be some form of corruption with SQLite?

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Posted: Wed Feb 05, 2020 8:32 pm
by rssfed23
SQLite doesn't support concurrency. There's not a lot of experience online of people using SQLite in production because traditionally it was never the "done thing", so in reality it's not as well proven out for production workloads.

But its lack of proven concurrency model is the biggest issue by far. Concurrency allows multiple users to read/write to a database at the same time. But don't think human users here (that's just one user - or 2/3/4/6/8 depending how you've got Gunicorn configured), think programmatic use as well. By default Mayan launches 3 celery worker processes (fast/medium/slow). Each one of those needs access to the DB. Not only that, but because the message queue can process things quite fast it's very common for 2 or more processes to want to read/write to the DB at the same time.
Sometimes even the same table or record if you're doing a batch operation like importing documents and running a workflow. If multiple processes can't use the database at once in a proven safe way then it's a solid no for production use. If we only stored basic data in the DB (such as only login information for example) then SQLite would perhaps be suitable, but sadly we store all critical persistent data aside from documents in it, so the risk is too high.

The reason I say lack of "proven" concurrency model is because in 2014 SQLite added an optional multi-threaded mode and a write ahead log. This solves many of the concurrency issues, BUT it's never really been proven in the market at scale. Definitely not as widely uses as Postgres/Mysql has. There's a lot of "history" here, where SQLite was always known for corruption when running at scale or needing concurrency and although I'm sure most of the most critical bugs have been fixed by now BUT that impression is ingrained in so many people that there's not enough existing experience out there to prove it safe for the kind of DB transactions Mayan needs. Over time this may change, as I know companies like Rancher are using it now instead of Etcd for their k3s Kubernetes distribution (and submitting fixes accordingly), but there's a difference between using SQLite as the DB for an app that was coded with SQLite in mind and using it for an existing app more thoroughly tested on Postgres/Mysql.

So for now we recommend and use it for testing/development purposes only. We do run our CI pipeline/tests through using SQLite so it does have some validation and we'll fix bugs existing in Mayan where possible if they crop up. Our overall database support plan is to support at least 2 commonly used major open database engines. My hope one day is that support for NoSQL DBs such as MongoDB will be proven enough in upstream Django that we can use it as well. You're always welcome to try using NoSQL and report back of course, but a few versions back we went through a lot of effort to migrate everyone off SQLite with migration scripts and it was definitely a challenging time for users as well as ourselves (ultimately it should be Django that takes care of migrations not Mayan but this code didn't exist at the time). We're always open to supporting more open databases as extra options, but I'd say until it's proven in the enterprise (Mayans primary market target) sufficiently it'll be a no.