Deploy Mayan EDMS to Windows 10 machine without Docker.

Reverse proxies, SSL termination, web servers.
Post Reply
ahmed
Posts: 2
Joined: Fri Oct 11, 2019 9:13 am

Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by ahmed »

First, many thanks to rosarior and rssfed23 who seem very active and support all users.

I read the guides on how to install Mayan EDMS. It was straight forward when I installed the docker image on a linux Ubuntu VM. But I am used to Windows not Linux. So can I run Mayan EDMS on my Windows Machine without Docker (for example by compiling the source code directly on my Windows 10 machine)? I want to customize the source code and run it directly from PyCharm or Visual Studio Code.

Also are there any advantages of deploying Mayan EDMS on Linux as compared to Windows?

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by rssfed23 »

Thanks for your kind words - it's much appreciated :)

At this point in time no, you can't run Mayan EDMS natively on windows.
The only option for windows users is to use docker for windows OR to install into WSL2.

This is because some of the Python packages Mayan depends on to function do not have Windows wheel (precompiled python binary) versions available and can not be compiled on Windows or have no support for windows. There's also missing Linux libraries that have no windows equivalents and cause quirks getting Mayan to run.

I will soon be trying to see if there's any way around this after I've written the MacOS deployment guide today but given the missing dependencies it is very, very unlikely to work successfully.

So for now you have to use docker for windows, a VM or WSL2. You don't need an Ubuntu VM for Docker though the native docker for windows (which manages a small Linux VM for you) works okay.

There's many benefits of using Linux to host critical services, but specifically for Mayan it relates to the dependencies all being available on Linux and performance. Of course there's security as well but that's a whole different topic and takes us into "why Linux over Windows for anything" debate.

Mayan does run on WSL2 though. So you can create a Windows Subsystem for Linux 2 Ubuntu Environment and deploy there. It is not as performant as running it inside of a virtual machine however (WSL2 is notorious for taking a performance hit on disk IO and CPU performance).
I actually do all my Mayan development/test work on a WSL2 environment (but host it in production on a standard Linux VM).


So to summarise the 3 options are (in my preferred order in terms of performance):
Docker for Windows
Ubuntu VM
WSL2

I will type up a guide for 1 and 3 at some point, but we already have a great docker for windows 8 guide available at viewtopic.php?f=15&t=1536 and most of the same concepts apply (the installing docker section you can follow the docker.com install instructions as they fully support windows 10)
Please don't PM for general support; start a new thread with your issue instead.

ahmed
Posts: 2
Joined: Fri Oct 11, 2019 9:13 am

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by ahmed »

Thanks a lot.

I will be working with Linux VM for the time being, but I think I will use a spare machine to run it when I am done.
I think it's time to learn more about Linux.

I am also learning Python and Django to be able to customize Mayan EDMS for my needs. Any other things that I need to learn to be able to customize Mayan EDMS?

Thanks again

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by rssfed23 »

ahmed wrote:
Tue Jan 07, 2020 8:04 am
I am also learning Python and Django to be able to customize Mayan EDMS for my needs. Any other things that I need to learn to be able to customize Mayan EDMS?
Mayan EDMS is quite a large project as you may have gathered. It has many different Django apps all working together to form one overall project. Checking out how that all works and fits together in the gitlab repo will be helpful: https://gitlab.com/mayan-edms/mayan-edms
Getting a local development version up and running is always a good learning experience as well. You can read through our developers documentation for this: https://docs.mayan-edms.com/chapters/de ... index.html

Aside from Python/Django/Virtualenvs it's also handy to learn about concepts relating to Tasks. We use Celery for this: http://www.celeryproject.org/. This is how Mayan runs as a distributed app with different workers (so you can have machines dedicated to running OCR) and how we make stuff happen in the backend without locking up the web UI.
Message brokers/queues also play an important role here so basic Redis and RabbitMQ is also handy transferable skills to have. Roberto actually did a great talk about this: https://www.youtube.com/watch?v=0UJTG5QU7Ss

Of course Docker also comes in handy to learn these days. But that's a whole different ball game in its own right!

Having said all that, it really depends what kind of customisations you're looking to do? - The above applies more from a development of Mayan Apps side of things. If it's purely UI/visual changes you want to make then learning Mayans templating engine and how you can override our default templates is all you need (and is a different skill set from the above).

I also ask about what customisations as if there's a requirement we're missing or room for improvement of course we want to know about it :) - It's possible the customisations you need are already on our roadmap.
Please don't PM for general support; start a new thread with your issue instead.

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by rssfed23 »

Okay, so it turns out if you set up a proper python 3.7 environment all the requirements of Mayan can be installed successfully and the app will start up in development mode.

HOWEVER, the mayan code has a dependency on sh to locate binaries. Sh does not exist in any way on windows and there's no alternative for it. Meaning; we'd have to do some modifications to the mayan code to work around this and they're not small changes.
That's just the first issue I've encountered. There's likely to be many, many others!

I think the best way forward is to use WSL/WSL2. I'm going to look and see if I can get Mayan running on Cygwin, but I don't know many people that would be comfortable using Cygwin in a production environment for critical document storage!
Please don't PM for general support; start a new thread with your issue instead.

Weasy
Posts: 2
Joined: Mon Feb 03, 2020 9:28 pm

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by Weasy »

I'm not sure if it would have been better to create a new thread, but because we are talking about a new deployment target here i'll just ask.

Have you ever considered to pack/bundle Mayan with/in Electron and SQLite as backend? I think that would be really nice for single person home use. Maybe you could also use this as a client for a Mayan server installation.

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by rssfed23 »

It’s a nice idea in principle, but Mayan really wouldn’t fit into the electron way of doing things. It’s not a node/JS based app but rather a python/django client and worker model.
Sure; you can get Django apps working inside electron, but that only works if your dependencies are also cross OS compatible/compatible with electron. Many of Mayans aren’t, and that’s not going to change.
I doubt we’ll ever see Mayan EDMS on windows unless an enterprise client specifically asks for it and puts up the $$ to pay for the significant amount of refactoring work required.

Even with only one user we’d likely still get SQLite corruption in many situations as well.

By all means the community is welcome to try and package something up and disable the features with missing electron/windows dependencies and share it here! - but for the core Mayan team home use isn’t a focus point right now.
Please don't PM for general support; start a new thread with your issue instead.

Weasy
Posts: 2
Joined: Mon Feb 03, 2020 9:28 pm

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by Weasy »

That's a shame, but understandable. Thanks for the insight.

I'm curious, why do you think there would be some form of corruption with SQLite?

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Deploy Mayan EDMS to Windows 10 machine without Docker.

Post by rssfed23 »

SQLite doesn't support concurrency. There's not a lot of experience online of people using SQLite in production because traditionally it was never the "done thing", so in reality it's not as well proven out for production workloads.

But its lack of proven concurrency model is the biggest issue by far. Concurrency allows multiple users to read/write to a database at the same time. But don't think human users here (that's just one user - or 2/3/4/6/8 depending how you've got Gunicorn configured), think programmatic use as well. By default Mayan launches 3 celery worker processes (fast/medium/slow). Each one of those needs access to the DB. Not only that, but because the message queue can process things quite fast it's very common for 2 or more processes to want to read/write to the DB at the same time.
Sometimes even the same table or record if you're doing a batch operation like importing documents and running a workflow. If multiple processes can't use the database at once in a proven safe way then it's a solid no for production use. If we only stored basic data in the DB (such as only login information for example) then SQLite would perhaps be suitable, but sadly we store all critical persistent data aside from documents in it, so the risk is too high.

The reason I say lack of "proven" concurrency model is because in 2014 SQLite added an optional multi-threaded mode and a write ahead log. This solves many of the concurrency issues, BUT it's never really been proven in the market at scale. Definitely not as widely uses as Postgres/Mysql has. There's a lot of "history" here, where SQLite was always known for corruption when running at scale or needing concurrency and although I'm sure most of the most critical bugs have been fixed by now BUT that impression is ingrained in so many people that there's not enough existing experience out there to prove it safe for the kind of DB transactions Mayan needs. Over time this may change, as I know companies like Rancher are using it now instead of Etcd for their k3s Kubernetes distribution (and submitting fixes accordingly), but there's a difference between using SQLite as the DB for an app that was coded with SQLite in mind and using it for an existing app more thoroughly tested on Postgres/Mysql.

So for now we recommend and use it for testing/development purposes only. We do run our CI pipeline/tests through using SQLite so it does have some validation and we'll fix bugs existing in Mayan where possible if they crop up. Our overall database support plan is to support at least 2 commonly used major open database engines. My hope one day is that support for NoSQL DBs such as MongoDB will be proven enough in upstream Django that we can use it as well. You're always welcome to try using NoSQL and report back of course, but a few versions back we went through a lot of effort to migrate everyone off SQLite with migration scripts and it was definitely a challenging time for users as well as ourselves (ultimately it should be Django that takes care of migrations not Mayan but this code didn't exist at the time). We're always open to supporting more open databases as extra options, but I'd say until it's proven in the enterprise (Mayans primary market target) sufficiently it'll be a no.
Please don't PM for general support; start a new thread with your issue instead.

Post Reply