Mayan-EDMS builds on various platforms

[KevinPawsey]

Hi all,

I was wondering what platforms people are using MayanEDMS on, my experience at the moment has been on a Rock64 (4Gb version).

The SBC is running OpenMediaVault (ARM64 version), which has a built in Docker... My issue is that I can't find a stable ARM64 docker build of MayanEDMS anywhere. This means that I am having to make the Docker Image each time, and it is both time consuming and also I am having to 'tweak' the build scripts to get it to work.

My latest venture has just successfully built v 3.0.3, but I am now getting "[CRITICAL] WORKER TIMEOUT" errors and the web page is not loading... it may be time to roll back to my old armhf version that was working (apart from not being able to email out anything).

Anyway, if anyone has had any experience with running MayanEDMS on Rock64, or ARM64 in general, I would like to hear from you


Kevin

[jeremymorel]

My 'rig' is probably an outside case, but I'm running a Docker instance inside of my Synology DS916+. It's running surprisingly well, (despite that Synology is 'behind' in docker versions, and I'm hesitant to upgrade).

I think I read that there are some tweaks upcoming to get abandoned/failed tasks to release memory, which I'm sure will improve my performance during all my try/fail cycles with figuring things out. One thing that struck me with your mention of tweaks for your setup and having to make the docker image each time-- would copying the original image, then customizing and doing a commit help you? (https://docs.docker.com/engine/referenc ... ne/commit/)

[KevinPawsey]

That is interesting, as we have been considering getting a NAS of some sort (although she was thinking QNAP! ).

I'll have a look at the commit document to see if that will do what I need... but I think that the crux of the issue is that the normal docker images that I have found so far are all AMD64-based, and the Rock64 is ARM64-based.

The main thing with building the image is that it means that it is compiled correctly for the CPU... but it isn't as convenient as just downloading and pressing the 'play' button!

I am so close to getting my ARM64 build working... I just need to figure out the Django/Gunicorn issue that is causing the timeouts...

[rosarior]

I've seen in Reddit's /selfhosted, a few people working Mayan on Synoglogy devices with a high rate of success. I don't own one so I can't add more to that.

To build the ARM64 images yourself you need a base image that is ARM64. Haven't tries the Rock64 but if it has 4GB (and as long as it has a multicore CPU) it should provide a good experience running Mayan.

My personal repository used to be a big server rack with redundancy. After hurricane Maria all that changed and I'm scaling down all my devices as much as possible. The ARM64 architecture has been the best performing one yet and are easy to convert to work on 12 volts DC to run from the solar system batteries. Here are some posts on the topic if you are interested:

https://medium.com/@siloraptor/solar-po ... 83027d20ac

https://medium.com/@siloraptor/adding-a ... f002ab6ba0

The Odroid C2 has been by far the best performing board in the price range. It has a 4 core, 64 bit cpu, 2GB of RAM and with an eMMC module for storage the IO performance is very good.

Producing Docker images for other than AMD-64 has been a disappointment. Multi architecture Docker build are still some time from being ready for production. If budget allows, in the future we might build a small cluster to build the images natively on ARM SBCs while Docker continues to fine tune the multi-arch via manifest support.

Also on the SBCs, there is a perceivable container performance penalty. For the Odroid C2 is use a bare OS install to get the most of the hardware.

This article outlines the experience of installing Mayan on the Odroid HC1: https://medium.com/@siloraptor/building ... aafdeb2f17

Performance was adequate but not as good as the Odroid C2. The HC1 is meant more as a storage solution and not an application server. I got a seconds HC1 and plan to do a two HC1 gluster FS cluster for storage with the Odroid C2 at the application server using the gluster FS for as a remote NAS.

If I'm able to get a Synology in the future (or the guys at Synology send one) to try an official install of Mayan, it could become the official "small" hardware for Mayan.

Bottom line at the moment: Building Docker for the ARM CPU automatically with the current build stack is not possible. There is a small performance penalty using Mayan inside Docker on ARM that pushes it into the less usable section of the spectrum.

My recommendation for small ARM SBCs is to do an OS install using the basic deployment: https://wiki.mayan-edms.com/index.php?t ... deployment

With the new setting option GUI changes we could reduce a bit more, and we are throwing ideas to have Mayan produce the supervisor config file itself so that's another step that could go away in the future.

[rosarior]

Just noticed the Rock64 is now officially supported by Armbian: https://www.armbian.com/rock64/

4 core, 64 bit ARM with eMMC support. Definitely will try Mayan on this board in the future.

[KevinPawsey]

Firstly, thank you for your time and effort in this project... so incredibly useful!!

Thank you also for taking the time to look into this, yes the Rock64 is a well-specced SBC, I have moved from the Raspberry Pi 3 to the Rock64, and the change in performance has been large. The Rock64 is indeed a 4 core processor, and I have the 4Gb version, so it seems to run Mayan well (on the version that I have going so far). It also has eMMC storage as a solution, but at the moment I am using a class 10 SD card, which seems to do the job well (need to look into eMMC to see if there are more benefits!).

I will certainly look into the threads that you have provided... at the moment I am running OpenMediaVault, which has Docker as an 'add-on', so that is the reason for trying to build the Docker image for it. At the moment it builds, but I am getting the timeouts in the logs... so close.. but no cigar

Having looked at the Synologies earlier, it looks like they are AMD64-based processors, so I am guessing that these will all work with the native docker image that is currently being built.

I shall do some more reading, and tweaking and let you know if I get it to run under ARM64 ok

Thanks once again for all you are doing for the project!

[KevinPawsey]

Spent some time today building/documenting the build of Mayan-EDMS on Rock64 with a Docker image if anyone is interested.

Still finalising the last little bits as to getting it working, and making it "straight forward" if there is any such thing when building docker images from scratch

Once I have the instructions completed I can post them here if there is interest.


Kevin

[rosarior]

You are most welcome Kevin and thank you for the discussion, helps us fine tune the priorities of the project.

Can you post the debug log of your image to see if there is any information that could help you in pinpointing the source of the timeout?

Definitely share anything that helps build the image, will helps us out advance the goal of an official ARM Docker image.

We are swamped right now with the database conversion work (SQLite to PostgreSQL) but once that is out of the way (or maybe after releasing 3.1) we can take another look at Docker's multiarch system and see if they've stabilized the support. I was able to build an image on an Odroid C2 a while back but took a lot of time and was not usable (can't remember the cause, but I ended up in your same situation). The goal is to be able to incorporate ARM images into the standard build pipeline.

[KevinPawsey]

Sounds great @rosarior, look forward to the new release

Once I have the process and the wording right, I will post a thread on here... I will make it as thorough as possible... but at the moment, my EDMS looks like it is working really well in ARM64. Tomorrow or over the weekend I may re-build it, just to make sure that all the steps work... hopefully it will help with the automated build of future releases

If I swap back to the old image (and remember where I pulled that image from!), I will see if I can post a debug for you... but I think it may have been an 'unofficial' one that I built.

I am glad you mentioned about the SQLite to PostGreSQL, as that was going to be my next question, as I started running it on SQLite in ARMHF, but now I have it running in ARM64, and this one seems to be working with PostGreSQL, I wouldn't mind porting my data over. If there is a way of doing it, I am willing to be a guinea pig

Thanks again for everything.

Kind regards


Kevin

[mrflippy]

My 'rig' is probably an outside case, but I'm running a Docker instance inside of my Synology DS916+. It's running surprisingly well, (despite that Synology is 'behind' in docker versions, and I'm hesitant to upgrade).

I think I read that there are some tweaks upcoming to get abandoned/failed tasks to release memory, which I'm sure will improve my performance during all my try/fail cycles with figuring things out. One thing that struck me with your mention of tweaks for your setup and having to make the docker image each time-- would copying the original image, then customizing and doing a commit help you? (https://docs.docker.com/engine/referenc ... ne/commit/)

I actually just installed Mayan on a DS718+ and it's running pretty well. I haven't started scanning in much yet though.

I did spend a few hours figuring out a couple of issues:

1. I set up a Docker volume for /var/lib/mayan that pointed to a local shared folder on the NAS. This setup gave me "permission denied" errors for "__init__.py" on startup. I finally gave up and granted read/write to "Everyone" which isn't ideal, but at least it runs now. I suspect I just don't understand how part of this works.
2. The system apparently already has a postgres instance running on port 5432 (which is what the docs say to use). Once I figured out what was happening, this was much simpler to resolve: On the postgres container, set up the container port 5432 to point to local port 5433 (or whatever) and then on the mayan container, add the MAYAN_DATABASE_PORT environment variable with the local port value.