Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Questions, comments, discussions. Over time certain topics might be moved to their own category.
Post Reply
pleblancq
Posts: 16
Joined: Sat Oct 26, 2019 2:09 pm

Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Post by pleblancq »

Hi, just want to tell everybody that you can run mayan on OSX natively without Docker. I installed all the required packages with brew.

What I had to modify:
- kombu.async to kombu.asynchronous directory
- changed all the async variables to asynchronous

Python 3 uses the async as a reserved keyword.

- all imports from celery to kombu.synchronous.x
- changed fcntl.lockf to fcntl.flock in check_sources function for watching files

- all paths from /usr/bin to /usr/local/bin

Works like a charm!

User avatar
rosarior
Developer
Developer
Posts: 522
Joined: Tue Aug 21, 2018 3:28 am
Location: Puerto Rico
Contact:

Re: Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Post by rosarior »

Hi,

Thanks for the feedback and instruction! Very appreciated. Mayan 3.3 includes an updated Celery and Kombu.

pleblancq
Posts: 16
Joined: Sat Oct 26, 2019 2:09 pm

Re: Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Post by pleblancq »

You're welcome.

When OSX Catalina will be more mature, I will upgrade my OS and do a full step by step installation guide for people that prefers using native installation instead of Docker.

Good for the 2 upgraded packages.

For the difference between lockf and flock, I looked on the net and for an unknown reason, lockf should work on OSX but doesn't works.

with lockf (default), I get "Error processing source: [Errno 45] Operation not supported" while using flock works flawlessly.

I tried both scenario, local watch directory or SMB/CIFS watch directory, same error with lockf and flockf works in both case.

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: A docker container that moves watch folder items automatically when ready for processing by Mayan

Post by rssfed23 »

Edit: moved here as I see Pleblancq has already posted about this. Since Mayan 3.3 supports python3 now (I'm trying 3.3.7 on Sierra) I'm hoping less changes are required but I'm 95% of the way there!
pleblancq wrote:
Wed Jan 01, 2020 10:05 pm
I like being able to use Mayan EDMS natively on my macOS without using docker. I would be sad if something is too linux specific that break the native OS installation.
I'm actually working on some documentation for MacOS installation right now and this will definitely become a part of it!

Although can I ask; how have you got your Mayan running on MacOS? - I'm on Catalina and got everything installed successfully (and updating locations of binaries etc) but the Gunicorn worker is throwing up:

billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 6 (SIGABRT).
2020-01-02 10:38:31,157 mayan.apps.common.middleware.error_logging <45027> [ERROR] "process_exception() line 18 Exception caught by request middleware; <WSGIRequest: GET '/api/documents/3/versions/3/pages/2/image/?width=&height=&zoom=&rotation=&_hash=92bcac02c4a72586e21044c0b244b052f5747c7d2c25e6086ca89ca64098e3f3'>, Worker exited prematurely: signal 6 (SIGABRT)."

I also had to change the 3 celery workers to "command = nice -n 1 /usr/local/opt/mayan-edms/bin/celery worker --pool solo -A mayan" as there seems to be some issues with Python + fork() on MacOS which I'm looking into now, but the main issue is the one above and that's what's holding me back from making a detailed MacOS post in the guide (and hopefully adding it into the official docs once proved out).
The workers are working there's just odd things happening on the forking side.
Last edited by rssfed23 on Thu Jan 02, 2020 11:34 am, edited 1 time in total.
Reason: Movet to more appropriate topic
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

pleblancq
Posts: 16
Joined: Sat Oct 26, 2019 2:09 pm

Re: Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Post by pleblancq »

Hi rssfed23,

I installed it on macOS High Sierra with v3.2.8 at first.

I never had celery issues, here is the command in my supervisor.conf

fast:
nice -n 1 /data/mayan-edms/.virtualenv/bin/celery worker -A mayan -Ofair -l ERROR -Q document_states_fast,converter,sources_fast -n mayan-worker-fast.%%h --concurrency=4

medium:
nice -n 18 /data/mayan-edms/.virtualenv/bin/celery worker -A mayan -Ofair -l ERROR -Q default,checkouts_periodic,indexing,signatures,documents_periodic,uploads,documents,file_metadata,metadata,sources,sources_periodic -n mayan-worker-medium.%%h --concurrency=1

slow:
nice -n 19 /data/mayan-edms/.virtualenv/bin/celery worker -A mayan -Ofair -l ERROR -Q statistics,tools,common_periodic,parsing,document_states,mailing,ocr -n mayan-worker-slow.%%h --concurrency=1


For WSGIRequest, I remember I had a similar issue, will look how I fixed it.

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Post by rssfed23 »

*doh* I said Sierra when I meant Catalina!
That would be awesome if you do and could share. I'm hoping to make OSX a first class citizen initially through community documentation. Mainly because a lot of people have a spare MacBook or Mac Mini floating around and they make great performant Mayan instances. Of course there's always Docker, but that just feels too easy ;).

You've not had to change your celery configs so that makes me think whatever is causing my WSGI/forking issue is causing those to fail. I have to use the --pool solo to prevent any kind of process spawning on the workers otherwise they die.

I submitted a patch yesterday as well to enable Darwin detection so we don't have to symlink binaries - it seems Apple now blocks you from doing anything in /opt even with Sudo so now with the patch the user just has to install the right dependencies, one libreoffice symlink then off they go
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Post by rssfed23 »

Update: I've managed to solve the forking errors. Turns out High Serria+ has a security limit that restricts multithreading.
Ensuring "export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" was added to the zshrc and the supervisor environment config has solved the fast worker concurrency issues so now workers don't need to be in a solo pool. Only the WSGI issue left now.


UPDATE: I cleared out brew and ran through my instructions from scratch. Turns out it was the above issue causing all the problems. That combined with using posgresql 12.
I'll type up the "proper" instructions tomorrow and put it in the Guides section for review before we think about putting them in the official docs. It's awesome to have a fully functioning Mayan environment now everything works as expected (nice and fast also!)
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

pleblancq
Posts: 16
Joined: Sat Oct 26, 2019 2:09 pm

Re: Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Post by pleblancq »

Good job! So it was harder to install on Catalina than High Sierra.

Did you have to fix lockf to flock for the watch directories?
you still have to change the celery commands in supervisor?

I'll spin a macOS VM soon to test it (I also have other servers running natively on my High Sierra) and report back.

Yeah, it feels great making it running natively on a plateform.

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Mayan EDMS 3.2.8 natively running on OSX High Sierra python 3.7

Post by rssfed23 »

Yeah definitely much harder. MacOS's new forbidding forking was causing a lot of headaches. Considering it was one simple environment variable to solve the mess it was one of those "the simple things".

But a warning; it may be a lot slower than on high Sierra if that's what you're used to. The security restrictions on forking slow down the starting of actual tasks and if you have non-pdfs that require libreoffice that seems to take a lot longer to start than on linux. But overall it's still pretty nippy and the responsiveness is fast (only ingesting is slower really).

I've not tested watch folders yet but at least it's "known" as such. If you've got a list of the files you edited/changed I can add that into the guide as well (or you can reply to the post when you have time and I'll make sure those changes end up in the final docs). It may be a week or so before we have an answer from the devs on the investigation/approach to flock/lockf due to other commitments.
In the end no changes to celery commands required aside from the usual path values. Of course they've changed a lot since 3.2.8 so if you're not running 3.3 yet there's a lot of small differences (variable names, queue names, celery is now called directly instead of via mayan-edms.py etc). Plus of course Python 3!
This setup will also use rabbitmq for queues and redis for the file lock backend. Reason: MacOS is more likely to get shut down than linux if people are running it on a laptop/personal device so I wanted things to be persistent and Redis lock decreased the fork slowdowns by around 20% (plus less writing to a SSD can only be a good thing :) )

I'll be publishing the guide post in the next 3-4 hours. If you are able to try it on the guide I'll link here it would be much appreciated (a fresh set of eyes to flesh out any final bugs/niggles :D).

One thing I'll look into later is the possibility of doing it all (+ OS dependencies) in Conda. People often have brew for other things so providing a way to "remove everything" without removing stuff they already have is advantageous (plus if all the deps are in Conda than this entire documentation process can be changed to "conda install mayan-edms"!)

EDIT: for anyone else stumbling across this post the guide is available at https://forum.mayan-edms.com/viewtopic.php?f=15&t=1574
Last edited by rssfed23 on Wed Feb 05, 2020 8:39 pm, edited 1 time in total.
Reason: Added guide
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

Post Reply