How to Install Mayan EDMS 3.3.x on MacOS Catalina

[rssfed23]

Traditionally Mayan EDMS has been primarily supported on Linux, BSD based platforms and platforms that can run Linux Containers.

MacOS is a Unix based operating system, so we have the capability to run it natively on MacOS as well. The steps however differ from our main deployment documentation in a number of ways, so we're sharing this here so people can install natively on MacOS

Important Note: If you are new to MacOS or the Linux command line then you may find it easier to use our Simple Docker Installation process. First you will need to install Docker for MacOS which will then allow you to run Linux containers inside of a Linux virtual machine on your Mac. It is a significantly easier process and much more in line with our officially supported methods. The guide below replicates the direct deployment instructions for MacOS allowing you to run Mayan EDMS natively on bare metal with no Virtualisation overhead.

Very Important Note: Although this process is known to work and has been validated by myself (and hopefully other forum members also), MacOS is NOT an officially supported platform for running Mayan EDMS. If you run into production issues and log a Gitlab Issue we cannot guarantee that it will be resolved or that we will help you in a timely fashion. I do not recommend this for production enterprise deployments.
If you are an Enterprise that uses MacOS Server and wish to use it to host Mayan EDMS then please reach out to sales@mayan-edms.com where custom support/consulting packages can be discussed.

Why run on MacOS?
This is a very personal decision. A lot of us use MacOS as our daily driver so it would be nice to have the option of running Mayan EDMS on it for development/testing purposes. Macs also have some very nice hardware in them that can create a very responsive Mayan EDMS environment.
An old Mac Mini for example could be the perfect host for Mayan. Discreet; efficient and powerful.
But primarily; we look into running Mayan EDMS on MacOS because... we can!

Install on MacOS Catalina - Part 1
These instructions are based on MacOS Catalina but should also apply to older MacOS versions. There are some security features in newer MacOS that we work around later in the tutorial and where those do not apply to older MacOS versions it will be called out and you can skip that step.

All steps in this tutorial should be run as the user you're logged in as (E.G "rob"). Only run "sudo" or commands as root when directed to in the guide. Failing to follow this or trying to install as a separate non-login user will result in permissions issues

What are we going to install?
At the end of this tutorial you'll have installed and configured the following Mayan EDMS Architecture:
- Python 3.7
- Mayan EDMS 3.3.7 inside a virtualenv
- RabbitMQ
- Redis
- Postgresql 9.6

We're going to set it up as an "Advanced Direct Deployment" using RabbitMQ over Redis because MacOS machines are typically desktops or laptops and they shutdown/reboot a lot more than servers do. If we used Redis for Mayan's task queue we would loose all pending tasks upon a reboot.
It also means scaling up the environment later on is easy as you don't have to migrate to RabbitMQ then.

1. Install Homebrew
Homebrew is a package manager for MacOS. It allows us to install Linux and Mac applications on MacOS natively. You may have homebrew already set up. If you do not, load up a terminal window and run the following to install MacOS

Code: Select all

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

This will install Homebrew. Homebrew installs most apps into /usr/local. It does this to both avoid conflicts with applications installed on MacOS already but also to give isolation between MacOS and Homebrew Apps. You will find /usr/local/etc and /usr/local/opt. These directories are similar to /etc/ and /opt on Linux. We will install Mayan into /usr/local/opt/mayan-edms for example where on Linux we would use /opt/mayan-edms.
Although MacOS has directories like /opt and /etc, MacOS security features prevent users (even root) from writing into them, so we have to use these "local" alternatives.

2. Install & Configure Python
MacOS comes with it's own version of Python. We want to avoid using that at all costs as it's usually not kept as up to date as the versions in homebrew. Managing everything in homebrew means we can update all dependencies at once.

Code: Select all

brew install python3

This will install Python 3.7 into /usr/local/bin/python3
Any time you want to run python3 for Mayan EDMS we must use the full path to the /usr/local/bin/python3 binary otherwise we will use the system default python in error

We can now create our python3 virtual environment:

Code: Select all

cd /usr/local/opt
/usr/local/bin/python3 -m venv mayan-edms

We're now ready to start installing Mayans dependencies

3. Install & Configure Dependencies
First, let's install Postgres, Redis and RabbitMQ:

Code: Select all

brew install postgresql@9.6 redis rabbitmq

Once completed, we need to apply some configuration changes to Redis:

Code: Select all

echo "maxmemory-policy allkeys-lru" >> /usr/local/etc/redis.conf
echo "save \"\"" >> /usr/local/etc/redis.conf
echo "databases 2" >> /usr/local/etc/redis.conf

We can now proceed to start those 3 services:

Code: Select all

brew services start rabbitmq
brew services start redis
brew services start postgresql@9.6

"brew services" is similar to systemctl on Linux. It acts as our process daemon to start/stop services. If you run the start commands with sudo they will start on boot

Now we need to configure RabbitMQ. We're going to remove the default insecure guest account, add an admin account that can access the web UI and add an account for Mayan to access RabbitMQ on a dedicated vhost.
If you want to change the passwords used feel free, but bear in mind you will have to update subsequent references in this tutorial also.

Code: Select all

rabbitmqctl delete_user guest
rabbitmqctl add_user admin youradminpassword
rabbitmqctl set_user_tags admin administrator
rabbitmqctl add_user mayan mayanrabbitmqpassword
rabbitmqctl add_vhost mayan
rabbitmqctl set_permissions -p mayan mayan ".*" ".*" ".*"
rabbitmqctl set_permissions -p mayan admin ".*" ".*" ".*"

Finally, we need to install some dependencies Mayan itself uses (or Python uses to build dependencies). Run the following command then go grab a coffee:

Code: Select all

brew install exiftool gcc ghostscript tesseract gnupg graphviz libjpeg libmagic libpq libpng libtiff poppler sane-backends supervisor zlib lzlib 

Now we need to install Libreoffice (so mayan can deal with doc/docx and other document types). We're going to use a brew capability called cask which is used to install binary applications into MacOS (such as web browsers):

Code: Select all

brew cask install libreoffice

If you want to be able to use Mayan's "mountindex" capability you'll also need to install fuse:

Code: Select all

brew cask install osxfuse

Finally, we need to create a symlink so Mayan can find the libreoffice binary:

Code: Select all

ln -s /usr/local/bin/soffice /usr/local/bin/libreoffice

We can now go ahead and install Mayan.

4. Install Mayan EDMS
Run the following to install Mayan. You may get some warnings in the console output about failure to build. This is okay as it will download precompiled versions for the dependencies it can't compile. If you get FAIL's, that's when you need to reply to this post for assistance.

Code: Select all

/usr/local/opt/mayan-edms/bin/pip install --no-use-pep517 mayan-edms

We also need to install the postgres, redis and rabbitmq clients for Mayan to be able to talk to those services.
First run

Code: Select all

export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"
export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"

This sets up your build environment so we can build a couple of the dependencies
Then run

Code: Select all

/usr/local/opt/mayan-edms/bin/pip install --no-use-pep517 amqp==2.5.2 psycopg2==2.8.4 redis==3.3.11 psutil==5.6.2

Once that's completed successfully, we can proceed to create our database. Run:

Code: Select all

/usr/local/opt/postgresql@9.6/bin/psql postgres

To drop into a postgres shell.
Then run:

Code: Select all

CREATE USER mayan WITH password 'mayanuserpass';
\q

To create the database role.
Then run:

Code: Select all

/usr/local/opt/postgresql@9.6/bin/createdb -O mayan mayan

To create the empty database.

5. Update literals (ONLY for Mayan 3.3.7 and below)
I submitted a patch to update Mayan to support the correct MacOS locations of certain dependencies (such as tesseract)
Until that is released with 3.3.8, you must update the following 3 files to add in Darwin as a supported OS.

Code: Select all

cd /usr/local/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps

Code: Select all

vim ocr/backends/literals.py

Change line 5:
from

Code: Select all

if platform.system() in ('FreeBSD', 'OpenBSD'):

to

Code: Select all

if platform.system() in ('FreeBSD', 'OpenBSD', 'Darwin'):

Then again for the following 2 other files:

Code: Select all

vim ocr/file_metadata/literals.py

Change line 5:
from

Code: Select all

if platform.system() in ('FreeBSD', 'OpenBSD'):

to

Code: Select all

if platform.system() in ('FreeBSD', 'OpenBSD', 'Darwin'):

Code: Select all

vim ocr/converter/literals.py

Change line 52:
from

Code: Select all

if platform.system() in ('FreeBSD', 'OpenBSD'):

to

Code: Select all

if platform.system() in ('FreeBSD', 'OpenBSD', 'Darwin'):

Again, if you're installing 3.3.8 the above step is not needed.

6. Mayan EDMS Initial Setup
We're now ready to run the initialsetup command and prepare static commands for Mayan. If either of these fail, then something earlier in the process must have gone wrong and you should reply to this thread for assistance

Run the following:

Code: Select all

MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}" MAYAN_MEDIA_ROOT=/usr/local/opt/mayan-edms/media /usr/local/opt/mayan-edms/bin/mayan-edms.py initialsetup

With a bit of luck, the last 2 lines should say:

Code: Select all

Installing package: Lato font ... Downloading... Extracting... Complete.
Superuser created successfully.

If it does, you can breathe a sigh of relief at this point as it means all the dependencies are installed and the rest of the guide should be plain sailing. If you've got this far successfully, give yourself a well deserved pat on the back and eat a bar of chocolate: You've earned it!

Next, we generate Mayans Static Media:

Code: Select all

MAYAN_MEDIA_ROOT=/usr/local/opt/mayan-edms/media \
/usr/local/opt/mayan-edms/bin/mayan-edms.py preparestatic --noinput

(ignore the "you are using SQLite" error. That's expected as we've not passed in database credentials as we don't need to)


This concludes part one. We're not done just yet! - There's just a limit on the number of characters in a forum post so I'm going to continue it below in part 2:

[rssfed23]

Part 2:

7. Supervisor config
Finally before we can start Mayan EDMS we need to configure supervisor. Supervisor will manage the Mayan web and task processes for us (it's a process manager for Python). We manage the Supervisor process with "brew services" and then Supervisor will start/stop/restart the Mayan services as needs be.
This is one step you absolutely cannot copy from the standard install documentation. You cannot use the platformtemplate command from that documentation as all the file paths and configuration items will be wrong. Instead; we will copy one I've made for you.

First:

Code: Select all

mkdir mkdir /usr/local/etc/supervisor.d

Then:

Code: Select all

vim /usr/local/etc/supervisor.d/mayan.ini

You can then press the "i" button to put vi into insert mode and paste in the following configuration.
You will need to update the "user" field at the end of each command (highlighted bold here) to match the local username you are logged in as/running Mayan as. If you don't know what the unix version of that name is run "id" on the command line and it'll tell you your short username in brackets after the uid section.
You will also need to update any other directories or username/passwords you changed during the setup process.

Code: Select all

[supervisord]
environment=
   PYTHONPATH="/usr/local/opt/mayan-edms/media/mayan_settings",
    DJANGO_SETTINGS_MODULE=mayan.settings.production,
    MAYAN_MEDIA_ROOT="/usr/local/opt/mayan-edms/media",
    OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES,
    MAYAN_ALLOWED_HOSTS="['*']",
    MAYAN_CELERY_RESULT_BACKEND="redis://127.0.0.1:6379/1",
    MAYAN_CELERY_BROKER_URL="amqp://mayan:mayanrabbitmqpassword@127.0.0.1:5672/mayan",
    MAYAN_DATABASES="{default: {ENGINE: django.db.backends.postgresql, HOST: 127.0.0.1, NAME: mayan, PASSWORD: mayanuserpass, USER: mayan}}"

[program:mayan-gunicorn]
autorestart = true
autostart = true
command = /usr/local/opt/mayan-edms/bin/gunicorn -w 4 mayan.wsgi --max-requests 500 --max-requests-jitter 50 --worker-class sync --bind 0.0.0.0:8000 --timeout 120
user = rob


[program:mayan-worker-fast]
autorestart = true
autostart = true
command = nice -n 1 /usr/local/opt/mayan-edms/bin/celery worker -A mayan -Ofair -l ERROR -Q document_states_fast,converter,sources_fast -n mayan-worker-fast.%%h
killasgroup = true
numprocs = 1
priority = 998
startsecs = 10
stopwaitsecs = 1
user = rob

[program:mayan-worker-medium]
autorestart = true
autostart = true
command = nice -n 18 /usr/local/opt/mayan-edms/bin/celery worker -A mayan -Ofair -l ERROR -Q default,checkouts_periodic,indexing,signatures,documents_periodic,uploads,documents,file_metadata,metadata,sources,sources_periodic -n mayan-worker-medium.%%h --concurrency=1
killasgroup = true
numprocs = 1
priority = 998
startsecs = 10
stopwaitsecs = 1
user = rob

[program:mayan-worker-slow]
autorestart = true
autostart = true
command = nice -n 19 /usr/local/opt/mayan-edms/bin/celery worker -A mayan -Ofair -l ERROR -Q statistics,tools,common_periodic,parsing,document_states,mailing,ocr -n mayan-worker-slow.%%h --concurrency=1
killasgroup = true
numprocs = 1
priority = 998
startsecs = 10
stopwaitsecs = 1
user = rob

I've included an example supervisor config inside the configs.zip file attached to this post for reference.

configs.zip

(3.91 KiB) Downloaded 123 times

Once you've saved the file (:wq), we have to generate a mayan configuration file and tweak a couple more items:

Code: Select all

cd /usr/local/opt/mayan-edms/media
cp config_backup.yml config.yml
vim config.yml

On line 228, change DOCUMENT_PARSING_PDFTOTEXT_PATH to:

Code: Select all

/usr/local/bin/pdftotext

On line 374, change SOURCES_SCANIMAGE_PATH: to:

Code: Select all

/usr/local/bin/scanimage

On line 324, change SIGNATURES_GPG_PATH to:

Code: Select all

/usr/local/bin/gpg

On line 380, change STORAGE_TEMPORARY_DIRECTORY to:

Code: Select all

/usr/local/opt/mayan-edms/media/tmp

Then run

Code: Select all

mkdir /usr/local/opt/mayan-edms/media/tmp

I've included an example config attached to this post for reference.

configs.zip

(3.91 KiB) Downloaded 123 times

That should be everything.
To start Mayan EDMS, run:

Code: Select all

brew services start supervisor

You can now access Mayan EDMS on http://localhost:8000
It's also accessible from any other computer that can access your Mac's IP address.

You should be presented with the login page containing the automatically generated admin credentials. Go ahead and login

You can run brew services stop supervisor to stop Mayan or brew services restart supervisor to restart it.

Where are the logs?
MacOS works a bit differently than other OS's, so to find your supervisor logs run:

Code: Select all

env | grep TMPDIR

You can then cd into the directory shown, and inside that directory will contain all your mayan-worker* log files.
Alternatively, enable PRODUCTION_ERROR_LOGGING in your config.yml file and restart mayan

[rssfed23]

How do I get help on this?

As mentioned at the start, this is an unofficial - but working - port of Mayan EDMS to MacOS. If you encounter issues you must put them into this thread and not the main forum (we'll move anything reported elsewhere to here).
In order to help you, when making your post in this thread you must include:

MacOS Version
The error message shown/symptom description
What steps you've tried to diagnose/fix it
The output of logs from either the production error log or your the errors shown in your Mayan task queue logs (TMPDIR=).

Without this information it will be a lot harder for us to help you.

As always, you're welcome to log bugs into Gitlab, but be aware that any bug unique to MacOS will not get priority from the development team as it's an unsupported OS.

Many thanks, and good luck!
Rob