Recommended Automated Backups

Questions, comments, discussions. Over time certain topics might be moved to their own category.
Post Reply
mamos
Posts: 10
Joined: Thu Jan 23, 2020 7:20 pm

Recommended Automated Backups

Post by mamos »

What automated backup method would be recommended for a Simple Docker installation of Mayan on Ubuntu using all of the installation defaults?

I currently have Barracuda Backup Agent performing a File System backup on my Mayan server. What would be the correct steps for restoring either everything and/or specific documents onto an identically configured Ubuntu server? Basically, can I restore the data without having to do a manual backup of the files? I do not want to have to perform manual backups.

Thanks!

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Recommended Automated Backups

Post by rssfed23 »

Hi Mamos. Great question!

We don't have a "recommended" backup solution as such. Ultimately as long as your backup tool/script is able to take a dump of the database (which is much easier to restore than the raw postgres files on the filesystem) and then copy that dump and your mayan media directory then you're good to go.

Everything else is really down to personal preference.

I will be writing a guide for this soon on how I personally do backups (and ones I recommend to our enterprise clients). It uses a tool called Restic, which is a snapshot based backup utility. It's similar in functionality to Borg but much much simpler to setup. Comes with de-duplication out of the box and as it's a snapshot based tool with file hashing I trust it much more than any kind of standard filesystem backup. We use it for our production systems for Mayan EDMS the company and it works a treat! Supports any storage backend you can think of or any that rclone supports.

Of course restic itself doesn't automate backups out of the box, but that's a good thing really as then you're not bound to some kind of server side tool for backups. A simple bash script that calls restic and then a cron job is all that's needed. What we do for Mayan EDMS the company is have restic backup to a local repository first, as then we can call rsync in our scripts to back that local repository up to as many remote locations as we like. It's this process I'll be basing my upcoming guide post on. But this is only a recommendation of course, we have no "official" tool as Mayan doesn't really need one. Backup/dump the database to a file then backup the DB file and Mayan media_root and you're good to go!
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

mamos
Posts: 10
Joined: Thu Jan 23, 2020 7:20 pm

Re: Recommended Automated Backups

Post by mamos »

So I have Mayan installed using the Simple Docker Install and I am not finding pg_dump anywhere. Isn't it supposed to be included in the /bin folder of the PostgreSQL install? I'd like to try using it to dump the database using a bash script automated with cron then I can just use Barracuda to back that up.

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Recommended Automated Backups

Post by rssfed23 »

As you're using docker, you have to run pg_dump within the postgres container itself not the Mayan container.
pg_dump should be running inside the postgres container, so you'll need to docker exec into that container first.

See https://docs.mayan-edms.com/chapters/do ... ckups.html for some other useful bits of information.
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

mamos
Posts: 10
Joined: Thu Jan 23, 2020 7:20 pm

Re: Recommended Automated Backups

Post by mamos »

I ended up installing Restic as you had mentioned. I was able to successfully obtain a snapshot with it of /docker-volumes and successfully restored to a test server. I'd like to automate Restic using Cron and Bash as well but I have noticed that Restic requires the repository password to be entered when it runs. Could you please point me in the right direction on how to get that password to be applied using Bash?

Thanks!

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Recommended Automated Backups

Post by rssfed23 »

Sure!
Create a file in /etc/restic/repo.env (the file can be anywhere and called anything).
Then put something like this into the file:

Code: Select all

export RESTIC_REPOSITORY=s3:http://yourbackuplocation
export RESTIC_PASSWORD=repopassword
The RESTIC_REPOSITORY value is what you provide after -r on the CLI (potentially won't be S3 based just put whatever line you enter on the CLI after the = sign), and the RESTIC_PASSWORD is the password to the repository.

But please ensure this file is only readable by root. Chmof 600 ought to do it.

Then to use it in your script (or before you run the CLI tool):

Code: Select all

source /etc/restic/repo.env
Job done :)
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

manux
Posts: 3
Joined: Sun Feb 16, 2020 8:57 am

Re: Recommended Automated Backups

Post by manux »

Hi

Basicly to make a succesfull backup and retore it later you need to copy the folder structure that docker uses outside of itself.
This can be done by shutting down docker at night and copy the files.

I wrote a script to do this:

Code: Select all

#/bin/bash

# to restore:
# clean up the broken mess
# copy over the data files, extraxt
# install mayan,prosgress and redis from docker 
# 

if [ -z "$1" ]
  then
    echo ""
    echo "No argument supplied"
    echo "You need to supply: backup PATH-TO-MAYAN-EMDS-FOLDER PATH-TO-BACKUP-LOCATION"
    echo "or"
    echo "You need to supply: restore FILENAME-OF-BACKUP.TAR.GZ"
    echo ""
    exit
fi

if [ "$1" == "backup" ]
	then
		bpath=$2
		
		if [ -z "$2" ]
		then
			echo "No Path supplied, Exiting"
			exit
		fi

		filepath=$3
		
		if [ -z "$3" ]
		then
			echo "No backup Path supplied, Exiting"
			exit
		fi
    echo "Backup in prosgress"
    
    # create semi uniq timestamp
	date_stamp=$(date +%s-%y%m%d%-H%M%S)

	# find container IDS
	cont_mayan=$(docker ps | grep mayanedms/mayanedms | cut -d' ' -f1)
	cont_redis=$(docker ps | grep redis:5.0-alpine | cut -d' ' -f1)
	cont_postgres=$(docker ps | grep postgres:9.6-alpine | cut -d' ' -f1)

	docker stop $cont_mayan
	docker stop $cont_redis
	docker stop $cont_postgres

	# backup data files and db dump
	tar -zcf $filepath$date_stamp-mayan-backup-datafiles-dbase.tar.gz $bpath
	chmod 777 $filepath$date_stamp-mayan-backup-datafiles-dbase.tar.gz
	
    echo "Backup finished"
	# restart the containers
    echo "Restarting mayan containers"
	docker start $cont_redis
	docker start $cont_postgres
	sleep 1
	docker start $cont_mayan
    
    echo "Cleaning older Backups (+7 days)"
    
    find $filepath -type f -mtime +7 -name '*.gz' -execdir rm -- '{}' \;
    
    exit
fi


if [ "$1" == "restore" ]
	then
		bfile=$2
		if [ -z "$2" ]
		then
			echo "No Backup File supplied. Exiting"
			exit
		fi
		
#delete everything  docker like on the machine
docker kill $(docker ps -q) # stop all containers
docker rm $(docker ps -a -q) # remove all containers 
docker rmi $(docker images -q) # remove all images
docker network prune # remove all networks
docker volume prune # remove all volumes 

docker pull mayanedms/mayanedms:latest
docker pull redis:5.0-alpine
docker pull postgres:9.6-alpine

# create network
docker network create mayan

# extraxt the data files 
cd /
tar xvf $bfile

docker run \
-d \
--name mayan-edms-postgres \
--network=mayan \
--restart=always \
-e POSTGRES_USER=mayan \
-e POSTGRES_DB=mayan \
-e POSTGRES_PASSWORD=mayanuserpass \
-v /nas/server/mayan_docker/mayan-edms/postgres:/var/lib/postgresql/data \
postgres:9.6-alpine

docker run \
-d \
--name mayan-edms-redis \
--network=mayan \
-v /nas/server/mayan_docker/mayan-edms/redis:/data \
redis:5.0-alpine \
redis-server \
--databases \
"2" \
--maxmemory-policy \
allkeys-lru \
--save \
"" \
--requirepass mayanredispassword

docker run \
-d \
--name mayan-edms \
--network=mayan \
--restart=always \
-p 8082:8000 \
-e MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'mayan-edms-postgres'}}" \
-e MAYAN_CELERY_BROKER_URL="redis://:mayanredispassword@mayan-edms-redis:6379/0" \
-e MAYAN_CELERY_RESULT_BACKEND="redis://:mayanredispassword@mayan-edms-redis:6379/1" \
-v /nas/server/mayan_docker/mayan-edms/media:/var/lib/mayan -v /nas/server/mayan_docker/scanned_files:/scanned_files -v /nas/server/mayan_docker/watched_files:/watched_files  \
mayanedms/mayanedms:latest

cont_mayan=$(docker ps | grep mayanedms/mayanedms:latest | cut -d' ' -f1)
docker exec -it $cont_mayan /bin/bash -c "apt update && apt install tesseract-ocr-deu tesseract-ocr vim nano htop -y"
docker exec -it $cont_mayan /bin/bash -c "chmod 755 /usr/share/tesseract-ocr/4.00/tessdata/deu.traineddata"

else
echo ""
echo "Wrong argument"
exit
fi

User avatar
rosarior
Developer
Developer
Posts: 513
Joined: Tue Aug 21, 2018 3:28 am
Location: Puerto Rico
Contact:

Re: Recommended Automated Backups

Post by rosarior »

This is very clever @manux! Thank you for sharing your backup and restore script!

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Recommended Automated Backups

Post by rssfed23 »

Thanks for sharing! - But a word of caution to others; it's usually recommended to backup via the dump and restore methods because if you run a backup against postgres 9.6.1 and then we migrate postgres over to v10, the restored directory won't work due to the different structures, and there's no easy way with docker to migrate/upgrade postgres tablespace versions (because it requires both versions of postgres running at the same time). For snapshot style restores with everything else being equal it's great, but a pg_dump will always help with portability :)
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

mamos
Posts: 10
Joined: Thu Jan 23, 2020 7:20 pm

Re: Recommended Automated Backups

Post by mamos »

I was able to do something similar for applying the Restic repo password, so my backups and purging of old snapshots is automated now. I have it writing to a daily log file when the script completes. Just trying to figure out how to purge the log files based on revision of the log file (like delete all but the most recent 5 or something like that). I'm also working on setting up email notifications when the backups complete. Thanks for the help so far!

Post Reply