Mayan EDMS Organisation Concepts Introduction

Community contributed guides or tutorials for multiple topics like installations for other operating systems or platforms, monitoring, log aggregation, etc.
Post Reply
User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Mayan EDMS Organisation Concepts Introduction

Post by rssfed23 »

A question was asked recently on the forum as to if we had any best practice or getting started documentation on how best to organise/structure a Mayan EDMS setup. The user specifically was referring to their home use of Mayan to bring paperless to their home.
I wrote a long response to that question but thought it might be better to move it here. These explain some introductory concepts as to the recommended approach to Document organisation within Mayan EDMS. As with many things in life; there is no "one size fits all" approach and your individual requirements should always take precedent, but this will at least serve as an introduction to users new to Mayan wondering "What's a document type? How's it different from a cabinet?" etc.

As always, I cannot recommend enough that you all purchase the Exploring Mayan EDMS book. It will be updated soon for v3.3.x and then again for
v4 and is a great resource for anyone both new and experienced to Mayan. You would also be supporting the project and the team behind this truely great piece of software.
https://www.mayan-edms.com/book/

These notes are ideal if you're getting started as a home user, but if you're beginning your adventure into Mayan EDMS on behalf of an enterprise organisation then these notes will not go far enough. Document management is a huge area and it's vital you get it right if your organisation is to succeed. That's why we have the Mayan EDMS Support Plans available for consideration.
Even more relevant to you though; we have a wide range of consulting and services capability within the team and would love to work with you on your implementation of Mayan EDMS or Document Management in general. If you're running Mayan in production then the chances are it will quickly become a critical part of your infrastructure that requires the same SLAs and Uptime as any other business critical app you have. Installing & Configuring Mayan EDMS in a Highly Available production ready way is what the consulting team do day to day, as well as help you design an implementation, plan a migration, provide training material to team members and advice on industry best practice for Document Management.
It can also be used as a way to sponsor features and get a bit of functionality you require implemented as part of the engagement. Don't hesitate to reach out to sales@mayan-edms.com to enquire about this or if you have any questions about the services we offer.

Now, let's take some look at the basic organisational structures within Mayan EDMS:
Please don't PM for general support; start a new thread with your issue instead.

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Mayan EDMS Organisation Concepts Introduction

Post by rssfed23 »

Document Type: Think of Document Types as "physical medium" rather than a subject category. So "Book", "Newspaper", "Letter", "Invoice" are document types but "Finance", "Banking", "Car Paperwork" aren't document types those are subjects/categories of documents for which you should use tags to track. Each document can only have one document type (which makes sense as it's physical medium).

Here's my document types for home use. I've not stuck strictly to the "physical medium" rule as you can see but by and large it's there


Image


Metadata: Metadata is a huge subject in it's own right. it's used to add....well metadata to documents! For example, adding "Company" to the document type "Letter" so you can add what company sent you a letter. There will be improvements in metadata so things can be automated as well as time goes on (like automatically reading barcodes to input that into the metadata field). Using metadata in indexes is also very powerful. Here's some example metadata items you might want to add to document types. Metadata fields can be shown on the document list screen but really come into power when used with Indexes. Metadata can also be validated against a specific type (such as "received date" must be a date) and by assigning the right set of metadata you'll never need to care about document titles/filenames as your indexes/searches will take care of finding the right documents for you.
At the moment you have to set metadata at the time a document is uploaded and before you actually choose which document is accepted. In the case of a watch folder everything in that folder goes to one document type with no metadata but some people have written scripts that use the Mayan API to automatically apply metadata based on what the OCR produced


Image


Cabinets: Think of these as folders. Very much like a filing cabinet or when browsing windows explorer. You can add sub-levels to cabinets so it matches a directory structure. Documents can be members of multible cabinets which is an awesome feature. For example; I have a cabinet for "future visa evidence" and one for "Monzo (my bank) Bills" as we need to evidence my husbands visa to prove our address.
Cabinets are the simplest way to organise documents. BUT: Cabinets are a manual activity. When you have thousands of documents manually managing cabinets becomes a real pain in the bum, which is why we always recommend people use indexes rather than cabinets. But for home use it's much more manageable as there's less documents. Heres mine....
When I scan a bank statement I can add it to both cabinets as part of the upload flow.


Image

Image


Tags: Tags are awesome. You can associate as many as you want to each document. They're separate from everything else and you can use tags to eventually build indexes. They show in the document view screen and can really help with visually organisation as they're colour coded. They can be used for categories or anything you want really:


Image


Indexes/ Indexes is the granddaddy of organisation automation! It's where Mayan can really excel. Think of an index being a real time always updating automated way to create a cabinet. You Basically write a small "query" value to search documents and then things that match them can be shown to you as the result. Pretty much any bit of metadata about a document can be queried in this way. Tags and metadata as we talked about above are super easy to query but you can also query on things such as document EXIF data to pull out details about the document you may not have known were embedded in the document. This is how we can get things like "created date" or "document author" and show all the results to users. Those values are usually stored inside a PDFs metadata so rather than relying on the filesystem last modified date or the date it was imported into Mayan we can use document file metadata (EXIF) instead.
But their most simple form is querying on document title, tag, cabinet or a metadata value that you've added to it.
If you remember the section on metadata above, you can see I have one for "Company". I assign the right company value to all letters/documents I scan in at home.
This means I can set up an index that creates a virtual folder-like-view pulling in ALL documents (or only specific document types if you prefer) that have a value set for "company" and then then list those values for me and let me browse all documents with that value assigned.


Image


I can then click "Barclays" and see all documents that have the company value set to barclays regardless of what document type/tag/cabinet they're in (I can also restrict it to certain document types if I want).
Pretty powerful, huh :)?
Here's all the indexes I've set up. They all have many sub-indexes:


Image


Going back to the exif data I mentioned earlier, here's one that organises documents by "file type":


Image
Please don't PM for general support; start a new thread with your issue instead.

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Mayan EDMS Organisation Concepts Introduction

Post by rssfed23 »

Continued on third post:

Smart links These are - as the name implies - ways to correlate one document to another. It's not something I've played with massively yet but think of it as a "similar to X" type feature.
For my example here, let's say I've created a smart link called "File Creation Date". If I put the following template snippet in:

Code: Select all

{{ document.file_metadata_value_of.exiftool_CreateDate|slice:"5:7" }}
This will search all documents exif metadata for the value of the created date and then discard the first few fields (as we don't want them - that's what the slice does). This templating engine is also how we create indexes (see the index reference post in the Guides forum for a more detailed walkthrough of indexes).

What this will do is add a smart link to every document type the smart link is assigned to so that I can see all other documents that share the same created date!

Just like indexes, you can create smart links on any piece of data/metadata about a document including metadata values you created yourself. I could put in a "by company" search again here so the smart links tab on the document view will show all other documents with the same value of "company="

Actually let's do that now. I'll add a smart link for by company:


Image


I then associate it with all document types:


Image


We need to add a "condition" to the smart link which basically says "if the value of this piece of metadata (company) is equal to (blackhorse) then show it.
There are SOO many different built in fields for the "foreign" document selector. They're the same ones available for indexes as well:


Image


So that when I view a document (let's say it was sent by a company called BlackHorse) and go to the smart links tab I can see the search template I just added:


Image


And when I click it see all other documents also sent to me by BlackHorse:


Image


Smart links + indexing are more complex than tags and cabinets for organisation but they really are very, very powerful. As long as you tag/put metadata on documents your document organisation basically runs itself. No other DCMS system out there does this to my knowledge free nor paid.


Thanks for reading!
Feel free to share how you personally organise Mayan EDMS below :)
Please don't PM for general support; start a new thread with your issue instead.

mcarlosro
Posts: 11
Joined: Fri Jan 10, 2020 3:23 pm

Re: Mayan EDMS Organisation Concepts Introduction

Post by mcarlosro »

Thanks for the explanation and sharing your own structure.

After reading it a few times, print (sorry) and study it, I have a few questions:
- Documents are assigned always to a "Document Type", but do you assign them as well to cabinets?

- I see a lot of repetitive information in document type, cabinet, tags. Can explain the reason?
- I'm starting with Payslips and Health information (analysis and reports). Regarding Health, I see a document type: Medical and a tag, and nothing else. also noticed Helthcare under cabinets. Why don't you keep the same terminlogy?
- Can you explain a bit about Payslips? Do you use Employment cabinet?
- I was expecting initially to have a lot more metadata fields. Are you removing metadata fields by adding tags?

I have a lot more questions, but answers to these will help to reduce my questions

Thanks a lot for your support.

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Mayan EDMS Organisation Concepts Introduction

Post by rssfed23 »

mcarlosro wrote:
Thu Jan 16, 2020 9:30 pm
- Documents are assigned always to a "Document Type", but do you assign them as well to cabinets?
Yes. All documents have to be assigned a Document Type but assigning them to Cabinets is optional.
Cabinets is an extra layer of organisation. Document types aren't really massively useful for organisation because all documents of a specific type (such as "letters") go into the same document type.
- I see a lot of repetitive information in document type, cabinet, tags. Can explain the reason?
They all serve slightly different purposes.
Every document needs a document type. The reason they exist is because as you see when you create a document type you can have the OCR, Parsing, Metadata generation etc enabled, disabled or customised for each document type.
Where you might want OCR on the document type "Letters" you likely wouldn't want it on "Emails" as they're already searchable by the Parsed data.
Conversely, it allows for fine grained permissions. You it means you can allow people in sales to see the document type "Quotes" but not see the document type "Payslips", which would be for HR.

Cabinets were a feature request from a paying customer that wanted a windows-explorer style interface to manage document hierarchy, so it was added to benefit the overall community as the code was already done.

Tags are quick ways to add a way to categorize documents and they're useful when creating indexes as well.
- I'm starting with Payslips and Health information (analysis and reports). Regarding Health, I see a document type: Medical and a tag, and nothing else. also noticed Helthcare under cabinets. Why don't you keep the same terminlogy?
No real reason. I use this for home use so it really doesn't matter what I call them as long as I know what it means.
I really should never have created a document type "Medical" as that's not a physical medium (which I called out in my post).
If I was a company doing a lot more documents then I would seperate medical out into its physical mediums and use document types such as "Test Results", "Radiological Scan", "Genomic File", "Report", etc.
- Can you explain a bit about Payslips? Do you use Employment cabinet?
I have a dedicated document type for Payslips because they're their own physical medium/type of document in real life.
I don't actually put my payslips into the Employment cabinet. There's a dedicated Cabinet for payslips called "Payslips". But I also add them to 2 other cabinets: "Future visa evidence" and "Financial > Income".
- I was expecting initially to have a lot more metadata fields. Are you removing metadata fields by adding tags?
Tags are completely separate from metadata and do not remove anything within Mayan. BUT; where I'm using tags it helps me use indexes to build everything I need to. In many ways that removes part of my need for metadata fields.
Mayan doesn't come with any built in metadata fields. That's entirely up to the user to define what metadata fields they need based on their use case.
Mayan does come with File Metadata fields (in the document view you'll see metadata and file metadata) but file metadata is for what is grabbed by the Exif utility about the file itself nothing to do with content.
I don't really bother much with metadata in my home environment as I don't need it and there's no way to automatically generate any fields I would like (yet). As you have to enter metadata before the document get's selected it would mean I'd have to type it in/choose from the dropdown every single time. That's why I only have company and tax year metadatafields. Those only apply to specific document types because I don't really need them for everything.
For my specific use case metadata fields would add a fair amount of workload to me without much gain as I've automated everything else with indexes and workflows.
But for your use case they perhaps might be vital. But it would be wrong for Mayan to ship with default metadata fields out of the box as there's countless ways someone can use Mayan.
Please don't PM for general support; start a new thread with your issue instead.

mcarlosro
Posts: 11
Joined: Fri Jan 10, 2020 3:23 pm

Re: Mayan EDMS Organisation Concepts Introduction

Post by mcarlosro »

Ummm, workflows for personal use. Isn't that overkill?

What workflows do you have?

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Mayan EDMS Organisation Concepts Introduction

Post by rssfed23 »

mcarlosro wrote:
Thu Jan 16, 2020 10:36 pm
Ummm, workflows for personal use. Isn't that overkill?
It's entirely down to an individual users use case if workflows are appropriate. You'll see I made a guide the other day on workflows - that was in response to a home user needing to be emailed when someone else in the family uploaded a document.

I just don't have a need for document metadata for what I do, but I do have a couple of workflows that add metadata and help me bulk process documents after upload (so I don't need to manually add metadata to them during the upload stage).

It's also worth remembering that my use case is a bit different from the average "home" user - I work on Mayan EDMS as a project with the team so need to be knowledgeable and be able to test any part of Mayan EDMS, including workflows :)

Again, it all depends on what someone wants to use Mayan EDMS for. I have 8 HPE DL360p's in my Garage running various self hosted apps. For some users that's overkill but for me I could do with a couple more.
Everyone has a slightly different use case which is why I posted the initial guide; to help people understand the basic concepts so they can figure out how it might fit into their environment and what they want to do.

Anything beyond those basic concepts..well that's what the book is for!
If they're deploying Mayan for their company/organisation then I'd recommend they check out our support plans and consulting packages and pay for one of us to help them identify their use case and help their organisation get the most benefit out of Mayan EDMS.
Please don't PM for general support; start a new thread with your issue instead.

stiphy
Posts: 2
Joined: Sun Dec 29, 2019 7:12 am

Re: Mayan EDMS Organisation Concepts Introduction

Post by stiphy »

Okay reading the concept introduction is helpful for me to see how I might implement the Mayan EDMS for my home use.
To further my user case I am looking into what I consider metadata, I would like to be able to add a note field (sometimes a bill will have something of note and in the future I would like quickly find it. Using tags, which are essentially binary can become excessive.. i.e. how many single use tags will I be creating for these exceptions). I note that a comment field is available but if I understand correctly this is for different revisions of a document, which is great but not what I am looking for. In addition I was thinking of the case of a system for rating files, ie. star rating, where a file (i.e. picture, cookbook, textbook, scientific paper) could be rated for future reference. At first I anticipated using metadata to enter this information however reading that metadata must be included while uploading a document appears to be too limiting for my case, my evaluation of a document could change over time. In addition reading how to create custom metadata, if I understand correctly Mayan documentation talk of using Django templates, is beyond me at the moment ( I am prepared to study Django to implement something for myself - and to pass it on to others if I find it works to my satisfaction). Is this so?
Are there any plans to simplify the creation of different types of metadata or at least make it mutable? Or perhaps it already is and I have interpreted the online guides and the pdf book incorrectly. Any help or a push in the appropriate direction would be greatly appreciated.

Stephan

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: Mayan EDMS Organisation Concepts Introduction

Post by rssfed23 »

Hi Stephan.

I agree; tags wouldn't be right for your use case here.

You can add comments to documents (the comments button on the right hand side) and they can be used for whatever you want really. You can also use the document description for notes if you want. I guess it depends on how you're going to use the notes afterwards.

You could add a rating using a numerical system yes, and there is work ongoing to improve the document upload flow so that metadata can be added after a document has been selected in the wizard/viewed in staging rather than having to be beforehand, but this will take time.

If you added notes as a metadata item you don't want sentences in there as they'll be too big to display on the documents list, but it'll likely be the most flexible way to do it. Metadata or comments or description can all be used in indexes later on if you need them as far as I know.
Mayan does use django templates yes, but although Mayan is a Django project it does push Django to its extremes and many concepts that apply in djangoland don't apply to Mayan. The templates side of things should be much simpler though than the actual Mayan code.

A potentially good idea would be to create a ratings app. Then you could use stars instead of numbers and give it room to grow and store different ratings for each document version so that old ratings are kept (as you mentioned tracking over time). I recommend having a browse of the dev documentation in the docs to see if that's something possible (but it is by far the most complex of the use cases). Metadata values are mutable - if by mutable you mean the value of the metadata can be changed once it's set. You do that by going into the document view clicking metadata and then you can change whatever value of whatever metadata label you want.
Please don't PM for general support; start a new thread with your issue instead.

Post Reply