Hierarchy / Organizational assistance

Questions, comments, discussions. Over time certain topics might be moved to their own category.
Post Reply
User avatar
jeremymorel
Posts: 8
Joined: Wed Aug 22, 2018 1:38 pm

Hierarchy / Organizational assistance

Post by jeremymorel »

Greetings to all! I am honored to find that I'm the first to use the new forum (aside from our esteemed admin, Mr. Rosario! :D )
I'm a developer, and have been following this project for some time, (and someday hope to contribute some value.) I have decided to implement version 3+ at home in order to go paperless, and have encountered some challenges. I am using the Docker implementation, (which I know has some small differences). I've searched the current (former?) forum extensively to try to find an answer, but can not, so please forgive me if this is a re-tread.

My question relates to both overall hierarchy and functionality. It is entirely possible that I am approaching organization wrong, so please allow me to express my intent, and feel free to tell me if I've got it all wrong!:

I'd like to start mass-scanning all of my documents to watched folders and ultimately have them land in a structure that is logical for both browsing and searching. After assessing the patterns available, (Cabinet, Document Type, Tags, Metadata etc.), I'm not certain where to base my focus, but believe that Type and Tag (or create a metadata type such as "Sub category") may be appropriate. In this case, a document type would be "Financial", then a tag could be "Receipt".

My setup is so that I will scan documents to appropriate watch folders based on document classification. So far, I've established document types for, "Financial" "Home", and "Medical". I would like to establish watch folders with a more detailed level of specificity, (i.e., "Receipt", "Tax Information" or "Banking", and have documents which come into each folder be processed and classified with a document type (in this case "Financial") and then receive the subcategory classification (i.e. Receipt).

I've tried every thing I can think to organize and accomplish this. I've even installed the document_analyzer plugin, (and have it working, but not sure it's right!), but I'm not certain it will help me to accomplish what I'm trying to do. Perhaps I'm overlooking a functionality in Metadata, Cabinet or even workflow? I'm anxious to get going, but want to get it right!

Could someone kindly take pity on me and tell me how I should handle this?
JeremyMorel
"Paper Intolerant" :lol:

KevinPawsey
50 Posts
50 Posts
Posts: 85
Joined: Wed Aug 22, 2018 2:52 pm

Re: Hierarchy / Organizational assistance

Post by KevinPawsey »

Hi,

I too am implementing this at home for exactly the same reasons as you... so far I am having "fun" with getting a stable version working properly... but that is a topic for another thread ;)

So far, I have come to the conclusion that with Cabinets/Doc Types/Tags, here is how I am implementing it:

Cabinets
I am using these like filing cabinets, so I have these as 'per person'

Tags
These are the "folders" within the cabinet, so medical, receipts, financial, etc

Doc Type
This I am using for how long the document is to be stored for, ie:
  • Indefinite storage - long term storage, such as certificates, pension documents, etc
  • 18 month storage - for such things as warranty certificates, or proof of purchase
  • 3 month storage - for appointment letters
The retention for the document types is reflected in the name, and after that time the documents then go into recycle bin for a period of time before complete deletion.

This is how I am looking at it at the moment... but this may change... I have literally been looking at this for a week now, so still unsure how to implement it. I haven't looked at workflows, or anything like that yet, but I am figuring that something could be done with a trigger of once a document is imported of Doc Type X, tag with Y and file in cabinet Z... maybe?

Anyway, would be interested to hear how others are implementing this in their homes :)


Kevin
Running Mayan-EDMS on: OpenMediaVault, (Docker plugin), on x86 dual-core

User avatar
jeremymorel
Posts: 8
Joined: Wed Aug 22, 2018 1:38 pm

Re: Hierarchy / Organizational assistance

Post by jeremymorel »

Hi Kevin,

Great to get your perspective, and thank you for sharing! Yes, I'm not certain that I understand the workflow feature well enough to perform the automation steps I need. After playing with it, I'm not sure it's intended to do what I'm looking for. (Or I'm just dense, which is more likely! :roll: ). Best I could see for organization actions might be something like a POST action to the REST API, but that might be an overreach. I was hoping I was just missing something simple!
JeremyMorel
"Paper Intolerant" :lol:

KevinPawsey
50 Posts
50 Posts
Posts: 85
Joined: Wed Aug 22, 2018 2:52 pm

Re: Hierarchy / Organizational assistance

Post by KevinPawsey »

Something else that I am going to have a look at is the Indexes, as apparently that is a way of dynamically "filing" items... just that I am not sure how well that would work.

If you look at the default index, it is done via Date as follows:
[YEAR] > [MONTH] > [DAY]
This folder tree is automatic built as documents are added.

What I was wondering for the indexes is if it is possible to index against OCR'd information... so for instance, on the first occurrence of a list of names (Mr A, Mrs A, Bob A, Jane A), then maybe by a list of companies (British Gas, nPower .... other energy companies are available! lol), so it would then build up a list of name/company/month Index.

I think more understanding of indexes may be required, but there is a start here:
https://wiki.mayan-edms.com/index.php?t ... ex_recipes

Maybe then it will just be a case of changing the way we view the documents, instead of going to "All Documents" or "Cabinets" all the time, then it is a case of referring to the index?

.. just random thinking :D
Running Mayan-EDMS on: OpenMediaVault, (Docker plugin), on x86 dual-core

User avatar
rosarior
Developer
Developer
Posts: 522
Joined: Tue Aug 21, 2018 3:28 am
Location: Puerto Rico
Contact:

Re: Hierarchy / Organizational assistance

Post by rosarior »

Hi jeremymorel, you are indeed the first in the new forum! Thanks for supporting the new platform.

Document types are meant to be atomic units. They are should mirror the actual physical paper type and not the usage of the paper. Some examples of document types are: invoices, receipts, prescriptions. The categorization units like: health, business, accounting build on top of document types and can be implemented with cabinets or indexes. Document types are the smaller unit of classification and everything else above them are containers for document types.

Here are some document types I use in my personal install:
  • Magazine (downloaded free PDFs like Odroid magazine). No metadata.
  • Receipt (Purchase evidence). Metadata: Date, Vendor.
  • eBook (Purchased eBooks). No metadata.
  • Manual (Downloaded or scanned manuals for product we own). No metadata.
  • Invoice (Invoice for products we purchased or services). Metadata: Data, Invoice Number, Vendor
  • * (For eye wear, repeat medicine). Metadata: Author, Date, Patient
  • Fine (Penalties, traffic violations). Metadata: Regarding, Amount, Paid.
  • Property title (scanned titles in case original is lost). Metadata: Item
  • Letter (Scanned letters recevied). Metadata: Sender, Recipient, Topic


Once scanned and with their specify document type they are then sorted into indexes by means of their metadata values. I use tags to track progress or state of a document if their number of state is small, otherwise I use a workflow to keep track of what's been done with the document and what remains to be done.

Hope this clears some of your questions.

User avatar
rosarior
Developer
Developer
Posts: 522
Joined: Tue Aug 21, 2018 3:28 am
Location: Puerto Rico
Contact:

Re: Hierarchy / Organizational assistance

Post by rosarior »

@KevinPawsey Yes that's the way to store documents in Mayan. If you need more control over the document type you can create similar document types for each purpose. Example: An invoice might have different fields if it is an invoice you make or one you receive. So create two invoice document types: Posted Invoice and Received Invoice, one for receivable and one for payable.

Cabinets and indexes are basically the same. Cabinets are multi level folders that are created by hand. Indexes are multi level folders that are created by the system based on a template and the value of the document properties including the attached metadata values.

Cabinets are good when your document count is low. After a few hundred document you reach "classification fatigue" and automation is the only way forwards.

Workflows are for storing a sequence of states over which the a document will progress. They were meant to avoid having to attach a tag and then remove the tag to add another one to keep track how a document has been processed so far. Workflows store a series of states to help you know the "status" of a document. Moving a document from a workflow state to another used to be purely a manual operation.

Over time we've added more features to workflows that allow them to work like a small programming language in itself. The workflow actions execute an action when a state is reached or when a document exists a state. There is a workflow to attach a tag to a document, there is another to do a POST request to an external IP address. This last one is used to have Mayan trigger processes in external systems based on the state of a document, like triggering the billing process of an accounting system when an invoice is scanned in Mayan.

The workflow triggers allow moving a document from one state to another automatically based on the events of the document. For example: move a document from "scanned" to "billed" when a tag is removed from the document. Workflow state actions and transitions triggers are new features and are still evolving. We are working now on workflow transition trigger filters to have the trigger move the state of the workflow on certain conditions. For example: Attach a tag if there is a specific word found in the OCR text.

ovidiu
Posts: 12
Joined: Tue Aug 21, 2018 12:53 pm

Re: Hierarchy / Organizational assistance

Post by ovidiu »

Thanks everyone for this enlightening discussion. Seeing that this is a general discussion about possible implementation of hierarchies, let me show you how I did it, maybe someone can give me feedback or ideas for improvement.

I intend to store ISMS documents for a couple of ISO 27001 projects. Since some documents are valid for every project and some are for a specific scope only I started with cabinets.

1 cabinet called ISO Projects (so in the future I can have other cabinets for other unrelated uses) contains documents which apply to all ISO scopes which are represented by 3 cabinets located inside this main cabinet.These 3 cabinets contain the documents which only apply to each of these 3 ISO projects.

Regarding document types, I called the 2 I use "ISO documents Workflow A" and "ISO documents Workflow B" to differentiate between the attached workflows. They are virtually identical types of documents except they get approved by a different role.

Every document uploaded in here (or rather belonging to one of these 2 document types) has a workflow attached.

I intend to use tags/metadata for security tagging each uploaded document levels with the corresponding security level but I am still clueless how this works (best). (after reading the handbook I think metadata types are better for security levels as one document can have multiple tags, which wouldn't work for security levels).

User avatar
jeremymorel
Posts: 8
Joined: Wed Aug 22, 2018 1:38 pm

Re: Hierarchy / Organizational assistance

Post by jeremymorel »

Hi rosarior, and thank you for your reply. Knowing the way you use it, and intended for it to be used, is very valuable insight!
While I have your attention, I want to be sure to say THANK YOU to you and those who have helped in the development of this amazing tool. I understand that sometimes we can forget to show appreciation while we're too busy complaining and making demands. :)

Since you mention workflow, can I ask: is there a guide that speaks more specifically to how to create/use the workflow functionality? I have attempted and failed several times to create one. I am not certain that I understand the concepts of 'initial state', 'state', and 'action' as you have intended them.

Again, I really appreciate this --all of it! :!:

Thank you to everyone who has responded. It's really useful to see how others are structuring their systems!
JeremyMorel
"Paper Intolerant" :lol:

ovidiu
Posts: 12
Joined: Tue Aug 21, 2018 12:53 pm

Re: Hierarchy / Organizational assistance

Post by ovidiu »

jeremymorel wrote:
Thu Aug 23, 2018 11:30 am
Since you mention workflow, can I ask: is there a guide that speaks more specifically to how to create/use the workflow functionality? I have attempted and failed several times to create one. I am not certain that I understand the concepts of 'initial state', 'state', and 'action' as you have intended them.
I think I can explain this as it works for me. Here's my example:

Workflow A
initial state: "waiting for review"
transition: "review passed" leads to state: "waiting for approval"
transition: "review failed" leads to state: "in progress"
state: "in progress"
transition: "edited" leads to state "waiting for review"
state: "waiting for approval"
transition: "rejected" leads to state "in progress"
transition: "passed approval" leads to state "approved"

I hope that helps.

User avatar
rosarior
Developer
Developer
Posts: 522
Joined: Tue Aug 21, 2018 3:28 am
Location: Puerto Rico
Contact:

Re: Hierarchy / Organizational assistance

Post by rosarior »

@ovidiu Wow! Just looked up ISO 27001. That is really cool! I would love to hear more that. It would be a good case study.
Yes, I concur, metadata types would be better for security levels. If the security levels are finite, you can add them in the lookup field and the metadata widget will change from a textbox to a dropdown that restricts entry to the items in the lookup field.

@jeremymorel Thank you for that. We appreciate it very much and we thank you for taking the time to use and understand the project. It has a steep learning curve and takes time to get it working just fine because it is very different than most solutions. So thank you all for taking the time to learn its paradigms and offering feedback.

To expand on ovidiu's explanation, workflow are implemented in Mayan as a finite state machines (https://en.wikipedia.org/wiki/Finite-state_machine). Ovidiu's example is a real life implementation of what's possible with a workflow. They allow translating business logic into a series of states. With the addition of state actions and transition triggers, the workflows in Mayan are no longer just informative but can be part of your actual business automation process.

The graphical representation of a workflow (or a finite state machine style in Mayan's case) is similar to a flowchart. This is an example:
It has two states. The states are the circles. The arrows are the transitions. The circle with the double border is the initial state. The user executes the transitions and the current state switches.
You can get these graphical representations using the "Preview" button of the workflow setup view. They are great for debugging your workflows.
Screenshot from 2018-08-23 19-20-29.png
Screenshot from 2018-08-23 19-20-29.png (24.69 KiB) Viewed 2373 times

Post Reply