Mayan EDMS Workflow Concepts Introduction and an Example Workflow to Send an Email on Every Document Upload

[rssfed23]

Based off a workflow for sending an email upon document upload as requested by a user at viewtopic.php?f=7&t=1607.

The first part is the introduction and terminology overview (which is vital for understanding workflows). If you want to skip straight to the demo then head to post number 2.

Introduction

Workflows are one of the most powerful features of Mayan EDMS. When utilised effectively they can be used to automate almost any business process, as well as to customise Mayan EDMS and the documents within it to your liking.

However, with great power often comes an increased learning curve. Workflows can take a bit of practice to optimise to your requirements. This guide is designed as a primer or introduction to workflows. I'll walk you through the basic concepts of the workflow process and how it all fits together, and then we'll produce a sample workflow to send an email upon every document upload to a user as a simple example that you can modify to suit your needs.

Note: Everything here, along with a whole lot more, is covered in the Exploring Mayan EDMS Book. I encourage everyone to consider purchasing a copy to get a more complete picture of workflows and a lot more. This forum guide is aimed as companion material to the book. It also has the added benefit of directly supporting the project, which allows us to continue adding the features you desire, fix the bugs you need fixing and provide the free support on the forum (I hope) you benefit from .

Very important note: The Mayan EDMS team have experience designing and implementing workflows to solve all sorts of business needs. You can give your Mayan EDMS installation a quickstart by considering a Mayan EDMS support plan, where you can get expert advice on workflows or any other topic relating to Enterprise Document Management. The team are also available for consulting engagements to get you up and running even faster. They're able to set up the foundations and produce documentation & training material for your teams so that they're empowered to set up their own workflows relevant for their own team & processes. That's in addition to helping you set up a production ready environment suitable for your critical workloads. Head to https://www.mayan-edms.com/support/ or email sales@mayan-edms.com to enqure about this.


Workflow terminology

It helps us to first define some terms which I'll refer to continually throughout the tutorial.

Workflow: Implemented using Mayan's Finite State Machine, Workflows are a collection of States, Transitions (between States), Actions (to take when a Transition happens) and Triggers (that invoke a Transition). The ultimate purpose of a Workflow is to perform a process. That could be a Business Process (such as approving a document), a Change (such as adding a tag to a document) or a Transformation (to modify a document in some way, such as adding a redaction). A Process can really be anything you desire to automate. In the example shown below the Process is "send an email whenever a user uploads a new document". Some other good examples:

- Change a documents ACL to be world readable when the metadata label "Visibility" has it's value changed to "Public"
- Email Accounts Payable when a document type "Purchase Order" is created, but only when that document has the tag "Pending Payment"
- When the metadata value of "Paid" is added to any document of the type "Pending Invoice", change the document type to "Paid Invoice", add a tag with the value of "Paid" and add today's date to the Documents metadata with a metadata label "Date Invoice Paid"
- Any document uploaded into the document type "Hiring Offer" get's automatically tagged as "Pending". When a document is first marked as "Pending", email the L1 approver the document and ask for approval. Once the L1 approver has entered the workflow and marked the document as "Approved", change the progress to 50% complete and email the L2 approver for approval. Once the L2 Approver marks the document as "approved" in the workflow, email a copy of the document to Human Resources and also email a copy of the document to the email address defined with the metadata label "Prospective Employee Email Address". Once Human Resources have confirmed acceptance of the document by the Prospective Employee, they mark the document as approved in the workflow, at which point the tag "pending" is removed, today's date is added to the metadata label "accepted date", and the tags "archived" and "accepted" are added.


States: A state is the current position of a Document within a workflow. Being a finite state machine, a document can only be in one state in a workflow at any point in time. Each workflow should have an initial state that the document starts in, and as various Transitions occur (which are invoked by Triggers) a document will move through the various states. States are associated with a % of completion of a specific workflow. As a document moves through a workflow it progresses towards the final state, which is 100% completion. However, different transitions can move a document towards the 100% completion state or take it backwards to a previous % completion state. One example of this:
1. Document enters the workflow at 0% Completion. This is marked as the initial state.
2. A document gets approved by a L1 manager and a Transition moves it to another state which is marked as 30% Completion.
3. The L2 manager rejects the document at this stage in the workflow, which depending on the requirements of the process could move the Document to the 100% completed state but add a Tag "Rejected" (using an action) OR it could move it back to 0% Completion and add the tag "Rejected" ready for someone to upload a new version of the document and start the process again.


Actions:
When it comes to States, a Document is either going to be stationary in a State, entering a State or exiting a State. This is where actions come in.
An action can happen when a document Enters, Exits a state or both. You can assign multiple actions to a state. Each action invokes a specific automated task. Here's the full list of possible actions that can happen when a document enters or exits a state:



One of the more powerful actions is the "Perform a POST request". This allows you to invoke ANY API, either one of Mayan's or any external API.
This means you can integrate your actions and workflows into any existing piece of software that has a workflow.
To give an example I use at my Organisation:
1. A customer orders from our website. When the order is completed an invoice is generated which is uploaded into Mayan into the "Pending Orders" document type. It's a customer order form for a cloud based product we offer. In the document metadata they've described various things about their purchase (such as size of the virtual machine and AWS region they want to launch the product into).
2. The workflow initial state is 0%. On entering that state (which happens automatically to all "Pending Orders", there is an Action that assigns the "Payment Pending" tag automatically
3. Our payment processor can send webhooks upon successful payment. We have it set up so that when payment is successful a webhook to the Mayan EDMS API is called that adds a tag "Payment Confirmed"
3. We have a transition trigger set (more on that below) that monitors for new Tags added. When it spots the "Payment Confirmed" tag added to the document it transitions it to the 50% Complete "Provisioning" State
4. As an entry Action to the Provisioning State, we've set up the "Perform a POST request" Action to send a POST to the AWS API that deploys the infrastructure. As everything in Mayan EDMS is templateable, we use the values the customer provided in the order (which is saved as document Metadata) as some of the variables in the POST request
5. The document is in the Provisioning state. A second Action we have as a document enters that state is to email our service desk that a new order is provisioning. That team will go and investigate the built environment to do a manual check before signing it off. During the time it takes them to get the email and check the environment the previous POST has already completed successfully and it's build
6. They manually progress the document to the 100% Provisioned state. This causes the document to Exit the Provisioning state and we have an Action set up on Exit from the Provisioning State to send an email to the customer (using metadata from the document again to get their name and email) saying their environment is ready to go and here are their credentials.

That's one example of what looks like a complex workflow, but only took a hour to implement and works reliably. It goes to show how diverse workflows can be!


Transitions:
Transitions move Documents from one state to another state. It is how we take a document from 0% Completion to 30% Completion to 100% Completion.
Transitions always have an Origion State (what state the document is transitioning from) and a Destination state (what state the document is transitioning to). Transitions do not have to be linear (as in 10% > 20% > 30% etc) and can have any origin or destination state. A workflow can have as many transitions as you require. This gives an unlimited number of combinations of states, transitions and actions and allows for complete customizability to suit your workflow process needs.

Fields: Transitions can be assigned Fields. These are pieces of additional metadata that provide context to the transition. They can be used for example to add internal notes to a transition or ask someone executing a transition a question. You can use the value of a field in any other part of a workflow, and the field data will always be associated with a specific workflow execution for a specific document but will not become a part of the document itself.


Transition Triggers:
Users can invoke a Transition on a document manually, but Transitions can also be configured to happen automatically. This is one of the core features of Mayan's automation.
Triggers are events that invoke a Transition (which moves a document to a different state, potentially invoking that states exit or entry action as it does so).
Almost anything that creates an event in Mayan can be used as a transition. Here's some examples taken from Mayan:



A transition can have multiple triggers enabled for it and any time any of those enabled events happen the transition will trigger. Remember a workflow can have multiple transitions associated to it and each transition can have any number of triggers that invoke that transition.

As I hope you're starting to see, the potential power and sheer flexibility of Workflows is why I recommended Mayan EDMS Support and Consulting earlier on. You really can achieve anything you want to with workflows.


Events:
Just like most things in Mayan, an event is logged any time a transition is triggered or action happens as the result of entering or exiting a State. You can subscribe to these events to be notified when they happen.

[rssfed23]

Now we've talked through what Workflows are and the basic concepts, we're going to walk through creating a real life workflow so you can get a feel for the basics.

Workflow: Send an Email upon Document Upload
This is a very simple workflow that will send a user you define an email whenever a new Document has been uploaded. As long as you have an existing Mailing Profile defined it should work for you to. As with all guides, click the picture to enlarge.

Go to System > Setup > Workflows.
Click Actions up the top right then Create workflow.

The label is a descriptive name to give the workflow so other users can easily see what the workflow does.
The Internal name is a shorthand reference that can be used elsewhere in Mayan. It should be concise and can't contain special characters.

Give your new workflow a suitable Label and Internal name:



We now need to choose which Document Types can run the workflow. Click document types on the newly created workflow.



I want this to apply to all Documents so I clicked Add all.

We've now got our blank workflow assigned to some document types. As mentioned in the previous post, any workflow is going to have some states assigned to it, so click the States button on the right hand side.

Initially we have no states, so click the Create state button

We need to create our starting state first.
As mentioned, all states have a % Completion value assigned to them. I'm going to give this one 0%. It doesn't matter really what % value you give the State as long as the Initial box is ticked. This box tells Mayan EDMS that it is the first step in the workflow and is the step documents should enter on.



As this is marked as Initial, all documents uploaded into the document type we selected earlier will enter this state.

Although we're calling it "Document has just been uploaded", it's important to note that we haven't actually set up a Trigger on document upload yet. Mayan will automatically put the document here regardless.

Now we set up a second state. We mark this one as 100% Completed and make a note that the email should be sent.
Click Actions up the top right again then Create state to create our second state:



We can save that second state. We now have both our states listed on our "States of workflow" page.

Now we need to add an action. Remember from earlier that Actions are events that can happen as a document enters or exits a state.
For the sake of sending an email, it really doesn't matter if the email is sent while exiting the Initial state or entering the Completed state, the net effect will be the same.
I'm going to go with adding the action to the completed step on Entry though as it makes more sense with the labels I've chosen.

So, on the "Document uploaded email sent" state click Actions, then Create action (and remember we can have multiple actions apply as a document enters or exits a state. Cool, huh!?!)
On this page we choose our Action. As mentioned earlier, this can be any number of tasks, but for this example we want the "Send email" action.

Select Send Email in the drop down box and click Submit

We're now taken to the screen where we define our action.

First, give it a Label (that describes the action).

Then, choose On entry for the When.
We're choosing On Entry for our Completed state so the action happens when it enters the completed state. If you want to choose On exit then you would apply that action to the first initial state instead.

Choose a valid Mailing profile so Mayan knows what Email server to use.

Define a Recipient
Remember these fields can all be templated as well. You could for example send the email to an email addressed noted as metadata in the document.
If for example you have a Metadata Label called "email" to template this field and send this email to that address you would use the template:

Code: Select all

{{ document.metadata_value_of.email}}

Then Mayan will send the email to whatever email address is put into the metadata value box for the label "email".

Choose a Subject for the email. Again; this could be templated. I'm going to use the Document Label (by default the filename) of the document in my Subject as well as the Document type the user choose. So my subject line is:

Code: Select all

New {{ document.document_type }} Uploaded to Mayan: {{ document.label }}

Finally, you need to choose a Body for the email. Again; this can be templated!

Personally I want to make sure I have a vague idea what people are uploading to make sure there's nothing inappropriate, so as well as a sentence in my email Body I'm also going to include a bunch of details about the document.
I'm also going to grab some values from the document file metadata (from the exif driver) to get the filetype (as people shouldn't be uploading EXEs to MY Mayan EDMS )
The template for this would be

Code: Select all

This is an email from Mayan EDMS - The best EDMS on the planet. A new document has been uploaded. <br>
The document title is {{ document.label }} <br><br>
The document language is {{ document.language }}  <br>
The document was uploaded with the following tags: {% for tag in document.tags.all%} {{tag.label}} {% endfor %}  <br>
The document was added to the following Cabinets: {% if document.cabinets.count == 0 %} None {% else %} {% for cab in document.cabinets.all %} {{ cab.label }} {% endfor %} {% endif %}  <br>
The document was added to Mayan on {{ document.date_added }} <br>
The document is of the format is {{ document.file_metadata_value_of.exiftool_FileType }} <br>
The document's exif data says it was initially created on  {{ document.file_metadata_value_of.exiftool_CreateDate }} <br>

The templating engine is the exact same as used for indexing, and is really powerful and dynamic as you can see. The <br> are to insert line breaks in the html email that's sent.

ProTip: You can use the Template Sandbox to figure out what template string you can use. If you view any document, then click Sandbox on the right hand side you can experiment with the various pre-filled template options, modify them and see what result that template string would give. Very, very, VERY handy for writing workflow or index templates

This is what my action looks like once all that is filled in:




So, We've now got 2 states and an action to actually send the email. We know a document will enter the Initial state automatically as we ticked the Initial box earlier, so now we need to set a Transition to move from 0% to 100% and then apply a Trigger to that transition so that it happens automatically.

Click Transitions on the right hand side, then Create transition

We give our Transition a name, and choose our 0% State as the Origin state and our 100% State as our Destination state.
As mentioned in the earlier post, a transition can go from any state to any other state!



Click Save.

Now the final piece to glue it all together: The Transition Trigger.

Transition Triggers are what trigger the automatic invocation of a transition.

Click Transition Triggers on the transition you just created.



This is where we can choose what events trigger the transition we just created in Mayan. Make sure you click Submit at the bottom before continuing

What you choose here really is up to you as we can send an email on anything. However, a couple of points to note:
- If you choose Document created, this will trigger the email as soon as the document is created but before Mayan has parsed, run metadata extraction/exif or OCRd the document. It also will only trigger when the document is created, so any new versions uploaded of the same document will not trigger the transition
- If you choose New version uploaded, this should cover both the initial document creation (as the "new version" is version 1) as well as subsequent updates to the same document, but again will be executed before any metadata/OCR/parsing has been completed
- If you want to be notified when a document is "Ready", this depends on your definition of "ready". For some that may be when OCR has finished, for some once the file metadata has generated. It really is up to you.
- A transition can only be triggered once (but you can of course have another transition set to reset the state back to 0% again if you want - this is how complex workflows form and it's very powerful)

For this exercise, I'm going to go with "Document version file metadata processing finished". My reason for this is that it will trigger on every new version of a document as long as I have another transition that moves the document back to 0% upon "new document version uploaded". If I don't have this additional transition, the workflow will get to 100% and then stay there as we won't have a transition to reset it back to 0% again. Make sense?

This is all optional of course, as you may not care about future versions of a document. If you don't want to have a second transition returning the document back to 0% on a new version being uploaded then skip to the next post, where we will run the transition.

If we want a workflow to trigger on every new document version we need a transition and trigger to reset that document workflow back to 0% again as it can't go from 100% (email sent) to 100% (email sent) it has to go back to 0% first. We'll set this up in a second.
By going with metadata finished it also means the document is largely ready and indexes will soon be updated but we don't have to wait for OCR to complete.

I'm going to click Actions then Create transition and create a new transition with the Label "Reset to 0 on new document version upload", but with the two states reversed to take things back to 0%



Click Transition triggers on the new transition we just created and click yes against Documents - "New version uploaded" then Submit at the bottom

What this will do is only reset the Workflow back to 0% when a new version of a document is uploaded.
This ties in nicely with us using the Document version file metadata processing finished above because if we had new version uploaded set for that trigger we would end up in a loop when a new document version is uploaded! - New document version uploaded will ALWAYS trigger before Document version file metadata processing finished!
See how this is starting to take shape?

Let's go to the next post to execute the workflow (as I've used my quota of uploaded images in this one )

[rssfed23]

So we've now got everything set up ready to go.

We've:
- Created a workflow to send emails on new document uploaded
- Created 2 states. 0% and 100%
- Created an action so that when a document enters the 100% state an email is sent using a template we defined using values from the document
- Created a transition to trigger an email once file metadata has completed processing
- Created a transition to trigger when a new version of a document is uploaded, to reset the process to 0% ready to hit metadata completed again once a new version is uploaded so that we get emailed on EVERY new version of a document as well as the original upload.

Now for the moment of truth! - Let's upload a document *eek!*.

I'm going to use "Upload Test" as the Document type



Assign a couple of tags:



Put it in my "Military Documents" Cabinet:



And choose a (totally hot) picture of myself as the actual document



Cross my fingers and click Open to start the upload.

Then VOLIA:



Pretty neat! - The templating worked perfectly

We can now go to the workflows section and see our document there:



We can also click states on the right and see how many documents are in each state (and drill down into viewing all the documents in a specific state if we want to):




That concludes our quick Workflows tutorial!
I hope you can use this as a foundation to build out some epic business processes with workflows.

If you want to see this workflow in real life then you can head to our demo environment: https://mayan.app - and login with the credentials shown (demo/mayan-edms).
You can browse the workflow I set up here and see the workflow documents and what documents are in each state. You can also see the basic set up of the workflow if you go to it via settings. However, as this environment is read only you won't be able to get into the actual configuration page of the transition or actions themselves as these pages only exist when a user has the edit ACL applied (and the demo user is read only for security purposes).
If you would like a more fully featured demonstration with the team as you're considering purchasing a support plan, then don't hesitate to reach out at sales@mayan-edms.com and we can organise a trial user for you as well as show a more fully featured demo.

Please, comment on the workflow shown below and please share any of your own awesome workflows. The sky really is the limit with Workflows (and they're only going to get even better with v4).

Let me know how your own experiments got on and if this worked for you

Happy Workflowing!
Rob

[Steve]

First of all many thanks for this tutorial, very clear and understandable.
I've got a problem anyway; copying your email template, no matter in which cabinet I upload my new document, on the received email, cabinet, format and created on informations are set to None;
this is what I get:

This is an email from Mayan EDMS - The best EDMS on the planet. A new document has been uploaded.
The document title is Test.jpg

The document language is eng
The document was uploaded with the following tags:
The document was added to the following Cabinets: None
The document was added to Mayan on Jan. 26, 2020, 5:19 p.m.
The document is of the format is None
The document's exif data says it was initially created on None

In Mayan, the document is uploaded in the wanted cabinet...

I have also tried Mayan demo installation as per your suggestion, to check what I've done wrong using test materials, but with the provided credentials (login page) it is impossible to check workflows internal working or to create new ones.

Thanks in advance

Steve

[rssfed23]

Hi Steve.

Yes, the demo environment user is read only for security reasons. I've just looked and although the workflow does show the actual drill down into the action and templates isn't shown like you say. I'll amend the initial post to reflect that.

I've just logged in to copy the template again (being applied on the entry to 100% state):

Code: Select all

This is an email from Mayan EDMS - The best EDMS on the planet. A new document has been uploaded. <br>
The document title is {{ document.label }} <br><br>
The document language is {{ document.language }}  <br>
The document was uploaded with the following tags: {% for tag in document.tags.all%} {{tag.label}} {% endfor %}  <br>
The document was added to the following Cabinets: {% if document.cabinets.count == 0 %} None {% else %} {% for cab in document.cabinets.all %} {{ cab.label }} {% endfor %} {% endif %}  <br>
The document was added to Mayan on {{ document.date_added }} <br>
The document is of the format is {{ document.file_metadata_value_of.exiftool_FileType }} <br>
The document's exif data says it was initially created on  {{ document.file_metadata_value_of.exiftool_CreateDate }} <br>

I've just uploaded another document and that had the cabinets and type my end.

2 possible things to check:
On what trigger are you triggering your transition on? In the tutorial we use "Document version file metadata processing finished".
If you choose something like "document uploaded" that won't work for cabinets/metadata, because as soon as a document is uploaded it hasn't yet had the metadata generated (where the "type" comes from) or been added to the cabinet. Those happen as later stages. So check the transition trigger to make sure it's happening at one of the final stages such as metadata processing finished or document indexing finished.
That's my suspicion as to why it's not showing for you.
If you are already using "Document version file metadata processing finished" as the only trigger for the workflow and your template matches the above then something else is going on.

Some documents also don't have a "Type" reported by the exif driver and will always show as none. Emails for example won't have a type. Anything that's not a jpg, png, doc, xls or pdf is unlikely to have a type.
What you can do is go to the document you uploaded, then click template Sandbox on the right hand side and put in {{ document.file_metadata_value_of.exiftool_FileType }}
If it shows None then that's why - the exif driver isn't able to determine a file type for that type of file. For example, this html email:



If that does show a filetype and you've checked the transition trigger as described above, then there must be something else going on beyond the scope of this thread/forum.

[Steve]

My apologies!
This example workflow works like a charm....
I have made a mistake with trigger selection (although checked many times)

Thanks again

Steve

[kozmi]

rssfed23 wrote:
> Hi Steve.
>
> Yes, the demo environment user is read only for security reasons. I've just looked
> and although the workflow does show the actual drill down into the action and
> templates isn't shown like you say. I'll amend the initial post to reflect that.
>
> I've just logged in to copy the template again (being applied on the entry to 100%
> state):
> [code]This is an email from Mayan EDMS - The best EDMS on the planet. A new document
> has been uploaded. <br>
> The document title is {{ document.label }} <br><br>
> The document language is {{ document.language }} <br>
> The document was uploaded with the following tags: {% for tag in document.tags.all%}
> {{tag.label}} {% endfor %} <br>
> The document was added to the following Cabinets: {% if document.cabinets.count == 0
> %} None {% else %} {% for cab in document.cabinets.all %} {{ cab.label }} {% endfor
> %} {% endif %} <br>
> The document was added to Mayan on {{ document.date_added }} <br>
> The document is of the format is {{ document.file_metadata_value_of.exiftool_FileType
> }} <br>
> The document's exif data says it was initially created on {{
> document.file_metadata_value_of.exiftool_CreateDate }} <br>[/code]
>
> I've just uploaded another document and that had the cabinets and type my end.
>
> 2 possible things to check:
> On what trigger are you triggering your transition on? In the tutorial we use
> "Document version file metadata processing finished".
> If you choose something like "document uploaded" that won't work for
> cabinets/metadata, because as soon as a document is uploaded it hasn't yet had the
> metadata generated (where the "type" comes from) or been added to the
> cabinet. Those happen as later stages. So check the transition trigger to make sure
> it's happening at one of the final stages such as metadata processing finished or
> document indexing finished.
> That's my suspicion as to why it's not showing for you.
> If you are already using "Document version file metadata processing
> finished" as the only trigger for the workflow and your template matches the
> above then something else is going on.
>
> Some documents also don't have a "Type" reported by the exif driver and
> will always show as none. Emails for example won't have a type. Anything that's not a
> jpg, png, doc, xls or pdf is unlikely to have a type.
> What you can do is go to the document you uploaded, then click template Sandbox on
> the right hand side and put in [i]{{
> document.file_metadata_value_of.exiftool_FileType }}[/i]
> If it shows None then that's why - the exif driver isn't able to determine a file
> type for that type of file. For example, this html email:
>
> [url=https://ibb.co/9GRnn80][img]https://i.ibb.co/ZNyddLr/image.png[/img][/url]
>
> If that does show a filetype and you've checked the transition trigger as described
> above, then there must be something else going on beyond the scope of this
> thread/forum.


Hi Rob and thank you very match indeed for your time explaining such a good example in depth !
The workflow is the most interesting function of Mayan EDMS in my opinion and I have to admit that it is not so clear and easy for me to understand it and even the e-book doesn't give good examples about.
Generally speaking, every workflow has the following structure :
1. States
2. Transitions
3. Actors
4. Actions
5. Triggers
6. Variables

In the Workflow of Mayan I have identified states, transitions, actions, triggers,but I cannot find one of the most important things: Actors to whom a state flow can be assigned.
In case I have 3 users : one operator that can upload,one controller, and one approver, how I'm supposed to assign task for them ?
For instance at the moment that a document is uploaded into a specific cabinet that document should go to the controller for approval/reject. If the controller decide to approve it then the workflow should assign the next task to the last approver person.
As far as I have read and understood so far it cannot be done in Mayan EDMS. Correct me if I'm wrong please.
I have tried such a scenario , thinking to play with tags (Approved, Rejected), but the user controller cannot see any document in workflow.

Thanks !
Kozmi