[SOLVED] - IntegrityError in a new fresh installation

When things don't work as they should.
Post Reply
mcarlosro
Posts: 11
Joined: Fri Jan 10, 2020 3:23 pm

[SOLVED] - IntegrityError in a new fresh installation

Post by mcarlosro »

Hi all,

I just installed a new mayan in my Synology in docker without any problem. Thanks for the documentation. Really, easy and clear to follow.

Then, I created a watch folder and I got the following error

Code: Select all

  
[2020-01-10 15:20:26,495: ERROR/ForkPoolWorker-2] Task mayan.apps.document_indexing.tasks.task_index_document[51bee2
e0-ac50-44aa-a66a-97bf6b8f6435] raised unexpected: IntegrityError('insert or update on table "document_indexing_inde
xinstancenode_documents" violates foreign key constraint "document_indexing_in_indexinstancenode_id_9f4e413e_fk_docu
ment_"\nDETAIL:  Key (indexinstancenode_id)=(13) is not present in table "document_indexing_indexinstancenode".\n') 
Traceback (most recent call last):                                                                                  
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/backends/base/base.py", line 236, in _commit          
    return self.connection.commit()                                                                                 
psycopg2.errors.ForeignKeyViolation: insert or update on table "document_indexing_indexinstancenode_documents" viola
tes foreign key constraint "document_indexing_in_indexinstancenode_id_9f4e413e_fk_document_"                        
DETAIL:  Key (indexinstancenode_id)=(13) is not present in table "document_indexing_indexinstancenode".             
                                                                                                                                                                                     
The above exception was the direct cause of the following exception:                                                

 File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task                   
    R = retval = fun(*args, **kwargs)                                                                               
  File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 648, in __protected_call__           
    return self.run(*args, **kwargs)                                                                                
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/document_indexing/tasks.py", line 45, in task_index_document                                                                                                             
    Index.objects.index_document(document=document)                                                                 
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/document_indexing/managers.py", line 19, in index_document                                                                                                               
    index.index_document(document=document)                                                                         
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/document_indexing/models.py", line 128, in index_document                                                                                                                
    self.template_root.index_document(document=document)                                                            
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/document_indexing/models.py", line 305, in index_document                                                                                                                
    index_instance_node_parent=index_instance_root_node                                                             
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/transaction.py", line 223, in __exit__                
    connection.commit()                                                                                             
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/backends/base/base.py", line 262, in commit           
    self._commit()                                                                                                  
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/backends/base/base.py", line 236, in _commit          
    return self.connection.commit()                                                                                 
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/utils.py", line 94, in __exit__                       
    six.reraise(dj_exc_type, dj_exc_value, traceback)                                                               
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/utils/six.py", line 685, in reraise                      
    raise value.with_traceback(tb)                                                                                  
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/backends/base/base.py", line 236, in _commit         
    return self.connection.commit()                                                                                 
django.db.utils.IntegrityError: insert or update on table "document_indexing_indexinstancenode_documents" violates foreign key constraint "document_indexing_in_indexinstancenode_id_9f4e413e_fk_document_"                             
DETAIL:  Key (indexinstancenode_id)=(13) is not present in table "document_indexing_indexinstancenode".                                                                                                                                
Do you know what it could be the reason? This is a complete new system where I just create the watch folder as my first step.

Thanks in advance
Last edited by mcarlosro on Sat Jan 11, 2020 6:57 am, edited 1 time in total.

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: IntegrityError in a new fresh installation

Post by rssfed23 »

So that error is being caused by a missing entry in the database (key (indexinstancenode_id)=(13) is not present in table "document_indexing_indexinstancenode").
It looks like it's trying to index a document using an index that doesn't exist (or a document that doesn't exist)

Can you try going to tools > reset indexes and then select and reset all of your indexes?

This could also be caused by a previously incompletely uploaded document (I've only seen it happen once before though).

If resetting the indexes doesn't work, can you try deleting any indexes you've set up and re-adding them again.

However I want to assure you that if the document is showing in your all documents list then it's only the indexing part that's failed.
Does the document show up in Mayan afterwards (in the all documents section)
If the document isn't showing in your all documents list then that is likely the cause of the indexing error (it's trying to index a document it thinks exist but doesn't) and we need to figure out why documents aren't uploading correctly for you.
Please don't PM for general support; start a new thread with your issue instead.

mcarlosro
Posts: 11
Joined: Fri Jan 10, 2020 3:23 pm

Re: IntegrityError in a new fresh installation

Post by mcarlosro »

Thanks for your quick response.

What you said makes sense. I have some problems with the watch folder because files were not deleted and also permssions when I tested manually. As this is a blank new system, I will delete all and start from scratch again to see if the error is solved.

At the end, I gave full access to that folder in my NAS in order to avoid any problem with the docker. Is there any better way to do it?

Also, I really like the idea of using mayan to remove papers, but I find it difficult to start with an empty system. Is there any best practices guide? running system could be used as a reference? I want to use it at home as most of the people post I read. Therefore, most of us will have a common point to start from.

Thanks a lot

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: IntegrityError in a new fresh installation

Post by rssfed23 »

No problem :)

That potentially would explain things! - I still recommend running the reset indexes but it's entirely possible there's stuff left "dangling" in the database as orphaned half-completed uploads. There is a process that goes through and cleans up these documents (once a day from memory).
Permissions can be a bit of a mess on Synology boxes. I have one myself (although don't run Mayan on it as it's only got 2gb of RAM. Just use it for NFS storage).

Getting started is definitely the hardest part. We have the mayan book which expands beyond the documentation and goes into more details on the concepts behind it. The forum also has a lot of archived info floating around (the indexes post in the guide is a particularly helpful reference). The book is the closest thing to a best practice guide I would say (plus you'll be supporting the project):
https://www.mayan-edms.com/book/

The main thing to get right from the start is document types, tags, metadata, cabinets and indexes (because going through and retrospectively adding those to thousands of documents can become a headache). The book covers those concepts what they are and how to implement them quite well.
The book will be updated soon as well for 3.3 (although the base concepts are the same in terms of organisation).

Funny you mention an existing running system....
I'm actually building a Mayan live demo environment right now as the Mayan docker demo starts off pretty blank.
If you go to https://mayan.app then you'll see the current state it's in. It's currently OCRing a zillion documents so it's not uber-fast yet.
Be warned though that this isn't "live" as yet and I've not set up much of a structure just imported a few thousand documents into it.

Going back to basic organisation;

I decided to write some explanations for each organisation type based on my home server but with all the screenshots I used I exceeded the forums limit so I'm going to add another post to this immediately after so i can have all the images in it I want to. I went a bit "overboard" on this so thanks for making me write it I may link it to the guides section as I'm sure it's useful for new users also!
Please don't PM for general support; start a new thread with your issue instead.

User avatar
rssfed23
Moderator
Moderator
Posts: 185
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: IntegrityError in a new fresh installation

Post by rssfed23 »

@mcarlosro I split off the 2 posts I made on introductory to organisation so I could move them to the guides section for everyone else to benefit. You can find them at viewtopic.php?f=15&t=1597

Here's what I typed at the end of the final post though before I split it off to a new topic:


So yeah; my quick intro to organisation went on a bit longer than I thought but I hope that helps explain things a bit more?
I strongly recommend you buy the book though. That's how I learnt Mayan. That and a lot of time playing around!

Although yes a lot of people use Mayan at home, we also have a lot of enterprise users as well and the enterprise is where our focus is.
That's why it has some of the features it does (such as workflows and indexes) that you'll never find in more basic document management tools.
BUT that of course means it also comes with some added complexity as a result and a learning curve. It took me many weeks before I became confident in using it for all my documents just playing around and experimenting. Now I use it for everything at the home office and help organisations transform their document management.

Of course, there's loads of other features as well I've not mentioned here as they're not related to organisation (like the build in scanning capabilities, GPG signing, Workflows, Permissions/Roles/ACLs, etc)

I do plan on making some kind of an introductory video at some point that talks through some of the basic concepts and shows how a system might be set up.

So that initial planned short reply took up 3 large posts. Thanks for asking this I hope it benefits others as well as yourself!

Rob
Please don't PM for general support; start a new thread with your issue instead.

mcarlosro
Posts: 11
Joined: Fri Jan 10, 2020 3:23 pm

Re: [SOLVED] - IntegrityError in a new fresh installation

Post by mcarlosro »

Thanks for the long response.

definitely, I will go the book way. As you are updating for version 3.3, it would make sense, in order to have home users to buy it and help the project, to include a chapter with basic setup for home users (just an idea). At the end, as you said in your response, there is a few cabinets (bills, kids grades, manuals/documentation, payslisps) that we need to track.

With those document we can evolve easily.

also link is viewtopic.php?f=15&t=1599

Post Reply