Wanted to start another topic, it stems from the post “Document Processing Times” but focus on the search backends in the context of running Mayan at home for home family use. (See my use case below and example case)
Thought I would get the ball rolling and ask to see if anyone in forum-landia has had experience with different search backends (I know of two I think that Mayan supports: whoosh the default and elasticsearch)
Looking for examples and pointers to documentation on how to change the default search backend, maybe tweaking the default to align more with the home use, use case. As I am sure there are others out there like me that just want a digital file cabinet.
My Use Case
My Use Case
I wanted to put out there my use case. As with any software there is no one size fits all my use case here is just for my families personal documents and the goal is to be able to search for a document’s OCR data. I will use indexes for commonly looked up by year/type and other added meta data. I share this to give context to the topic. I am not looking to processing 5,000 documents a day. More like 1-15 and as quick and low energy usage as possible.
So It seems my Mayan server is very CPU hungry with in regards to the search workers and search message queue. As it was pointed out in the above referenced post, one change can require a lot of search updates.
Example document processing times
Example, as part of my email submission work flow, I need to change the document type and update the meta data fields for that type. Just updating one, two page document and three meta fields, moving the workflow to completed (it also changes tags and file cabinets, so more changes I presume) that processing took two hours to complete and was showing 50,000+ messages in the search queue. I feel this is a but much I think, but maybe I am off base and have unreal expectations. My specific setup is below to provide context with this example.
My Server Setup for Context
- Manual Install from (Turnkey Linux v17) and upgraded to v4.4.4
- Running as a VM in Proxmox v7.3
- 6 x vCPUs (Xeon(R) CPU E3-1245 v5 @ 3.50GHz )
- 10GB vRAM (DDR4)
- 500GB vHDD (NVME backed)
Note this is a manual install and it not technically supported, however I hope to be moving to the docker install once I get my mind around the docker environment, and that may play a role in this issue/question as well.