I have more than 100GB of PDF files. I tried to use watch folder.
However the Watch folder uploads one for each checking interval.
Changing from 600s to 10s not helped.
If you become a supporter of the Forum, you can read a Knowledgebase Article that Roberto wrote about using Staging Sources to load large amounts of files.
Here’s a link to the article
https://forum.mayan-edms.com/docs?topic=1058
Hello, thank you for your response.
I am already a supporter of the forum.
I have another question: Is the option you mentioned the only one available for large uploads? I am seeking an option that enables a single mass upload, after which I can seamlessly utilize the Watch Folder feature. Typically, a document is updated every hour, and I need a solution that accommodates this workflow effortlessly.
1 Like
100 GB of existing documents is best served with our custom migration tool. We cover those under the custom contract option.
We create a custom import backend for the technology storing your documents and launch the import tool to process the bulk of documents. This tool can do stopping and resuming of migrations, as well as capture relevant information of the document stored also in the source (like internal identifiers, folder location, API IDs).
If you don’t want to go with a custom contract to use the migration tool, you can compress your documents into multiple zip files and set the watch folder to uncompress zip files automatically.
Another option is to write a simple API client that scans the folder using Python’s pathlib and the requests library. If you are familiar with Python programming this would be very trivial to you and you can customized it to also handle metadata or tags as the documents are uploaded.