Thank you for this thread, @talbottech! I’ve got exactly the same problem, down to the number of documents (1.5k with ~17k pages). Just to be clear, for us, 1.5k is simply a testing run, the real amount of documents in production is expected to be magnitudes higher.
I’ve run into this problem on an updated installation (4.3 → 4.4), and just to be on the safe side, 3 days ago I did a clean 4.4.6 Docker install (no cabinets, no indices, no tags, no workflows, one default document type), and the search queue is still sitting at 600.000+ messages.
I’ve tried increasing the search chunk size, I’ve tried “spinning up” (starting, really) additional B and A workers just for the search queue. The servers we’re testing on both have a 100GB SSD; one is 6 cores, 12 GB RAM, the other is 12 cores, 24 GB RAM. In both cases, Mayan’s shitting into the search queue like there’s no tomorrow.
Adding resources doesn’t seem to help–Mayan simply starts spamming the queue faster.
To make matters worse, this seems to be an old problem:
(notice how the thread starts with the high CPU load, but then moves to “just like in the above thread I have the search queue eating up all my resources endless”)
Some other guy having pretty much the same problem:
@roberto.rosario, you’re saying “it is getting addressed,” but the reality is, current version of the system is completely unusable. Well, maybe it is–for a hundred documents or so. Are you doing any QA? How could this slip through?
Honestly, I’m completely flabbergasted and after months of working with Mayan will probably be recommending to my organization that we try something else.