150k pages taking 2+ minutes to search

Questions, comments, discussions. Over time certain topics might be moved to their own category.
User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: 150k pages taking 2+ minutes to search

Post by rssfed23 »

Docker will always add some overhead yes. It also can't take advantage of some of the optimisations available for bare metal.

I just ran some test queries on version 11 (upgraded from 9.6) and it did not add any significant improvement to speed on search.

It should really only be searching that gives you trouble. Postgres has to look through every 150k page and search through for the line of text in order to find the result.

So In summary:

Postgres is running the search query with only 1 thread. To speed things up any or all of the below can be tried and each item alone will help slightly:
- Don't run postgres containerised
- Ensure you've set up Postgres Hugepages and other Optimisation parameters (the pgtune utility can help here)
- Apply other postgres optimisations such as adding in a memcached cache (https://dzone.com/articles/three-steps- ... erformance)
- Run postgres on a faster server

In the background, we'll investigate what we can do as far as tweaking the search. I've logged https://gitlab.com/mayan-edms/mayan-edms/issues/741 to see what investigations can be done on this
As mentioned before, enterprise customers get priority support and feature requests so don't hesitate to get in touch.
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: 150k pages taking 2+ minutes to search

Post by rssfed23 »

Hybred.

As per the github issue (https://gitlab.com/mayan-edms/mayan-edm ... _268258869), we won't be making any immediate changes in terms of the current search setup/postgres configuration as there are broader changes planned for 4.0

Initially that will be to add backend support to the Mayan search engine. That will enable pluggable backends using drivers. Initially the Target will be woosh, but in theory that could be extended to others in the future (e.g. Elasticsearch). Based on my own experience in the past switching to woosh (which is designed for full text search) will significantly improve search speed and likely solve all your performance issues.
I should highlight that this is a two step process: adding support for backends and then adding our first additional backend (woosh). It is hoped that backend will appear in 4.0 this year and then woosh will be added soon after. Of course there is no guarantee it will end up in 4.0 as priorities can change and release dates move, but it is on the plan.

You can see the gitlab issue for all the details but that's the current plan. I hope that gives you some confidence. Prioritising this functionality can definitely be a part of any enterprise consulting agreement discussion.
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

Hybred
Posts: 8
Joined: Fri Jan 03, 2020 3:12 pm

Re: 150k pages taking 2+ minutes to search

Post by Hybred »

Thanks for making it an official concern and feature improvement.

I ran the database optimization and it still continues to crash after 120 seconds, so it seems like I need to pull the database out of the docker container and into it's own home.

I'll shoot for that early next week and see what comes of it.

Thanks for your help! I just submitted a PR for enterprise support, I'm sure it will be needed.

User avatar
rssfed23
Moderator
Moderator
Posts: 213
Joined: Mon Oct 14, 2019 1:18 pm
Location: United Kingdom
Contact:

Re: 150k pages taking 2+ minutes to search

Post by rssfed23 »

You’re more than welcome!
We’ll be happy to help support you :)
Please bear with us during the current global situation. The team all have families and local communities to look after as well as the community here. Responses may be delayed during this time, but rest assured we will get to your query eventually.

Hybred
Posts: 8
Joined: Fri Jan 03, 2020 3:12 pm

Re: 150k pages taking 2+ minutes to search

Post by Hybred »

Hello again!

After a project was completed I'm able to jump back into this. I was able to dump the database and import it into a separate PostGRE instance. What settings would I need to update to point it to the new DB? Or would it be better to just start a new install and point the data to the db?

Hybred
Posts: 8
Joined: Fri Jan 03, 2020 3:12 pm

Re: 150k pages taking 2+ minutes to search

Post by Hybred »

Also signed up for support - let me know if the base tier is enough, can easily upgrade if not.

ID: I-KCN9BW1Y93YN

Post Reply