ElasticSearch search backend configuration

Questions, comments, discussions. Over time certain topics might be moved to their own category.
Post Reply
joekhoobyar
Posts: 17
Joined: Mon Feb 24, 2020 3:49 am

ElasticSearch search backend configuration

Post by joekhoobyar »

Is there any documentation for the ElasticSearch search backend configuration in Mayan?

Or should I go read the source to try and reverse engineer?

I done a fair amount of googling, board searching, etc. on the topic and wasn't able to find anything.
joekhoobyar
Posts: 17
Joined: Mon Feb 24, 2020 3:49 am

Re: ElasticSearch search backend configuration

Post by joekhoobyar »

Note, using the following env vars:

Code: Select all

MAYAN_SEARCH_BACKEND='mayan.apps.dynamic_search.backends.elasticsearch.ElasticSearchBackend'
MAYAN_SEARCH_BACKEND_ARGUMENTS: {"client_host":"http://elasticsearch:9200","client_http_auth":false}
I get a promising initial result: mayan is able to talk to ElasticSearch and create some initial indexes there.

However, even after initiating a rebuild of search indexes, I always get a "Server Error" trying even a simple one word search.

In the logs I see:

Code: Select all

mayan.apps.logging.middleware.error_logging <36> [ERROR] "process_exception() line 15 Exception caught by request middleware; <WSGIRequest: GET '/search/results/?_search_model_pk=documents.documentsearchresult&q=test'>, NotFoundError(404, 'index_not_found_exception', 'no such index [mayan-documentsearchresult]', mayan-documentsearchresult, index_or_alias)"
Traceback (most recent call last):
  File "/opt/mayan-edms/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
 File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/view_mixins.py", line 11, in dispatch
    return super().dispatch(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/views/mixins.py", line 607, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/django/views/generic/list.py", line 142, in get
    self.object_list = self.get_queryset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/views/generics.py", line 898, in get_queryset
    queryset = super().get_queryset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/views/mixins.py", line 578, in get_queryset
    queryset = super().get_queryset(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/views/mixins.py", line 539, in get_queryset
    queryset = self.get_source_queryset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/views.py", line 63, in get_source_queryset
    queryset = SearchBackend.get_instance().search(
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/classes.py", line 419, in search
    queryset = self.solve_scope(
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/classes.py", line 477, in solve_scope
    return self._search(
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/elasticsearch.py", line 128, in _search
    client.indices.refresh(index=index_name)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 68, in refresh
    return self.transport.perform_request(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/transport.py", line 466, in perform_request
    raise e
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/transport.py", line 427, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 291, in perform_request
    self._raise_error(response.status, raw_data)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 328, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [mayan-documentsearchresult]', mayan-documentsearchresult, index_or_alias)
Is this a sign of incomplete index creation in ElasticSearch by Mayan?

Code: Select all

$ curl http://localhost:9200/_cat/indices
green open mayan-documentversionsearchresult     fI60-tpCSSmidi6G4FPtMA 1 1  2328  349 128.7mb  58.3mb
green open mayan-cabinetsearchresult             0WfaBJKZTn-e-uK2IXx9ZA 1 1     9  751 512.6mb 266.8mb
green open mayan-indexinstancenodesearchresult   Z44I4TqRS5uDRdVzj9OelA 1 1  6152 6616 196.4mb 115.7mb
green open mayan-documentfilesearchresult        LTZclrcmSaep3Z3GXNA8sA 1 1  2116  729   126mb  63.5mb
green open mayan-tag                             ufpl5GBdSU2shzIVtCpsJg 1 1    10  497 564.3mb 342.5mb
green open mayan-documentfilepagesearchresult    EKlrd-voR_-awb1IXwdumA 1 1 12811 2946 165.8mb  79.4mb
green open mayan-documentversionpagesearchresult kW7BTSKBT4mhUC-4Vt30-Q 1 1 13775 5200 176.9mb  89.1mb
Do I need to wait for the reindexing to complete?
joekhoobyar
Posts: 17
Joined: Mon Feb 24, 2020 3:49 am

Re: ElasticSearch search backend configuration

Post by joekhoobyar »

Perhaps this is the source of the problem?

When I begin the rebuild search indexes operation, it starts with this error:

Code: Select all

[2022-07-04 01:43:50,305: ERROR/ForkPoolWorker-16] Task mayan.apps.dynamic_search.tasks.task_reindex_backend[45d6517a-c3ec-4a39-bb9b-b3156e699e01] raised unexpected: RequestError(400, 'illegal_argument_exception', {'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'Wildcard expressions or all indices are not allowed'}], 'type': 'illegal_argument_exception', 'reason': 'Wildcard expressions or all indices are not allowed'}, 'status': 400})
Traceback (most recent call last):
  File "/opt/mayan-edms/lib/python3.9/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/celery/app/trace.py", line 731, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/tasks.py", line 151, in task_reindex_backend
    backend.reset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/elasticsearch.py", line 226, in reset
    self.tear_down(search_model=search_model)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/elasticsearch.py", line 236, in tear_down
    client.indices.delete(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 334, in delete
    return self.transport.perform_request(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/transport.py", line 466, in perform_request
    raise e
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/transport.py", line 427, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 291, in perform_request
    self._raise_error(response.status, raw_data)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 328, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Wildcard expressions or all indices are not allowed')
joekhoobyar
Posts: 17
Joined: Mon Feb 24, 2020 3:49 am

Re: ElasticSearch search backend configuration

Post by joekhoobyar »

I was able to get past the last error posted above by changing my elasticsearch configuration:

Code: Select all

action.destructive_requires_name=false
User avatar
michael
Developer
Developer
Posts: 276
Joined: Sun Apr 19, 2020 6:21 am

Re: ElasticSearch search backend configuration

Post by michael »

The command "search_initialize" needs to be executed to create the ElasticSearch indices.

The command "search_status" will show the indices for each model and how many object the contain. A number of "-1" means the index for that model is not found. After running "search_initialize" the count for each model index should be 0.

Then from the Tools menu execute "Reindex search backend". This will launch a background task that will crawl all objects and add them to the ElasticSearch indices.

Not sure what the setting "destructive_requires_name" does, but in version 4.3 we updated the "search_initialize" command to also remove existing indices to guarantee a pristine state.
justkoss
Posts: 3
Joined: Tue Aug 10, 2021 12:00 pm

Re: ElasticSearch search backend configuration

Post by justkoss »

hello, is there an equivalent of search_initialize for Mayan 4.0.15?
User avatar
michael
Developer
Developer
Posts: 276
Joined: Sun Apr 19, 2020 6:21 am

Re: ElasticSearch search backend configuration

Post by michael »

Hello,

It is unlikely that this will happen. Our versions are immutable. Once a version is released, no functional changes are made to it, only bug fixes and security updates.

We do offer backports of selected features for specialized forks as part of our commercial services agreements.
woec
Posts: 25
Joined: Sun Jun 02, 2019 7:11 am

Re: ElasticSearch search backend configuration

Post by woec »

Hi,

I encounter the same errors reported above (Mayan version 4.3.1):

[2022-09-19 13:03:03,895: ERROR/ForkPoolWorker-16] Task mayan.apps.dynamic_search.tasks.task_reindex_backend[2d5f3a21-d1b7-44fa-9f5a-0c9749357ecc] raised unexpected: RequestError(400, 'illegal_argument_exception', {'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'Wildcard expressions or all indices are not allowed'}], 'type': 'illegal_argument_exception', 'reason': 'Wildcard expressions or all indices are not allowed'}, 'status': 400})
........

Executing search_initialize and search_status afterwards yields to 0 indexes.
But starting rebuild search index from the tools menu yields again the same errors.
How to solve that?

woec
User avatar
michael
Developer
Developer
Posts: 276
Joined: Sun Apr 19, 2020 6:21 am

Re: ElasticSearch search backend configuration

Post by michael »

What version of ElasticSearch are you using?
woec wrote: Mon Sep 19, 2022 1:18 pm Hi,

I encounter the same errors reported above (Mayan version 4.3.1):

[2022-09-19 13:03:03,895: ERROR/ForkPoolWorker-16] Task mayan.apps.dynamic_search.tasks.task_reindex_backend[2d5f3a21-d1b7-44fa-9f5a-0c9749357ecc] raised unexpected: RequestError(400, 'illegal_argument_exception', {'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'Wildcard expressions or all indices are not allowed'}], 'type': 'illegal_argument_exception', 'reason': 'Wildcard expressions or all indices are not allowed'}, 'status': 400})
........

Executing search_initialize and search_status afterwards yields to 0 indexes.
But starting rebuild search index from the tools menu yields again the same errors.
How to solve that?

woec
woec
Posts: 25
Joined: Sun Jun 02, 2019 7:11 am

Re: ElasticSearch search backend configuration

Post by woec »

Elastic search version is 8.4.1
Post Reply