ElasticSearch search backend configuration

[joekhoobyar]

Is there any documentation for the ElasticSearch search backend configuration in Mayan?

Or should I go read the source to try and reverse engineer?

I done a fair amount of googling, board searching, etc. on the topic and wasn't able to find anything.

[joekhoobyar]

Note, using the following env vars:

Code: Select all

MAYAN_SEARCH_BACKEND='mayan.apps.dynamic_search.backends.elasticsearch.ElasticSearchBackend'
MAYAN_SEARCH_BACKEND_ARGUMENTS: {"client_host":"http://elasticsearch:9200","client_http_auth":false}

I get a promising initial result: mayan is able to talk to ElasticSearch and create some initial indexes there.

However, even after initiating a rebuild of search indexes, I always get a "Server Error" trying even a simple one word search.

In the logs I see:

Code: Select all

mayan.apps.logging.middleware.error_logging <36> [ERROR] "process_exception() line 15 Exception caught by request middleware; <WSGIRequest: GET '/search/results/?_search_model_pk=documents.documentsearchresult&q=test'>, NotFoundError(404, 'index_not_found_exception', 'no such index [mayan-documentsearchresult]', mayan-documentsearchresult, index_or_alias)"
Traceback (most recent call last):
  File "/opt/mayan-edms/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
 File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/view_mixins.py", line 11, in dispatch
    return super().dispatch(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/views/mixins.py", line 607, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/django/views/generic/list.py", line 142, in get
    self.object_list = self.get_queryset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/views/generics.py", line 898, in get_queryset
    queryset = super().get_queryset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/views/mixins.py", line 578, in get_queryset
    queryset = super().get_queryset(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/views/mixins.py", line 539, in get_queryset
    queryset = self.get_source_queryset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/views.py", line 63, in get_source_queryset
    queryset = SearchBackend.get_instance().search(
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/classes.py", line 419, in search
    queryset = self.solve_scope(
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/classes.py", line 477, in solve_scope
    return self._search(
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/elasticsearch.py", line 128, in _search
    client.indices.refresh(index=index_name)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 68, in refresh
    return self.transport.perform_request(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/transport.py", line 466, in perform_request
    raise e
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/transport.py", line 427, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 291, in perform_request
    self._raise_error(response.status, raw_data)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 328, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [mayan-documentsearchresult]', mayan-documentsearchresult, index_or_alias)

Is this a sign of incomplete index creation in ElasticSearch by Mayan?

Code: Select all

$ curl http://localhost:9200/_cat/indices
green open mayan-documentversionsearchresult     fI60-tpCSSmidi6G4FPtMA 1 1  2328  349 128.7mb  58.3mb
green open mayan-cabinetsearchresult             0WfaBJKZTn-e-uK2IXx9ZA 1 1     9  751 512.6mb 266.8mb
green open mayan-indexinstancenodesearchresult   Z44I4TqRS5uDRdVzj9OelA 1 1  6152 6616 196.4mb 115.7mb
green open mayan-documentfilesearchresult        LTZclrcmSaep3Z3GXNA8sA 1 1  2116  729   126mb  63.5mb
green open mayan-tag                             ufpl5GBdSU2shzIVtCpsJg 1 1    10  497 564.3mb 342.5mb
green open mayan-documentfilepagesearchresult    EKlrd-voR_-awb1IXwdumA 1 1 12811 2946 165.8mb  79.4mb
green open mayan-documentversionpagesearchresult kW7BTSKBT4mhUC-4Vt30-Q 1 1 13775 5200 176.9mb  89.1mb

Do I need to wait for the reindexing to complete?

[joekhoobyar]

Perhaps this is the source of the problem?

When I begin the rebuild search indexes operation, it starts with this error:

Code: Select all

[2022-07-04 01:43:50,305: ERROR/ForkPoolWorker-16] Task mayan.apps.dynamic_search.tasks.task_reindex_backend[45d6517a-c3ec-4a39-bb9b-b3156e699e01] raised unexpected: RequestError(400, 'illegal_argument_exception', {'error': {'root_cause': [{'type': 'illegal_argument_exception', 'reason': 'Wildcard expressions or all indices are not allowed'}], 'type': 'illegal_argument_exception', 'reason': 'Wildcard expressions or all indices are not allowed'}, 'status': 400})
Traceback (most recent call last):
  File "/opt/mayan-edms/lib/python3.9/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/celery/app/trace.py", line 731, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/tasks.py", line 151, in task_reindex_backend
    backend.reset()
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/elasticsearch.py", line 226, in reset
    self.tear_down(search_model=search_model)
  File "/opt/mayan-edms/lib/python3.9/site-packages/mayan/apps/dynamic_search/backends/elasticsearch.py", line 236, in tear_down
    client.indices.delete(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 334, in delete
    return self.transport.perform_request(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/transport.py", line 466, in perform_request
    raise e
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/transport.py", line 427, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 291, in perform_request
    self._raise_error(response.status, raw_data)
  File "/opt/mayan-edms/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 328, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Wildcard expressions or all indices are not allowed')

[joekhoobyar]

I was able to get past the last error posted above by changing my elasticsearch configuration:

Code: Select all

action.destructive_requires_name=false

[michael]

The command "search_initialize" needs to be executed to create the ElasticSearch indices.

The command "search_status" will show the indices for each model and how many object the contain. A number of "-1" means the index for that model is not found. After running "search_initialize" the count for each model index should be 0.

Then from the Tools menu execute "Reindex search backend". This will launch a background task that will crawl all objects and add them to the ElasticSearch indices.

Not sure what the setting "destructive_requires_name" does, but in version 4.3 we updated the "search_initialize" command to also remove existing indices to guarantee a pristine state.