Watchfolder and Log Errors with 3.5 [SOLVED]

When things don't work as they should.
Post Reply
Lffy
Posts: 2
Joined: Sat Oct 10, 2020 7:48 am

Watchfolder and Log Errors with 3.5 [SOLVED]

Post by Lffy »

Hi,

I upgraded Mayan with Docker to 3.5. Now I am not able to delete the watchfolders I created and just getting an error. Additionally the docker protocol/log gets spammed with errors. When I disable the watchfolders the errors are gone, but other errors appear (see 2nd code).
Despite the errors, if enabled the watchfolders still work.

I use PostgreSQL 12.3 container und Redis 5.0 alpine container. I did not touch them after updating the Mayan container.
Can anyone help me?

P.S.: Mayan is great, thx!

Code: Select all

[2020-10-10 07:59:40,120: ERROR/ForkPoolWorker-2] Error processing source id: 4; get() returned more than one ErrorLog -- it returned 2!,
[2020-10-10 07:59:40,123: ERROR/ForkPoolWorker-2] Task mayan.apps.sources.tasks.task_check_interval_source[5f76e187-9c23-4847-ac9d-941c0dcd5115] raised unexpected: AttributeError("'WatchFolderSource' object has no attribute 'logs'"),
Traceback (most recent call last):,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/sources/tasks.py", line 43, in task_check_interval_source,
    source.check_source(test=test),
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/sources/models/base.py", line 224, in check_source,
    self.error_log.all().delete(),
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/logging/classes.py", line 42, in method_instance_logs,
    error_log_partition, created = error_log_instance.model.partitions.get_or_create(,
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/logging/classes.py", line 34, in model,
    model, created = ErrorLogModel.objects.get_or_create(name=self.app_config.name),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method,
    return getattr(self.get_queryset(), name)(*args, **kwargs),
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/models/query.py", line 538, in get_or_create,
    return self.get(**kwargs), False,
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/models/query.py", line 412, in get,
    (self.model._meta.object_name, num),
mayan.apps.logging.models.ErrorLog.MultipleObjectsReturned: get() returned more than one ErrorLog -- it returned 2!,
,
During handling of the above exception, another exception occurred:,
,
Traceback (most recent call last):,
  File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task,
    R = retval = fun(*args, **kwargs),
  File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__,
    return self.run(*args, **kwargs),
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/sources/tasks.py", line 48, in task_check_interval_source,
    source.logs.create(,
AttributeError: 'WatchFolderSource' object has no attribute 'logs',
mayan.apps.sources.tasks <405> [ERROR] "task_check_interval_source() line 46 Error processing source id: 4; get() returned more than one ErrorLog -- it returned 2!",

Code: Select all

edms/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/views/mixins.py", line 128, in dispatch
    return super(ExternalObjectMixin, self).dispatch(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/views/mixins.py", line 554, in dispatch
    ).dispatch(request, *args, **kwargs)
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/views/generic/base.py", line 97, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/views/generic/list.py", line 142, in get
    self.object_list = self.get_queryset()
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/views/generics.py", line 809, in get_queryset
    queryset = super(SingleObjectListView, self).get_queryset()
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/views/mixins.py", line 508, in get_queryset
    queryset = self.get_source_queryset()
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/logging/views.py", line 47, in get_source_queryset
    return self.external_object.error_log.all()
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/logging/classes.py", line 42, in method_instance_logs
    error_log_partition, created = error_log_instance.model.partitions.get_or_create(
  File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/logging/classes.py", line 34, in model
    model, created = ErrorLogModel.objects.get_or_create(name=self.app_config.name)
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/opt/mayan-edms/lib/python3.7/site-packages/django/db/models/query.py", line 412, in get
    (self.model._meta.object_name, num)
mayan.apps.logging.models.ErrorLog.MultipleObjectsReturned: get() returned more than one ErrorLog -- it returned 2!
Attachments
sources.png
sources.png (61.28 KiB) Viewed 722 times
error2.png
error2.png (7.38 KiB) Viewed 722 times
error1.png
error1.png (27.47 KiB) Viewed 722 times
User avatar
michael
Developer
Developer
Posts: 78
Joined: Sun Apr 19, 2020 6:21 am

Re: Watchfolder and Log Errors with 3.5

Post by michael »

Thanks for the report and the debug information. Ticket #894 created. As soon as it is triaged it will be queued to fix.

https://gitlab.com/mayan-edms/mayan-edms/-/issues/894
User avatar
michael
Developer
Developer
Posts: 78
Joined: Sun Apr 19, 2020 6:21 am

Re: Watchfolder and Log Errors with 3.5

Post by michael »

Version 3.5.1 is out and includes a fix for this issue. Thanks for the report!
Lffy
Posts: 2
Joined: Sat Oct 10, 2020 7:48 am

Re: Watchfolder and Log Errors with 3.5 [SOLVED]

Post by Lffy »

Just wanted to report back. The update fixed the log spamming and the error button works again.
Thx for the blazing fast fix!
DrRSatzteil
Posts: 6
Joined: Mon Oct 26, 2020 11:38 pm

Re: Watchfolder and Log Errors with 3.5 [SOLVED]

Post by DrRSatzteil »

Hi,

I do experience a similar behaviour (error messages in the log but upload from watch folder seems to work nevertheless) though the problem seems to be unrelated to Lffy's problem as the error message seems to point to the whoosh search provider:

Code: Select all

app_1               | [2020-11-13 18:14:20,457: ERROR/ForkPoolWorker-1] Task mayan.apps.dynamic_search.tasks.task_index_instance[9b31f83f-8fb7-48ac-bc6b-be621a4e9a9a] raised unexpected: TypeError('utf_8_encode() argument 1 must be str, not None')
app_1               | Traceback (most recent call last):
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
app_1               |     R = retval = fun(*args, **kwargs)
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
app_1               |     return self.run(*args, **kwargs)
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/dynamic_search/tasks.py", line 61, in task_index_instance
app_1               |     SearchBackend.get_instance().index_instance(instance=instance)
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/dynamic_search/backends/whoosh.py", line 181, in index_instance
app_1               |     instance=instance, exclude_set=exclude_set
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/dynamic_search/backends/whoosh.py", line 237, in _index_instance
app_1               |     instance=instance, exclude_set=exclude_set
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/dynamic_search/backends/whoosh.py", line 237, in _index_instance
app_1               |     instance=instance, exclude_set=exclude_set
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/mayan/apps/dynamic_search/backends/whoosh.py", line 213, in _index_instance
app_1               |     writer.add_document(**kwargs)
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/whoosh/writing.py", line 750, in add_document
app_1               |     for tbytes, freq, weight, vbytes in items:
app_1               |   File "/opt/mayan-edms/lib/python3.7/site-packages/whoosh/fields.py", line 165, in index
app_1               |     yield (utf8encode(tstring)[0], freq, wt, vbytes)
app_1               | TypeError: utf_8_encode() argument 1 must be str, not None
app_1               | mayan.apps.sources.tasks <409> [ERROR] "task_check_interval_source() line 47 Error processing source id: 5; "
app_1               | mayan.apps.sources.tasks <410> [ERROR] "task_check_interval_source() line 47 Error processing source id: 4; "
app_1               | [2020-11-13 18:15:46,046: ERROR/ForkPoolWorker-4] Error processing source id: 4;
app_1               | [2020-11-13 18:15:46,045: ERROR/ForkPoolWorker-3] Error processing source id: 5;
User avatar
rosarior
Developer
Developer
Posts: 624
Joined: Tue Aug 21, 2018 3:28 am
Location: Puerto Rico
Contact:

Re: Watchfolder and Log Errors with 3.5 [SOLVED]

Post by rosarior »

Hi,

Thanks for the report.

Yes, the error origin is different, in your case it seems to be related to the search backend indexing job that is triggered after a new document is created. That helps narrow things down and help investigate the underlying cause. Added for the next bug fix releases.

Thanks!

https://gitlab.com/mayan-edms/mayan-edms/-/issues/913
DrRSatzteil
Posts: 6
Joined: Mon Oct 26, 2020 11:38 pm

Re: Watchfolder and Log Errors with 3.5 [SOLVED]

Post by DrRSatzteil »

Hi,

thank you for looking into it!

I just did some more testing on this. It appears that the documents are in fact NOT uploaded. Mayan tried to import a file without success for a couple of hours (every ten minutes). However after I recreated my docker container the file gets uploaded even though I see an error in the log.

What surprises me is that I do not see the detailed stack trace in the log but only the error message so there might even be another problem. I do not know why I saw the stack trace the other day. Can I somehow configure the log output?

And one more observation: I do see errors in the logs for empty watch folders as well...
Post Reply