Page 1 of 2

File Upload Limit

Posted: Mon Feb 11, 2019 10:53 am
by gareththered
Hi,

I seem to get a 413 error each time I attempt to upload a file over 1MB. This used to be configurable in NGINX (and I believe the default was 500MB), but since the move to a Python powered web-server, I'm at a loss as to how to increase the limit.

I'm using the Docker image.

Any advice would be appreciated.

Thanks,

Gareth

Re: File Upload Limit

Posted: Tue Feb 12, 2019 8:41 pm
by rosarior
Looking at GUNICORN's documentation there doesn't seem to be a setting to control de HTTP request body size (http://docs.gunicorn.org/en/latest/settings.html). Apparently this is not possible (https://github.com/benoitc/gunicorn/issues/1659). Continuing to investigate.

Re: File Upload Limit

Posted: Tue Feb 12, 2019 8:54 pm
by rosarior
This post suggest it could be Dropzone.js, the widget used to upload the files (https://stackoverflow.com/questions/516 ... 30-seconds).

Re: File Upload Limit

Posted: Tue Feb 12, 2019 9:35 pm
by gareththered
The linked page in your last posts suggests it's a 30 second limit. I get a 413 error as soon as a file is 1MB in size, which is a matter of a couple of seconds or so - definitely not 30 seconds.

If I click on the drag-and-drop area to open the file-open dialog, and select a 1MB file, I get the same results. Presumably, that's using the same JS to upload?

Re: File Upload Limit

Posted: Tue Feb 12, 2019 10:20 pm
by rosarior
Yes, it is the same widget that services interactive uploads using a drop or by clicking the "Browse" button.

I'm investigating DropZone's maxFilesize setting (https://www.dropzonejs.com/#config-maxFilesize) the description is confusing because it appears to be a file size limit but the description say it limits the number of files.

I have not been able to find any reference to a file size limit in GUNICORN and the responses seem to suggest it doesn't and instead relies on the frontend webserver to do the HTTP body size limiting. Not discarding it yet but looking at all the alternatives.

Re: File Upload Limit

Posted: Wed Feb 13, 2019 6:48 am
by gareththered
I've just been reading the Stackoverflow entries tagged dropzone.js and they seem to suggest that there is a `maxFile` and a `maxFileSize` option available, with the former being the number of files and the latter the size. The documentation you link to states that the default is 256, but doesn't state the units - 256 bytes, 256 TB, or as you said 256 files? None of those align with my single 1MB file though.

Also, the HTTP 413 error that shows up is a server error: https://httpstatuses.com/413 suggesting that this isn't a dropzone.js error.

File Upload Limit

Posted: Sun Mar 24, 2019 12:17 am
by EdwardZem
Hi Panos,

If you want to increase the load of up to 20 MB, then you need to do:

Change the file php.ini:
upload_max_filesize = 20M
post_max_size = 20M

Or

Change the file .htaccess:
php_value upload_max_filesize 20M
php_value post_max_size 20M

Re: File Upload Limit

Posted: Sat Mar 30, 2019 12:07 am
by jorgeegomez
Rosario, I recently encountered this error, and it doesn't seem to be an upload limit, but some internal processing limit.

What I tried: upload a 16 MB .zip file (after applying my fix to #572), with several documents. The file uploaded fine, but all documents over 1MB did not show up in /recent_added. All documents under 1MB show up fine.

The only thing I can see in the logs is a LockError in task_index_document. Hope this helps you diagnose.

Re: File Upload Limit

Posted: Wed Apr 03, 2019 10:38 pm
by rosarior
Finally managed to trigger this on a production install. The issue seems to be an out of memory problem with a subsequent OOM kill by the operating system kernel.

[2019-04-03 22:11:46,876: ERROR/MainProcess] Task sources.tasks.task_upload_document[1387c04b-8487-4193-b3c4-217f723f057c] raised unexpected: MagicException('cannot allocate 1436672320 bytes (Cannot allocate memory)',)
Traceback (most recent call last):
File "/opt/mayan-edms/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/mayan-edms/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/sources/tasks.py", line 84, in task_upload_document
querystring=querystring, user=user,
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/sources/models.py", line 147, in upload_document
file_object=file_object, _user=user,
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/documents/models.py", line 298, in new_version
document_version.save(_user=_user)
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/documents/models.py", line 638, in save
self.update_page_count(save=False)
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/documents/models.py", line 729, in update_page_count
file_object=file_object, mime_type=self.mimetype
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/converter/classes.py", line 90, in __init__
file_object=file_object, mimetype_only=False
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/mimetype/api.py", line 16, in get_mimetype
file_mimetype = mime.from_buffer(file_object.read())
File "/opt/mayan-edms/lib/python2.7/site-packages/magic.py", line 82, in from_buffer
return self._handle509Bug(e)
File "/opt/mayan-edms/lib/python2.7/site-packages/magic.py", line 101, in _handle509Bug
raise e
MagicException: cannot allocate 1436672320 bytes (Cannot allocate memory)

...
[2308813.280915] [ 4929] 1001 4929 206983 116165 1343488 36207 0 mayan-edms.py
...
[2308813.281195] Out of memory: Kill process 4929 (mayan-edms.py) score 201 or sacrifice child
[2308813.281207] Killed process 4929 (mayan-edms.py) total-vm:827932kB, anon-rss:464648kB, file-rss:12kB, shmem-rss:0kB
[2308813.477890] oom_reaper: reaped process 4929 (mayan-edms.py), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

The system ran out of memory when attempting to load the entire document to perform MIME type introspection using the Python magic library, which in turn call the system's libmagic library. Need to investigate If Python magic tries to load the entire file at a time or if uses (can use) a generator to load it in chunks.

The same file doesn't cause a problem in development. More investigation is needed.

Re: File Upload Limit

Posted: Fri Apr 12, 2019 5:24 pm
by laramy2020
I am in a similar boat with mayan right now. Been looking in both django and gunicorn as to where to change file upload limits.
My setup is 3.1.9 docker deployment (one mayan prod and one mayan prod postgres)

Users get hit with a max upload of 256mib error
They get the same error through nginx and direct connection to mayan docker
nginx reverse proxy is set to 1000M

thanks for any help.