File Upload Limit

When things doesn't work as they should.
gareththered
Posts: 5
Joined: Mon Feb 11, 2019 10:48 am

File Upload Limit

Post by gareththered » Mon Feb 11, 2019 10:53 am

Hi,

I seem to get a 413 error each time I attempt to upload a file over 1MB. This used to be configurable in NGINX (and I believe the default was 500MB), but since the move to a Python powered web-server, I'm at a loss as to how to increase the limit.

I'm using the Docker image.

Any advice would be appreciated.

Thanks,

Gareth

User avatar
rosarior
Posts: 274
Joined: Tue Aug 21, 2018 3:28 am

Re: File Upload Limit

Post by rosarior » Tue Feb 12, 2019 8:41 pm

Looking at GUNICORN's documentation there doesn't seem to be a setting to control de HTTP request body size (http://docs.gunicorn.org/en/latest/settings.html). Apparently this is not possible (https://github.com/benoitc/gunicorn/issues/1659). Continuing to investigate.

User avatar
rosarior
Posts: 274
Joined: Tue Aug 21, 2018 3:28 am

Re: File Upload Limit

Post by rosarior » Tue Feb 12, 2019 8:54 pm

This post suggest it could be Dropzone.js, the widget used to upload the files (https://stackoverflow.com/questions/516 ... 30-seconds).

gareththered
Posts: 5
Joined: Mon Feb 11, 2019 10:48 am

Re: File Upload Limit

Post by gareththered » Tue Feb 12, 2019 9:35 pm

The linked page in your last posts suggests it's a 30 second limit. I get a 413 error as soon as a file is 1MB in size, which is a matter of a couple of seconds or so - definitely not 30 seconds.

If I click on the drag-and-drop area to open the file-open dialog, and select a 1MB file, I get the same results. Presumably, that's using the same JS to upload?

User avatar
rosarior
Posts: 274
Joined: Tue Aug 21, 2018 3:28 am

Re: File Upload Limit

Post by rosarior » Tue Feb 12, 2019 10:20 pm

Yes, it is the same widget that services interactive uploads using a drop or by clicking the "Browse" button.

I'm investigating DropZone's maxFilesize setting (https://www.dropzonejs.com/#config-maxFilesize) the description is confusing because it appears to be a file size limit but the description say it limits the number of files.

I have not been able to find any reference to a file size limit in GUNICORN and the responses seem to suggest it doesn't and instead relies on the frontend webserver to do the HTTP body size limiting. Not discarding it yet but looking at all the alternatives.

gareththered
Posts: 5
Joined: Mon Feb 11, 2019 10:48 am

Re: File Upload Limit

Post by gareththered » Wed Feb 13, 2019 6:48 am

I've just been reading the Stackoverflow entries tagged dropzone.js and they seem to suggest that there is a `maxFile` and a `maxFileSize` option available, with the former being the number of files and the latter the size. The documentation you link to states that the default is 256, but doesn't state the units - 256 bytes, 256 TB, or as you said 256 files? None of those align with my single 1MB file though.

Also, the HTTP 413 error that shows up is a server error: https://httpstatuses.com/413 suggesting that this isn't a dropzone.js error.

EdwardZem
Posts: 2
Joined: Wed Mar 20, 2019 6:00 am

File Upload Limit

Post by EdwardZem » Sun Mar 24, 2019 12:17 am

Hi Panos,

If you want to increase the load of up to 20 MB, then you need to do:

Change the file php.ini:
upload_max_filesize = 20M
post_max_size = 20M

Or

Change the file .htaccess:
php_value upload_max_filesize 20M
php_value post_max_size 20M

jorgeegomez
Posts: 1
Joined: Fri Mar 29, 2019 11:45 pm

Re: File Upload Limit

Post by jorgeegomez » Sat Mar 30, 2019 12:07 am

Rosario, I recently encountered this error, and it doesn't seem to be an upload limit, but some internal processing limit.

What I tried: upload a 16 MB .zip file (after applying my fix to #572), with several documents. The file uploaded fine, but all documents over 1MB did not show up in /recent_added. All documents under 1MB show up fine.

The only thing I can see in the logs is a LockError in task_index_document. Hope this helps you diagnose.

User avatar
rosarior
Posts: 274
Joined: Tue Aug 21, 2018 3:28 am

Re: File Upload Limit

Post by rosarior » Wed Apr 03, 2019 10:38 pm

Finally managed to trigger this on a production install. The issue seems to be an out of memory problem with a subsequent OOM kill by the operating system kernel.

[2019-04-03 22:11:46,876: ERROR/MainProcess] Task sources.tasks.task_upload_document[1387c04b-8487-4193-b3c4-217f723f057c] raised unexpected: MagicException('cannot allocate 1436672320 bytes (Cannot allocate memory)',)
Traceback (most recent call last):
File "/opt/mayan-edms/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/mayan-edms/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/sources/tasks.py", line 84, in task_upload_document
querystring=querystring, user=user,
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/sources/models.py", line 147, in upload_document
file_object=file_object, _user=user,
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/documents/models.py", line 298, in new_version
document_version.save(_user=_user)
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/documents/models.py", line 638, in save
self.update_page_count(save=False)
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/documents/models.py", line 729, in update_page_count
file_object=file_object, mime_type=self.mimetype
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/converter/classes.py", line 90, in __init__
file_object=file_object, mimetype_only=False
File "/opt/mayan-edms/lib/python2.7/site-packages/mayan/apps/mimetype/api.py", line 16, in get_mimetype
file_mimetype = mime.from_buffer(file_object.read())
File "/opt/mayan-edms/lib/python2.7/site-packages/magic.py", line 82, in from_buffer
return self._handle509Bug(e)
File "/opt/mayan-edms/lib/python2.7/site-packages/magic.py", line 101, in _handle509Bug
raise e
MagicException: cannot allocate 1436672320 bytes (Cannot allocate memory)

...
[2308813.280915] [ 4929] 1001 4929 206983 116165 1343488 36207 0 mayan-edms.py
...
[2308813.281195] Out of memory: Kill process 4929 (mayan-edms.py) score 201 or sacrifice child
[2308813.281207] Killed process 4929 (mayan-edms.py) total-vm:827932kB, anon-rss:464648kB, file-rss:12kB, shmem-rss:0kB
[2308813.477890] oom_reaper: reaped process 4929 (mayan-edms.py), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

The system ran out of memory when attempting to load the entire document to perform MIME type introspection using the Python magic library, which in turn call the system's libmagic library. Need to investigate If Python magic tries to load the entire file at a time or if uses (can use) a generator to load it in chunks.

The same file doesn't cause a problem in development. More investigation is needed.

laramy2020
Posts: 1
Joined: Mon Oct 29, 2018 3:40 pm

Re: File Upload Limit

Post by laramy2020 » Fri Apr 12, 2019 5:24 pm

I am in a similar boat with mayan right now. Been looking in both django and gunicorn as to where to change file upload limits.
My setup is 3.1.9 docker deployment (one mayan prod and one mayan prod postgres)

Users get hit with a max upload of 256mib error
They get the same error through nginx and direct connection to mayan docker
nginx reverse proxy is set to 1000M

thanks for any help.

Post Reply