Unable to upload files greater than 128mb via http

Avatar
bigcomposer
  • Answered
We have a weird issue that popped up on our website and after several rounds with inMotion Support, they recommended I post here We have an upload form that abruptly resets the connection when trying to upload files over 128mb. Please know ahead of time that inMotion Support has verified that our php.ini file is correctly configured. Also, this script worked perfectly last year -- Our members only post files from March to June, so I can't pinpoint a time when this behavior began.

On our site, we allow members to upload files to the server. This is in a secure area, but for testing purposes we have also created a very simple PHP upload script, but the behavior is the same.

â?¢ The problem only manifests with files larger than 128 MB. We have verified a file of ~127MB uploads successfully even on a slow connection with the whole transaction lasting >5:00, but a file of ~130MB fails ~:15 after the form has submitted. This is very specific behavior.

â?¢ There are no decisions made regarding acceptance of a file upload based on its size or anything else that would explain this very specific behavior in our php code.

â?¢ There is no JS running on the client that could make a decision based on file size or account for this very specific behavior.

â?¢ There are no 3rd party libraries or unknown quantities in play here. This is straight-up, kind of boring PHP...

â?¢ We are experiencing the same very specific behavior uploading a file >128MB using a completely different php script on your server that is outside of our main codebase.

â?¢ This script is in use on other websites with other hosting services.

Because we are on a shared hosting package at inMotion, changing any Apache configurations (even if they're incorrect) does not seem to be an option. I would greatly appreciate any help you might be able to give.
Replies 1
Avatar
0
JacobIMH
Hello bigcomposer, thanks for the detailed question, Your thoroughness helped me narrow down the issue relatively easily. I created some 1MB, 127MB, and 128MB test .zip files, and sure enough had no issues with the first two. However with the last 128MB one, I had no such luck. Strangely enough, the request seemed to just die off. In Google Chrome, I could see in the bottom-left status area Uploading (1%).... It got up to Uploading (4%)... then reset back to Uploading (0%)..., got back up to Uploading (4%)..., then I got the This webpage is not available error in Chrome. Taking a look at your site's access logs I noticed that when I tried to upload the 1MB or 127MB files, the POST request from my browser got a 200 OK response from the server, stating your PHP script was found and accepting the POST data. When I attempted to upload a 128MB file, I could see the server was instead sending back a 404 Not Found response right away. So I started the file upload, but then my web-browser couldn't communicate to the PHP script to properly upload it, it bugs out tries again, and then serves me a default 404 error page.

Apache error logs and ModSecurity

To troubleshoot further I took a look at the tcmpdump info for my domain during the request, I also had a grep running for my IP to your website's access log, and the Apache error logs: 
tcpdump -n host 1.2.3.4 | tee -a IP_LOG &
 tail -f ~userna5/access-logs/example.com | grep --color 1.2.3.4 &
 tail -f /usr/local/apache/logs/error_log | grep --color 1.2.3.4 &
This is what I noticed in the Apache error log: 
[error] ModSecurity: Request body (Content-Length) is larger than the configured limit (134217728).
So it's ModSecurity that's intercepting your large POSTs, as 134217728 bytes is 128MB. This is a setting that can be raised at the server-wide level with the SecRequestBodyInMemoryLimit directive. However, I believe this is the default ModSecurity setting, and not something I'm sure we'll change on shared hosting. I can check for you tomorrow to find out for sure, and update this answer accordingly. Sorry for the troubles, and thanks again for providing a great deal of info in your question. - Jacob