This OSError: write error
issue has been frustrating me for years — ever since
I began switching Python/Django web applications across from gunicorn to uWSGI
Emperor. Up until recently, I didn't understand what caused these errors,
whether they represented a problem worth solving, and if so how to deal with
them.
Problem
I deploy Python/Django sites with an Nginx frontend server, which proxies to a
Unix domain socket where uWSGI Emperor listening using the uWSGI protocol
(uwsgi_pass
). Something like this for Nginx:
# Nginx config (truncated for clarity)
location / {
uwsgi_pass unix:/tmp/site_name.sock;
include uwsgi_params;
}
And this for the uWSGI:
# uWSGI Emperor vassel config (truncated for clarity)
[uwsgi]
plugins = python3
chdir = /srv/site_name
home = /srv/venvs/site_name
module = project.wsgi
master = true
socket = /tmp/site_name.sock
processes = 3
vacuum = true
enable-threads = true # for Sentry
A few times a week, I would see an error like below, both on my Sentry error
tracking service, as well as in the uWSGI logs at /var/log/uwsgi/emperor.log
:
mysite - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request / (ip 1.2.3.4) !!!
mysite - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET / (1.2.3.4)
OSError: write error
Noteably, I did not receive any Django error emails about this, only notifications from Sentry.
From my literal reading of the above, I gather that the client web browser had closed the HTTP connection with Nginx and Nginx had closed the output stream that uWSGI was trying to write the response to. Seemed reasonable enough. I hadn't received any reports from customers of issues, so assumed the issue was not causing any serious inconvenience. Still, I couldn't reproduce it, which bothered me. And while I could mark the issues in Sentry as "ignore", they still counted towards the total weekly errors report.
I tried mobile devices, command-line browsers and load testing to see if I could pin down the cause of the issue. Nothing. The errors in the logs were sporadic and unpredictable.
Over the years I made a few blind attempts to fix the issue by adjusting Nginx's
uwsgi_ignore_client_abort
, uwsgi_read_timeout
, uwsgi_send_timeout
settings
and uWSGI's harakiri
setting. Despite what some StackOverflow posts would
suggest, none of these made any difference.
How to reproduce
I recently managed to replicate this issue. Turns out that all you need to do is open a non-trivial Django page in a regular desktop browser and hammer on the F5 (refresh) key a few times in rapid succession. Bingo!
Solution
Being able to reproduce the issue gave me confidence that the error was part of
normal operation and did not represent a negative experience for the visitor.
Given that the default Django mail_admins
logging handler didn't report the
error, it was clear that only Sentry was really a problem here.
After further testing I found that uWSGI's disable-write-exception
would
prevent the errors being logged by Sentry.
To prevent the (harmless) errors in the uWSGI log, you need all three of these lines in your uWSGI config:
ignore-sigpipe
ignore-write-errors
disable-write-exception
Each corresponds to one of the three error lines mentioned above.
Since applying the change, I've had no further notifications from Sentry about the issue and lovely clean weekly error reports.
An alternate fix is to just filter out these errors before they are sent to
Sentry. I added the following before_send
handler to the Sentry setup in the
production Django settings, returning None
when encountering an OSError:
write error
:
def before_send(event, hint):
"""Don't report "OSError: write error" to Sentry."""
exc_type, exc_value, _ = hint.get('exc_info', [None, None, None])
if exc_type == OSError and str(exc_value) == 'write error':
return None
else:
return event
sentry_sdk.init(
dsn='...',
before_send=before_send,
)