Published: Tue 21 September 2021
By Ben Sturmfels
In Technology .
tags: python django coding work
This OSError: write error issue has been frustrating me for years — ever since
I began switching Python/Django web applications across from gunicorn to uWSGI
Emperor. Up until recently, I didn't understand what caused these errors,
whether they represented a problem worth solving, and if so how to deal with
them.
Problem
I deploy Python/Django sites with an Nginx frontend server, which proxies to a
Unix domain socket where uWSGI Emperor listening using the uWSGI protocol
(uwsgi_pass). Something like this for Nginx:
# Nginx config (truncated for clarity)
location / { uwsgi_pass unix:/tmp/site_name.sock ; include uwsgi_params ; } And this for the uWSGI:
# uWSGI Emperor vassel config (truncated for clarity) [uwsgi] plugins = python3 chdir = /srv/site_name home = /srv/venvs/site_name module = project.wsgi master = true socket = /tmp/site_name.sock processes = 3 vacuum = true enable-threads = true # for Sentry A few times a week, I would see an error like below, both on my Sentry error
tracking service, as well as in the uWSGI logs at /var/log/uwsgi/emperor.log:
mysite - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request / (ip 1.2.3.4) !!!
mysite - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET / (1.2.3.4)
OSError: write error
Noteably, I did not receive any Django error emails about this, only notifications
from Sentry.
From my literal reading of the above, I gather that the client web browser had
closed the HTTP connection with Nginx and Nginx had closed the output stream
that uWSGI was trying to write the response to. Seemed reasonable enough. I
hadn't received any reports from customers of issues, so assumed the issue was
not causing any serious inconvenience. Still, I couldn't reproduce it, which
bothered me. And while I could mark the issues in Sentry as "ignore", they still
counted towards the total weekly errors report.
I tried mobile devices, command-line browsers and load testing to see if I could
pin down the cause of the issue. Nothing. The errors in the logs were sporadic
and unpredictable.
Over the years I made a few blind attempts to fix the issue by adjusting Nginx's
uwsgi_ignore_client_abort, uwsgi_read_timeout, uwsgi_send_timeout settings
and uWSGI's harakiri setting. Despite what some StackOverflow posts would
suggest, none of these made any difference.
How to reproduce
I recently managed to replicate this issue. Turns out that all you need
to do is open a non-trivial Django page in a regular desktop browser and hammer
on the F5 (refresh) key a few times in rapid succession. Bingo!
Solution
Being able to reproduce the issue gave me confidence that the error was part of
normal operation and did not represent a negative experience for the visitor.
Given that the default Django mail_admins logging handler didn't report the
error, it was clear that only Sentry was really a problem here.
After further testing I found that uWSGI's disable-write-exception would
prevent the errors being logged by Sentry.
To prevent the (harmless) errors in the uWSGI log, you need all three of these
lines in your uWSGI config:
ignore-sigpipe ignore-write-errors disable-write-exception Each corresponds to one of the three error lines mentioned above.
Since applying the change, I've had no further notifications from Sentry about
the issue and lovely clean weekly error reports.
An alternate fix is to just filter out these errors before they are sent to
Sentry. I added the following before_send handler to the Sentry setup in the
production Django settings, returning None when encountering an OSError:
write error:
def before_send ( event , hint ):
"""Don't report "OSError: write error" to Sentry."""
exc_type , exc_value , _ = hint . get ( 'exc_info' , [ None , None , None ])
if exc_type == OSError and str ( exc_value ) == 'write error' :
return None
else :
return event
sentry_sdk . init (
dsn = '...' ,
before_send = before_send ,
)
