How to fix uWSGI "OSError: write error"

This OSError: write error issue has been frustrating me for years — ever since I began switching Python/Django web applications across from gunicorn to uWSGI Emperor. Up until recently, I didn't understand what caused these errors, whether they represented a problem worth solving, and if so how to deal with them.

Problem

I deploy Python/Django sites with an Nginx frontend server, which proxies to a Unix domain socket where uWSGI Emperor listening using the uWSGI protocol (uwsgi_pass). Something like this for Nginx:

# Nginx config (truncated for clarity)
location / {
     uwsgi_pass unix:/tmp/site_name.sock;
     include uwsgi_params;
}

And this for the uWSGI:

# uWSGI Emperor vassel config (truncated for clarity)
[uwsgi]
plugins = python3
chdir = /srv/site_name
home = /srv/venvs/site_name
module = project.wsgi
master = true
socket = /tmp/site_name.sock
processes = 3
vacuum = true
enable-threads = true  # for Sentry

A few times a week, I would see an error like below, both on my Sentry error tracking service, as well as in the uWSGI logs at /var/log/uwsgi/emperor.log:

mysite - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request / (ip 1.2.3.4) !!!
mysite - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET / (1.2.3.4)
OSError: write error

Noteably, I did not receive any Django error emails about this, only notifications from Sentry.

From my literal reading of the above, I gather that the client web browser had closed the HTTP connection with Nginx and Nginx had closed the output stream that uWSGI was trying to write the response to. Seemed reasonable enough. I hadn't received any reports from customers of issues, so assumed the issue was not causing any serious inconvenience. Still, I couldn't reproduce it, which bothered me. And while I could mark the issues in Sentry as "ignore", they still counted towards the total weekly errors report.

I tried mobile devices, command-line browsers and load testing to see if I could pin down the cause of the issue. Nothing. The errors in the logs were sporadic and unpredictable.

Over the years I made a few blind attempts to fix the issue by adjusting Nginx's uwsgi_ignore_client_abort, uwsgi_read_timeout, uwsgi_send_timeout settings and uWSGI's harakiri setting. Despite what some StackOverflow posts would suggest, none of these made any difference.

How to reproduce

I recently managed to replicate this issue. Turns out that all you need to do is open a non-trivial Django page in a regular desktop browser and hammer on the F5 (refresh) key a few times in rapid succession. Bingo!

Solution

Being able to reproduce the issue gave me confidence that the error was part of normal operation and did not represent a negative experience for the visitor. Given that the default Django mail_admins logging handler didn't report the error, it was clear that only Sentry was really a problem here.

After further testing I found that uWSGI's disable-write-exception would prevent the errors being logged by Sentry.

To prevent the (harmless) errors in the uWSGI log, you need all three of these lines in your uWSGI config:

ignore-sigpipe
ignore-write-errors
disable-write-exception

Each corresponds to one of the three error lines mentioned above.

Since applying the change, I've had no further notifications from Sentry about the issue and lovely clean weekly error reports.

An alternate fix is to just filter out these errors before they are sent to Sentry. I added the following before_send handler to the Sentry setup in the production Django settings, returning None when encountering an OSError: write error:

def before_send(event, hint):
    """Don't report "OSError: write error" to Sentry."""
    exc_type, exc_value, _ = hint.get('exc_info', [None, None, None])
    if exc_type == OSError and str(exc_value) == 'write error':
        return None
    else:
        return event

sentry_sdk.init(
    dsn='...',
    before_send=before_send,
)

(my software/consulting business)

links

social

Supporting

FSF member since 2006 Become a Conservancy Supporter!