Graceful Gunicorn Timeouts
Update (2022-04-22): The described behaviour below is no longer accurate. Instead,
--graceful-timeout seems completely broken. Today --timeout sends a
SIGABRT as before but then 2 seconds later it sends a SIGKILL which you cannot
trap. A GracefulExit
handler may still be useful, just know you don't seem to be able
to modify the grace period anymore.
In case you're using Gunicorn and haven't read the manual, I
figured I'd let you know how your code can more gracefully handle timeouts.
Gunicorn timeouts are signaled using
the Unix SIGABRT signal.
Gunicorn provides an option called
graceful-timeout
which allows you to have it send a second timeout signal using
SIGTERM. This
means you can trap the SIGABRT and have a few precious seconds to try and more gracefully
handle a timeout rather than just abandoning mid routine to return a generic timeout error.
First, the config. While there are a couple ways to configure Gunicorn, this is how you'd configure it using command line arguments:
# Send SIGABRT at 5 seconds, send SIGTERM at 7 seconds
gunicorn --timeout 5 --graceful-timeout 7 …
# Send SIGABRT at 5 seconds followed by SIGKILL 2 seconds later
gunicorn --timeout 5 …
When it comes to writing Python code to use this config, here's an exception based system to handle graceful exits. First, we define a new interrupt object we can raise and a helper method to raise it.
import signal
import types
class GracefulExit(KeyboardInterrupt): """ An interrupt class to allow subroutines to try-except a block of code that should have a chance to perform logic in the event of a graceful exit signal (like keyboard interrupt or trapped Unix signal).
Properties: signum: int The Unix signal number trapped. frame: frame The python stack frame at the moment of interrupt. """
def __init__(self, signum: int, frame: types.FrameType): """ Store the values for handlers.
Arguments: signum: int The Unix signal number trapped. frame: frame Python stack frame of execution. """ super().__init__(f"{signal.Signals(signum).name} received.") self.signum = signum self.frame = frame
@staticmethod def throw(signum: int, frame: types.FrameType): """ Pass to signal.signal() to raise a GracefulExit exception. Eg:
signal.signal(signal.SIGTERM, GracefulExit.throw)
Arguments: signum: int The Unix signal number trapped. frame: frame Python stack frame of execution.
Raises: GracefulExit This is the point… """ raise GracefulExit(signum, frame)
Next, when initializing everything we also need to register our handler for the signal:
signal.signal(signal.SIGABRT, GracefulExit.throw)
Finally, whenever we have a sensitive section of code, we can use a try-except block to provide a
graceful exit strategy. Our interrupt class inherits from KeyboardInterrupt
which
provides us three options:
- catch
KeyboardInterrupt
which will also handle our new interrupt; - catch
GracefulExit
to ignoreKeyboardInterrupt
issues; or - catch both in their own block to provide different clean up routines.
try: … timeout code …
except GracefulExit: … clean up code …
In case the inheritance from KeyboardInterrupt
doesn't make sense, you should know that
pressing Ctrl+D while using a
POSIX terminal sends the current process a
SIGINT. This includes
things running in CPython which by default
raises a KeyboardInterrupt
when this happens.