Graceful Gunicorn Timeouts

Update (2022-04-22): The described behaviour below is no longer accurate. Instead, --graceful-timeout seems completely broken. Today --timeout sends a SIGABRT as before but then 2 seconds later it sends a SIGKILL which you cannot trap. A GracefulExit handler may still be useful, just know you don't seem to be able to modify the grace period anymore.

In case you're using Gunicorn and haven't read the manual, I figured I'd let you know how your code can more gracefully handle timeouts. Gunicorn timeouts are signaled using the Unix SIGABRT signal. Gunicorn provides an option called graceful-timeout which allows you to have it send a second timeout signal using SIGTERM. This means you can trap the SIGABRT and have a few precious seconds to try and more gracefully handle a timeout rather than just abandoning mid routine to return a generic timeout error.

First, the config. While there are a couple ways to configure Gunicorn, this is how you'd configure it using command line arguments:

# Send SIGABRT at 5 seconds, send SIGTERM at 7 seconds
gunicorn --timeout 5 --graceful-timeout 7 …
# Send SIGABRT at 5 seconds followed by SIGKILL 2 seconds later
gunicorn --timeout 5 …

When it comes to writing Python code to use this config, here's an exception based system to handle graceful exits. First, we define a new interrupt object we can raise and a helper method to raise it.

import signal
import types


class GracefulExit(KeyboardInterrupt): """ An interrupt class to allow subroutines to try-except a block of code that should have a chance to perform logic in the event of a graceful exit signal (like keyboard interrupt or trapped Unix signal).
 Properties: signum: int The Unix signal number trapped. frame: frame The python stack frame at the moment of interrupt. """
 def __init__(self, signum: int, frame: types.FrameType): """ Store the values for handlers.
 Arguments: signum: int The Unix signal number trapped. frame: frame Python stack frame of execution. """ super().__init__(f"{signal.Signals(signum).name} received.") self.signum = signum self.frame = frame
 @staticmethod def throw(signum: int, frame: types.FrameType): """ Pass to signal.signal() to raise a GracefulExit exception. Eg:
 signal.signal(signal.SIGTERM, GracefulExit.throw)
 Arguments: signum: int The Unix signal number trapped. frame: frame Python stack frame of execution.
 Raises: GracefulExit This is the point… """ raise GracefulExit(signum, frame)

Next, when initializing everything we also need to register our handler for the signal:

signal.signal(signal.SIGABRT, GracefulExit.throw)

Finally, whenever we have a sensitive section of code, we can use a try-except block to provide a graceful exit strategy. Our interrupt class inherits from KeyboardInterrupt which provides us three options:

  1. catch KeyboardInterrupt which will also handle our new interrupt;
  2. catch GracefulExit to ignore KeyboardInterrupt issues; or
  3. catch both in their own block to provide different clean up routines.
try: … timeout code …
except GracefulExit: … clean up code …

In case the inheritance from KeyboardInterrupt doesn't make sense, you should know that pressing Ctrl+D while using a POSIX terminal sends the current process a SIGINT. This includes things running in CPython which by default raises a KeyboardInterrupt when this happens.