Graceful Gunicorn Timeouts

Update (2022-04-22): The described behaviour below is no longer accurate. Instead, --graceful-timeout seems completely broken. Today --timeout sends a SIGABRT as before but then 2 seconds later it sends a SIGKILL which you cannot trap. A GracefulExit handler may still be useful, just know you don't seem to be able to modify the grace period anymore.

In case you're using Gunicorn and haven't read the manual, I figured I'd let you know how your code can more gracefully handle timeouts. Gunicorn timeouts are signaled using the Unix SIGABRT signal. Gunicorn provides an option called graceful-timeout which allows you to have it send a second timeout signal using SIGTERM. This means you can trap the SIGABRT and have a few precious seconds to try and more gracefully handle a timeout rather than just abandoning mid routine to return a generic timeout error.

First, the config. While there are a couple ways to configure Gunicorn, this is how you'd configure it using command line arguments:

# Send SIGABRT at 5 seconds, send SIGTERM at 7 seconds
gunicorn --timeout 5 --graceful-timeout 7 ...
# Send SIGABRT at 5 seconds followed by SIGKILL 2 seconds later
gunicorn --timeout 5 ...

When it comes to writing Python code to use this config, here's an exception based system to handle graceful exits. First, we define a new interrupt object we can raise and a helper method to raise it.

import signal
import types


class GracefulExit(KeyboardInterrupt):
	"""
	An interrupt class to allow subroutines to try-except a block of code
	that should have a chance to perform logic in the event of a graceful
	exit signal (like keyboard interrupt or trapped Unix signal).

	Properties:
		signum: int
			The Unix signal number trapped.
		frame: frame
			The python stack frame at the moment of interrupt.
	"""

	def __init__(self, signum: int, frame: types.FrameType):
		"""
		Store the values for handlers.

		Arguments:
			signum: int
				The Unix signal number trapped.
			frame: frame
				Python stack frame of execution.
		"""
		super().__init__(f"{signal.Signals(signum).name} received.")
		self.signum = signum
		self.frame = frame

	@staticmethod
	def throw(signum: int, frame: types.FrameType):
		"""
		Pass to signal.signal() to raise a GracefulExit exception. Eg:

			signal.signal(signal.SIGTERM, GracefulExit.throw)

		Arguments:
			signum: int
				The Unix signal number trapped.
			frame: frame
				Python stack frame of execution.

		Raises:
			GracefulExit
				This is the point...
		"""
		raise GracefulExit(signum, frame)

Next, when initialising everything we also need to register our handler for the signal:

signal.signal(signal.SIGABRT, GracefulExit.throw)

Finally, whenever we have a sensitive section of code, we can use a try-except block to provide a graceful exit strategy. Our interrupt class inherits from KeyboardInterrupt which provides us three options:

1. catch KeyboardInterrupt which will also handle our new interrupt;
2. catch GracefulExit to ignore KeyboardInterrupt issues; or
3. catch both in their own block to provide different clean up routines.

try:
	... timeout code ...
except GracefulExit:
	... clean up code ...

In case the inheritance from KeyboardInterrupt doesn't make sense, you should know that pressing Ctrl+D while using a POSIX terminal sends the current process a SIGINT. This includes things running in cpython which by default raises a KeyboardInterrupt when this happens.