How to monitor Python scheduled jobs

To monitor a Python scheduled job, ping a monitoring URL when the job starts and when it finishes, and send /fail with the traceback if it raises. Cronomonitor alerts you on a failure, a hang, or a run that never happens.

Why monitor it

Python jobs run in many ways: cron calling a script, a Django management command, Celery beat, APScheduler or a cloud scheduler calling a function. The same few lines work in all of them, because the job only needs to make an HTTPS request.

1. Create a check

Sign up for Cronomonitor, create a project and add a check named after the job. Give it the job's schedule, here the cron expression 0 3 * * * (every day at 3am, UTC), and a grace period that covers how long the job normally takes.

The check gives you a unique ping URL. Every snippet below is also on the check's page in the dashboard, already filled in with that URL and schedule.

2. Add the ping to your job

Example

import traceback
import urllib.request

PING_URL = "https://cronomonitor.com/api/ping/<your-check-id>"


def ping(path="", data=None):
    try:
        urllib.request.urlopen(PING_URL + path, data=data, timeout=10)
    except OSError:
        pass  # never let monitoring break the job


ping("/start")
try:
    run_job()  # your code here
except Exception:
    ping("/fail", traceback.format_exc()[-10000:].encode())
    raise
ping()

How it works

The ping helper uses only the standard library and swallows network errors, so monitoring can never break the job. If the job raises, the last 10 KB of the traceback goes with the /fail ping and appears in the check's ping log, then the exception is re-raised as before.

Put your job's code where run_job() is called. If you already use the requests library, requests.post(PING_URL + path, data=data, timeout=10) works just as well.

Celery beat and APScheduler

Wrap the body of the scheduled task in the same start, success and fail pings. Give each periodic task its own check, so you know exactly which one stopped.

Exit codes still count

If a script reports failure with sys.exit(1) rather than raising, the pattern above doesn't see it. Either ping /fail before exiting, or run the script from cron with the exit-code form of the ping URL.

3. Test it

Run the job once by hand, or use Send a test ping on the check. The check turns green as soon as the first ping arrives, and from then on Cronomonitor expects the next one on schedule.

To check where alerts go, use Send test alert on the check. To see a real failure alert, make the job exit with an error once.

If the check says the job didn't run

Run the script by hand with the same Python and virtualenv the scheduler uses, to catch import errors.

Read the check's ping log: a /start ping with no finish means the job hung or was killed.