How to monitor Kubernetes CronJobs

To monitor a Kubernetes CronJob, have the job's container ping a monitoring URL when it starts and when it finishes, with its exit code. Cronomonitor then alerts you if a run fails, hangs, or never starts at all.

Why monitor it

Kubernetes keeps failed Jobs around, but nothing tells you about them, and some failures leave no Job behind: a suspended CronJob, a missed schedule while the control plane was busy, an image that can't be pulled, or a pod stuck Pending because the cluster ran out of capacity.

1. Create a check

Sign up for Cronomonitor, create a project and add a check named after the job. Give it the job's schedule, here the cron expression 0 3 * * * (every day at 3am, UTC), and a grace period that covers how long the job normally takes.

The check gives you a unique ping URL. Every snippet below is also on the check's page in the dashboard, already filled in with that URL and schedule.

2. Add the ping to your job

The image needs curl. timeZone needs Kubernetes 1.27 or later.

nightly-report-cronjob.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 3 * * *"
  timeZone: "UTC"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: nightly-report
              image: your-image:latest
              env:
                - name: PING_URL
                  value: "https://cronomonitor.com/api/ping/<your-check-id>"
              command: ["/bin/sh", "-c"]
              args:
                - |
                  curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL/start"
                  /path/to/your-job.sh
                  STATUS=$?
                  curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL/$STATUS"
                  exit $STATUS

How it works

The container pings /start, runs the job, then pings again with the job's exit code, so a non-zero exit reports a failure straight away. concurrencyPolicy: Forbid stops a slow run from overlapping the next one.

With a cron schedule, the manifest sets timeZone to the check's time zone, so Kubernetes and Cronomonitor agree on when the job is due.

Retries send a ping each

A Job retries a failed pod up to backoffLimit times (6 by default), and each attempt pings. You'll get one failure alert, then a recovery if a retry succeeds. Set backoffLimit: 0 if a failed run shouldn't be retried.

Missed schedules

If the CronJob controller misses a schedule, for example during an upgrade, it starts the job late only within startingDeadlineSeconds. If it has missed more than 100 schedules it stops trying until you fix it. In both cases no pod runs, so no ping arrives and Cronomonitor alerts you once the grace period passes.

Keep the ping URL in a Secret

Anyone with the ping URL can ping the check, so in a shared cluster put it in a Secret and reference it with valueFrom.secretKeyRef rather than writing it into the manifest.

3. Test it

Run the job once by hand, or use Send a test ping on the check. The check turns green as soon as the first ping arrives, and from then on Cronomonitor expects the next one on schedule.

To check where alerts go, use Send test alert on the check. To see a real failure alert, make the job exit with an error once.

If the check says the job didn't run

kubectl get cronjob <name> shows the last schedule time and whether the CronJob is suspended.

kubectl get jobs --sort-by=.metadata.creationTimestamp lists recent runs; kubectl describe job <job> shows why pods failed.

kubectl logs job/<job> prints the output of the job's pod.