How to Run Cron in Kubernetes So Jobs Never Overlap or Vanish

Scheduled work fails quietly. A billing run that takes longer than usual starts a second copy of itself and double charges. A schedule stops firing after a control plane hiccup and nobody notices until the weekly report is missing. Neither is bad luck, both are default behaviour you can change.

Here is the manifest that gets it right, followed by why each field is there.

# k8s/cronjob-nightly-invoices.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-invoices
spec:
  schedule: "15 2 * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 3600
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: invoices
              image: registry.example.com/billing:1.4.2
              command: ["/app/bin/invoices", "--run-date=yesterday"]

Overlap

concurrencyPolicy defaults to Allow, which lets concurrent Jobs run. That default is right for a stateless cache warmer and wrong for anything that writes.

Forbid skips the new run if the previous one has not finished. Replace kills the running Job and starts the new one. For financial or reporting work Forbid is almost always the answer, because a skipped run is visible and recoverable while two concurrent runs are neither.

Even with Forbid, keep the job itself idempotent. The documentation is direct about this, a CronJob creates a Job approximately once per execution time, circumstances exist where two Jobs or no Job get created, and the Jobs you define should be idempotent. Guard the work with a database transaction or a run key rather than trusting the scheduler.

Vanishing schedules

This is the failure that costs an afternoon of confusion. For every CronJob, the controller checks how many schedules it missed between its last scheduled time and now. If there are more than 100 missed schedules it does not start the Job at all, and logs an error reading "too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew".

A CronJob suspended for a week, or a control plane that was down long enough, lands in exactly that state and then stays stuck. The fix is startingDeadlineSeconds, which changes what the controller counts. With a deadline set, it measures the time between when a Job was expected and now, and skips the execution if the difference is larger than the limit, so old misses stop accumulating into the lockout.

Do not set it below ten seconds. The CronJob controller checks things every ten seconds, so a deadline shorter than that can mean the Job is never scheduled at all. Five minutes is a reasonable value for a nightly job.

Time zones

.spec.timeZone takes an IANA name such as Etc/UTC or Europe/Berlin and has been stable since v1.27. Putting CRON_TZ or TZ variables inside .spec.schedule is not officially supported and produces a validation error.

Daylight saving is the reason to care. A job scheduled at 02:30 in a zone that observes DST will run twice on one night of the year and not at all on another. Schedule anything financial in UTC and do the timezone conversion inside the job.

History, so there is something to read

successfulJobsHistoryLimit defaults to 3 and failedJobsHistoryLimit defaults to 1. One failed Job is not much of an audit trail when a job has been failing intermittently for a fortnight, so raise both for anything important. Setting either to 0 keeps none.

Job history is the primary debugging surface, since Pod logs disappear with the Pod.

kubectl get jobs -l batch.kubernetes.io/cronjob-name=nightly-invoices --sort-by=.metadata.creationTimestamp
kubectl logs job/nightly-invoices-29001234

Since Kubernetes v1.32 the CronJob controller also stamps created Jobs with a batch.kubernetes.io/cronjob-scheduled-timestamp annotation, which tells you the schedule slot a Job belongs to rather than only when it happened to start.

If you prefer time based cleanup, ttlSecondsAfterFinished on the Job spec deletes finished Jobs after a fixed delay. Both mechanisms delete Jobs, so if you set both, the one that fires first wins. That feature is also sensitive to clock skew in the cluster, which can cause cleanup at the wrong time.

Failure handling

backoffLimit defaults to 6 retries. Failed Pods are recreated with an exponential back-off delay of 10s, 20s, 40s and so on, capped at six minutes, so six retries can stretch across a long window. Lower it for a job that either works immediately or needs a human.

activeDeadlineSeconds is the wall clock stop. It takes precedence over backoffLimit, so a Job that hits the time limit stops deploying new Pods even if retries remain. Every scheduled job should have one, chosen a comfortable margin above the longest healthy run, otherwise a hung job holds a Forbid lock forever and every subsequent schedule is skipped.

restartPolicy in a Job's Pod template only accepts Never or OnFailure. OnFailure restarts the container in place, which is faster; Never creates a fresh Pod per attempt, which leaves clearer evidence behind.

Running one by hand

Testing a schedule by waiting for it is a bad use of an evening. Trigger a run from the CronJob definition instead.

kubectl create job invoices-manual-001 --from=cronjob/nightly-invoices

And when a job needs to stop without being deleted, suspend it rather than commenting out the schedule.

kubectl patch cronjob nightly-invoices -p '{"spec":{"suspend":true}}'

Remember the missed schedule counter when you unsuspend a job that has been paused for a long time, since that is precisely the situation startingDeadlineSeconds exists to survive.

Alert on absence

Nothing above tells you a job stopped running, and Kubernetes will not either. A job that never starts produces no failed Pod and no alert. Have each run touch a heartbeat, then alert when the heartbeat is older than the interval plus a margin. Absence is the failure mode that scheduled work is worst at reporting on its own.

Getting this right across a fleet of scheduled work is standard practice in our SRE services and our ongoing Kubernetes management.