How to Patch on a Schedule Without a Maintenance Window

The window is the problem. A quarterly maintenance window means three months of known vulnerabilities waiting for a Saturday night, and then a Saturday night on which forty machines change at once while the people who could diagnose the fallout are asleep.

Patching continuously and restarting in waves gets fixes on faster and takes the outage out of the process. The pieces are unglamorous and they are all in the operating system already.

Turn automatic updates on, then prove they run

On Debian and Ubuntu the package is unattended-upgrades and the switch is two lines in a separate file.

// /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

Installing the package and enabling it are separate acts, and a host can sit with the first and not the second for years while a dashboard reports the package as present. Ubuntu's documentation gives the dry run that shows what it would really do.

sudo unattended-upgrade -v --dry-run

Know what the default actually allows

The shipped Ubuntu configuration allows more than security updates, and it explains itself in its own comments. Unattended-Upgrade::Allowed-Origins lists the release pocket next to the security pocket and the expanded maintenance pockets, with -updates present but commented out. The reasoning in the file is that "in Ubuntu security updates may pull in new dependencies from non-security sources", so allowing the release pocket lets those dependencies come in rather than blocking the security fix that needs them.

// /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";
        "${distro_id}ESMApps:${distro_codename}-apps-security";
        "${distro_id}ESM:${distro_codename}-infra-security";
};

If a package must never move without a human, Unattended-Upgrade::Package-Blacklist takes Python regular expressions. The shipped file warns about the trap in its own examples, that without a $ anchor an entry of libc6 also matches libc6-dev and libc6-i686.

The schedule already staggers, which is not the same as rolling

Two systemd timers drive this. apt-daily.timer handles the download side twice a day with a randomised delay of up to twelve hours, and apt-daily-upgrade.timer runs the upgrade at 06:00 with a randomised delay of up to an hour, ordered after the download timer.

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true

That randomisation stops a fleet hitting the mirror in the same second. It knows nothing about which of your hosts sit behind the same load balancer, so it will happily pick two of them at once. To control that, override the timer per group. Note the empty assignment on the first line, which systemd requires because "if the empty string is assigned to any of these options, the list of timers is reset".

# /etc/systemd/system/apt-daily-upgrade.timer.d/override.conf
[Timer]
OnCalendar=
OnCalendar=Tue *-*-* 02:00
RandomizedDelaySec=15m

sudo systemctl daemon-reload
systemctl list-timers apt-daily-upgrade.timer

Wave A on Tuesday, wave B on Thursday. Two rules build the waves. No two hosts serving the same traffic may share a wave, and the first wave is a single host that carries real production traffic so a bad package meets one machine instead of the fleet.

The reboot question, answered by the machine rather than by a habit

Installing a package does not restart what is already running, and the kernel is a separate question from the services. Debian and Ubuntu answer the first with a file. A package that needs a reboot touches /var/run/reboot-required from its postinst script and appends its own name to /run/reboot-required.pkgs.

if [ -f /var/run/reboot-required ]; then
  echo "reboot pending for: $(tr '\n' ' ' < /var/run/reboot-required.pkgs 2>/dev/null)"
fi

unattended-upgrades can act on that file by itself. Unattended-Upgrade::Automatic-Reboot defaults to false, and when set true it reboots without confirmation if the file exists after an upgrade. Automatic-Reboot-Time defaults to now and Automatic-Reboot-WithUsers defaults to true, so a logged-in session will not hold it back. On any fleet with a load balancer in front, leave that off and let your wave tooling take the host out, reboot it, health check it, and put it back.

On RHEL and its rebuilds, dnf needs-restarting answers both questions. With -r it will "only report whether a reboot is required (exit code 1) or not (exit code 0)", which is the exact shape a health check wants, and -s lists the affected systemd services so you can restart those instead of rebooting.

dnf needs-restarting -r || echo "this host needs a reboot"
dnf needs-restarting -s

The automatic side there is dnf-automatic, configured in /etc/dnf/automatic.conf and driven by dnf-automatic.timer, alongside separate notify-only, download and install timers that override the configured behaviour. The shipped defaults are conservative and worth knowing before assuming anything is being applied.

# /etc/dnf/automatic.conf
[commands]
upgrade_type = security
random_sleep = 0
download_updates = yes
apply_updates = yes
reboot = never

upgrade_type takes default or security and ships as default. apply_updates ships as no, and reboot takes never, when-changed or when-needed and ships as never. A host that downloads updates and never applies them looks busy in a log and is not patched.

Kernel fixes that buy time before the reboot

Ubuntu's Livepatch service patches "the Linux kernel between security maintenance windows, while the system runs", covering kernel vulnerabilities at the critical and high CVSS and Ubuntu Priority ratings, and it comes with Ubuntu Pro. Canonical is explicit that it is "not a replacement for rebooting". Treat it as cover for the gap between a kernel fix landing and the host's turn coming round, rather than as permission to stop rebooting.

Alert on the decay, not on the runs

A patch pipeline fails quietly, so alert on state rather than on events. A host whose /var/log/unattended-upgrades/unattended-upgrades.log has not changed in a week is not patching. A host carrying /var/run/reboot-required for longer than one full wave cycle has missed its turn. A host whose last successful upgrade is far older than the rest of the fleet is usually a machine that somebody excluded during an incident and never put back.

Draining a host cleanly before its reboot is its own piece of work, covered in how to drain a node without dropping a single request. If you would rather this ran without anyone remembering to check it, that is the routine part of our infrastructure management and servers management engagements.