Differences

This shows you the differences between two versions of the page.

--- docker:reliable-compose-startup-after-reboot [2026/07/06 19:12] – created 127.0.0.1
+++ docker:reliable-compose-startup-after-reboot [Unknown date] (current) – external edit (Unknown date) 127.0.0.1
@@ Line 1: / Line 1: @@
+====== Reliable Docker Compose Startup After a Reboot ======
+A common and confusing failure mode: a Docker Compose stack that runs perfectly when you start it by hand starts misbehaving **after the server reboots**. One container pins a CPU core, the machine runs hot and the fans spin up — even though the overall system load is near zero. This page explains **why it happens** and shows a clean, general way to prevent it.
+===== The trap: ''depends_on'' is not honored at reboot =====
+Compose lets you declare start-up order and readiness gating:
+<code yaml>
+services:
+  aggregator:
+    depends_on:
+      backend:
+        condition: service_healthy
+</code>
+This tells Compose: //"do not start ''aggregator'' until ''backend'' reports healthy."// It works exactly as expected — **but only when you run ''docker compose up''.**
+The catch: after a **host reboot**, your containers are usually brought back not by Compose, but by the **Docker daemon's restart policy** (''restart: always'' or ''restart: unless-stopped''). The daemon restarts containers in an **arbitrary order and ignores ''depends_on'' entirely.**
+> **Key fact:** ''depends_on: condition: service_healthy'' is respected by ''docker compose up'' — **never** by the Docker daemon when it restarts containers at boot.
+===== Why this causes a runaway (hot fans, pinned core) =====
+Many "aggregator", "proxy" or "gateway" style services discover their backends **once at start-up** and never retry. If such a service starts **before** its backend is ready — exactly what the reboot race produces — it can end up:
+  * holding a dead or half-open connection,
+  * retrying in a tight loop with no back-off,
+  * pinning a full CPU core at ~100%.
+The result: the box is essentially idle (load near zero) yet **one core runs flat out**, temperatures climb and the fans ramp up — with no obvious "busy" process at a casual glance.
+**Analogy:** it is like a courier who memorises the delivery list once, at the door, //before// the addresses are posted — then drives in circles forever because the list came up blank.
+===== The clean fix: let systemd own start-up =====
+The reliable pattern is to stop relying on the daemon's restart policy for boot ordering, and hand that job to **systemd** — which //does// respect Compose's health gating, because it runs ''docker compose up'' itself.
+=== 1. Stop the daemon from race-starting containers ===
+Change the restart policy in your ''docker-compose.yml'' from ''unless-stopped'' to ''on-failure'':
+<code yaml>
+services:
+  backend:
+    restart: on-failure
+  aggregator:
+    restart: on-failure
+</code>
+**Why:** the daemon auto-starts ''always'' and ''unless-stopped'' containers at boot (that is the race). It does **not** auto-start ''on-failure'' containers at boot — yet ''on-failure'' still restarts a container that crashes while running. So you keep crash recovery but remove the boot race.
+=== 2. Make systemd start the whole stack, in order ===
+Create ''/etc/systemd/system/mystack.service'':
+<code ini>
+[Unit]
+Description=My application stack (docker compose)
+Requires=docker.service
+After=docker.service network-online.target
+Wants=network-online.target
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/mystack
+ExecStart=/usr/bin/docker compose up -d --wait
+ExecStop=/usr/bin/docker compose stop
+[Install]
+WantedBy=multi-user.target
+</code>
+Enable it once:
+<code bash>
+sudo systemctl daemon-reload
+sudo systemctl enable --now mystack.service
+</code>
+Now systemd is the **single** thing that starts the stack at boot, and ''--wait'' blocks until every service is healthy — so the aggregator always comes up **after** its backends.
+=== 3. Handle cold-boot transients with a retry ===
+On a cold boot, a backend that needs the network (DNS, a remote API) may crash-loop for a few seconds until networking is fully up. ''docker compose up --wait'' is **intolerant** of this: the moment any container reports //unhealthy//, it aborts — it will not wait for the container to recover on its own.
+Make the unit retry, so the stack converges by itself:
+<code ini>
+[Unit]
+# ...
+StartLimitIntervalSec=900
+StartLimitBurst=6
+[Service]
+# ...
+Restart=on-failure
+RestartSec=20
+</code>
+If the first attempt fails because a backend was still settling, systemd waits and re-runs ''compose up --wait''. By the retry the backend is healthy, the aggregator starts, and the unit goes ''active'' — with **zero manual intervention**. Seeing a handful of restarts in the boot log is normal, not a fault:
+<code bash>
+systemctl show mystack.service -p NRestarts --value
+</code>
+===== How to diagnose the runaway =====
+If a box runs hot after a reboot but overall load is low, look for a single pinned container:
+<code bash>
+# per-container CPU — the culprit sits near a full core
+docker stats --no-stream
+# confirm it is a spin, not real work, then read its logs
+# for a tight retry / error loop (e.g. the same request repeating)
+docker logs --tail 50 <container>
+</code>
+A quick manual recovery (until the boot fix is in place) is simply to restart the affected container //once its dependencies are up//:
+<code bash>
+docker restart <container>
+</code>
+===== Optional: a runtime watchdog =====
+The steps above fix **boot** ordering. A separate, rarer problem is a long-running proxy that holds a **persistent session** to a backend and has no working auto-reconnect: if the backend restarts //during normal operation//, the session dies and the proxy may spin again. If your software has this limitation, a small periodic health-check script (driven by a systemd timer or cron) that restarts the proxy when it detects the fault is a reasonable backstop. Guard it with a **minimum interval** between restarts so it can never storm.
+===== Summary =====
+  * ''depends_on: service_healthy'' orders **''compose up''**, not the daemon at reboot.
+  * Use ''restart: on-failure'' so the daemon does not race-start containers at boot.
+  * Let a **systemd oneshot unit** run ''docker compose up -d --wait'' — it respects health ordering.
+  * Add ''Restart=on-failure'' + ''RestartSec'' so cold-boot transients self-heal.
+  * Result: the stack comes up in the correct order every time, and a single misbehaving service can no longer pin a core and roast your fans.