Med Java

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Reliable Docker Compose Startup After a Reboot ======

A common and confusing failure mode: a Docker Compose stack that runs perfectly when you start it by hand starts misbehaving **after the server reboots**. One container pins a CPU core, the machine runs hot and the fans spin up — even though the overall system load is near zero. This page explains **why it happens** and shows a clean, general way to prevent it.

===== The trap: ''depends_on'' is not honored at reboot =====

Compose lets you declare start-up order and readiness gating:

<code yaml>
services:
  aggregator:
    depends_on:
      backend:
        condition: service_healthy
</code>

This tells Compose: //"do not start ''aggregator'' until ''backend'' reports healthy."// It works exactly as expected — **but only when you run ''docker compose up''.**

The catch: after a **host reboot**, your containers are usually brought back not by Compose, but by the **Docker daemon's restart policy** (''restart: always'' or ''restart: unless-stopped''). The daemon restarts containers in an **arbitrary order and ignores ''depends_on'' entirely.**

> **Key fact:** ''depends_on: condition: service_healthy'' is respected by ''docker compose up'' — **never** by the Docker daemon when it restarts containers at boot.

===== Why this causes a runaway (hot fans, pinned core) =====

Many "aggregator", "proxy" or "gateway" style services discover their backends **once at start-up** and never retry. If such a service starts **before** its backend is ready — exactly what the reboot race produces — it can end up:

  * holding a dead or half-open connection,
  * retrying in a tight loop with no back-off,
  * pinning a full CPU core at ~100%.

The result: the box is essentially idle (load near zero) yet **one core runs flat out**, temperatures climb and the fans ramp up — with no obvious "busy" process at a casual glance.

**Analogy:** it is like a courier who memorises the delivery list once, at the door, //before// the addresses are posted — then drives in circles forever because the list came up blank.

===== The clean fix: let systemd own start-up =====

The reliable pattern is to stop relying on the daemon's restart policy for boot ordering, and hand that job to **systemd** — which //does// respect Compose's health gating, because it runs ''docker compose up'' itself.

=== 1. Stop the daemon from race-starting containers ===

Change the restart policy in your ''docker-compose.yml'' from ''unless-stopped'' to ''on-failure'':

<code yaml>
services:
  backend:
    restart: on-failure
  aggregator:
    restart: on-failure
</code>

**Why:** the daemon auto-starts ''always'' and ''unless-stopped'' containers at boot (that is the race). It does **not** auto-start ''on-failure'' containers at boot — yet ''on-failure'' still restarts a container that crashes while running. So you keep crash recovery but remove the boot race.

=== 2. Make systemd start the whole stack, in order ===

Create ''/etc/systemd/system/mystack.service'':

<code ini>
[Unit]
Description=My application stack (docker compose)
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/mystack
ExecStart=/usr/bin/docker compose up -d --wait
ExecStop=/usr/bin/docker compose stop

[Install]
WantedBy=multi-user.target
</code>

Enable it once:

<code bash>
sudo systemctl daemon-reload
sudo systemctl enable --now mystack.service
</code>

Now systemd is the **single** thing that starts the stack at boot, and ''--wait'' blocks until every service is healthy — so the aggregator always comes up **after** its backends.

=== 3. Handle cold-boot transients with a retry ===

On a cold boot, a backend that needs the network (DNS, a remote API) may crash-loop for a few seconds until networking is fully up. ''docker compose up --wait'' is **intolerant** of this: the moment any container reports //unhealthy//, it aborts — it will not wait for the container to recover on its own.

Make the unit retry, so the stack converges by itself:

<code ini>
[Unit]
# ...
StartLimitIntervalSec=900
StartLimitBurst=6

[Service]
# ...
Restart=on-failure
RestartSec=20
</code>

If the first attempt fails because a backend was still settling, systemd waits and re-runs ''compose up --wait''. By the retry the backend is healthy, the aggregator starts, and the unit goes ''active'' — with **zero manual intervention**. Seeing a handful of restarts in the boot log is normal, not a fault:

<code bash>
systemctl show mystack.service -p NRestarts --value
</code>

===== How to diagnose the runaway =====

If a box runs hot after a reboot but overall load is low, look for a single pinned container:

<code bash>
# per-container CPU — the culprit sits near a full core
docker stats --no-stream

# confirm it is a spin, not real work, then read its logs
# for a tight retry / error loop (e.g. the same request repeating)
docker logs --tail 50 <container>
</code>

A quick manual recovery (until the boot fix is in place) is simply to restart the affected container //once its dependencies are up//:

<code bash>
docker restart <container>
</code>

===== Optional: a runtime watchdog =====

The steps above fix **boot** ordering. A separate, rarer problem is a long-running proxy that holds a **persistent session** to a backend and has no working auto-reconnect: if the backend restarts //during normal operation//, the session dies and the proxy may spin again. If your software has this limitation, a small periodic health-check script (driven by a systemd timer or cron) that restarts the proxy when it detects the fault is a reasonable backstop. Guard it with a **minimum interval** between restarts so it can never storm.

===== Summary =====

  * ''depends_on: service_healthy'' orders **''compose up''**, not the daemon at reboot.
  * Use ''restart: on-failure'' so the daemon does not race-start containers at boot.
  * Let a **systemd oneshot unit** run ''docker compose up -d --wait'' — it respects health ordering.
  * Add ''Restart=on-failure'' + ''RestartSec'' so cold-boot transients self-heal.
  * Result: the stack comes up in the correct order every time, and a single misbehaving service can no longer pin a core and roast your fans.