Hetzner Cloud

Facts that drift (prices, SKUs, locations, limits) go stale — prefer retrieval over pre-training: hcloud server-type list -o json, hcloud location list, docs.hetzner.com. Everything below is behavior, not price sheets. If the project documents a topology (e.g. docs/architecture/), read it before provisioning anything.

Provisioning Workflow

1. Pick type + location together — CAX (Ampere arm64, best RAM/€) exists only in EU DCs (fsn1/nbg1/hel1); verify with hcloud server-type list. arm64 requires multi-arch container images end-to-end.
   • Checkpoint: hcloud server-type describe <type> shows the target location.
2. Create network + firewall before servers — attach at create time; firewall via label selector (role=web) scales better than per-server IDs.
   • Checkpoint: firewall rules list SSH restricted to admin IPs, never 0.0.0.0/0.
3. Create server with cloud-init user-data — runs once at first boot only.
   • Recovery: broken cloud-init → don't patch by hand; fix the template, server rebuild (keeps IPs) or replace via IaC.
4. Attach volumes, then mount explicitly — see Volumes below.
   • Checkpoint: mount survives reboot (fstab entry, not just automount).
5. Enable protection on stateful resources — hcloud server enable-protection <s> delete rebuild + same for volumes.
   • Gate: protection verified via hcloud server describe before real data lands.

Non-Obvious Behaviors

Networking

• Cloud Firewalls filter ONLY the public interface. Private-network traffic is never touched by them — segment the private net with host firewalls (ufw) or not at all. This is the most common Hetzner security misconception.
• Private network MTU is 1450, not 1500 — override in Docker (daemon.json "mtu": 1450) and any overlay/VPN on top, or suffer silent large-packet stalls.
• Private networks are not encrypted — treat as shared fabric, keep TLS for sensitive cross-node traffic.
• Network zones (e.g. eu-central = fsn1/nbg1/hel1) span DCs — cross-DC private traffic works and is free.
• Floating IPs need manual OS config (netplan alias); primary IPs configure themselves. Floating IP ≠ instant failover without host-side automation.
• IPv4 primary IPs bill separately and survive server deletion unless auto_delete is set — orphaned IPs keep billing.

Servers & billing

• Hourly billing capped at the monthly price. Deleting a server stops its billing but volumes, snapshots, and primary IPs keep billing — sweep for orphans after teardowns.
• change-type (resize) requires ACPI shutdown first (hcloud server shutdown, not poweroff — that's a hard cut risking fs damage). CPU/RAM can downgrade later only with --keep-disk; disk never shrinks.
• server rebuild wipes the disk but keeps all IPs — the fast DR path when the root cause is software.
• 20 TB egress included per cloud server; ingress + private-net traffic free.
• Metadata endpoint: http://169.254.169.254/hetzner/v1/metadata (useful in cloud-init to self-discover private IP).

Volumes

• Location-bound — attach only within the same DC; a "move to another region" is snapshot-less (rsync) by design.
• Grow-only, online — after hcloud volume resize, run resize2fs/xfs_growfs yourself; nothing automatic.
• automount mounts under /mnt/<volume-name> via udev — fine for experiments, wrong for services. Use an explicit fstab entry (discard,nofail) at a stable path; nofail so a detached volume doesn't hang boot.

Backups vs snapshots

Backup→snapshot conversion exists — do it before deleting a server whose history matters. Neither replaces app-level data exports — Hetzner backups are crash-consistent block copies, not application backups.

IaC (Pulumi / Terraform hcloud provider)

• Changing userData replaces the server. Deliberate for cattle; catastrophic for stateful nodes. Keep volatile config out of cloud-init (runtime config management owns it) and treat any cloud-init diff as a planned node replacement — or ignoreChanges: ["userData"] on pet servers.
• serverType change is a replacement in IaC even though the console/CLI resizes in place. To resize without replacement: hcloud server change-type manually, then refresh state + update code.
• Pulumi: firewallIds wants numbers — firewall.id.apply(Number). Inline networks on the Server conflicts with a separate ServerNetwork resource — pick one.
• backups: true toggles in place (safe anytime). IaC-level protect only blocks the IaC tool — set hcloud-side delete/rebuild protection too; they guard different doors.
• Auth: HCLOUD_TOKEN env (CI/agents) — per-project tokens, read-write only where needed.

CLI Patterns

export HCLOUD_TOKEN=...                       # stateless auth for CI/agents; `hcloud context` for humans
hcloud server list -o columns=name,status,ipv4,type -o noheader
hcloud server ssh my-server                   # no IP lookup, uses your agent keys
hcloud server create-image my-server --type snapshot --description "pre-change $(date +%F)"
hcloud volume list -o json | jq -r '.[] | select(.server==null) | .name'   # orphaned volumes
hcloud primary-ip list -o json | jq -r '.[] | select(.assignee_id==null) | .ip'  # orphaned IPs
hcloud server shutdown my-server \
  && hcloud server change-type my-server --server-type cax31 --keep-disk \
  && hcloud server poweron my-server          # in-place upgrade, downgrade-capable

Anti-Patterns

References