Debug Hetzner Node via Rescue Console

Overview

When a Hetzner Cloud server is unreachable (SSH hangs, provisioning stuck, cloud-init failure), this skill uses Hetzner's rescue mode to mount the node's filesystem and inspect everything from the outside — no working SSH required.

Usage

/debug-node

When invoked, ask for:

1. The server name or IP (can be found from hcloud server list)
2. What symptom they're seeing (SSH timeout, provisioning hang, etc.)

Prerequisites

• hcloud CLI installed and configured with a valid token
• The server must exist in Hetzner Cloud

hcloud server list

Leap Micro Filesystem Model

Leap Micro uses a transactional-update system on btrfs. This is the mental model for everything below.

Rule: Any /etc change that must survive MUST go through transactional-update --continue shell.

Packer build phases:

1. Rescue mode: Write qcow2 to disk, reboot
2. install_packages: Inside transactional-update — changes persist
3. clean_up: Volatile overlay — /etc changes are lost in the Hetzner snapshot

Step 1: Identify the Server

hcloud server list -o columns=id,name,status,ipv4 | grep <pattern>

Step 2: Enable Rescue Mode & Reboot

hcloud server enable-rescue <SERVER_ID> --type linux64
hcloud server reboot <SERVER_ID>
sleep 30

Save the rescue root password from the output (usually key auth works, but just in case).

Step 3: SSH into Rescue

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@<SERVER_IP>

Step 4: Mount the Filesystem

Leap Micro / MicroOS (btrfs)

# Mount btrfs top-level
mount -o subvolid=5 /dev/sda3 /mnt

# List snapshots — highest number is active
ls /mnt/@/.snapshots/

Layout:

/mnt/@/.snapshots/N/snapshot/    latest active snapshot — /etc lives here
/mnt/@/root/                     /root home (/root/.ssh/authorized_keys)
/mnt/@/var/                      /var (logs, cloud-init state, journal)

Key: /etc is inside the snapshot. /var and /root are separate subvolumes at @/var and @/root.

Unsupported/custom ext4 images

mount /dev/sda1 /mnt

Step 5: Diagnostic Checklist

Set this once and use throughout:

SNAP=/mnt/@/.snapshots/N/snapshot   # replace N with highest snapshot number

5a. Cloud-Init Status

Start here — most provisioning failures trace back to cloud-init.

cat /mnt/@/var/lib/cloud/data/result.json
cat /mnt/@/var/lib/cloud/data/status.json
cat /mnt/@/var/lib/cloud/instance/datasource

# What Terraform actually sent
cat /mnt/@/var/lib/cloud/instance/user-data.txt
zcat /mnt/@/var/lib/cloud/instance/user-data.txt.i 2>/dev/null

# Logs
tail -100 /mnt/@/var/log/cloud-init.log
tail -100 /mnt/@/var/log/cloud-init-output.log

Expected: DataSourceHetzner, no errors. Watch for: Skipping modules — means cloud-init already ran for this instance-id.

Cloud-init facts on Hetzner + Leap Micro:

• Datasource: DataSourceHetzner (metadata API)
• Terraform's cloudinit_config → gzip+base64 multipart MIME → user_data
• disable_root: false prevents cloud-init from disabling root but does NOT unlock a locked account
• ssh_authorized_keys writes keys to /root/.ssh/authorized_keys

5b. SSH Keys

cat /mnt/@/root/.ssh/authorized_keys

Compare with your local pubkey. If missing, cloud-init failed to inject — check 5a logs.

5c. SSHD Configuration

Config loading order (first match wins):

1. /etc/ssh/sshd_config.d/40-kube-hetzner-authorized-keys-command.conf
2. /etc/ssh/sshd_config.d/50-cloud-init.conf
3. /etc/ssh/sshd_config.d/kube-hetzner.conf           (MaxAuthTries 2)
4. /usr/etc/ssh/sshd_config.d/40-suse-crypto-policies.conf
5. /usr/etc/ssh/sshd_config                            (UsePAM yes)

ls $SNAP/etc/ssh/sshd_config.d/
cat $SNAP/etc/ssh/sshd_config.d/*.conf
cat $SNAP/usr/etc/ssh/sshd_config
ls -la $SNAP/etc/ssh/ssh_host_*

5d. Account Status

grep '^root:' $SNAP/etc/shadow

This is fixed in packer (usermod -p '*' root inside transactional-update) with a cloud-init bootcmd safety net. If you see a locked account on a fresh node, the packer snapshot needs rebuilding.

5e. Journal Logs

journalctl -D /mnt/@/var/log/journal/ -u sshd --no-pager | tail -50
journalctl -D /mnt/@/var/log/journal/ -u k3s --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-server --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-agent --no-pager | tail -30

5f. Network

ls $SNAP/etc/NetworkManager/system-connections/
cat $SNAP/etc/NetworkManager/system-connections/*.nmconnection 2>/dev/null

5g. Kubernetes

cat $SNAP/etc/rancher/k3s/config.yaml 2>/dev/null
cat $SNAP/etc/rancher/rke2/config.yaml 2>/dev/null
cat /mnt/@/var/lib/rancher/k3s/server/token 2>/dev/null

5h. SELinux

cat $SNAP/etc/selinux/config
chroot $SNAP rpm -qa | grep -iE 'selinux|k3s|rke2'
grep -i 'avc:.*denied' /mnt/@/var/log/audit/audit.log | tail -50
journalctl -D /mnt/@/var/log/journal/ --no-pager | grep -i 'avc:.*denied' | tail -50

For workload denials, follow docs/selinux.md: collect the AVC lines, workload name/version, k3s/RKE2 distribution, OS image, and udica result before proposing upstream policy changes. Do not globally disable SELinux as the first answer; use per-pool selinux = false only as the last resort for a workload or nodepool that cannot run under policy.

Step 6: Apply a Fix

Edit files in the active snapshot ($SNAP), not in @/ base.

# Unlock root account (if locked)
sed -i 's/^root:!*/root:*/' $SNAP/etc/shadow

# Fix authorized_keys
mkdir -p /mnt/@/root/.ssh
echo "ssh-ed25519 AAAA..." > /mnt/@/root/.ssh/authorized_keys
chmod 700 /mnt/@/root/.ssh && chmod 600 /mnt/@/root/.ssh/authorized_keys

# Regenerate host keys
mount --bind /proc $SNAP/proc && mount --bind /sys $SNAP/sys && mount --bind /dev $SNAP/dev
chroot $SNAP ssh-keygen -A
umount $SNAP/proc $SNAP/sys $SNAP/dev

Note: Rescue-mode edits are immediate fixes. The proper long-term fix belongs in the packer template or cloud-init.

Step 7: Reboot to Normal

umount /mnt 2>/dev/null
exit

hcloud server disable-rescue <SERVER_ID>
hcloud server reboot <SERVER_ID>
sleep 60
ssh -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP> 'echo ok'

Common Diagnoses

Debugging SSH Manually

# Verbose with specific key (avoids agent key spray hitting MaxAuthTries 2)
ssh -vvv -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP>

# In -vvv output:
# "Offering public key: ..."              → key was offered
# "Server accepts key: ..."               → success path
# "Authentications that can continue: ..."  → key was REJECTED
# "Too many authentication failures"       → agent sent too many keys

Pro Tips

1. Mount with subvolid=5 — gets the real btrfs root, navigate to @/.snapshots/N/snapshot/
2. Highest snapshot = active — that's where /etc lives
3. /var is separate — logs and cloud-init are at /mnt/@/var/, not inside the snapshot
4. Journal without a running system — journalctl -D /path/to/journal/
5. Use -o IdentitiesOnly=yes — kube-hetzner sets MaxAuthTries 2
6. Volatile overlay trap — if rescue shows different content than the live system did, it was running on a volatile overlay that never got committed
7. After fixing packer, rebuild snapshots — verify build logs show changes inside the transactional-update output
8. Rescue mode is non-destructive — you're just reading/writing files on the disk
9. Destroy with the wrapper — for full-cluster teardown, run scripts/destroy.sh from the Terraform root; it retries only the known ingress-LB detach race and prints a read-only orphan report. Use scripts/cleanup.sh only as the forceful fallback.