Jon Ferrer

Kubernetes & cloud infrastructure — this site runs on a cluster I built and operate myself.

k3s OCI Always Free (Ampere A1 / ARM64) Traefik + Let's Encrypt Prometheus / Grafana

01Architecture

Two-node k3s cluster on two Oracle Cloud "Always Free" ARM instances (1 OCPU / 6GB each — the entire Ampere A1 free allotment, split across both nodes). Traefik (bundled with k3s) handles ingress and TLS termination directly via its ACME client — no cloud load balancer, no cert-manager, $0 beyond the free tier.

┌─────────────────────────────┐ ┌─────────────────────────────┐ │ k8s-control-plane │ │ k8s-worker-arm1 │ │ 1 OCPU / 6GB (ARM64) │◄──────►│ 1 OCPU / 6GB (ARM64) │ │ │ VXLAN │ │ │ ┌──────────────────────┐ │ (flannel, UDP 8472) │ │ │ Traefik (ingress) │ │ │ ┌─────────────────────┐ │ │ │ + Let's Encrypt ACME│ │ │ │ portfolio-site pod │ │ │ └──────────────────────┘ │ │ └─────────────────────┘ │ │ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │ │ │ Prometheus │ │ │ │ Grafana │ │ │ └─────────────────────┘ │ │ └─────────────────────┘ │ └─────────────────────────────┘ └─────────────────────────────┘ ▲ ▲ └──────────────── public internet ───────┘ (DNS → node IP, HTTPS via Traefik)

02Debug log

Most portfolio projects show a finished, working demo. This section is the opposite — the actual incidents hit while standing this cluster back up, in the order they happened, because diagnosing this kind of thing is the actual job.

TLS SAN mismatch on the k3s API server
Reinstalling k3s with --node-external-ip alone doesn't add the public IP to the server certificate. Fixed by adding --tls-san explicitly, and for a live cluster, by appending tls-san to /etc/rancher/k3s/config.yaml and restarting — no need to reinstall.
"certificate signed by unknown authority" after merging kubeconfigs
kubectl config view --flatten keeps the first-seen cluster/context/user when names collide across files listed in KUBECONFIG. A stale local entry from before the reinstall silently won over the new cluster CA.
Cross-node pod traffic: 502, then "No route to host"
firewalld's trusted zone allowed the node subnet but not the pod/service CIDRs. Host-to-host VXLAN traffic (source = node IP) passed fine, but once decapsulated, the inner pod-to-pod packet (source/dest in the pod CIDR) fell into the restrictive default zone and got dropped. Fixed by adding the pod and service CIDRs to the trusted zone on both nodes.
Ingress traffic hit a wall — a completely different nginx answered
A pre-existing system nginx, left running from an earlier project on the same box, already owned ports 80/443. Traefik's ServiceLB daemonset never got real traffic on that node until it was stopped.
Still broken after freeing the port
k3s's klipper-lb doesn't bind a socket — it inserts raw iptables DNAT/FORWARD rules at pod startup. A firewall-cmd --reload run minutes earlier (to fix the CIDR issue above) had rebuilt the ruleset and wiped those rules out from under it. Recreating the pods reinserted them.
Grafana stuck in CrashLoopBackOff
Exit code 137 — OOMKilled. A hand-tuned 192Mi memory limit (sized against an older Grafana release) was too tight for the current version to even finish booting. Raised the limit and re-applied via helm upgrade.
DNS pointed at the right IP, but nothing matched
DuckDNS's real service is at duckdns.org — duckdns.com is an unrelated domain that resolves any subdomain, real or invented, to its own generic parking IPs. Every check "worked" right up until the TLS certificate came back for the wrong domain.

03Stack

k3sorchestration
Traefikingress + TLS
Let's EncryptACME certs
Prometheusmetrics
Grafanadashboards
Helmpackaging
OCI Ampere A1ARM64, free tier
flannelpod networking

04Live cluster metrics

Grafana runs in-cluster (kept off the public internet deliberately — access is via kubectl port-forward only), fed by Prometheus scraping node and cluster metrics every 30s.

TODO: drop a real Grafana dashboard screenshot (site/grafana-dashboard.png) once there's a few days of real data to show.