container-from-scratch is a minimal container runtime, one Go file, stdlib plus golang.org/x/sys/unix and nothing else — no container libraries. The goal wasn't to compete with Docker, it was to stop treating it as a black box: implement, by hand, the actual kernel primitives it's built on. PID namespaces, mount namespaces with a private /proc, UTS namespaces, pivot_root, cgroups v2, network namespaces, OverlayFS, and a real PID 1 that reaps its zombies. Each stage adds exactly one mechanism on top of the last, and each has a "done" condition that isn't "it compiles" but "here is the terminal output proving this isolation is real." Source is on GitHub.
Deliberately out of scope: image registries, a long-running daemon, real bridge networking with iptables NAT (documented as a manual exercise, not automated), and user namespaces — flagged below as the natural next project rather than bolted on at the end of an already long session.
The trick every stage depends on: re-executing yourself via /proc/self/exe
Before any of the stages, one problem has to be solved or nothing else works. PID namespaces can only be entered by a process at creation time, via clone(2) — unlike most other namespace types, calling unshare(CLONE_NEWPID) does not move the calling process into a new namespace, it only affects children forked afterward. And Go specifically can't just call a raw fork() to get that child: the runtime is heavily multithreaded (goroutine scheduler, GC workers, the sysmon thread), and a bare fork() only duplicates the calling OS thread — every other thread, and any lock state it held, vanishes in the child, leaving a structurally broken runtime until an exec() replaces the whole process image. Go deliberately doesn't expose a safe raw fork.
The fix, used here and in real runtimes like runc: re-exec the same binary via /proc/self/exe — a magic symlink the kernel always points at the current process's own executable — passing the target clone flags through exec.Cmd's SysProcAttr.Cloneflags. The new process is born directly inside the new namespace as its PID 1, via a full execve(), no fork, no broken runtime. Since it's the same compiled binary running twice, a sentinel argv value tells the two branches apart:
const reexecMarker = "__namespace_init__"
func main() {
if len(os.Args) > 1 && os.Args[1] == reexecMarker {
runInNamespace(os.Args[2:])
return
}
spawn(os.Args[1:])
}Stage 1 — PID namespace
A process's private view of PID space: the first process born inside a new PID namespace becomes PID 1 from that namespace's point of view, while still having a real, different PID on the host — dual identity, same task. Namespaces nest asymmetrically on purpose: a child can't see its parent's namespace, but the parent can always see into it, so the host can still supervise every container. One flag does it:
cmd.SysProcAttr = &syscall.SysProcAttr{
Cloneflags: unix.CLONE_NEWPID,
}Done: echo $$ inside reports 1. ps aux on the host, in a second terminal, still shows the process under its real, large PID. But ls /proc inside the namespace at this point still misleadingly lists the whole host process tree — not a bug. Procfs binds itself to a specific PID namespace at mount time, stored in the superblock, not re-evaluated per reader. The /proc mount here is the one inherited from the host, still bound to the host's namespace. Fixing that is the next stage.
Stage 2 — mount namespace and a private /proc
CLONE_NEWNS gives a private mount table, but there's a subtlety that trips up almost everyone on the first attempt: mount points have a propagation type, not just namespace membership. Default propagation on most distros, Ubuntu included, is MS_SHARED for root and everything under it — a shared mount belongs to a peer group, and cloning off a shared mount keeps that membership, so mount/unmount events still propagate bidirectionally between the "private" namespace and the host even though the table is technically separate. CLONE_NEWNS alone does not stop this; you have to explicitly, recursively remount root as MS_PRIVATE to sever peer-group membership. It's the first thing every real container runtime does after creating the mount namespace.
Then mount a fresh procfs at /proc — a new procfs mount always captures the PID namespace of whoever is mounting it, right now, fixing the Stage 1 cliffhanger:
func setupMounts() error {
if err := unix.Mount("none", "/", "", unix.MS_REC|unix.MS_PRIVATE, ""); err != nil {
return fmt.Errorf("making / private: %w", err)
}
return unix.Mount("proc", "/proc", "proc", 0, "")
}Done, with real output: ps aux inside shows only the shell and ps itself — PIDs 1 and 7. From the host, mount | grep /proc shows exactly one entry, nothing leaked from the container. Both directions of isolation proven. One small confusion worth naming: running cat /proc/mounts | head -3 inside shows the old host mounts first (ext4 root, udev, devpts), which looks like the private /proc didn't take. It did — a new mount namespace starts as a copy of the parent's table, not empty, so those entries are legitimately part of the private table too, just listed before the fresh proc mount, which was added last. head -3 just cuts off before reaching it.
Stage 3 — UTS namespace
The simplest namespace in the set, on purpose. "UTS" is Unix Timesharing System, named after the struct utsname from uname(2) that hostname happens to live in. It isolates two strings — hostname and NIS domain name — but that matters because so much tooling keys off hostname (shell prompts, logs, service discovery, TLS SANs, split-brain detection) that shared hostnames across containers would cause silent collisions. No propagation semantics, no filesystem interaction, no ordering constraints with anything else: one clone flag, one syscall.
// added to the same Cloneflags bitmask:
syscall.CLONE_NEWUTS
// then, before exec, inside the new namespace:
unix.Sethostname([]byte("container"))Done: hostname inside prints container; the host's own hostname is untouched.
Stage 4 — chroot vs. pivot_root
This is the stage with the best "here's something people get wrong" content in the whole project. chroot(2), from 1979, only changes where a process's / appears to point for path resolution — it never touches the mount table. The old root stays mounted exactly where it was, just unreachable via chroot-relative paths. It's famously not a security boundary: root can escape it — open a file descriptor outside the jail before chrooting, or the classic double-chroot()-plus-fchdir() trick, or mknod a device node to reach raw disk.
pivot_root(2) is what runc and Docker actually use. It's fundamentally a mount operation: it swaps which mount is the root of the mount tree for the current mount namespace, and relocates the old root to a path you specify so you can genuinely unmount it afterward. This only makes sense on top of Stage 2's private mount namespace — pivoting the host's real root would be catastrophic, and the kernel disallows it in most cases anyway. The mechanism is a five-step dance, each step satisfying a real kernel constraint:
func pivotRoot(newRoot string) error {
// pivot_root requires newRoot to already BE a mount point —
// bind-mounting it onto itself satisfies that without copying data
if err := unix.Mount(newRoot, newRoot, "", unix.MS_BIND|unix.MS_REC, ""); err != nil {
return fmt.Errorf("bind-mounting new root: %w", err)
}
oldRootDir := filepath.Join(newRoot, ".pivot_root_old")
if err := os.MkdirAll(oldRootDir, 0700); err != nil {
return fmt.Errorf("creating old root holder: %w", err)
}
if err := unix.PivotRoot(newRoot, oldRootDir); err != nil {
return fmt.Errorf("pivot_root: %w", err)
}
if err := unix.Chdir("/"); err != nil {
return fmt.Errorf("chdir to new root: %w", err)
}
// lazy unmount: detaches immediately even with an open fd still
// referencing it, cleaned up once nothing holds it — a plain
// unmount can fail EBUSY here
if err := unix.Unmount("/.pivot_root_old", unix.MNT_DETACH); err != nil {
return fmt.Errorf("unmounting old root: %w", err)
}
return os.RemoveAll("/.pivot_root_old")
}This stage needs an actual root filesystem to pivot into — a one-time shell setup step, kept deliberately out of the Go program to respect the "no image pulling" scope boundary: install busybox-static, copy the binary in, symlink sh/ls/ps/etc. to it. Skip that step and the failure is exactly as informative as it should be — pivot root failed: bind-mounting new root: no such file or directory, because the rootfs directory genuinely doesn't exist yet. Done: ls / inside shows only bin and proc; anything host-only fails with "no such file or directory," not a permission error — it genuinely isn't in this mount tree anymore.
Stage 5 — cgroups v2: memory, CPU, and the OOM debugging saga
Everything so far controls what a container can see. None of it controls what it can consume — a runaway loop or a leak inside the namespace is exactly as dangerous to the host as running it directly. Namespaces are a visibility mechanism; cgroups are the resource-accounting mechanism, and together they're what actually makes this a container rather than a process with a funny /proc. cgroups v2 isn't a syscall API at all — it's a virtual filesystem at /sys/fs/cgroup. Making a directory there creates a cgroup, auto-populated with control files (cgroup.procs, memory.max, cpu.max); writing a PID into cgroup.procs joins it, writing numbers into the limit files sets real kernel-enforced ceilings. No new clone flag — pure file I/O, done from the launcher process using the child's real host PID (the namespace's own view of itself, PID 1, would be useless as a cgroup key — the same host-PID-vs-namespace-PID duality from Stage 1's ps aux confusion, again):
const (
cgroupPath = "/sys/fs/cgroup/container-from-scratch"
memoryLimitBytes = "20971520" // 20 MB
cpuMax = "10000 100000" // 10ms per 100ms period = 10% of one core
)
func setupCgroup(pid int) (cleanup func(), err error) {
os.MkdirAll(cgroupPath, 0755)
cleanup = func() { os.Remove(cgroupPath) }
os.WriteFile(filepath.Join(cgroupPath, "memory.max"), []byte(memoryLimitBytes), 0644)
os.WriteFile(filepath.Join(cgroupPath, "memory.swap.max"), []byte("0"), 0644)
os.WriteFile(filepath.Join(cgroupPath, "cpu.max"), []byte(cpuMax), 0644)
os.WriteFile(filepath.Join(cgroupPath, "cgroup.procs"), []byte(strconv.Itoa(pid)), 0644)
return cleanup, nil
}Verifying this one turned into the richest debugging chain in the whole project. First memory-stress attempt, head -c 50000000 /dev/zero | tr '\0' x, failed immediately — the minimal busybox rootfs never had /dev set up, no devtmpfs, no device nodes. Switched to busybox's own yes builtin instead. Second attempt ran the stress command without capturing its output, so x characters just flooded the terminal instead of building up in memory — memory.current sat near 5MB and memory.events showed oom 0, proof nothing was actually accumulating. Switched to a pipe-free, substitution-free shell doubling loop instead, which can't fail on quoting:
x="a"; while true; do x="$x$x"; done # doubles every iteration — past 20MB in ~25 loopsFirst run of that also didn't die. memory.current sat right at the limit indefinitely — no OOM kill. The real cause is a genuinely well-known and poorly-documented cgroups v2 gotcha: memory.max only caps memory, not swap, separately. With swap enabled on the host, once the cgroup hits its memory ceiling the kernel pushes pages into swap instead of invoking the OOM killer — the process doesn't die, it thrashes. The fix is the memory.swap.max = 0 write above, forcing the kernel to OOM-kill instead of swap-stall since there's nowhere left to overflow into. After that fix, the same test triggered a real kill — verified with the correct diagnostic tool, not dmesg | tail (full of unrelated AppArmor noise from other running apps, drowning the real signal) but the cgroup's own accounting:
cat /sys/fs/cgroup/container-from-scratch/memory.events # oom_kill count went nonzeroCPU throttling needed no debugging: a while true; do :; done busy-loop inside, checked from the host with ps aux | grep sh, showed 8.2% CPU — capped near the configured 10% quota instead of pinning a full core.
Stage 6 — network namespace
CLONE_NEWNET gives a process its own complete network stack — interfaces, routing table, ARP table, port space — so containers can't collide over ports the way two unrelated host programs never would either. It's the entire code change:
Cloneflags: syscall.CLONE_NEWPID | syscall.CLONE_NEWNS | syscall.CLONE_NEWUTS | syscall.CLONE_NEWNETNothing else needed — everything observable is a direct consequence of the flag. Worth a paragraph: a fresh netns isn't empty, the kernel auto-creates a loopback interface, but leaves it administratively down. Even curl localhost inside a brand-new netns fails until something explicitly brings lo up; Docker's entrypoint does this silently, here it's done by hand and directly observable. Connecting the namespace to the outside world — a veth pair, a bridge, iptables NAT for outbound masquerading — is a large amount of plumbing that dockerd/CNI automate, and was deliberately left as a documented, manual shell exercise rather than built into the Go program:
sudo ip link add veth0 type veth peer name veth1
sudo ip link set veth1 netns $PID
sudo ip addr add 10.200.1.1/24 dev veth0
sudo ip link set veth0 up
sudo nsenter -t $PID -n ip addr add 10.200.1.2/24 dev veth1
sudo nsenter -t $PID -n ip link set veth1 up
sudo nsenter -t $PID -n ip link set lo upDone: ip addr inside shows only lo, down; the host's real interfaces are untouched; ping -c1 127.0.0.1 inside fails until lo is brought up by hand, then succeeds.
Stage 7 — OverlayFS, and three unrelated bugs wearing the same disguise
This is the payoff stage for "how do image layers actually work." Every prior stage pivoted into one flat rootfs directory. Real images are built from stacked layers — a base, an install layer, an app layer — and two containers from the same image share the base layer's blocks on disk instead of each getting a full copy; a container's writes never touch the shared base at all. That's why docker run from an existing image is near-instant instead of a filesystem copy, and why layer caching works. OverlayFS combines directories into one view via four roles: lowerdir (read-only layers, never written), upperdir (the single writable layer every write lands in), workdir (internal scratch space for atomic renames, must be empty and on the same filesystem as upperdir), and merged (the actual mountpoint — this is what gets pivot_root'd into, indistinguishable from an ordinary filesystem from inside). A file deleted from inside the container that exists in the lower layer isn't actually deleted — it's recorded in upperdir as a whiteout, a character device with major/minor 0/0.
opts := fmt.Sprintf("lowerdir=%s,upperdir=%s,workdir=%s", absLower, absUpper, absWork)
unix.Mount("overlay", merged, "overlay", 0, opts)This is the debugging chain worth reading in full, because it's rarely one clean root cause and this wasn't either — three distinct, mostly unrelated bugs, all surfacing as the same symptom, mount failures, each needing a different diagnostic to isolate:
Bug one. First run: mounting overlay: invalid argument, no clue why from the Go error alone. Reproducing the same mount manually from a host shell (more verbose errors) hit a different failure, mount: bad usage — a shell issue, not overlayfs: the project directory's path contains a literal space, and the manual reproduction used an unquoted $(pwd) inside the -o value, so word-splitting corrupted the argument into garbage tokens. Quoting it (-o "lowerdir=$(pwd)/...,upperdir=...,workdir=...") fixed the manual command and proved the overlay mechanism itself was fine — syscalls don't care about spaces the way shells do, so this wasn't the real blocker for the Go program.
Bug two, the real one behind the original error. Going back to dmesg from an earlier failed attempt turned up the actual smoking gun: overlay: Unknown parameter ' upperdir' — note the leading space. Traced by reading the Go source directly to a one-character typo introduced while hand-typing the code: fmt.Sprintf("lowerdir=%s, upperdir=%s,workdir=%s", ...), a stray space after the first comma. Overlayfs's option parser splits strictly on commas with zero tolerance for surrounding whitespace, so ", upperdir=..." became a literally unparseable " upperdir" token — the kernel error was telling the exact truth the whole time. Removing the space fixed it.
Bug three, completely unrelated, surfacing right after. mount failed: no such file or directory — this time from the /proc mount, not overlay, which now succeeded. The layers/lower setup script had never actually been run: the directory was empty, auto-created by os.MkdirAll inside the sudo-run Go program and so owned by root. No /bin, no /proc mountpoint existed in the base layer, so after pivoting into an empty merged view there was nowhere to mount /proc onto. Fixing it directly wasn't possible without a sudo password to write into a root-owned directory — the fix had to be handed back as a shell command instead.
Done, once all three were fixed: writing a file inside the container appears in layers/upper on the host, layers/lower stays untouched; deleting a file that exists in the lower layer makes it disappear from inside the container while the original remains on the host, now visible in layers/upper as a whiteout.
Two more additions, chosen for what they catch
pids.max — fork-bomb protection, the same filesystem-write pattern as the other cgroup limits, capping total process count at 64. A shell loop spawning background processes continuously hits the cap and stops, instead of exhausting the host.
PID 1 as a real init, reaping zombies — the more structurally interesting one, and a genuine gotcha that's literally why docker run --init and tini exist. Up to this point, PID 1 handed off to the target command via unix.Exec — an execve() that replaces the process image entirely, meaning PID 1 became the shell directly, with nothing left running to ever call wait() on orphaned descendants. Any process reparented onto PID 1 (a background job whose direct parent exited) would finish and become a permanent zombie, since nobody was waiting on it. The fix keeps PID 1 as a small Go supervisor: start the target as a child instead of exec-replacing into it, then loop wait4(-1, ...) forever, reaping any child — the main one or any reparented orphan — until there are none left:
func runAsInit(args []string) {
child := exec.Command(args[0], args[1:]...)
child.Stdin, child.Stdout, child.Stderr = os.Stdin, os.Stdout, os.Stderr
child.Start()
mainPid := child.Process.Pid
exitCode := 0
for {
var ws unix.WaitStatus
pid, err := unix.Wait4(-1, &ws, 0, nil)
if err == unix.EINTR {
continue
}
if err != nil {
break // ECHILD: nothing left to reap
}
if pid == mainPid {
exitCode = ws.ExitStatus()
}
}
os.Exit(exitCode)
}This waits for every descendant to exit before returning, rather than exiting the instant the main process does and letting namespace teardown SIGKILL the rest, which is what tini actually does. Deliberately the simpler, slightly less production-faithful choice — good enough for the concept.
What's left out, on purpose
Image registries and pulling images, a long-running daemon or REST API, and real bridge networking with iptables NAT are all explained conceptually here but never implemented — each one is a lot of plumbing whose value, for this project, was understanding that it exists and roughly how, not re-deriving dockerd. User namespaces (rootless containers) are the one deliberately deferred for a different reason: they're the single most security-relevant piece not covered, and they interact with everything above — mounts, cgroup permissions — in ways that deserve a slower, dedicated pass rather than being bolted onto the end of an already long session. Natural part two.
What actually mattered
None of the individual mechanisms here are secret — they're documented in man pages and in runc's own source. What a from-scratch implementation buys is the debugging: watching ls /proc half-work in Stage 1 and understanding exactly why via superfs superblock binding; hitting a memory limit that silently does nothing until you learn cgroups v2 caps memory and swap separately; chasing an overlay mount failure through a shell-quoting artifact, a one-character typo, and a skipped setup step before finding the real cause each time. That's the actual shape of systems debugging — rarely one clean root cause, always resolved by reading the error message literally and reaching for the more specific diagnostic (memory.events over dmesg, a manual mount reproduction over trusting the Go error) instead of the generic one.