Zero-OS Builder – Working Notes and Repository Map

Purpose
- This document captures operational knowledge of this repository: build flow, key files, flags, and recent behavior decisions (e.g., passwordless root).
- Links below point to exact functions and files for fast triage, using code navigation-friendly anchors.

Repository Overview
- Build entrypoint: [scripts/build.sh](scripts/build.sh)
  - Orchestrates incremental stages using stage markers.
  - Runs inside a container defined by [Dockerfile](Dockerfile) for reproducibility.
- Common utilities and config loading: [bash.common.sh](scripts/lib/common.sh:1)
  - Loads [config/build.conf](config/build.conf), normalizes directory paths, provides logging and safe execution wrappers.
- Initramfs assembly and finalization: [bash.initramfs_* functions](scripts/lib/initramfs.sh:1)
  - Copies components, sets up zinit configs, finalizes branding, creates CPIO archive, validates contents.
- Kernel integration (optional embedded initramfs): [bash.kernel_* functions](scripts/lib/kernel.sh:1)
  - Downloads/configures/builds kernel and modules, embeds initramfs, runs depmod.
- zinit configuration: [config/zinit/](config/zinit/)
  - YAML service definitions and init scripts used by zinit inside the initramfs rootfs.
- RFS tooling (modules/firmware flists): [scripts/rfs/](scripts/rfs/)
  - Packs module/firmware flists and embeds them into initramfs at /etc/rfs.

Container Tooling (dev-container)
- Base image: Alpine 3.22 in [Dockerfile](Dockerfile:1)
- Tools:
  - shadow (passwd/chpasswd): required for root password management in initramfs.
  - openssl, openssl-dev: kept for other build steps and potential hashing utilities.
  - build-base, rustup, kmod, upx, etc.: required by various build stages.
- Removed: perl, not required for password handling after switching to passwd/chpasswd workflow.

Configuration – build.conf
- File: [config/build.conf](config/build.conf)
- Key variables:
  - Versions: ALPINE_VERSION, KERNEL_VERSION
  - Directories (relative in config, normalized to absolute during runtime):
    - INSTALL_DIR="initramfs"
    - COMPONENTS_DIR="components"
    - KERNEL_DIR="kernel"
    - DIST_DIR="dist"
  - Flags:
    - ZEROOS_BRANDING="true"
    - ZEROOS_REBRANDING="true"
  - Branding behavior:
    - ZEROOS_PASSWORDLESS_ROOT="true" (default for branded builds in current policy)
    - ZEROOS_ROOT_PASSWORD_HASH / ROOT_PASSWORD_HASH (not used in current policy)
    - ZEROOS_ROOT_PASSWORD / ROOT_PASSWORD (not used in current policy)
  - FIRMWARE_TAG optional for reproducible firmware flist naming.

Absolute Path Normalization
- Location: [bash.common.sh](scripts/lib/common.sh:236)
- After sourcing build.conf, the following variables are normalized to absolute paths anchored at PROJECT_ROOT:
  - INSTALL_DIR, COMPONENTS_DIR, KERNEL_DIR, DIST_DIR
- Rationale: Prevents path resolution errors when CWD changes (e.g., when kernel build operates in /workspace/kernel/current, validation now resolves to /workspace/initramfs instead of /workspace/kernel/current/initramfs).

Build Pipeline – High Level
- Orchestrator: [bash.main_build_process()](scripts/build.sh:214)
  - Stage list:
    - alpine_extract
    - alpine_configure
    - alpine_packages
    - alpine_firmware
    - components_build
    - components_verify
    - kernel_modules
    - init_script
    - components_copy
    - zinit_setup
    - modules_setup
    - modules_copy
    - cleanup
    - rfs_flists
    - validation
    - initramfs_create
    - initramfs_test
    - kernel_build
    - boot_tests
  - Each stage wrapped with [bash.stage_run()](scripts/lib/stages.sh:99) and tracked under .build-stages/
- Container use:
  - Always run in container for stable toolchain (podman/docker auto-detected).
  - Inside container, CWD normalized to PROJECT_ROOT.

Initramfs Assembly – Key Functions
- zinit setup: [bash.initramfs_setup_zinit()](scripts/lib/initramfs.sh:12)
  - Copies [config/zinit](config/zinit/) into /etc/zinit, fixes perms, removes OpenRC remnants.
- Install init script: [bash.initramfs_install_init_script()](scripts/lib/initramfs.sh:74)
  - Installs [config/init](config/init) as /init in initramfs root.
- Components copy: [bash.initramfs_copy_components()](scripts/lib/initramfs.sh:101)
  - Installs built components (zinit/rfs/mycelium/corex) into proper locations, strips/UPX where applicable.
- Modules setup: [bash.initramfs_setup_modules()](scripts/lib/initramfs.sh:229)
  - Reads [config/modules.conf](config/modules.conf), resolves deps via [bash.initramfs_resolve_module_dependencies()](scripts/lib/initramfs.sh:318), generates stage1 list (firmware hints in modules.conf are ignored; firmware.conf is authoritative).
- Create module scripts: [bash.initramfs_create_module_scripts()](scripts/lib/initramfs.sh:427)
  - Writes /etc/zinit/init/stage1-modules.sh and stage2-modules.sh for zinit to load modules.
- Binary size optimization: [bash.initramfs_strip_and_upx()](scripts/lib/initramfs.sh:491)
- Final customization: [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:575)
  - See “Branding behavior” below.
- Create archive: [bash.initramfs_create_cpio()](scripts/lib/initramfs.sh:688)
  - Calls finalize, runs sanity checks, and creates initramfs.cpio.xz.
- Validate: [bash.initramfs_validate()](scripts/lib/initramfs.sh:799)
  - Ensures essential items exist, logs debug info:
    - Prints “Validation debug:” lines showing input, PWD, PROJECT_ROOT, INSTALL_DIR, and resolved path.

Kernel Integration
- Get full kernel version: [bash.kernel_get_full_version()](scripts/lib/kernel.sh:13)
- Apply config (embed initramfs): [bash.kernel_apply_config()](scripts/lib/kernel.sh:81)
  - Updates CONFIG_INITRAMFS_SOURCE to the archive’s absolute path via [bash.kernel_modify_config_for_initramfs()](scripts/lib/kernel.sh:130).
- Build kernel: [bash.kernel_build_with_initramfs()](scripts/lib/kernel.sh:173)
- Build and install modules in container: [bash.kernel_build_modules()](scripts/lib/kernel.sh:228)
  - Installs modules to /lib/modules/$FULL_VERSION in container, runs depmod -a.

RFS Flists (modules/firmware)
- Packing scripts:
  - Modules: [bash.pack-modules.sh](scripts/rfs/pack-modules.sh:1)
  - Firmware: [bash.pack-firmware.sh](scripts/rfs/pack-firmware.sh:1)
- Firmware policy:
  - For initramfs: [config/firmware.conf](config/firmware.conf) is the single source of truth for preinstalled firmware; modules.conf hints are ignored.
  - For RFS: install all Alpine linux-firmware* packages into the build container and pack from /lib/firmware (full set for runtime).
- Integrated in stage_rfs_flists:
  - Embeds /etc/rfs/modules-<FULL_KERNEL_VERSION>.fl
  - Embeds /etc/rfs/firmware-latest.fl (or tagged by FIRMWARE_TAG)
  - See [bash.main_build_process() — stage_rfs_flists](scripts/build.sh:298)
- Runtime mount/readiness:
  - Firmware flist mounts over /lib/firmware (overmount hides any initramfs firmware).
  - Modules flist mounts at /lib/modules/$(uname -r).
  - Init scripts probe BASE_URL reachability (accepts FLISTS_BASE_URL or FLIST_BASE_URL) and wait for HTTP(S) before fetching:
    - Firmware: [sh.firmware.sh](config/zinit/init/firmware.sh:1)
    - Modules: [sh.modules.sh](config/zinit/init/modules.sh:1)

Branding Behavior (Passwordless Root, motd/issue)
- Finalization hook: [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:575)
- Behavior (current policy):
  - Passwordless root enforced using passwd for shadow-aware deletion:
    - [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:616):
      - passwd -d -R "${initramfs_dir}" root
      - Ensures /etc/shadow has root:: (empty password) inside initramfs root, not host.
  - Branding toggles: ZEROOS_BRANDING and ZEROOS_REBRANDING (branding guard printed in logs).
  - Branding also updates /etc/motd and /etc/issue to Zero-OS.

Console and getty
- Early keyboard and debug:
  - [config/init](config/init) preloads input/HID and USB HCD modules (i8042, atkbd, usbhid, hid, hid_generic, evdev, xhci/ehci/ohci/uhci) so console input works before zinit/rfs.
  - Kernel cmdline initdebug=true opens an early interactive shell; if /init-debug exists and is executable, it runs preferentially.
- Serial and console getty configs (zinit service YAML):
  - [config/zinit/getty-tty1.yaml](config/zinit/getty-tty1.yaml)
  - [config/zinit/getty-console.yaml](config/zinit/getty-console.yaml)
- Optional ash login loop (not enabled unless referenced):
  - [bash.ashloging.sh](config/zinit/init/ashloging.sh:1)

Validation Diagnostics and Triage
- Common error previously observed:
  - “Initramfs directory not found: initramfs (resolved: /workspace/kernel/current/initramfs)”
- Root cause:
  - INSTALL_DIR re-sourced in a different CWD and interpreted as relative.
- Fix:
  - Absolute path normalization of INSTALL_DIR/COMPONENTS_DIR/KERNEL_DIR/DIST_DIR after sourcing build.conf in [bash.common.sh](scripts/lib/common.sh:236).
  - Additional “Validation debug” prints added in [bash.initramfs_validate()](scripts/lib/initramfs.sh:799).
- Expected logs now:
  - “Validation debug: input='initramfs' PWD=/workspace PROJECT_ROOT=/workspace INSTALL_DIR=/workspace/initramfs”
  - Resolves correctly even if called from a different stage CWD.

How to Verify Passwordless Root
- After build, check archive:
  - mkdir -p dist/_inspect && cd dist/_inspect
  - xz -dc ../initramfs.cpio.xz | cpio -idmv
  - grep '^root:' ./etc/shadow
  - Expect root:: (empty field) indicating passwordless root.
- At runtime on console:
  - When prompted for root’s password, press Enter.

Stage System and Incremental Rebuilds
- Stage markers stored in .build-stages/ (one file per stage).
- Minimal rebuild helper (host or container):
  - [scripts/rebuild-after-zinit.sh](scripts/rebuild-after-zinit.sh) clears only: modules_setup, modules_copy, init_script, zinit_setup, validation, initramfs_create, initramfs_test (kernel_build only with --with-kernel; kernel_modules only with --refresh-container-mods).
  - Flags:
    - --with-kernel (also rebuild kernel; ensures cpio is recreated right before embedding)
    - --refresh-container-mods (rebuild container /lib/modules for fresh containers)
    - --verify-only (report changed files and stage status; no rebuild)
  - Shows stage status before/after marker removal; no --rebuild-from is passed by default (relies on markers only).
- Manual minimal rebuild:
  - Remove relevant .done files, e.g.: initramfs_create.done initramfs_test.done validation.done
  - Rerun: DEBUG=1 ./scripts/build.sh --skip-tests
- Show status:
  - ./scripts/build.sh --show-stages

Key Decisions (current)
- Firmware selection for initramfs comes exclusively from [config/firmware.conf](config/firmware.conf); firmware hints in modules.conf are ignored to avoid duplication/mismatch.
- Runtime firmware flist overmounts /lib/firmware after network readiness; init scripts wait for FLISTS_BASE_URL/FLIST_BASE_URL HTTP reachability before fetching.
- Early keyboard and debug shell added to [config/init](config/init) as described above.
- Branding enforces passwordless root via passwd -d -R inside initramfs finalization, avoiding direct edits of passwd/shadow files.
- Directory paths normalized to absolute after loading config to avoid CWD-sensitive behavior.
- Container image contains shadow suite to ensure passwd/chpasswd availability; perl removed.

File Pointers (quick jump)
- Orchestrator: [scripts/build.sh](scripts/build.sh)
- Common and config loading: [bash.common.sh](scripts/lib/common.sh:1)
- Finalization hook: [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:575)
- Validation entry: [bash.initramfs_validate()](scripts/lib/initramfs.sh:799)
- CPIO creation: [bash.initramfs_create_cpio()](scripts/lib/initramfs.sh:688)
- Kernel embed config: [bash.kernel_modify_config_for_initramfs()](scripts/lib/kernel.sh:130)
- RFS packers: [bash.pack-modules.sh](scripts/rfs/pack-modules.sh:1), [bash.pack-firmware.sh](scripts/rfs/pack-firmware.sh:1)
- USB creator: [scripts/make-grub-usb.sh](scripts/make-grub-usb.sh)

Change Log
- 2025-09-09:
  - Enforce passwordless root using passwd -d -R in finalization.
  - Normalize INSTALL_DIR/COMPONENTS_DIR/KERNEL_DIR/DIST_DIR to absolute paths post-config load.
  - Add validation diagnostics prints (input/PWD/PROJECT_ROOT/INSTALL_DIR/resolved).
  - Ensure shadow package in container for passwd/chpasswd; keep openssl and openssl-dev; remove perl earlier.