diff --git a/README.md b/README.md index 8132664..f299815 100644 --- a/README.md +++ b/README.md @@ -7,130 +7,152 @@ Modern Alpine Linux based approach to building Zero-OS initramfs with source com This system uses: - **Alpine Linux 3.22** as base system with latest packages - **Individual Rust workspaces** with git subtrees for Zero-OS components (zinit, mycelium, rfs) -- **Component-based compilation** - each component built in its own workspace +- **Component-based compilation** - each component built from source in its own workspace - **Alpine packages** for system tools (busybox, openssh, btrfs-progs, etc.) -- **Smart caching** with Docker layers and volumes -- **Same init flow** as original Zero-OS +- **Smart caching** with Docker layers and volumes for fast rebuilds +- **Single-stage initramfs** with minimal (~50MB) or full (~700MB) variants ## Directory Structure ``` alpine-initramfs/ ├── build/ # Build orchestration -│ ├── Dockerfile.alpine # Alpine build environment +│ ├── Dockerfile.alpine # Alpine build environment │ ├── Dockerfile.cached # Multi-stage cached build │ └── docker-compose.yml # Build orchestration with caching ├── components/ # Git subtrees (individual workspaces) -│ ├── zinit/ # Init system (Rust subtree with workspace) -│ ├── mycelium/ # Networking layer (Rust subtree with workspace) -│ └── rfs/ # Filesystem (Rust subtree with workspace) +│ ├── zinit/ # Init system (Rust workspace) +│ ├── mycelium/ # Networking layer (Rust workspace) +│ └── rfs/ # Filesystem (Rust workspace) ├── configs/ # Configuration files -│ ├── packages-alpine.txt # Alpine packages to install +│ ├── packages-alpine.txt # Full Alpine packages (~700MB) +│ ├── packages-minimal.txt # Minimal packages (~50MB) │ ├── kernel-version # Kernel version to build │ ├── kernel-config-generic # Kernel configuration │ ├── init # Init script (Alpine → zinit) │ └── zinit/ # Zinit service configurations ├── scripts/ # Build scripts -│ ├── build-initramfs.sh # Main build orchestrator -│ ├── compile-components.sh # Compile Rust workspace -│ ├── fetch-github.sh # Download CoreX binary +│ ├── build-smart.sh # Smart build with caching (main orchestrator) +│ ├── compile-components.sh # Compile Rust components from source │ ├── install-packages.sh # Install Alpine packages -│ ├── setup-initramfs.sh # Create initramfs structure +│ ├── setup-initramfs.sh # Create initramfs structure and archive │ ├── build-kernel.sh # Build kernel with embedded initramfs -│ ├── build-smart.sh # Smart build with caching │ └── cleanup.sh # Clean build artifacts -├── target/ # Rust workspace build artifacts ├── output/ # Final build outputs │ ├── vmlinuz.efi # Final bootable kernel -│ ├── initramfs.cpio.xz # Generated initramfs -│ └── version.txt # Build information -└── docs/ # Documentation - ├── BUILD.md # Build process guide - ├── OVERVIEW.md # Architecture overview - └── *.md # Additional documentation +│ ├── initramfs.cpio.xz # Generated initramfs archive +│ └── compiled binaries # Individual component binaries +└── cache/ # Build cache for smart rebuilds + ├── components-marker # Component build cache marker + └── kernel-marker # Kernel build cache marker ``` ## Build Flow -1. **Environment Setup**: Alpine Docker container with Rust and build tools -2. **Component Compilation**: Individual cargo builds for each Rust component -3. **Binary Downloads**: Download CoreX (Go binary) from GitHub releases -4. **Package Installation**: Install Alpine packages to initramfs root -5. **Initramfs Assembly**: Create filesystem structure with compiled binaries -6. **Kernel Build**: Compile kernel with embedded initramfs -7. **Output**: Generate bootable vmlinuz.efi +The build system uses intelligent caching to minimize rebuild time: + +1. **Smart Analysis**: Check what components need rebuilding based on file changes +2. **Component Compilation**: Build only changed Rust components (zinit, mycelium, rfs) +3. **Package Installation**: Install Alpine packages to initramfs root +4. **Initramfs Assembly**: Create filesystem structure with compiled binaries +5. **Kernel Build**: Compile kernel only if config/version changed +6. **Final Assembly**: Embed initramfs into kernel → bootable vmlinuz.efi ## Quick Start -### Full Build +### Build Everything ```bash -# Build everything +# Build with default settings (debug mode, full packages) ./build.sh -# Or with Docker Compose -cd build/ -docker-compose up +# Build minimal embedded initramfs (~50MB) +./build.sh --minimal + +# Build release mode +./build.sh --release + +# Clean build without cache +./build.sh --clean ``` -### Development Build +### Build Variants ```bash -cd build/ -# Interactive development shell -docker-compose run dev-shell +# Minimal embedded initramfs (~50MB) +./build.sh --minimal -# Build all components (automated) -../scripts/compile-components.sh +# Full featured initramfs (~700MB) +./build.sh + +# Development with clean rebuild +./build.sh --clean --dev +``` + +### Development Workflow +```bash +# Start interactive development shell +./build.sh --dev + +# Inside container: +/build/scripts/compile-components.sh # Build all components +/build/scripts/build-smart.sh # Smart build with caching # Build individual components manually: -cd ../components/zinit && cargo build --release --target x86_64-unknown-linux-musl -cd ../components/mycelium && cargo build --release --target x86_64-unknown-linux-musl -cd ../components/rfs && cargo build --release --target x86_64-unknown-linux-musl -``` - -### Smart Build (with caching) -```bash -cd build/ -docker-compose run builder # Uses build-smart.sh automatically +cd /build/components/zinit && cargo build --release --target x86_64-unknown-linux-musl +cd /build/components/mycelium/myceliumd && cargo build --release --target x86_64-unknown-linux-musl +cd /build/components/rfs && cargo build --release --target x86_64-unknown-linux-musl ``` ## Key Features +- ✅ **Smart caching** - Only rebuild components that changed - ✅ **Component workspaces** - Each component maintains its own workspace - ✅ **Git subtrees** - Source code included, no submodule complexity -- ✅ **Automated build** - `scripts/compile-components.sh` builds all components -- ✅ **Shared target directory** - Efficient build artifact sharing -- ✅ **Smart caching** - Docker layer and volume caching for fast rebuilds -- ✅ **Alpine packages** - System tools from Alpine repositories +- ✅ **Source compilation** - All Zero-OS components built from source +- ✅ **Dual size modes** - Minimal (~50MB) or full (~700MB) initramfs +- ✅ **Docker layer caching** - Fast rebuilds with persistent cache volumes +- ✅ **Alpine packages** - Latest system tools from Alpine repositories - ✅ **Development mode** - Full IDE support and integrated tooling -- ✅ **Same functionality** - 100% compatible with original Zero-OS + +## Build Options + +### Package Modes +- **Minimal Mode** (`--minimal`): ~50MB embedded initramfs with essential packages only +- **Full Mode** (default): ~700MB initramfs with complete package set + +### Build Modes +- **Debug** (default): Faster compilation with debug symbols +- **Release** (`--release`): Optimized binaries for production + +### Cache Control +- **Smart Build** (default): Use cache when possible +- **Clean Build** (`--clean`): Force complete rebuild ## Development ### Building Individual Components ```bash -cd build/ -docker-compose run dev-shell +# Start development shell +./build.sh --dev -# Inside container - build all components: -../scripts/compile-components.sh +# Build all components +/build/scripts/compile-components.sh -# Or build specific components manually: -cd ../components/zinit && cargo build --release --target x86_64-unknown-linux-musl -cd ../components/mycelium/myceliumd && cargo build --release --target x86_64-unknown-linux-musl -cd ../components/rfs && cargo build --release --target x86_64-unknown-linux-musl -``` +# Or build individual components: +cd /build/components/zinit +cargo build --release --target x86_64-unknown-linux-musl -### Cleanup -```bash -# Clean all build artifacts -./scripts/cleanup.sh +cd /build/components/mycelium/myceliumd +cargo build --release --target x86_64-unknown-linux-musl + +cd /build/components/rfs +cargo build --release --target x86_64-unknown-linux-musl ``` ### Cache Management Smart build uses Docker volumes for caching: -- `build-cache`: Build state markers -- `source-cache`: Downloaded sources -- `kernel-cache`: Kernel builds +- `build-cache`: Build state markers for smart rebuilds +- `source-cache`: Downloaded sources and dependencies +- `kernel-cache`: Compiled kernel builds - `target-cache`: Rust workspace build artifacts ### Updating Components @@ -139,9 +161,59 @@ Update git subtree components: # Update zinit to latest git subtree pull --prefix=components/zinit https://github.com/threefoldtech/zinit.git master --squash -# Update mycelium to latest +# Update mycelium to latest git subtree pull --prefix=components/mycelium https://github.com/threefoldtech/mycelium.git master --squash # Update rfs to latest git subtree pull --prefix=components/rfs https://github.com/threefoldtech/rfs.git master --squash -``` \ No newline at end of file +``` + +### Cleanup +```bash +# Clean all build artifacts and cache +./build.sh --clean + +# Or clean specific components +./scripts/cleanup.sh +``` + +## Testing + +Test the built kernel: +```bash +# Boot with QEMU +qemu-system-x86_64 \ + -kernel output/vmlinuz.efi \ + -m 2048 \ + -enable-kvm \ + -cpu host \ + -net nic,model=e1000 \ + -net bridge,br=vm0 \ + -nographic \ + -serial null \ + -serial mon:stdio \ + -append "console=ttyS1,115200n8" +``` + +## Architecture Notes + +### Single-Stage Boot +This system uses a single-stage initramfs approach where the entire Zero-OS environment is embedded in the kernel. This provides: +- Simpler deployment (single vmlinuz.efi file) +- Faster boot times (no network dependencies) +- Better reliability (no network mount failures) + +### Component Compilation +All Zero-OS components (zinit, mycelium, rfs) are compiled from source rather than downloaded as binaries. This ensures: +- Consistent build environment +- Version control over all components +- Easy development and debugging +- Security through reproducible builds + +### Smart Caching +The build system tracks file changes and only rebuilds what's necessary: +- Component sources unchanged → use cached binaries +- Kernel config unchanged → use cached kernel +- Only initramfs assembly runs every time (fast) + +This typically reduces rebuild time from 10+ minutes to under 2 minutes. \ No newline at end of file diff --git a/scripts/build-boot.sh b/scripts/build-boot.sh deleted file mode 100755 index 977cf8a..0000000 --- a/scripts/build-boot.sh +++ /dev/null @@ -1,179 +0,0 @@ -#!/bin/bash -set -e - -BOOT_ROOT="/build/boot" -OUTPUT_DIR="/build/output" -CONFIG_DIR="/build/configs" - -echo "[+] Building minimal boot initramfs..." -echo " Target: ~10-20MB embedded in kernel" -echo " Purpose: Boot + network + mount Alpine runtime" - -# Clean and create boot root -rm -rf "$BOOT_ROOT" -mkdir -p "$BOOT_ROOT" - -echo "[+] Installing minimal boot packages..." - -# Set up APK for boot root -mkdir -p "$BOOT_ROOT/etc/apk" -cp /etc/apk/repositories "$BOOT_ROOT/etc/apk/" - -# Copy APK keys -if [ -d /etc/apk/keys ]; then - mkdir -p "$BOOT_ROOT/etc/apk/keys" - cp /etc/apk/keys/* "$BOOT_ROOT/etc/apk/keys/" -fi - -# Create APK database -mkdir -p "$BOOT_ROOT/lib/apk/db" -mkdir -p "$BOOT_ROOT/var/cache/apk" - -# Install absolute minimal packages -echo " Installing boot packages from packages-boot.txt..." -apk --root "$BOOT_ROOT" --clean-protected --update-cache --initdb add alpine-baselayout busybox apk-tools musl - -# Install additional boot packages -while read -r package; do - # Skip comments and empty lines - case "$package" in - \#*|"") continue ;; - esac - - echo " Installing: $package" - if apk --root "$BOOT_ROOT" add --no-cache "$package"; then - echo " Success: $package" - else - echo " Warning: Failed to install $package" - fi -done < "$CONFIG_DIR/packages-boot.txt" - -echo "[+] Installing minimal ethernet drivers..." -# Install essential ethernet firmware and drivers using our minimal script -INITRAMFS_ROOT="$BOOT_ROOT" /build/scripts/install-firmware-minimal.sh - -echo "[+] Setting up boot filesystem structure..." - -# Create essential directories -mkdir -p "$BOOT_ROOT"/{dev,proc,sys,mnt,tmp,var/run,var/log} -mkdir -p "$BOOT_ROOT"/{root,home,opt} -mkdir -p "$BOOT_ROOT"/mnt/{alpine,debug} - -# Create device nodes -echo " Creating device nodes..." -[ ! -e "$BOOT_ROOT/dev/console" ] && mknod "$BOOT_ROOT/dev/console" c 5 1 -[ ! -e "$BOOT_ROOT/dev/null" ] && mknod "$BOOT_ROOT/dev/null" c 1 3 -[ ! -e "$BOOT_ROOT/dev/zero" ] && mknod "$BOOT_ROOT/dev/zero" c 1 5 - -# Set proper permissions -chmod 666 "$BOOT_ROOT/dev/null" "$BOOT_ROOT/dev/zero" -chmod 600 "$BOOT_ROOT/dev/console" - -echo "[+] Creating boot init script..." -# Create minimal init script for mounting Alpine runtime -cat > "$BOOT_ROOT/init" << 'EOF' -#!/bin/bash -# Minimal boot init - establish network and mount Alpine runtime - -echo "[BOOT] Zero-OS minimal boot starting..." - -# Mount essential filesystems -mount -t proc proc /proc -mount -t sysfs sysfs /sys -mount -t devtmpfs devtmpfs /dev - -# Load essential modules -echo "[BOOT] Loading ethernet drivers..." -modprobe -a e1000 e1000e igb ixgbe r8169 8139too bnx2 tg3 2>/dev/null || true - -# Start eudev for hardware detection -echo "[BOOT] Starting hardware detection..." -/sbin/udevd --daemon -udevadm trigger -udevadm settle - -# Network configuration -echo "[BOOT] Configuring network..." -for iface in $(ls /sys/class/net/ | grep -v lo); do - echo " Bringing up interface: $iface" - ip link set "$iface" up - # Try DHCP with timeout - timeout 30 dhcpcd "$iface" && break -done - -# Mount Alpine runtime system -echo "[BOOT] Mounting Alpine runtime system..." - -# Check for VirtioFS mount (development) -if [ -d /sys/fs/virtio_fs ]; then - echo " Attempting VirtioFS mount..." - mount -t virtiofs alpine-root /mnt/alpine && ALPINE_MOUNTED=true -fi - -# If VirtioFS failed, try network mount (production) -if [ "$ALPINE_MOUNTED" != "true" ]; then - echo " VirtioFS not available, attempting network mount..." - # Add network mounting logic here (NFS, HTTP, etc.) - echo " Network mount not implemented yet" -fi - -# Switch to Alpine runtime if mounted -if [ "$ALPINE_MOUNTED" = "true" ]; then - echo "[BOOT] Switching to Alpine runtime system..." - - # Move boot mounts to Alpine - mount --move /proc /mnt/alpine/proc - mount --move /sys /mnt/alpine/sys - mount --move /dev /mnt/alpine/dev - - # Switch root to Alpine - exec switch_root /mnt/alpine /sbin/zinit -else - echo "[BOOT] Failed to mount Alpine runtime - dropping to shell" - exec /bin/sh -fi -EOF - -chmod +x "$BOOT_ROOT/init" - -echo "[+] Setting up symlinks..." -cd "$BOOT_ROOT" - -# Standard symlinks -ln -sf usr/lib lib || true -ln -sf usr/lib lib64 || true - -# Fixed run symlink (not circular) -mkdir -p run -cd var -ln -sf ../run run || true -cd .. - -echo "[+] Cleaning up boot initramfs..." -# Remove unnecessary files -rm -rf var/cache/apk/* -rm -rf usr/share/doc usr/share/man usr/share/info 2>/dev/null || true - -# Aggressive cleanup for embedded -find . -name "*.a" -delete 2>/dev/null || true -find . -name "*.la" -delete 2>/dev/null || true - -echo "[+] Creating boot initramfs archive..." -cd "$BOOT_ROOT" - -# Create cpio archive -find . | cpio -H newc -o > "$OUTPUT_DIR/boot-initramfs.cpio" - -if [ -f "$OUTPUT_DIR/boot-initramfs.cpio" ]; then - size=$(stat -c%s "$OUTPUT_DIR/boot-initramfs.cpio") - size_mb=$((size / 1024 / 1024)) - echo "[+] Boot initramfs created: $OUTPUT_DIR/boot-initramfs.cpio (${size_mb}MB)" -else - echo "[-] Error: Failed to create boot initramfs" - exit 1 -fi - -echo "[+] Minimal boot initramfs build complete" -echo " Size: ${size_mb}MB (target: ~10-20MB)" -echo " Contains: Boot + network + minimal drivers" -echo " Purpose: Mount and switch to Alpine runtime" \ No newline at end of file diff --git a/scripts/cleanup.sh b/scripts/cleanup.sh index 678d910..f094b9d 100755 --- a/scripts/cleanup.sh +++ b/scripts/cleanup.sh @@ -176,5 +176,4 @@ echo "[+] Cleanup completed successfully!" echo "" echo "To rebuild the project after cleanup:" echo " 1. Run: ./build.sh # Full rebuild" -echo " 2. Or: docker-compose up --build # Docker rebuild" -echo " 3. Or: scripts/setup-submodules.sh # Restore submodules first" \ No newline at end of file +echo " 2. Or: cd build && docker-compose up --build # Docker rebuild" \ No newline at end of file diff --git a/scripts/setup-permissions.sh b/scripts/setup-permissions.sh deleted file mode 100755 index e873d36..0000000 --- a/scripts/setup-permissions.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash -# Ensure all scripts are executable - -chmod +x /build/scripts/*.sh - -echo "Script permissions updated:" -ls -la /build/scripts/*.sh | grep -E '\.(sh)$' \ No newline at end of file