tfgrid/zosbuilder
11
0
Fork 1
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Clean up build system and update documentation

Browse Source 
- Updated README.md to reflect current single-stage build architecture
- Removed obsolete scripts/build-boot.sh (unused two-stage boot approach)
- Updated cleanup.sh to remove reference to non-existent setup-submodules.sh
- Documented smart caching and component compilation workflow
- Added proper build variants (minimal vs full) documentation
This commit is contained in:
Jan De Landtsheer
2025-08-18 14:41:46 +02:00
parent e402de625d
commit e13034810d
4 changed files with 143 additions and 258 deletions
Download Patch File Download Diff File Expand all files Collapse all files

212
README.md
View File

@@ -7,130 +7,152 @@ Modern Alpine Linux based approach to building Zero-OS initramfs with source com
This system uses:
- **Alpine Linux 3.22** as base system with latest packages
- **Individual Rust workspaces** with git subtrees for Zero-OS components (zinit, mycelium, rfs)
- **Component-based compilation** - each component built in its own workspace
- **Component-based compilation** - each component built from source in its own workspace
- **Alpine packages** for system tools (busybox, openssh, btrfs-progs, etc.)
- **Smart caching** with Docker layers and volumes
- **Same init flow** as original Zero-OS
- **Smart caching** with Docker layers and volumes for fast rebuilds
- **Single-stage initramfs** with minimal (~50MB) or full (~700MB) variants
## Directory Structure
```
alpine-initramfs/
├── build/ # Build orchestration
│ ├── Dockerfile.alpine # Alpine build environment
│ ├── Dockerfile.alpine # Alpine build environment
│ ├── Dockerfile.cached # Multi-stage cached build
│ └── docker-compose.yml # Build orchestration with caching
├── components/ # Git subtrees (individual workspaces)
│ ├── zinit/ # Init system (Rust subtree with workspace)
│ ├── mycelium/ # Networking layer (Rust subtree with workspace)
│ └── rfs/ # Filesystem (Rust subtree with workspace)
│ ├── zinit/ # Init system (Rust workspace)
│ ├── mycelium/ # Networking layer (Rust workspace)
│ └── rfs/ # Filesystem (Rust workspace)
├── configs/ # Configuration files
│ ├── packages-alpine.txt # Alpine packages to install
│ ├── packages-alpine.txt # Full Alpine packages (~700MB)
│ ├── packages-minimal.txt # Minimal packages (~50MB)
│ ├── kernel-version # Kernel version to build
│ ├── kernel-config-generic # Kernel configuration
│ ├── init # Init script (Alpine → zinit)
│ └── zinit/ # Zinit service configurations
├── scripts/ # Build scripts
│ ├── build-initramfs.sh # Main build orchestrator
│ ├── compile-components.sh # Compile Rust workspace
│ ├── fetch-github.sh # Download CoreX binary
│ ├── build-smart.sh # Smart build with caching (main orchestrator)
│ ├── compile-components.sh # Compile Rust components from source
│ ├── install-packages.sh # Install Alpine packages
│ ├── setup-initramfs.sh # Create initramfs structure
│ ├── setup-initramfs.sh # Create initramfs structure and archive
│ ├── build-kernel.sh # Build kernel with embedded initramfs
│ ├── build-smart.sh # Smart build with caching
│ └── cleanup.sh # Clean build artifacts
├── target/ # Rust workspace build artifacts
├── output/ # Final build outputs
│ ├── vmlinuz.efi # Final bootable kernel
│ ├── initramfs.cpio.xz # Generated initramfs
│ └── version.txt # Build information
└── docs/ # Documentation
├── BUILD.md # Build process guide
├── OVERVIEW.md # Architecture overview
└── *.md # Additional documentation
│ ├── initramfs.cpio.xz # Generated initramfs archive
│ └── compiled binaries # Individual component binaries
└── cache/ # Build cache for smart rebuilds
├── components-marker # Component build cache marker
└── kernel-marker # Kernel build cache marker
```
## Build Flow
1. **Environment Setup**: Alpine Docker container with Rust and build tools
2. **Component Compilation**: Individual cargo builds for each Rust component
3. **Binary Downloads**: Download CoreX (Go binary) from GitHub releases
4. **Package Installation**: Install Alpine packages to initramfs root
5. **Initramfs Assembly**: Create filesystem structure with compiled binaries
6. **Kernel Build**: Compile kernel with embedded initramfs
7. **Output**: Generate bootable vmlinuz.efi
The build system uses intelligent caching to minimize rebuild time:
1. **Smart Analysis**: Check what components need rebuilding based on file changes
2. **Component Compilation**: Build only changed Rust components (zinit, mycelium, rfs)
3. **Package Installation**: Install Alpine packages to initramfs root
4. **Initramfs Assembly**: Create filesystem structure with compiled binaries
5. **Kernel Build**: Compile kernel only if config/version changed
6. **Final Assembly**: Embed initramfs into kernel → bootable vmlinuz.efi
## Quick Start
### Full Build
### Build Everything
```bash
# Build everything
# Build with default settings (debug mode, full packages)
./build.sh
# Or with Docker Compose
cd build/
docker-compose up
# Build minimal embedded initramfs (~50MB)
./build.sh --minimal
# Build release mode
./build.sh --release
# Clean build without cache
./build.sh --clean
```
### Development Build
### Build Variants
```bash
cd build/
# Interactive development shell
docker-compose run dev-shell
# Minimal embedded initramfs (~50MB)
./build.sh --minimal
# Build all components (automated)
../scripts/compile-components.sh
# Full featured initramfs (~700MB)
./build.sh
# Development with clean rebuild
./build.sh --clean --dev
```
### Development Workflow
```bash
# Start interactive development shell
./build.sh --dev
# Inside container:
/build/scripts/compile-components.sh # Build all components
/build/scripts/build-smart.sh # Smart build with caching
# Build individual components manually:
cd ../components/zinit && cargo build --release --target x86_64-unknown-linux-musl
cd ../components/mycelium && cargo build --release --target x86_64-unknown-linux-musl
cd ../components/rfs && cargo build --release --target x86_64-unknown-linux-musl
```
### Smart Build (with caching)
```bash
cd build/
docker-compose run builder # Uses build-smart.sh automatically
cd /build/components/zinit && cargo build --release --target x86_64-unknown-linux-musl
cd /build/components/mycelium/myceliumd && cargo build --release --target x86_64-unknown-linux-musl
cd /build/components/rfs && cargo build --release --target x86_64-unknown-linux-musl
```
## Key Features
- ✅ **Smart caching** - Only rebuild components that changed
- ✅ **Component workspaces** - Each component maintains its own workspace
- ✅ **Git subtrees** - Source code included, no submodule complexity
- ✅ **Automated build** - `scripts/compile-components.sh` builds all components
- ✅ **Shared target directory** - Efficient build artifact sharing
- ✅ **Smart caching** - Docker layer and volume caching for fast rebuilds
- ✅ **Alpine packages** - System tools from Alpine repositories
- ✅ **Source compilation** - All Zero-OS components built from source
- ✅ **Dual size modes** - Minimal (~50MB) or full (~700MB) initramfs
- ✅ **Docker layer caching** - Fast rebuilds with persistent cache volumes
- ✅ **Alpine packages** - Latest system tools from Alpine repositories
- ✅ **Development mode** - Full IDE support and integrated tooling
- ✅ **Same functionality** - 100% compatible with original Zero-OS
## Build Options
### Package Modes
- **Minimal Mode** (`--minimal`): ~50MB embedded initramfs with essential packages only
- **Full Mode** (default): ~700MB initramfs with complete package set
### Build Modes
- **Debug** (default): Faster compilation with debug symbols
- **Release** (`--release`): Optimized binaries for production
### Cache Control
- **Smart Build** (default): Use cache when possible
- **Clean Build** (`--clean`): Force complete rebuild
## Development
### Building Individual Components
```bash
cd build/
docker-compose run dev-shell
# Start development shell
./build.sh --dev
# Inside container - build all components:
../scripts/compile-components.sh
# Build all components
/build/scripts/compile-components.sh
# Or build specific components manually:
cd ../components/zinit && cargo build --release --target x86_64-unknown-linux-musl
cd ../components/mycelium/myceliumd && cargo build --release --target x86_64-unknown-linux-musl
cd ../components/rfs && cargo build --release --target x86_64-unknown-linux-musl
```
# Or build individual components:
cd /build/components/zinit
cargo build --release --target x86_64-unknown-linux-musl
### Cleanup
```bash
# Clean all build artifacts
./scripts/cleanup.sh
cd /build/components/mycelium/myceliumd
cargo build --release --target x86_64-unknown-linux-musl
cd /build/components/rfs
cargo build --release --target x86_64-unknown-linux-musl
```
### Cache Management
Smart build uses Docker volumes for caching:
- `build-cache`: Build state markers
- `source-cache`: Downloaded sources
- `kernel-cache`: Kernel builds
- `build-cache`: Build state markers for smart rebuilds
- `source-cache`: Downloaded sources and dependencies
- `kernel-cache`: Compiled kernel builds
- `target-cache`: Rust workspace build artifacts
### Updating Components
@@ -139,9 +161,59 @@ Update git subtree components:
# Update zinit to latest
git subtree pull --prefix=components/zinit https://github.com/threefoldtech/zinit.git master --squash
# Update mycelium to latest
# Update mycelium to latest
git subtree pull --prefix=components/mycelium https://github.com/threefoldtech/mycelium.git master --squash
# Update rfs to latest
git subtree pull --prefix=components/rfs https://github.com/threefoldtech/rfs.git master --squash
```
```
### Cleanup
```bash
# Clean all build artifacts and cache
./build.sh --clean
# Or clean specific components
./scripts/cleanup.sh
```
## Testing
Test the built kernel:
```bash
# Boot with QEMU
qemu-system-x86_64 \
-kernel output/vmlinuz.efi \
-m 2048 \
-enable-kvm \
-cpu host \
-net nic,model=e1000 \
-net bridge,br=vm0 \
-nographic \
-serial null \
-serial mon:stdio \
-append "console=ttyS1,115200n8"
```
## Architecture Notes
### Single-Stage Boot
This system uses a single-stage initramfs approach where the entire Zero-OS environment is embedded in the kernel. This provides:
- Simpler deployment (single vmlinuz.efi file)
- Faster boot times (no network dependencies)
- Better reliability (no network mount failures)
### Component Compilation
All Zero-OS components (zinit, mycelium, rfs) are compiled from source rather than downloaded as binaries. This ensures:
- Consistent build environment
- Version control over all components
- Easy development and debugging
- Security through reproducible builds
### Smart Caching
The build system tracks file changes and only rebuilds what's necessary:
- Component sources unchanged → use cached binaries
- Kernel config unchanged → use cached kernel
- Only initramfs assembly runs every time (fast)
This typically reduces rebuild time from 10+ minutes to under 2 minutes.