tfgrid/zosstorage
11
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refine default orchestration flow and documentation

Browse Source 
- Document defaults-only configuration, kernel topology override, and deprecated CLI flags in README
- Mark schema doc as deprecated per ADR-0002
- Warn that --topology/--config are ignored; adjust loader/main/context flow
- Refactor orchestrator run() to auto-select mount/apply, reuse state when already provisioned, and serialize topology via Display
- Add Callgraph/FUNCTION_LIST/ADR docs tracking the new behavior
- Derive Eq for Topology to satisfy updated CLI handling
This commit is contained in:
Jan De Landtsheer
2025-10-09 16:51:12 +02:00
parent d374176c0b
commit c8b76a2a3d
11 changed files with 5405 additions and 297 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1716
docs/Callgraph.svg Normal file
View File

File diff suppressed because it is too large Load Diff
Side by Side

After

Width:  |  Height:  |  Size: 122 KiB

294
docs/FUNCTION_LIST.md Normal file
View File

@@ -0,0 +1,294 @@
# Function Reference - Call Graph Analysis
> This documentation is automatically derived from [`Callgraph.svg`](Callgraph.svg) and provides a comprehensive overview of all functions in the zosstorage project, organized by module.
## Table of Contents
- [Main Entry Points](#main-entry-points)
- [CLI & Configuration](#cli--configuration)
- [Orchestration](#orchestration)
- [Device Discovery](#device-discovery)
- [Partition Management](#partition-management)
- [Filesystem Operations](#filesystem-operations)
- [Mount Operations](#mount-operations)
- [Idempotency & State](#idempotency--state)
- [Reporting](#reporting)
- [Utilities](#utilities)
- [Logging](#logging)
- [Type Definitions](#type-definitions)
---
## Main Entry Points
### [`src/main.rs`](../src/main.rs)
| Function | Purpose |
|----------|---------|
| `main()` | Application entry point; initializes the program and handles top-level errors |
| `real_main()` | Core application logic; orchestrates the main workflow after initialization |
---
## CLI & Configuration
### [`src/cli/args.rs`](../src/cli/args.rs)
**Structs:** `Cli`, `LogLevelArg` (enum)
| Function | Purpose |
|----------|---------|
| `from_args()` | Parses command-line arguments and returns a `Cli` configuration object |
### [`src/config/loader.rs`](../src/config/loader.rs)
| Function | Purpose |
|----------|---------|
| `load_and_merge()` | Loads configuration from multiple sources and merges them into a unified config |
| `validate()` | Validates the merged configuration for correctness and completeness |
| `to_value()` | Converts configuration structures to internal value representation |
| `merge_value()` | Recursively merges configuration values, handling conflicts appropriately |
| `cli_overlay_value()` | Overlays CLI-provided values onto existing configuration |
| `kernel_cmdline_topology()` | Extracts topology information from kernel command line parameters |
| `parse_topology_token()` | Parses individual topology tokens from kernel cmdline |
| `default_config()` | Generates default configuration values when no config file is present |
---
## Orchestration
### [`src/orchestrator/run.rs`](../src/orchestrator/run.rs)
**Structs:** `Context`
| Function | Purpose |
|----------|---------|
| `Context::new()` | Creates a new orchestration context with default settings |
| `Context::with_show()` | Builder method to enable show/dry-run mode |
| `Context::with_apply()` | Builder method to enable apply mode (actual execution) |
| `Context::with_report_path()` | Builder method to set the report output path |
| `Context::with_mount_existing()` | Builder method to configure mounting of existing filesystems |
| `Context::with_report_current()` | Builder method to enable reporting of current system state |
| `Context::with_topology_from_cli()` | Builder method to set topology from CLI arguments |
| `Context::with_topology_from_cmdline()` | Builder method to set topology from kernel cmdline |
| `run()` | Main orchestration function; coordinates all storage operations |
| `build_device_filter()` | Constructs device filter based on configuration and user input |
| `enforce_empty_disks()` | Validates that target disks are empty before proceeding |
| `role_str()` | Converts partition role enum to human-readable string |
| `build_summary_json()` | Builds a JSON summary of operations performed |
---
## Device Discovery
### [`src/device/discovery.rs`](../src/device/discovery.rs)
**Structs:** `Disk`, `DeviceFilter`, `SysProvider`
**Traits:** `DeviceProvider`
| Function | Purpose |
|----------|---------|
| `DeviceFilter::matches()` | Checks if a device matches the configured filter criteria |
| `SysProvider::new()` | Creates a new sysfs-based device provider |
| `SysProvider::list_block_devices()` | Lists all block devices found via sysfs |
| `SysProvider::probe_properties()` | Probes detailed properties of a specific device |
| `discover()` | Entry point for device discovery using default provider |
| `discover_with_provider()` | Device discovery with custom provider (for testing/flexibility) |
| `is_ignored_name()` | Checks if device name should be ignored (loop, ram, etc.) |
| `sys_block_path()` | Constructs sysfs path for a given block device |
| `base_name()` | Extracts base device name from path |
| `is_removable_sysfs()` | Checks if device is removable via sysfs |
| `is_partition_sysfs()` | Checks if device is a partition via sysfs |
| `read_disk_size_bytes()` | Reads disk size in bytes from sysfs |
| `read_rotational()` | Determines if disk is rotational (HDD) or not (SSD) |
| `read_model_serial()` | Reads device model and serial number |
| `read_optional_string()` | Utility to safely read optional string values from sysfs |
---
## Partition Management
### [`src/partition/plan.rs`](../src/partition/plan.rs)
**Structs:** `PartitionSpec`, `DiskPlan`, `PartitionPlan`, `PartitionResult`
**Enums:** `PartRole`
| Function | Purpose |
|----------|---------|
| `plan_partitions()` | Creates partition plans for all target disks based on topology |
| `apply_partitions()` | Executes partition plans using sgdisk tool |
| `type_code()` | Returns GPT partition type code for a given partition role |
| `part_dev_path()` | Constructs device path for a partition (e.g., /dev/sda1) |
| `sector_size_bytes()` | Reads logical sector size of disk |
| `parse_sgdisk_info()` | Parses output from sgdisk to extract partition information |
---
## Filesystem Operations
### [`src/fs/plan.rs`](../src/fs/plan.rs)
**Structs:** `FsSpec`, `FsPlan`, `FsResult`
**Enums:** `FsKind`
| Function | Purpose |
|----------|---------|
| `plan_filesystems()` | Plans filesystem creation for all partitions |
| `make_filesystems()` | Creates filesystems according to plan (mkfs.* tools) |
| `capture_uuid()` | Captures UUID of newly created filesystem |
| `parse_blkid_export()` | Parses blkid export format to extract filesystem metadata |
| `probe_existing_filesystems()` | Detects existing filesystems on partitions |
---
## Mount Operations
### [`src/mount/ops.rs`](../src/mount/ops.rs)
**Structs:** `PlannedMount`, `PlannedSubvolMount`, `MountPlan`, `MountResult`
| Function | Purpose |
|----------|---------|
| `fstype_str()` | Converts FsKind enum to mount filesystem type string |
| `plan_mounts()` | Creates mount plans for all filesystems |
| `apply_mounts()` | Executes mount operations and creates mount points |
| `maybe_write_fstab()` | Conditionally writes /etc/fstab entries for persistent mounts |
---
## Idempotency & State
### [`src/idempotency/mod.rs`](../src/idempotency/mod.rs)
| Function | Purpose |
|----------|---------|
| `detect_existing_state()` | Detects existing partitions and filesystems to avoid destructive operations |
| `is_empty_disk()` | Checks if a disk has no partition table or filesystems |
| `parse_blkid_export()` | Parses blkid output to identify existing filesystems |
| `read_proc_partitions_names()` | Reads partition names from /proc/partitions |
| `base_name()` | Extracts base name from device path |
| `is_partition_of()` | Checks if one device is a partition of another |
---
## Reporting
### [`src/report/state.rs`](../src/report/state.rs)
**Structs:** `StateReport`
| Function | Purpose |
|----------|---------|
| `build_report()` | Builds comprehensive state report of operations performed |
| `write_report()` | Writes report to specified output path (JSON format) |
---
## Utilities
### [`src/util/mod.rs`](../src/util/mod.rs)
**Structs:** `CmdOutput`
| Function | Purpose |
|----------|---------|
| `which_tool()` | Locates external tool in PATH (sgdisk, mkfs.*, etc.) |
| `run_cmd()` | Executes shell command and returns exit status |
| `run_cmd_capture()` | Executes command and captures stdout/stderr |
| `udev_settle()` | Waits for udev to process device events |
| `is_efi_boot()` | Detects if system booted in EFI mode |
---
## Logging
### [`src/logging/mod.rs`](../src/logging/mod.rs)
**Structs:** `LogOptions`
| Function | Purpose |
|----------|---------|
| `LogOptions::from_cli()` | Creates logging configuration from CLI arguments |
| `level_from_str()` | Converts string to log level enum |
| `init_logging()` | Initializes logging subsystem with configured options |
---
## Type Definitions
### [`src/types.rs`](../src/types.rs)
**Core Configuration Structures:**
- `Config` - Top-level configuration structure
- `LoggingConfig` - Logging configuration
- `DeviceSelection` - Device selection criteria
- `Topology` - Storage topology definition (enum)
- `Partitioning` - Partition layout specification
- `BiosBootSpec`, `EspSpec`, `DataSpec`, `CacheSpec` - Partition type specifications
- `FsOptions`, `BtrfsOptions`, `BcachefsOptions`, `VfatOptions` - Filesystem options
- `MountScheme`, `MountSchemeKind` - Mount configuration
- `ReportOptions` - Report generation configuration
### [`src/errors.rs`](../src/errors.rs)
**Error Types:**
- `Error` - Main error enum for all error conditions
- `Result<T>` - Type alias for `std::result::Result<T, Error>`
---
## Call Graph Relationships
### Main Execution Flow
```
main() → real_main() → orchestrator::run()
↓
├─→ cli::from_args()
├─→ config::load_and_merge()
├─→ logging::init_logging()
├─→ device::discover()
├─→ partition::plan_partitions()
├─→ partition::apply_partitions()
├─→ fs::plan_filesystems()
├─→ fs::make_filesystems()
├─→ mount::plan_mounts()
├─→ mount::apply_mounts()
└─→ report::build_report() / write_report()
```
### Key Dependencies
- **Orchestrator** (`run()`) calls: All major subsystems
- **Device Discovery** uses: Utilities for system probing
- **Partition/FS/Mount** operations use: Utilities for command execution
- **All operations** call: `util::run_cmd()` or `util::run_cmd_capture()`
- **Idempotency checks** called by: Orchestrator before destructive operations
---
## Function Count Summary
- **Main Entry**: 2 functions
- **CLI & Config**: 9 functions
- **Orchestration**: 13 functions
- **Device Discovery**: 14 functions
- **Partition Management**: 6 functions
- **Filesystem Operations**: 5 functions
- **Mount Operations**: 4 functions
- **Idempotency**: 6 functions
- **Reporting**: 2 functions
- **Utilities**: 6 functions
- **Logging**: 3 functions
**Total: 70 documented functions** across 15 source files
---
## References
- Original call graph visualization: [`docs/Callgraph.svg`](Callgraph.svg)
- Architecture documentation: [`docs/ARCHITECTURE.md`](ARCHITECTURE.md)
- API documentation: [`docs/API.md`](API.md)

31
docs/SCHEMA.md
View File

@@ -1,27 +1,16 @@
# zosstorage Configuration Schema
# zosstorage Configuration (Deprecated schema)
This document defines the YAML configuration for the initramfs-only disk provisioning utility and the exact precedence rules between configuration sources. It complements [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
This schema document is deprecated per [docs/adr/0002-defaults-only-no-external-config.md](docs/adr/0002-defaults-only-no-external-config.md). Runtime now uses defaults-only with a single optional kernel cmdline override. The YAML configuration file is not read at boot.
Canonical paths and keys
- Kernel cmdline key: zosstorage.config=
- Default config file path: /etc/zosstorage/config.yaml
- JSON state report path: /run/zosstorage/state.json
- Optional log file path: /run/zosstorage/zosstorage.log
- fstab generation: disabled by default
- Reserved filesystem labels: ZOSBOOT (ESP), ZOSDATA (all data filesystems)
- GPT partition names: zosboot, zosdata, zoscache
Active behavior (ADR-0002)
- Defaults-only: all settings are defined in code. No /etc/zosstorage/config.yaml is read.
- Optional kernel cmdline override: `zosstorage.topology=VALUE` can override only the topology. Legacy alias `zosstorage.topo=` is accepted.
- CLI: `--config` and `--topology` are deprecated and ignored (warnings emitted). Operational flags remain (`--apply`, `--show`, `--report`, `--fstab`, logging).
- Report path: `/run/zosstorage/state.json`. Optional log file: `/run/zosstorage/zosstorage.log`.
- Reserved labels: `ZOSBOOT` (ESP), `ZOSDATA` (data). GPT names: `zosboot`, `zosdata`, `zoscache`.
Precedence and merge strategy
1. Start from built-in defaults documented here.
2. Merge in the on-disk config file if present at /etc/zosstorage/config.yaml.
3. Merge CLI flags next; these override file values.
4. Merge kernel cmdline last; zosstorage.config= overrides CLI and file.
5. No interactive prompts are permitted.
The kernel cmdline key zosstorage.config= accepts:
- A path to a YAML file inside the initramfs root (preferred).
- A file: absolute path (e.g., file:/run/config/zos.yaml).
- A data: URL containing base64 YAML (optional extension).
Historical reference (original YAML-based schema, no longer used at runtime)
The remainder of this document preserves the previous YAML schema for archival purposes only.
Top-level YAML structure

109
docs/adr/0002-defaults-only-no-external-config.md Normal file
View File

@@ -0,0 +1,109 @@
# ADR 0002: Defaults-Only Configuration; Remove External YAML Config
Status
- Accepted
- Date: 2025-10-06
Context
- Running from initramfs at first boot provides no reliable access to an on-disk configuration file (e.g., /etc/zosstorage/config.yaml). An external file cannot be assumed to exist or be mounted.
- The previous design added precedence and merge complexity across file, CLI, and kernel cmdline as documented in [docs/SCHEMA.md](../SCHEMA.md) and implemented via [fn load_and_merge()](../../src/config/loader.rs:1), increasing maintenance burden and risks of drift.
- YAML introduces misconfiguration risk in early boot, adds I/O, and complicates idempotency guarantees without meaningful benefits for the intended minimal-first initializer.
- The desired model is to ship with sane built-in defaults, selected automatically from detected hardware topology; optional kernel cmdline may override only the topology choice for VM/lab scenarios.
Decision
- Remove all dependency on an on-disk configuration file:
- Do not read /etc/zosstorage/config.yaml or any file-based config.
- Deprecate and ignore repository-local config files for runtime (e.g., config/zosstorage.yaml). The example file [config/zosstorage.example.yaml](../../config/zosstorage.example.yaml) remains as historical reference only and may be removed later.
- Deprecate the --config CLI flag in [struct Cli](../../src/cli/args.rs:1). If present, emit a deprecation warning and ignore it.
- Retain operational CLI flags and logging controls for usability:
- --apply, --show, --report PATH, --fstab, --log-level LEVEL, --log-to-file
- Replace the prior file/CLI/kernel precedence with a defaults-only policy plus a single optional kernel cmdline override:
- Recognized key: zosstorage.topology=VALUE
- The key may override only the topology selection; all other settings use built-in defaults.
- Topology defaults and override policy:
- 1 eligible disk:
- Default: btrfs_single
- Allowed cmdline overrides: btrfs_single, bcachefs_single
- 2 eligible disks:
- Default: dual_independent
- Allowed cmdline overrides: dual_independent, ssd_hdd_bcachefs, btrfs_raid1, bcachefs_2copy
- >2 eligible disks:
- Default: btrfs_raid1
- Allowed cmdline overrides: btrfs_raid1, bcachefs_2copy
- Accept both snake_case and hyphenated forms for VALUE; normalize to [enum Topology](../../src/types.rs:1):
- btrfs_single | btrfs-single
- bcachefs_single | bcachefs-single
- dual_independent | dual-independent
- ssd_hdd_bcachefs | ssd-hdd-bcachefs
- btrfs_raid1 | btrfs-raid1
- bcachefs_2copy | bcachefs-2copy
- Kernel cmdline parsing beyond topology is deferred; future extensions for VM workflows may be proposed separately.
Rationale
- Eliminates unreachable configuration paths at first boot and simplifies the mental model.
- Reduces maintenance overhead by removing schema and precedence logic.
- Minimizes early-boot I/O and failure modes while preserving a targeted override for lab/VMs.
- Keeps the tool safe-by-default and fully idempotent without depending on external files.
Consequences
- Documentation:
- Mark [docs/SCHEMA.md](../SCHEMA.md) as deprecated for runtime behavior; retain only as historical reference.
- Update [docs/ARCHITECTURE.md](../ARCHITECTURE.md) and [docs/SPECS.md](../SPECS.md) to reflect defaults-only configuration.
- Update [docs/API.md](../API.md) and [docs/API-SKELETONS.md](../API-SKELETONS.md) where they reference file-based config.
- CLI:
- [struct Cli](../../src/cli/args.rs:1) keeps operational flags; --config becomes a no-op with a deprecation warning.
- Code:
- Replace [fn load_and_merge()](../../src/config/loader.rs:1) with a minimal loader that:
- Builds a [struct Config](../../src/types.rs:1) entirely from baked-in defaults.
- Reads /proc/cmdline to optionally parse zosstorage.topology and normalize to [enum Topology](../../src/types.rs:1).
- Removes YAML parsing, file reads, and merge logic.
- Tests:
- Remove tests that depend on external YAML; add tests for cmdline override normalization and disk-count defaults.
Defaults (authoritative)
- Partitioning:
- GPT only, 1 MiB alignment, BIOS boot 1 MiB first unless UEFI detected via [fn is_efi_boot()](../../src/util/mod.rs:1).
- ESP 512 MiB labeled ZOSBOOT (GPT name: zosboot), data uses GPT name zosdata.
- Filesystems:
- ESP: vfat labeled ZOSBOOT
- Data: label ZOSDATA
- Backend per topology (btrfs for btrfs_*; bcachefs for ssd_hdd_bcachefs and bcachefs_2copy)
- Mount scheme:
- Root-mount all data filesystems under /var/mounts/{UUID}; final subvolume/subdir mounts from the primary data FS to /var/cache/{system,etc,modules,vm-meta}; fstab remains optional.
- Idempotency:
- Unchanged: already-provisioned signals exit success-without-changes via [fn detect_existing_state()](../../src/idempotency/mod.rs:1).
Implementation Plan
1) Introduce a minimal defaults loader in [src/config/loader.rs](../../src/config/loader.rs:1):
- new internal fn parse_topology_from_cmdline() -> Option<Topology>
- new internal fn normalize_topology(s: &str) -> Option<Topology>
- refactor load to construct Config from constants + optional topology override
2) CLI:
- Emit deprecation warning when --config is provided; ignore its value.
3) Docs:
- Add deprecation banner to [docs/SCHEMA.md](../SCHEMA.md).
- Adjust [README.md](../../README.md) to describe defaults and the zosstorage.topology override.
4) Tests:
- Add unit tests for normalization and disk-count policy; remove YAML-based tests.
Backward Compatibility
- External YAML configuration is no longer supported at runtime.
- Kernel cmdline key zosstorage.config= is removed. Only zosstorage.topology remains recognized.
- The JSON report, labels, GPT names, and mount behavior remain unchanged.
Security and Safety
- By eliminating external configuration input, we reduce attack surface and misconfiguration risk in early boot.
- The emptiness and idempotency checks continue to gate destructive operations.
Open Items
- Decide whether to accept additional synonyms (e.g., “bcachefs-raid1”) and map them to existing [enum Topology](../../src/types.rs:1) variants; default is to reject unknown values with a clear error.
- Potential future kernel cmdline keys (e.g., logging level) may be explored via a separate ADR.
Links
- Architecture: [docs/ARCHITECTURE.md](../ARCHITECTURE.md)
- API Index: [docs/API-SKELETONS.md](../API-SKELETONS.md)
- Specs: [docs/SPECS.md](../SPECS.md)
- CLI: [src/cli/args.rs](../../src/cli/args.rs)
- Config loader: [src/config/loader.rs](../../src/config/loader.rs)
- Types: [src/types.rs](../../src/types.rs)
- Util: [src/util/mod.rs](../../src/util/mod.rs)

2932
docs/adr/callgraph.html Normal file
View File

File diff suppressed because it is too large Load Diff