# RFS flist creation and runtime overmounts (design)

Goal
- Produce two flists without modifying existing build scripts:
  - firmware-VERSION.fl
  - modules-KERNEL_FULL_VERSION.fl
- Store blobs in S3 via rfs store; upload .fl manifest (sqlite) separately to S3.
- Overmount these at runtime later to enable extended hardware, then depmod + udev trigger.

Scope of this change
- Add standalone scripts under [scripts/rfs](scripts/rfs) (no changes in existing libs or stages).
- Add a config file [config/rfs.conf](config/rfs.conf) for S3 credentials and addressing.
- Document the flow and usage here; scripting comes next.

Inputs
- Built kernel modules present in the dev-container (from kernel build stages):
  - Preferred: /lib/modules/KERNEL_FULL_VERSION
- Firmware tree:
  - Preferred: $PROJECT_ROOT/firmware (prepopulated tree from dev-container: “$root/firmware”)
  - Fallback: initramfs/lib/firmware created by apk install of firmware packages
- Kernel version derivation (never use uname -r in container):
  - Combine KERNEL_VERSION from [config/build.conf](config/build.conf) and LOCALVERSION from [config/kernel.config](config/kernel.config).
  - This matches [kernel_get_full_version()](scripts/lib/kernel.sh:14).

Outputs and locations
- Flists:
  - [dist/flists/firmware-VERSION.fl](dist/flists/firmware-VERSION.fl)
  - [dist/flists/modules-KERNEL_FULL_VERSION.fl](dist/flists/modules-KERNEL_FULL_VERSION.fl)
- Blobs are uploaded by rfs to the configured S3 store.
- Manifests (.fl sqlite) are uploaded by script as S3 objects (separate from blob store).

Configuration: [config/rfs.conf](config/rfs.conf)
Required values:
- S3_ENDPOINT=https://s3.example.com:9000
- S3_REGION=us-east-1
- S3_BUCKET=zos
- S3_PREFIX=flists/zosbuilder
- S3_ACCESS_KEY=AKIA...
- S3_SECRET_KEY=...

Notes:
- We construct an rfs S3 store URI for pack operations (for blob uploads during pack):
  - s3://S3_ACCESS_KEY:S3_SECRET_KEY@HOST:PORT/S3_BUCKET/S3_PREFIX?region=S3_REGION
- After pack, we correct the flist route URL to include READ-ONLY credentials so mounts can read directly from Garage:
  - UPDATE route SET url='s3://READ_ACCESS_KEY:READ_SECRET_KEY@HOST:PORT/ROUTE_PATH?region=ROUTE_REGION'
  - Defaults: ROUTE_PATH=/blobs, ROUTE_REGION=garage, ROUTE_ENDPOINT=S3_ENDPOINT (overridable)

Scripts to add (standalone)
- [scripts/rfs/common.sh](scripts/rfs/common.sh)
  - Read [config/build.conf](config/build.conf) and [config/kernel.config](config/kernel.config).
  - Compute FULL_KERNEL_VERSION exactly as [kernel_get_full_version()](scripts/lib/kernel.sh:14).
  - Read and validate [config/rfs.conf](config/rfs.conf).
  - Build S3 store URI for rfs.
  - Locate module and firmware source trees (with priority rules).
  - Locate rfs binary (PATH first, fallback to [components/rfs/target/x86_64-unknown-linux-musl/release/rfs](components/rfs/target/x86_64-unknown-linux-musl/release/rfs)).

- [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh)
  - Name: modules-KERNEL_FULL_VERSION.fl (e.g., modules-6.12.44-Zero-OS.fl).
  - rfs pack -m dist/flists/modules-...fl -s s3://... /lib/modules/KERNEL_FULL_VERSION
  - Optional: upload dist/flists/modules-...fl to s3://S3_BUCKET/S3_PREFIX/manifests/ using MinIO Client (mc) if present.

- [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh)
  - Source: $PROJECT_ROOT/firmware if exists, else initramfs/lib/firmware.
  - Name: firmware-YYYYMMDD.fl by default; override with FIRMWARE_TAG env to firmware-FIRMWARE_TAG.fl.
  - rfs pack as above; optional upload of the .fl manifest using MinIO Client (mc) if present.

- [scripts/rfs/verify-flist.sh](scripts/rfs/verify-flist.sh)
  - rfs flist inspect dist/flists/NAME.fl
  - rfs flist tree dist/flists/NAME.fl | head
  - Optional: test mount if run with --mount (mountpoint under /tmp).

Runtime (deferred to a follow-up)
- New zinit units to mount and coldplug:
  - Mount firmware flist read-only at /usr/lib/firmware
  - Mount modules flist at /lib/modules/KERNEL_FULL_VERSION
  - Run depmod -a KERNEL_FULL_VERSION
  - udevadm control --reload; udevadm trigger --action=add; udevadm settle
- Placement examples (to be created later):
  - [config/zinit/rfs-modules.yaml](config/zinit/rfs-modules.yaml)
  - [config/zinit/rfs-firmware.yaml](config/zinit/rfs-firmware.yaml)
  - Keep in correct dependency order before [config/zinit/udev-trigger.yaml](config/zinit/udev-trigger.yaml).

Naming policy
- modules flist:
  - modules-KERNEL_FULL_VERSION.fl
- firmware flist:
  - firmware-YYYYMMDD.fl by default
  - firmware-FIRMWARE_TAG.fl if env FIRMWARE_TAG is set

Usage flow (after your normal build inside dev-container)
1) Create config for S3: [config/rfs.conf](config/rfs.conf)
2) Generate modules flist: [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh)
3) Generate firmware flist: [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh)
4) Verify manifests: [scripts/rfs/verify-flist.sh](scripts/rfs/verify-flist.sh) dist/flists/modules-...fl

Assumptions
- rfs supports s3 store URIs as described (per [components/rfs/README.md](components/rfs/README.md)).
- The dev-container has the built kernel modules in /lib/modules/KERNEL_FULL_VERSION (as produced via [kernel_build_modules()](scripts/lib/kernel.sh:228)).
- No changes are made to existing build scripts. The new scripts are run on-demand.

Open question for confirm
- Confirm S3 endpoint form (with or without explicit port) and whether we should prefer AWS_REGION env over query param; scripts will support both patterns.

Note on route URL vs HTTP endpoint
- rfs mount reads blobs via s3:// URLs, not via an arbitrary HTTP(S) endpoint. A reverse proxy is not required if you embed read-only S3 credentials in the flist.
- This project now patches the flist after pack to set route.url to a read-only Garage S3 URL:
  - Example SQL equivalent:
    - UPDATE route SET url='s3://READ_ACCESS_KEY:READ_SECRET_KEY@[HOST]:3900/blobs?region=garage';
- Configure these in config/rfs.conf:
  - READ_ACCESS_KEY / READ_SECRET_KEY: read-only credentials
  - ROUTE_ENDPOINT (defaults to S3_ENDPOINT), ROUTE_PATH=/blobs, ROUTE_REGION=garage
- Do not set ROUTE_PATH to S3_PREFIX. ROUTE_PATH is the gateway’s blob route (usually /blobs). S3_PREFIX is only for the pack-time store path.