Jan De Landtsheer
afd4f4c6f9
Summary
- Implemented plain S3-only flist workflow (no web endpoint). rfs pack uploads blobs using write creds; flist route.url is patched to embed read-only S3 credentials so rfs mount reads directly from S3.
Changes
1) New RFS tooling (scripts/rfs/)
- common.sh:
- Compute FULL_KERNEL_VERSION from configs (no uname).
- Load S3 config and construct pack store URI.
- Build read-only S3 route URL and patch flist (sqlite).
- Helpers to locate modules/firmware trees and rfs binary.
- pack-modules.sh:
- Pack /lib/modules/<FULL_KERNEL_VERSION> to dist/flists/modules-<FULL_KERNEL_VERSION>.fl
- Patch flist route to s3://READ:READ@host:port/ROUTE_PATH?region=ROUTE_REGION (default /blobs, garage).
- Optional upload of .fl using MinIO client (mcli/mc).
- pack-firmware.sh:
- Source firmware from $PROJECT_ROOT/firmware (fallback to initramfs/lib/firmware).
- Pack to dist/flists/firmware-<TAG_OR_DATE>.fl (FIRMWARE_TAG or YYYYMMDD).
- Patch flist route to read-only S3; optional .fl upload via mcli/mc.
- verify-flist.sh:
- rfs flist inspect/tree; optional mount test (best effort).
- patch-stores.sh:
- Helper to patch stores (kept though not used by default).
2) Dev-container (Dockerfile)
- Added sqlite and MinIO client package for manifest patching/upload (expect mcli binary at runtime; scripts support both mcli/mc).
- Retains rustup and musl target for building rfs/zinit/mycelium.
3) Config and examples
- config/rfs.conf.example:
- S3_ENDPOINT/S3_REGION/S3_BUCKET/S3_PREFIX
- S3_ACCESS_KEY/S3_SECRET_KEY (write)
- READ_ACCESS_KEY/READ_SECRET_KEY (read-only)
- ROUTE_ENDPOINT (defaults to S3_ENDPOINT), ROUTE_PATH=/blobs, ROUTE_REGION=garage
- MANIFESTS_SUBPATH, UPLOAD_MANIFESTS (mcli upload optional)
- config/rfs.conf updated by user with real values (not committed here; example included).
- config/modules.conf minor tweak (staged).
4) Zinit mount scripts (config/zinit/init/)
- firmware.sh:
- Mounts firmware-latest.fl over /usr/lib/firmware using rfs mount (env override FIRMWARE_FLIST supported).
- modules.sh:
- Mounts modules-$(uname -r).fl over /lib/modules/$(uname -r) (env override MODULES_FLIST supported).
- Both skip if target already mounted and respect RFS_BIN env.
5) Documentation
- docs/rfs-flists.md:
- End-to-end flow, S3-only route URL patching, mcli upload notes.
- docs/review-rfs-integration.md:
- Integration points, build flow, and post-build standalone usage.
- docs/depmod-behavior.md:
- depmod reads .modinfo; recommend prebuilt modules.*(.bin); use depmod -A only on mismatch.
6) Utility
- scripts/functionlist.md synced with current functions.
Behavioral details
- Pack (write):
s3://S3_ACCESS_KEY:S3_SECRET_KEY@HOST:PORT/S3_BUCKET/S3_PREFIX?region=REGION
- Flist route (read, post-patch):
s3://READ_ACCESS_KEY:READ_SECRET_KEY@HOST:PORT/ROUTE_PATH?region=ROUTE_REGION
Defaults: ROUTE_PATH=/blobs, ROUTE_REGION=garage; ROUTE_ENDPOINT derived from S3_ENDPOINT if not set.
Runtime mount examples
- Modules:
rfs mount -m dist/flists/modules-6.12.44-Zero-OS.fl /lib/modules/6.12.44-Zero-OS
- Firmware:
rfs mount -m dist/flists/firmware-YYYYMMDD.fl /usr/lib/firmware
Notes
- FUSE policy: If "allow_other" error occurs, enable user_allow_other in /etc/fuse.conf or run mounts as root.
- WEB_ENDPOINT rewrite is disabled by default (set WEB_ENDPOINT=""). Plain S3 route is embedded in flists.
- MinIO client binary in dev-container is mcli; scripts support mcli (preferred) and mc (fallback).
Files added/modified
- Added: scripts/rfs/{common.sh,pack-modules.sh,pack-firmware.sh,verify-flist.sh,patch-stores.sh}
- Added: config/zinit/init/{firmware.sh,modules.sh}
- Added: docs/{rfs-flists.md,review-rfs-integration.md,depmod-behavior.md}
- Added: config/rfs.conf.example
- Modified: Dockerfile, scripts/functionlist.md, config/modules.conf, config/zinit/sshd-setup.yaml, .gitignore
79 lines
3.2 KiB
Bash
Executable File
79 lines
3.2 KiB
Bash
Executable File
#!/bin/bash
|
|
# Pack firmware tree into an RFS flist and patch manifest stores for Garage web endpoint.
|
|
# - Computes FULL_KERNEL_VERSION from configs (not strictly needed for firmware, but kept uniform)
|
|
# - Selects firmware directory with priority:
|
|
# 1) $PROJECT_ROOT/firmware
|
|
# 2) $PROJECT_ROOT/initramfs/lib/firmware
|
|
# 3) /lib/firmware
|
|
# - Manifest name: firmware-<FIRMWARE_TAG or YYYYMMDD>.fl
|
|
# - Uploads blobs to S3 (Garage) via rfs store URI
|
|
# - Patches .fl sqlite stores table to use WEB_ENDPOINT for read-only fetches
|
|
# - Optionally uploads the .fl manifest to S3 manifests/ using aws CLI
|
|
|
|
set -euo pipefail
|
|
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# shellcheck source=/dev/null
|
|
source "${HERE}/common.sh"
|
|
|
|
section() { echo -e "\n==== $* ====\n"; }
|
|
|
|
section "Loading configuration (kernel + RFS S3) and locating rfs"
|
|
# Kernel version is computed for consistency/logging (not required to pack firmware)
|
|
rfs_common_load_build_kernel_version
|
|
rfs_common_load_rfs_s3_config
|
|
rfs_common_build_s3_store_uri
|
|
rfs_common_locate_rfs
|
|
|
|
section "Locating firmware directory"
|
|
rfs_common_locate_firmware_dir
|
|
|
|
TAG="$(rfs_common_firmware_tag)"
|
|
MANIFEST_NAME="firmware-${TAG}.fl"
|
|
MANIFEST_PATH="$(rfs_common_prepare_output "${MANIFEST_NAME}")"
|
|
|
|
section "Packing firmware to flist"
|
|
log_info "Firmware dir: ${FIRMWARE_DIR}"
|
|
log_info "rfs pack -m ${MANIFEST_PATH} -s ${RFS_S3_STORE_URI} ${FIRMWARE_DIR}"
|
|
safe_execute "${RFS_BIN}" pack -m "${MANIFEST_PATH}" -s "${RFS_S3_STORE_URI}" "${FIRMWARE_DIR}"
|
|
|
|
# Patch manifest route URL to include read-only S3 credentials (Garage)
|
|
section "Updating route URL in manifest to include read-only S3 credentials"
|
|
rfs_common_build_route_url
|
|
rfs_common_patch_flist_route_url "${MANIFEST_PATH}"
|
|
# Patch manifest stores to HTTPS web endpoint if provided
|
|
if [[ -n "${WEB_ENDPOINT:-}" ]]; then
|
|
section "Patching manifest stores to HTTPS web endpoint"
|
|
log_info "Patching ${MANIFEST_PATH} stores to: ${WEB_ENDPOINT} (keep_s3_fallback=${KEEP_S3_FALLBACK:-false})"
|
|
rfs_common_patch_flist_stores "${MANIFEST_PATH}" "${WEB_ENDPOINT}" "${KEEP_S3_FALLBACK:-false}"
|
|
else
|
|
log_warn "WEB_ENDPOINT not set in config; manifest will reference only s3:// store"
|
|
fi
|
|
|
|
# Optional: upload .fl manifest to Garage via MinIO Client (separate from blobs)
|
|
if [[ "${UPLOAD_MANIFESTS:-false}" == "true" ]]; then
|
|
section "Uploading manifest .fl via MinIO Client to S3 manifests/"
|
|
# Support both mcli (new) and mc (legacy) binaries
|
|
if command -v mcli >/dev/null 2>&1; then
|
|
MCLI_BIN="mcli"
|
|
elif command -v mc >/dev/null 2>&1; then
|
|
MCLI_BIN="mc"
|
|
else
|
|
log_warn "MinIO Client not found (expected mcli or mc); skipping manifest upload"
|
|
MCLI_BIN=""
|
|
fi
|
|
|
|
if [[ -n "${MCLI_BIN}" ]]; then
|
|
local_subpath="${MANIFESTS_SUBPATH:-manifests}"
|
|
# Configure alias and upload using MinIO client
|
|
safe_execute "${MCLI_BIN}" alias set rfs "${S3_ENDPOINT}" "${S3_ACCESS_KEY}" "${S3_SECRET_KEY}"
|
|
mcli_dst="rfs/${S3_BUCKET}/${S3_PREFIX%/}/${local_subpath%/}/${MANIFEST_NAME}"
|
|
log_info "${MCLI_BIN} cp ${MANIFEST_PATH} ${mcli_dst}"
|
|
safe_execute "${MCLI_BIN}" cp "${MANIFEST_PATH}" "${mcli_dst}"
|
|
fi
|
|
else
|
|
log_info "UPLOAD_MANIFESTS=false; skipping manifest upload"
|
|
fi
|
|
|
|
section "Done"
|
|
log_info "Manifest: ${MANIFEST_PATH}" |