rfs(firmware): pack full Alpine linux-firmware set from container and overmount /lib/firmware

Change pack source: install all linux-firmware* packages in container and pack from /lib/firmware via [bash.rfs_common_install_all_alpine_firmware_packages()](scripts/rfs/common.sh:290) used by [bash.pack-firmware.sh](scripts/rfs/pack-firmware.sh:21). At runtime, overmount firmware flist on /lib/firmware by updating [sh.firmware.sh](config/zinit/init/firmware.sh:10). Update docs to reflect /lib/firmware mount and new pack strategy.
2025-09-19 08:27:10 +02:00
parent 79ed723303
commit 310e11d2bf
5 changed files with 100 additions and 13 deletions
--- a/config/zinit/init/firmware.sh
+++ b/config/zinit/init/firmware.sh
@@ -1,5 +1,5 @@
 #!/bin/sh
-# rfs mount firmware flist over /usr/lib/firmware (plain S3 route inside the .fl)
+# rfs mount firmware flist over /lib/firmware (plain S3 route inside the .fl)
 # Looks for firmware-latest.fl in known locations; can be overridden via FIRMWARE_FLIST env

 set -eu
@@ -7,7 +7,7 @@ set -eu
 log() { echo "[rfs-firmware] $*"; }

 RFS_BIN="${RFS_BIN:-rfs}"
-TARGET="/usr/lib/firmware"
+TARGET="/lib/firmware"
 BASE_URL="${FLISTS_BASE_URL:-https://zos.grid.tf/store/flists}"

 # Allow override via env
--- a/docs/depmod-behavior.md
+++ b/docs/depmod-behavior.md
@@ -31,7 +31,7 @@ Key property: depmod’s default operation opens many .ko files to read .modinfo
  - Option C: Run depmod -a only if you changed the module set or path layout. Expect many small reads on first run.

 3) Firmware implications
- No depmod impact, but udev coldplug will probe devices. Keep firmware files accessible via the firmware flist mount (e.g., /usr/lib/firmware).
+- No depmod impact, but udev coldplug will probe devices. Keep firmware files accessible via the firmware flist mount (e.g., /lib/firmware).
 - Since firmware loads on-demand by the kernel/driver, the lazy store will fetch only needed blobs.

 4) Post-processing .fl to use a web endpoint (garage S3 private)
--- a/docs/rfs-flists.md
+++ b/docs/rfs-flists.md
@@ -15,9 +15,10 @@ Scope of this change
 Inputs
 - Built kernel modules present in the dev-container (from kernel build stages):
  - Preferred: /lib/modules/KERNEL_FULL_VERSION
- Firmware tree:
-  - Preferred: $PROJECT_ROOT/firmware (prepopulated tree from dev-container: “$root/firmware”)
-  - Fallback: initramfs/lib/firmware created by apk install of firmware packages
+- Firmware source for RFS pack:
+-   Install all Alpine linux-firmware* packages into the build container and use /lib/firmware as the source (full set).
+- Initramfs fallback (build-time):
+-   Selective firmware packages installed by [bash.alpine_install_firmware()](scripts/lib/alpine.sh:392) into initramfs/lib/firmware (kept inside the initramfs).
 - Kernel version derivation (never use uname -r in container):
  - Combine KERNEL_VERSION from [config/build.conf](config/build.conf) and LOCALVERSION from [config/kernel.config](config/kernel.config).
  - This matches [kernel_get_full_version()](scripts/lib/kernel.sh:14).
@@ -71,7 +72,7 @@ Scripts to add (standalone)

 Runtime (deferred to a follow-up)
 - New zinit units to mount and coldplug:
-  - Mount firmware flist read-only at /usr/lib/firmware
+  -   Mount firmware flist read-only at /lib/firmware
  - Mount modules flist at /lib/modules/KERNEL_FULL_VERSION
  - Run depmod -a KERNEL_FULL_VERSION
  - udevadm control --reload; udevadm trigger --action=add; udevadm settle
--- a/scripts/rfs/common.sh
+++ b/scripts/rfs/common.sh
@@ -287,6 +287,92 @@ rfs_common_validate_modules_metadata() {
  fi
 }

+# -----------------------------------------------------------------------------
+# Alpine firmware installation in container (for flist packing)
+# -----------------------------------------------------------------------------
+# Install all available linux-firmware* packages into the build container root.
+# This prepares a full /lib/firmware tree to be packed into an RFS flist.
+# Notes:
+# - Requires root inside the container (the builder image runs scripts as root).
+# - Uses apk search to enumerate all linux-firmware packages; installs them all.
+rfs_common_install_all_alpine_firmware_packages() {
+  log_info "Installing all Alpine linux-firmware* packages into container (/lib/firmware)"
+  # Ensure apk index is fresh
+  safe_execute apk update
+
+  # Enumerate firmware packages. Output of 'apk search' looks like:
+  #   linux-firmware-rtl_nic-20231030-r0
+  # Strip version suffix to get the package names acceptable to apk add.
+  local pkgs_raw
+  pkgs_raw="$(apk search 'linux-firmware*' 2>/dev/null || true)"
+
+  if [[ -z "${pkgs_raw// }" ]]; then
+    log_warn "No linux-firmware* packages found via apk search"
+  fi
+
+  # Build unique package list without versions
+  # 1) take first column (package-with-version)
+  # 2) strip trailing '-<digits...>' (version/revision)
+  # 3) dedupe
+  local pkgs=()
+  while IFS= read -r line; do
+    [[ -z "${line// }" ]] && continue
+    local name="${line%% *}"
+    name="$(echo "$name" | sed -E 's/-[0-9].*$//')"
+    if [[ -n "$name" ]]; then
+      pkgs+=("$name")
+    fi
+  done <<< "$pkgs_raw"
+
+  if [[ ${#pkgs[@]} -eq 0 ]]; then
+    log_warn "Firmware package list is empty after parsing; attempting base meta-package 'linux-firmware'"
+    pkgs=("linux-firmware")
+  fi
+
+  # Deduplicate while preserving order
+  local seen="" final_pkgs=()
+  for p in "${pkgs[@]}"; do
+    if ! grep -qx "$p" <<< "$seen"; then
+      final_pkgs+=("$p")
+      seen+=$'\n'"$p"
+    fi
+  done
+
+  log_info "Installing ${#final_pkgs[@]} firmware packages:"
+  for p in "${final_pkgs[@]}"; do
+    log_info "  - $p"
+  done
+
+  # Install all firmware packages; allow some failures (not all subpackages exist on all arches)
+  local failed=()
+  for p in "${final_pkgs[@]}"; do
+    if apk add --no-cache "$p" >/dev/null 2>&1; then
+      log_debug "Installed: $p"
+    else
+      log_warn "Failed to install: $p"
+      failed+=("$p")
+    fi
+  done
+
+  # Quick check that /lib/firmware exists and is populated
+  if [[ -d "/lib/firmware" ]]; then
+    local cnt
+    cnt=$(find /lib/firmware -type f | wc -l || echo 0)
+    log_info "/lib/firmware population: ${cnt} files"
+    if [[ "$cnt" -eq 0 ]]; then
+      log_warn "/lib/firmware exists but is empty after installation"
+    fi
+  else
+    log_error "/lib/firmware directory not found after firmware installation"
+    return 1
+  fi
+
+  if [[ ${#failed[@]} -gt 0 ]]; then
+    log_warn "Some firmware packages failed to install (${#failed[@]} failures); proceeding with available set"
+  fi
+  return 0
+}
+
 # -----------------------------------------------------------------------------
 # Manifest patching (sqlite .fl)
 # -----------------------------------------------------------------------------
--- a/scripts/rfs/pack-firmware.sh
+++ b/scripts/rfs/pack-firmware.sh
@@ -1,10 +1,8 @@
 #!/bin/bash
 # Pack firmware tree into an RFS flist and patch manifest stores for Garage web endpoint.
 # - Computes FULL_KERNEL_VERSION from configs (not strictly needed for firmware, but kept uniform)
-# - Selects firmware directory with priority:
-#     1) $PROJECT_ROOT/firmware
-#     2) $PROJECT_ROOT/initramfs/lib/firmware
-#     3) /lib/firmware
+# - Installs all Alpine linux-firmware* packages into the build container (/lib/firmware)
+# - Packs from container's /lib/firmware into an RFS flist (reproducible, full set)
 # - Manifest name: firmware-<FIRMWARE_TAG or YYYYMMDD>.fl
 # - Uploads blobs to S3 (Garage) via rfs store URI
 # - Patches .fl sqlite stores table to use WEB_ENDPOINT for read-only fetches
@@ -25,8 +23,10 @@ rfs_common_load_rfs_s3_config
 rfs_common_build_s3_store_uri
 rfs_common_locate_rfs

-section "Locating firmware directory"
-rfs_common_locate_firmware_dir
+section "Installing full Alpine firmware set in container (source: /lib/firmware)"
+rfs_common_install_all_alpine_firmware_packages
+export FIRMWARE_DIR="/lib/firmware"
+log_info "Using firmware source dir: ${FIRMWARE_DIR}"

 TAG="$(rfs_common_firmware_tag)"
 MANIFEST_NAME="firmware-${TAG}.fl"