forked from tfgrid/zosbuilder
docs: regenerate scripts/functionlist.md; refresh NOTES with jump-points and roadmap; extend rfs-flists with RESP backend design. config: add RESP placeholders to rfs.conf.example. components: keep previous non-destructive git clone logic.
This commit is contained in:
@@ -203,4 +203,79 @@ Change Log
|
||||
- Enforce passwordless root using passwd -d -R in finalization.
|
||||
- Normalize INSTALL_DIR/COMPONENTS_DIR/KERNEL_DIR/DIST_DIR to absolute paths post-config load.
|
||||
- Add validation diagnostics prints (input/PWD/PROJECT_ROOT/INSTALL_DIR/resolved).
|
||||
- Ensure shadow package in container for passwd/chpasswd; keep openssl and openssl-dev; remove perl earlier.
|
||||
- Ensure shadow package in container for passwd/chpasswd; keep openssl and openssl-dev; remove perl earlier.
|
||||
Updates 2025-10-01
|
||||
|
||||
- Function index regenerated: see [scripts/functionlist.md](scripts/functionlist.md) for an authoritative map of all functions with current line numbers. Use it alongside the quick links below to jump into code fast.
|
||||
- Key jump-points (current lines):
|
||||
- Finalization: [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:568)
|
||||
- CPIO creation: [bash.initramfs_create_cpio()](scripts/lib/initramfs.sh:691)
|
||||
- Validation: [bash.initramfs_validate()](scripts/lib/initramfs.sh:820)
|
||||
- Kernel embed config: [bash.kernel_modify_config_for_initramfs()](scripts/lib/kernel.sh:130)
|
||||
- Stage orchestrator entry: [bash.main_build_process()](scripts/build.sh:214)
|
||||
- Repo-wide index: [scripts/functionlist.md](scripts/functionlist.md)
|
||||
|
||||
Roadmap / TODO (tracked in tool todo list)
|
||||
|
||||
- Zosception (zinit service graph and ordering)
|
||||
- Define additional services and ordering for nested/recursive orchestration.
|
||||
- Likely integration points:
|
||||
- Networking readiness before RFS: [config/zinit/network.yaml](config/zinit/network.yaml)
|
||||
- Early udev coldplug: [config/zinit/udev-trigger.yaml](config/zinit/udev-trigger.yaml)
|
||||
- Post-RFS coldplug: [config/zinit/udev-rfs.yaml](config/zinit/udev-rfs.yaml)
|
||||
- Ensure dependency edges are correct in the service DAG image (see docs/img_*.png).
|
||||
|
||||
- Add zosstorage to initramfs
|
||||
- Source:
|
||||
- If packaged: add to [config/packages.list](config/packages.list).
|
||||
- If built from source: extend [bash.components_parse_sources_conf()](scripts/lib/components.sh:13) and add a build_* function; install via [bash.initramfs_copy_components()](scripts/lib/initramfs.sh:102).
|
||||
- Zinit unit:
|
||||
- Add YAML under [config/zinit/](config/zinit/) and hook into the network-ready path.
|
||||
- Ordering:
|
||||
- Start after "network" and before/with RFS mounts if it provides storage functionality used by rfs.
|
||||
|
||||
- RFS blob store backends (design + docs; http and s3 exist)
|
||||
- Current S3 store URI construction: [bash.rfs_common_build_s3_store_uri()](scripts/rfs/common.sh:137)
|
||||
- Flist manifest store patching: [bash.rfs_common_patch_flist_stores()](scripts/rfs/common.sh:385)
|
||||
- Route URL patching: [bash.rfs_common_patch_flist_route_url()](scripts/rfs/common.sh:494)
|
||||
- Packers entrypoints:
|
||||
- [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh:1)
|
||||
- [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh:1)
|
||||
- Proposed additional backend: RESP/DB-style store
|
||||
- Goal: Allow rfs to push/fetch content-addressed blobs via a RESP-compatible endpoint (e.g., Redis/KeyDB/Dragonfly-like), or a thin HTTP/RESP adapter.
|
||||
- Draft URI scheme examples:
|
||||
- resp://host:port/db?tls=0&prefix=blobs
|
||||
- resp+tls://host:port/db?prefix=blobs&ca=/etc/ssl/certs/ca.pem
|
||||
- resp+sentinel://sentinelHost:26379/mymaster?prefix=blobs
|
||||
- Minimum operations:
|
||||
- PUT blob: SETEX prefix/ab/cd/hash ttl file-bytes or HSET prefix/hash data file-bytes
|
||||
- GET blob: GET or HGET
|
||||
- HEAD/exists: EXISTS
|
||||
- Optional: pipelined/mget for batch prefetch
|
||||
- Client integration layers:
|
||||
- Pack-time: extend rfs CLI store resolver (design doc first; scripts/rfs/common.sh can map scheme→uploader if CLI not ready).
|
||||
- Manifest post-process: still supported; stores table may include multiple URIs (s3 + resp) for redundancy.
|
||||
- Caching and retries:
|
||||
- Local on-disk cache under dist/.rfs-cache keyed by hash with LRU GC.
|
||||
- Exponential backoff on GET failures; fall back across stores in order.
|
||||
- Auth:
|
||||
- RESP: optional username/password in URI; TLS with cert pinning parameters.
|
||||
- Keep secrets in config/rfs.conf or env; do not embed write creds in manifests (read-credential routes only).
|
||||
- Deliverables:
|
||||
- Design section in docs/rfs-flists.md (to be added)
|
||||
- Config keys in config/rfs.conf.example for RESP endpoints
|
||||
- Optional shim uploader script if CLI support lags.
|
||||
|
||||
- Documentation refresh tasks
|
||||
- Cross-check this file’s clickable references against [scripts/functionlist.md](scripts/functionlist.md) after changes in lib files.
|
||||
- Keep “Branding behavior” and “Absolute Path Normalization” pointers aligned with:
|
||||
- [bash.common.sh normalization](scripts/lib/common.sh:244)
|
||||
- [bash.initramfs_finalize_customization()](scripts/lib/initramfs.sh:568)
|
||||
|
||||
Diagnostics-first reminder
|
||||
|
||||
- Use DEBUG=1 and stage markers for minimal rebuilds.
|
||||
- Quick commands:
|
||||
- Show stages: ./scripts/build.sh --show-stages
|
||||
- Minimal rebuild after zinit/init edits: [scripts/rebuild-after-zinit.sh](scripts/rebuild-after-zinit.sh)
|
||||
- Validate archive: [bash.initramfs_create_cpio()](scripts/lib/initramfs.sh:691), then [bash.initramfs_test_archive()](scripts/lib/initramfs.sh:953)
|
||||
|
||||
@@ -165,3 +165,45 @@ Use the helper to inspect a manifest, optionally listing entries and testing a l
|
||||
- scripts/rfs/verify-flist.sh -m dist/flists/firmware-latest.fl --tree
|
||||
- Inspect + mount test to a temp dir:
|
||||
- sudo scripts/rfs/verify-flist.sh -m dist/flists/modules-6.12.44-Zero-OS.fl --mount
|
||||
|
||||
## Additional blob store backends (design)
|
||||
|
||||
This extends the existing S3/HTTP approach with a RESP/DB-style backend option for rfs blob storage. It is a design-only addition; CLI and scripts will be extended in a follow-up.
|
||||
|
||||
Scope
|
||||
- Keep S3 flow intact via [scripts/rfs/common.sh](scripts/rfs/common.sh:137), [scripts/rfs/common.sh](scripts/rfs/common.sh:385), and [scripts/rfs/common.sh](scripts/rfs/common.sh:494).
|
||||
- Introduce RESP URIs that can be encoded in config and, later, resolved by rfs or a thin uploader shim invoked by:
|
||||
- [scripts/rfs/pack-modules.sh](scripts/rfs/pack-modules.sh:1)
|
||||
- [scripts/rfs/pack-firmware.sh](scripts/rfs/pack-firmware.sh:1)
|
||||
|
||||
URI schemes (draft)
|
||||
- resp://host:port/db?prefix=blobs
|
||||
- resp+tls://host:port/db?prefix=blobs&ca=/etc/ssl/certs/ca.pem
|
||||
- resp+sentinel://sentinelHost:26379/mymaster?prefix=blobs
|
||||
- Credentials may be provided via URI userinfo or config (recommended: config only).
|
||||
|
||||
Operations (minimal set)
|
||||
- PUT blob: write content-addressed key (e.g., prefix/ab/cd/hash)
|
||||
- GET blob: fetch by exact key
|
||||
- Exists/HEAD: presence test by key
|
||||
- Optional batching: pipelined MGET for prefetch
|
||||
|
||||
Config keys (see example additions in config/rfs.conf.example)
|
||||
- RESP_ENDPOINT (host:port), RESP_DB (integer), RESP_PREFIX (path namespace)
|
||||
- RESP_USERNAME/RESP_PASSWORD (optional), RESP_TLS=0/1 (+ RESP_CA if needed)
|
||||
- RESP_SENTINEL and RESP_MASTER for sentinel deployments
|
||||
|
||||
Manifests and routes
|
||||
- Keep S3 store in flist stores table (fallback) while enabling route.url patching to HTTP/S3 for read-only access:
|
||||
- Patch stores table as today via [scripts/rfs/common.sh](scripts/rfs/common.sh:385)
|
||||
- Patch route.url as today via [scripts/rfs/common.sh](scripts/rfs/common.sh:494)
|
||||
- RESP may be used primarily for pack-time blob uploads or as an additional store the CLI can consume later.
|
||||
|
||||
Security
|
||||
- Do not embed write credentials in manifests.
|
||||
- Read-only credentials may be embedded in route.url if required, mirroring S3 pattern.
|
||||
|
||||
Next steps
|
||||
- Implement RESP uploader shim called from pack scripts; keep the CLI S3 flow unchanged.
|
||||
- Extend config loader in [scripts/rfs/common.sh](scripts/rfs/common.sh:82) to parse RESP_* variables.
|
||||
- Add verification routines to sanity-check connectivity before pack.
|
||||
|
||||
Reference in New Issue
Block a user