geomind_code/my_fs
23
0
Fork 0
Code Issues 1 Pull requests Projects Releases 2 Packages Wiki Activity Actions

merge-all-branches #28

Closed
despiegk wants to merge 30 commits from merge-all-branches into development
pull from: merge-all-branches
merge into: geomind_code:development
Conversation 1 Commits 30 Files changed 105 +10423 -6355
despiegk commented 2026-03-04 06:08:34 +00:00
Owner
Copy link
No description provided. 
- Remove code_from_herolib/ directory (legacy herolib integration)
- Remove duplicate 'README MyFS.md' (consolidated into README.md)
- Add FEATURE_VERIFICATION.md (feature testing guide)
- Sync frontend version to 0.2.0 to match backend
Major changes:
- ServerStore now properly uses client-provided hash for block storage
- Server block upload API now accepts optional hash, file_hash, and idx params
- Added upload_block_simple() for simplified blob storage without file tracking
- Unified block size to 512KB across pack.rs, upload.rs, and server
- Added DEFAULT_BLOCK_SIZE constant in lib.rs
- Simplified publish_website() to use upload_dir() internally
- Added documentation clarifying encrypted (pack) vs unencrypted (upload) paths

Architecture:
- Server is now a pure key-value blob store (zero-knowledge capable)
- Client handles all encryption/decryption via BlockStore wrapper
- File structure tracked only in client-side flist metadata
- Two clear modes: encrypted pack/unpack vs simple upload/download
- Remove --create-flist flag from upload-dir command (always create flist)
- Simplify upload_dir() to always use pack() with encryption
- Remove unused collect_files() function
- Update publish_website() to delegate to upload_dir()
- This ensures all uploaded directories have proper metadata and encryption
New test files:
- tests/core_tests.rs: 17 tests covering encryption, deduplication,
  pack/unpack edge cases (symlinks, permissions, special filenames,
  large files, empty dirs, nested dirs), DirStore, Router
- tests/cli_tests.rs: 12 tests covering CLI help commands, error
  handling, and integration roundtrips (binary, many files, deep dirs)
- tests/server_api_tests.rs: 8 tests (4 run, 4 ignored requiring server)
  covering error handling for corrupt blocks/flists/truncated data

ACCEPTANCE_MATRIX.md: Documents all test cases organized by category
with current coverage status (~48% overall, ~85% for unit tests)

Test coverage summary:
- Encryption & Security: 100% (5/5)
- Core Operations: 92% (12/13)
- CLI Commands: 100% (8/8)
- DirStore: 75% (3/4)
- Router: 67% (2/3)
- Edge Cases: 43% (3/7)
- Integration: 60% (6/10)

Total: 62 tests, 56 passing, 6 ignored (need external services)
Provides targets for:
- Build: build, build-release, lib, check, clean
- Test: test, test-all, test-core, test-cli, test-server, test-verbose
- Code Quality: fmt, fmt-check, lint, lint-fix
- Documentation: doc, doc-open
- Development: install, run-server, watch, watch-test
- CI/CD: ci, validate, release, dist
- Examples: pack-example, unpack-example
- Docker: docker-build, docker-run

Run 'make help' for full command list.
Split the monolithic myfs crate into a workspace as per ADR-002:

- myfs-core: Shared library with store backends, fungi metadata,
  cache, error handling, pack/unpack, upload/download, and utilities
- myfs-server: Standalone HTTP server binary with REST API and JWT auth
- myfstool: Advanced tools binary (container, clone, merge, sync)
- myfs: FUSE mount binary with core flist operations

This separation provides clearer ownership, smaller binaries,
faster builds, and independent versioning for each component.

All 21 tests pass and clippy is clean.
Implement testing strategy with 176 tests covering:
- myfs-core: server_api, upload, download, cache, store/dir (88 tests)
- myfs-server: auth, sqlite db, block_handlers (53 tests)
- myfstool: clone, merge, sync operations (35 tests)

Add shared test utilities in myfs-core/src/testutils.rs with InMemoryStore
and helper functions. Add integration tests for server API and pack/unpack
round-trips. Add wiremock and tokio-test dev-dependencies for HTTP mocking.
New test file: tests/container_mount_tests.rs (10 tests)
- Container tests (4): alpine conversion, filesystem structure,
  CLI help, invalid image handling
- Mount tests (5): CLI help, invalid flist, invalid mountpoint,
  container flist unpack, full mount cycle (ignored - needs FUSE)
- Integration test (1): full container flow (busybox → flist → unpack)

Updated ACCEPTANCE_MATRIX.md:
- Added Container Conversion section (K1-K5): 100% coverage
- Added FUSE Mount section (U1-U7): 43% coverage (rest needs FUSE)
- Extended CLI section with container/mount help tests
- Overall coverage: ~53% (50 of 96 test cases)
- Unit tests without external deps: ~90% coverage

Updated Makefile:
- Added test-container target (requires podman)
- Added test-mount target
- Updated help text
Container runtime detection:
- Added ContainerRuntime enum with Podman, Docker, Nerdctl variants
- ContainerRuntime::detect() finds first available runtime
- ContainerRuntime::is_available() checks if runtime is functional
- ContainerImageToFlist::with_runtime() allows specifying runtime
- ContainerImageToFlist::new() auto-detects runtime

Updated all container functions to use runtime parameter:
- pull_image(), create_container(), export_container()
- clean_container(), clean_image(), write_startup_toml()

Updated tests:
- is_container_runtime_available() replaces is_podman_available()
- Tests print which runtime is being used
- container_test.rs uses with_runtime() for explicit runtime

This allows users with docker or nerdctl (but not podman) to use
container-to-flist conversion.
docs: Add system requirements for FUSE mount and debug builds
Some checks failed
Unit and Integration Test / test (push) Failing after 20s
Details
8ced03684b 
- Add System Requirements section to README with:
  - Stack size requirements for debug builds (ulimit -s unlimited)
  - FUSE mount prerequisites (fuse3 package, user_allow_other config)
- Add module-level documentation in main.rs with requirements
- Update Mount command help text to mention FUSE requirements
- Add helpful error context in fs/mod.rs for mount failures

The FUSE on-demand filesystem requires:
1. fuse3 package installed
2. /etc/fuse.conf with 'user_allow_other' uncommented
3. Debug builds need unlimited stack size due to async runtime overhead
docs: Add system requirements for FUSE mount and debug builds
All checks were successful
Unit and Integration Test / test (push) Successful in 5m9s
Details
62495991f5 
- Add System Requirements section to README with:
  - Stack size requirements for debug builds (ulimit -s unlimited)
  - FUSE mount prerequisites (fuse3 package, user_allow_other config)
- Add module-level documentation in main.rs with requirements
- Update Mount command help text to mention FUSE requirements
- Add helpful error context in fs/mod.rs for mount failures

The FUSE on-demand filesystem requires:
1. fuse3 package installed
2. /etc/fuse.conf with 'user_allow_other' uncommented
3. Debug builds need unlimited stack size due to async runtime overhead
Update README to reflect current workspace structure
Some checks failed
Unit and Integration Test / test (push) Failing after 20s
Details
2721004c92
 
- Document 4-crate workspace architecture (myfs-core, myfs-server, myfs, myfstool)
- Update command reference for myfs and myfstool CLIs
- Add accurate server configuration example with all required fields
- Document storage backends with URL formats
- Add project structure overview
- Remove outdated README MyFS.md (contained incorrect command references)
update with parallel unpacking and refine the tests
Some checks failed
Unit and Integration Test / test (push) Failing after 20s
Details
64a335e28f
fix fmt and tests
All checks were successful
Unit and Integration Test / test (push) Successful in 4m50s
Details
765f6587e9
Add support to buildah
Some checks failed
Unit and Integration Test / test (push) Failing after 21s
Details
d7dcf6bc58
chore: migrate polyfuse source and comprehensively realign docs to current CLI/server
Some checks failed
Unit and Integration Test / test (push) Failing after 6s
Details
3e1c2fec34
 
This commit consolidates a full documentation and build-tooling cleanup while
also switching the myfs FUSE dependency to the requested upstream branch.

Dependency/tooling updates:
- Switch myfs polyfuse dependency from
  https://github.com/muhamadazmy/polyfuse (master)
  to ssh://git@github.com/ubnt-intrepid/polyfuse.git (0.5.x)
- Refresh Cargo.lock to the new polyfuse/polyfuse-kernel revisions and
  related transitive dependency graph changes
- Add a root Makefile with practical release workflows:
  - build / test
  - build-musl
  - strip-binaries
  - upx-binaries (upx --best)
  - release-musl pipeline and dist output handling

Documentation realignment (broad sweep):
- Correct server startup docs from legacy myfs server/config-path usage to
  current myfs-server --config flow
- Align command naming to implemented CLI surface:
  - myfs flist-create (not myfs flist create)
  - advanced operations under myfstool (container/clone/merge/sync)
  - remove stale references to removed myfs subcommands (docker/clone/server/
    website-publish/token/track* where no longer available)
- Update server API references to the current routes:
  - auth via /api/v1/signin
  - current /api/v1/* endpoint family
  - website serving via /website/{flist_hash}/...
- Rewrite/refresh high-drift docs so examples are now executable with current
  binaries and flags

Files updated include:
- root README and docs/README command summaries
- user guides (myfs-cli, fl-server, frontend, troubleshooting,
  performance-tuning)
- tutorials (server-setup, website-publishing, docker-conversion,
  end-to-end-docker-workflow)
- architecture and concepts pages
- zdb and legacy guidance pages
- ADR wording to explicitly label migration tables as historical mappings

Outcome:
- Docs now match actual binaries and command behavior
- release build/compression workflow is standardized in-repo
- dependency source change requested for polyfuse is fully applied
follow consistent error structure across all crate modules
Some checks failed
Build and Test / build (push) Has been cancelled
Details
a08590f553
fix: update workers crate to hanging_myfs_fixes branch and adapt to new Work trait API
Some checks failed
Unit and Integration Test / test (push) Failing after 8s
Details
49e8f39e35
 
Switch tokio-worker-pool dependency to the hanging_myfs_fixes branch which
removes async_trait from the Work trait in favor of impl Future return types.
Adapt Uploader (pack.rs) and BlobCloner (clone.rs) implementations accordingly,
and handle the new Option return from WorkerPool::get().
refactor: remove async-trait crate, upgrade bb8-redis to 0.26
Some checks failed
Unit and Integration Test / test (push) Failing after 25s
Details
590f250557
 
Remove all async_trait usage across the codebase by leveraging native
async fn in traits (stable since Rust 1.75). The Store and WalkVisitor
traits now use desugared return types with explicit Send bounds.

Upgrade bb8-redis 0.13 → 0.26 (bb8 0.8 → 0.9, redis 0.23 → 1.0) to
eliminate bb8's async_trait dependency. Rework zdb.rs for redis 1.0 API
changes (private ConnectionInfo fields, MultiplexedConnection type).
(cherry-picked from development 8383c6f)
(cherry-picked from development ed9e617)
Replace manual workflow scripts with build_lib pattern: buildenv.sh for
config, scripts/build_lib.sh for shared logic, Makefile for dev targets.
Add Linux ARM64 cross-compilation and macOS build workflows.
Fix Makefile cargo and shell compatibility.

(cherry-picked from development a0f9fdd + 289cf64)
(cherry-picked from development 0d47b7d)
(cherry-picked from development abbc34b)
(cherry-picked from development 453c6d4)
(cherry-picked from development 7f81d77)
(cherry-picked from development a9f84b5)
Generate unique random nonces for each AES-GCM encryption instead of
deriving from the key. Add nonce column to block schema with backward
compatibility for old flists.

(cherry-picked from development 2e699fb)
(cherry-picked from development cdd4319)
Switch from std::sync::Mutex to tokio::sync::Mutex for async-safe
locking of jobs_state and flists_progress.

(cherry-picked from development 1450ed8)
fix: update tests for nonce and atomic write changes
Some checks failed
Build and Test / build (push) Failing after 7s
Details
be8ced9c3c
 
Add nonce parameters to meta.block() test calls and Block struct
initializer. Add Writer::finalize() calls before Reader creation
in all meta.rs tests to match the atomic write pattern.
fix: remove build-binary feature flag from build system
Some checks failed
Build and Test / build (push) Failing after 10s
Details
c0846b40d1
 
The workspace restructuring eliminated the need for the build-binary
feature gate — binary targets are now separate crates. Update Makefile,
buildenv.sh, and build_lib.sh accordingly.
fix: use HTTPS URL for polyfuse dependency
All checks were successful
Build and Test / build (push) Successful in 2m52s
Details
37636e1618
 
SSH git URL fails in CI runners without SSH keys configured.
Switch to HTTPS which works without authentication.
fix parallel functions and tests
All checks were successful
Build and Test / build (push) Successful in 4m45s
Details
3870c30109 
# Conflicts:
#	src/server/auth.rs
# Conflicts:
#	Cargo.lock
#	src/lib.rs
# Conflicts:
#	src/container.rs
#	src/main.rs
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	Makefile
#	README.md
#	docs/tutorials/docker-conversion.md
#	docs/user-guides/myfs-cli.md
#	myfs-core/src/bin/rhai_runner.rs
#	myfs-core/src/pack.rs
#	myfs-core/src/rhai_bindings.rs
#	myfs-server/src/server/config.rs
#	myfs-server/src/server/db/map.rs
#	myfs-server/src/server/db/mod.rs
#	myfs-server/src/server/db/sqlite.rs
#	myfs-server/src/server/handlers.rs
#	myfs-server/src/server/response.rs
#	myfs-server/src/server/serve_flists.rs
#	myfs-server/src/server/website_handlers.rs
#	myfs/src/main.rs
#	src/cache/mod.rs
#	src/lib.rs
#	src/merge.rs
#	src/server/auth.rs
#	src/server_api.rs
# Conflicts:
#	.forgejo/workflows/build.yaml
#	Cargo.toml
#	Makefile
#	buildenv.sh
#	myfs-core/src/container.rs
#	myfs-core/src/error.rs
#	myfs-core/src/fungi/meta.rs
#	myfs-core/src/pack.rs
#	myfs-server/src/server/config.rs
#	myfs-server/src/server/handlers.rs
#	myfs-server/src/server/mod.rs
#	myfs-server/src/server/website_handlers.rs
#	scripts/build_lib.sh
fix: list all workspace binaries in buildenv.sh
Some checks failed
Build and Test / build (push) Failing after 0s
Details
71ac31205d 
Include myfs-server and myfstool alongside myfs so the build system
installs all three binaries produced by the workspace.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
thabeta commented 2026-03-04 12:54:35 +00:00
Owner
Copy link

merged all of the the rest, only 2 branches from Jan to be resolved

merged all of the the rest, only 2 branches from Jan to be resolved
thabeta closed this pull request 2026-03-04 12:54:35 +00:00
thabeta deleted branch merge-all-branches 2026-03-04 12:54:56 +00:00
Some checks failed
Build and Test / build (push) Failing after 0s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
geomind_code/my_fs!28
No description provided.