feature/workspace-restructuring-worker-pool-fixes #27

Open

delandtj wants to merge 20 commits from feature/workspace-restructuring-worker-pool-fixes into main

pull from: feature/workspace-restructuring-worker-pool-fixes

merge into: geomind_code:main

geomind_code:development

geomind_code:main

geomind_code:feature/workspace-restructuring

Conversation 1 Commits 20 Files changed 102 +10687 -5918

delandtj commented 2026-02-26 20:58:54 +00:00

Owner

Copy link

Summary

• Restructure the codebase from a flat src/ layout into a Cargo workspace with 4 crates: myfs-core, myfs-server, myfs (FUSE mount), and myfstool (CLI utilities)
• Remove async-trait crate entirely, using native Rust async fn in traits (requires Rust 1.75+)
• Upgrade bb8-redis 0.13 → 0.26 (bb8 0.8 → 0.9, redis 0.23 → 1.0) to eliminate last async-trait dependency
• Update workers (tokio-worker-pool) to hanging_myfs_fixes branch with new Work trait API
• Port all commits from the development branch into the new workspace layout
• Add comprehensive test suite (174 tests across all crates)
• Add build system (Makefile, build_lib.sh, Forgejo workflows)

Ported from development

All meaningful commits from development have been cherry-picked and adapted to the workspace layout:

• Local image check before container conversion
• String entrypoint support
• Recursive tree traversal → iterative (stack overflow fix)
• Path traversal prevention in website_handlers
• Configurable server concurrency/timeouts
• Atomic metadata writes (temp file + rename pattern)
• Complete Error-to-Errno mapping for FUSE
• AES-GCM random nonce per block (security fix)
• Constant-time password comparison (timing attack fix)
• std::sync::Mutex → tokio::sync::Mutex for async safety

Not integrated

The development_consistent_bugs branch commit a08590f ("follow consistent error structure across all crate modules") is not included. This is a sweeping refactor that replaces
anyhow::Result with crate::error::Result across 19 files. It conflicts with the workspace restructuring and the error/errno changes already ported here. It should be re-applied on top of
the workspace layout after this PR lands.

Post-merge

After merging, the development branch can be reset to main — all its changes are already here. Development continues from the workspace layout going forward.

Test plan

• cargo test — 174 tests pass (86 core + 53 server + 35 tool)
• cargo check / cargo clippy clean
• Manual smoke test of FUSE mount + server

Summary - Restructure the codebase from a flat src/ layout into a Cargo workspace with 4 crates: myfs-core, myfs-server, myfs (FUSE mount), and myfstool (CLI utilities) - Remove async-trait crate entirely, using native Rust async fn in traits (requires Rust 1.75+) - Upgrade bb8-redis 0.13 → 0.26 (bb8 0.8 → 0.9, redis 0.23 → 1.0) to eliminate last async-trait dependency - Update workers (tokio-worker-pool) to hanging_myfs_fixes branch with new Work trait API - Port all commits from the development branch into the new workspace layout - Add comprehensive test suite (174 tests across all crates) - Add build system (Makefile, build_lib.sh, Forgejo workflows) Ported from development All meaningful commits from development have been cherry-picked and adapted to the workspace layout: - Local image check before container conversion - String entrypoint support - Recursive tree traversal → iterative (stack overflow fix) - Path traversal prevention in website_handlers - Configurable server concurrency/timeouts - Atomic metadata writes (temp file + rename pattern) - Complete Error-to-Errno mapping for FUSE - AES-GCM random nonce per block (security fix) - Constant-time password comparison (timing attack fix) - std::sync::Mutex → tokio::sync::Mutex for async safety Not integrated The development_consistent_bugs branch commit a08590f ("follow consistent error structure across all crate modules") is not included. This is a sweeping refactor that replaces anyhow::Result with crate::error::Result across 19 files. It conflicts with the workspace restructuring and the error/errno changes already ported here. It should be re-applied on top of the workspace layout after this PR lands. Post-merge After merging, the development branch can be reset to main — all its changes are already here. Development continues from the workspace layout going forward. Test plan - cargo test — 174 tests pass (86 core + 53 server + 35 tool) - cargo check / cargo clippy clean - Manual smoke test of FUSE mount + server

delandtj added 18 commits 2026-02-26 20:58:55 +00:00

Restructure codebase into Cargo workspace with 4 crates ... 0a9aea0a65

Split the monolithic myfs crate into a workspace as per ADR-002:

- myfs-core: Shared library with store backends, fungi metadata,
  cache, error handling, pack/unpack, upload/download, and utilities
- myfs-server: Standalone HTTP server binary with REST API and JWT auth
- myfstool: Advanced tools binary (container, clone, merge, sync)
- myfs: FUSE mount binary with core flist operations

This separation provides clearer ownership, smaller binaries,
faster builds, and independent versioning for each component.

All 21 tests pass and clippy is clean.

Add comprehensive test suite across all workspace crates ... 1644747a99

Implement testing strategy with 176 tests covering:
- myfs-core: server_api, upload, download, cache, store/dir (88 tests)
- myfs-server: auth, sqlite db, block_handlers (53 tests)
- myfstool: clone, merge, sync operations (35 tests)

Add shared test utilities in myfs-core/src/testutils.rs with InMemoryStore
and helper functions. Add integration tests for server API and pack/unpack
round-trips. Add wiremock and tokio-test dev-dependencies for HTTP mocking.

Update README to reflect current workspace structure ... 2721004c92

- Document 4-crate workspace architecture (myfs-core, myfs-server, myfs, myfstool)
- Update command reference for myfs and myfstool CLIs
- Add accurate server configuration example with all required fields
- Document storage backends with URL formats
- Add project structure overview
- Remove outdated README MyFS.md (contained incorrect command references)

chore: migrate polyfuse source and comprehensively realign docs to current CLI/server ... 3e1c2fec34

This commit consolidates a full documentation and build-tooling cleanup while
also switching the myfs FUSE dependency to the requested upstream branch.

Dependency/tooling updates:
- Switch myfs polyfuse dependency from
  https://github.com/muhamadazmy/polyfuse (master)
  to ssh://git@github.com/ubnt-intrepid/polyfuse.git (0.5.x)
- Refresh Cargo.lock to the new polyfuse/polyfuse-kernel revisions and
  related transitive dependency graph changes
- Add a root Makefile with practical release workflows:
  - build / test
  - build-musl
  - strip-binaries
  - upx-binaries (upx --best)
  - release-musl pipeline and dist output handling

Documentation realignment (broad sweep):
- Correct server startup docs from legacy myfs server/config-path usage to
  current myfs-server --config flow
- Align command naming to implemented CLI surface:
  - myfs flist-create (not myfs flist create)
  - advanced operations under myfstool (container/clone/merge/sync)
  - remove stale references to removed myfs subcommands (docker/clone/server/
    website-publish/token/track* where no longer available)
- Update server API references to the current routes:
  - auth via /api/v1/signin
  - current /api/v1/* endpoint family
  - website serving via /website/{flist_hash}/...
- Rewrite/refresh high-drift docs so examples are now executable with current
  binaries and flags

Files updated include:
- root README and docs/README command summaries
- user guides (myfs-cli, fl-server, frontend, troubleshooting,
  performance-tuning)
- tutorials (server-setup, website-publishing, docker-conversion,
  end-to-end-docker-workflow)
- architecture and concepts pages
- zdb and legacy guidance pages
- ADR wording to explicitly label migration tables as historical mappings

Outcome:
- Docs now match actual binaries and command behavior
- release build/compression workflow is standardized in-repo
- dependency source change requested for polyfuse is fully applied

fix: update workers crate to hanging_myfs_fixes branch and adapt to new Work trait API ... 49e8f39e35

Switch tokio-worker-pool dependency to the hanging_myfs_fixes branch which
removes async_trait from the Work trait in favor of impl Future return types.
Adapt Uploader (pack.rs) and BlobCloner (clone.rs) implementations accordingly,
and handle the new Option return from WorkerPool::get().

refactor: remove async-trait crate, upgrade bb8-redis to 0.26 ... 590f250557

Remove all async_trait usage across the codebase by leveraging native
async fn in traits (stable since Rust 1.75). The Store and WalkVisitor
traits now use desugared return types with explicit Send bounds.

Upgrade bb8-redis 0.13 → 0.26 (bb8 0.8 → 0.9, redis 0.23 → 1.0) to
eliminate bb8's async_trait dependency. Rework zdb.rs for redis 1.0 API
changes (private ConnectionInfo fields, MultiplexedConnection type).

check local images before conversion in myfs ... c581f8c601

(cherry-picked from development 8383c6f)

Allow string entrypoint ... dd30d01a49

(cherry-picked from development ed9e617)

Add build_lib build system with Makefile and updated workflows ... 572e967f47

Replace manual workflow scripts with build_lib pattern: buildenv.sh for
config, scripts/build_lib.sh for shared logic, Makefile for dev targets.
Add Linux ARM64 cross-compilation and macOS build workflows.
Fix Makefile cargo and shell compatibility.

(cherry-picked from development a0f9fdd + 289cf64)

[BUG] Refactor recursive tree traversal to avoid Stack Overflow ... ec2c3177d3

(cherry-picked from development 0d47b7d)

[SECURITY] Prevent Path Traversal in website_handlers ... 27ef8a1f55

(cherry-picked from development abbc34b)

[ENHANCEMENT] Make server concurrency/timeouts configurable ... b37998c427

(cherry-picked from development 453c6d4)

[ROBUSTNESS] Implement atomic metadata writes (Write-Rename pattern) ... 769ca9082d

(cherry-picked from development 7f81d77)

[FUSE] Complete Error to Errno mapping for filesystem mounts ... 572563d071

(cherry-picked from development a9f84b5)

[SECURITY] Insecure AES-GCM nonce usage in website_handlers ... ae198508d0

Generate unique random nonces for each AES-GCM encryption instead of
deriving from the key. Add nonce column to block schema with backward
compatibility for old flists.

(cherry-picked from development 2e699fb)

[SECURITY] Timing Attack Vulnerability in Authentication ... d1cbfdd218

(cherry-picked from development cdd4319)

[RELIABILITY] Unbounded Temporary File Creation in website_handlers ... 62d76134b5

Switch from std::sync::Mutex to tokio::sync::Mutex for async-safe
locking of jobs_state and flists_progress.

(cherry-picked from development 1450ed8)

fix: update tests for nonce and atomic write changes ... be8ced9c3c

Add nonce parameters to meta.block() test calls and Block struct
initializer. Add Writer::finalize() calls before Reader creation
in all meta.rs tests to match the atomic write pattern.

delandtj added 1 commit 2026-02-26 21:01:24 +00:00

fix: remove build-binary feature flag from build system ... c0846b40d1

The workspace restructuring eliminated the need for the build-binary
feature gate — binary targets are now separate crates. Update Makefile,
buildenv.sh, and build_lib.sh accordingly.

delandtj added 1 commit 2026-02-26 21:03:43 +00:00

fix: use HTTPS URL for polyfuse dependency ... 37636e1618

SSH git URL fails in CI runners without SSH keys configured.
Switch to HTTPS which works without authentication.

rawdaGastan commented 2026-03-04 11:48:03 +00:00

Member

Copy link

Can you please solve conflicts?

Can you please solve conflicts?

All checks were successful

Hide all checks

Build and Test / build (push) Successful in 2m52s

Details

This pull request has changes conflicting with the target branch.

• .forgejo/workflows/build.yaml
• Cargo.lock
• Cargo.toml
• Makefile
• README.md
• buildenv.sh
• docs/user-guides/myfs-cli.md
• myfs-core/src/bin/rhai_runner.rs
• myfs-core/src/container.rs
• myfs-core/src/error.rs
• myfs-core/src/fungi/meta.rs
• myfs-core/src/pack.rs
• myfs-core/src/rhai_bindings.rs
• myfs-server/src/server/config.rs
• myfs-server/src/server/db/map.rs
• myfs-server/src/server/db/mod.rs
• myfs-server/src/server/db/sqlite.rs
• myfs-server/src/server/handlers.rs
• myfs-server/src/server/mod.rs
• myfs-server/src/server/response.rs
• myfs-server/src/server/serve_flists.rs
• myfs-server/src/server/website_handlers.rs
• myfs/src/main.rs
• scripts/build_lib.sh
• src/cache/mod.rs
• src/lib.rs
• src/merge.rs
• src/server/auth.rs
• src/server_api.rs

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feature/workspace-restructuring-worker-pool-fixes:feature/workspace-restructuring-worker-pool-fixes

git switch feature/workspace-restructuring-worker-pool-fixes

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

geomind_code/my_fs!27

No description provided.