myfs

A flist-based filesystem toolkit and server with deduplicated, content-addressed storage. MyFS (formerly RFS) packages directories and container images into FungiList (flist) metadata, stores blocks in pluggable backends, and serves content over HTTP or a read-only FUSE mount.

Overview

MyFS provides:

• Flist metadata format: SQLite-backed metadata for files, directories, and block references
• Deduplicated, content-addressed storage: block-based storage with encryption and compression
• Pluggable stores: local directory, ZDB, S3, HTTP, and server-backed stores
• Cache + FUSE mount: on-demand file access with local caching
• Server API + web UI: manage blocks, files, flists, and websites
• Client utilities: pack/unpack, upload/download, sync, and tracking tools
• Container conversion: build flists from container images

Architecture

• Metadata (fungi): flist metadata read/write and traversal
• Store layer: router abstraction over multiple backends
• Block layer: encryption + compression wrapping stores
• Cache: local block cache for reads
• FUSE filesystem: read-only mount from flists
• Server: Axum REST API + web UI

Features

• Pack/unpack: create flists from directories and reconstruct data
• Mount: read-only FUSE mount for flists
• Inspect flists: tree and metadata inspection
• Clone/merge: clone stores or merge multiple flists
• Server APIs:
  • Blocks: upload, download, verify, list, track
  • Files: upload, download
  • Flists: create, list, preview, track creation jobs
  • Directories: browse flist directories via HTTP
  • Websites: serve static sites from flists
  • Auth: JWT-based sign-in
• Client workflows: upload/download files and directories, hash existence checks
• Sync: copy blocks or files between servers
• Container conversion: create flists from container images
• Website publishing: serve static sites from flists
• Tracking: block and website download tracking

System Requirements

Stack Size (Debug Builds)

Debug builds require a larger stack size due to async runtime and SQLx overhead. Set unlimited stack before running:

ulimit -s unlimited

For production use, always use release builds which don't have this requirement.

FUSE Mount Requirements

To use the FUSE mount feature, you need:

1. FUSE installed: fusermount3 (FUSE3) or fusermount (FUSE2)

   # Ubuntu/Debian
   sudo apt install fuse3
   
   # Fedora/RHEL
   sudo dnf install fuse3
   

2. Enable user_allow_other: Edit /etc/fuse.conf and uncomment or add:

   user_allow_other
   

   This allows the mount to be accessible to other users on the system.

Building

Option 1: Install from Release (Recommended)

Download the pre-built binary from the latest release:

# Download the latest release (replace v0.0.1 with the latest version)
wget https://forge.ourworld.tf/geomind_code/my_fs/releases/download/v0.0.1/myfs-linux-amd64-musl

# Make it executable
chmod +x myfs-linux-amd64-musl

# Move to system path
sudo mv myfs-linux-amd64-musl /usr/local/bin/myfs

# Verify installation
myfs --help

Option 2: Build from Source

Prerequisites

• Rust 1.92
• Linux (primary supported platform)

Linux Build

# Build the project in release mode with binary features
cargo build --release --features build-binary

# The binary will be available at:
./target/release/myfs

Development Build

# Build in debug mode (includes debug symbols)
cargo build --features build-binary

# Run tests
cargo test

Usage

CLI quickstart

# Pack a directory into a flist (and upload blocks)
myfs pack --meta ./rootfs.fl --store-spec <store-url> ./rootfs

# Unpack a flist
myfs unpack --meta ./rootfs.fl ./output

# Mount a flist (read-only FUSE)
myfs mount --meta ./rootfs.fl ./mnt

# Inspect flist contents
myfs flist tree <flist-path-or-hash>
myfs flist inspect <flist-path-or-hash>

# Convert a container image to flist
myfs container --image-name docker.io/library/nginx:latest --store <store-url>

# Publish a static website
myfs website-publish <dir> --server http://localhost:8080 --token <token>

# Upload or download files
myfs upload ./file.txt --server http://localhost:8080 --token <token>
myfs download <hash> --output ./file.txt --server http://localhost:8080

# Sync between servers
myfs sync --source http://localhost:8080 --destination http://localhost:8081 --token <token>

Server

# Start the server
myfs server --config-path ./server.toml

The server exposes REST APIs for blocks, files, flists, and websites via Axum/Tokio. Key endpoints:

• Flist management: POST /api/v1/fl (create), GET /api/v1/fl (list), GET /api/v1/fl/preview/:path (preview)
• Block storage: POST /api/v1/block (upload), GET /api/v1/block/:hash (download), HEAD /api/v1/block/:hash (check), POST /api/v1/block/verify (batch verify)
• File storage: POST /api/v1/file (upload), GET /api/v1/file/:hash (download)
• Directory browsing: GET /*path (browse flist directories, list files with progress)
• Website serving: GET /website/:website_hash/*path (serve static sites)
• Auth: POST /api/v1/signin (JWT token)

OpenAPI docs available at /swagger-ui.

Configuration

• Store specs: select backend, location, and options using store URLs.
• Server config: configure routes, auth, and storage in the server config file.

Future Roadmap

• Integrate RFS codebase from GitHub
• Performance optimizations for large images
• Enhanced deduplication algorithms