sal/MONOREPO_CONVERSION_PLAN.md

SAL Monorepo Conversion Plan

🎯 Objective

Convert the SAL (System Abstraction Layer) project from a single-crate structure with modules in src/ to a proper Rust monorepo with independent packages, following Rust best practices for workspace management.

📊 Current State Analysis

Current Structure

sal/
├── Cargo.toml (single package + workspace with vault, git)
├── src/
│   ├── lib.rs (main library)
│   ├── bin/herodo.rs (binary)
│   ├── mycelium/ (module)
│   ├── net/ (module)
│   ├── os/ (module)
│   ├── postgresclient/ (module)
│   ├── process/ (module)
│   ├── redisclient/ (module)
│   ├── rhai/ (module - depends on ALL others, now imports git from sal-git)
│   ├── text/ (module)
│   ├── vault/ (module)
│   ├── virt/ (module)
│   └── zinit_client/ (module)
├── vault/ (converted package)
├── git/ (converted package) ✅ COMPLETED

Issues with Current Structure

1. Monolithic dependencies: All external crates are listed in root Cargo.toml even if only used by specific modules
2. Tight coupling: All modules are compiled together, making it hard to use individual components
3. Testing complexity: Cannot test individual packages in isolation
4. Version management: Cannot version packages independently
5. Build inefficiency: Changes to one module trigger rebuilds of entire crate

🏗️ Target Architecture

Final Monorepo Structure

sal/
├── Cargo.toml (workspace only)
├── git/           (sal-git package)
├── mycelium/      (sal-mycelium package)
├── net/           (sal-net package)
├── os/            (sal-os package)
├── postgresclient/ (sal-postgresclient package)
├── process/       (sal-process package)
├── redisclient/   (sal-redisclient package)
├── text/          (sal-text package)
├── vault/         (sal-vault package) ✅ already done
├── virt/          (sal-virt package)
├── zinit_client/  (sal-zinit-client package)
├── rhai/          (sal-rhai package - aggregates all others)
└── herodo/        (herodo binary package)

📋 Detailed Conversion Plan

Phase 1: Analysis & Dependency Mapping

• Analyze each package's source code for dependencies
  • Examine imports and usage in each src/ package
  • Identify external crates actually used by each module
• Map inter-package dependencies
  • Identify which packages depend on other packages within the project
• Identify shared vs package-specific dependencies
  • Categorize dependencies as common across packages or specific to individual packages
• Create dependency tree and conversion order
  • Determine the order for converting packages based on their dependency relationships

Phase 2: Package Structure Design

• Design workspace structure
  • Keep packages at root level (not in src/ or crates/ subdirectory)
  • Follow Rust monorepo best practices
• Plan individual package Cargo.toml structure
  • Design template for individual package Cargo.toml files
  • Include proper metadata (name, version, description, etc.)
• Handle version management strategy
  • Use unified versioning (0.1.0) across all packages initially
  • Plan for independent versioning in the future
• Plan rhai module handling
  • The rhai module depends on ALL other packages
  • Convert it last as an aggregation package

Phase 3: Incremental Package Conversion

Convert packages in dependency order (leaf packages first):

3.1 Leaf Packages (no internal dependencies)

• redisclient → sal-redisclient
• text → sal-text
• mycelium → sal-mycelium
• net → sal-net
• os → sal-os

3.2 Mid-level Packages (depend on leaf packages)

• git → sal-git (depends on redisclient) ✅ COMPLETED WITH FULL INTEGRATION
  • ✅ Independent package with comprehensive test suite (27 tests)
  • ✅ Rhai integration moved to git package
  • ✅ Circular dependency resolved (direct redis client implementation)
  • ✅ Old src/git/ removed and references updated
  • ✅ Test infrastructure moved to git/tests/rhai/
• process → sal-process (depends on text)
• zinit_client → sal-zinit-client

3.3 Higher-level Packages

• virt → sal-virt (depends on process, os)
• postgresclient → sal-postgresclient (depends on virt)

3.4 Aggregation Package

• rhai → sal-rhai (depends on ALL other packages)

3.5 Binary Package

• herodo → herodo (binary package)

Phase 4: Cleanup & Validation

• Clean up root Cargo.toml
  • Remove old dependencies that are now in individual packages
  • Keep only workspace configuration
• Remove old src/ modules
  • After confirming all packages work independently
• Update documentation
  • Update README.md with new structure
  • Update examples to use new package structure
• Validate builds
  • Ensure all packages build independently
  • Ensure workspace builds successfully
  • Run all tests

🔧 Implementation Strategy

Package Conversion Template

For each package conversion:

1. Create package directory (e.g., git/)
2. Create Cargo.toml with:

   [package]
   name = "sal-{package}"
   version = "0.1.0"
   edition = "2021"
   authors = ["PlanetFirst <info@incubaid.com>"]
   description = "SAL {Package} - {description}"
   repository = "https://git.threefold.info/herocode/sal"
   license = "Apache-2.0"
   
   [dependencies]
   # Only dependencies actually used by this package
   

3. Move source files from src/{package}/ to {package}/src/
4. Update imports in moved files
5. Add to workspace in root Cargo.toml
6. Test package builds independently
7. Update dependent packages to use new package

Advanced Package Conversion (Git Package Example)

For packages with Rhai integration and complex dependencies:

1. Handle Rhai Integration:

   • Move rhai wrappers from src/rhai/{package}.rs to {package}/src/rhai.rs
   • Add rhai dependency to package Cargo.toml
   • Update main SAL rhai module to import from new package
   • Export rhai module from package lib.rs

2. Resolve Circular Dependencies:

   • Identify circular dependency patterns (e.g., package → sal → redisclient)
   • Implement direct dependencies or minimal client implementations
   • Remove dependency on main sal crate where possible

3. Comprehensive Testing:

   • Create {package}/tests/ directory with separate test files
   • Keep source files clean (no inline tests)
   • Add both Rust unit tests and Rhai integration tests
   • Move package-specific rhai script tests to {package}/tests/rhai/

4. Update Test Infrastructure:

   • Update run_rhai_tests.sh to find tests in new locations
   • Update documentation to reflect new test paths
   • Ensure both old and new test locations are supported during transition

5. Clean Migration:

   • Remove old src/{package}/ directory completely
   • Remove package-specific tests from main SAL test files
   • Update all import references in main SAL crate
   • Verify no broken references remain

Dependency Management Rules

• Minimize dependencies: Only include crates actually used by each package
• Use workspace dependencies: For common dependencies, consider workspace-level dependency management
• Version consistency: Keep versions consistent across packages for shared dependencies

🧪 Testing Strategy

Package-level Testing

• Rust Unit Tests: Each package should have tests in {package}/tests/ directory
  • Keep source files clean (no inline #[cfg(test)] modules)
  • Separate test files for different modules (e.g., git_tests.rs, git_executor_tests.rs)
  • Tests should be runnable independently: cd {package} && cargo test
• Rhai Integration Tests: For packages with rhai wrappers
  • Rust tests for rhai function registration in {package}/tests/rhai_tests.rs
  • Rhai script tests in {package}/tests/rhai/ directory
  • Include comprehensive test runner scripts

Integration Testing

• Workspace-level tests for cross-package functionality
• Test Infrastructure Updates:
  • Update run_rhai_tests.sh to support both old (rhai_tests/) and new ({package}/tests/rhai/) locations
  • Ensure smooth transition during conversion process
• Documentation Updates: Update test documentation to reflect new paths

Validation Checklist

• Each package builds independently
• All packages build together in workspace
• All existing tests pass
• Examples work with new structure
• herodo binary still works
• Rhai integration works for converted packages
• Test infrastructure supports new package locations
• No circular dependencies exist
• Old source directories completely removed
• Documentation updated for new structure

🚨 Risk Mitigation

Potential Issues

1. Circular dependencies: Carefully analyze dependencies to avoid cycles
2. Feature flags: Some packages might need conditional compilation
3. External git dependencies: Handle external dependencies like kvstore
4. Build performance: Monitor build times after conversion

Rollback Plan

• Keep original src/ structure until full validation
• Use git branches for incremental changes
• Test each phase thoroughly before proceeding

📚 Lessons Learned (Git Package Conversion)

Key Insights from Git Package Implementation

1. Rhai Integration Complexity: Moving rhai wrappers to individual packages provides better cohesion but requires careful dependency management
2. Circular Dependency Resolution: Main SAL crate depending on packages that depend on SAL creates cycles - resolve by implementing direct dependencies
3. Test Organization: Separating tests into dedicated directories keeps source files clean and follows Rust best practices
4. Infrastructure Updates: Test runners and documentation need updates to support new package locations
5. Comprehensive Validation: Need both Rust unit tests AND rhai script tests to ensure full functionality

Best Practices Established

• Source File Purity: Keep source files identical to original, move all tests to separate files
• Comprehensive Test Coverage: Include unit tests, integration tests, and rhai script tests
• Dependency Minimization: Implement minimal clients rather than depending on main crate
• Smooth Transition: Support both old and new test locations during conversion
• Documentation Consistency: Update all references to new package structure

📈 Success Metrics

• ✅ All packages build independently
• ✅ Workspace builds successfully
• ✅ All tests pass
• ✅ Build times are reasonable or improved
• ✅ Individual packages can be used independently
• ✅ Clear separation of concerns between packages
• ✅ Proper dependency management (no unnecessary dependencies)