sal/MONOREPO_CONVERSION_PLAN.md

# SAL Monorepo Conversion Plan

## 🎯 **Objective**

Convert the SAL (System Abstraction Layer) project from a single-crate structure with modules in `src/` to a proper Rust monorepo with independent packages, following Rust best practices for workspace management.

## 📊 **Current State Analysis**

### Current Structure
```
sal/
├── Cargo.toml (single package + workspace with vault, git)
├── src/
│   ├── lib.rs (main library)
│   ├── bin/herodo.rs (binary)
│   ├── mycelium/ (module)
│   ├── net/ (module)
│   ├── os/ (module)
│   ├── postgresclient/ (module)
│   ├── process/ (module)
│   ├── redisclient/ (module)
│   ├── rhai/ (module - depends on ALL others, now imports git from sal-git)
│   ├── text/ (module)
│   ├── vault/ (module)
│   ├── virt/ (module)
│   └── zinit_client/ (module)
├── vault/ (converted package)
├── git/ (converted package) ✅ COMPLETED
```

### Issues with Current Structure
1. **Monolithic dependencies**: All external crates are listed in root Cargo.toml even if only used by specific modules
2. **Tight coupling**: All modules are compiled together, making it hard to use individual components
3. **Testing complexity**: Cannot test individual packages in isolation
4. **Version management**: Cannot version packages independently
5. **Build inefficiency**: Changes to one module trigger rebuilds of entire crate

## 🏗️ **Target Architecture**

### Final Monorepo Structure
```
sal/
├── Cargo.toml (workspace only)
├── git/           (sal-git package)
├── mycelium/      (sal-mycelium package)
├── net/           (sal-net package)
├── os/            (sal-os package)
├── postgresclient/ (sal-postgresclient package)
├── process/       (sal-process package)
├── redisclient/   (sal-redisclient package)
├── text/          (sal-text package)
├── vault/         (sal-vault package) ✅ already done
├── virt/          (sal-virt package)
├── zinit_client/  (sal-zinit-client package)
├── rhai/          (sal-rhai package - aggregates all others)
└── herodo/        (herodo binary package)
```

## 📋 **Detailed Conversion Plan**

### Phase 1: Analysis & Dependency Mapping
- [x] **Analyze each package's source code for dependencies**
  - Examine imports and usage in each src/ package
  - Identify external crates actually used by each module
- [x] **Map inter-package dependencies**
  - Identify which packages depend on other packages within the project
- [x] **Identify shared vs package-specific dependencies**
  - Categorize dependencies as common across packages or specific to individual packages
- [x] **Create dependency tree and conversion order**
  - Determine the order for converting packages based on their dependency relationships

### Phase 2: Package Structure Design
- [x] **Design workspace structure**
  - Keep packages at root level (not in src/ or crates/ subdirectory)
  - Follow Rust monorepo best practices
- [x] **Plan individual package Cargo.toml structure**
  - Design template for individual package Cargo.toml files
  - Include proper metadata (name, version, description, etc.)
- [x] **Handle version management strategy**
  - Use unified versioning (0.1.0) across all packages initially
  - Plan for independent versioning in the future
- [x] **Plan rhai module handling**
  - The rhai module depends on ALL other packages
  - Convert it last as an aggregation package

### Phase 3: Incremental Package Conversion
Convert packages in dependency order (leaf packages first):

#### 3.1 Leaf Packages (no internal dependencies)
- [x] **redisclient** → sal-redisclient
- [x] **text** → sal-text  
- [x] **mycelium** → sal-mycelium
- [x] **net** → sal-net
- [x] **os** → sal-os

#### 3.2 Mid-level Packages (depend on leaf packages)
- [x] **git** → sal-git (depends on redisclient) ✅ **COMPLETED WITH FULL INTEGRATION**
  - ✅ Independent package with comprehensive test suite (27 tests)
  - ✅ Rhai integration moved to git package
  - ✅ Circular dependency resolved (direct redis client implementation)
  - ✅ Old src/git/ removed and references updated
  - ✅ Test infrastructure moved to git/tests/rhai/
- [x] **process** → sal-process (depends on text)
- [x] **zinit_client** → sal-zinit-client

#### 3.3 Higher-level Packages
- [x] **virt** → sal-virt (depends on process, os)
- [x] **postgresclient** → sal-postgresclient (depends on virt)

#### 3.4 Aggregation Package
- [ ] **rhai** → sal-rhai (depends on ALL other packages)

#### 3.5 Binary Package
- [ ] **herodo** → herodo (binary package)

### Phase 4: Cleanup & Validation
- [ ] **Clean up root Cargo.toml**
  - Remove old dependencies that are now in individual packages
  - Keep only workspace configuration
- [ ] **Remove old src/ modules**
  - After confirming all packages work independently
- [ ] **Update documentation**
  - Update README.md with new structure
  - Update examples to use new package structure
- [ ] **Validate builds**
  - Ensure all packages build independently
  - Ensure workspace builds successfully
  - Run all tests

## 🔧 **Implementation Strategy**

### Package Conversion Template
For each package conversion:

1. **Create package directory** (e.g., `git/`)
2. **Create Cargo.toml** with:
   ```toml
   [package]
   name = "sal-{package}"
   version = "0.1.0"
   edition = "2021"
   authors = ["PlanetFirst <info@incubaid.com>"]
   description = "SAL {Package} - {description}"
   repository = "https://git.threefold.info/herocode/sal"
   license = "Apache-2.0"

   [dependencies]
   # Only dependencies actually used by this package
   ```
3. **Move source files** from `src/{package}/` to `{package}/src/`
4. **Update imports** in moved files
5. **Add to workspace** in root Cargo.toml
6. **Test package** builds independently
7. **Update dependent packages** to use new package

### Advanced Package Conversion (Git Package Example)
For packages with Rhai integration and complex dependencies:

1. **Handle Rhai Integration**:
   - Move rhai wrappers from `src/rhai/{package}.rs` to `{package}/src/rhai.rs`
   - Add rhai dependency to package Cargo.toml
   - Update main SAL rhai module to import from new package
   - Export rhai module from package lib.rs

2. **Resolve Circular Dependencies**:
   - Identify circular dependency patterns (e.g., package → sal → redisclient)
   - Implement direct dependencies or minimal client implementations
   - Remove dependency on main sal crate where possible

3. **Comprehensive Testing**:
   - Create `{package}/tests/` directory with separate test files
   - Keep source files clean (no inline tests)
   - Add both Rust unit tests and Rhai integration tests
   - Move package-specific rhai script tests to `{package}/tests/rhai/`

4. **Update Test Infrastructure**:
   - Update `run_rhai_tests.sh` to find tests in new locations
   - Update documentation to reflect new test paths
   - Ensure both old and new test locations are supported during transition

5. **Clean Migration**:
   - Remove old `src/{package}/` directory completely
   - Remove package-specific tests from main SAL test files
   - Update all import references in main SAL crate
   - Verify no broken references remain

### Dependency Management Rules
- **Minimize dependencies**: Only include crates actually used by each package
- **Use workspace dependencies**: For common dependencies, consider workspace-level dependency management
- **Version consistency**: Keep versions consistent across packages for shared dependencies

## 🧪 **Testing Strategy**

### Package-level Testing
- **Rust Unit Tests**: Each package should have tests in `{package}/tests/` directory
  - Keep source files clean (no inline `#[cfg(test)]` modules)
  - Separate test files for different modules (e.g., `git_tests.rs`, `git_executor_tests.rs`)
  - Tests should be runnable independently: `cd {package} && cargo test`
- **Rhai Integration Tests**: For packages with rhai wrappers
  - Rust tests for rhai function registration in `{package}/tests/rhai_tests.rs`
  - Rhai script tests in `{package}/tests/rhai/` directory
  - Include comprehensive test runner scripts

### Integration Testing
- Workspace-level tests for cross-package functionality
- **Test Infrastructure Updates**:
  - Update `run_rhai_tests.sh` to support both old (`rhai_tests/`) and new (`{package}/tests/rhai/`) locations
  - Ensure smooth transition during conversion process
- **Documentation Updates**: Update test documentation to reflect new paths

### Validation Checklist
- [ ] Each package builds independently
- [ ] All packages build together in workspace
- [ ] All existing tests pass
- [ ] Examples work with new structure
- [ ] herodo binary still works
- [ ] Rhai integration works for converted packages
- [ ] Test infrastructure supports new package locations
- [ ] No circular dependencies exist
- [ ] Old source directories completely removed
- [ ] Documentation updated for new structure

## 🚨 **Risk Mitigation**

### Potential Issues
1. **Circular dependencies**: Carefully analyze dependencies to avoid cycles
2. **Feature flags**: Some packages might need conditional compilation
3. **External git dependencies**: Handle external dependencies like kvstore
4. **Build performance**: Monitor build times after conversion

### Rollback Plan
- Keep original src/ structure until full validation
- Use git branches for incremental changes
- Test each phase thoroughly before proceeding

## 📚 **Lessons Learned (Git Package Conversion)**

### Key Insights from Git Package Implementation
1. **Rhai Integration Complexity**: Moving rhai wrappers to individual packages provides better cohesion but requires careful dependency management
2. **Circular Dependency Resolution**: Main SAL crate depending on packages that depend on SAL creates cycles - resolve by implementing direct dependencies
3. **Test Organization**: Separating tests into dedicated directories keeps source files clean and follows Rust best practices
4. **Infrastructure Updates**: Test runners and documentation need updates to support new package locations
5. **Comprehensive Validation**: Need both Rust unit tests AND rhai script tests to ensure full functionality

### Best Practices Established
- **Source File Purity**: Keep source files identical to original, move all tests to separate files
- **Comprehensive Test Coverage**: Include unit tests, integration tests, and rhai script tests
- **Dependency Minimization**: Implement minimal clients rather than depending on main crate
- **Smooth Transition**: Support both old and new test locations during conversion
- **Documentation Consistency**: Update all references to new package structure

## 📈 **Success Metrics**

- ✅ All packages build independently
- ✅ Workspace builds successfully  
- ✅ All tests pass
- ✅ Build times are reasonable or improved
- ✅ Individual packages can be used independently
- ✅ Clear separation of concerns between packages
- ✅ Proper dependency management (no unnecessary dependencies)