sal/README.md

# SAL (System Abstraction Layer)

**Version: 0.1.0**

SAL is a comprehensive Rust library designed to provide a unified and simplified interface for a wide array of system-level operations and interactions. It abstracts platform-specific details, enabling developers to write robust, cross-platform code with greater ease. SAL also includes `herodo`, a powerful command-line tool for executing Rhai scripts that leverage SAL's capabilities for automation and system management tasks.

## 🏗️ **Cargo Workspace Structure**

SAL is organized as a **Cargo workspace** with 16 specialized crates:

- **Root Package**: `sal` - Umbrella crate that re-exports all modules
- **13 Library Crates**: Specialized SAL modules (git, text, os, net, etc.)
- **1 Binary Crate**: `herodo` - Rhai script execution engine
- **1 Integration Crate**: `rhai` - Rhai scripting integration layer

This workspace structure provides excellent build performance, dependency management, and maintainability.

### **🚀 Workspace Benefits**
- **Unified Dependency Management**: Shared dependencies across all crates with consistent versions
- **Optimized Build Performance**: Parallel compilation and shared build artifacts
- **Simplified Testing**: Run tests across all modules with a single command
- **Modular Architecture**: Each module is independently maintainable while sharing common infrastructure
- **Production Ready**: 100% test coverage with comprehensive Rhai integration tests

## 📦 Installation

SAL is designed to be modular - install only the components you need!

### Option 1: Individual Crates (Recommended)

Install only the modules you need:

```bash
# Core system operations
cargo add sal-os sal-process sal-text sal-net

# Database clients
cargo add sal-redisclient sal-postgresclient

# Infrastructure tools
cargo add sal-git sal-vault sal-kubernetes sal-virt

# Service clients
cargo add sal-zinit-client sal-mycelium

# Scripting support
cargo add sal-rhai
```

### Option 2: Meta-crate with Features

Use the main `sal` crate with specific features:

```bash
# Install specific modules
cargo add sal --features os,process,text

# Install feature groups
cargo add sal --features core              # os, process, text, net
cargo add sal --features clients           # redisclient, postgresclient, zinit_client, mycelium
cargo add sal --features infrastructure    # git, vault, kubernetes, virt
cargo add sal --features scripting         # rhai

# Install everything
cargo add sal --features all
```

### Quick Start Examples

#### Using Individual Crates (Recommended)

```rust
use sal_os::fs;
use sal_process::run;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // File system operations
    let files = fs::list_files(".")?;
    println!("Found {} files", files.len());

    // Process execution
    let result = run::command("echo hello")?;
    println!("Output: {}", result.stdout);

    Ok(())
}
```

#### Using Meta-crate with Features

```rust
// In Cargo.toml: sal = { version = "0.1.0", features = ["os", "process"] }
use sal::os::fs;
use sal::process::run;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // File system operations
    let files = fs::list_files(".")?;
    println!("Found {} files", files.len());

    // Process execution
    let result = run::command("echo hello")?;
    println!("Output: {}", result.stdout);

    Ok(())
}
```

#### Using Herodo for Scripting

```bash
# Build and install herodo
git clone https://github.com/PlanetFirst/sal.git
cd sal
./build_herodo.sh

# Create a script file
cat > example.rhai << 'EOF'
// File operations
let files = find_files(".", "*.rs");
print("Found " + files.len() + " Rust files");

// Process execution
let result = run("echo 'Hello from SAL!'");
print("Output: " + result.stdout);

// Network operations
let reachable = http_check("https://github.com");
print("GitHub reachable: " + reachable);
EOF

# Execute the script
herodo example.rhai
```

## 📦 Available Packages

SAL is published as individual crates, allowing you to install only what you need:

| Package | Description | Install Command |
|---------|-------------|-----------------|
| [`sal-os`](https://crates.io/crates/sal-os) | Operating system operations | `cargo add sal-os` |
| [`sal-process`](https://crates.io/crates/sal-process) | Process management | `cargo add sal-process` |
| [`sal-text`](https://crates.io/crates/sal-text) | Text processing utilities | `cargo add sal-text` |
| [`sal-net`](https://crates.io/crates/sal-net) | Network operations | `cargo add sal-net` |
| [`sal-git`](https://crates.io/crates/sal-git) | Git repository management | `cargo add sal-git` |
| [`sal-vault`](https://crates.io/crates/sal-vault) | Cryptographic operations | `cargo add sal-vault` |
| [`sal-kubernetes`](https://crates.io/crates/sal-kubernetes) | Kubernetes management | `cargo add sal-kubernetes` |
| [`sal-virt`](https://crates.io/crates/sal-virt) | Virtualization tools | `cargo add sal-virt` |
| `sal-redisclient` | Redis database client | `cargo add sal-redisclient` ⏳ |
| `sal-postgresclient` | PostgreSQL client | `cargo add sal-postgresclient` ⏳ |
| `sal-zinit-client` | Zinit process supervisor | `cargo add sal-zinit-client` ⏳ |
| `sal-mycelium` | Mycelium network client | `cargo add sal-mycelium` ⏳ |
| `sal-rhai` | Rhai scripting integration | `cargo add sal-rhai` ⏳ |
| `sal` | Meta-crate with features | `cargo add sal --features all` ⏳ |
| `herodo` | Script executor binary | Build from source ⏳ |

**Legend**: ✅ Published | ⏳ Publishing soon (rate limited)

### 📢 **Publishing Status**

**Currently Available on crates.io:**
- ✅ [`sal-os`](https://crates.io/crates/sal-os) - Operating system operations
- ✅ [`sal-text`](https://crates.io/crates/sal-text) - Text processing utilities
- ✅ [`sal-net`](https://crates.io/crates/sal-net) - Network operations
- ✅ [`sal-git`](https://crates.io/crates/sal-git) - Git repository management
- ✅ [`sal-vault`](https://crates.io/crates/sal-vault) - Cryptographic operations
- ✅ [`sal-kubernetes`](https://crates.io/crates/sal-kubernetes) - Kubernetes management

**Publishing Soon** (hit crates.io rate limit):
- ⏳ `sal-redisclient`, `sal-postgresclient`, `sal-zinit-client`, `sal-mycelium`
- ⏳ `sal-process`, `sal-virt`, `sal-rhai`
- ⏳ `sal` (meta-crate), `herodo` (binary)

**Estimated Timeline**: Remaining packages will be published within 24 hours once the rate limit resets.

## Core Features

SAL offers a broad spectrum of functionalities, including:

- **System Operations**: File and directory management, environment variable access, system information retrieval, and OS-specific commands.
- **Process Management**: Create, monitor, control, and interact with system processes.
- **Containerization Tools**: 
    - Integration with **Buildah** for building OCI/Docker-compatible container images.
    - Integration with **nerdctl** for managing containers (run, stop, list, build, etc.).
- **Version Control**: Programmatic interaction with Git repositories (clone, commit, push, pull, status, etc.).
- **Database Clients**:
    - **Redis**: Robust client for interacting with Redis servers.
    - **PostgreSQL**: Client for executing queries and managing PostgreSQL databases.
- **Scripting Engine**: In-built support for the **Rhai** scripting language, allowing SAL functionalities to be scripted and automated, primarily through the `herodo` tool.
- **Networking & Services**:
    - **Mycelium**: Tools for Mycelium network peer management and message passing.
    - **Zinit**: Client for interacting with the Zinit process supervision system.
    - **RFS (Remote/Virtual Filesystem)**: Mount, manage, pack, and unpack various types of filesystems (local, SSH, S3, WebDAV).
- **Text Processing**: A suite of utilities for text manipulation, formatting, and regular expressions.
- **Cryptography (`vault`)**: Functions for common cryptographic operations.

## `herodo`: The SAL Scripting Tool

`herodo` is a command-line utility bundled with SAL that executes Rhai scripts. It empowers users to automate tasks and orchestrate complex workflows by leveraging SAL's diverse modules directly from scripts.

### Usage

```bash
# Execute a single Rhai script
herodo script.rhai

# Execute a script with arguments
herodo script.rhai arg1 arg2

# Execute all .rhai scripts in a directory
herodo /path/to/scripts/
```

If a directory is provided, `herodo` will execute all `.rhai` scripts within that directory (and its subdirectories) in alphabetical order.

### Scriptable SAL Modules via `herodo`

The following SAL modules and functionalities are exposed to the Rhai scripting environment through `herodo`:

- **OS (`os`)**: Comprehensive file system operations, file downloading & installation, and system package management. [Documentation](os/README.md)
- **Process (`process`)**: Robust command and script execution, plus process management (listing, finding, killing, checking command existence). [Documentation](process/README.md)
- **Text (`text`)**: String manipulation, prefixing, path/name fixing, text replacement, and templating. [Documentation](text/README.md)
- **Net (`net`)**: Network operations, HTTP requests, and connectivity utilities. [Documentation](net/README.md)
- **Git (`git`)**: High-level repository management and generic Git command execution with Redis-backed authentication (clone, pull, push, commit, etc.). [Documentation](git/README.md)
- **Vault (`vault`)**: Cryptographic operations, keypair management, encryption, decryption, hashing, etc. [Documentation](vault/README.md)
- **Redis Client (`redisclient`)**: Execute Redis commands (`redis_get`, `redis_set`, `redis_execute`, etc.). [Documentation](redisclient/README.md)
- **PostgreSQL Client (`postgresclient`)**: Execute SQL queries against PostgreSQL databases. [Documentation](postgresclient/README.md)
- **Zinit (`zinit_client`)**: Client for Zinit process supervisor (service management, logs). [Documentation](zinit_client/README.md)
- **Mycelium (`mycelium`)**: Client for Mycelium decentralized networking API (node info, peer management, messaging). [Documentation](mycelium/README.md)
- **Virtualization (`virt`)**:
  - **Buildah**: OCI/Docker image building functions. [Documentation](virt/README.md)
  - **nerdctl**: Container lifecycle management (`nerdctl_run`, `nerdctl_stop`, `nerdctl_images`, `nerdctl_image_build`, etc.)
  - **RFS**: Mount various filesystems (local, SSH, S3, etc.), pack/unpack filesystem layers.

### Example `herodo` Rhai Script

```rhai
// file: /opt/scripts/example_task.rhai

// OS operations
println("Checking for /tmp/my_app_data...");
if !exist("/tmp/my_app_data") {
    mkdir("/tmp/my_app_data");
    println("Created directory /tmp/my_app_data");
}

// Redis operations
println("Setting Redis key 'app_status' to 'running'");
redis_set("app_status", "running");
let status = redis_get("app_status");
println("Current app_status from Redis: " + status);

// Process execution
println("Listing files in /tmp:");
let output = run("ls -la /tmp");
println(output.stdout);

println("Script finished.");
```

Run with: `herodo /opt/scripts/example_task.rhai`

For more examples, check the individual module test directories (e.g., `text/tests/rhai/`, `os/tests/rhai/`, etc.) in this repository.

## Using SAL as a Rust Library

Add SAL as a dependency to your `Cargo.toml`:

```toml
[dependencies]
sal = "0.1.0" # Or the latest version
```

### Rust Example: Using Redis Client

```rust
use sal::redisclient::{get_global_client, execute_cmd_with_args};
use redis::RedisResult;

async fn example_redis_interaction() -> RedisResult<()> {
    // Get a connection from the global pool
    let mut conn = get_global_client().await?.get_async_connection().await?;

    // Set a value
    execute_cmd_with_args(&mut conn, "SET", vec!["my_key", "my_value"]).await?;
    println!("Set 'my_key' to 'my_value'");

    // Get a value
    let value: String = execute_cmd_with_args(&mut conn, "GET", vec!["my_key"]).await?;
    println!("Retrieved value for 'my_key': {}", value);

    Ok(())
}

#[tokio::main]
async fn main() {
    if let Err(e) = example_redis_interaction().await {
        eprintln!("Redis Error: {}", e);
    }
}
```
*(Note: The Redis client API might have evolved; please refer to `src/redisclient/mod.rs` and its documentation for the most current usage.)*

## 🎯 **Why Choose SAL?**

### **Modular Architecture**
- **Install Only What You Need**: Each package is independent - no bloated dependencies
- **Faster Compilation**: Smaller dependency trees mean faster build times
- **Smaller Binaries**: Only include the functionality you actually use
- **Clear Dependencies**: Explicit about what functionality your project uses

### **Developer Experience**
- **Consistent APIs**: All packages follow the same design patterns and conventions
- **Comprehensive Documentation**: Each package has detailed documentation and examples
- **Real-World Tested**: All functionality is production-tested, no placeholder code
- **Type Safety**: Leverages Rust's type system for safe, reliable operations

### **Scripting Power**
- **Herodo Integration**: Execute Rhai scripts with full access to SAL functionality
- **Cross-Platform**: Works consistently across Windows, macOS, and Linux
- **Automation Ready**: Perfect for DevOps, CI/CD, and system administration tasks

## 📦 **Workspace Modules Overview**

SAL is organized as a Cargo workspace with the following crates:

### **Core Library Modules**
- **`sal-os`**: Core OS interactions, file system operations, environment access
- **`sal-process`**: Process creation, management, and control
- **`sal-text`**: Utilities for text processing and manipulation
- **`sal-net`**: Network operations, HTTP requests, and connectivity utilities

### **Integration Modules**
- **`sal-git`**: Git repository management and operations
- **`sal-vault`**: Cryptographic functions and keypair management
- **`sal-rhai`**: Integration layer for the Rhai scripting engine, used by `herodo`

### **Client Modules**
- **`sal-redisclient`**: Client for Redis database interactions
- **`sal-postgresclient`**: Client for PostgreSQL database interactions
- **`sal-zinit-client`**: Client for Zinit process supervisor
- **`sal-mycelium`**: Client for Mycelium network operations

### **Specialized Modules**
- **`sal-virt`**: Virtualization-related utilities (buildah, nerdctl, rfs)

### **Root Package & Binary**
- **`sal`**: Root umbrella crate that re-exports all modules
- **`herodo`**: Command-line binary for executing Rhai scripts

## 🔨 **Building SAL**

Build the entire workspace (all crates) using Cargo:

```bash
# Build all workspace members
cargo build --workspace

# Build for release
cargo build --workspace --release

# Build specific crate
cargo build -p sal-text
cargo build -p herodo
```

The `herodo` executable will be located at `target/debug/herodo` or `target/release/herodo`.

## 🧪 **Running Tests**

### **Rust Unit Tests**
```bash
# Run all workspace tests
cargo test --workspace

# Run tests for specific crate
cargo test -p sal-text
cargo test -p sal-os

# Run only library tests (faster)
cargo test --workspace --lib
```

### **Rhai Integration Tests**
Run comprehensive Rhai script tests that exercise `herodo` and SAL's scripted functionalities:

```bash
# Run all Rhai integration tests (16 modules)
./run_rhai_tests.sh

# Results: 16/16 modules pass with 100% success rate
```

The Rhai tests validate real-world functionality across all SAL modules and provide comprehensive integration testing.

## License

SAL is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.