heroagent/pkg/servers/heroagent/README.md

# HeroAgent Server Factory

The HeroAgent Server Factory is a comprehensive server management system that integrates multiple services:

- Redis Server
- WebDAV Server
- UI Server
- Job Management System

## Overview

The server factory provides a unified interface for starting, managing, and stopping these services. Each service can be enabled or disabled independently through configuration.

## Job Management System

The job management system provides a robust solution for handling asynchronous tasks with persistence and reliability. It combines the strengths of OurDB for persistent storage and Redis for active job queuing.

### Architecture

The job system follows a specific flow:

1. **Job Creation**:
   - When a job is created, it's stored in both OurDB and Redis
   - OurDB provides persistent storage with history tracking
   - Redis stores the job data and adds the job ID to a queue for processing
   - Each job is stored in Redis using a key pattern: `herojobs:<topic>:<jobID>`
   - Each job ID is added to a queue using a key pattern: `heroqueue:<topic>`

2. **Job Processing**:
   - Workers continuously poll Redis queues for new jobs
   - When a job is found, it's fetched from Redis and updated to "active" status
   - The updated job is stored in both OurDB and Redis
   - The job is processed based on its parameters

3. **Job Completion**:
   - When a job completes (success or error), it's updated in OurDB
   - The job is removed from Redis to keep only active jobs in memory
   - This approach ensures efficient memory usage while maintaining a complete history

### Data Flow Diagram

```
Job Creation:
  ┌─────────┐     ┌─────────┐     ┌─────────┐
  │  Client │────▶│  OurDB  │     │  Redis  │
  └─────────┘     └────┬────┘     └────┬────┘
                       │               │
                       │ Store Job     │ Store Job
                       │               │
                       ▼               ▼
                  ┌─────────┐     ┌─────────┐
                  │ Job Data│     │ Job Data│
                  └─────────┘     └─────────┘
                                      │
                                      │ Add to Queue
                                      │
                                      ▼
                                 ┌─────────┐
                                 │  Queue  │
                                 └─────────┘

Job Processing:
  ┌─────────┐     ┌─────────┐     ┌─────────┐
  │  Worker │────▶│  Redis  │────▶│  OurDB  │
  └─────────┘     └────┬────┘     └────┬────┘
                       │               │
                       │ Fetch Job     │ Update Job
                       │ from Queue    │
                       ▼               ▼
                  ┌─────────┐     ┌─────────┐
                  │ Job Data│     │ Job Data│
                  └─────────┘     └─────────┘
                       │
                       │ Process Job
                       │
                       ▼
                  ┌─────────┐
                  │ Result  │
                  └─────────┘

Job Completion:
  ┌─────────┐     ┌─────────┐     ┌─────────┐
  │  Worker │────▶│  OurDB  │     │  Redis  │
  └─────────┘     └────┬────┘     └────┬────┘
                       │               │
                       │ Update Job    │ Remove Job
                       │               │
                       ▼               ▼
                  ┌─────────┐     ┌─────────┐
                  │ Job Data│     │   ✓     │
                  └─────────┘     └─────────┘
```

### Components

- **JobManager**: Coordinates job operations between OurDB and Redis
- **RedisJobManager**: Handles Redis-specific operations for jobs
- **JobWorker**: Processes jobs from Redis queues
- **OurDB**: Provides persistent storage for all jobs

### Job States

Jobs can be in one of four states:

- **New**: Job has been created but not yet processed
- **Active**: Job is currently being processed
- **Done**: Job has completed successfully
- **Error**: Job encountered an error during processing

## Usage

### Configuration

```go
config := heroagent.DefaultConfig()

// Configure Redis
config.Redis.TCPPort = 6379
config.Redis.UnixSocketPath = "/tmp/redis.sock"

// Configure job system
config.Jobs.OurDBPath = "./data/jobsdb"
config.Jobs.WorkerCount = 5
config.Jobs.QueuePollInterval = 100 * time.Millisecond

// Enable/disable services
config.EnableRedis = true
config.EnableWebDAV = true
config.EnableUI = true
config.EnableJobs = true
```

### Starting the Server Factory

```go
// Create server factory
factory := heroagent.New(config)

// Start servers
if err := factory.Start(); err != nil {
    log.Fatalf("Failed to start servers: %v", err)
}
```

### Creating and Managing Jobs

```go
// Get job manager
jobManager := factory.GetJobManager()

// Create a job
job, err := jobManager.CreateJob("email", `{"to": "user@example.com", "subject": "Hello"}`)
if err != nil {
    log.Fatalf("Failed to create job: %v", err)
}

// Get job status
job, err = jobManager.GetJob(job.JobID)
if err != nil {
    log.Fatalf("Failed to get job: %v", err)
}

// Update job status
err = jobManager.UpdateJobStatus(job.JobID, heroagent.JobStatusActive)
if err != nil {
    log.Fatalf("Failed to update job status: %v", err)
}

// Complete a job
err = jobManager.CompleteJob(job.JobID, "Job completed successfully")
if err != nil {
    log.Fatalf("Failed to complete job: %v", err)
}

// Mark a job as failed
err = jobManager.FailJob(job.JobID, "Job failed due to network error")
if err != nil {
    log.Fatalf("Failed to mark job as failed: %v", err)
}
```

### Stopping the Server Factory

```go
// Stop servers
if err := factory.Stop(); err != nil {
    log.Fatalf("Failed to stop servers: %v", err)
}
```

## Implementation Details

### OurDB Integration

OurDB provides persistent storage for all jobs, including their complete history. It uses an auto-incrementing ID system to assign unique IDs to jobs.

### Redis Integration

Redis is used for active job queuing and temporary storage. Jobs are stored in Redis using the following key patterns:

- Queue keys: `heroqueue:<topic>`
- Job storage keys: `herojobs:<topic>:<jobID>`

When a job reaches a terminal state (done or error), it's removed from Redis but remains in OurDB for historical reference.

### Worker Pool

The job system uses a configurable worker pool to process jobs concurrently. Each worker polls Redis queues for new jobs and processes them independently.