...

pkg/servers/heroagent/README.md

@@ -22,11 +22,14 @@ The job system follows a specific flow:
 1. **Job Creation**:
    - When a job is created, it's stored in both OurDB and Redis
    - OurDB provides persistent storage with history tracking
-   - Redis provides fast access and queuing capabilities
+   - Redis stores the job data and adds the job ID to a queue for processing
+   - Each job is stored in Redis using a key pattern: `herojobs:<topic>:<jobID>`
+   - Each job ID is added to a queue using a key pattern: `heroqueue:<topic>`
 
 2. **Job Processing**:
    - Workers continuously poll Redis queues for new jobs
-   - When a job is found, it's updated to "active" status in both OurDB and Redis
+   - When a job is found, it's fetched from Redis and updated to "active" status
+   - The updated job is stored in both OurDB and Redis
    - The job is processed based on its parameters
 
 3. **Job Completion**:
@@ -34,6 +37,60 @@ The job system follows a specific flow:
    - The job is removed from Redis to keep only active jobs in memory
    - This approach ensures efficient memory usage while maintaining a complete history
 
+### Data Flow Diagram
+
+```
+Job Creation:
+  ┌─────────┐     ┌─────────┐     ┌─────────┐
+  │  Client │────▶│  OurDB  │     │  Redis  │
+  └─────────┘     └────┬────┘     └────┬────┘
+                       │               │
+                       │ Store Job     │ Store Job
+                       │               │
+                       ▼               ▼
+                  ┌─────────┐     ┌─────────┐
+                  │ Job Data│     │ Job Data│
+                  └─────────┘     └─────────┘
+                                      │
+                                      │ Add to Queue
+                                      │
+                                      ▼
+                                 ┌─────────┐
+                                 │  Queue  │
+                                 └─────────┘
+
+Job Processing:
+  ┌─────────┐     ┌─────────┐     ┌─────────┐
+  │  Worker │────▶│  Redis  │────▶│  OurDB  │
+  └─────────┘     └────┬────┘     └────┬────┘
+                       │               │
+                       │ Fetch Job     │ Update Job
+                       │ from Queue    │
+                       ▼               ▼
+                  ┌─────────┐     ┌─────────┐
+                  │ Job Data│     │ Job Data│
+                  └─────────┘     └─────────┘
+                       │
+                       │ Process Job
+                       │
+                       ▼
+                  ┌─────────┐
+                  │ Result  │
+                  └─────────┘
+
+Job Completion:
+  ┌─────────┐     ┌─────────┐     ┌─────────┐
+  │  Worker │────▶│  OurDB  │     │  Redis  │
+  └─────────┘     └────┬────┘     └────┬────┘
+                       │               │
+                       │ Update Job    │ Remove Job
+                       │               │
+                       ▼               ▼
+                  ┌─────────┐     ┌─────────┐
+                  │ Job Data│     │   ✓     │
+                  └─────────┘     └─────────┘
+```
+
 ### Components
 
 - **JobManager**: Coordinates job operations between OurDB and Redis