Hero Indexer

Full-text search service for the Hero OS ecosystem, powered by Tantivy.

Hero Indexer provides multi-database full-text search with dynamic schemas, 9 query types, and batch operations — exposed via JSON-RPC over Unix sockets with an admin web UI.

Architecture

hero_proxy ──HTTP──> hero_indexer_ui.sock ──raw JSON-RPC──> hero_indexer_server.sock
                     (admin UI + /rpc proxy)    (newline-delimited JSON-RPC 2.0)

Crate	Type	Purpose
hero_indexer	library	Core types, Tantivy index manager, RPC handlers
hero_indexer_server	binary	Backend — raw JSON-RPC over Unix socket
hero_indexer_sdk	library	Async client SDK (talks to backend socket)
hero_indexer_ui	binary	Admin UI + HTTP proxy to backend
hero_indexer_examples	examples	Example programs using the SDK

Quick Start

git clone ssh://git@forge.ourworld.tf/lhumina_code/hero_indexer.git
cd hero_indexer
make run

This starts both services via hero_proc:

1. hero_indexer_server — backend, creates a demo database on first run
2. hero_indexer_ui — admin UI (Unix socket only, use hero_proxy for HTTP access)

Make Targets

make run           # Build release and run both services via hero_proc
make rundev        # Run with debug logging via hero_proc
make stop          # Stop all services
make restart       # Restart all services
make status        # Show service status
make logs          # View server logs
make logs-ui       # View UI logs
make install       # Build release and install to ~/hero/bin
make installdev    # Install debug build (faster compile)
make build         # Build release binaries
make check         # cargo check
make lint          # cargo clippy
make test          # Run all tests
make test-all      # Run all tests including doc-tests
make fmt           # Format code
make clean         # Remove build artifacts
make help          # Show all commands

Sockets

Service	Socket	Protocol
Server	~/hero/var/sockets/hero_indexer_server.sock	Raw JSON-RPC 2.0 (newline-delimited)
UI	~/hero/var/sockets/hero_indexer_ui.sock	HTTP

Both services bind exclusively to Unix domain sockets. Use hero_proxy for external TCP access.

JSON-RPC Methods

The backend exposes 18 methods over its Unix socket:

Method	Description
rpc.discover	OpenRPC specification
rpc.health	Health check ({"status":"ok"})
server.ping	Ping / version info
server.stats	Uptime, database count, total docs
server.exit	Graceful shutdown
db.list	List all databases
db.create	Create database with schema
db.delete	Delete a database
db.close	Close database (free memory)
db.select	Select database for operations
db.info	Info about selected database
schema.get	Get schema of selected database
doc.add	Add a single document
doc.add_batch	Add documents in batch
doc.delete	Delete documents by field/value
index.commit	Commit pending changes
index.reload	Reload index reader
search.query	Execute a search query
search.count	Count matching documents

UI Endpoints

The UI service proxies requests to the backend:

Endpoint	Method	Description
/health	GET	Health check
/rpc	POST	JSON-RPC proxy to backend
/mcp	POST	MCP-to-OpenRPC translation proxy
/openrpc.json	GET	OpenRPC specification
/*	GET	Admin dashboard (static assets)

Query Types

Type	Description	Example
all	Match all documents	{"type": "all"}
match	Full-text match (tokenized)	{"type": "match", "field": "body", "value": "search terms"}
term	Exact match (keyword)	{"type": "term", "field": "id", "value": "abc123"}
fuzzy	Typo-tolerant match	{"type": "fuzzy", "field": "title", "value": "serch", "distance": 1}
phrase	Exact phrase match	{"type": "phrase", "field": "body", "value": "exact phrase"}
prefix	Prefix / autocomplete	{"type": "prefix", "field": "title", "value": "hel"}
range	Numeric/date range	{"type": "range", "field": "price", "gte": 10, "lt": 100}
regex	Regex pattern	{"type": "regex", "field": "title", "value": "test.*"}
boolean	Combine with must/should/must_not	{"type": "boolean", "must": [...], "must_not": [...]}

Field Types

Type	Description	Example Value
text	Full-text searchable (tokenized)	"Hello World"
str	Exact match keyword (not tokenized)	"user-123"
u64	Unsigned 64-bit integer	42
i64	Signed 64-bit integer	-10
f64	64-bit floating point	3.14
date	DateTime (RFC 3339)	"2024-01-15T10:30:00Z"
bool	Boolean	true
json	JSON object	{"key": "value"}
bytes	Binary data (base64)	"SGVsbG8="
ip	IP address	"192.168.1.1"

Field options: stored (retrieve in results), indexed (searchable), fast (sorting/aggregations), tokenizer (e.g. "en_stem").

SDK

Add hero_indexer_sdk to your project:

[dependencies]
hero_indexer_sdk = { git = "ssh://git@forge.ourworld.tf/lhumina_code/hero_indexer.git" }
tokio = { version = "1", features = ["full"] }
serde_json = "1"

use hero_indexer_sdk::HeroIndexClient;
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Connect to the backend socket
    let mut client = HeroIndexClient::connect_default().await?;

    // Create a database with schema
    client.db_create("articles", json!({
        "fields": [
            {"name": "title", "type": "text", "stored": true, "indexed": true},
            {"name": "body", "type": "text", "stored": true, "indexed": true}
        ]
    })).await?;

    // Select, add documents, commit
    client.db_select("articles").await?;
    client.doc_add(json!({"title": "Hello", "body": "Rust is awesome"})).await?;
    client.commit().await?;
    client.reload().await?;

    // Search
    let results = client.search(
        json!({"type": "match", "field": "body", "value": "rust"}),
        10, 0
    ).await?;
    println!("Found {} results", results.total_hits);

    Ok(())
}

Configuration

Server (hero_indexer_server)

Flag	Default	Description
--dir	~/hero/var/index	Base directory for all indexes
--socket	~/hero/var/sockets/hero_indexer_server.sock	Unix socket path

UI (hero_indexer_ui)

Flag	Default	Description
--bind	~/hero/var/sockets/hero_indexer_ui.sock	Unix socket path

Project Structure

hero_indexer/
├── Cargo.toml                         # Workspace (5 crates)
├── Makefile
├── buildenv.sh
├── crates/
│   ├── hero_indexer/                   # Core library
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── error.rs
│   │       └── modules/
│   │           ├── handlers.rs         # 18 RPC method handlers
│   │           ├── index_manager.rs    # Tantivy index lifecycle
│   │           ├── schema.rs           # Schema types
│   │           ├── query.rs            # Query types
│   │           └── rpc.rs              # JSON-RPC types + OpenRPC spec
│   ├── hero_indexer_server/            # Server binary
│   │   ├── openrpc.json               # OpenRPC specification
│   │   └── src/main.rs                # Unix socket server + demo DB
│   ├── hero_indexer_sdk/               # Client SDK
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── client.rs
│   │       ├── types.rs
│   │       └── error.rs
│   ├── hero_indexer_ui/                # Admin UI binary
│   │   ├── src/main.rs                # Unix socket listener + proxy
│   │   └── static/index.html          # Admin dashboard
│   └── hero_indexer_examples/          # Example programs
│       └── examples/
│           ├── health.rs
│           └── basic_usage.rs
├── docs/
│   ├── specs.md                       # Interface specification
│   └── OPENRPC.md                     # Full API documentation
└── .forgejo/workflows/                # CI/CD (Linux + macOS)

Requirements

• Rust 1.92.0+
• Unix-like OS (Linux, macOS)

License

Apache-2.0