herodb/docs/lance.md

Lance Vector Backend (RESP + JSON-RPC)

This document explains how to use HeroDB’s Lance-backed vector store. It is text-first: users provide text, and HeroDB computes embeddings server-side (no manual vectors). It includes copy-pasteable RESP (redis-cli) and JSON-RPC examples for:

• Creating a Lance database
• Embedding provider configuration (OpenAI, Azure OpenAI, or deterministic test provider)
• Dataset lifecycle: CREATE, LIST, INFO, DROP
• Ingestion: STORE text (+ optional metadata)
• Search: QUERY with K, optional FILTER and RETURN
• Delete by id
• Index creation (currently a placeholder/no-op)

References:

• Implementation: src/lance_store.rs, src/cmd.rs, src/rpc.rs, src/server.rs, src/embedding.rs

Notes:

• Admin DB 0 cannot be Lance (or Tantivy). Only databases with id >= 1 can use Lance.
• Permissions:
  • Read operations (SEARCH, LIST, INFO) require read permission.
  • Mutating operations (CREATE, STORE, CREATEINDEX, DEL, DROP, EMBEDDING CONFIG SET) require readwrite permission.
• Backend gating:
  • If a DB is Lance, only LANCE.* and basic control commands (PING, ECHO, SELECT, INFO, CLIENT, etc.) are permitted.
  • If a DB is not Lance, LANCE.* commands return an error.

Storage layout and schema:

• Files live at: <base_dir>/lance/<db_id>/.lance
• Records schema:
  • id: Utf8 (non-null)
  • vector: FixedSizeList<Float32, dim> (non-null)
  • text: Utf8 (nullable)
  • meta: Utf8 JSON (nullable)
• Search is an L2 KNN brute-force scan for now (lower score = better). Index creation is a no-op placeholder to be implemented later.

Prerequisites:

• Start HeroDB with RPC enabled (for management calls):
  • See docs/basics.md for flags. Example:

    ./target/release/herodb --dir /tmp/herodb --admin-secret mysecret --port 6379 --enable-rpc
    

0) Create a Lance-backed database (JSON-RPC)

Use the management API to create a database with backend "Lance". DB 0 is reserved for admin and cannot be Lance.

Request:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "herodb_createDatabase",
  "params": [
    "Lance",
    { "name": "vectors-db", "storage_path": null, "max_size": null, "redis_version": null },
    null
  ]
}

• Response contains the allocated db_id (>= 1). Use that id below (replace 1 with your actual id).

Select the database over RESP:

redis-cli -p 6379 SELECT 1
# → OK

1) Configure embedding provider (server-side embeddings)

HeroDB embeds text internally at STORE/SEARCH time using a per-dataset EmbeddingConfig sidecar. Configure provider before creating a dataset to choose dimensions and provider.

Supported providers:

• openai (standard OpenAI or Azure OpenAI)
• testhash (deterministic, CI-friendly; no network)

Environment variables for OpenAI:

• Standard OpenAI: export OPENAI_API_KEY=sk-...
• Azure OpenAI: export AZURE_OPENAI_API_KEY=...

RESP examples:

# Standard OpenAI with default dims (model-dependent, e.g. 1536)
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER openai MODEL text-embedding-3-small

# OpenAI with reduced output dimension (e.g., 512) when supported
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER openai MODEL text-embedding-3-small PARAM dim 512

# Azure OpenAI (set env: AZURE_OPENAI_API_KEY)
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER openai MODEL text-embedding-3-small \
  PARAM use_azure true \
  PARAM azure_endpoint https://myresource.openai.azure.com \
  PARAM azure_deployment my-embed-deploy \
  PARAM azure_api_version 2024-02-15 \
  PARAM dim 512

# Deterministic test provider (no network, stable vectors)
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER testhash MODEL any

Read config:

redis-cli -p 6379 LANCE.EMBEDDING CONFIG GET myset
# → JSON blob describing provider/model/params

JSON-RPC examples:

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "herodb_lanceSetEmbeddingConfig",
  "params": [
    1,
    "myset",
    "openai",
    "text-embedding-3-small",
    { "dim": "512" }
  ]
}

{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "herodb_lanceGetEmbeddingConfig",
  "params": [1, "myset"]
}

2) Create a dataset

Choose a dimension that matches your embedding configuration. For OpenAI text-embedding-3-small without dimension override, typical dimension is 1536; when dim is set (e.g., 512), use that. The current API requires an explicit DIM.

RESP:

redis-cli -p 6379 LANCE.CREATE myset DIM 512
# → OK

JSON-RPC:

{
  "jsonrpc": "2.0",
  "id": 4,
  "method": "herodb_lanceCreate",
  "params": [1, "myset", 512]
}

3) Store text documents (server-side embedding)

Provide your id, the text to embed, and optional META fields. The server computes the embedding using the configured provider and stores id/vector/text/meta in the Lance dataset. Upserts by id are supported via delete-then-append semantics.

RESP:

redis-cli -p 6379 LANCE.STORE myset ID doc-1 TEXT "Hello vector world" META title "Hello" category "demo"
# → OK

JSON-RPC:

{
  "jsonrpc": "2.0",
  "id": 5,
  "method": "herodb_lanceStoreText",
  "params": [
    1,
    "myset",
    "doc-1",
    "Hello vector world",
    { "title": "Hello", "category": "demo" }
  ]
}

4) Search with a text query

Provide a query string; the server embeds it and performs KNN search. Optional: FILTER expression and RETURN subset of fields.

RESP:

# K nearest neighbors for the query text
redis-cli -p 6379 LANCE.SEARCH myset K 5 QUERY "greetings to vectors"
# → Array of hits: [id, score, [k,v, ...]] pairs, lower score = closer

# With a filter on meta fields and return only title
redis-cli -p 6379 LANCE.SEARCH myset K 3 QUERY "greetings to vectors" FILTER "category = 'demo'" RETURN 1 title

JSON-RPC:

{
  "jsonrpc": "2.0",
  "id": 6,
  "method": "herodb_lanceSearchText",
  "params": [1, "myset", "greetings to vectors", 5, null, null]
}

With filter and selected fields:

{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "herodb_lanceSearchText",
  "params": [1, "myset", "greetings to vectors", 3, "category = 'demo'", ["title"]]
}

Response shape:

• RESP over redis-cli: an array of hits [id, score, [k, v, ...]].
• JSON-RPC returns an object containing the RESP-encoded wire format string or a structured result depending on implementation. See src/rpc.rs for details.

5) Create an index (placeholder)

Index creation currently returns OK but is a no-op. It will integrate Lance vector indices in a future update.

RESP:

redis-cli -p 6379 LANCE.CREATEINDEX myset TYPE "ivf_pq" PARAM nlist 100 PARAM pq_m 16
# → OK (no-op for now)

JSON-RPC:

{
  "jsonrpc": "2.0",
  "id": 8,
  "method": "herodb_lanceCreateIndex",
  "params": [1, "myset", "ivf_pq", { "nlist": "100", "pq_m": "16" }]
}

6) Inspect datasets

RESP:

# List datasets in current Lance DB
redis-cli -p 6379 LANCE.LIST

# Get dataset info
redis-cli -p 6379 LANCE.INFO myset

JSON-RPC:

{
  "jsonrpc": "2.0",
  "id": 9,
  "method": "herodb_lanceList",
  "params": [1]
}

{
  "jsonrpc": "2.0",
  "id": 10,
  "method": "herodb_lanceInfo",
  "params": [1, "myset"]
}

7) Delete and drop

RESP:

# Delete by id
redis-cli -p 6379 LANCE.DEL myset doc-1
# → OK

# Drop the entire dataset
redis-cli -p 6379 LANCE.DROP myset
# → OK

JSON-RPC:

{
  "jsonrpc": "2.0",
  "id": 11,
  "method": "herodb_lanceDel",
  "params": [1, "myset", "doc-1"]
}

{
  "jsonrpc": "2.0",
  "id": 12,
  "method": "herodb_lanceDrop",
  "params": [1, "myset"]
}

8) End-to-end example (RESP)

# 1. Select Lance DB (assume db_id=1 created via RPC)
redis-cli -p 6379 SELECT 1

# 2. Configure embedding provider (OpenAI small model at 512 dims)
redis-cli -p 6379 LANCE.EMBEDDING CONFIG SET myset PROVIDER openai MODEL text-embedding-3-small PARAM dim 512

# 3. Create dataset
redis-cli -p 6379 LANCE.CREATE myset DIM 512

# 4. Store documents
redis-cli -p 6379 LANCE.STORE myset ID doc-1 TEXT "The quick brown fox jumps over the lazy dog" META title "Fox" category "animal"
redis-cli -p 6379 LANCE.STORE myset ID doc-2 TEXT "A fast auburn fox vaulted a sleepy canine" META title "Fox paraphrase" category "animal"

# 5. Search
redis-cli -p 6379 LANCE.SEARCH myset K 2 QUERY "quick brown fox" RETURN 1 title

# 6. Dataset info and listing
redis-cli -p 6379 LANCE.INFO myset
redis-cli -p 6379 LANCE.LIST

# 7. Delete and drop
redis-cli -p 6379 LANCE.DEL myset doc-2
redis-cli -p 6379 LANCE.DROP myset

9) End-to-end example (JSON-RPC)

Assume RPC server on port 8080. Replace ids and ports as needed.

1. Create Lance DB:

{
  "jsonrpc": "2.0",
  "id": 100,
  "method": "herodb_createDatabase",
  "params": ["Lance", { "name": "vectors-db", "storage_path": null, "max_size": null, "redis_version": null }, null]
}

2. Set embedding config:

{
  "jsonrpc": "2.0",
  "id": 101,
  "method": "herodb_lanceSetEmbeddingConfig",
  "params": [1, "myset", "openai", "text-embedding-3-small", { "dim": "512" }]
}

3. Create dataset:

{
  "jsonrpc": "2.0",
  "id": 102,
  "method": "herodb_lanceCreate",
  "params": [1, "myset", 512]
}

4. Store text:

{
  "jsonrpc": "2.0",
  "id": 103,
  "method": "herodb_lanceStoreText",
  "params": [1, "myset", "doc-1", "The quick brown fox jumps over the lazy dog", { "title": "Fox", "category": "animal" }]
}

5. Search text:

{
  "jsonrpc": "2.0",
  "id": 104,
  "method": "herodb_lanceSearchText",
  "params": [1, "myset", "quick brown fox", 2, null, ["title"]]
}

6. Info/list:

{
  "jsonrpc": "2.0",
  "id": 105,
  "method": "herodb_lanceInfo",
  "params": [1, "myset"]
}

{
  "jsonrpc": "2.0",
  "id": 106,
  "method": "herodb_lanceList",
  "params": [1]
}

7. Delete/drop:

{
  "jsonrpc": "2.0",
  "id": 107,
  "method": "herodb_lanceDel",
  "params": [1, "myset", "doc-1"]
}

{
  "jsonrpc": "2.0",
  "id": 108,
  "method": "herodb_lanceDrop",
  "params": [1, "myset"]
}

10) Operational notes and troubleshooting

• If using OpenAI and you see “missing API key env”, set:
  • Standard: export OPENAI_API_KEY=sk-...
  • Azure: export AZURE_OPENAI_API_KEY=... and pass use_azure true, azure_endpoint, azure_deployment, azure_api_version.
• Dimensions mismatch:
  • Ensure the dataset DIM equals the provider’s embedding dim. For OpenAI text-embedding-3 models, set PARAM dim 512 (or another supported size) and use that same DIM for LANCE.CREATE.
• DB 0 restriction:
  • Lance is not allowed on DB 0. Use db_id >= 1.
• Permissions:
  • Read operations (SEARCH, LIST, INFO) require read permission.
  • Mutations (CREATE, STORE, CREATEINDEX, DEL, DROP, EMBEDDING CONFIG SET) require readwrite permission.
• Backend gating:
  • On Lance DBs, only LANCE.* commands are accepted (plus basic control).
• Current index behavior:
  • LANCE.CREATEINDEX returns OK but is a no-op. Future versions will integrate Lance vector indices.
• Implementation files for reference:
  • src/lance_store.rs, src/cmd.rs, src/rpc.rs, src/server.rs, src/embedding.rs