253 lines
7.0 KiB
Markdown
253 lines
7.0 KiB
Markdown
# Tantivy Full‑Text Backend (JSON‑RPC)
|
||
|
||
This document explains how to use HeroDB’s Tantivy-backed full‑text search as a dedicated database backend and provides copy‑pasteable JSON‑RPC requests. Tantivy is available only for non‑admin databases (db_id >= 1). Admin DB 0 always uses Redb/Sled and rejects FT operations.
|
||
|
||
Important characteristics:
|
||
- Tantivy is a third backend alongside Redb and Sled. It provides search indexes only; there is no KV store backing it.
|
||
- On Tantivy databases, Redis KV/list/hash commands are rejected; only FT commands and basic control (SELECT, CLIENT, INFO, etc.) are allowed.
|
||
- FT JSON‑RPC is namespaced as "herodb" and methods are named with underscore: herodb_ftCreate, herodb_ftAdd, herodb_ftSearch, herodb_ftDel, herodb_ftInfo, herodb_ftDrop.
|
||
|
||
Reference to server implementation:
|
||
- RPC methods are defined in [rust.trait Rpc()](src/rpc.rs:70):
|
||
- [rust.fn ft_create()](src/rpc.rs:121)
|
||
- [rust.fn ft_add()](src/rpc.rs:130)
|
||
- [rust.fn ft_search()](src/rpc.rs:141)
|
||
- [rust.fn ft_del()](src/rpc.rs:154)
|
||
- [rust.fn ft_info()](src/rpc.rs:158)
|
||
- [rust.fn ft_drop()](src/rpc.rs:162)
|
||
|
||
Notes on responses:
|
||
- ftCreate/ftAdd/ftDel/ftDrop return a JSON boolean: true on success.
|
||
- ftSearch/ftInfo return a JSON object with a single key "resp" containing a RESP‑encoded string (wire format used by Redis). You can display or parse it on the client side as needed.
|
||
|
||
RESP usage (redis-cli):
|
||
- For RESP clients, you must SELECT the Tantivy database first. SELECT now succeeds for Tantivy DBs without opening KV storage.
|
||
- After SELECT, you can run FT.* commands within that DB context.
|
||
|
||
Example with redis-cli:
|
||
```bash
|
||
# Connect to server
|
||
redis-cli -p 6379
|
||
|
||
# Select Tantivy DB 1 (public by default)
|
||
SELECT 1
|
||
# → OK
|
||
|
||
# Create index
|
||
FT.CREATE product_catalog SCHEMA title TEXT description TEXT category TAG price NUMERIC rating NUMERIC location GEO
|
||
# → OK
|
||
|
||
# Add a document
|
||
FT.ADD product_catalog product:1 1.0 title "Wireless Bluetooth Headphones" description "Premium noise-canceling headphones with 30-hour battery life" category "electronics,audio" price 299.99 rating 4.5 location "-122.4194,37.7749"
|
||
# → OK
|
||
|
||
# Search
|
||
FT.SEARCH product_catalog wireless LIMIT 0 3
|
||
# → RESP array with hits
|
||
```
|
||
|
||
Storage layout (on disk):
|
||
- Indices are stored per database under:
|
||
- <base_dir>/search_indexes/<db_id>/<index_name>
|
||
- Example: /tmp/test/search_indexes/1/product_catalog
|
||
|
||
0) Create a new Tantivy database
|
||
|
||
Use herodb_createDatabase with backend "Tantivy". DB 0 cannot be Tantivy.
|
||
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 1,
|
||
"method": "herodb_createDatabase",
|
||
"params": [
|
||
"Tantivy",
|
||
{ "name": "search-db", "storage_path": null, "max_size": null, "redis_version": null },
|
||
null
|
||
]
|
||
}
|
||
```
|
||
|
||
The response contains the allocated db_id (>= 1). Use that id in the calls below.
|
||
|
||
1) FT.CREATE — create an index with schema
|
||
|
||
Method: herodb_ftCreate → [rust.fn ft_create()](src/rpc.rs:121)
|
||
|
||
Schema format is an array of tuples: [ [field_name, field_type, [options...] ], ... ]
|
||
Supported field types: "TEXT", "NUMERIC" (defaults to F64), "TAG", "GEO"
|
||
Supported options (subset): "WEIGHT", "SORTABLE", "NOINDEX", "SEPARATOR", "CASESENSITIVE"
|
||
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 2,
|
||
"method": "herodb_ftCreate",
|
||
"params": [
|
||
1,
|
||
"product_catalog",
|
||
[
|
||
["title", "TEXT", ["SORTABLE"]],
|
||
["description", "TEXT", []],
|
||
["category", "TAG", ["SEPARATOR", ","]],
|
||
["price", "NUMERIC", ["SORTABLE"]],
|
||
["rating", "NUMERIC", []],
|
||
["location", "GEO", []]
|
||
]
|
||
]
|
||
}
|
||
```
|
||
|
||
Returns: true on success.
|
||
|
||
2) FT.ADD — add or replace a document
|
||
|
||
Method: herodb_ftAdd → [rust.fn ft_add()](src/rpc.rs:130)
|
||
|
||
Fields is an object (map) of field_name → value (all values are sent as strings). GEO expects "lat,lon".
|
||
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 3,
|
||
"method": "herodb_ftAdd",
|
||
"params": [
|
||
1,
|
||
"product_catalog",
|
||
"product:1",
|
||
1.0,
|
||
{
|
||
"title": "Wireless Bluetooth Headphones",
|
||
"description": "Premium noise-canceling headphones with 30-hour battery life",
|
||
"category": "electronics,audio",
|
||
"price": "299.99",
|
||
"rating": "4.5",
|
||
"location": "-122.4194,37.7749"
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
Returns: true on success.
|
||
|
||
3) FT.SEARCH — query an index
|
||
|
||
Method: herodb_ftSearch → [rust.fn ft_search()](src/rpc.rs:141)
|
||
|
||
Parameters: (db_id, index_name, query, filters?, limit?, offset?, return_fields?)
|
||
- filters: array of [field, value] pairs (Equals filter)
|
||
- limit/offset: numbers (defaults: limit=10, offset=0)
|
||
- return_fields: array of field names to include (optional)
|
||
|
||
Simple query:
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 4,
|
||
"method": "herodb_ftSearch",
|
||
"params": [1, "product_catalog", "wireless", null, 10, 0, null]
|
||
}
|
||
```
|
||
|
||
Pagination + filters + selected fields:
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 5,
|
||
"method": "herodb_ftSearch",
|
||
"params": [
|
||
1,
|
||
"product_catalog",
|
||
"mouse",
|
||
[["category", "electronics"]],
|
||
5,
|
||
0,
|
||
["title", "price", "rating"]
|
||
]
|
||
}
|
||
```
|
||
|
||
Response shape:
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 5,
|
||
"result": { "resp": "*...RESP encoded array..." }
|
||
}
|
||
```
|
||
|
||
4) FT.INFO — index metadata
|
||
|
||
Method: herodb_ftInfo → [rust.fn ft_info()](src/rpc.rs:158)
|
||
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 6,
|
||
"method": "herodb_ftInfo",
|
||
"params": [1, "product_catalog"]
|
||
}
|
||
```
|
||
|
||
Response shape:
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 6,
|
||
"result": { "resp": "*...RESP encoded array with fields and counts..." }
|
||
}
|
||
```
|
||
|
||
5) FT.DEL — delete by doc id
|
||
|
||
Method: herodb_ftDel → [rust.fn ft_del()](src/rpc.rs:154)
|
||
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 7,
|
||
"method": "herodb_ftDel",
|
||
"params": [1, "product_catalog", "product:1"]
|
||
}
|
||
```
|
||
|
||
Returns: true on success. Note: current implementation logs and returns success; physical delete may be a no‑op until delete is finalized in the engine.
|
||
|
||
6) FT.DROP — drop an index
|
||
|
||
Method: herodb_ftDrop → [rust.fn ft_drop()](src/rpc.rs:162)
|
||
|
||
```json
|
||
{
|
||
"jsonrpc": "2.0",
|
||
"id": 8,
|
||
"method": "herodb_ftDrop",
|
||
"params": [1, "product_catalog"]
|
||
}
|
||
```
|
||
|
||
Returns: true on success.
|
||
|
||
Field types and options
|
||
|
||
- TEXT: stored/indexed/tokenized text. "SORTABLE" marks it fast (stored + fast path in our wrapper).
|
||
- NUMERIC: stored/indexed numeric; default precision F64. "SORTABLE" enables fast column.
|
||
- TAG: exact matching terms. Options: "SEPARATOR" (default ","), "CASESENSITIVE" (default false).
|
||
- GEO: "lat,lon" string; stored as two numeric fields internally.
|
||
|
||
Backend and permission gating
|
||
|
||
- FT methods are rejected on DB 0.
|
||
- FT methods require the database backend to be Tantivy; otherwise RPC returns an error.
|
||
- Write‑like FT methods (create/add/del/drop) follow the same permission model as Redis writes on selected databases.
|
||
|
||
Troubleshooting
|
||
|
||
- "DB backend is not Tantivy": ensure the database was created with backend "Tantivy".
|
||
- "FT not allowed on DB 0": use a non‑admin database id (>= 1).
|
||
- Empty search results: confirm that the queried fields are tokenized/indexed (TEXT) and that documents were added successfully.
|
||
|
||
Related docs
|
||
|
||
- Command‑level search overview: [docs/search.md](./search.md)
|
||
- RPC definitions: [src/rpc.rs](../src/rpc.rs) |