herocode/herodb
12
0
Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

...

Browse Source
This commit is contained in:
despiegk
2025-08-23 04:57:47 +02:00
parent 56699b9abb
commit c9e1dcdb6c
47 changed files with 9267 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files

99
specs/backgroundinfo/encrypt.md Normal file
View File

@@ -0,0 +1,99 @@
### Cargo.toml
```toml
[dependencies]
chacha20poly1305 = { version = "0.10", features = ["xchacha20"] }
rand = "0.8"
sha2 = "0.10"
```
### `crypto_factory.rs`
```rust
use chacha20poly1305::{
aead::{Aead, KeyInit, OsRng},
XChaCha20Poly1305, Key, XNonce,
};
use rand::RngCore;
use sha2::{Digest, Sha256};
const VERSION: u8 = 1;
const NONCE_LEN: usize = 24;
const TAG_LEN: usize = 16;
#[derive(Debug)]
pub enum CryptoError {
Format, // wrong length / header
Version(u8), // unknown version
Decrypt, // wrong key or corrupted data
}
/// Super-simple factory: new(secret) + encrypt(bytes) + decrypt(bytes)
pub struct CryptoFactory {
key: Key<XChaCha20Poly1305>,
}
impl CryptoFactory {
/// Accepts any secret bytes; turns them into a 32-byte key (SHA-256).
/// (If your secret is already 32 bytes, this is still fine.)
pub fn new<S: AsRef<[u8]>>(secret: S) -> Self {
let mut h = Sha256::new();
h.update(b"xchacha20poly1305-factory:v1"); // domain separation
h.update(secret.as_ref());
let digest = h.finalize(); // 32 bytes
let key = Key::<XChaCha20Poly1305>::from_slice(&digest).to_owned();
Self { key }
}
/// Output layout: [version:1][nonce:24][ciphertext||tag]
pub fn encrypt(&self, plaintext: &[u8]) -> Vec<u8> {
let cipher = XChaCha20Poly1305::new(&self.key);
let mut nonce_bytes = [0u8; NONCE_LEN];
OsRng.fill_bytes(&mut nonce_bytes);
let nonce = XNonce::from_slice(&nonce_bytes);
let mut out = Vec::with_capacity(1 + NONCE_LEN + plaintext.len() + TAG_LEN);
out.push(VERSION);
out.extend_from_slice(&nonce_bytes);
let ct = cipher.encrypt(nonce, plaintext).expect("encrypt");
out.extend_from_slice(&ct);
out
}
pub fn decrypt(&self, blob: &[u8]) -> Result<Vec<u8>, CryptoError> {
if blob.len() < 1 + NONCE_LEN + TAG_LEN {
return Err(CryptoError::Format);
}
let ver = blob[0];
if ver != VERSION {
return Err(CryptoError::Version(ver));
}
let nonce = XNonce::from_slice(&blob[1..1 + NONCE_LEN]);
let ct = &blob[1 + NONCE_LEN..];
let cipher = XChaCha20Poly1305::new(&self.key);
cipher.decrypt(nonce, ct).map_err(|_| CryptoError::Decrypt)
}
}
```
### Tiny usage example
```rust
fn main() {
let f = CryptoFactory::new(b"super-secret-key-material");
let val = b"\x00\xFFbinary\x01\x02\x03";
let blob = f.encrypt(val);
let roundtrip = f.decrypt(&blob).unwrap();
assert_eq!(roundtrip, val);
}
```
That’s it: `new(secret)`, `encrypt(bytes)`, `decrypt(bytes)`.
You can stash the returned `blob` directly in your storage layer behind Redis.

1251
specs/backgroundinfo/lance.md Normal file
View File

File diff suppressed because it is too large Load Diff

6847
specs/backgroundinfo/lancedb.md Normal file
View File

File diff suppressed because it is too large Load Diff

80
specs/backgroundinfo/redb.md Normal file
View File

@@ -0,0 +1,80 @@
========================
CODE SNIPPETS
========================
TITLE: 1PC+C Commit Strategy Vulnerability Example
DESCRIPTION: Illustrates a scenario where a partially committed transaction might appear complete due to the non-cryptographic checksum (XXH3) used in the 1PC+C commit strategy. This requires controlling page flush order, introducing a crash during fsync, and ensuring valid checksums for partially written data.
SOURCE: https://github.com/cberner/redb/blob/master/docs/design.md#_snippet_9
LANGUAGE: rust
CODE:
```
table.insert(malicious_key, malicious_value);
table.insert(good_key, good_value);
txn.commit();
```
LANGUAGE: rust
CODE:
```
table.insert(malicious_key, malicious_value);
txn.commit();
```
----------------------------------------
TITLE: Basic Key-Value Operations in redb
DESCRIPTION: Demonstrates the fundamental usage of redb for creating a database, opening a table, inserting a key-value pair, and retrieving the value within separate read and write transactions.
SOURCE: https://github.com/cberner/redb/blob/master/README.md#_snippet_0
LANGUAGE: rust
CODE:
```
use redb::{Database, Error, ReadableTable, TableDefinition};
const TABLE: TableDefinition<&str, u64> = TableDefinition::new("my_data");
fn main() -> Result<(), Error> {
let db = Database::create("my_db.redb")?;
let write_txn = db.begin_write()?;
{
let mut table = write_txn.open_table(TABLE)?;
table.insert("my_key", &123)?;
}
write_txn.commit()?;
let read_txn = db.begin_read()?;
let table = read_txn.open_table(TABLE)?;
assert_eq!(table.get("my_key")?.unwrap().value(), 123);
Ok(())
}
```
## What *redb* currently supports:
* Simple operations like creating databases, inserting key-value pairs, opening and reading tables ([GitHub][1]).
* No mention of operations such as:
* Iterating over keys with a given prefix.
* Range queries based on string prefixes.
* Specialized prefix‑filtered lookups.
## implement range scans as follows
You can implement prefix-like functionality using **range scans** combined with manual checks, similar to using a `BTreeSet` in Rust:
```rust
for key in table.range(prefix..).keys() {
if !key.starts_with(prefix) {
break;
}
// process key
}
```
This pattern iterates keys starting at the prefix, and stops once a key no longer matches the prefix—this works because the keys are sorted ([GitHub][1]).

150
specs/backgroundinfo/redis_basic_client.md Normal file
View File

@@ -0,0 +1,150 @@
]
# INFO
**What it does**
Returns server stats in a human-readable text block, optionally filtered by sections. Typical sections: `server`, `clients`, `memory`, `persistence`, `stats`, `replication`, `cpu`, `commandstats`, `latencystats`, `cluster`, `modules`, `keyspace`, `errorstats`. Special args: `all`, `default`, `everything`. The reply is a **Bulk String** with `# <Section>` headers and `key:value` lines. ([Redis][1])
**Syntax**
```
INFO [section [section ...]]
```
**Return (RESP2/RESP3)**: Bulk String. ([Redis][1])
**RESP request/response**
```
# Request: whole default set
*1\r\n$4\r\nINFO\r\n
# Request: a specific section, e.g., clients
*2\r\n$4\r\nINFO\r\n$7\r\nclients\r\n
# Response (prefix shown; body is long)
$1234\r\n# Server\r\nredis_version:7.4.0\r\n...\r\n# Clients\r\nconnected_clients:3\r\n...\r\n
```
(Reply type/format per RESP spec and the INFO page.) ([Redis][2])
---
# Connection “name” (there is **no** top-level `NAME` command)
Redis doesn’t have a standalone `NAME` command. Connection names are handled via `CLIENT SETNAME` and retrieved via `CLIENT GETNAME`. ([Redis][3])
## CLIENT SETNAME
Assigns a human label to the current connection (shown in `CLIENT LIST`, logs, etc.). No spaces allowed in the name; empty string clears it. Length is limited by Redis string limits (practically huge). **Reply**: Simple String `OK`. ([Redis][4])
**Syntax**
```
CLIENT SETNAME connection-name
```
**RESP**
```
# Set the name "myapp"
*3\r\n$6\r\nCLIENT\r\n$7\r\nSETNAME\r\n$5\r\nmyapp\r\n
# Reply
+OK\r\n
```
## CLIENT GETNAME
Returns the current connection’s name or **Null Bulk String** if unset. ([Redis][5])
**Syntax**
```
CLIENT GETNAME
```
**RESP**
```
# Before SETNAME:
*2\r\n$6\r\nCLIENT\r\n$7\r\nGETNAME\r\n
$-1\r\n # nil (no name)
# After SETNAME myapp:
*2\r\n$6\r\nCLIENT\r\n$7\r\nGETNAME\r\n
$5\r\nmyapp\r\n
```
(Null/Bulk String encoding per RESP spec.) ([Redis][2])
---
# CLIENT (container command + key subcommands)
`CLIENT` is a **container**; use subcommands like `CLIENT LIST`, `CLIENT INFO`, `CLIENT ID`, `CLIENT KILL`, `CLIENT TRACKING`, etc. Call `CLIENT HELP` to enumerate them. ([Redis][3])
## CLIENT LIST
Shows all connections as a single **Bulk String**: one line per client with `field=value` pairs (includes `id`, `addr`, `name`, `db`, `user`, `resp`, and more). Filters: `TYPE` and `ID`. **Return**: Bulk String (RESP2/RESP3). ([Redis][6])
**Syntax**
```
CLIENT LIST [TYPE <NORMAL|MASTER|REPLICA|PUBSUB>] [ID client-id ...]
```
**RESP**
```
*2\r\n$6\r\nCLIENT\r\n$4\r\nLIST\r\n
# Reply (single Bulk String; example with one line shown)
$188\r\nid=7 addr=127.0.0.1:60840 laddr=127.0.0.1:6379 fd=8 name=myapp age=12 idle=3 flags=N db=0 ...\r\n
```
## CLIENT INFO
Returns info for **this** connection only (same format/fields as a single line of `CLIENT LIST`). **Return**: Bulk String. Available since 6.2.0. ([Redis][7])
**Syntax**
```
CLIENT INFO
```
**RESP**
```
*2\r\n$6\r\nCLIENT\r\n$4\r\nINFO\r\n
$160\r\nid=7 addr=127.0.0.1:60840 laddr=127.0.0.1:6379 fd=8 name=myapp db=0 user=default resp=2 ...\r\n
```
---
# RESP notes you’ll need for your parser
* **Requests** are Arrays: `*N\r\n` followed by `N` Bulk Strings for verb/args.
* **Common replies here**: Simple String (`+OK\r\n`), Bulk String (`$<len>\r\n...\r\n`), and **Null Bulk String** (`$-1\r\n`). (These cover `INFO`, `CLIENT LIST/INFO`, `CLIENT GETNAME`, `CLIENT SETNAME`.) ([Redis][2])
---
## Sources (checked)
* INFO command (syntax, sections, behavior). ([Redis][1])
* RESP spec (request/response framing, Bulk/Null Bulk Strings). ([Redis][2])
* CLIENT container + subcommands index. ([Redis][3])
* CLIENT LIST (fields, bulk-string return, filters). ([Redis][6])
* CLIENT INFO (exists since 6.2, reply format). ([Redis][7])
* CLIENT SETNAME (no spaces; clears with empty string; huge length OK). ([Redis][4])
* CLIENT GETNAME (nil if unset). ([Redis][5])
If you want, I can fold this into a tiny Rust “command + RESP” test harness that exercises `INFO`, `CLIENT SETNAME/GETNAME`, `CLIENT LIST`, and `CLIENT INFO` against your in-mem RESP parser.
[1]: https://redis.io/docs/latest/commands/info/ "INFO | Docs"
[2]: https://redis.io/docs/latest/develop/reference/protocol-spec/?utm_source=chatgpt.com "Redis serialization protocol specification | Docs"
[3]: https://redis.io/docs/latest/commands/client/ "CLIENT | Docs"
[4]: https://redis.io/docs/latest/commands/client-setname/?utm_source=chatgpt.com "CLIENT SETNAME | Docs"
[5]: https://redis.io/docs/latest/commands/client-getname/?utm_source=chatgpt.com "CLIENT GETNAME | Docs"
[6]: https://redis.io/docs/latest/commands/client-list/ "CLIENT LIST | Docs"
[7]: https://redis.io/docs/latest/commands/client-info/?utm_source=chatgpt.com "CLIENT INFO | Docs"

251
specs/backgroundinfo/redis_basics.md Normal file
View File

@@ -0,0 +1,251 @@
Got it 👍 — let’s break this down properly.
Redis has two broad classes you’re asking about:
1. **Basic key-space functions** (SET, GET, DEL, EXISTS, etc.)
2. **Iteration commands** (`SCAN`, `SSCAN`, `HSCAN`, `ZSCAN`)
And for each I’ll show:
* What it does
* How it works at a high level
* Its **RESP protocol implementation** (the actual wire format).
---
# 1. Basic Key-Space Commands
### `SET key value`
* Stores a string value at a key.
* Overwrites if the key already exists.
**Protocol (RESP2):**
```
*3
$3
SET
$3
foo
$3
bar
```
(client sends: array of 3 bulk strings: `["SET", "foo", "bar"]`)
**Reply:**
```
+OK
```
---
### `GET key`
* Retrieves the string value stored at the key.
* Returns `nil` if key doesn’t exist.
**Protocol:**
```
*2
$3
GET
$3
foo
```
**Reply:**
```
$3
bar
```
(or `$-1` for nil)
---
### `DEL key [key ...]`
* Removes one or more keys.
* Returns number of keys actually removed.
**Protocol:**
```
*2
$3
DEL
$3
foo
```
**Reply:**
```
:1
```
(integer reply = number of deleted keys)
---
### `EXISTS key [key ...]`
* Checks if one or more keys exist.
* Returns count of existing keys.
**Protocol:**
```
*2
$6
EXISTS
$3
foo
```
**Reply:**
```
:1
```
---
### `KEYS pattern`
* Returns all keys matching a glob-style pattern.
⚠️ Not efficient in production (O(N)), better to use `SCAN`.
**Protocol:**
```
*2
$4
KEYS
$1
*
```
**Reply:**
```
*2
$3
foo
$3
bar
```
(array of bulk strings with key names)
---
# 2. Iteration Commands (`SCAN` family)
### `SCAN cursor [MATCH pattern] [COUNT n]`
* Iterates the keyspace incrementally.
* Client keeps sending back the cursor from previous call until it returns `0`.
**Protocol example:**
```
*2
$4
SCAN
$1
0
```
**Reply:**
```
*2
$1
0
*2
$3
foo
$3
bar
```
Explanation:
* First element = new cursor (`"0"` means iteration finished).
* Second element = array of keys returned in this batch.
---
### `HSCAN key cursor [MATCH pattern] [COUNT n]`
* Like `SCAN`, but iterates fields of a hash.
**Protocol:**
```
*3
$5
HSCAN
$3
myh
$1
0
```
**Reply:**
```
*2
$1
0
*4
$5
field
$5
value
$5
age
$2
42
```
(Array of alternating field/value pairs)
---
### `SSCAN key cursor [MATCH pattern] [COUNT n]`
* Iterates members of a set.
Protocol and reply structure same as SCAN.
---
### `ZSCAN key cursor [MATCH pattern] [COUNT n]`
* Iterates members of a sorted set with scores.
* Returns alternating `member`, `score`.
---
# Quick Comparison
| Command | Purpose | Return Type |
| -------- | ----------------------------- | --------------------- |
| `SET` | Store a string value | Simple string `+OK` |
| `GET` | Retrieve a string value | Bulk string / nil |
| `DEL` | Delete keys | Integer (count) |
| `EXISTS` | Check existence | Integer (count) |
| `KEYS` | List all matching keys (slow) | Array of bulk strings |
| `SCAN` | Iterate over keys (safe) | `[cursor, array]` |
| `HSCAN` | Iterate over hash fields | `[cursor, array]` |
| `SSCAN` | Iterate over set members | `[cursor, array]` |
| `ZSCAN` | Iterate over sorted set | `[cursor, array]` |
##

307
specs/backgroundinfo/redis_hset_functions.md Normal file
View File

@@ -0,0 +1,307 @@
# 🔑 Redis `HSET` and Related Hash Commands
## 1. `HSET`
* **Purpose**: Set the value of one or more fields in a hash.
* **Syntax**:
```bash
HSET key field value [field value ...]
```
* **Return**:
* Integer: number of fields that were newly added.
* **RESP Protocol**:
```
*4
$4
HSET
$3
key
$5
field
$5
value
```
(If multiple field-value pairs: `*6`, `*8`, etc.)
---
## 2. `HSETNX`
* **Purpose**: Set the value of a hash field only if it does **not** exist.
* **Syntax**:
```bash
HSETNX key field value
```
* **Return**:
* `1` if field was set.
* `0` if field already exists.
* **RESP Protocol**:
```
*4
$6
HSETNX
$3
key
$5
field
$5
value
```
---
## 3. `HGET`
* **Purpose**: Get the value of a hash field.
* **Syntax**:
```bash
HGET key field
```
* **Return**:
* Bulk string (value) or `nil` if field does not exist.
* **RESP Protocol**:
```
*3
$4
HGET
$3
key
$5
field
```
---
## 4. `HGETALL`
* **Purpose**: Get all fields and values in a hash.
* **Syntax**:
```bash
HGETALL key
```
* **Return**:
* Array of `[field1, value1, field2, value2, ...]`.
* **RESP Protocol**:
```
*2
$7
HGETALL
$3
key
```
---
## 5. `HMSET` (⚠️ Deprecated, use `HSET`)
* **Purpose**: Set multiple field-value pairs.
* **Syntax**:
```bash
HMSET key field value [field value ...]
```
* **Return**:
* Always `OK`.
* **RESP Protocol**:
```
*6
$5
HMSET
$3
key
$5
field
$5
value
$5
field2
$5
value2
```
---
## 6. `HMGET`
* **Purpose**: Get values of multiple fields.
* **Syntax**:
```bash
HMGET key field [field ...]
```
* **Return**:
* Array of values (bulk strings or nils).
* **RESP Protocol**:
```
*4
$5
HMGET
$3
key
$5
field1
$5
field2
```
---
## 7. `HDEL`
* **Purpose**: Delete one or more fields from a hash.
* **Syntax**:
```bash
HDEL key field [field ...]
```
* **Return**:
* Integer: number of fields removed.
* **RESP Protocol**:
```
*3
$4
HDEL
$3
key
$5
field
```
---
## 8. `HEXISTS`
* **Purpose**: Check if a field exists.
* **Syntax**:
```bash
HEXISTS key field
```
* **Return**:
* `1` if exists, `0` if not.
* **RESP Protocol**:
```
*3
$7
HEXISTS
$3
key
$5
field
```
---
## 9. `HKEYS`
* **Purpose**: Get all field names in a hash.
* **Syntax**:
```bash
HKEYS key
```
* **Return**:
* Array of field names.
* **RESP Protocol**:
```
*2
$5
HKEYS
$3
key
```
---
## 10. `HVALS`
* **Purpose**: Get all values in a hash.
* **Syntax**:
```bash
HVALS key
```
* **Return**:
* Array of values.
* **RESP Protocol**:
```
*2
$5
HVALS
$3
key
```
---
## 11. `HLEN`
* **Purpose**: Get number of fields in a hash.
* **Syntax**:
```bash
HLEN key
```
* **Return**:
* Integer: number of fields.
* **RESP Protocol**:
```
*2
$4
HLEN
$3
key
```
## 12. `HSCAN`
* **Purpose**: Iterate fields/values of a hash (cursor-based scan).
* **Syntax**:
```bash
HSCAN key cursor [MATCH pattern] [COUNT count]
```
* **Return**:
* Array: `[new-cursor, [field1, value1, ...]]`
* **RESP Protocol**:
```
*3
$5
HSCAN
$3
key
$1
0
```

259
specs/backgroundinfo/redis_lists.md Normal file
View File

@@ -0,0 +1,259 @@
# 1) Data model & basics
* A **queue** is a List at key `queue:<name>`.
* Common patterns:
* **Producer**: `LPUSH queue item` (or `RPUSH`)
* **Consumer (non-blocking)**: `RPOP queue` (or `LPOP`)
* **Consumer (blocking)**: `BRPOP queue timeout` (or `BLPOP`)
* If a key doesn’t exist, it’s treated as an **empty list**; push **creates** the list; when the **last element is popped, the key is deleted**. ([Redis][1])
---
# 2) Commands to implement (queues via Lists)
## LPUSH / RPUSH
Prepend/append one or more elements. Create the list if it doesn’t exist.
**Return**: Integer = new length of the list.
**Syntax**
```
LPUSH key element [element ...]
RPUSH key element [element ...]
```
**RESP (example)**
```
*3\r\n$5\r\nLPUSH\r\n$5\r\nqueue\r\n$5\r\njob-1\r\n
:1\r\n
```
Refs: semantics & multi-arg ordering. ([Redis][1])
### LPUSHX / RPUSHX (optional but useful)
Like LPUSH/RPUSH, **but only if the list exists**.
**Return**: Integer = new length (0 if key didn’t exist).
```
LPUSHX key element [element ...]
RPUSHX key element [element ...]
```
Refs: command index. ([Redis][2])
---
## LPOP / RPOP
Remove & return one (default) or **up to COUNT** elements since Redis 6.2.
If the list is empty or missing, **Null** is returned (Null Bulk or Null Array if COUNT>1).
**Return**:
* No COUNT: Bulk String or Null Bulk.
* With COUNT: Array of Bulk Strings (possibly empty) or Null Array if key missing.
**Syntax**
```
LPOP key [count]
RPOP key [count]
```
**RESP (no COUNT)**
```
*2\r\n$4\r\nRPOP\r\n$5\r\nqueue\r\n
$5\r\njob-1\r\n # or $-1\r\n if empty
```
**RESP (COUNT=2)**
```
*3\r\n$4\r\nLPOP\r\n$5\r\nqueue\r\n$1\r\n2\r\n
*2\r\n$5\r\njob-2\r\n$5\r\njob-3\r\n # or *-1\r\n if key missing
```
Refs: LPOP w/ COUNT; general pop semantics. ([Redis][3])
---
## BLPOP / BRPOP (blocking consumers)
Block until an element is available in any of the given lists or until `timeout` (seconds, **double**, `0` = forever).
**Return** on success: **Array \[key, element]**.
**Return** on timeout: **Null Array**.
**Syntax**
```
BLPOP key [key ...] timeout
BRPOP key [key ...] timeout
```
**RESP**
```
*3\r\n$5\r\nBRPOP\r\n$5\r\nqueue\r\n$1\r\n0\r\n # block forever
# Success reply
*2\r\n$5\r\nqueue\r\n$5\r\njob-4\r\n
# Timeout reply
*-1\r\n
```
**Implementation notes**
* If any listed key is non-empty at call time, reply **immediately** from the first non-empty key **by the command’s key order**.
* Otherwise, put the client into a **blocked state** (register per-key waiters). On any `LPUSH/RPUSH` to those keys, **wake the earliest waiter** and serve it atomically.
* If timeout expires, return **Null Array** and clear the blocked state.
Refs: timeout semantics and return shape. ([Redis][4])
---
## LMOVE / BLMOVE (atomic move; replaces RPOPLPUSH/BRPOPLPUSH)
Atomically **pop from one side** of `source` and **push to one side** of `destination`.
* Use for **reliable queues** (move to a *processing* list).
* `BLMOVE` blocks like `BLPOP` when `source` is empty.
**Syntax**
```
LMOVE source destination LEFT|RIGHT LEFT|RIGHT
BLMOVE source destination LEFT|RIGHT LEFT|RIGHT timeout
```
**Return**: Bulk String element moved, or Null if `source` empty (LMOVE); `BLMOVE` blocks/Null on timeout.
**RESP (LMOVE RIGHT->LEFT)**
```
*5\r\n$5\r\nLMOVE\r\n$6\r\nsource\r\n$3\r\ndst\r\n$5\r\nRIGHT\r\n$4\r\nLEFT\r\n
$5\r\njob-5\r\n
```
**Notes**
* Prefer `LMOVE/BLMOVE` over deprecated `RPOPLPUSH/BRPOPLPUSH`.
* Pattern: consumer `LMOVE queue processing RIGHT LEFT` → work → `LREM processing 1 <elem>` to ACK; a reaper can requeue stale items.
Refs: LMOVE/BLMOVE behavior and reliable-queue pattern; deprecation of RPOPLPUSH. ([Redis][5])
*(Compat: you can still implement `RPOPLPUSH source dest` and `BRPOPLPUSH source dest timeout`, but mark them deprecated and map to LMOVE/BLMOVE.)* ([Redis][6])
---
## LLEN (length)
Useful for metrics/backpressure.
```
LLEN key
```
**RESP**
```
*2\r\n$4\r\nLLEN\r\n$5\r\nqueue\r\n
:3\r\n
```
Refs: list overview mentioning LLEN. ([Redis][7])
---
## LREM (ack for “reliable” processing)
Remove occurrences of `element` from the list (head→tail scan).
Use `count=1` to ACK a single processed item from `processing`.
```
LREM key count element
```
**RESP**
```
*4\r\n$4\r\nLREM\r\n$9\r\nprocessing\r\n$1\r\n1\r\n$5\r\njob-5\r\n
:1\r\n
```
Refs: reliable pattern mentions LREM to ACK. ([Redis][5])
---
## LTRIM (bounded queues / retention)
Keep only `[start, stop]` range; everything else is dropped.
Use to cap queue length after pushes.
```
LTRIM key start stop
```
**RESP**
```
*4\r\n$5\r\nLTRIM\r\n$5\r\nqueue\r\n$2\r\n0\r\n$3\r\n999\r\n
+OK\r\n
```
Refs: list overview includes LTRIM for retention. ([Redis][7])
---
## LRANGE / LINDEX (debugging / peeking)
* `LRANGE key start stop` → Array of elements (non-destructive).
* `LINDEX key index` → one element or Null.
These aren’t required for queue semantics, but handy. ([Redis][7])
---
# 3) Errors & types
* Wrong type: `-WRONGTYPE Operation against a key holding the wrong kind of value\r\n`
* Non-existing key:
* Push: creates the list (returns new length).
* Pop (non-blocking): returns **Null**.
* Blocking pop: **Null Array** on timeout. ([Redis][1])
---
# 4) Blocking engine (implementation sketch)
1. **Call time**: scan keys in user order. If a non-empty list is found, pop & reply immediately.
2. **Otherwise**: register the client as **blocked** on those keys with `deadline = now + timeout` (or infinite).
3. **On push to any key**: if waiters exist, **wake one** (FIFO) and serve its pop **atomically** with the push result.
4. **On timer**: for each blocked client whose deadline passed, reply `Null Array` and clear state.
5. **Connection close**: remove from any wait queues.
Refs for timeout/block semantics. ([Redis][4])
---
# 5) Reliable queue pattern (recommended)
* **Consume**: `LMOVE queue processing RIGHT LEFT` (or `BLMOVE ... 0`).
* **Process** the job.
* **ACK**: `LREM processing 1 <job>` when done.
* **Reaper**: auxiliary task that detects stale jobs (e.g., track job IDs + timestamps in a ZSET) and requeues them. (Lists don’t include timestamps; pairing with a ZSET is standard practice.)
Refs: LMOVE doc’s pattern. ([Redis][5])
---
# 6) Minimal test matrix
* Push/pop happy path (both ends), with/without COUNT.
* Blocking pop: immediate availability, block + timeout, wake on push, multiple keys order, FIFO across multiple waiters.
* LMOVE/BLMOVE: RIGHT→LEFT pipeline, block + wake, cross-list atomicity, ACK via LREM.
* Type errors and key deletion on last pop.

113
specs/backgroundinfo/sled.md Normal file
View File

@@ -0,0 +1,113 @@
========================
CODE SNIPPETS
========================
TITLE: Basic Database Operations with sled in Rust
DESCRIPTION: This snippet demonstrates fundamental operations using the `sled` embedded database in Rust. It covers opening a database tree, inserting and retrieving key-value pairs, performing range queries, deleting entries, and executing an atomic compare-and-swap operation. It also shows how to flush changes to disk for durability.
SOURCE: https://github.com/spacejam/sled/blob/main/README.md#_snippet_0
LANGUAGE: Rust
CODE:
```
let tree = sled::open("/tmp/welcome-to-sled")?;
// insert and get, similar to std's BTreeMap
let old_value = tree.insert("key", "value")?;
assert_eq!(
tree.get(&"key")?,
Some(sled::IVec::from("value")),
);
// range queries
for kv_result in tree.range("key_1".."key_9") {}
// deletion
let old_value = tree.remove(&"key")?;
// atomic compare and swap
tree.compare_and_swap(
"key",
Some("current_value"),
Some("new_value"),
)?;
// block until all operations are stable on disk
// (flush_async also available to get a Future)
tree.flush()?;
```
----------------------------------------
TITLE: Subscribing to sled Events Asynchronously (Rust)
DESCRIPTION: This snippet demonstrates how to asynchronously subscribe to events on key prefixes in a `sled` database. It initializes a `sled` database, creates a `Subscriber` for all key prefixes, inserts a key-value pair to trigger an event, and then uses `extreme::run` to await and process incoming events. The `Subscriber` struct implements `Future<Output=Option<Event>>`, allowing it to be awaited in an async context.
SOURCE: https://github.com/spacejam/sled/blob/main/README.md#_snippet_1
LANGUAGE: Rust
CODE:
```
let sled = sled::open("my_db").unwrap();
let mut sub = sled.watch_prefix("");
sled.insert(b"a", b"a").unwrap();
extreme::run(async move {
while let Some(event) = (&mut sub).await {
println!("got event {:?}", event);
}
});
```
----------------------------------------
TITLE: Iterating Subscriber Events with Async/Await in Rust
DESCRIPTION: This snippet demonstrates how to asynchronously iterate over events from a `Subscriber` instance in Rust. Since `Subscriber` now implements `Future`, it can be awaited in a loop to process incoming events, enabling efficient prefix watching. The loop continues as long as new events are available.
SOURCE: https://github.com/spacejam/sled/blob/main/CHANGELOG.md#_snippet_0
LANGUAGE: Rust
CODE:
```
while let Some(event) = (&mut subscriber).await {}
```
----------------------------------------
TITLE: Suppressing TSAN Race on Arc::drop in Rust
DESCRIPTION: This suppression addresses a false positive race detection by ThreadSanitizer in Rust's `Arc::drop` implementation. TSAN fails to correctly reason about the raw atomic `Acquire` fence used after the strong-count atomic subtraction with a `Release` fence in the `Drop` implementation, leading to an erroneous race report.
SOURCE: https://github.com/spacejam/sled/blob/main/tsan_suppressions.txt#_snippet_0
LANGUAGE: TSAN Suppression
CODE:
```
race:Arc*drop
```
----------------------------------------
TITLE: Suppressing TSAN Race on std::thread::local in Rust
DESCRIPTION: This suppression addresses ThreadSanitizer false positives when using Rust's `std::thread::local`. Similar to `lazy_static`, `std::thread::local` utilizes implicit barriers that TSAN fails to recognize, leading to incorrect race condition reports.
SOURCE: https://github.com/spacejam/sled/blob/main/tsan_suppressions.txt#_snippet_2
LANGUAGE: TSAN Suppression
CODE:
```
race:std::thread::local
```
----------------------------------------
TITLE: Suppressing TSAN Race on lazy_static in Rust
DESCRIPTION: This suppression targets ThreadSanitizer false positives related to the `lazy_static` crate in Rust. `lazy_static` relies on implicit memory barriers that TSAN does not correctly detect, causing it to report races where none exist.
SOURCE: https://github.com/spacejam/sled/blob/main/tsan_suppressions.txt#_snippet_1
LANGUAGE: TSAN Suppression
CODE:
```
race:lazy_static
```

0
specs/backgroundinfo/tantivy.md Normal file
View File

735
specs/lance_implementation.md Normal file
View File

@@ -0,0 +1,735 @@
Looking at your request to ensure embedding happens with an external embedding provider configured in Redis at `config:core:aiembed:url`, I'll modify the implementation to use an external embedding service instead of local models.
## Updated Implementation with External Embedding Provider
### 1. Update Dependencies in `Cargo.toml`
```toml
[dependencies]
# ... existing dependencies ...
lance = "0.18"
lance-index = "0.18"
lance-linalg = "0.18"
arrow = "52"
arrow-array = "52"
arrow-schema = "52"
parquet = "52"
uuid = { version = "1.10", features = ["v4"] }
reqwest = { version = "0.11", features = ["json"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
base64 = "0.22"
image = "0.25"
```
### 2. Create Enhanced Lance Module with External Embedding
Create `src/lance_store.rs`:
```rust
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::Arc;
use tokio::sync::RwLock;
use arrow::array::{Float32Array, StringArray, BinaryArray, ArrayRef};
use arrow::datatypes::{DataType, Field, Schema};
use arrow::record_batch::RecordBatch;
use lance::dataset::{Dataset, WriteParams, WriteMode};
use lance::index::vector::VectorIndexParams;
use lance_index::vector::pq::PQBuildParams;
use lance_index::vector::ivf::IvfBuildParams;
use serde::{Deserialize, Serialize};
use crate::error::DBError;
use crate::cmd::Protocol;
#[derive(Debug, Serialize, Deserialize)]
struct EmbeddingRequest {
texts: Option<Vec<String>>,
images: Option<Vec<String>>, // base64 encoded
model: Option<String>,
}
#[derive(Debug, Serialize, Deserialize)]
struct EmbeddingResponse {
embeddings: Vec<Vec<f32>>,
model: String,
usage: Option<HashMap<String, u32>>,
}
pub struct LanceStore {
datasets: Arc<RwLock<HashMap<String, Arc<Dataset>>>>,
data_dir: PathBuf,
http_client: reqwest::Client,
}
impl LanceStore {
pub async fn new(data_dir: PathBuf) -> Result<Self, DBError> {
// Create data directory if it doesn't exist
std::fs::create_dir_all(&data_dir)
.map_err(|e| DBError(format!("Failed to create Lance data directory: {}", e)))?;
let http_client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.build()
.map_err(|e| DBError(format!("Failed to create HTTP client: {}", e)))?;
Ok(Self {
datasets: Arc::new(RwLock::new(HashMap::new())),
data_dir,
http_client,
})
}
/// Get embedding service URL from Redis config
async fn get_embedding_url(&self, server: &crate::server::Server) -> Result<String, DBError> {
// Get the embedding URL from Redis config
let key = "config:core:aiembed:url";
// Use HGET to retrieve the URL from Redis hash
let cmd = crate::cmd::Cmd::HGet {
key: key.to_string(),
field: "url".to_string(),
};
// Execute command to get the config
let result = cmd.run(server).await?;
match result {
Protocol::BulkString(url) => Ok(url),
Protocol::SimpleString(url) => Ok(url),
Protocol::Nil => Err(DBError(
"Embedding service URL not configured. Set it with: HSET config:core:aiembed:url url <YOUR_EMBEDDING_SERVICE_URL>".to_string()
)),
_ => Err(DBError("Invalid embedding URL configuration".to_string())),
}
}
/// Call external embedding service
async fn call_embedding_service(
&self,
server: &crate::server::Server,
texts: Option<Vec<String>>,
images: Option<Vec<String>>,
) -> Result<Vec<Vec<f32>>, DBError> {
let url = self.get_embedding_url(server).await?;
let request = EmbeddingRequest {
texts,
images,
model: None, // Let the service use its default
};
let response = self.http_client
.post(&url)
.json(&request)
.send()
.await
.map_err(|e| DBError(format!("Failed to call embedding service: {}", e)))?;
if !response.status().is_success() {
let status = response.status();
let error_text = response.text().await.unwrap_or_default();
return Err(DBError(format!(
"Embedding service returned error {}: {}",
status, error_text
)));
}
let embedding_response: EmbeddingResponse = response
.json()
.await
.map_err(|e| DBError(format!("Failed to parse embedding response: {}", e)))?;
Ok(embedding_response.embeddings)
}
pub async fn embed_text(
&self,
server: &crate::server::Server,
texts: Vec<String>
) -> Result<Vec<Vec<f32>>, DBError> {
if texts.is_empty() {
return Ok(Vec::new());
}
self.call_embedding_service(server, Some(texts), None).await
}
pub async fn embed_image(
&self,
server: &crate::server::Server,
image_bytes: Vec<u8>
) -> Result<Vec<f32>, DBError> {
// Convert image bytes to base64
let base64_image = base64::encode(&image_bytes);
let embeddings = self.call_embedding_service(
server,
None,
Some(vec![base64_image])
).await?;
embeddings.into_iter()
.next()
.ok_or_else(|| DBError("No embedding returned for image".to_string()))
}
pub async fn create_dataset(
&self,
name: &str,
schema: Schema,
) -> Result<(), DBError> {
let dataset_path = self.data_dir.join(format!("{}.lance", name));
// Create empty dataset with schema
let write_params = WriteParams {
mode: WriteMode::Create,
..Default::default()
};
// Create an empty RecordBatch with the schema
let empty_batch = RecordBatch::new_empty(Arc::new(schema));
let batches = vec![empty_batch];
let dataset = Dataset::write(
batches,
dataset_path.to_str().unwrap(),
Some(write_params)
).await
.map_err(|e| DBError(format!("Failed to create dataset: {}", e)))?;
let mut datasets = self.datasets.write().await;
datasets.insert(name.to_string(), Arc::new(dataset));
Ok(())
}
pub async fn write_vectors(
&self,
dataset_name: &str,
vectors: Vec<Vec<f32>>,
metadata: Option<HashMap<String, Vec<String>>>,
) -> Result<usize, DBError> {
let dataset_path = self.data_dir.join(format!("{}.lance", dataset_name));
// Open or get cached dataset
let dataset = self.get_or_open_dataset(dataset_name).await?;
// Build RecordBatch
let num_vectors = vectors.len();
if num_vectors == 0 {
return Ok(0);
}
let dim = vectors.first()
.ok_or_else(|| DBError("Empty vectors".to_string()))?
.len();
// Flatten vectors
let flat_vectors: Vec<f32> = vectors.into_iter().flatten().collect();
let vector_array = Float32Array::from(flat_vectors);
let vector_array = arrow::array::FixedSizeListArray::try_new_from_values(
vector_array,
dim as i32
).map_err(|e| DBError(format!("Failed to create vector array: {}", e)))?;
let mut arrays: Vec<ArrayRef> = vec![Arc::new(vector_array)];
let mut fields = vec![Field::new(
"vector",
DataType::FixedSizeList(
Arc::new(Field::new("item", DataType::Float32, true)),
dim as i32
),
false
)];
// Add metadata columns if provided
if let Some(metadata) = metadata {
for (key, values) in metadata {
if values.len() != num_vectors {
return Err(DBError(format!(
"Metadata field '{}' has {} values but expected {}",
key, values.len(), num_vectors
)));
}
let array = StringArray::from(values);
arrays.push(Arc::new(array));
fields.push(Field::new(&key, DataType::Utf8, true));
}
}
let schema = Arc::new(Schema::new(fields));
let batch = RecordBatch::try_new(schema, arrays)
.map_err(|e| DBError(format!("Failed to create RecordBatch: {}", e)))?;
// Append to dataset
let write_params = WriteParams {
mode: WriteMode::Append,
..Default::default()
};
Dataset::write(
vec![batch],
dataset_path.to_str().unwrap(),
Some(write_params)
).await
.map_err(|e| DBError(format!("Failed to write to dataset: {}", e)))?;
// Refresh cached dataset
let mut datasets = self.datasets.write().await;
datasets.remove(dataset_name);
Ok(num_vectors)
}
pub async fn search_vectors(
&self,
dataset_name: &str,
query_vector: Vec<f32>,
k: usize,
nprobes: Option<usize>,
refine_factor: Option<usize>,
) -> Result<Vec<(f32, HashMap<String, String>)>, DBError> {
let dataset = self.get_or_open_dataset(dataset_name).await?;
// Build query
let mut query = dataset.scan();
query = query.nearest(
"vector",
&query_vector,
k,
).map_err(|e| DBError(format!("Failed to build search query: {}", e)))?;
if let Some(nprobes) = nprobes {
query = query.nprobes(nprobes);
}
if let Some(refine) = refine_factor {
query = query.refine_factor(refine);
}
// Execute search
let results = query
.try_into_stream()
.await
.map_err(|e| DBError(format!("Failed to execute search: {}", e)))?
.try_collect::<Vec<_>>()
.await
.map_err(|e| DBError(format!("Failed to collect results: {}", e)))?;
// Process results
let mut output = Vec::new();
for batch in results {
// Get distances
let distances = batch
.column_by_name("_distance")
.ok_or_else(|| DBError("No distance column".to_string()))?
.as_any()
.downcast_ref::<Float32Array>()
.ok_or_else(|| DBError("Invalid distance type".to_string()))?;
// Get metadata
for i in 0..batch.num_rows() {
let distance = distances.value(i);
let mut metadata = HashMap::new();
for field in batch.schema().fields() {
if field.name() != "vector" && field.name() != "_distance" {
if let Some(col) = batch.column_by_name(field.name()) {
if let Some(str_array) = col.as_any().downcast_ref::<StringArray>() {
if !str_array.is_null(i) {
metadata.insert(
field.name().to_string(),
str_array.value(i).to_string()
);
}
}
}
}
}
output.push((distance, metadata));
}
}
Ok(output)
}
pub async fn store_multimodal(
&self,
server: &crate::server::Server,
dataset_name: &str,
text: Option<String>,
image_bytes: Option<Vec<u8>>,
metadata: HashMap<String, String>,
) -> Result<String, DBError> {
// Generate ID
let id = uuid::Uuid::new_v4().to_string();
// Generate embeddings using external service
let embedding = if let Some(text) = text.as_ref() {
self.embed_text(server, vec![text.clone()]).await?
.into_iter()
.next()
.ok_or_else(|| DBError("No embedding returned".to_string()))?
} else if let Some(img) = image_bytes.as_ref() {
self.embed_image(server, img.clone()).await?
} else {
return Err(DBError("No text or image provided".to_string()));
};
// Prepare metadata
let mut full_metadata = metadata;
full_metadata.insert("id".to_string(), id.clone());
if let Some(text) = text {
full_metadata.insert("text".to_string(), text);
}
if let Some(img) = image_bytes {
full_metadata.insert("image_base64".to_string(), base64::encode(img));
}
// Convert metadata to column vectors
let mut metadata_cols = HashMap::new();
for (key, value) in full_metadata {
metadata_cols.insert(key, vec![value]);
}
// Write to dataset
self.write_vectors(dataset_name, vec![embedding], Some(metadata_cols)).await?;
Ok(id)
}
pub async fn search_with_text(
&self,
server: &crate::server::Server,
dataset_name: &str,
query_text: String,
k: usize,
nprobes: Option<usize>,
refine_factor: Option<usize>,
) -> Result<Vec<(f32, HashMap<String, String>)>, DBError> {
// Embed the query text using external service
let embeddings = self.embed_text(server, vec![query_text]).await?;
let query_vector = embeddings.into_iter()
.next()
.ok_or_else(|| DBError("No embedding returned for query".to_string()))?;
// Search with the embedding
self.search_vectors(dataset_name, query_vector, k, nprobes, refine_factor).await
}
pub async fn create_index(
&self,
dataset_name: &str,
index_type: &str,
num_partitions: Option<usize>,
num_sub_vectors: Option<usize>,
) -> Result<(), DBError> {
let dataset = self.get_or_open_dataset(dataset_name).await?;
let mut params = VectorIndexParams::default();
match index_type.to_uppercase().as_str() {
"IVF_PQ" => {
params.ivf = IvfBuildParams {
num_partitions: num_partitions.unwrap_or(256),
..Default::default()
};
params.pq = PQBuildParams {
num_sub_vectors: num_sub_vectors.unwrap_or(16),
..Default::default()
};
}
_ => return Err(DBError(format!("Unsupported index type: {}", index_type))),
}
dataset.create_index(
&["vector"],
lance::index::IndexType::Vector,
None,
&params,
true
).await
.map_err(|e| DBError(format!("Failed to create index: {}", e)))?;
Ok(())
}
async fn get_or_open_dataset(&self, name: &str) -> Result<Arc<Dataset>, DBError> {
let mut datasets = self.datasets.write().await;
if let Some(dataset) = datasets.get(name) {
return Ok(dataset.clone());
}
let dataset_path = self.data_dir.join(format!("{}.lance", name));
if !dataset_path.exists() {
return Err(DBError(format!("Dataset '{}' does not exist", name)));
}
let dataset = Dataset::open(dataset_path.to_str().unwrap())
.await
.map_err(|e| DBError(format!("Failed to open dataset: {}", e)))?;
let dataset = Arc::new(dataset);
datasets.insert(name.to_string(), dataset.clone());
Ok(dataset)
}
pub async fn list_datasets(&self) -> Result<Vec<String>, DBError> {
let mut datasets = Vec::new();
let entries = std::fs::read_dir(&self.data_dir)
.map_err(|e| DBError(format!("Failed to read data directory: {}", e)))?;
for entry in entries {
let entry = entry.map_err(|e| DBError(format!("Failed to read entry: {}", e)))?;
let path = entry.path();
if path.is_dir() {
if let Some(name) = path.file_name() {
if let Some(name_str) = name.to_str() {
if name_str.ends_with(".lance") {
let dataset_name = name_str.trim_end_matches(".lance");
datasets.push(dataset_name.to_string());
}
}
}
}
}
Ok(datasets)
}
pub async fn drop_dataset(&self, name: &str) -> Result<(), DBError> {
// Remove from cache
let mut datasets = self.datasets.write().await;
datasets.remove(name);
// Delete from disk
let dataset_path = self.data_dir.join(format!("{}.lance", name));
if dataset_path.exists() {
std::fs::remove_dir_all(dataset_path)
.map_err(|e| DBError(format!("Failed to delete dataset: {}", e)))?;
}
Ok(())
}
pub async fn get_dataset_info(&self, name: &str) -> Result<HashMap<String, String>, DBError> {
let dataset = self.get_or_open_dataset(name).await?;
let mut info = HashMap::new();
info.insert("name".to_string(), name.to_string());
info.insert("version".to_string(), dataset.version().to_string());
info.insert("num_rows".to_string(), dataset.count_rows().await?.to_string());
// Get schema info
let schema = dataset.schema();
let fields: Vec<String> = schema.fields()
.iter()
.map(|f| format!("{}:{}", f.name(), f.data_type()))
.collect();
info.insert("schema".to_string(), fields.join(", "));
Ok(info)
}
}
```
### 3. Update Command Implementations
Update the command implementations to pass the server reference for embedding service access:
```rust
// In cmd.rs, update the lance command implementations
async fn lance_store_cmd(
server: &Server,
dataset: &str,
text: Option<String>,
image_base64: Option<String>,
metadata: HashMap<String, String>,
) -> Result<Protocol, DBError> {
let lance_store = server.lance_store()?;
// Decode image if provided
let image_bytes = if let Some(b64) = image_base64 {
Some(base64::decode(b64).map_err(|e|
DBError(format!("Invalid base64 image: {}", e)))?)
} else {
None
};
// Pass server reference for embedding service access
let id = lance_store.store_multimodal(
server, // Pass server to access Redis config
dataset,
text,
image_bytes,
metadata,
).await?;
Ok(Protocol::BulkString(id))
}
async fn lance_embed_text_cmd(
server: &Server,
texts: &[String],
) -> Result<Protocol, DBError> {
let lance_store = server.lance_store()?;
// Pass server reference for embedding service access
let embeddings = lance_store.embed_text(server, texts.to_vec()).await?;
// Return as array of vectors
let mut output = Vec::new();
for embedding in embeddings {
let vector_str = format!("[{}]",
embedding.iter()
.map(|f| f.to_string())
.collect::<Vec<_>>()
.join(",")
);
output.push(Protocol::BulkString(vector_str));
}
Ok(Protocol::Array(output))
}
async fn lance_search_text_cmd(
server: &Server,
dataset: &str,
query_text: &str,
k: usize,
nprobes: Option<usize>,
refine_factor: Option<usize>,
) -> Result<Protocol, DBError> {
let lance_store = server.lance_store()?;
// Search using text query (will be embedded automatically)
let results = lance_store.search_with_text(
server,
dataset,
query_text.to_string(),
k,
nprobes,
refine_factor,
).await?;
// Format results
let mut output = Vec::new();
for (distance, metadata) in results {
let metadata_json = serde_json::to_string(&metadata)
.unwrap_or_else(|_| "{}".to_string());
output.push(Protocol::Array(vec![
Protocol::BulkString(distance.to_string()),
Protocol::BulkString(metadata_json),
]));
}
Ok(Protocol::Array(output))
}
// Add new command for text-based search
pub enum Cmd {
// ... existing commands ...
LanceSearchText {
dataset: String,
query_text: String,
k: usize,
nprobes: Option<usize>,
refine_factor: Option<usize>,
},
}
```
## Usage Examples
### 1. Configure the Embedding Service
First, users need to configure the embedding service URL:
```bash
# Configure the embedding service endpoint
redis-cli> HSET config:core:aiembed:url url "http://localhost:8000/embeddings"
OK
# Or use a cloud service
redis-cli> HSET config:core:aiembed:url url "https://api.openai.com/v1/embeddings"
OK
```
### 2. Use Lance Commands with Automatic External Embedding
```bash
# Create a dataset
redis-cli> LANCE.CREATE products DIM 1536 SCHEMA name:string price:float category:string
OK
# Store text with automatic embedding (calls external service)
redis-cli> LANCE.STORE products TEXT "Wireless noise-canceling headphones with 30-hour battery" name:AirPods price:299.99 category:Electronics
"uuid-123-456"
# Search using text query (automatically embeds the query)
redis-cli> LANCE.SEARCH.TEXT products "best headphones for travel" K 5
1) "0.92"
2) "{\"id\":\"uuid-123\",\"name\":\"AirPods\",\"price\":\"299.99\"}"
# Get embeddings directly
redis-cli> LANCE.EMBED.TEXT "This text will be embedded"
1) "[0.123, 0.456, 0.789, ...]"
```
## External Embedding Service API Specification
The external embedding service should accept POST requests with this format:
```json
// Request
{
"texts": ["text1", "text2"], // Optional
"images": ["base64_img1"], // Optional
"model": "text-embedding-ada-002" // Optional
}
// Response
{
"embeddings": [[0.1, 0.2, ...], [0.3, 0.4, ...]],
"model": "text-embedding-ada-002",
"usage": {
"prompt_tokens": 100,
"total_tokens": 100
}
}
```
## Error Handling
The implementation includes comprehensive error handling:
1. **Missing Configuration**: Clear error message if embedding URL not configured
2. **Service Failures**: Graceful handling of embedding service errors
3. **Timeout Protection**: 30-second timeout for embedding requests
4. **Retry Logic**: Could be added for resilience
## Benefits of This Approach
1. **Flexibility**: Supports any embedding service with compatible API
2. **Cost Control**: Use your preferred embedding provider
3. **Scalability**: Embedding service can be scaled independently
4. **Consistency**: All embeddings use the same configured service
5. **Security**: API keys and endpoints stored securely in Redis
This implementation ensures that all embedding operations go through the external service configured in Redis, providing a clean separation between the vector database functionality and the embedding generation.
TODO EXTRA:
- secret for the embedding service API key