Compare commits
18 Commits
tantivy_im
...
tantivy
Author | SHA1 | Date | |
---|---|---|---|
|
e5fcaf81b5
|
||
|
65c297ca94
|
||
|
8decbf3375
|
||
|
ff0659b933
|
||
|
e9675aafed
|
||
|
142084c60f
|
||
4b3a86d73d | |||
fbcaafc86b | |||
ce1be0369a | |||
4b8216bfdb | |||
8bc372ea64 | |||
7920945986 | |||
d4d3660bac | |||
b68325016d | |||
2743cd9c81 | |||
eb07386cf4 | |||
fc7672c78a | |||
46f96fa8cf |
941
Cargo.lock
generated
941
Cargo.lock
generated
File diff suppressed because it is too large
Load Diff
@@ -1,8 +1,8 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "herodb"
|
name = "herodb"
|
||||||
version = "0.0.1"
|
version = "0.0.1"
|
||||||
authors = ["ThreeFold Tech NV"]
|
authors = ["Pin Fang <fpfangpin@hotmail.com>"]
|
||||||
edition = "2024"
|
edition = "2021"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
anyhow = "1.0.59"
|
anyhow = "1.0.59"
|
||||||
@@ -23,9 +23,7 @@ sha2 = "0.10"
|
|||||||
age = "0.10"
|
age = "0.10"
|
||||||
secrecy = "0.8"
|
secrecy = "0.8"
|
||||||
ed25519-dalek = "2"
|
ed25519-dalek = "2"
|
||||||
x25519-dalek = "2"
|
|
||||||
base64 = "0.22"
|
base64 = "0.22"
|
||||||
jsonrpsee = { version = "0.26.0", features = ["http-client", "ws-client", "server", "macros"] }
|
|
||||||
tantivy = "0.25.0"
|
tantivy = "0.25.0"
|
||||||
|
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
|
41
README.md
41
README.md
@@ -17,8 +17,6 @@ The main purpose of HeroDB is to offer a lightweight, embeddable, and Redis-comp
|
|||||||
- **Expiration**: Time-to-live (TTL) functionality for keys.
|
- **Expiration**: Time-to-live (TTL) functionality for keys.
|
||||||
- **Scanning**: Cursor-based iteration for keys and hash fields (`SCAN`, `HSCAN`).
|
- **Scanning**: Cursor-based iteration for keys and hash fields (`SCAN`, `HSCAN`).
|
||||||
- **AGE Cryptography Commands**: HeroDB-specific extensions for cryptographic operations.
|
- **AGE Cryptography Commands**: HeroDB-specific extensions for cryptographic operations.
|
||||||
- **Symmetric Encryption**: Stateless symmetric encryption using XChaCha20-Poly1305.
|
|
||||||
- **Admin Database 0**: Centralized control for database management, access control, and per-database encryption.
|
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
@@ -32,14 +30,31 @@ cargo build --release
|
|||||||
|
|
||||||
### Running HeroDB
|
### Running HeroDB
|
||||||
|
|
||||||
Launch HeroDB with the required `--admin-secret` flag, which encrypts the admin database (DB 0) and authorizes admin access. Optional flags include `--dir` for the database directory, `--port` for the TCP port (default 6379), `--sled` for the sled backend, and `--enable-rpc` to start the JSON-RPC management server on port 8080.
|
You can start HeroDB with different backends and encryption options:
|
||||||
|
|
||||||
|
#### Default `redb` Backend
|
||||||
|
|
||||||
Example:
|
|
||||||
```bash
|
```bash
|
||||||
./target/release/herodb --dir /tmp/herodb --admin-secret myadminsecret --port 6379 --enable-rpc
|
./target/release/herodb --dir /tmp/herodb_redb --port 6379
|
||||||
```
|
```
|
||||||
|
|
||||||
For detailed launch options, see [Basics](docs/basics.md).
|
#### `sled` Backend
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./target/release/herodb --dir /tmp/herodb_sled --port 6379 --sled
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `redb` with Encryption
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./target/release/herodb --dir /tmp/herodb_encrypted --port 6379 --encrypt --key mysecretkey
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `sled` with Encryption
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./target/release/herodb --dir /tmp/herodb_sled_encrypted --port 6379 --sled --encrypt --key mysecretkey
|
||||||
|
```
|
||||||
|
|
||||||
## Usage with Redis Clients
|
## Usage with Redis Clients
|
||||||
|
|
||||||
@@ -61,18 +76,6 @@ redis-cli -p 6379 SCAN 0 MATCH user:* COUNT 10
|
|||||||
# 2) 1) "user:1"
|
# 2) 1) "user:1"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Cryptography
|
|
||||||
|
|
||||||
HeroDB supports asymmetric encryption/signatures via AGE commands (X25519 for encryption, Ed25519 for signatures) in stateless or key-managed modes, and symmetric encryption via SYM commands. Keys are persisted in the admin database (DB 0) for managed modes.
|
|
||||||
|
|
||||||
For details, see [AGE Cryptography](docs/age.md) and [Basics](docs/basics.md).
|
|
||||||
|
|
||||||
## Database Management
|
|
||||||
|
|
||||||
Databases are managed via JSON-RPC API, with metadata stored in the encrypted admin database (DB 0). Databases are public by default upon creation; use RPC to set them private, requiring access keys for SELECT operations (read or readwrite based on permissions). This includes per-database encryption keys, access control, and lifecycle management.
|
|
||||||
|
|
||||||
For examples, see [JSON-RPC Examples](docs/rpc_examples.md) and [Admin DB 0 Model](docs/admin.md).
|
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
For more detailed information on commands, features, and advanced usage, please refer to the documentation:
|
For more detailed information on commands, features, and advanced usage, please refer to the documentation:
|
||||||
@@ -80,5 +83,3 @@ For more detailed information on commands, features, and advanced usage, please
|
|||||||
- [Basics](docs/basics.md)
|
- [Basics](docs/basics.md)
|
||||||
- [Supported Commands](docs/cmds.md)
|
- [Supported Commands](docs/cmds.md)
|
||||||
- [AGE Cryptography](docs/age.md)
|
- [AGE Cryptography](docs/age.md)
|
||||||
- [Admin DB 0 Model (access control, per-db encryption)](docs/admin.md)
|
|
||||||
- [JSON-RPC Examples (management API)](docs/rpc_examples.md)
|
|
181
docs/admin.md
181
docs/admin.md
@@ -1,181 +0,0 @@
|
|||||||
# Admin Database 0 (`0.db`)
|
|
||||||
|
|
||||||
This page explains what the Admin Database `DB 0` is, why HeroDB uses it, and how to work with it as a developer and end-user. It’s a practical guide covering how databases are created, listed, secured with access keys, and encrypted using per-database secrets.
|
|
||||||
|
|
||||||
## What is `DB 0`?
|
|
||||||
|
|
||||||
`DB 0` is the control-plane for a HeroDB instance. It stores metadata for all user databases (`db_id >= 1`) so the server can:
|
|
||||||
- Know which databases exist (without scanning the filesystem)
|
|
||||||
- Enforce access control (public/private with access keys)
|
|
||||||
- Enforce per-database encryption (whether a given database must be opened encrypted and with which write-only key)
|
|
||||||
|
|
||||||
`DB 0` itself is always encrypted with the admin secret (the process-level secret provided at startup).
|
|
||||||
|
|
||||||
## How `DB 0` is created and secured
|
|
||||||
|
|
||||||
- `DB 0` lives at `<base_dir>/0.db`
|
|
||||||
- It is always encrypted using the `admin secret` provided at process startup (using the `--admin-secret <secret>` CLI flag)
|
|
||||||
- Only clients that provide the correct admin secret can `SELECT 0` (see “`SELECT` + `KEY`” below)
|
|
||||||
|
|
||||||
At startup, the server bootstraps `DB 0` (initializes counters and structures) if it’s missing.
|
|
||||||
|
|
||||||
## Metadata stored in `DB 0`
|
|
||||||
|
|
||||||
Keys in `DB 0` (internal layout, but useful to understand how things work):
|
|
||||||
|
|
||||||
- `admin:next_id`
|
|
||||||
- String counter holding the next id to allocate (initialized to `"1"`)
|
|
||||||
|
|
||||||
- `admin:dbs`
|
|
||||||
- A hash acting as a set of existing database ids
|
|
||||||
- field = id (as string), value = `"1"`
|
|
||||||
|
|
||||||
- `meta:db:<id>`
|
|
||||||
- A hash holding db-level metadata
|
|
||||||
- field `public` = `"true"` or `"false"` (defaults to `true` if missing)
|
|
||||||
|
|
||||||
- `meta:db:<id>:keys`
|
|
||||||
- A hash mapping access-key hashes to the string `Permission:created_at_seconds`
|
|
||||||
- Examples: `Read:1713456789` or `ReadWrite:1713456789`
|
|
||||||
- The plaintext access keys are never stored; only their `SHA-256` hashes are kept
|
|
||||||
|
|
||||||
- `meta:db:<id>:enc`
|
|
||||||
- A string holding the per-database encryption key used to open `<id>.db` encrypted
|
|
||||||
- This value is write-only from the perspective of the management APIs (it’s set at creation and never returned)
|
|
||||||
|
|
||||||
- `age:key:<name>`
|
|
||||||
- Base64-encoded X25519 recipient (public encryption key) for named AGE keys
|
|
||||||
- `age:privkey:<name>`
|
|
||||||
- Base64-encoded X25519 identity (secret encryption key) for named AGE keys
|
|
||||||
- `age:signpub:<name>`
|
|
||||||
- Base64-encoded Ed25519 verify public key for named AGE keys
|
|
||||||
- `age:signpriv:<name>`
|
|
||||||
- Base64-encoded Ed25519 signing secret key for named AGE keys
|
|
||||||
|
|
||||||
> You don’t need to manipulate these keys directly; they’re listed to clarify the model. AGE keys are managed via AGE commands.
|
|
||||||
|
|
||||||
## Database lifecycle
|
|
||||||
|
|
||||||
1) Create a database (via JSON-RPC)
|
|
||||||
- The server allocates an id from `admin:next_id`, registers it in `admin:dbs`, and defaults the database to `public=true`
|
|
||||||
- If you pass an optional `encryption_key` during creation, the server persists it in `meta:db:<id>:enc`. That database will be opened in encrypted mode from then on
|
|
||||||
|
|
||||||
2) Open and use a database
|
|
||||||
- Clients select a database over RESP using `SELECT`
|
|
||||||
- Authorization and encryption state are enforced using `DB 0` metadata
|
|
||||||
|
|
||||||
3) Delete database files
|
|
||||||
- Removing `<id>.db` removes the physical storage
|
|
||||||
- `DB 0` remains the source of truth for existence and may be updated by future management methods as the system evolves
|
|
||||||
|
|
||||||
## Access control model
|
|
||||||
|
|
||||||
- Public database (default)
|
|
||||||
- Anyone can `SELECT <id>` with no key, and will get `ReadWrite` permission
|
|
||||||
- Private database
|
|
||||||
- You must provide an access key when selecting the database
|
|
||||||
- The server hashes the provided key with `SHA-256` and checks membership in `meta:db:<id>:keys`
|
|
||||||
- Permissions are `Read` or `ReadWrite` depending on how the key was added
|
|
||||||
- Admin `DB 0`
|
|
||||||
- Requires the exact admin secret as the `KEY` argument to `SELECT 0`
|
|
||||||
- Permission is `ReadWrite` when the secret matches
|
|
||||||
|
|
||||||
### How to select databases with optional `KEY`
|
|
||||||
|
|
||||||
- Public DB (no key required)
|
|
||||||
- `SELECT <id>`
|
|
||||||
|
|
||||||
- Private DB (access key required)
|
|
||||||
- `SELECT <id> KEY <plaintext_key>`
|
|
||||||
|
|
||||||
- Admin `DB 0` (admin secret required)
|
|
||||||
- `SELECT 0 KEY <admin_secret>`
|
|
||||||
|
|
||||||
Examples (using `redis-cli`):
|
|
||||||
```bash
|
|
||||||
# Public database
|
|
||||||
redis-cli -p $PORT SELECT 1
|
|
||||||
# → OK
|
|
||||||
|
|
||||||
# Private database
|
|
||||||
redis-cli -p $PORT SELECT 2 KEY my-db2-access-key
|
|
||||||
# → OK
|
|
||||||
|
|
||||||
# Admin DB 0
|
|
||||||
redis-cli -p $PORT SELECT 0 KEY my-admin-secret
|
|
||||||
# → OK
|
|
||||||
```
|
|
||||||
|
|
||||||
## Per-database encryption
|
|
||||||
|
|
||||||
- At database creation, you can provide an optional per-db encryption key
|
|
||||||
- If provided, the server persists that key in `DB 0` as `meta:db:<id>:enc`
|
|
||||||
- When you later open the database, the engine checks whether `meta:db:<id>:enc` exists to decide if it must open `<id>.db` in encrypted mode
|
|
||||||
- The per-db key is not returned by RPC—it is considered write-only configuration data
|
|
||||||
|
|
||||||
Operationally:
|
|
||||||
- Create with encryption: pass a non-null `encryption_key` to the `createDatabase` RPC
|
|
||||||
- Open later: simply `SELECT` the database; encryption is transparent to clients
|
|
||||||
|
|
||||||
## Management via JSON-RPC
|
|
||||||
|
|
||||||
You can manage databases using the management RPC (namespaced `herodb.*`). Typical operations:
|
|
||||||
- `createDatabase(backend, config, encryption_key?)`
|
|
||||||
- Allocates a new id, sets optional encryption key
|
|
||||||
- `listDatabases()`
|
|
||||||
- Lists database ids and info (including whether storage is currently encrypted)
|
|
||||||
- `getDatabaseInfo(db_id)`
|
|
||||||
- Returns details: backend, encrypted flag, size on disk, `key_count`, timestamps, etc.
|
|
||||||
- `addAccessKey(db_id, key, permissions)`
|
|
||||||
- Adds a `Read` or `ReadWrite` access key (permissions = `"read"` | `"readwrite"`)
|
|
||||||
- `listAccessKeys(db_id)`
|
|
||||||
- Returns hashes and permissions; you can use these hashes to delete keys
|
|
||||||
- `deleteAccessKey(db_id, key_hash)`
|
|
||||||
- Removes a key by its hash
|
|
||||||
- `setDatabasePublic(db_id, public)`
|
|
||||||
- Toggles public/private
|
|
||||||
|
|
||||||
Copyable JSON examples are provided in the [RPC examples documentation](./rpc_examples.md).
|
|
||||||
|
|
||||||
## Typical flows
|
|
||||||
|
|
||||||
1) Public, unencrypted database
|
|
||||||
- Create a new database without an encryption key
|
|
||||||
- Clients can immediately `SELECT <id>` without a key
|
|
||||||
- You can later make it private and add keys if needed
|
|
||||||
|
|
||||||
2) Private, encrypted database
|
|
||||||
- Create passing an `encryption_key`
|
|
||||||
- Mark it private (`setDatabasePublic false`) and add access keys
|
|
||||||
- Clients must use `SELECT <id> KEY <plaintext_access_key>`
|
|
||||||
- Storage opens in encrypted mode automatically
|
|
||||||
|
|
||||||
## Security notes
|
|
||||||
|
|
||||||
- Only `SHA-256` hashes of access keys are stored in `DB 0`; keep plaintext keys safe on the client side
|
|
||||||
- The per-db encryption key is never exposed via the API after it is set
|
|
||||||
- The admin secret must be kept secure; anyone with it can `SELECT 0` and perform administrative actions
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
- `ERR invalid access key` when selecting a private db
|
|
||||||
- Ensure you passed the `KEY` argument: `SELECT <id> KEY <plaintext_key>`
|
|
||||||
- If you recently added the key, confirm the permissions and that you used the exact plaintext (hash must match)
|
|
||||||
|
|
||||||
- `Database X not found`
|
|
||||||
- The id isn’t registered in `DB 0` (`admin:dbs`). Use the management APIs to create or list databases
|
|
||||||
|
|
||||||
- Cannot `SELECT 0`
|
|
||||||
- The `KEY` must be the exact admin secret passed at server startup
|
|
||||||
|
|
||||||
## Reference
|
|
||||||
|
|
||||||
- Admin metadata lives in `DB 0` (`0.db`) and controls:
|
|
||||||
- Existence: `admin:dbs`
|
|
||||||
- Access: `meta:db:<id>.public` and `meta:db:<id>:keys`
|
|
||||||
- Encryption: `meta:db:<id>:enc`
|
|
||||||
|
|
||||||
For command examples and management payloads:
|
|
||||||
- RESP command basics: `docs/basics.md`
|
|
||||||
- Supported commands: `docs/cmds.md`
|
|
||||||
- JSON-RPC examples: `docs/rpc_examples.md`
|
|
242
docs/age.md
242
docs/age.md
@@ -1,96 +1,188 @@
|
|||||||
# HeroDB AGE Cryptography
|
# HeroDB AGE usage: Stateless vs Key‑Managed
|
||||||
|
|
||||||
HeroDB provides AGE-based asymmetric encryption and digital signatures over the Redis protocol using X25519 for encryption and Ed25519 for signatures. Keys can be used in stateless (ephemeral) or key-managed (persistent, named) modes.
|
This document explains how to use the AGE cryptography commands exposed by HeroDB over the Redis protocol in two modes:
|
||||||
|
- Stateless (ephemeral keys; nothing stored on the server)
|
||||||
|
- Key‑managed (server‑persisted, named keys)
|
||||||
|
|
||||||
In key-managed mode, HeroDB uses a unified keypair concept: a single Ed25519 signing key is deterministically derived into X25519 keys for encryption, allowing one keypair to handle both encryption and signatures transparently.
|
If you are new to the codebase, the exact tests that exercise these behaviors are:
|
||||||
|
- [rust.test_07_age_stateless_suite()](herodb/tests/usage_suite.rs:495)
|
||||||
|
- [rust.test_08_age_persistent_named_suite()](herodb/tests/usage_suite.rs:555)
|
||||||
|
|
||||||
## Cryptographic Algorithms
|
Implementation entry points:
|
||||||
|
- [herodb/src/age.rs](herodb/src/age.rs)
|
||||||
|
- Dispatch from [herodb/src/cmd.rs](herodb/src/cmd.rs)
|
||||||
|
|
||||||
### X25519 (Encryption)
|
Note: Database-at-rest encryption flags in the test harness are unrelated to AGE commands; those flags control storage-level encryption of DB files. See the harness near [rust.start_test_server()](herodb/tests/usage_suite.rs:10).
|
||||||
- Elliptic-curve Diffie-Hellman key exchange for symmetric key derivation.
|
|
||||||
- Used for encrypting/decrypting messages.
|
|
||||||
|
|
||||||
### Ed25519 (Signatures)
|
## Quick start
|
||||||
- EdDSA digital signatures for message authentication.
|
|
||||||
- Used for signing/verifying messages.
|
|
||||||
|
|
||||||
### Key Derivation
|
Assuming the server is running on localhost on some $PORT:
|
||||||
Ed25519 signing keys are deterministically converted to X25519 keys for encryption. This enables a single keypair to support both operations without additional keys. Derivation uses the Ed25519 secret scalar clamped for X25519.
|
|
||||||
|
|
||||||
In named keypairs, Ed25519 keys are stored, and X25519 keys are derived on-demand and cached.
|
|
||||||
|
|
||||||
## Stateless Mode (Ephemeral Keys)
|
|
||||||
No server-side storage; keys are provided with each command.
|
|
||||||
|
|
||||||
Available commands:
|
|
||||||
- `AGE GENENC`: Generate ephemeral X25519 keypair. Returns `[recipient, identity]`.
|
|
||||||
- `AGE GENSIGN`: Generate ephemeral Ed25519 keypair. Returns `[verify_pub, sign_secret]`.
|
|
||||||
- `AGE ENCRYPT <recipient> <message>`: Encrypt message. Returns base64 ciphertext.
|
|
||||||
- `AGE DECRYPT <identity> <ciphertext_b64>`: Decrypt ciphertext. Returns plaintext.
|
|
||||||
- `AGE SIGN <sign_secret> <message>`: Sign message. Returns base64 signature.
|
|
||||||
- `AGE VERIFY <verify_pub> <message> <signature_b64>`: Verify signature. Returns 1 (valid) or 0 (invalid).
|
|
||||||
|
|
||||||
Example:
|
|
||||||
```bash
|
```bash
|
||||||
redis-cli AGE GENENC
|
~/code/git.ourworld.tf/herocode/herodb/herodb/build.sh
|
||||||
# → 1) "age1qz..." # recipient (X25519 public)
|
~/code/git.ourworld.tf/herocode/herodb/target/release/herodb --dir /tmp/data --debug --$PORT 6381 --encryption-key 1234 --encrypt
|
||||||
# 2) "AGE-SECRET-KEY-1..." # identity (X25519 secret)
|
```
|
||||||
|
|
||||||
redis-cli AGE ENCRYPT "age1qz..." "hello"
|
|
||||||
# → base64_ciphertext
|
|
||||||
|
|
||||||
redis-cli AGE DECRYPT "AGE-SECRET-KEY-1..." base64_ciphertext
|
```bash
|
||||||
|
export PORT=6381
|
||||||
|
# Generate an ephemeral keypair and encrypt/decrypt a message (stateless mode)
|
||||||
|
redis-cli -p $PORT AGE GENENC
|
||||||
|
# → returns an array: [recipient, identity]
|
||||||
|
|
||||||
|
redis-cli -p $PORT AGE ENCRYPT <recipient> "hello world"
|
||||||
|
# → returns ciphertext (base64 in a bulk string)
|
||||||
|
|
||||||
|
redis-cli -p $PORT AGE DECRYPT <identity> <ciphertext_b64>
|
||||||
|
# → returns "hello world"
|
||||||
|
```
|
||||||
|
|
||||||
|
For key‑managed mode, generate a named key once and reference it by name afterwards:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
redis-cli -p $PORT AGE KEYGEN app1
|
||||||
|
# → persists encryption keypair under name "app1"
|
||||||
|
|
||||||
|
redis-cli -p $PORT AGE ENCRYPTNAME app1 "hello"
|
||||||
|
redis-cli -p $PORT AGE DECRYPTNAME app1 <ciphertext_b64>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stateless AGE (ephemeral)
|
||||||
|
|
||||||
|
Characteristics
|
||||||
|
|
||||||
|
- No server‑side storage of keys.
|
||||||
|
- You pass the actual key material with every call.
|
||||||
|
- Not listable via AGE LIST.
|
||||||
|
|
||||||
|
Commands and examples
|
||||||
|
|
||||||
|
1) Ephemeral encryption keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate an ephemeral encryption keypair
|
||||||
|
redis-cli -p $PORT AGE GENENC
|
||||||
|
# Example output (abridged):
|
||||||
|
# 1) "age1qz..." # recipient (public key) = can be used by others e.g. to verify what I sign
|
||||||
|
# 2) "AGE-SECRET-KEY-1..." # identity (secret) = is like my private, cannot lose this one
|
||||||
|
|
||||||
|
# Encrypt with the recipient public key
|
||||||
|
redis-cli -p $PORT AGE ENCRYPT "age1qz..." "hello world"
|
||||||
|
|
||||||
|
# → returns bulk string payload: base64 ciphertext (encrypted content)
|
||||||
|
|
||||||
|
# Decrypt with the identity (secret) in other words your private key
|
||||||
|
redis-cli -p $PORT AGE DECRYPT "AGE-SECRET-KEY-1..." "<ciphertext_b64>"
|
||||||
|
# → "hello world"
|
||||||
|
```
|
||||||
|
|
||||||
|
2) Ephemeral signing keys
|
||||||
|
|
||||||
|
> ? is this same as my private key
|
||||||
|
|
||||||
|
```bash
|
||||||
|
|
||||||
|
# Generate an ephemeral signing keypair
|
||||||
|
redis-cli -p $PORT AGE GENSIGN
|
||||||
|
# Example output:
|
||||||
|
# 1) "<verify_pub_b64>"
|
||||||
|
# 2) "<sign_secret_b64>"
|
||||||
|
|
||||||
|
# Sign a message with the secret
|
||||||
|
redis-cli -p $PORT AGE SIGN "<sign_secret_b64>" "msg"
|
||||||
|
# → returns "<signature_b64>"
|
||||||
|
|
||||||
|
# Verify with the public key
|
||||||
|
redis-cli -p $PORT AGE VERIFY "<verify_pub_b64>" "msg" "<signature_b64>"
|
||||||
|
# → 1 (valid) or 0 (invalid)
|
||||||
|
```
|
||||||
|
|
||||||
|
When to use
|
||||||
|
- You do not want the server to store private keys.
|
||||||
|
- You already manage key material on the client side.
|
||||||
|
- You need ad‑hoc operations without persistence.
|
||||||
|
|
||||||
|
Reference test: [rust.test_07_age_stateless_suite()](herodb/tests/usage_suite.rs:495)
|
||||||
|
|
||||||
|
## Key‑managed AGE (persistent, named)
|
||||||
|
|
||||||
|
Characteristics
|
||||||
|
- Server generates and persists keypairs under a chosen name.
|
||||||
|
- Clients refer to keys by name; raw secrets are not supplied on each call.
|
||||||
|
- Keys are discoverable via AGE LIST.
|
||||||
|
|
||||||
|
Commands and examples
|
||||||
|
|
||||||
|
1) Named encryption keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create/persist a named encryption keypair
|
||||||
|
redis-cli -p $PORT AGE KEYGEN app1
|
||||||
|
# → returns [recipient, identity] but also stores them under name "app1"
|
||||||
|
|
||||||
|
> TODO: should not return identity (security, but there can be separate function to export it e.g. AGE EXPORTKEY app1)
|
||||||
|
|
||||||
|
# Encrypt using the stored public key
|
||||||
|
redis-cli -p $PORT AGE ENCRYPTNAME app1 "hello"
|
||||||
|
# → returns bulk string payload: base64 ciphertext
|
||||||
|
|
||||||
|
# Decrypt using the stored secret
|
||||||
|
redis-cli -p $PORT AGE DECRYPTNAME app1 "<ciphertext_b64>"
|
||||||
# → "hello"
|
# → "hello"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Key-Managed Mode (Persistent Named Keys)
|
2) Named signing keys
|
||||||
Keys are stored server-side under names. Supports unified keypairs for both encryption and signatures.
|
|
||||||
|
|
||||||
Available commands:
|
|
||||||
- `AGE KEYGEN <name>`: Generate and store unified keypair. Returns `[recipient, identity]` in age format.
|
|
||||||
- `AGE SIGNKEYGEN <name>`: Generate and store Ed25519 signing keypair. Returns `[verify_pub, sign_secret]`.
|
|
||||||
- `AGE ENCRYPTNAME <name> <message>`: Encrypt with named key. Returns base64 ciphertext.
|
|
||||||
- `AGE DECRYPTNAME <name> <ciphertext_b64>`: Decrypt with named key. Returns plaintext.
|
|
||||||
- `AGE SIGNNAME <name> <message>`: Sign with named key. Returns base64 signature.
|
|
||||||
- `AGE VERIFYNAME <name> <message> <signature_b64>`: Verify with named key. Returns 1 or 0.
|
|
||||||
- `AGE LIST`: List all stored key names. Returns sorted array of names.
|
|
||||||
|
|
||||||
### AGE LIST Output
|
|
||||||
Returns a flat, deduplicated, sorted array of key names (strings). Each name corresponds to a stored keypair, which may include encryption keys (X25519), signing keys (Ed25519), or both.
|
|
||||||
|
|
||||||
Output format: `["name1", "name2", ...]`
|
|
||||||
|
|
||||||
Example:
|
|
||||||
```bash
|
```bash
|
||||||
redis-cli AGE LIST
|
# Create/persist a named signing keypair
|
||||||
# → 1) "<named_keypair_1>"
|
redis-cli -p $PORT AGE SIGNKEYGEN app1
|
||||||
# 2) "<named_keypair_2>"
|
# → returns [verify_pub_b64, sign_secret_b64] and stores under name "app1"
|
||||||
|
|
||||||
|
> TODO: should not return sign_secret_b64 (for security, but there can be separate function to export it e.g. AGE EXPORTSIGNKEY app1)
|
||||||
|
|
||||||
|
# Sign using the stored secret
|
||||||
|
redis-cli -p $PORT AGE SIGNNAME app1 "msg"
|
||||||
|
# → returns "<signature_b64>"
|
||||||
|
|
||||||
|
# Verify using the stored public key
|
||||||
|
redis-cli -p $PORT AGE VERIFYNAME app1 "msg" "<signature_b64>"
|
||||||
|
# → 1 (valid) or 0 (invalid)
|
||||||
```
|
```
|
||||||
|
|
||||||
For unified keypairs (from `AGE KEYGEN`), the name handles both encryption (derived X25519) and signatures (stored Ed25519) transparently.
|
3) List stored AGE keys
|
||||||
|
|
||||||
Example with named keys:
|
|
||||||
```bash
|
```bash
|
||||||
redis-cli AGE KEYGEN app1
|
redis-cli -p $PORT AGE LIST
|
||||||
# → 1) "age1..." # recipient
|
# Example output includes labels such as "encpub" and your key names (e.g., "app1")
|
||||||
# 2) "AGE-SECRET-KEY-1..." # identity
|
|
||||||
|
|
||||||
redis-cli AGE ENCRYPTNAME app1 "secret message"
|
|
||||||
# → base64_ciphertext
|
|
||||||
|
|
||||||
redis-cli AGE DECRYPTNAME app1 base64_ciphertext
|
|
||||||
# → "secret message"
|
|
||||||
|
|
||||||
redis-cli AGE SIGNNAME app1 "message"
|
|
||||||
# → base64_signature
|
|
||||||
|
|
||||||
redis-cli AGE VERIFYNAME app1 "message" base64_signature
|
|
||||||
# → 1
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Choosing a Mode
|
When to use
|
||||||
- **Stateless**: For ad-hoc operations without persistence; client manages keys.
|
- You want centralized key storage/rotation and fewer secrets on the client.
|
||||||
- **Key-managed**: For centralized key lifecycle; server stores keys for convenience and discoverability.
|
- You need names/labels for workflows and can trust the server with secrets.
|
||||||
|
- You want discoverability (AGE LIST) and simpler client commands.
|
||||||
|
|
||||||
Implementation: [herodb/src/age.rs](herodb/src/age.rs) <br>
|
Reference test: [rust.test_08_age_persistent_named_suite()](herodb/tests/usage_suite.rs:555)
|
||||||
Tests: [herodb/tests/usage_suite.rs](herodb/tests/usage_suite.rs)
|
|
||||||
|
## Choosing a mode
|
||||||
|
|
||||||
|
- Prefer Stateless when:
|
||||||
|
- Minimizing server trust for secret material is the priority.
|
||||||
|
- Clients already have a secure mechanism to store/distribute keys.
|
||||||
|
- Prefer Key‑managed when:
|
||||||
|
- Centralized lifecycle, naming, and discoverability are beneficial.
|
||||||
|
- You plan to integrate rotation, ACLs, or auditability on the server side.
|
||||||
|
|
||||||
|
## Security notes
|
||||||
|
|
||||||
|
- Treat identities and signing secrets as sensitive; avoid logging them.
|
||||||
|
- For key‑managed mode, ensure server storage (and backups) are protected.
|
||||||
|
- AGE operations here are application‑level crypto and are distinct from database-at-rest encryption configured in the test harness.
|
||||||
|
|
||||||
|
## Repository pointers
|
||||||
|
|
||||||
|
- Stateless examples in tests: [rust.test_07_age_stateless_suite()](herodb/tests/usage_suite.rs:495)
|
||||||
|
- Key‑managed examples in tests: [rust.test_08_age_persistent_named_suite()](herodb/tests/usage_suite.rs:555)
|
||||||
|
- AGE implementation: [herodb/src/age.rs](herodb/src/age.rs)
|
||||||
|
- Command dispatch: [herodb/src/cmd.rs](herodb/src/cmd.rs)
|
||||||
|
- Bash demo: [herodb/examples/age_bash_demo.sh](herodb/examples/age_bash_demo.sh)
|
||||||
|
- Rust persistent demo: [herodb/examples/age_persist_demo.rs](herodb/examples/age_persist_demo.rs)
|
||||||
|
- Additional notes: [herodb/instructions/encrypt.md](herodb/instructions/encrypt.md)
|
103
docs/basics.md
103
docs/basics.md
@@ -1,58 +1,4 @@
|
|||||||
# HeroDB Basics
|
Here's an expanded version of the cmds.md documentation to include the list commands:
|
||||||
|
|
||||||
## Launching HeroDB
|
|
||||||
|
|
||||||
To launch HeroDB, use the binary with required and optional flags. The `--admin-secret` flag is mandatory, encrypting the admin database (DB 0) and authorizing admin access.
|
|
||||||
|
|
||||||
### Launch Flags
|
|
||||||
- `--dir <path>`: Directory for database files (default: current directory).
|
|
||||||
- `--port <port>`: TCP port for Redis protocol (default: 6379).
|
|
||||||
- `--debug`: Enable debug logging.
|
|
||||||
- `--sled`: Use Sled backend (default: Redb).
|
|
||||||
- `--enable-rpc`: Start JSON-RPC management server on port 8080.
|
|
||||||
- `--rpc-port <port>`: Custom RPC port (default: 8080).
|
|
||||||
- `--admin-secret <secret>`: Required secret for DB 0 encryption and admin access.
|
|
||||||
|
|
||||||
Example:
|
|
||||||
```bash
|
|
||||||
./target/release/herodb --dir /tmp/herodb --admin-secret mysecret --port 6379 --enable-rpc
|
|
||||||
```
|
|
||||||
|
|
||||||
Deprecated flags (`--encrypt`, `--encryption-key`) are ignored for data DBs; per-database encryption is managed via RPC.
|
|
||||||
|
|
||||||
## Admin Database (DB 0)
|
|
||||||
|
|
||||||
DB 0 acts as the administrative database instance, storing metadata for all user databases (IDs >= 1). It controls existence, access control, and per-database encryption. DB 0 is always encrypted with the `--admin-secret`.
|
|
||||||
|
|
||||||
When creating a new database, DB 0 allocates an ID, registers it, and optionally stores a per-database encryption key (write-only). Databases are public by default; use RPC to set them private, requiring access keys for SELECT (read or readwrite based on permissions). Keys are persisted in DB 0 for managed AGE operations.
|
|
||||||
|
|
||||||
Access DB 0 with `SELECT 0 KEY <admin-secret>`.
|
|
||||||
|
|
||||||
## Symmetric Encryption
|
|
||||||
|
|
||||||
HeroDB supports stateless symmetric encryption via SYM commands, using XChaCha20-Poly1305 AEAD.
|
|
||||||
|
|
||||||
Commands:
|
|
||||||
- `SYM KEYGEN`: Generate 32-byte key. Returns base64-encoded key.
|
|
||||||
- `SYM ENCRYPT <key_b64> <message>`: Encrypt message. Returns base64 ciphertext.
|
|
||||||
- `SYM DECRYPT <key_b64> <ciphertext_b64>`: Decrypt. Returns plaintext.
|
|
||||||
|
|
||||||
Example:
|
|
||||||
```bash
|
|
||||||
redis-cli SYM KEYGEN
|
|
||||||
# → base64_key
|
|
||||||
|
|
||||||
redis-cli SYM ENCRYPT base64_key "secret"
|
|
||||||
# → base64_ciphertext
|
|
||||||
|
|
||||||
redis-cli SYM DECRYPT base64_key base64_ciphertext
|
|
||||||
# → "secret"
|
|
||||||
```
|
|
||||||
|
|
||||||
## RPC Options
|
|
||||||
|
|
||||||
Enable the JSON-RPC server with `--enable-rpc` for database management. Methods include creating databases, managing access keys, and setting encryption. See [JSON-RPC Examples](./rpc_examples.md) for payloads.
|
|
||||||
|
|
||||||
# HeroDB Commands
|
# HeroDB Commands
|
||||||
|
|
||||||
HeroDB implements a subset of Redis commands over the Redis protocol. This document describes the available commands and their usage.
|
HeroDB implements a subset of Redis commands over the Redis protocol. This document describes the available commands and their usage.
|
||||||
@@ -629,29 +575,6 @@ redis-cli -p $PORT AGE LIST
|
|||||||
# 2) "keyname2"
|
# 2) "keyname2"
|
||||||
```
|
```
|
||||||
|
|
||||||
## SYM Commands
|
|
||||||
|
|
||||||
### SYM KEYGEN
|
|
||||||
Generate a symmetric encryption key.
|
|
||||||
```bash
|
|
||||||
redis-cli -p $PORT SYM KEYGEN
|
|
||||||
# → base64_encoded_32byte_key
|
|
||||||
```
|
|
||||||
|
|
||||||
### SYM ENCRYPT
|
|
||||||
Encrypt a message with a symmetric key.
|
|
||||||
```bash
|
|
||||||
redis-cli -p $PORT SYM ENCRYPT <key_b64> "message"
|
|
||||||
# → base64_encoded_ciphertext
|
|
||||||
```
|
|
||||||
|
|
||||||
### SYM DECRYPT
|
|
||||||
Decrypt a ciphertext with a symmetric key.
|
|
||||||
```bash
|
|
||||||
redis-cli -p $PORT SYM DECRYPT <key_b64> <ciphertext_b64>
|
|
||||||
# → decrypted_message
|
|
||||||
```
|
|
||||||
|
|
||||||
## Server Information Commands
|
## Server Information Commands
|
||||||
|
|
||||||
### INFO
|
### INFO
|
||||||
@@ -698,27 +621,3 @@ This expanded documentation includes all the list commands that were implemented
|
|||||||
10. LINDEX - get element by index
|
10. LINDEX - get element by index
|
||||||
11. LRANGE - get range of elements
|
11. LRANGE - get range of elements
|
||||||
|
|
||||||
|
|
||||||
## Updated Database Selection and Access Keys
|
|
||||||
|
|
||||||
HeroDB uses an `Admin DB 0` to control database existence, access, and encryption. Access to data DBs can be public (no key) or private (requires a key). See detailed model in `docs/admin.md`.
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Public database (no key required)
|
|
||||||
redis-cli -p $PORT SELECT 1
|
|
||||||
# → OK
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Private database (requires access key)
|
|
||||||
redis-cli -p $PORT SELECT 2 KEY my-db2-access-key
|
|
||||||
# → OK
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Admin DB 0 (requires admin secret)
|
|
||||||
redis-cli -p $PORT SELECT 0 KEY my-admin-secret
|
|
||||||
# → OK
|
|
||||||
```
|
|
||||||
|
23
docs/cmds.md
23
docs/cmds.md
@@ -123,26 +123,3 @@ redis-cli -p 6379 --rdb dump.rdb
|
|||||||
# Import to sled
|
# Import to sled
|
||||||
redis-cli -p 6381 --pipe < dump.rdb
|
redis-cli -p 6381 --pipe < dump.rdb
|
||||||
```
|
```
|
||||||
|
|
||||||
## Authentication and Database Selection
|
|
||||||
|
|
||||||
HeroDB uses an `Admin DB 0` to govern database existence, access and per-db encryption. Access control is enforced via `Admin DB 0` metadata. See the full model in `docs/admin.md`.
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
```bash
|
|
||||||
# Public database (no key required)
|
|
||||||
redis-cli -p $PORT SELECT 1
|
|
||||||
# → OK
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Private database (requires access key)
|
|
||||||
redis-cli -p $PORT SELECT 2 KEY my-db2-access-key
|
|
||||||
# → OK
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Admin DB 0 (requires admin secret)
|
|
||||||
redis-cli -p $PORT SELECT 0 KEY my-admin-secret
|
|
||||||
# → OK
|
|
||||||
```
|
|
@@ -1,141 +0,0 @@
|
|||||||
# HeroDB JSON-RPC Examples
|
|
||||||
|
|
||||||
These examples show full JSON-RPC 2.0 payloads for managing HeroDB via the RPC API (enable with `--enable-rpc`). Methods are named as `hero_<function>`. Params are positional arrays; enum values are strings (e.g., `"Redb"`). Copy-paste into Postman or similar clients.
|
|
||||||
|
|
||||||
## Database Management
|
|
||||||
|
|
||||||
### Create Database
|
|
||||||
Creates a new database with optional per-database encryption key (stored write-only in Admin DB 0).
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "hero_createDatabase",
|
|
||||||
"params": [
|
|
||||||
"Redb",
|
|
||||||
{ "name": null, "storage_path": null, "max_size": null, "redis_version": null },
|
|
||||||
null
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
With encryption:
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 2,
|
|
||||||
"method": "hero_createDatabase",
|
|
||||||
"params": [
|
|
||||||
"Sled",
|
|
||||||
{ "name": "secure-db", "storage_path": null, "max_size": null, "redis_version": null },
|
|
||||||
"my-per-db-encryption-key"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### List Databases
|
|
||||||
Returns array of database infos (id, backend, encrypted status, size, etc.).
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 3,
|
|
||||||
"method": "hero_listDatabases",
|
|
||||||
"params": []
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Get Database Info
|
|
||||||
Retrieves detailed info for a specific database.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 4,
|
|
||||||
"method": "hero_getDatabaseInfo",
|
|
||||||
"params": [1]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Delete Database
|
|
||||||
Removes physical database file; metadata remains in Admin DB 0.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 5,
|
|
||||||
"method": "hero_deleteDatabase",
|
|
||||||
"params": [1]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Access Control
|
|
||||||
|
|
||||||
### Add Access Key
|
|
||||||
Adds a hashed access key for private databases. Permissions: `"read"` or `"readwrite"`.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 6,
|
|
||||||
"method": "hero_addAccessKey",
|
|
||||||
"params": [2, "my-access-key", "readwrite"]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### List Access Keys
|
|
||||||
Returns array of key hashes, permissions, and creation timestamps.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 7,
|
|
||||||
"method": "hero_listAccessKeys",
|
|
||||||
"params": [2]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Delete Access Key
|
|
||||||
Removes key by its SHA-256 hash.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 8,
|
|
||||||
"method": "hero_deleteAccessKey",
|
|
||||||
"params": [2, "0123abcd...keyhash..."]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Set Database Public/Private
|
|
||||||
Toggles public access (default true). Private databases require access keys.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 9,
|
|
||||||
"method": "hero_setDatabasePublic",
|
|
||||||
"params": [2, false]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Server Info
|
|
||||||
|
|
||||||
### Get Server Stats
|
|
||||||
Returns stats like total databases and uptime.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 10,
|
|
||||||
"method": "hero_getServerStats",
|
|
||||||
"params": []
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
- Per-database encryption keys are write-only; set at creation and used transparently.
|
|
||||||
- Access keys are hashed (SHA-256) for storage; provide plaintext in requests.
|
|
||||||
- Backend options: `"Redb"` (default) or `"Sled"`.
|
|
||||||
- Config object fields (name, storage_path, etc.) are optional and currently ignored but positional.
|
|
253
docs/tantivy.md
253
docs/tantivy.md
@@ -1,253 +0,0 @@
|
|||||||
# Tantivy Full‑Text Backend (JSON‑RPC)
|
|
||||||
|
|
||||||
This document explains how to use HeroDB’s Tantivy-backed full‑text search as a dedicated database backend and provides copy‑pasteable JSON‑RPC requests. Tantivy is available only for non‑admin databases (db_id >= 1). Admin DB 0 always uses Redb/Sled and rejects FT operations.
|
|
||||||
|
|
||||||
Important characteristics:
|
|
||||||
- Tantivy is a third backend alongside Redb and Sled. It provides search indexes only; there is no KV store backing it.
|
|
||||||
- On Tantivy databases, Redis KV/list/hash commands are rejected; only FT commands and basic control (SELECT, CLIENT, INFO, etc.) are allowed.
|
|
||||||
- FT JSON‑RPC is namespaced as "herodb" and methods are named with underscore: herodb_ftCreate, herodb_ftAdd, herodb_ftSearch, herodb_ftDel, herodb_ftInfo, herodb_ftDrop.
|
|
||||||
|
|
||||||
Reference to server implementation:
|
|
||||||
- RPC methods are defined in [rust.trait Rpc()](src/rpc.rs:70):
|
|
||||||
- [rust.fn ft_create()](src/rpc.rs:121)
|
|
||||||
- [rust.fn ft_add()](src/rpc.rs:130)
|
|
||||||
- [rust.fn ft_search()](src/rpc.rs:141)
|
|
||||||
- [rust.fn ft_del()](src/rpc.rs:154)
|
|
||||||
- [rust.fn ft_info()](src/rpc.rs:158)
|
|
||||||
- [rust.fn ft_drop()](src/rpc.rs:162)
|
|
||||||
|
|
||||||
Notes on responses:
|
|
||||||
- ftCreate/ftAdd/ftDel/ftDrop return a JSON boolean: true on success.
|
|
||||||
- ftSearch/ftInfo return a JSON object with a single key "resp" containing a RESP‑encoded string (wire format used by Redis). You can display or parse it on the client side as needed.
|
|
||||||
|
|
||||||
RESP usage (redis-cli):
|
|
||||||
- For RESP clients, you must SELECT the Tantivy database first. SELECT now succeeds for Tantivy DBs without opening KV storage.
|
|
||||||
- After SELECT, you can run FT.* commands within that DB context.
|
|
||||||
|
|
||||||
Example with redis-cli:
|
|
||||||
```bash
|
|
||||||
# Connect to server
|
|
||||||
redis-cli -p 6379
|
|
||||||
|
|
||||||
# Select Tantivy DB 1 (public by default)
|
|
||||||
SELECT 1
|
|
||||||
# → OK
|
|
||||||
|
|
||||||
# Create index
|
|
||||||
FT.CREATE product_catalog SCHEMA title TEXT description TEXT category TAG price NUMERIC rating NUMERIC location GEO
|
|
||||||
# → OK
|
|
||||||
|
|
||||||
# Add a document
|
|
||||||
FT.ADD product_catalog product:1 1.0 title "Wireless Bluetooth Headphones" description "Premium noise-canceling headphones with 30-hour battery life" category "electronics,audio" price 299.99 rating 4.5 location "-122.4194,37.7749"
|
|
||||||
# → OK
|
|
||||||
|
|
||||||
# Search
|
|
||||||
FT.SEARCH product_catalog wireless LIMIT 0 3
|
|
||||||
# → RESP array with hits
|
|
||||||
```
|
|
||||||
|
|
||||||
Storage layout (on disk):
|
|
||||||
- Indices are stored per database under:
|
|
||||||
- <base_dir>/search_indexes/<db_id>/<index_name>
|
|
||||||
- Example: /tmp/test/search_indexes/1/product_catalog
|
|
||||||
|
|
||||||
0) Create a new Tantivy database
|
|
||||||
|
|
||||||
Use herodb_createDatabase with backend "Tantivy". DB 0 cannot be Tantivy.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "herodb_createDatabase",
|
|
||||||
"params": [
|
|
||||||
"Tantivy",
|
|
||||||
{ "name": "search-db", "storage_path": null, "max_size": null, "redis_version": null },
|
|
||||||
null
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
The response contains the allocated db_id (>= 1). Use that id in the calls below.
|
|
||||||
|
|
||||||
1) FT.CREATE — create an index with schema
|
|
||||||
|
|
||||||
Method: herodb_ftCreate → [rust.fn ft_create()](src/rpc.rs:121)
|
|
||||||
|
|
||||||
Schema format is an array of tuples: [ [field_name, field_type, [options...] ], ... ]
|
|
||||||
Supported field types: "TEXT", "NUMERIC" (defaults to F64), "TAG", "GEO"
|
|
||||||
Supported options (subset): "WEIGHT", "SORTABLE", "NOINDEX", "SEPARATOR", "CASESENSITIVE"
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 2,
|
|
||||||
"method": "herodb_ftCreate",
|
|
||||||
"params": [
|
|
||||||
1,
|
|
||||||
"product_catalog",
|
|
||||||
[
|
|
||||||
["title", "TEXT", ["SORTABLE"]],
|
|
||||||
["description", "TEXT", []],
|
|
||||||
["category", "TAG", ["SEPARATOR", ","]],
|
|
||||||
["price", "NUMERIC", ["SORTABLE"]],
|
|
||||||
["rating", "NUMERIC", []],
|
|
||||||
["location", "GEO", []]
|
|
||||||
]
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Returns: true on success.
|
|
||||||
|
|
||||||
2) FT.ADD — add or replace a document
|
|
||||||
|
|
||||||
Method: herodb_ftAdd → [rust.fn ft_add()](src/rpc.rs:130)
|
|
||||||
|
|
||||||
Fields is an object (map) of field_name → value (all values are sent as strings). GEO expects "lat,lon".
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 3,
|
|
||||||
"method": "herodb_ftAdd",
|
|
||||||
"params": [
|
|
||||||
1,
|
|
||||||
"product_catalog",
|
|
||||||
"product:1",
|
|
||||||
1.0,
|
|
||||||
{
|
|
||||||
"title": "Wireless Bluetooth Headphones",
|
|
||||||
"description": "Premium noise-canceling headphones with 30-hour battery life",
|
|
||||||
"category": "electronics,audio",
|
|
||||||
"price": "299.99",
|
|
||||||
"rating": "4.5",
|
|
||||||
"location": "-122.4194,37.7749"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Returns: true on success.
|
|
||||||
|
|
||||||
3) FT.SEARCH — query an index
|
|
||||||
|
|
||||||
Method: herodb_ftSearch → [rust.fn ft_search()](src/rpc.rs:141)
|
|
||||||
|
|
||||||
Parameters: (db_id, index_name, query, filters?, limit?, offset?, return_fields?)
|
|
||||||
- filters: array of [field, value] pairs (Equals filter)
|
|
||||||
- limit/offset: numbers (defaults: limit=10, offset=0)
|
|
||||||
- return_fields: array of field names to include (optional)
|
|
||||||
|
|
||||||
Simple query:
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 4,
|
|
||||||
"method": "herodb_ftSearch",
|
|
||||||
"params": [1, "product_catalog", "wireless", null, 10, 0, null]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Pagination + filters + selected fields:
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 5,
|
|
||||||
"method": "herodb_ftSearch",
|
|
||||||
"params": [
|
|
||||||
1,
|
|
||||||
"product_catalog",
|
|
||||||
"mouse",
|
|
||||||
[["category", "electronics"]],
|
|
||||||
5,
|
|
||||||
0,
|
|
||||||
["title", "price", "rating"]
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response shape:
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 5,
|
|
||||||
"result": { "resp": "*...RESP encoded array..." }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
4) FT.INFO — index metadata
|
|
||||||
|
|
||||||
Method: herodb_ftInfo → [rust.fn ft_info()](src/rpc.rs:158)
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 6,
|
|
||||||
"method": "herodb_ftInfo",
|
|
||||||
"params": [1, "product_catalog"]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Response shape:
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 6,
|
|
||||||
"result": { "resp": "*...RESP encoded array with fields and counts..." }
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
5) FT.DEL — delete by doc id
|
|
||||||
|
|
||||||
Method: herodb_ftDel → [rust.fn ft_del()](src/rpc.rs:154)
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 7,
|
|
||||||
"method": "herodb_ftDel",
|
|
||||||
"params": [1, "product_catalog", "product:1"]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Returns: true on success. Note: current implementation logs and returns success; physical delete may be a no‑op until delete is finalized in the engine.
|
|
||||||
|
|
||||||
6) FT.DROP — drop an index
|
|
||||||
|
|
||||||
Method: herodb_ftDrop → [rust.fn ft_drop()](src/rpc.rs:162)
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 8,
|
|
||||||
"method": "herodb_ftDrop",
|
|
||||||
"params": [1, "product_catalog"]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Returns: true on success.
|
|
||||||
|
|
||||||
Field types and options
|
|
||||||
|
|
||||||
- TEXT: stored/indexed/tokenized text. "SORTABLE" marks it fast (stored + fast path in our wrapper).
|
|
||||||
- NUMERIC: stored/indexed numeric; default precision F64. "SORTABLE" enables fast column.
|
|
||||||
- TAG: exact matching terms. Options: "SEPARATOR" (default ","), "CASESENSITIVE" (default false).
|
|
||||||
- GEO: "lat,lon" string; stored as two numeric fields internally.
|
|
||||||
|
|
||||||
Backend and permission gating
|
|
||||||
|
|
||||||
- FT methods are rejected on DB 0.
|
|
||||||
- FT methods require the database backend to be Tantivy; otherwise RPC returns an error.
|
|
||||||
- Write‑like FT methods (create/add/del/drop) follow the same permission model as Redis writes on selected databases.
|
|
||||||
|
|
||||||
Troubleshooting
|
|
||||||
|
|
||||||
- "DB backend is not Tantivy": ensure the database was created with backend "Tantivy".
|
|
||||||
- "FT not allowed on DB 0": use a non‑admin database id (>= 1).
|
|
||||||
- Empty search results: confirm that the queried fields are tokenized/indexed (TEXT) and that documents were added successfully.
|
|
||||||
|
|
||||||
Related docs
|
|
||||||
|
|
||||||
- Command‑level search overview: [docs/search.md](docs/search.md:1)
|
|
||||||
- RPC definitions: [src/rpc.rs](src/rpc.rs:1)
|
|
@@ -14,25 +14,31 @@ fn read_reply(s: &mut TcpStream) -> String {
|
|||||||
let n = s.read(&mut buf).unwrap();
|
let n = s.read(&mut buf).unwrap();
|
||||||
String::from_utf8_lossy(&buf[..n]).to_string()
|
String::from_utf8_lossy(&buf[..n]).to_string()
|
||||||
}
|
}
|
||||||
fn parse_two_bulk(reply: &str) -> Option<(String,String)> {
|
fn parse_two_bulk(reply: &str) -> Option<(String, String)> {
|
||||||
let mut lines = reply.split("\r\n");
|
let mut lines = reply.split("\r\n");
|
||||||
if lines.next()? != "*2" { return None; }
|
if lines.next()? != "*2" {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
let _n = lines.next()?;
|
let _n = lines.next()?;
|
||||||
let a = lines.next()?.to_string();
|
let a = lines.next()?.to_string();
|
||||||
let _m = lines.next()?;
|
let _m = lines.next()?;
|
||||||
let b = lines.next()?.to_string();
|
let b = lines.next()?.to_string();
|
||||||
Some((a,b))
|
Some((a, b))
|
||||||
}
|
}
|
||||||
fn parse_bulk(reply: &str) -> Option<String> {
|
fn parse_bulk(reply: &str) -> Option<String> {
|
||||||
let mut lines = reply.split("\r\n");
|
let mut lines = reply.split("\r\n");
|
||||||
let hdr = lines.next()?;
|
let hdr = lines.next()?;
|
||||||
if !hdr.starts_with('$') { return None; }
|
if !hdr.starts_with('$') {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
Some(lines.next()?.to_string())
|
Some(lines.next()?.to_string())
|
||||||
}
|
}
|
||||||
fn parse_simple(reply: &str) -> Option<String> {
|
fn parse_simple(reply: &str) -> Option<String> {
|
||||||
let mut lines = reply.split("\r\n");
|
let mut lines = reply.split("\r\n");
|
||||||
let hdr = lines.next()?;
|
let hdr = lines.next()?;
|
||||||
if !hdr.starts_with('+') { return None; }
|
if !hdr.starts_with('+') {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
Some(hdr[1..].to_string())
|
Some(hdr[1..].to_string())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -45,37 +51,43 @@ fn main() {
|
|||||||
let mut s = TcpStream::connect(addr).expect("connect");
|
let mut s = TcpStream::connect(addr).expect("connect");
|
||||||
|
|
||||||
// Generate & persist X25519 enc keys under name "alice"
|
// Generate & persist X25519 enc keys under name "alice"
|
||||||
s.write_all(arr(&["age","keygen","alice"]).as_bytes()).unwrap();
|
s.write_all(arr(&["age", "keygen", "alice"]).as_bytes())
|
||||||
|
.unwrap();
|
||||||
let (_alice_recip, _alice_ident) = parse_two_bulk(&read_reply(&mut s)).expect("gen enc");
|
let (_alice_recip, _alice_ident) = parse_two_bulk(&read_reply(&mut s)).expect("gen enc");
|
||||||
|
|
||||||
// Generate & persist Ed25519 signing key under name "signer"
|
// Generate & persist Ed25519 signing key under name "signer"
|
||||||
s.write_all(arr(&["age","signkeygen","signer"]).as_bytes()).unwrap();
|
s.write_all(arr(&["age", "signkeygen", "signer"]).as_bytes())
|
||||||
|
.unwrap();
|
||||||
let (_verify, _secret) = parse_two_bulk(&read_reply(&mut s)).expect("gen sign");
|
let (_verify, _secret) = parse_two_bulk(&read_reply(&mut s)).expect("gen sign");
|
||||||
|
|
||||||
// Encrypt by name
|
// Encrypt by name
|
||||||
let msg = "hello from persistent keys";
|
let msg = "hello from persistent keys";
|
||||||
s.write_all(arr(&["age","encryptname","alice", msg]).as_bytes()).unwrap();
|
s.write_all(arr(&["age", "encryptname", "alice", msg]).as_bytes())
|
||||||
|
.unwrap();
|
||||||
let ct_b64 = parse_bulk(&read_reply(&mut s)).expect("ct b64");
|
let ct_b64 = parse_bulk(&read_reply(&mut s)).expect("ct b64");
|
||||||
println!("ciphertext b64: {}", ct_b64);
|
println!("ciphertext b64: {}", ct_b64);
|
||||||
|
|
||||||
// Decrypt by name
|
// Decrypt by name
|
||||||
s.write_all(arr(&["age","decryptname","alice", &ct_b64]).as_bytes()).unwrap();
|
s.write_all(arr(&["age", "decryptname", "alice", &ct_b64]).as_bytes())
|
||||||
|
.unwrap();
|
||||||
let pt = parse_bulk(&read_reply(&mut s)).expect("pt");
|
let pt = parse_bulk(&read_reply(&mut s)).expect("pt");
|
||||||
assert_eq!(pt, msg);
|
assert_eq!(pt, msg);
|
||||||
println!("decrypted ok");
|
println!("decrypted ok");
|
||||||
|
|
||||||
// Sign by name
|
// Sign by name
|
||||||
s.write_all(arr(&["age","signname","signer", msg]).as_bytes()).unwrap();
|
s.write_all(arr(&["age", "signname", "signer", msg]).as_bytes())
|
||||||
|
.unwrap();
|
||||||
let sig_b64 = parse_bulk(&read_reply(&mut s)).expect("sig b64");
|
let sig_b64 = parse_bulk(&read_reply(&mut s)).expect("sig b64");
|
||||||
|
|
||||||
// Verify by name
|
// Verify by name
|
||||||
s.write_all(arr(&["age","verifyname","signer", msg, &sig_b64]).as_bytes()).unwrap();
|
s.write_all(arr(&["age", "verifyname", "signer", msg, &sig_b64]).as_bytes())
|
||||||
|
.unwrap();
|
||||||
let ok = parse_simple(&read_reply(&mut s)).expect("verify");
|
let ok = parse_simple(&read_reply(&mut s)).expect("verify");
|
||||||
assert_eq!(ok, "1");
|
assert_eq!(ok, "1");
|
||||||
println!("signature verified");
|
println!("signature verified");
|
||||||
|
|
||||||
// List names
|
// List names
|
||||||
s.write_all(arr(&["age","list"]).as_bytes()).unwrap();
|
s.write_all(arr(&["age", "list"]).as_bytes()).unwrap();
|
||||||
let list = read_reply(&mut s);
|
let list = read_reply(&mut s);
|
||||||
println!("LIST -> {list}");
|
println!("LIST -> {list}");
|
||||||
|
|
||||||
|
239
examples/tantivy_search_demo.sh
Executable file
239
examples/tantivy_search_demo.sh
Executable file
@@ -0,0 +1,239 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# HeroDB Tantivy Search Demo
|
||||||
|
# This script demonstrates full-text search capabilities using Redis commands
|
||||||
|
# HeroDB server should be running on port 6381
|
||||||
|
|
||||||
|
set -e # Exit on any error
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
REDIS_HOST="localhost"
|
||||||
|
REDIS_PORT="6382"
|
||||||
|
REDIS_CLI="redis-cli -h $REDIS_HOST -p $REDIS_PORT"
|
||||||
|
|
||||||
|
# Start the herodb server in the background
|
||||||
|
echo "Starting herodb server..."
|
||||||
|
cargo run -p herodb -- --dir /tmp/herodbtest --port ${REDIS_PORT} --debug &
|
||||||
|
SERVER_PID=$!
|
||||||
|
echo
|
||||||
|
sleep 2 # Give the server a moment to start
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Function to print colored output
|
||||||
|
print_header() {
|
||||||
|
echo -e "${BLUE}=== $1 ===${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
print_success() {
|
||||||
|
echo -e "${GREEN}✓ $1${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
print_info() {
|
||||||
|
echo -e "${YELLOW}ℹ $1${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
print_error() {
|
||||||
|
echo -e "${RED}✗ $1${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to check if HeroDB is running
|
||||||
|
check_herodb() {
|
||||||
|
print_info "Checking if HeroDB is running on port $REDIS_PORT..."
|
||||||
|
if ! $REDIS_CLI ping > /dev/null 2>&1; then
|
||||||
|
print_error "HeroDB is not running on port $REDIS_PORT"
|
||||||
|
print_info "Please start HeroDB with: cargo run -- --port $REDIS_PORT"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
print_success "HeroDB is running and responding"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to execute Redis command with error handling
|
||||||
|
execute_cmd() {
|
||||||
|
local description="${@: -1}"
|
||||||
|
set -- "${@:1:$(($#-1))}"
|
||||||
|
|
||||||
|
echo -e "${YELLOW}Command:${NC} $(printf '%q ' "$@")"
|
||||||
|
if result=$($REDIS_CLI "$@" 2>&1); then
|
||||||
|
echo -e "${GREEN}Result:${NC} $result"
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
print_error "Failed: $description"
|
||||||
|
echo "Error: $result"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to pause for readability
|
||||||
|
pause() {
|
||||||
|
echo
|
||||||
|
read -p "Press Enter to continue..."
|
||||||
|
echo
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main demo function
|
||||||
|
main() {
|
||||||
|
clear
|
||||||
|
print_header "HeroDB Tantivy Search Demonstration"
|
||||||
|
echo "This demo shows full-text search capabilities using Redis commands"
|
||||||
|
echo "HeroDB runs on port $REDIS_PORT (instead of Redis default 6379)"
|
||||||
|
echo
|
||||||
|
|
||||||
|
# Check if HeroDB is running
|
||||||
|
check_herodb
|
||||||
|
echo
|
||||||
|
|
||||||
|
print_header "Step 1: Create Search Index"
|
||||||
|
print_info "Creating a product catalog search index with various field types"
|
||||||
|
|
||||||
|
# Create search index with schema
|
||||||
|
execute_cmd FT.CREATE product_catalog SCHEMA title TEXT description TEXT category TAG price NUMERIC rating NUMERIC location GEO \
|
||||||
|
"Creating search index"
|
||||||
|
|
||||||
|
print_success "Search index 'product_catalog' created successfully"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 2: Add Sample Products"
|
||||||
|
print_info "Adding sample products to demonstrate different search scenarios"
|
||||||
|
|
||||||
|
# Add sample products using FT.ADD
|
||||||
|
execute_cmd FT.ADD product_catalog product:1 1.0 title 'Wireless Bluetooth Headphones' description 'Premium noise-canceling headphones with 30-hour battery life' category 'electronics,audio' price 299.99 rating 4.5 location '-122.4194,37.7749' "Adding product 1"
|
||||||
|
execute_cmd FT.ADD product_catalog product:2 1.0 title 'Organic Coffee Beans' description 'Single-origin Ethiopian coffee beans, medium roast' category 'food,beverages,organic' price 24.99 rating 4.8 location '-74.0060,40.7128' "Adding product 2"
|
||||||
|
execute_cmd FT.ADD product_catalog product:3 1.0 title 'Yoga Mat Premium' description 'Eco-friendly yoga mat with superior grip and cushioning' category 'fitness,wellness,eco-friendly' price 89.99 rating 4.3 location '-118.2437,34.0522' "Adding product 3"
|
||||||
|
execute_cmd FT.ADD product_catalog product:4 1.0 title 'Smart Home Speaker' description 'Voice-controlled smart speaker with AI assistant' category 'electronics,smart-home' price 149.99 rating 4.2 location '-87.6298,41.8781' "Adding product 4"
|
||||||
|
execute_cmd FT.ADD product_catalog product:5 1.0 title 'Organic Green Tea' description 'Premium organic green tea leaves from Japan' category 'food,beverages,organic,tea' price 18.99 rating 4.7 location '139.6503,35.6762' "Adding product 5"
|
||||||
|
execute_cmd FT.ADD product_catalog product:6 1.0 title 'Wireless Gaming Mouse' description 'High-precision gaming mouse with RGB lighting' category 'electronics,gaming' price 79.99 rating 4.4 location '-122.3321,47.6062' "Adding product 6"
|
||||||
|
execute_cmd FT.ADD product_catalog product:7 1.0 title 'Comfortable meditation cushion for mindfulness practice' description 'Meditation cushion with premium materials' category 'wellness,meditation' price 45.99 rating 4.6 location '-122.4194,37.7749' "Adding product 7"
|
||||||
|
execute_cmd FT.ADD product_catalog product:8 1.0 title 'Bluetooth Earbuds' description 'True wireless earbuds with active noise cancellation' category 'electronics,audio' price 199.99 rating 4.1 location '-74.0060,40.7128' "Adding product 8"
|
||||||
|
|
||||||
|
print_success "Added 8 products to the index"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 3: Basic Text Search"
|
||||||
|
print_info "Searching for 'wireless' products"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog wireless "Basic text search"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 4: Search with Filters"
|
||||||
|
print_info "Searching for 'organic' products"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog organic "Filtered search"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 5: Numeric Range Search"
|
||||||
|
print_info "Searching for 'premium' products"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog premium "Text search"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 6: Sorting Results"
|
||||||
|
print_info "Searching for electronics"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog electronics "Category search"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 7: Limiting Results"
|
||||||
|
print_info "Searching for wireless products with limit"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog wireless LIMIT 0 3 "Limited results"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 8: Complex Query"
|
||||||
|
print_info "Finding audio products with noise cancellation"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog 'noise cancellation' "Complex query"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 9: Geographic Search"
|
||||||
|
print_info "Searching for meditation products"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog meditation "Text search"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 10: Aggregation Example"
|
||||||
|
print_info "Getting index information and statistics"
|
||||||
|
|
||||||
|
execute_cmd FT.INFO product_catalog "Index information"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 11: Search Comparison"
|
||||||
|
print_info "Comparing Tantivy search vs simple key matching"
|
||||||
|
|
||||||
|
echo -e "${YELLOW}Tantivy Full-Text Search:${NC}"
|
||||||
|
execute_cmd FT.SEARCH product_catalog 'battery life' "Full-text search for 'battery life'"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo -e "${YELLOW}Simple Key Pattern Matching:${NC}"
|
||||||
|
execute_cmd KEYS *battery* "Simple pattern matching for 'battery'"
|
||||||
|
|
||||||
|
print_info "Notice how full-text search finds relevant results even when exact words don't match keys"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 12: Fuzzy Search"
|
||||||
|
print_info "Searching for headphones"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog headphones "Text search"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 13: Phrase Search"
|
||||||
|
print_info "Searching for coffee products"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog coffee "Text search"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 14: Boolean Queries"
|
||||||
|
print_info "Searching for gaming products"
|
||||||
|
|
||||||
|
execute_cmd FT.SEARCH product_catalog gaming "Text search"
|
||||||
|
echo
|
||||||
|
execute_cmd FT.SEARCH product_catalog tea "Text search"
|
||||||
|
pause
|
||||||
|
|
||||||
|
print_header "Step 15: Cleanup"
|
||||||
|
print_info "Removing test data"
|
||||||
|
|
||||||
|
# Delete the search index
|
||||||
|
execute_cmd FT.DROP product_catalog "Dropping search index"
|
||||||
|
|
||||||
|
# Clean up documents from search index
|
||||||
|
for i in {1..8}; do
|
||||||
|
execute_cmd FT.DEL product_catalog product:$i "Deleting product:$i from index"
|
||||||
|
done
|
||||||
|
|
||||||
|
print_success "Cleanup completed"
|
||||||
|
echo
|
||||||
|
|
||||||
|
print_header "Demo Summary"
|
||||||
|
echo "This demonstration showed:"
|
||||||
|
echo "• Creating search indexes with different field types"
|
||||||
|
echo "• Adding documents to the search index"
|
||||||
|
echo "• Basic and advanced text search queries"
|
||||||
|
echo "• Filtering by categories and numeric ranges"
|
||||||
|
echo "• Sorting and limiting results"
|
||||||
|
echo "• Geographic searches"
|
||||||
|
echo "• Fuzzy matching and phrase searches"
|
||||||
|
echo "• Boolean query operators"
|
||||||
|
echo "• Comparison with simple pattern matching"
|
||||||
|
echo
|
||||||
|
print_success "HeroDB Tantivy search demo completed successfully!"
|
||||||
|
echo
|
||||||
|
print_info "Key advantages of Tantivy full-text search:"
|
||||||
|
echo " - Relevance scoring and ranking"
|
||||||
|
echo " - Fuzzy matching and typo tolerance"
|
||||||
|
echo " - Complex boolean queries"
|
||||||
|
echo " - Field-specific searches and filters"
|
||||||
|
echo " - Geographic and numeric range queries"
|
||||||
|
echo " - Much faster than pattern matching on large datasets"
|
||||||
|
echo
|
||||||
|
print_info "To run HeroDB server: cargo run -- --port 6381"
|
||||||
|
print_info "To connect with redis-cli: redis-cli -h localhost -p 6381"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run the demo
|
||||||
|
main "$@"
|
101
examples/test_tantivy_integration.sh
Executable file
101
examples/test_tantivy_integration.sh
Executable file
@@ -0,0 +1,101 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Simple Tantivy Search Integration Test for HeroDB
|
||||||
|
# This script tests the full-text search functionality we just integrated
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "🔍 Testing Tantivy Search Integration..."
|
||||||
|
|
||||||
|
# Build the project first
|
||||||
|
echo "📦 Building HeroDB..."
|
||||||
|
cargo build --release
|
||||||
|
|
||||||
|
# Start the server in the background
|
||||||
|
echo "🚀 Starting HeroDB server on port 6379..."
|
||||||
|
cargo run --release -- --port 6379 --dir ./test_data &
|
||||||
|
SERVER_PID=$!
|
||||||
|
|
||||||
|
# Wait for server to start
|
||||||
|
sleep 3
|
||||||
|
|
||||||
|
# Function to cleanup on exit
|
||||||
|
cleanup() {
|
||||||
|
echo "🧹 Cleaning up..."
|
||||||
|
kill $SERVER_PID 2>/dev/null || true
|
||||||
|
rm -rf ./test_data
|
||||||
|
exit
|
||||||
|
}
|
||||||
|
|
||||||
|
# Set trap for cleanup
|
||||||
|
trap cleanup EXIT INT TERM
|
||||||
|
|
||||||
|
# Function to execute Redis command
|
||||||
|
execute_cmd() {
|
||||||
|
local cmd="$1"
|
||||||
|
local description="$2"
|
||||||
|
|
||||||
|
echo "📝 $description"
|
||||||
|
echo " Command: $cmd"
|
||||||
|
|
||||||
|
if result=$(redis-cli -p 6379 $cmd 2>&1); then
|
||||||
|
echo " ✅ Result: $result"
|
||||||
|
echo
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
echo " ❌ Failed: $result"
|
||||||
|
echo
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "🧪 Running Tantivy Search Tests..."
|
||||||
|
echo
|
||||||
|
|
||||||
|
# Test 1: Create a search index
|
||||||
|
execute_cmd "ft.create books SCHEMA title TEXT description TEXT author TEXT category TAG price NUMERIC" \
|
||||||
|
"Creating search index 'books'"
|
||||||
|
|
||||||
|
# Test 2: Add documents to the index
|
||||||
|
execute_cmd "ft.add books book1 1.0 title \"The Great Gatsby\" description \"A classic American novel about the Jazz Age\" author \"F. Scott Fitzgerald\" category \"fiction,classic\" price \"12.99\"" \
|
||||||
|
"Adding first book"
|
||||||
|
|
||||||
|
execute_cmd "ft.add books book2 1.0 title \"To Kill a Mockingbird\" description \"A novel about racial injustice in the American South\" author \"Harper Lee\" category \"fiction,classic\" price \"14.99\"" \
|
||||||
|
"Adding second book"
|
||||||
|
|
||||||
|
execute_cmd "ft.add books book3 1.0 title \"Programming Rust\" description \"A comprehensive guide to Rust programming language\" author \"Jim Blandy\" category \"programming,technical\" price \"49.99\"" \
|
||||||
|
"Adding third book"
|
||||||
|
|
||||||
|
execute_cmd "ft.add books book4 1.0 title \"The Rust Programming Language\" description \"The official book on Rust programming\" author \"Steve Klabnik\" category \"programming,technical\" price \"39.99\"" \
|
||||||
|
"Adding fourth book"
|
||||||
|
|
||||||
|
# Test 3: Basic search
|
||||||
|
execute_cmd "ft.search books Rust" \
|
||||||
|
"Searching for 'Rust'"
|
||||||
|
|
||||||
|
# Test 4: Search with filters
|
||||||
|
execute_cmd "ft.search books programming FILTER category programming" \
|
||||||
|
"Searching for 'programming' with category filter"
|
||||||
|
|
||||||
|
# Test 5: Search with limit
|
||||||
|
execute_cmd "ft.search books \"*\" LIMIT 0 2" \
|
||||||
|
"Getting first 2 documents"
|
||||||
|
|
||||||
|
# Test 6: Get index info
|
||||||
|
execute_cmd "ft.info books" \
|
||||||
|
"Getting index information"
|
||||||
|
|
||||||
|
# Test 7: Delete a document
|
||||||
|
execute_cmd "ft.del books book1" \
|
||||||
|
"Deleting book1"
|
||||||
|
|
||||||
|
# Test 8: Search again to verify deletion
|
||||||
|
execute_cmd "ft.search books Gatsby" \
|
||||||
|
"Searching for deleted book"
|
||||||
|
|
||||||
|
# Test 9: Drop the index
|
||||||
|
execute_cmd "ft.drop books" \
|
||||||
|
"Dropping the index"
|
||||||
|
|
||||||
|
echo "🎉 All tests completed successfully!"
|
||||||
|
echo "✅ Tantivy search integration is working correctly"
|
143
run.sh
143
run.sh
@@ -1,143 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
set -euo pipefail
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
cd "$SCRIPT_DIR"
|
|
||||||
|
|
||||||
|
|
||||||
# Test script for HeroDB - Redis-compatible database with redb backend
|
|
||||||
# This script starts the server and runs comprehensive tests
|
|
||||||
|
|
||||||
# Colors for output
|
|
||||||
RED='\033[0;31m'
|
|
||||||
GREEN='\033[0;32m'
|
|
||||||
YELLOW='\033[1;33m'
|
|
||||||
BLUE='\033[0;34m'
|
|
||||||
NC='\033[0m' # No Color
|
|
||||||
|
|
||||||
# Configuration
|
|
||||||
DB_DIR="/tmp/test_db"
|
|
||||||
PORT=6381
|
|
||||||
SERVER_PID=""
|
|
||||||
|
|
||||||
# Function to print colored output
|
|
||||||
print_status() {
|
|
||||||
echo -e "${BLUE}[INFO]${NC} $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
print_success() {
|
|
||||||
echo -e "${GREEN}[SUCCESS]${NC} $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
print_error() {
|
|
||||||
echo -e "${RED}[ERROR]${NC} $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
print_warning() {
|
|
||||||
echo -e "${YELLOW}[WARNING]${NC} $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Function to cleanup on exit
|
|
||||||
cleanup() {
|
|
||||||
if [ ! -z "$SERVER_PID" ]; then
|
|
||||||
print_status "Stopping HeroDB server (PID: $SERVER_PID)..."
|
|
||||||
kill $SERVER_PID 2>/dev/null || true
|
|
||||||
wait $SERVER_PID 2>/dev/null || true
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Clean up test database
|
|
||||||
if [ -d "$DB_DIR" ]; then
|
|
||||||
print_status "Cleaning up test database directory..."
|
|
||||||
rm -rf "$DB_DIR"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Set trap to cleanup on script exit
|
|
||||||
trap cleanup EXIT
|
|
||||||
|
|
||||||
# Function to wait for server to start
|
|
||||||
wait_for_server() {
|
|
||||||
local max_attempts=30
|
|
||||||
local attempt=1
|
|
||||||
|
|
||||||
print_status "Waiting for server to start on port $PORT..."
|
|
||||||
|
|
||||||
while [ $attempt -le $max_attempts ]; do
|
|
||||||
if nc -z localhost $PORT 2>/dev/null; then
|
|
||||||
print_success "Server is ready!"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo -n "."
|
|
||||||
sleep 1
|
|
||||||
attempt=$((attempt + 1))
|
|
||||||
done
|
|
||||||
|
|
||||||
print_error "Server failed to start within $max_attempts seconds"
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Function to send Redis command and get response
|
|
||||||
redis_cmd() {
|
|
||||||
local cmd="$1"
|
|
||||||
local expected="$2"
|
|
||||||
|
|
||||||
print_status "Testing: $cmd"
|
|
||||||
|
|
||||||
local result=$(echo "$cmd" | redis-cli -p $PORT --raw 2>/dev/null || echo "ERROR")
|
|
||||||
|
|
||||||
if [ "$expected" != "" ] && [ "$result" != "$expected" ]; then
|
|
||||||
print_error "Expected: '$expected', Got: '$result'"
|
|
||||||
return 1
|
|
||||||
else
|
|
||||||
print_success "✓ $cmd -> $result"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Main execution
|
|
||||||
main() {
|
|
||||||
print_status "Starting HeroDB"
|
|
||||||
|
|
||||||
# Build the project
|
|
||||||
print_status "Building HeroDB..."
|
|
||||||
if ! cargo build -p herodb --release; then
|
|
||||||
print_error "Failed to build HeroDB"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Create test database directory
|
|
||||||
mkdir -p "$DB_DIR"
|
|
||||||
|
|
||||||
# Start the server
|
|
||||||
print_status "Starting HeroDB server..."
|
|
||||||
${SCRIPT_DIR}/target/release/herodb --dir "$DB_DIR" --port $PORT &
|
|
||||||
SERVER_PID=$!
|
|
||||||
|
|
||||||
# Wait for server to start
|
|
||||||
if ! wait_for_server; then
|
|
||||||
print_error "Failed to start server"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
}
|
|
||||||
|
|
||||||
# Check dependencies
|
|
||||||
check_dependencies() {
|
|
||||||
if ! command -v cargo &> /dev/null; then
|
|
||||||
print_error "cargo is required but not installed"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! command -v nc &> /dev/null; then
|
|
||||||
print_warning "netcat (nc) not found - some tests may not work properly"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! command -v redis-cli &> /dev/null; then
|
|
||||||
print_warning "redis-cli not found - using netcat fallback"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Run dependency check and main function
|
|
||||||
check_dependencies
|
|
||||||
main "$@"
|
|
||||||
tail -f /dev/null
|
|
@@ -1,7 +1,4 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
set -euo pipefail
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
cd "$SCRIPT_DIR"
|
|
||||||
|
|
||||||
echo "🧪 Running HeroDB Redis Compatibility Tests"
|
echo "🧪 Running HeroDB Redis Compatibility Tests"
|
||||||
echo "=========================================="
|
echo "=========================================="
|
||||||
|
@@ -1,495 +0,0 @@
|
|||||||
use std::path::PathBuf;
|
|
||||||
use std::sync::{Arc, OnceLock, Mutex, RwLock};
|
|
||||||
use std::collections::HashMap;
|
|
||||||
|
|
||||||
use crate::error::DBError;
|
|
||||||
use crate::options;
|
|
||||||
use crate::rpc::Permissions;
|
|
||||||
use crate::storage::Storage;
|
|
||||||
use crate::storage_sled::SledStorage;
|
|
||||||
use crate::storage_trait::StorageBackend;
|
|
||||||
|
|
||||||
// Key builders
|
|
||||||
fn k_admin_next_id() -> &'static str {
|
|
||||||
"admin:next_id"
|
|
||||||
}
|
|
||||||
fn k_admin_dbs() -> &'static str {
|
|
||||||
"admin:dbs"
|
|
||||||
}
|
|
||||||
fn k_meta_db(id: u64) -> String {
|
|
||||||
format!("meta:db:{}", id)
|
|
||||||
}
|
|
||||||
fn k_meta_db_keys(id: u64) -> String {
|
|
||||||
format!("meta:db:{}:keys", id)
|
|
||||||
}
|
|
||||||
fn k_meta_db_enc(id: u64) -> String {
|
|
||||||
format!("meta:db:{}:enc", id)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Global cache of admin DB 0 handles per base_dir to avoid sled/reDB file-lock contention
|
|
||||||
// and to correctly isolate different test instances with distinct directories.
|
|
||||||
static ADMIN_STORAGES: OnceLock<RwLock<HashMap<String, Arc<dyn StorageBackend>>>> = OnceLock::new();
|
|
||||||
|
|
||||||
// Global registry for data DB storages to avoid double-open across process.
|
|
||||||
static DATA_STORAGES: OnceLock<RwLock<HashMap<u64, Arc<dyn StorageBackend>>>> = OnceLock::new();
|
|
||||||
static DATA_INIT_LOCK: Mutex<()> = Mutex::new(());
|
|
||||||
|
|
||||||
fn init_admin_storage(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
) -> Result<Arc<dyn StorageBackend>, DBError> {
|
|
||||||
let db_file = PathBuf::from(base_dir).join("0.db");
|
|
||||||
if let Some(parent_dir) = db_file.parent() {
|
|
||||||
std::fs::create_dir_all(parent_dir).map_err(|e| {
|
|
||||||
DBError(format!("Failed to create directory {}: {}", parent_dir.display(), e))
|
|
||||||
})?;
|
|
||||||
}
|
|
||||||
let storage: Arc<dyn StorageBackend> = match backend {
|
|
||||||
options::BackendType::Redb => Arc::new(Storage::new(&db_file, true, Some(admin_secret))?),
|
|
||||||
options::BackendType::Sled => Arc::new(SledStorage::new(&db_file, true, Some(admin_secret))?),
|
|
||||||
options::BackendType::Tantivy => {
|
|
||||||
return Err(DBError("Admin DB 0 cannot use Tantivy backend".to_string()))
|
|
||||||
}
|
|
||||||
};
|
|
||||||
Ok(storage)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get or initialize a cached handle to admin DB 0 per base_dir (thread-safe, no double-open race)
|
|
||||||
pub fn open_admin_storage(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
) -> Result<Arc<dyn StorageBackend>, DBError> {
|
|
||||||
let map = ADMIN_STORAGES.get_or_init(|| RwLock::new(HashMap::new()));
|
|
||||||
// Fast path
|
|
||||||
if let Some(st) = map.read().unwrap().get(base_dir) {
|
|
||||||
return Ok(st.clone());
|
|
||||||
}
|
|
||||||
// Slow path with write lock
|
|
||||||
{
|
|
||||||
let mut w = map.write().unwrap();
|
|
||||||
if let Some(st) = w.get(base_dir) {
|
|
||||||
return Ok(st.clone());
|
|
||||||
}
|
|
||||||
|
|
||||||
// Detect existing 0.db backend by filesystem, if present.
|
|
||||||
let admin_path = PathBuf::from(base_dir).join("0.db");
|
|
||||||
let detected = if admin_path.exists() {
|
|
||||||
if admin_path.is_file() {
|
|
||||||
Some(options::BackendType::Redb)
|
|
||||||
} else if admin_path.is_dir() {
|
|
||||||
Some(options::BackendType::Sled)
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
};
|
|
||||||
|
|
||||||
let effective_backend = match detected {
|
|
||||||
Some(d) if d != backend => {
|
|
||||||
eprintln!(
|
|
||||||
"warning: Admin DB 0 at {} appears to be {:?}, but process default is {:?}. Using detected backend.",
|
|
||||||
admin_path.display(),
|
|
||||||
d,
|
|
||||||
backend
|
|
||||||
);
|
|
||||||
d
|
|
||||||
}
|
|
||||||
Some(d) => d,
|
|
||||||
None => backend, // First boot: use requested backend to initialize 0.db
|
|
||||||
};
|
|
||||||
|
|
||||||
let st = init_admin_storage(base_dir, effective_backend, admin_secret)?;
|
|
||||||
w.insert(base_dir.to_string(), st.clone());
|
|
||||||
Ok(st)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Ensure admin structures exist in encrypted DB 0
|
|
||||||
pub fn ensure_bootstrap(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
) -> Result<(), DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
|
|
||||||
// Initialize next id if missing
|
|
||||||
if !admin.exists(k_admin_next_id())? {
|
|
||||||
admin.set(k_admin_next_id().to_string(), "1".to_string())?;
|
|
||||||
}
|
|
||||||
// admin:dbs is a hash; it's fine if it doesn't exist (hlen -> 0)
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get or initialize a shared handle to a data DB (> 0), avoiding double-open across subsystems
|
|
||||||
pub fn open_data_storage(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
) -> Result<Arc<dyn StorageBackend>, DBError> {
|
|
||||||
if id == 0 {
|
|
||||||
return open_admin_storage(base_dir, backend, admin_secret);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Validate existence in admin metadata
|
|
||||||
if !db_exists(base_dir, backend.clone(), admin_secret, id)? {
|
|
||||||
return Err(DBError(format!(
|
|
||||||
"Cannot open database instance {}, as that database instance does not exist.",
|
|
||||||
id
|
|
||||||
)));
|
|
||||||
}
|
|
||||||
|
|
||||||
let map = DATA_STORAGES.get_or_init(|| RwLock::new(HashMap::new()));
|
|
||||||
// Fast path
|
|
||||||
if let Some(st) = map.read().unwrap().get(&id) {
|
|
||||||
return Ok(st.clone());
|
|
||||||
}
|
|
||||||
|
|
||||||
// Slow path with init lock
|
|
||||||
let _guard = DATA_INIT_LOCK.lock().unwrap();
|
|
||||||
if let Some(st) = map.read().unwrap().get(&id) {
|
|
||||||
return Ok(st.clone());
|
|
||||||
}
|
|
||||||
|
|
||||||
// Resolve effective backend for this db id:
|
|
||||||
// 1) Try admin meta "backend" field
|
|
||||||
// 2) If missing, sniff filesystem (file => Redb, dir => Sled), then persist into admin meta
|
|
||||||
// 3) Fallback to requested 'backend' (startup default) if nothing else is known
|
|
||||||
let meta_backend = get_database_backend(base_dir, backend.clone(), admin_secret, id).ok().flatten();
|
|
||||||
let db_path = PathBuf::from(base_dir).join(format!("{}.db", id));
|
|
||||||
let sniffed_backend = if db_path.exists() {
|
|
||||||
if db_path.is_file() {
|
|
||||||
Some(options::BackendType::Redb)
|
|
||||||
} else if db_path.is_dir() {
|
|
||||||
Some(options::BackendType::Sled)
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
};
|
|
||||||
let effective_backend = meta_backend.clone().or(sniffed_backend).unwrap_or(backend.clone());
|
|
||||||
|
|
||||||
// If we had to sniff (i.e., meta missing), persist it for future robustness
|
|
||||||
if meta_backend.is_none() {
|
|
||||||
let _ = set_database_backend(base_dir, backend.clone(), admin_secret, id, effective_backend.clone());
|
|
||||||
}
|
|
||||||
|
|
||||||
// Warn if caller-provided backend differs from effective
|
|
||||||
if effective_backend != backend {
|
|
||||||
eprintln!(
|
|
||||||
"notice: Database {} backend resolved to {:?} (caller requested {:?}). Using resolved backend.",
|
|
||||||
id, effective_backend, backend
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Determine per-db encryption (from admin meta)
|
|
||||||
let enc = get_enc_key(base_dir, backend.clone(), admin_secret, id)?;
|
|
||||||
let should_encrypt = enc.is_some();
|
|
||||||
|
|
||||||
// Build database file path and ensure parent dir exists
|
|
||||||
let db_file = PathBuf::from(base_dir).join(format!("{}.db", id));
|
|
||||||
if let Some(parent_dir) = db_file.parent() {
|
|
||||||
std::fs::create_dir_all(parent_dir).map_err(|e| {
|
|
||||||
DBError(format!("Failed to create directory {}: {}", parent_dir.display(), e))
|
|
||||||
})?;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Open storage using the effective backend
|
|
||||||
let storage: Arc<dyn StorageBackend> = match effective_backend {
|
|
||||||
options::BackendType::Redb => Arc::new(Storage::new(&db_file, should_encrypt, enc.as_deref())?),
|
|
||||||
options::BackendType::Sled => Arc::new(SledStorage::new(&db_file, should_encrypt, enc.as_deref())?),
|
|
||||||
options::BackendType::Tantivy => {
|
|
||||||
return Err(DBError("Tantivy backend has no KV storage; use FT.* commands only".to_string()))
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
// Publish to registry
|
|
||||||
map.write().unwrap().insert(id, storage.clone());
|
|
||||||
Ok(storage)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Allocate the next DB id and persist new pointer
|
|
||||||
pub fn allocate_next_id(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
) -> Result<u64, DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let cur = admin
|
|
||||||
.get(k_admin_next_id())?
|
|
||||||
.unwrap_or_else(|| "1".to_string());
|
|
||||||
let id: u64 = cur.parse().unwrap_or(1);
|
|
||||||
let next = id.checked_add(1).ok_or_else(|| DBError("next_id overflow".into()))?;
|
|
||||||
admin.set(k_admin_next_id().to_string(), next.to_string())?;
|
|
||||||
|
|
||||||
// Register into admin:dbs set/hash
|
|
||||||
let _ = admin.hset(k_admin_dbs(), vec![(id.to_string(), "1".to_string())])?;
|
|
||||||
|
|
||||||
// Default meta for the new db: public true
|
|
||||||
let meta_key = k_meta_db(id);
|
|
||||||
let _ = admin.hset(&meta_key, vec![("public".to_string(), "true".to_string())])?;
|
|
||||||
|
|
||||||
Ok(id)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check existence of a db id in admin:dbs
|
|
||||||
pub fn db_exists(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
) -> Result<bool, DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
Ok(admin.hexists(k_admin_dbs(), &id.to_string())?)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get per-db encryption key, if any
|
|
||||||
pub fn get_enc_key(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
) -> Result<Option<String>, DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
admin.get(&k_meta_db_enc(id))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Set per-db encryption key (called during create)
|
|
||||||
pub fn set_enc_key(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
key: &str,
|
|
||||||
) -> Result<(), DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
admin.set(k_meta_db_enc(id), key.to_string())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Set database public flag
|
|
||||||
pub fn set_database_public(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
public: bool,
|
|
||||||
) -> Result<(), DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let mk = k_meta_db(id);
|
|
||||||
let _ = admin.hset(&mk, vec![("public".to_string(), public.to_string())])?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Persist per-db backend type in admin metadata (module-scope)
|
|
||||||
pub fn set_database_backend(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
db_backend: options::BackendType,
|
|
||||||
) -> Result<(), DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let mk = k_meta_db(id);
|
|
||||||
let val = match db_backend {
|
|
||||||
options::BackendType::Redb => "Redb",
|
|
||||||
options::BackendType::Sled => "Sled",
|
|
||||||
options::BackendType::Tantivy => "Tantivy",
|
|
||||||
};
|
|
||||||
let _ = admin.hset(&mk, vec![("backend".to_string(), val.to_string())])?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn get_database_backend(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
) -> Result<Option<options::BackendType>, DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let mk = k_meta_db(id);
|
|
||||||
match admin.hget(&mk, "backend")? {
|
|
||||||
Some(s) if s == "Redb" => Ok(Some(options::BackendType::Redb)),
|
|
||||||
Some(s) if s == "Sled" => Ok(Some(options::BackendType::Sled)),
|
|
||||||
Some(s) if s == "Tantivy" => Ok(Some(options::BackendType::Tantivy)),
|
|
||||||
_ => Ok(None),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Set database name
|
|
||||||
pub fn set_database_name(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
name: &str,
|
|
||||||
) -> Result<(), DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let mk = k_meta_db(id);
|
|
||||||
let _ = admin.hset(&mk, vec![("name".to_string(), name.to_string())])?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get database name
|
|
||||||
pub fn get_database_name(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
) -> Result<Option<String>, DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let mk = k_meta_db(id);
|
|
||||||
admin.hget(&mk, "name")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Internal: load public flag; default to true when meta missing
|
|
||||||
fn load_public(
|
|
||||||
admin: &Arc<dyn StorageBackend>,
|
|
||||||
id: u64,
|
|
||||||
) -> Result<bool, DBError> {
|
|
||||||
let mk = k_meta_db(id);
|
|
||||||
match admin.hget(&mk, "public")? {
|
|
||||||
Some(v) => Ok(v == "true"),
|
|
||||||
None => Ok(true),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Add access key for db (value format: "Read:ts" or "ReadWrite:ts")
|
|
||||||
pub fn add_access_key(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
key_plain: &str,
|
|
||||||
perms: Permissions,
|
|
||||||
) -> Result<(), DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let hash = crate::rpc::hash_key(key_plain);
|
|
||||||
let v = match perms {
|
|
||||||
Permissions::Read => format!("Read:{}", now_secs()),
|
|
||||||
Permissions::ReadWrite => format!("ReadWrite:{}", now_secs()),
|
|
||||||
};
|
|
||||||
let _ = admin.hset(&k_meta_db_keys(id), vec![(hash, v)])?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Delete access key by hash
|
|
||||||
pub fn delete_access_key(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
key_hash: &str,
|
|
||||||
) -> Result<bool, DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let n = admin.hdel(&k_meta_db_keys(id), vec![key_hash.to_string()])?;
|
|
||||||
Ok(n > 0)
|
|
||||||
}
|
|
||||||
|
|
||||||
// List access keys, returning (hash, perms, created_at_secs)
|
|
||||||
pub fn list_access_keys(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
) -> Result<Vec<(String, Permissions, u64)>, DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let pairs = admin.hgetall(&k_meta_db_keys(id))?;
|
|
||||||
let mut out = Vec::new();
|
|
||||||
for (hash, val) in pairs {
|
|
||||||
let (perm, ts) = parse_perm_value(&val);
|
|
||||||
out.push((hash, perm, ts));
|
|
||||||
}
|
|
||||||
Ok(out)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Verify access permission for db id with optional key
|
|
||||||
// Returns:
|
|
||||||
// - Ok(Some(Permissions)) when access is allowed
|
|
||||||
// - Ok(None) when not allowed or db missing (caller can distinguish by calling db_exists)
|
|
||||||
pub fn verify_access(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
id: u64,
|
|
||||||
key_opt: Option<&str>,
|
|
||||||
) -> Result<Option<Permissions>, DBError> {
|
|
||||||
// Admin DB 0: require exact admin_secret
|
|
||||||
if id == 0 {
|
|
||||||
if let Some(k) = key_opt {
|
|
||||||
if k == admin_secret {
|
|
||||||
return Ok(Some(Permissions::ReadWrite));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return Ok(None);
|
|
||||||
}
|
|
||||||
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
if !admin.hexists(k_admin_dbs(), &id.to_string())? {
|
|
||||||
return Ok(None);
|
|
||||||
}
|
|
||||||
|
|
||||||
let is_public = load_public(&admin, id)?;
|
|
||||||
|
|
||||||
// If a key is explicitly provided, enforce its validity strictly.
|
|
||||||
// Do NOT fall back to public when an invalid key is supplied.
|
|
||||||
if let Some(k) = key_opt {
|
|
||||||
let hash = crate::rpc::hash_key(k);
|
|
||||||
if let Some(v) = admin.hget(&k_meta_db_keys(id), &hash)? {
|
|
||||||
let (perm, _ts) = parse_perm_value(&v);
|
|
||||||
return Ok(Some(perm));
|
|
||||||
}
|
|
||||||
// Invalid key
|
|
||||||
return Ok(None);
|
|
||||||
}
|
|
||||||
|
|
||||||
// No key provided: allow access if DB is public, otherwise deny
|
|
||||||
if is_public {
|
|
||||||
Ok(Some(Permissions::ReadWrite))
|
|
||||||
} else {
|
|
||||||
Ok(None)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Enumerate all db ids
|
|
||||||
pub fn list_dbs(
|
|
||||||
base_dir: &str,
|
|
||||||
backend: options::BackendType,
|
|
||||||
admin_secret: &str,
|
|
||||||
) -> Result<Vec<u64>, DBError> {
|
|
||||||
let admin = open_admin_storage(base_dir, backend, admin_secret)?;
|
|
||||||
let ids = admin.hkeys(k_admin_dbs())?;
|
|
||||||
let mut out = Vec::new();
|
|
||||||
for s in ids {
|
|
||||||
if let Ok(v) = s.parse() {
|
|
||||||
out.push(v);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(out)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Helper: parse permission value "Read:ts" or "ReadWrite:ts"
|
|
||||||
fn parse_perm_value(v: &str) -> (Permissions, u64) {
|
|
||||||
let mut parts = v.split(':');
|
|
||||||
let p = parts.next().unwrap_or("Read");
|
|
||||||
let ts = parts
|
|
||||||
.next()
|
|
||||||
.and_then(|s| s.parse().ok())
|
|
||||||
.unwrap_or(0u64);
|
|
||||||
let perm = match p {
|
|
||||||
"ReadWrite" => Permissions::ReadWrite,
|
|
||||||
_ => Permissions::Read,
|
|
||||||
};
|
|
||||||
(perm, ts)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn now_secs() -> u64 {
|
|
||||||
use std::time::{SystemTime, UNIX_EPOCH};
|
|
||||||
SystemTime::now()
|
|
||||||
.duration_since(UNIX_EPOCH)
|
|
||||||
.unwrap_or_default()
|
|
||||||
.as_secs()
|
|
||||||
}
|
|
406
src/age.rs
406
src/age.rs
@@ -12,19 +12,17 @@
|
|||||||
|
|
||||||
use std::str::FromStr;
|
use std::str::FromStr;
|
||||||
|
|
||||||
use secrecy::ExposeSecret;
|
|
||||||
use age::{Decryptor, Encryptor};
|
|
||||||
use age::x25519;
|
use age::x25519;
|
||||||
|
use age::{Decryptor, Encryptor};
|
||||||
|
use secrecy::ExposeSecret;
|
||||||
|
|
||||||
use ed25519_dalek::{Signature, Signer, Verifier, SigningKey, VerifyingKey};
|
use ed25519_dalek::{Signature, Signer, SigningKey, Verifier, VerifyingKey};
|
||||||
|
|
||||||
use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
|
use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
|
||||||
use std::collections::HashSet;
|
|
||||||
use std::convert::TryInto;
|
|
||||||
|
|
||||||
|
use crate::error::DBError;
|
||||||
use crate::protocol::Protocol;
|
use crate::protocol::Protocol;
|
||||||
use crate::server::Server;
|
use crate::server::Server;
|
||||||
use crate::error::DBError;
|
|
||||||
|
|
||||||
// ---------- Internal helpers ----------
|
// ---------- Internal helpers ----------
|
||||||
|
|
||||||
@@ -76,125 +74,6 @@ fn parse_ed25519_verifying_key(s: &str) -> Result<VerifyingKey, AgeWireError> {
|
|||||||
VerifyingKey::from_bytes(&key_bytes).map_err(|_| AgeWireError::ParseKey)
|
VerifyingKey::from_bytes(&key_bytes).map_err(|_| AgeWireError::ParseKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------- Derivation + Raw X25519 (Ed25519 -> X25519) ----------
|
|
||||||
//
|
|
||||||
// We deterministically derive an X25519 keypair from an Ed25519 SigningKey.
|
|
||||||
// We persist the X25519 public/secret as base64-encoded 32-byte raw values
|
|
||||||
// (no "age1..."/"AGE-SECRET-KEY-1..." formatting). Name-based encrypt/decrypt
|
|
||||||
// uses these raw values directly via x25519-dalek + ChaCha20Poly1305.
|
|
||||||
|
|
||||||
use chacha20poly1305::{aead::{Aead, KeyInit}, ChaCha20Poly1305, Key, Nonce};
|
|
||||||
use sha2::{Digest, Sha256};
|
|
||||||
use x25519_dalek::{PublicKey as XPublicKey, StaticSecret as XStaticSecret};
|
|
||||||
|
|
||||||
fn derive_x25519_raw_from_ed25519(sk: &SigningKey) -> ([u8; 32], [u8; 32]) {
|
|
||||||
// X25519 secret scalar (clamped) from Ed25519 secret
|
|
||||||
let scalar: [u8; 32] = sk.to_scalar_bytes();
|
|
||||||
// Build X25519 secret/public using dalek
|
|
||||||
let xsec = XStaticSecret::from(scalar);
|
|
||||||
let xpub = XPublicKey::from(&xsec);
|
|
||||||
(xpub.to_bytes(), xsec.to_bytes())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn derive_x25519_raw_b64_from_ed25519(sk: &SigningKey) -> (String, String) {
|
|
||||||
let (xpub, xsec) = derive_x25519_raw_from_ed25519(sk);
|
|
||||||
(B64.encode(xpub), B64.encode(xsec))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Helper: detect whether a stored key looks like an age-formatted string
|
|
||||||
fn looks_like_age_format(s: &str) -> bool {
|
|
||||||
s.starts_with("age1") || s.starts_with("AGE-SECRET-KEY-1")
|
|
||||||
}
|
|
||||||
|
|
||||||
// Our container format for name-based raw X25519 encryption:
|
|
||||||
// bytes = "HDBX1" (5) || eph_pub(32) || nonce(12) || ciphertext(..)
|
|
||||||
// Entire blob is base64-encoded for transport.
|
|
||||||
const HDBX1_MAGIC: &[u8; 5] = b"HDBX1";
|
|
||||||
|
|
||||||
fn encrypt_b64_with_x25519_raw(recip_pub_b64: &str, msg: &str) -> Result<String, AgeWireError> {
|
|
||||||
use rand::RngCore;
|
|
||||||
use rand::rngs::OsRng;
|
|
||||||
|
|
||||||
// Parse recipient public key (raw 32 bytes, base64)
|
|
||||||
let recip_pub_bytes = B64.decode(recip_pub_b64).map_err(|_| AgeWireError::ParseKey)?;
|
|
||||||
if recip_pub_bytes.len() != 32 { return Err(AgeWireError::ParseKey); }
|
|
||||||
let recip_pub_arr: [u8; 32] = recip_pub_bytes.as_slice().try_into().map_err(|_| AgeWireError::ParseKey)?;
|
|
||||||
let recip_pub: XPublicKey = XPublicKey::from(recip_pub_arr);
|
|
||||||
|
|
||||||
// Generate ephemeral X25519 keypair
|
|
||||||
let mut eph_sec_bytes = [0u8; 32];
|
|
||||||
OsRng.fill_bytes(&mut eph_sec_bytes);
|
|
||||||
let eph_sec = XStaticSecret::from(eph_sec_bytes);
|
|
||||||
let eph_pub = XPublicKey::from(&eph_sec);
|
|
||||||
|
|
||||||
// ECDH
|
|
||||||
let shared = eph_sec.diffie_hellman(&recip_pub);
|
|
||||||
// Derive symmetric key via SHA-256 over context + shared + parties
|
|
||||||
let mut hasher = Sha256::default();
|
|
||||||
hasher.update(b"herodb-x25519-v1");
|
|
||||||
hasher.update(shared.as_bytes());
|
|
||||||
hasher.update(eph_pub.as_bytes());
|
|
||||||
hasher.update(recip_pub.as_bytes());
|
|
||||||
let key_bytes = hasher.finalize();
|
|
||||||
let key = Key::from_slice(&key_bytes[..32]);
|
|
||||||
|
|
||||||
// Nonce (12 bytes)
|
|
||||||
let mut nonce_bytes = [0u8; 12];
|
|
||||||
OsRng.fill_bytes(&mut nonce_bytes);
|
|
||||||
let nonce = Nonce::from_slice(&nonce_bytes);
|
|
||||||
|
|
||||||
// Encrypt
|
|
||||||
let cipher = ChaCha20Poly1305::new(key);
|
|
||||||
let ct = cipher.encrypt(nonce, msg.as_bytes())
|
|
||||||
.map_err(|e| AgeWireError::Crypto(format!("encrypt: {e}")))?;
|
|
||||||
|
|
||||||
// Assemble container
|
|
||||||
let mut out = Vec::with_capacity(5 + 32 + 12 + ct.len());
|
|
||||||
out.extend_from_slice(HDBX1_MAGIC);
|
|
||||||
out.extend_from_slice(eph_pub.as_bytes());
|
|
||||||
out.extend_from_slice(&nonce_bytes);
|
|
||||||
out.extend_from_slice(&ct);
|
|
||||||
|
|
||||||
Ok(B64.encode(out))
|
|
||||||
}
|
|
||||||
|
|
||||||
fn decrypt_b64_with_x25519_raw(identity_sec_b64: &str, ct_b64: &str) -> Result<String, AgeWireError> {
|
|
||||||
// Parse X25519 secret (raw 32 bytes, base64)
|
|
||||||
let sec_bytes = B64.decode(identity_sec_b64).map_err(|_| AgeWireError::ParseKey)?;
|
|
||||||
if sec_bytes.len() != 32 { return Err(AgeWireError::ParseKey); }
|
|
||||||
let sec_arr: [u8; 32] = sec_bytes.as_slice().try_into().map_err(|_| AgeWireError::ParseKey)?;
|
|
||||||
let xsec = XStaticSecret::from(sec_arr);
|
|
||||||
let xpub = XPublicKey::from(&xsec); // self public
|
|
||||||
|
|
||||||
// Decode container
|
|
||||||
let blob = B64.decode(ct_b64.as_bytes()).map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
|
||||||
if blob.len() < 5 + 32 + 12 { return Err(AgeWireError::Crypto("ciphertext too short".to_string())); }
|
|
||||||
if &blob[..5] != HDBX1_MAGIC { return Err(AgeWireError::Crypto("bad header".to_string())); }
|
|
||||||
|
|
||||||
let eph_pub_arr: [u8; 32] = blob[5..5+32].try_into().map_err(|_| AgeWireError::Crypto("bad eph pub".to_string()))?;
|
|
||||||
let eph_pub = XPublicKey::from(eph_pub_arr);
|
|
||||||
let nonce_bytes: [u8; 12] = blob[5+32..5+32+12].try_into().unwrap();
|
|
||||||
let ct = &blob[5+32+12..];
|
|
||||||
|
|
||||||
// Recompute shared + key
|
|
||||||
let shared = xsec.diffie_hellman(&eph_pub);
|
|
||||||
let mut hasher = Sha256::default();
|
|
||||||
hasher.update(b"herodb-x25519-v1");
|
|
||||||
hasher.update(shared.as_bytes());
|
|
||||||
hasher.update(eph_pub.as_bytes());
|
|
||||||
hasher.update(xpub.as_bytes());
|
|
||||||
let key_bytes = hasher.finalize();
|
|
||||||
let key = Key::from_slice(&key_bytes[..32]);
|
|
||||||
|
|
||||||
// Decrypt
|
|
||||||
let cipher = ChaCha20Poly1305::new(key);
|
|
||||||
let nonce = Nonce::from_slice(&nonce_bytes);
|
|
||||||
let pt = cipher.decrypt(nonce, ct)
|
|
||||||
.map_err(|e| AgeWireError::Crypto(format!("decrypt: {e}")))?;
|
|
||||||
|
|
||||||
String::from_utf8(pt).map_err(|_| AgeWireError::Utf8)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------- Stateless crypto helpers (string in/out) ----------
|
// ---------- Stateless crypto helpers (string in/out) ----------
|
||||||
|
|
||||||
pub fn gen_enc_keypair() -> (String, String) {
|
pub fn gen_enc_keypair() -> (String, String) {
|
||||||
@@ -204,8 +83,8 @@ pub fn gen_enc_keypair() -> (String, String) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
pub fn gen_sign_keypair() -> (String, String) {
|
pub fn gen_sign_keypair() -> (String, String) {
|
||||||
use rand::RngCore;
|
|
||||||
use rand::rngs::OsRng;
|
use rand::rngs::OsRng;
|
||||||
|
use rand::RngCore;
|
||||||
|
|
||||||
// Generate random 32 bytes for the signing key
|
// Generate random 32 bytes for the signing key
|
||||||
let mut secret_bytes = [0u8; 32];
|
let mut secret_bytes = [0u8; 32];
|
||||||
@@ -224,14 +103,18 @@ pub fn gen_sign_keypair() -> (String, String) {
|
|||||||
/// Encrypt `msg` for `recipient_str` (X25519). Returns base64(ciphertext).
|
/// Encrypt `msg` for `recipient_str` (X25519). Returns base64(ciphertext).
|
||||||
pub fn encrypt_b64(recipient_str: &str, msg: &str) -> Result<String, AgeWireError> {
|
pub fn encrypt_b64(recipient_str: &str, msg: &str) -> Result<String, AgeWireError> {
|
||||||
let recipient = parse_recipient(recipient_str)?;
|
let recipient = parse_recipient(recipient_str)?;
|
||||||
let enc = Encryptor::with_recipients(vec![Box::new(recipient)])
|
let enc =
|
||||||
.expect("failed to create encryptor"); // Handle Option<Encryptor>
|
Encryptor::with_recipients(vec![Box::new(recipient)]).expect("failed to create encryptor"); // Handle Option<Encryptor>
|
||||||
let mut out = Vec::new();
|
let mut out = Vec::new();
|
||||||
{
|
{
|
||||||
use std::io::Write;
|
use std::io::Write;
|
||||||
let mut w = enc.wrap_output(&mut out).map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
let mut w = enc
|
||||||
w.write_all(msg.as_bytes()).map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
.wrap_output(&mut out)
|
||||||
w.finish().map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
.map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
||||||
|
w.write_all(msg.as_bytes())
|
||||||
|
.map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
||||||
|
w.finish()
|
||||||
|
.map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
||||||
}
|
}
|
||||||
Ok(B64.encode(out))
|
Ok(B64.encode(out))
|
||||||
}
|
}
|
||||||
@@ -239,19 +122,27 @@ pub fn encrypt_b64(recipient_str: &str, msg: &str) -> Result<String, AgeWireErro
|
|||||||
/// Decrypt base64(ciphertext) with `identity_str`. Returns plaintext String.
|
/// Decrypt base64(ciphertext) with `identity_str`. Returns plaintext String.
|
||||||
pub fn decrypt_b64(identity_str: &str, ct_b64: &str) -> Result<String, AgeWireError> {
|
pub fn decrypt_b64(identity_str: &str, ct_b64: &str) -> Result<String, AgeWireError> {
|
||||||
let id = parse_identity(identity_str)?;
|
let id = parse_identity(identity_str)?;
|
||||||
let ct = B64.decode(ct_b64.as_bytes()).map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
let ct = B64
|
||||||
|
.decode(ct_b64.as_bytes())
|
||||||
|
.map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
||||||
let dec = Decryptor::new(&ct[..]).map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
let dec = Decryptor::new(&ct[..]).map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
||||||
|
|
||||||
// The decrypt method returns a Result<StreamReader, DecryptError>
|
// The decrypt method returns a Result<StreamReader, DecryptError>
|
||||||
let mut r = match dec {
|
let mut r = match dec {
|
||||||
Decryptor::Recipients(d) => d.decrypt(std::iter::once(&id as &dyn age::Identity))
|
Decryptor::Recipients(d) => d
|
||||||
|
.decrypt(std::iter::once(&id as &dyn age::Identity))
|
||||||
.map_err(|e| AgeWireError::Crypto(e.to_string()))?,
|
.map_err(|e| AgeWireError::Crypto(e.to_string()))?,
|
||||||
Decryptor::Passphrase(_) => return Err(AgeWireError::Crypto("Expected recipients, got passphrase".to_string())),
|
Decryptor::Passphrase(_) => {
|
||||||
|
return Err(AgeWireError::Crypto(
|
||||||
|
"Expected recipients, got passphrase".to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut pt = Vec::new();
|
let mut pt = Vec::new();
|
||||||
use std::io::Read;
|
use std::io::Read;
|
||||||
r.read_to_end(&mut pt).map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
r.read_to_end(&mut pt)
|
||||||
|
.map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
||||||
String::from_utf8(pt).map_err(|_| AgeWireError::Utf8)
|
String::from_utf8(pt).map_err(|_| AgeWireError::Utf8)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -265,7 +156,9 @@ pub fn sign_b64(signing_secret_str: &str, msg: &str) -> Result<String, AgeWireEr
|
|||||||
/// Verify detached signature (base64) for `msg` with pubkey.
|
/// Verify detached signature (base64) for `msg` with pubkey.
|
||||||
pub fn verify_b64(verify_pub_str: &str, msg: &str, sig_b64: &str) -> Result<bool, AgeWireError> {
|
pub fn verify_b64(verify_pub_str: &str, msg: &str, sig_b64: &str) -> Result<bool, AgeWireError> {
|
||||||
let verifying_key = parse_ed25519_verifying_key(verify_pub_str)?;
|
let verifying_key = parse_ed25519_verifying_key(verify_pub_str)?;
|
||||||
let sig_bytes = B64.decode(sig_b64.as_bytes()).map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
let sig_bytes = B64
|
||||||
|
.decode(sig_b64.as_bytes())
|
||||||
|
.map_err(|e| AgeWireError::Crypto(e.to_string()))?;
|
||||||
if sig_bytes.len() != 64 {
|
if sig_bytes.len() != 64 {
|
||||||
return Err(AgeWireError::SignatureLen);
|
return Err(AgeWireError::SignatureLen);
|
||||||
}
|
}
|
||||||
@@ -276,30 +169,49 @@ pub fn verify_b64(verify_pub_str: &str, msg: &str, sig_b64: &str) -> Result<bool
|
|||||||
// ---------- Storage helpers ----------
|
// ---------- Storage helpers ----------
|
||||||
|
|
||||||
fn sget(server: &Server, key: &str) -> Result<Option<String>, AgeWireError> {
|
fn sget(server: &Server, key: &str) -> Result<Option<String>, AgeWireError> {
|
||||||
let st = server.current_storage().map_err(|e| AgeWireError::Storage(e.0))?;
|
let st = server
|
||||||
|
.current_storage()
|
||||||
|
.map_err(|e| AgeWireError::Storage(e.0))?;
|
||||||
st.get(key).map_err(|e| AgeWireError::Storage(e.0))
|
st.get(key).map_err(|e| AgeWireError::Storage(e.0))
|
||||||
}
|
}
|
||||||
fn sset(server: &Server, key: &str, val: &str) -> Result<(), AgeWireError> {
|
fn sset(server: &Server, key: &str, val: &str) -> Result<(), AgeWireError> {
|
||||||
let st = server.current_storage().map_err(|e| AgeWireError::Storage(e.0))?;
|
let st = server
|
||||||
st.set(key.to_string(), val.to_string()).map_err(|e| AgeWireError::Storage(e.0))
|
.current_storage()
|
||||||
|
.map_err(|e| AgeWireError::Storage(e.0))?;
|
||||||
|
st.set(key.to_string(), val.to_string())
|
||||||
|
.map_err(|e| AgeWireError::Storage(e.0))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn enc_pub_key_key(name: &str) -> String { format!("age:key:{name}") }
|
fn enc_pub_key_key(name: &str) -> String {
|
||||||
fn enc_priv_key_key(name: &str) -> String { format!("age:privkey:{name}") }
|
format!("age:key:{name}")
|
||||||
fn sign_pub_key_key(name: &str) -> String { format!("age:signpub:{name}") }
|
}
|
||||||
fn sign_priv_key_key(name: &str) -> String { format!("age:signpriv:{name}") }
|
fn enc_priv_key_key(name: &str) -> String {
|
||||||
|
format!("age:privkey:{name}")
|
||||||
|
}
|
||||||
|
fn sign_pub_key_key(name: &str) -> String {
|
||||||
|
format!("age:signpub:{name}")
|
||||||
|
}
|
||||||
|
fn sign_priv_key_key(name: &str) -> String {
|
||||||
|
format!("age:signpriv:{name}")
|
||||||
|
}
|
||||||
|
|
||||||
// ---------- Command handlers (RESP Protocol) ----------
|
// ---------- Command handlers (RESP Protocol) ----------
|
||||||
// Basic (stateless) ones kept for completeness
|
// Basic (stateless) ones kept for completeness
|
||||||
|
|
||||||
pub async fn cmd_age_genenc() -> Protocol {
|
pub async fn cmd_age_genenc() -> Protocol {
|
||||||
let (recip, ident) = gen_enc_keypair();
|
let (recip, ident) = gen_enc_keypair();
|
||||||
Protocol::Array(vec![Protocol::BulkString(recip), Protocol::BulkString(ident)])
|
Protocol::Array(vec![
|
||||||
|
Protocol::BulkString(recip),
|
||||||
|
Protocol::BulkString(ident),
|
||||||
|
])
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn cmd_age_gensign() -> Protocol {
|
pub async fn cmd_age_gensign() -> Protocol {
|
||||||
let (verify, secret) = gen_sign_keypair();
|
let (verify, secret) = gen_sign_keypair();
|
||||||
Protocol::Array(vec![Protocol::BulkString(verify), Protocol::BulkString(secret)])
|
Protocol::Array(vec![
|
||||||
|
Protocol::BulkString(verify),
|
||||||
|
Protocol::BulkString(secret),
|
||||||
|
])
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn cmd_age_encrypt(recipient: &str, message: &str) -> Protocol {
|
pub async fn cmd_age_encrypt(recipient: &str, message: &str) -> Protocol {
|
||||||
@@ -331,159 +243,66 @@ pub async fn cmd_age_verify(verify_pub: &str, message: &str, sig_b64: &str) -> P
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------- NEW: unified stateless generator (Ed25519 + derived X25519 raw) ----------
|
|
||||||
//
|
|
||||||
// Returns 4-tuple:
|
|
||||||
// [ verify_pub_b64 (32B), signpriv_b64 (32B), x25519_pub_b64 (32B), x25519_sec_b64 (32B) ]
|
|
||||||
// No persistence (stateless).
|
|
||||||
pub async fn cmd_age_genkey() -> Protocol {
|
|
||||||
use rand::RngCore;
|
|
||||||
use rand::rngs::OsRng;
|
|
||||||
|
|
||||||
let mut secret_bytes = [0u8; 32];
|
|
||||||
OsRng.fill_bytes(&mut secret_bytes);
|
|
||||||
|
|
||||||
let signing_key = SigningKey::from_bytes(&secret_bytes);
|
|
||||||
let verifying_key = signing_key.verifying_key();
|
|
||||||
|
|
||||||
let verify_b64 = B64.encode(verifying_key.to_bytes());
|
|
||||||
let sign_b64 = B64.encode(signing_key.to_bytes());
|
|
||||||
|
|
||||||
let (xpub_b64, xsec_b64) = derive_x25519_raw_b64_from_ed25519(&signing_key);
|
|
||||||
|
|
||||||
Protocol::Array(vec![
|
|
||||||
Protocol::BulkString(verify_b64),
|
|
||||||
Protocol::BulkString(sign_b64),
|
|
||||||
Protocol::BulkString(xpub_b64),
|
|
||||||
Protocol::BulkString(xsec_b64),
|
|
||||||
])
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------- NEW: Persistent, named-key commands ----------
|
// ---------- NEW: Persistent, named-key commands ----------
|
||||||
|
|
||||||
pub async fn cmd_age_keygen(server: &Server, name: &str) -> Protocol {
|
pub async fn cmd_age_keygen(server: &Server, name: &str) -> Protocol {
|
||||||
use rand::RngCore;
|
let (recip, ident) = gen_enc_keypair();
|
||||||
use rand::rngs::OsRng;
|
if let Err(e) = sset(server, &enc_pub_key_key(name), &recip) {
|
||||||
|
return e.to_protocol();
|
||||||
// Generate Ed25519 keypair
|
}
|
||||||
let mut secret_bytes = [0u8; 32];
|
if let Err(e) = sset(server, &enc_priv_key_key(name), &ident) {
|
||||||
OsRng.fill_bytes(&mut secret_bytes);
|
return e.to_protocol();
|
||||||
let signing_key = SigningKey::from_bytes(&secret_bytes);
|
}
|
||||||
let verifying_key = signing_key.verifying_key();
|
|
||||||
|
|
||||||
// Encode Ed25519 as base64 (32 bytes)
|
|
||||||
let verify_b64 = B64.encode(verifying_key.to_bytes());
|
|
||||||
let sign_b64 = B64.encode(signing_key.to_bytes());
|
|
||||||
|
|
||||||
// Derive X25519 raw (32-byte) keys and encode as base64
|
|
||||||
let (xpub_b64, xsec_b64) = derive_x25519_raw_b64_from_ed25519(&signing_key);
|
|
||||||
|
|
||||||
// Decode to create age-formatted strings
|
|
||||||
let xpub_bytes = B64.decode(&xpub_b64).unwrap();
|
|
||||||
let xsec_bytes = B64.decode(&xsec_b64).unwrap();
|
|
||||||
let xpub_arr: [u8; 32] = xpub_bytes.as_slice().try_into().unwrap();
|
|
||||||
let xsec_arr: [u8; 32] = xsec_bytes.as_slice().try_into().unwrap();
|
|
||||||
let recip_str = format!("age1{}", B64.encode(xpub_arr));
|
|
||||||
let ident_str = format!("AGE-SECRET-KEY-1{}", B64.encode(xsec_arr));
|
|
||||||
|
|
||||||
// Persist Ed25519 and derived X25519 (key-managed mode)
|
|
||||||
if let Err(e) = sset(server, &sign_pub_key_key(name), &verify_b64) { return e.to_protocol(); }
|
|
||||||
if let Err(e) = sset(server, &sign_priv_key_key(name), &sign_b64) { return e.to_protocol(); }
|
|
||||||
if let Err(e) = sset(server, &enc_pub_key_key(name), &xpub_b64) { return e.to_protocol(); }
|
|
||||||
if let Err(e) = sset(server, &enc_priv_key_key(name), &xsec_b64) { return e.to_protocol(); }
|
|
||||||
|
|
||||||
// Return [recipient, identity] in age format
|
|
||||||
Protocol::Array(vec![
|
Protocol::Array(vec![
|
||||||
Protocol::BulkString(recip_str),
|
Protocol::BulkString(recip),
|
||||||
Protocol::BulkString(ident_str),
|
Protocol::BulkString(ident),
|
||||||
])
|
])
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn cmd_age_signkeygen(server: &Server, name: &str) -> Protocol {
|
pub async fn cmd_age_signkeygen(server: &Server, name: &str) -> Protocol {
|
||||||
let (verify, secret) = gen_sign_keypair();
|
let (verify, secret) = gen_sign_keypair();
|
||||||
if let Err(e) = sset(server, &sign_pub_key_key(name), &verify) { return e.to_protocol(); }
|
if let Err(e) = sset(server, &sign_pub_key_key(name), &verify) {
|
||||||
if let Err(e) = sset(server, &sign_priv_key_key(name), &secret) { return e.to_protocol(); }
|
return e.to_protocol();
|
||||||
Protocol::Array(vec![Protocol::BulkString(verify), Protocol::BulkString(secret)])
|
}
|
||||||
|
if let Err(e) = sset(server, &sign_priv_key_key(name), &secret) {
|
||||||
|
return e.to_protocol();
|
||||||
|
}
|
||||||
|
Protocol::Array(vec![
|
||||||
|
Protocol::BulkString(verify),
|
||||||
|
Protocol::BulkString(secret),
|
||||||
|
])
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn cmd_age_encrypt_name(server: &Server, name: &str, message: &str) -> Protocol {
|
pub async fn cmd_age_encrypt_name(server: &Server, name: &str, message: &str) -> Protocol {
|
||||||
// Load stored recipient (could be raw b64 32-byte or "age1..." from legacy)
|
let recip = match sget(server, &enc_pub_key_key(name)) {
|
||||||
let recip_or_b64 = match sget(server, &enc_pub_key_key(name)) {
|
|
||||||
Ok(Some(v)) => v,
|
Ok(Some(v)) => v,
|
||||||
Ok(None) => {
|
|
||||||
// Derive from stored Ed25519 if present, then persist
|
|
||||||
match sget(server, &sign_priv_key_key(name)) {
|
|
||||||
Ok(Some(sign_b64)) => {
|
|
||||||
let sk = match parse_ed25519_signing_key(&sign_b64) {
|
|
||||||
Ok(k) => k,
|
|
||||||
Err(e) => return e.to_protocol(),
|
|
||||||
};
|
|
||||||
let (xpub_b64, xsec_b64) = derive_x25519_raw_b64_from_ed25519(&sk);
|
|
||||||
if let Err(e) = sset(server, &enc_pub_key_key(name), &xpub_b64) { return e.to_protocol(); }
|
|
||||||
if let Err(e) = sset(server, &enc_priv_key_key(name), &xsec_b64) { return e.to_protocol(); }
|
|
||||||
xpub_b64
|
|
||||||
}
|
|
||||||
Ok(None) => return AgeWireError::NotFound("recipient (age:key:{name})").to_protocol(),
|
Ok(None) => return AgeWireError::NotFound("recipient (age:key:{name})").to_protocol(),
|
||||||
Err(e) => return e.to_protocol(),
|
Err(e) => return e.to_protocol(),
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => return e.to_protocol(),
|
|
||||||
};
|
};
|
||||||
|
match encrypt_b64(&recip, message) {
|
||||||
if looks_like_age_format(&recip_or_b64) {
|
|
||||||
match encrypt_b64(&recip_or_b64, message) {
|
|
||||||
Ok(ct) => Protocol::BulkString(ct),
|
Ok(ct) => Protocol::BulkString(ct),
|
||||||
Err(e) => e.to_protocol(),
|
Err(e) => e.to_protocol(),
|
||||||
}
|
}
|
||||||
} else {
|
|
||||||
match encrypt_b64_with_x25519_raw(&recip_or_b64, message) {
|
|
||||||
Ok(ct) => Protocol::BulkString(ct),
|
|
||||||
Err(e) => e.to_protocol(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn cmd_age_decrypt_name(server: &Server, name: &str, ct_b64: &str) -> Protocol {
|
pub async fn cmd_age_decrypt_name(server: &Server, name: &str, ct_b64: &str) -> Protocol {
|
||||||
// Load stored identity (could be raw b64 32-byte or "AGE-SECRET-KEY-1..." from legacy)
|
let ident = match sget(server, &enc_priv_key_key(name)) {
|
||||||
let ident_or_b64 = match sget(server, &enc_priv_key_key(name)) {
|
|
||||||
Ok(Some(v)) => v,
|
Ok(Some(v)) => v,
|
||||||
Ok(None) => {
|
|
||||||
// Derive from stored Ed25519 if present, then persist
|
|
||||||
match sget(server, &sign_priv_key_key(name)) {
|
|
||||||
Ok(Some(sign_b64)) => {
|
|
||||||
let sk = match parse_ed25519_signing_key(&sign_b64) {
|
|
||||||
Ok(k) => k,
|
|
||||||
Err(e) => return e.to_protocol(),
|
|
||||||
};
|
|
||||||
let (xpub_b64, xsec_b64) = derive_x25519_raw_b64_from_ed25519(&sk);
|
|
||||||
if let Err(e) = sset(server, &enc_pub_key_key(name), &xpub_b64) { return e.to_protocol(); }
|
|
||||||
if let Err(e) = sset(server, &enc_priv_key_key(name), &xsec_b64) { return e.to_protocol(); }
|
|
||||||
xsec_b64
|
|
||||||
}
|
|
||||||
Ok(None) => return AgeWireError::NotFound("identity (age:privkey:{name})").to_protocol(),
|
Ok(None) => return AgeWireError::NotFound("identity (age:privkey:{name})").to_protocol(),
|
||||||
Err(e) => return e.to_protocol(),
|
Err(e) => return e.to_protocol(),
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => return e.to_protocol(),
|
|
||||||
};
|
};
|
||||||
|
match decrypt_b64(&ident, ct_b64) {
|
||||||
if looks_like_age_format(&ident_or_b64) {
|
|
||||||
match decrypt_b64(&ident_or_b64, ct_b64) {
|
|
||||||
Ok(pt) => Protocol::BulkString(pt),
|
Ok(pt) => Protocol::BulkString(pt),
|
||||||
Err(e) => e.to_protocol(),
|
Err(e) => e.to_protocol(),
|
||||||
}
|
}
|
||||||
} else {
|
|
||||||
match decrypt_b64_with_x25519_raw(&ident_or_b64, ct_b64) {
|
|
||||||
Ok(pt) => Protocol::BulkString(pt),
|
|
||||||
Err(e) => e.to_protocol(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn cmd_age_sign_name(server: &Server, name: &str, message: &str) -> Protocol {
|
pub async fn cmd_age_sign_name(server: &Server, name: &str, message: &str) -> Protocol {
|
||||||
let sec = match sget(server, &sign_priv_key_key(name)) {
|
let sec = match sget(server, &sign_priv_key_key(name)) {
|
||||||
Ok(Some(v)) => v,
|
Ok(Some(v)) => v,
|
||||||
Ok(None) => return AgeWireError::NotFound("signing secret (age:signpriv:{name})").to_protocol(),
|
Ok(None) => {
|
||||||
|
return AgeWireError::NotFound("signing secret (age:signpriv:{name})").to_protocol()
|
||||||
|
}
|
||||||
Err(e) => return e.to_protocol(),
|
Err(e) => return e.to_protocol(),
|
||||||
};
|
};
|
||||||
match sign_b64(&sec, message) {
|
match sign_b64(&sec, message) {
|
||||||
@@ -492,10 +311,17 @@ pub async fn cmd_age_sign_name(server: &Server, name: &str, message: &str) -> Pr
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn cmd_age_verify_name(server: &Server, name: &str, message: &str, sig_b64: &str) -> Protocol {
|
pub async fn cmd_age_verify_name(
|
||||||
|
server: &Server,
|
||||||
|
name: &str,
|
||||||
|
message: &str,
|
||||||
|
sig_b64: &str,
|
||||||
|
) -> Protocol {
|
||||||
let pubk = match sget(server, &sign_pub_key_key(name)) {
|
let pubk = match sget(server, &sign_pub_key_key(name)) {
|
||||||
Ok(Some(v)) => v,
|
Ok(Some(v)) => v,
|
||||||
Ok(None) => return AgeWireError::NotFound("verify pubkey (age:signpub:{name})").to_protocol(),
|
Ok(None) => {
|
||||||
|
return AgeWireError::NotFound("verify pubkey (age:signpub:{name})").to_protocol()
|
||||||
|
}
|
||||||
Err(e) => return e.to_protocol(),
|
Err(e) => return e.to_protocol(),
|
||||||
};
|
};
|
||||||
match verify_b64(&pubk, message, sig_b64) {
|
match verify_b64(&pubk, message, sig_b64) {
|
||||||
@@ -506,8 +332,11 @@ pub async fn cmd_age_verify_name(server: &Server, name: &str, message: &str, sig
|
|||||||
}
|
}
|
||||||
|
|
||||||
pub async fn cmd_age_list(server: &Server) -> Protocol {
|
pub async fn cmd_age_list(server: &Server) -> Protocol {
|
||||||
// Return a flat, deduplicated, sorted list of managed key names (no labels)
|
// Returns 4 arrays: ["encpub", <names...>], ["encpriv", ...], ["signpub", ...], ["signpriv", ...]
|
||||||
let st = match server.current_storage() { Ok(s) => s, Err(e) => return Protocol::err(&e.0) };
|
let st = match server.current_storage() {
|
||||||
|
Ok(s) => s,
|
||||||
|
Err(e) => return Protocol::err(&e.0),
|
||||||
|
};
|
||||||
|
|
||||||
let pull = |pat: &str, prefix: &str| -> Result<Vec<String>, DBError> {
|
let pull = |pat: &str, prefix: &str| -> Result<Vec<String>, DBError> {
|
||||||
let keys = st.keys(pat)?;
|
let keys = st.keys(pat)?;
|
||||||
@@ -519,18 +348,35 @@ pub async fn cmd_age_list(server: &Server) -> Protocol {
|
|||||||
Ok(names)
|
Ok(names)
|
||||||
};
|
};
|
||||||
|
|
||||||
let encpub = match pull("age:key:*", "age:key:") { Ok(v) => v, Err(e)=> return Protocol::err(&e.0) };
|
let encpub = match pull("age:key:*", "age:key:") {
|
||||||
let encpriv = match pull("age:privkey:*", "age:privkey:") { Ok(v) => v, Err(e)=> return Protocol::err(&e.0) };
|
Ok(v) => v,
|
||||||
let signpub = match pull("age:signpub:*", "age:signpub:") { Ok(v) => v, Err(e)=> return Protocol::err(&e.0) };
|
Err(e) => return Protocol::err(&e.0),
|
||||||
let signpriv = match pull("age:signpriv:*", "age:signpriv:") { Ok(v) => v, Err(e)=> return Protocol::err(&e.0) };
|
};
|
||||||
|
let encpriv = match pull("age:privkey:*", "age:privkey:") {
|
||||||
|
Ok(v) => v,
|
||||||
|
Err(e) => return Protocol::err(&e.0),
|
||||||
|
};
|
||||||
|
let signpub = match pull("age:signpub:*", "age:signpub:") {
|
||||||
|
Ok(v) => v,
|
||||||
|
Err(e) => return Protocol::err(&e.0),
|
||||||
|
};
|
||||||
|
let signpriv = match pull("age:signpriv:*", "age:signpriv:") {
|
||||||
|
Ok(v) => v,
|
||||||
|
Err(e) => return Protocol::err(&e.0),
|
||||||
|
};
|
||||||
|
|
||||||
let mut set: HashSet<String> = HashSet::new();
|
let to_arr = |label: &str, v: Vec<String>| {
|
||||||
for n in encpub.into_iter().chain(encpriv).chain(signpub).chain(signpriv) {
|
let mut out = vec![Protocol::BulkString(label.to_string())];
|
||||||
set.insert(n);
|
out.push(Protocol::Array(
|
||||||
}
|
v.into_iter().map(Protocol::BulkString).collect(),
|
||||||
|
));
|
||||||
|
Protocol::Array(out)
|
||||||
|
};
|
||||||
|
|
||||||
let mut names: Vec<String> = set.into_iter().collect();
|
Protocol::Array(vec![
|
||||||
names.sort();
|
to_arr("encpub", encpub),
|
||||||
|
to_arr("encpriv", encpriv),
|
||||||
Protocol::Array(names.into_iter().map(Protocol::BulkString).collect())
|
to_arr("signpub", signpub),
|
||||||
|
to_arr("signpriv", signpriv),
|
||||||
|
])
|
||||||
}
|
}
|
1109
src/cmd.rs
1109
src/cmd.rs
File diff suppressed because it is too large
Load Diff
@@ -1,9 +1,8 @@
|
|||||||
use std::num::ParseIntError;
|
use std::num::ParseIntError;
|
||||||
|
|
||||||
use tokio::sync::mpsc;
|
|
||||||
use redb;
|
|
||||||
use bincode;
|
use bincode;
|
||||||
|
use redb;
|
||||||
|
use tokio::sync::mpsc;
|
||||||
|
|
||||||
// todo: more error types
|
// todo: more error types
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
|
12
src/lib.rs
12
src/lib.rs
@@ -1,16 +1,12 @@
|
|||||||
pub mod age;
|
pub mod age; // NEW
|
||||||
pub mod sym;
|
|
||||||
pub mod cmd;
|
pub mod cmd;
|
||||||
pub mod crypto;
|
pub mod crypto;
|
||||||
pub mod error;
|
pub mod error;
|
||||||
pub mod options;
|
pub mod options;
|
||||||
pub mod protocol;
|
pub mod protocol;
|
||||||
pub mod rpc;
|
pub mod search_cmd; // Add this
|
||||||
pub mod rpc_server;
|
|
||||||
pub mod server;
|
pub mod server;
|
||||||
pub mod storage;
|
pub mod storage;
|
||||||
pub mod storage_trait;
|
pub mod storage_sled; // Add this
|
||||||
pub mod storage_sled;
|
pub mod storage_trait; // Add this
|
||||||
pub mod admin_meta;
|
|
||||||
pub mod tantivy_search;
|
pub mod tantivy_search;
|
||||||
pub mod search_cmd;
|
|
||||||
|
57
src/main.rs
57
src/main.rs
@@ -3,7 +3,6 @@
|
|||||||
use tokio::net::TcpListener;
|
use tokio::net::TcpListener;
|
||||||
|
|
||||||
use herodb::server;
|
use herodb::server;
|
||||||
use herodb::rpc_server;
|
|
||||||
|
|
||||||
use clap::Parser;
|
use clap::Parser;
|
||||||
|
|
||||||
@@ -23,29 +22,17 @@ struct Args {
|
|||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
debug: bool,
|
debug: bool,
|
||||||
|
|
||||||
/// Master encryption key for encrypted databases (deprecated; ignored for data DBs)
|
/// Master encryption key for encrypted databases
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
encryption_key: Option<String>,
|
encryption_key: Option<String>,
|
||||||
|
|
||||||
/// Encrypt the database (deprecated; ignored for data DBs)
|
/// Encrypt the database
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
encrypt: bool,
|
encrypt: bool,
|
||||||
|
|
||||||
/// Enable RPC management server
|
|
||||||
#[arg(long)]
|
|
||||||
enable_rpc: bool,
|
|
||||||
|
|
||||||
/// RPC server port (default: 8080)
|
|
||||||
#[arg(long, default_value = "8080")]
|
|
||||||
rpc_port: u16,
|
|
||||||
|
|
||||||
/// Use the sled backend
|
/// Use the sled backend
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
sled: bool,
|
sled: bool,
|
||||||
|
|
||||||
/// Admin secret used to encrypt DB 0 and authorize admin access (required)
|
|
||||||
#[arg(long)]
|
|
||||||
admin_secret: String,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::main]
|
#[tokio::main]
|
||||||
@@ -60,19 +47,9 @@ async fn main() {
|
|||||||
.await
|
.await
|
||||||
.unwrap();
|
.unwrap();
|
||||||
|
|
||||||
// deprecation warnings for legacy flags
|
|
||||||
if args.encrypt || args.encryption_key.is_some() {
|
|
||||||
eprintln!("warning: --encrypt and --encryption-key are deprecated and ignored for data DBs. Admin DB 0 is always encrypted with --admin-secret.");
|
|
||||||
}
|
|
||||||
// basic validation for admin secret
|
|
||||||
if args.admin_secret.trim().is_empty() {
|
|
||||||
eprintln!("error: --admin-secret must not be empty");
|
|
||||||
std::process::exit(2);
|
|
||||||
}
|
|
||||||
|
|
||||||
// new DB option
|
// new DB option
|
||||||
let option = herodb::options::DBOption {
|
let option = herodb::options::DBOption {
|
||||||
dir: args.dir.clone(),
|
dir: args.dir,
|
||||||
port,
|
port,
|
||||||
debug: args.debug,
|
debug: args.debug,
|
||||||
encryption_key: args.encryption_key,
|
encryption_key: args.encryption_key,
|
||||||
@@ -82,42 +59,14 @@ async fn main() {
|
|||||||
} else {
|
} else {
|
||||||
herodb::options::BackendType::Redb
|
herodb::options::BackendType::Redb
|
||||||
},
|
},
|
||||||
admin_secret: args.admin_secret.clone(),
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let backend = option.backend.clone();
|
|
||||||
|
|
||||||
// Bootstrap admin DB 0 before opening any server storage
|
|
||||||
if let Err(e) = herodb::admin_meta::ensure_bootstrap(&args.dir, backend.clone(), &args.admin_secret) {
|
|
||||||
eprintln!("Failed to bootstrap admin DB 0: {}", e.0);
|
|
||||||
std::process::exit(2);
|
|
||||||
}
|
|
||||||
|
|
||||||
// new server
|
// new server
|
||||||
let server = server::Server::new(option).await;
|
let server = server::Server::new(option).await;
|
||||||
|
|
||||||
// Add a small delay to ensure the port is ready
|
// Add a small delay to ensure the port is ready
|
||||||
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
|
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
|
||||||
|
|
||||||
// Start RPC server if enabled
|
|
||||||
let _rpc_handle = if args.enable_rpc {
|
|
||||||
let rpc_addr = format!("127.0.0.1:{}", args.rpc_port).parse().unwrap();
|
|
||||||
let base_dir = args.dir.clone();
|
|
||||||
|
|
||||||
match rpc_server::start_rpc_server(rpc_addr, base_dir, backend, args.admin_secret.clone()).await {
|
|
||||||
Ok(handle) => {
|
|
||||||
println!("RPC management server started on port {}", args.rpc_port);
|
|
||||||
Some(handle)
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
eprintln!("Failed to start RPC server: {}", e);
|
|
||||||
None
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
};
|
|
||||||
|
|
||||||
// accept new connections
|
// accept new connections
|
||||||
loop {
|
loop {
|
||||||
let stream = listener.accept().await;
|
let stream = listener.accept().await;
|
||||||
|
@@ -1,8 +1,7 @@
|
|||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
#[derive(Debug, Clone)]
|
||||||
pub enum BackendType {
|
pub enum BackendType {
|
||||||
Redb,
|
Redb,
|
||||||
Sled,
|
Sled,
|
||||||
Tantivy, // Full-text search backend (no KV storage)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
@@ -10,11 +9,7 @@ pub struct DBOption {
|
|||||||
pub dir: String,
|
pub dir: String,
|
||||||
pub port: u16,
|
pub port: u16,
|
||||||
pub debug: bool,
|
pub debug: bool,
|
||||||
// Deprecated for data DBs; retained for backward-compat on CLI parsing
|
|
||||||
pub encrypt: bool,
|
pub encrypt: bool,
|
||||||
// Deprecated for data DBs; retained for backward-compat on CLI parsing
|
|
||||||
pub encryption_key: Option<String>,
|
pub encryption_key: Option<String>,
|
||||||
pub backend: BackendType,
|
pub backend: BackendType,
|
||||||
// New: required admin secret, used to encrypt DB 0 and authorize admin operations
|
|
||||||
pub admin_secret: String,
|
|
||||||
}
|
}
|
||||||
|
@@ -92,7 +92,10 @@ impl Protocol {
|
|||||||
|
|
||||||
fn parse_simple_string_sfx(protocol: &str) -> Result<(Self, &str), DBError> {
|
fn parse_simple_string_sfx(protocol: &str) -> Result<(Self, &str), DBError> {
|
||||||
match protocol.find("\r\n") {
|
match protocol.find("\r\n") {
|
||||||
Some(x) => Ok((Self::SimpleString(protocol[..x].to_string()), &protocol[x + 2..])),
|
Some(x) => Ok((
|
||||||
|
Self::SimpleString(protocol[..x].to_string()),
|
||||||
|
&protocol[x + 2..],
|
||||||
|
)),
|
||||||
_ => Err(DBError(format!(
|
_ => Err(DBError(format!(
|
||||||
"[new simple string] unsupported protocol: {:?}",
|
"[new simple string] unsupported protocol: {:?}",
|
||||||
protocol
|
protocol
|
||||||
|
678
src/rpc.rs
678
src/rpc.rs
@@ -1,678 +0,0 @@
|
|||||||
use std::collections::HashMap;
|
|
||||||
use std::sync::Arc;
|
|
||||||
use tokio::sync::RwLock;
|
|
||||||
use jsonrpsee::{core::RpcResult, proc_macros::rpc};
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
use sha2::{Digest, Sha256};
|
|
||||||
|
|
||||||
use crate::server::Server;
|
|
||||||
use crate::options::DBOption;
|
|
||||||
use crate::admin_meta;
|
|
||||||
|
|
||||||
/// Database backend types
|
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
||||||
pub enum BackendType {
|
|
||||||
Redb,
|
|
||||||
Sled,
|
|
||||||
Tantivy, // Full-text search backend (no KV storage)
|
|
||||||
// Future: InMemory, Custom(String)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Database configuration
|
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
||||||
pub struct DatabaseConfig {
|
|
||||||
pub name: Option<String>,
|
|
||||||
pub storage_path: Option<String>,
|
|
||||||
pub max_size: Option<u64>,
|
|
||||||
pub redis_version: Option<String>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Database information returned by metadata queries
|
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
||||||
pub struct DatabaseInfo {
|
|
||||||
pub id: u64,
|
|
||||||
pub name: Option<String>,
|
|
||||||
pub backend: BackendType,
|
|
||||||
pub encrypted: bool,
|
|
||||||
pub redis_version: Option<String>,
|
|
||||||
pub storage_path: Option<String>,
|
|
||||||
pub size_on_disk: Option<u64>,
|
|
||||||
pub key_count: Option<u64>,
|
|
||||||
pub created_at: u64,
|
|
||||||
pub last_access: Option<u64>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Access permissions for database keys
|
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
|
||||||
pub enum Permissions {
|
|
||||||
Read,
|
|
||||||
ReadWrite,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
/// Access key information returned by RPC
|
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
||||||
pub struct AccessKeyInfo {
|
|
||||||
pub hash: String,
|
|
||||||
pub permissions: Permissions,
|
|
||||||
pub created_at: u64,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Hash a plaintext key using SHA-256
|
|
||||||
pub fn hash_key(key: &str) -> String {
|
|
||||||
let mut hasher = Sha256::new();
|
|
||||||
hasher.update(key.as_bytes());
|
|
||||||
format!("{:x}", hasher.finalize())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// RPC trait for HeroDB management
|
|
||||||
#[rpc(server, client, namespace = "herodb")]
|
|
||||||
pub trait Rpc {
|
|
||||||
/// Create a new database with specified configuration
|
|
||||||
#[method(name = "createDatabase")]
|
|
||||||
async fn create_database(
|
|
||||||
&self,
|
|
||||||
backend: BackendType,
|
|
||||||
config: DatabaseConfig,
|
|
||||||
encryption_key: Option<String>,
|
|
||||||
) -> RpcResult<u64>;
|
|
||||||
|
|
||||||
/// Set encryption for an existing database (write-only key)
|
|
||||||
#[method(name = "setEncryption")]
|
|
||||||
async fn set_encryption(&self, db_id: u64, encryption_key: String) -> RpcResult<bool>;
|
|
||||||
|
|
||||||
/// List all managed databases
|
|
||||||
#[method(name = "listDatabases")]
|
|
||||||
async fn list_databases(&self) -> RpcResult<Vec<DatabaseInfo>>;
|
|
||||||
|
|
||||||
/// Get detailed information about a specific database
|
|
||||||
#[method(name = "getDatabaseInfo")]
|
|
||||||
async fn get_database_info(&self, db_id: u64) -> RpcResult<DatabaseInfo>;
|
|
||||||
|
|
||||||
/// Delete a database
|
|
||||||
#[method(name = "deleteDatabase")]
|
|
||||||
async fn delete_database(&self, db_id: u64) -> RpcResult<bool>;
|
|
||||||
|
|
||||||
/// Get server statistics
|
|
||||||
#[method(name = "getServerStats")]
|
|
||||||
async fn get_server_stats(&self) -> RpcResult<HashMap<String, serde_json::Value>>;
|
|
||||||
|
|
||||||
/// Add an access key to a database
|
|
||||||
#[method(name = "addAccessKey")]
|
|
||||||
async fn add_access_key(&self, db_id: u64, key: String, permissions: String) -> RpcResult<bool>;
|
|
||||||
|
|
||||||
/// Delete an access key from a database
|
|
||||||
#[method(name = "deleteAccessKey")]
|
|
||||||
async fn delete_access_key(&self, db_id: u64, key_hash: String) -> RpcResult<bool>;
|
|
||||||
|
|
||||||
/// List all access keys for a database
|
|
||||||
#[method(name = "listAccessKeys")]
|
|
||||||
async fn list_access_keys(&self, db_id: u64) -> RpcResult<Vec<AccessKeyInfo>>;
|
|
||||||
|
|
||||||
/// Set database public/private status
|
|
||||||
#[method(name = "setDatabasePublic")]
|
|
||||||
async fn set_database_public(&self, db_id: u64, public: bool) -> RpcResult<bool>;
|
|
||||||
|
|
||||||
// ----- Full-text (Tantivy) minimal RPC endpoints -----
|
|
||||||
|
|
||||||
/// Create a new FT index in a Tantivy-backed DB
|
|
||||||
#[method(name = "ftCreate")]
|
|
||||||
async fn ft_create(
|
|
||||||
&self,
|
|
||||||
db_id: u64,
|
|
||||||
index_name: String,
|
|
||||||
schema: Vec<(String, String, Vec<String>)>,
|
|
||||||
) -> RpcResult<bool>;
|
|
||||||
|
|
||||||
/// Add or replace a document in an FT index
|
|
||||||
#[method(name = "ftAdd")]
|
|
||||||
async fn ft_add(
|
|
||||||
&self,
|
|
||||||
db_id: u64,
|
|
||||||
index_name: String,
|
|
||||||
doc_id: String,
|
|
||||||
score: f64,
|
|
||||||
fields: HashMap<String, String>,
|
|
||||||
) -> RpcResult<bool>;
|
|
||||||
|
|
||||||
/// Search an FT index
|
|
||||||
#[method(name = "ftSearch")]
|
|
||||||
async fn ft_search(
|
|
||||||
&self,
|
|
||||||
db_id: u64,
|
|
||||||
index_name: String,
|
|
||||||
query: String,
|
|
||||||
filters: Option<Vec<(String, String)>>,
|
|
||||||
limit: Option<usize>,
|
|
||||||
offset: Option<usize>,
|
|
||||||
return_fields: Option<Vec<String>>,
|
|
||||||
) -> RpcResult<serde_json::Value>;
|
|
||||||
|
|
||||||
/// Delete a document by id from an FT index
|
|
||||||
#[method(name = "ftDel")]
|
|
||||||
async fn ft_del(&self, db_id: u64, index_name: String, doc_id: String) -> RpcResult<bool>;
|
|
||||||
|
|
||||||
/// Get FT index info
|
|
||||||
#[method(name = "ftInfo")]
|
|
||||||
async fn ft_info(&self, db_id: u64, index_name: String) -> RpcResult<serde_json::Value>;
|
|
||||||
|
|
||||||
/// Drop an FT index
|
|
||||||
#[method(name = "ftDrop")]
|
|
||||||
async fn ft_drop(&self, db_id: u64, index_name: String) -> RpcResult<bool>;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// RPC Server implementation
|
|
||||||
pub struct RpcServerImpl {
|
|
||||||
/// Base directory for database files
|
|
||||||
base_dir: String,
|
|
||||||
/// Managed database servers
|
|
||||||
servers: Arc<RwLock<HashMap<u64, Arc<Server>>>>,
|
|
||||||
/// Default backend type
|
|
||||||
backend: crate::options::BackendType,
|
|
||||||
/// Admin secret used to encrypt DB 0 and authorize admin access
|
|
||||||
admin_secret: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl RpcServerImpl {
|
|
||||||
/// Create a new RPC server instance
|
|
||||||
pub fn new(base_dir: String, backend: crate::options::BackendType, admin_secret: String) -> Self {
|
|
||||||
Self {
|
|
||||||
base_dir,
|
|
||||||
servers: Arc::new(RwLock::new(HashMap::new())),
|
|
||||||
backend,
|
|
||||||
admin_secret,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Get or create a server instance for the given database ID
|
|
||||||
async fn get_or_create_server(&self, db_id: u64) -> Result<Arc<Server>, jsonrpsee::types::ErrorObjectOwned> {
|
|
||||||
// Check if server already exists
|
|
||||||
{
|
|
||||||
let servers = self.servers.read().await;
|
|
||||||
if let Some(server) = servers.get(&db_id) {
|
|
||||||
return Ok(server.clone());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Validate existence via admin DB 0 (metadata), not filesystem presence
|
|
||||||
let exists = admin_meta::db_exists(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id)
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
if !exists {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(
|
|
||||||
-32000,
|
|
||||||
format!("Database {} not found", db_id),
|
|
||||||
None::<()>
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
// Resolve effective backend for this db from admin meta or filesystem; fallback to default
|
|
||||||
let meta_backend = admin_meta::get_database_backend(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id)
|
|
||||||
.ok()
|
|
||||||
.flatten();
|
|
||||||
let db_path = std::path::PathBuf::from(&self.base_dir).join(format!("{}.db", db_id));
|
|
||||||
let sniffed_backend = if db_path.exists() {
|
|
||||||
if db_path.is_file() {
|
|
||||||
Some(crate::options::BackendType::Redb)
|
|
||||||
} else if db_path.is_dir() {
|
|
||||||
Some(crate::options::BackendType::Sled)
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
};
|
|
||||||
let effective_backend = meta_backend.clone().or(sniffed_backend).unwrap_or(self.backend.clone());
|
|
||||||
if effective_backend != self.backend {
|
|
||||||
eprintln!(
|
|
||||||
"notice: get_or_create_server: db {} backend resolved to {:?} (server default {:?})",
|
|
||||||
db_id, effective_backend, self.backend
|
|
||||||
);
|
|
||||||
}
|
|
||||||
// If we had to sniff (no meta), persist the resolved backend
|
|
||||||
if meta_backend.is_none() {
|
|
||||||
let _ = admin_meta::set_database_backend(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id, effective_backend.clone());
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create server instance with resolved backend
|
|
||||||
let is_tantivy = matches!(effective_backend, crate::options::BackendType::Tantivy);
|
|
||||||
let db_option = DBOption {
|
|
||||||
dir: self.base_dir.clone(),
|
|
||||||
port: 0, // Not used for RPC-managed databases
|
|
||||||
debug: false,
|
|
||||||
encryption_key: None,
|
|
||||||
encrypt: false,
|
|
||||||
backend: effective_backend.clone(),
|
|
||||||
admin_secret: self.admin_secret.clone(),
|
|
||||||
};
|
|
||||||
|
|
||||||
let mut server = Server::new(db_option).await;
|
|
||||||
|
|
||||||
// Set the selected database to the db_id
|
|
||||||
server.selected_db = db_id;
|
|
||||||
|
|
||||||
// Lazily open/create physical storage according to admin meta (per-db encryption)
|
|
||||||
// Skip for Tantivy backend (no KV storage to open)
|
|
||||||
if !is_tantivy {
|
|
||||||
let _ = server.current_storage();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Store the server
|
|
||||||
let mut servers = self.servers.write().await;
|
|
||||||
servers.insert(db_id, Arc::new(server.clone()));
|
|
||||||
|
|
||||||
Ok(Arc::new(server))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Discover existing database IDs from admin DB 0
|
|
||||||
async fn discover_databases(&self) -> Vec<u64> {
|
|
||||||
admin_meta::list_dbs(&self.base_dir, self.backend.clone(), &self.admin_secret)
|
|
||||||
.unwrap_or_default()
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
/// Build database file path for given server/db_id
|
|
||||||
fn db_file_path(&self, server: &Server, db_id: u64) -> std::path::PathBuf {
|
|
||||||
std::path::PathBuf::from(&server.option.dir).join(format!("{}.db", db_id))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Recursively compute size on disk for the database path
|
|
||||||
fn compute_size_on_disk(&self, path: &std::path::Path) -> Option<u64> {
|
|
||||||
fn dir_size(p: &std::path::Path) -> u64 {
|
|
||||||
if p.is_file() {
|
|
||||||
std::fs::metadata(p).map(|m| m.len()).unwrap_or(0)
|
|
||||||
} else if p.is_dir() {
|
|
||||||
let mut total = 0u64;
|
|
||||||
if let Ok(read) = std::fs::read_dir(p) {
|
|
||||||
for entry in read.flatten() {
|
|
||||||
total += dir_size(&entry.path());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
total
|
|
||||||
} else {
|
|
||||||
0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Some(dir_size(path))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extract created and last access times (secs) from a path, with fallbacks
|
|
||||||
fn get_file_times_secs(path: &std::path::Path) -> (u64, Option<u64>) {
|
|
||||||
let now = std::time::SystemTime::now();
|
|
||||||
let created = std::fs::metadata(path)
|
|
||||||
.and_then(|m| m.created().or_else(|_| m.modified()))
|
|
||||||
.unwrap_or(now)
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap_or_default()
|
|
||||||
.as_secs();
|
|
||||||
|
|
||||||
let last_access = std::fs::metadata(path)
|
|
||||||
.and_then(|m| m.accessed())
|
|
||||||
.ok()
|
|
||||||
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok().map(|d| d.as_secs()));
|
|
||||||
|
|
||||||
(created, last_access)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Compose a DatabaseInfo by probing storage and filesystem, with admin meta for access key count
|
|
||||||
async fn build_database_info(&self, db_id: u64, server: &Server) -> DatabaseInfo {
|
|
||||||
// Probe storage to determine encryption state
|
|
||||||
let storage = server.current_storage().ok();
|
|
||||||
let encrypted = storage.as_ref().map(|s| s.is_encrypted()).unwrap_or(server.option.encrypt);
|
|
||||||
|
|
||||||
// Get actual key count from storage
|
|
||||||
let key_count = storage.as_ref()
|
|
||||||
.and_then(|s| s.dbsize().ok())
|
|
||||||
.map(|count| count as u64);
|
|
||||||
|
|
||||||
// Get database name from admin meta
|
|
||||||
let name = admin_meta::get_database_name(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id)
|
|
||||||
.ok()
|
|
||||||
.flatten();
|
|
||||||
|
|
||||||
// Compute size on disk and timestamps from the DB file path
|
|
||||||
let db_path = self.db_file_path(server, db_id);
|
|
||||||
let size_on_disk = self.compute_size_on_disk(&db_path);
|
|
||||||
let (created_at, last_access) = Self::get_file_times_secs(&db_path);
|
|
||||||
|
|
||||||
let backend = match server.option.backend {
|
|
||||||
crate::options::BackendType::Redb => BackendType::Redb,
|
|
||||||
crate::options::BackendType::Sled => BackendType::Sled,
|
|
||||||
crate::options::BackendType::Tantivy => BackendType::Tantivy,
|
|
||||||
};
|
|
||||||
|
|
||||||
DatabaseInfo {
|
|
||||||
id: db_id,
|
|
||||||
name,
|
|
||||||
backend,
|
|
||||||
encrypted,
|
|
||||||
redis_version: Some("7.0".to_string()),
|
|
||||||
storage_path: Some(server.option.dir.clone()),
|
|
||||||
size_on_disk,
|
|
||||||
key_count,
|
|
||||||
created_at,
|
|
||||||
last_access,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[jsonrpsee::core::async_trait]
|
|
||||||
impl RpcServer for RpcServerImpl {
|
|
||||||
async fn create_database(
|
|
||||||
&self,
|
|
||||||
backend: BackendType,
|
|
||||||
config: DatabaseConfig,
|
|
||||||
encryption_key: Option<String>,
|
|
||||||
) -> RpcResult<u64> {
|
|
||||||
// Allocate new ID via admin DB 0
|
|
||||||
let db_id = admin_meta::allocate_next_id(&self.base_dir, self.backend.clone(), &self.admin_secret)
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
|
|
||||||
// Persist per-db encryption key in admin DB 0 if provided
|
|
||||||
if let Some(ref key) = encryption_key {
|
|
||||||
admin_meta::set_enc_key(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id, key)
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Persist database name if provided
|
|
||||||
if let Some(ref name) = config.name {
|
|
||||||
admin_meta::set_database_name(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id, name)
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Ensure base dir exists
|
|
||||||
if let Err(e) = std::fs::create_dir_all(&self.base_dir) {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, format!("Failed to ensure base dir: {}", e), None::<()>));
|
|
||||||
}
|
|
||||||
|
|
||||||
// Map RPC backend to options backend and persist it in admin meta for this db id
|
|
||||||
let opt_backend = match backend {
|
|
||||||
BackendType::Redb => crate::options::BackendType::Redb,
|
|
||||||
BackendType::Sled => crate::options::BackendType::Sled,
|
|
||||||
BackendType::Tantivy => crate::options::BackendType::Tantivy,
|
|
||||||
};
|
|
||||||
admin_meta::set_database_backend(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id, opt_backend.clone())
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
|
|
||||||
// Create server instance using base_dir, chosen backend and admin secret
|
|
||||||
let is_tantivy_new = matches!(opt_backend, crate::options::BackendType::Tantivy);
|
|
||||||
let option = DBOption {
|
|
||||||
dir: self.base_dir.clone(),
|
|
||||||
port: 0, // Not used for RPC-managed databases
|
|
||||||
debug: false,
|
|
||||||
encryption_key: None, // per-db key is stored in admin DB 0
|
|
||||||
encrypt: false, // encryption decided per-db at open time
|
|
||||||
backend: opt_backend.clone(),
|
|
||||||
admin_secret: self.admin_secret.clone(),
|
|
||||||
};
|
|
||||||
|
|
||||||
let mut server = Server::new(option).await;
|
|
||||||
server.selected_db = db_id;
|
|
||||||
|
|
||||||
// Initialize storage to create physical <id>.db with proper encryption from admin meta
|
|
||||||
// Skip for Tantivy backend (no KV storage to initialize)
|
|
||||||
if !is_tantivy_new {
|
|
||||||
let _ = server.current_storage();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Store the server in cache
|
|
||||||
let mut servers = self.servers.write().await;
|
|
||||||
servers.insert(db_id, Arc::new(server));
|
|
||||||
|
|
||||||
Ok(db_id)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn set_encryption(&self, _db_id: u64, _encryption_key: String) -> RpcResult<bool> {
|
|
||||||
// For now, return false as encryption can only be set during creation
|
|
||||||
let _servers = self.servers.read().await;
|
|
||||||
// TODO: Implement encryption setting for existing databases
|
|
||||||
Ok(false)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn list_databases(&self) -> RpcResult<Vec<DatabaseInfo>> {
|
|
||||||
let db_ids = self.discover_databases().await;
|
|
||||||
let mut result = Vec::new();
|
|
||||||
|
|
||||||
for db_id in db_ids {
|
|
||||||
if let Ok(server) = self.get_or_create_server(db_id).await {
|
|
||||||
// Build accurate info from storage/meta/fs
|
|
||||||
let info = self.build_database_info(db_id, &server).await;
|
|
||||||
result.push(info);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(result)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn get_database_info(&self, db_id: u64) -> RpcResult<DatabaseInfo> {
|
|
||||||
let server = self.get_or_create_server(db_id).await?;
|
|
||||||
// Build accurate info from storage/meta/fs
|
|
||||||
let info = self.build_database_info(db_id, &server).await;
|
|
||||||
Ok(info)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn delete_database(&self, db_id: u64) -> RpcResult<bool> {
|
|
||||||
let mut servers = self.servers.write().await;
|
|
||||||
|
|
||||||
if let Some(_server) = servers.remove(&db_id) {
|
|
||||||
// Clean up database files
|
|
||||||
let db_path = std::path::PathBuf::from(&self.base_dir).join(format!("{}.db", db_id));
|
|
||||||
if db_path.exists() {
|
|
||||||
if db_path.is_dir() {
|
|
||||||
std::fs::remove_dir_all(&db_path).ok();
|
|
||||||
} else {
|
|
||||||
std::fs::remove_file(&db_path).ok();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(true)
|
|
||||||
} else {
|
|
||||||
Ok(false)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn get_server_stats(&self) -> RpcResult<HashMap<String, serde_json::Value>> {
|
|
||||||
let db_ids = self.discover_databases().await;
|
|
||||||
let mut stats = HashMap::new();
|
|
||||||
|
|
||||||
|
|
||||||
stats.insert("total_databases".to_string(), serde_json::json!(db_ids.len()));
|
|
||||||
stats.insert("uptime".to_string(), serde_json::json!(
|
|
||||||
std::time::SystemTime::now()
|
|
||||||
.duration_since(std::time::UNIX_EPOCH)
|
|
||||||
.unwrap()
|
|
||||||
.as_secs()
|
|
||||||
));
|
|
||||||
|
|
||||||
Ok(stats)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ----- Full-text (Tantivy) minimal RPC endpoints -----
|
|
||||||
|
|
||||||
async fn ft_create(
|
|
||||||
&self,
|
|
||||||
db_id: u64,
|
|
||||||
index_name: String,
|
|
||||||
schema: Vec<(String, String, Vec<String>)>,
|
|
||||||
) -> RpcResult<bool> {
|
|
||||||
let server = self.get_or_create_server(db_id).await?;
|
|
||||||
if db_id == 0 {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "FT not allowed on DB 0", None::<()>));
|
|
||||||
}
|
|
||||||
if !matches!(server.option.backend, crate::options::BackendType::Tantivy) {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "DB backend is not Tantivy", None::<()>));
|
|
||||||
}
|
|
||||||
let proto = crate::search_cmd::ft_create_cmd(&*server, index_name, schema)
|
|
||||||
.await
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
match proto {
|
|
||||||
crate::protocol::Protocol::Error(msg) => {
|
|
||||||
Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, msg, None::<()>))
|
|
||||||
}
|
|
||||||
_ => Ok(true),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn ft_add(
|
|
||||||
&self,
|
|
||||||
db_id: u64,
|
|
||||||
index_name: String,
|
|
||||||
doc_id: String,
|
|
||||||
score: f64,
|
|
||||||
fields: HashMap<String, String>,
|
|
||||||
) -> RpcResult<bool> {
|
|
||||||
let server = self.get_or_create_server(db_id).await?;
|
|
||||||
if db_id == 0 {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "FT not allowed on DB 0", None::<()>));
|
|
||||||
}
|
|
||||||
if !matches!(server.option.backend, crate::options::BackendType::Tantivy) {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "DB backend is not Tantivy", None::<()>));
|
|
||||||
}
|
|
||||||
let proto = crate::search_cmd::ft_add_cmd(&*server, index_name, doc_id, score, fields)
|
|
||||||
.await
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
match proto {
|
|
||||||
crate::protocol::Protocol::Error(msg) => {
|
|
||||||
Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, msg, None::<()>))
|
|
||||||
}
|
|
||||||
_ => Ok(true),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn ft_search(
|
|
||||||
&self,
|
|
||||||
db_id: u64,
|
|
||||||
index_name: String,
|
|
||||||
query: String,
|
|
||||||
filters: Option<Vec<(String, String)>>,
|
|
||||||
limit: Option<usize>,
|
|
||||||
offset: Option<usize>,
|
|
||||||
return_fields: Option<Vec<String>>,
|
|
||||||
) -> RpcResult<serde_json::Value> {
|
|
||||||
let server = self.get_or_create_server(db_id).await?;
|
|
||||||
if db_id == 0 {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "FT not allowed on DB 0", None::<()>));
|
|
||||||
}
|
|
||||||
if !matches!(server.option.backend, crate::options::BackendType::Tantivy) {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "DB backend is not Tantivy", None::<()>));
|
|
||||||
}
|
|
||||||
let proto = crate::search_cmd::ft_search_cmd(
|
|
||||||
&*server,
|
|
||||||
index_name,
|
|
||||||
query,
|
|
||||||
filters.unwrap_or_default(),
|
|
||||||
limit,
|
|
||||||
offset,
|
|
||||||
return_fields,
|
|
||||||
)
|
|
||||||
.await
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
match proto {
|
|
||||||
crate::protocol::Protocol::Error(msg) => {
|
|
||||||
Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, msg, None::<()>))
|
|
||||||
}
|
|
||||||
_ => Ok(serde_json::json!({ "resp": proto.encode() })),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn ft_del(&self, db_id: u64, index_name: String, doc_id: String) -> RpcResult<bool> {
|
|
||||||
let server = self.get_or_create_server(db_id).await?;
|
|
||||||
if db_id == 0 {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "FT not allowed on DB 0", None::<()>));
|
|
||||||
}
|
|
||||||
if !matches!(server.option.backend, crate::options::BackendType::Tantivy) {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "DB backend is not Tantivy", None::<()>));
|
|
||||||
}
|
|
||||||
let proto = crate::search_cmd::ft_del_cmd(&*server, index_name, doc_id)
|
|
||||||
.await
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
match proto {
|
|
||||||
crate::protocol::Protocol::Error(msg) => {
|
|
||||||
Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, msg, None::<()>))
|
|
||||||
}
|
|
||||||
crate::protocol::Protocol::SimpleString(s) => Ok(s == "1"),
|
|
||||||
_ => Ok(false),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn ft_info(&self, db_id: u64, index_name: String) -> RpcResult<serde_json::Value> {
|
|
||||||
let server = self.get_or_create_server(db_id).await?;
|
|
||||||
if db_id == 0 {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "FT not allowed on DB 0", None::<()>));
|
|
||||||
}
|
|
||||||
if !matches!(server.option.backend, crate::options::BackendType::Tantivy) {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "DB backend is not Tantivy", None::<()>));
|
|
||||||
}
|
|
||||||
let proto = crate::search_cmd::ft_info_cmd(&*server, index_name)
|
|
||||||
.await
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
match proto {
|
|
||||||
crate::protocol::Protocol::Error(msg) => {
|
|
||||||
Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, msg, None::<()>))
|
|
||||||
}
|
|
||||||
_ => Ok(serde_json::json!({ "resp": proto.encode() })),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn ft_drop(&self, db_id: u64, index_name: String) -> RpcResult<bool> {
|
|
||||||
let server = self.get_or_create_server(db_id).await?;
|
|
||||||
if db_id == 0 {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "FT not allowed on DB 0", None::<()>));
|
|
||||||
}
|
|
||||||
if !matches!(server.option.backend, crate::options::BackendType::Tantivy) {
|
|
||||||
return Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, "DB backend is not Tantivy", None::<()>));
|
|
||||||
}
|
|
||||||
let proto = crate::search_cmd::ft_drop_cmd(&*server, index_name)
|
|
||||||
.await
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
match proto {
|
|
||||||
crate::protocol::Protocol::Error(msg) => {
|
|
||||||
Err(jsonrpsee::types::ErrorObjectOwned::owned(-32000, msg, None::<()>))
|
|
||||||
}
|
|
||||||
crate::protocol::Protocol::SimpleString(s) => Ok(s.eq_ignore_ascii_case("OK")),
|
|
||||||
_ => Ok(false),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn add_access_key(&self, db_id: u64, key: String, permissions: String) -> RpcResult<bool> {
|
|
||||||
let perms = match permissions.to_lowercase().as_str() {
|
|
||||||
"read" => Permissions::Read,
|
|
||||||
"readwrite" => Permissions::ReadWrite,
|
|
||||||
_ => return Err(jsonrpsee::types::ErrorObjectOwned::owned(
|
|
||||||
-32000,
|
|
||||||
"Invalid permissions: use 'read' or 'readwrite'",
|
|
||||||
None::<()>
|
|
||||||
)),
|
|
||||||
};
|
|
||||||
|
|
||||||
admin_meta::add_access_key(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id, &key, perms)
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
Ok(true)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn delete_access_key(&self, db_id: u64, key_hash: String) -> RpcResult<bool> {
|
|
||||||
let ok = admin_meta::delete_access_key(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id, &key_hash)
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
Ok(ok)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn list_access_keys(&self, db_id: u64) -> RpcResult<Vec<AccessKeyInfo>> {
|
|
||||||
let pairs = admin_meta::list_access_keys(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id)
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
let keys: Vec<AccessKeyInfo> = pairs.into_iter().map(|(hash, perm, ts)| AccessKeyInfo {
|
|
||||||
hash,
|
|
||||||
permissions: perm,
|
|
||||||
created_at: ts,
|
|
||||||
}).collect();
|
|
||||||
Ok(keys)
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn set_database_public(&self, db_id: u64, public: bool) -> RpcResult<bool> {
|
|
||||||
admin_meta::set_database_public(&self.base_dir, self.backend.clone(), &self.admin_secret, db_id, public)
|
|
||||||
.map_err(|e| jsonrpsee::types::ErrorObjectOwned::owned(-32000, e.0, None::<()>))?;
|
|
||||||
Ok(true)
|
|
||||||
}
|
|
||||||
}
|
|
@@ -1,49 +0,0 @@
|
|||||||
use std::net::SocketAddr;
|
|
||||||
use jsonrpsee::server::{ServerBuilder, ServerHandle};
|
|
||||||
use jsonrpsee::RpcModule;
|
|
||||||
|
|
||||||
use crate::rpc::{RpcServer, RpcServerImpl};
|
|
||||||
|
|
||||||
/// Start the RPC server on the specified address
|
|
||||||
pub async fn start_rpc_server(addr: SocketAddr, base_dir: String, backend: crate::options::BackendType, admin_secret: String) -> Result<ServerHandle, Box<dyn std::error::Error + Send + Sync>> {
|
|
||||||
// Create the RPC server implementation
|
|
||||||
let rpc_impl = RpcServerImpl::new(base_dir, backend, admin_secret);
|
|
||||||
|
|
||||||
// Create the RPC module
|
|
||||||
let mut module = RpcModule::new(());
|
|
||||||
module.merge(RpcServer::into_rpc(rpc_impl))?;
|
|
||||||
|
|
||||||
// Build the server with both HTTP and WebSocket support
|
|
||||||
let server = ServerBuilder::default()
|
|
||||||
.build(addr)
|
|
||||||
.await?;
|
|
||||||
|
|
||||||
// Start the server
|
|
||||||
let handle = server.start(module);
|
|
||||||
|
|
||||||
println!("RPC server started on {}", addr);
|
|
||||||
|
|
||||||
Ok(handle)
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
use std::time::Duration;
|
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn test_rpc_server_startup() {
|
|
||||||
let addr = "127.0.0.1:0".parse().unwrap(); // Use port 0 for auto-assignment
|
|
||||||
let base_dir = "/tmp/test_rpc".to_string();
|
|
||||||
let backend = crate::options::BackendType::Redb; // Default for test
|
|
||||||
|
|
||||||
let handle = start_rpc_server(addr, base_dir, backend, "test-admin".to_string()).await.unwrap();
|
|
||||||
|
|
||||||
// Give the server a moment to start
|
|
||||||
tokio::time::sleep(Duration::from_millis(100)).await;
|
|
||||||
|
|
||||||
// Stop the server
|
|
||||||
handle.stop().unwrap();
|
|
||||||
handle.stopped().await;
|
|
||||||
}
|
|
||||||
}
|
|
@@ -14,57 +14,32 @@ pub async fn ft_create_cmd(
|
|||||||
index_name: String,
|
index_name: String,
|
||||||
schema: Vec<(String, String, Vec<String>)>,
|
schema: Vec<(String, String, Vec<String>)>,
|
||||||
) -> Result<Protocol, DBError> {
|
) -> Result<Protocol, DBError> {
|
||||||
if server.selected_db == 0 {
|
|
||||||
return Ok(Protocol::err("FT commands are not allowed on DB 0"));
|
|
||||||
}
|
|
||||||
// Enforce Tantivy backend for selected DB
|
|
||||||
let is_tantivy = crate::admin_meta::get_database_backend(
|
|
||||||
&server.option.dir,
|
|
||||||
server.option.backend.clone(),
|
|
||||||
&server.option.admin_secret,
|
|
||||||
server.selected_db,
|
|
||||||
)
|
|
||||||
.ok()
|
|
||||||
.flatten()
|
|
||||||
.map(|b| matches!(b, crate::options::BackendType::Tantivy))
|
|
||||||
.unwrap_or(false);
|
|
||||||
if !is_tantivy {
|
|
||||||
return Ok(Protocol::err("ERR DB backend is not Tantivy; FT.* commands are not allowed"));
|
|
||||||
}
|
|
||||||
|
|
||||||
if !server.has_write_permission() {
|
|
||||||
return Ok(Protocol::err("ERR write permission denied"));
|
|
||||||
}
|
|
||||||
|
|
||||||
// Parse schema into field definitions
|
// Parse schema into field definitions
|
||||||
let mut field_definitions = Vec::new();
|
let mut field_definitions = Vec::new();
|
||||||
|
|
||||||
for (field_name, field_type, options) in schema {
|
for (field_name, field_type, options) in schema {
|
||||||
let field_def = match field_type.to_uppercase().as_str() {
|
let field_def = match field_type.to_uppercase().as_str() {
|
||||||
"TEXT" => {
|
"TEXT" => {
|
||||||
|
let mut weight = 1.0;
|
||||||
let mut sortable = false;
|
let mut sortable = false;
|
||||||
let mut no_index = false;
|
let mut no_index = false;
|
||||||
// Weight is not used in current implementation
|
|
||||||
let mut _weight = 1.0f32;
|
for opt in &options {
|
||||||
let mut i = 0;
|
match opt.to_uppercase().as_str() {
|
||||||
while i < options.len() {
|
|
||||||
match options[i].to_uppercase().as_str() {
|
|
||||||
"WEIGHT" => {
|
"WEIGHT" => {
|
||||||
if i + 1 < options.len() {
|
// Next option should be the weight value
|
||||||
_weight = options[i + 1].parse::<f32>().unwrap_or(1.0);
|
if let Some(idx) = options.iter().position(|x| x == opt) {
|
||||||
i += 2;
|
if idx + 1 < options.len() {
|
||||||
continue;
|
weight = options[idx + 1].parse().unwrap_or(1.0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
"SORTABLE" => {
|
|
||||||
sortable = true;
|
|
||||||
}
|
|
||||||
"NOINDEX" => {
|
|
||||||
no_index = true;
|
|
||||||
}
|
}
|
||||||
|
"SORTABLE" => sortable = true,
|
||||||
|
"NOINDEX" => no_index = true,
|
||||||
_ => {}
|
_ => {}
|
||||||
}
|
}
|
||||||
i += 1;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
FieldDef::Text {
|
FieldDef::Text {
|
||||||
stored: true,
|
stored: true,
|
||||||
indexed: !no_index,
|
indexed: !no_index,
|
||||||
@@ -73,13 +48,14 @@ pub async fn ft_create_cmd(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
"NUMERIC" => {
|
"NUMERIC" => {
|
||||||
// default to F64
|
|
||||||
let mut sortable = false;
|
let mut sortable = false;
|
||||||
|
|
||||||
for opt in &options {
|
for opt in &options {
|
||||||
if opt.to_uppercase() == "SORTABLE" {
|
if opt.to_uppercase() == "SORTABLE" {
|
||||||
sortable = true;
|
sortable = true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
FieldDef::Numeric {
|
FieldDef::Numeric {
|
||||||
stored: true,
|
stored: true,
|
||||||
indexed: true,
|
indexed: true,
|
||||||
@@ -90,23 +66,19 @@ pub async fn ft_create_cmd(
|
|||||||
"TAG" => {
|
"TAG" => {
|
||||||
let mut separator = ",".to_string();
|
let mut separator = ",".to_string();
|
||||||
let mut case_sensitive = false;
|
let mut case_sensitive = false;
|
||||||
let mut i = 0;
|
|
||||||
while i < options.len() {
|
for i in 0..options.len() {
|
||||||
match options[i].to_uppercase().as_str() {
|
match options[i].to_uppercase().as_str() {
|
||||||
"SEPARATOR" => {
|
"SEPARATOR" => {
|
||||||
if i + 1 < options.len() {
|
if i + 1 < options.len() {
|
||||||
separator = options[i + 1].clone();
|
separator = options[i + 1].clone();
|
||||||
i += 2;
|
|
||||||
continue;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
"CASESENSITIVE" => {
|
"CASESENSITIVE" => case_sensitive = true,
|
||||||
case_sensitive = true;
|
|
||||||
}
|
|
||||||
_ => {}
|
_ => {}
|
||||||
}
|
}
|
||||||
i += 1;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
FieldDef::Tag {
|
FieldDef::Tag {
|
||||||
stored: true,
|
stored: true,
|
||||||
separator,
|
separator,
|
||||||
@@ -118,12 +90,20 @@ pub async fn ft_create_cmd(
|
|||||||
return Err(DBError(format!("Unknown field type: {}", field_type)));
|
return Err(DBError(format!("Unknown field type: {}", field_type)));
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
field_definitions.push((field_name, field_def));
|
field_definitions.push((field_name, field_def));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create the search index
|
// Create the search index
|
||||||
let search_path = server.search_index_path();
|
let search_path = server.search_index_path();
|
||||||
let config = IndexConfig::default();
|
let config = IndexConfig::default();
|
||||||
|
|
||||||
|
println!(
|
||||||
|
"Creating search index '{}' at path: {:?}",
|
||||||
|
index_name, search_path
|
||||||
|
);
|
||||||
|
println!("Field definitions: {:?}", field_definitions);
|
||||||
|
|
||||||
let search_index = TantivySearch::new_with_schema(
|
let search_index = TantivySearch::new_with_schema(
|
||||||
search_path,
|
search_path,
|
||||||
index_name.clone(),
|
index_name.clone(),
|
||||||
@@ -131,6 +111,8 @@ pub async fn ft_create_cmd(
|
|||||||
Some(config),
|
Some(config),
|
||||||
)?;
|
)?;
|
||||||
|
|
||||||
|
println!("Search index '{}' created successfully", index_name);
|
||||||
|
|
||||||
// Store in registry
|
// Store in registry
|
||||||
let mut indexes = server.search_indexes.write().unwrap();
|
let mut indexes = server.search_indexes.write().unwrap();
|
||||||
indexes.insert(index_name, Arc::new(search_index));
|
indexes.insert(index_name, Arc::new(search_index));
|
||||||
@@ -145,31 +127,14 @@ pub async fn ft_add_cmd(
|
|||||||
_score: f64,
|
_score: f64,
|
||||||
fields: HashMap<String, String>,
|
fields: HashMap<String, String>,
|
||||||
) -> Result<Protocol, DBError> {
|
) -> Result<Protocol, DBError> {
|
||||||
if server.selected_db == 0 {
|
|
||||||
return Ok(Protocol::err("FT commands are not allowed on DB 0"));
|
|
||||||
}
|
|
||||||
// Enforce Tantivy backend for selected DB
|
|
||||||
let is_tantivy = crate::admin_meta::get_database_backend(
|
|
||||||
&server.option.dir,
|
|
||||||
server.option.backend.clone(),
|
|
||||||
&server.option.admin_secret,
|
|
||||||
server.selected_db,
|
|
||||||
)
|
|
||||||
.ok()
|
|
||||||
.flatten()
|
|
||||||
.map(|b| matches!(b, crate::options::BackendType::Tantivy))
|
|
||||||
.unwrap_or(false);
|
|
||||||
if !is_tantivy {
|
|
||||||
return Ok(Protocol::err("ERR DB backend is not Tantivy; FT.* commands are not allowed"));
|
|
||||||
}
|
|
||||||
if !server.has_write_permission() {
|
|
||||||
return Ok(Protocol::err("ERR write permission denied"));
|
|
||||||
}
|
|
||||||
let indexes = server.search_indexes.read().unwrap();
|
let indexes = server.search_indexes.read().unwrap();
|
||||||
|
|
||||||
let search_index = indexes
|
let search_index = indexes
|
||||||
.get(&index_name)
|
.get(&index_name)
|
||||||
.ok_or_else(|| DBError(format!("Index '{}' not found", index_name)))?;
|
.ok_or_else(|| DBError(format!("Index '{}' not found", index_name)))?;
|
||||||
|
|
||||||
search_index.add_document_with_fields(&doc_id, fields)?;
|
search_index.add_document_with_fields(&doc_id, fields)?;
|
||||||
|
|
||||||
Ok(Protocol::SimpleString("OK".to_string()))
|
Ok(Protocol::SimpleString("OK".to_string()))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -182,31 +147,13 @@ pub async fn ft_search_cmd(
|
|||||||
offset: Option<usize>,
|
offset: Option<usize>,
|
||||||
return_fields: Option<Vec<String>>,
|
return_fields: Option<Vec<String>>,
|
||||||
) -> Result<Protocol, DBError> {
|
) -> Result<Protocol, DBError> {
|
||||||
if server.selected_db == 0 {
|
|
||||||
return Ok(Protocol::err("FT commands are not allowed on DB 0"));
|
|
||||||
}
|
|
||||||
// Enforce Tantivy backend for selected DB
|
|
||||||
let is_tantivy = crate::admin_meta::get_database_backend(
|
|
||||||
&server.option.dir,
|
|
||||||
server.option.backend.clone(),
|
|
||||||
&server.option.admin_secret,
|
|
||||||
server.selected_db,
|
|
||||||
)
|
|
||||||
.ok()
|
|
||||||
.flatten()
|
|
||||||
.map(|b| matches!(b, crate::options::BackendType::Tantivy))
|
|
||||||
.unwrap_or(false);
|
|
||||||
if !is_tantivy {
|
|
||||||
return Ok(Protocol::err("ERR DB backend is not Tantivy; FT.* commands are not allowed"));
|
|
||||||
}
|
|
||||||
if !server.has_read_permission() {
|
|
||||||
return Ok(Protocol::err("ERR read permission denied"));
|
|
||||||
}
|
|
||||||
let indexes = server.search_indexes.read().unwrap();
|
let indexes = server.search_indexes.read().unwrap();
|
||||||
|
|
||||||
let search_index = indexes
|
let search_index = indexes
|
||||||
.get(&index_name)
|
.get(&index_name)
|
||||||
.ok_or_else(|| DBError(format!("Index '{}' not found", index_name)))?;
|
.ok_or_else(|| DBError(format!("Index '{}' not found", index_name)))?;
|
||||||
|
|
||||||
|
// Convert filters to search filters
|
||||||
let search_filters = filters
|
let search_filters = filters
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.map(|(field, value)| Filter {
|
.map(|(field, value)| Filter {
|
||||||
@@ -226,26 +173,33 @@ pub async fn ft_search_cmd(
|
|||||||
|
|
||||||
let results = search_index.search_with_options(&query, options)?;
|
let results = search_index.search_with_options(&query, options)?;
|
||||||
|
|
||||||
// Format results as a flattened Redis protocol array to match client expectations:
|
// Format results as Redis protocol
|
||||||
// [ total, doc_id, score, field, value, field, value, ... , doc_id, score, ... ]
|
|
||||||
let mut response = Vec::new();
|
let mut response = Vec::new();
|
||||||
|
|
||||||
// First element is the total count
|
// First element is the total count
|
||||||
response.push(Protocol::BulkString(results.total.to_string()));
|
response.push(Protocol::SimpleString(results.total.to_string()));
|
||||||
// Then each document flattened
|
|
||||||
for mut doc in results.documents {
|
// Then each document
|
||||||
|
for doc in results.documents {
|
||||||
|
let mut doc_array = Vec::new();
|
||||||
|
|
||||||
// Add document ID if it exists
|
// Add document ID if it exists
|
||||||
if let Some(id) = doc.fields.get("_id") {
|
if let Some(id) = doc.fields.get("_id") {
|
||||||
response.push(Protocol::BulkString(id.clone()));
|
doc_array.push(Protocol::BulkString(id.clone()));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Add score
|
// Add score
|
||||||
response.push(Protocol::BulkString(doc.score.to_string()));
|
doc_array.push(Protocol::BulkString(doc.score.to_string()));
|
||||||
|
|
||||||
// Add fields as key-value pairs
|
// Add fields as key-value pairs
|
||||||
for (field_name, field_value) in std::mem::take(&mut doc.fields) {
|
for (field_name, field_value) in doc.fields {
|
||||||
if field_name != "_id" {
|
if field_name != "_id" {
|
||||||
response.push(Protocol::BulkString(field_name));
|
doc_array.push(Protocol::BulkString(field_name));
|
||||||
response.push(Protocol::BulkString(field_value));
|
doc_array.push(Protocol::BulkString(field_value));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
response.push(Protocol::Array(doc_array));
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(Protocol::Array(response))
|
Ok(Protocol::Array(response))
|
||||||
@@ -256,69 +210,40 @@ pub async fn ft_del_cmd(
|
|||||||
index_name: String,
|
index_name: String,
|
||||||
doc_id: String,
|
doc_id: String,
|
||||||
) -> Result<Protocol, DBError> {
|
) -> Result<Protocol, DBError> {
|
||||||
if server.selected_db == 0 {
|
|
||||||
return Ok(Protocol::err("FT commands are not allowed on DB 0"));
|
|
||||||
}
|
|
||||||
// Enforce Tantivy backend for selected DB
|
|
||||||
let is_tantivy = crate::admin_meta::get_database_backend(
|
|
||||||
&server.option.dir,
|
|
||||||
server.option.backend.clone(),
|
|
||||||
&server.option.admin_secret,
|
|
||||||
server.selected_db,
|
|
||||||
)
|
|
||||||
.ok()
|
|
||||||
.flatten()
|
|
||||||
.map(|b| matches!(b, crate::options::BackendType::Tantivy))
|
|
||||||
.unwrap_or(false);
|
|
||||||
if !is_tantivy {
|
|
||||||
return Ok(Protocol::err("ERR DB backend is not Tantivy; FT.* commands are not allowed"));
|
|
||||||
}
|
|
||||||
if !server.has_write_permission() {
|
|
||||||
return Ok(Protocol::err("ERR write permission denied"));
|
|
||||||
}
|
|
||||||
let indexes = server.search_indexes.read().unwrap();
|
let indexes = server.search_indexes.read().unwrap();
|
||||||
let search_index = indexes
|
|
||||||
|
let _search_index = indexes
|
||||||
.get(&index_name)
|
.get(&index_name)
|
||||||
.ok_or_else(|| DBError(format!("Index '{}' not found", index_name)))?;
|
.ok_or_else(|| DBError(format!("Index '{}' not found", index_name)))?;
|
||||||
let existed = search_index.delete_document_by_id(&doc_id)?;
|
|
||||||
Ok(Protocol::SimpleString(if existed { "1".to_string() } else { "0".to_string() }))
|
// For now, return success
|
||||||
|
// In a full implementation, we'd need to add a delete method to TantivySearch
|
||||||
|
println!("Deleting document '{}' from index '{}'", doc_id, index_name);
|
||||||
|
|
||||||
|
Ok(Protocol::SimpleString("1".to_string()))
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn ft_info_cmd(server: &Server, index_name: String) -> Result<Protocol, DBError> {
|
pub async fn ft_info_cmd(server: &Server, index_name: String) -> Result<Protocol, DBError> {
|
||||||
if server.selected_db == 0 {
|
|
||||||
return Ok(Protocol::err("FT commands are not allowed on DB 0"));
|
|
||||||
}
|
|
||||||
// Enforce Tantivy backend for selected DB
|
|
||||||
let is_tantivy = crate::admin_meta::get_database_backend(
|
|
||||||
&server.option.dir,
|
|
||||||
server.option.backend.clone(),
|
|
||||||
&server.option.admin_secret,
|
|
||||||
server.selected_db,
|
|
||||||
)
|
|
||||||
.ok()
|
|
||||||
.flatten()
|
|
||||||
.map(|b| matches!(b, crate::options::BackendType::Tantivy))
|
|
||||||
.unwrap_or(false);
|
|
||||||
if !is_tantivy {
|
|
||||||
return Ok(Protocol::err("ERR DB backend is not Tantivy; FT.* commands are not allowed"));
|
|
||||||
}
|
|
||||||
if !server.has_read_permission() {
|
|
||||||
return Ok(Protocol::err("ERR read permission denied"));
|
|
||||||
}
|
|
||||||
let indexes = server.search_indexes.read().unwrap();
|
let indexes = server.search_indexes.read().unwrap();
|
||||||
|
|
||||||
let search_index = indexes
|
let search_index = indexes
|
||||||
.get(&index_name)
|
.get(&index_name)
|
||||||
.ok_or_else(|| DBError(format!("Index '{}' not found", index_name)))?;
|
.ok_or_else(|| DBError(format!("Index '{}' not found", index_name)))?;
|
||||||
|
|
||||||
let info = search_index.get_info()?;
|
let info = search_index.get_info()?;
|
||||||
|
|
||||||
// Format info as Redis protocol
|
// Format info as Redis protocol
|
||||||
let mut response = Vec::new();
|
let mut response = Vec::new();
|
||||||
|
|
||||||
response.push(Protocol::BulkString("index_name".to_string()));
|
response.push(Protocol::BulkString("index_name".to_string()));
|
||||||
response.push(Protocol::BulkString(info.name));
|
response.push(Protocol::BulkString(info.name));
|
||||||
|
|
||||||
response.push(Protocol::BulkString("num_docs".to_string()));
|
response.push(Protocol::BulkString("num_docs".to_string()));
|
||||||
response.push(Protocol::BulkString(info.num_docs.to_string()));
|
response.push(Protocol::BulkString(info.num_docs.to_string()));
|
||||||
|
|
||||||
response.push(Protocol::BulkString("num_fields".to_string()));
|
response.push(Protocol::BulkString("num_fields".to_string()));
|
||||||
response.push(Protocol::BulkString(info.fields.len().to_string()));
|
response.push(Protocol::BulkString(info.fields.len().to_string()));
|
||||||
|
|
||||||
response.push(Protocol::BulkString("fields".to_string()));
|
response.push(Protocol::BulkString("fields".to_string()));
|
||||||
let fields_str = info
|
let fields_str = info
|
||||||
.fields
|
.fields
|
||||||
@@ -327,52 +252,22 @@ pub async fn ft_info_cmd(server: &Server, index_name: String) -> Result<Protocol
|
|||||||
.collect::<Vec<_>>()
|
.collect::<Vec<_>>()
|
||||||
.join(", ");
|
.join(", ");
|
||||||
response.push(Protocol::BulkString(fields_str));
|
response.push(Protocol::BulkString(fields_str));
|
||||||
|
|
||||||
Ok(Protocol::Array(response))
|
Ok(Protocol::Array(response))
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn ft_drop_cmd(server: &Server, index_name: String) -> Result<Protocol, DBError> {
|
pub async fn ft_drop_cmd(server: &Server, index_name: String) -> Result<Protocol, DBError> {
|
||||||
if server.selected_db == 0 {
|
|
||||||
return Ok(Protocol::err("FT commands are not allowed on DB 0"));
|
|
||||||
}
|
|
||||||
// Enforce Tantivy backend for selected DB
|
|
||||||
let is_tantivy = crate::admin_meta::get_database_backend(
|
|
||||||
&server.option.dir,
|
|
||||||
server.option.backend.clone(),
|
|
||||||
&server.option.admin_secret,
|
|
||||||
server.selected_db,
|
|
||||||
)
|
|
||||||
.ok()
|
|
||||||
.flatten()
|
|
||||||
.map(|b| matches!(b, crate::options::BackendType::Tantivy))
|
|
||||||
.unwrap_or(false);
|
|
||||||
if !is_tantivy {
|
|
||||||
return Ok(Protocol::err("ERR DB backend is not Tantivy; FT.* commands are not allowed"));
|
|
||||||
}
|
|
||||||
|
|
||||||
if !server.has_write_permission() {
|
|
||||||
return Ok(Protocol::err("ERR write permission denied"));
|
|
||||||
}
|
|
||||||
|
|
||||||
// Remove from registry and files; report error if nothing to drop
|
|
||||||
let mut existed = false;
|
|
||||||
{
|
|
||||||
let mut indexes = server.search_indexes.write().unwrap();
|
let mut indexes = server.search_indexes.write().unwrap();
|
||||||
if indexes.remove(&index_name).is_some() {
|
|
||||||
existed = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Remove the index files from disk
|
if indexes.remove(&index_name).is_some() {
|
||||||
|
// Also remove the index files from disk
|
||||||
let index_path = server.search_index_path().join(&index_name);
|
let index_path = server.search_index_path().join(&index_name);
|
||||||
if index_path.exists() {
|
if index_path.exists() {
|
||||||
std::fs::remove_dir_all(&index_path)
|
std::fs::remove_dir_all(index_path)
|
||||||
.map_err(|e| DBError(format!("Failed to remove index files: {}", e)))?;
|
.map_err(|e| DBError(format!("Failed to remove index files: {}", e)))?;
|
||||||
existed = true;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if !existed {
|
|
||||||
return Ok(Protocol::err(&format!("ERR Index '{}' not found", index_name)));
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(Protocol::SimpleString("OK".to_string()))
|
Ok(Protocol::SimpleString("OK".to_string()))
|
||||||
|
} else {
|
||||||
|
Err(DBError(format!("Index '{}' not found", index_name)))
|
||||||
|
}
|
||||||
}
|
}
|
133
src/server.rs
133
src/server.rs
@@ -1,9 +1,10 @@
|
|||||||
use core::str;
|
use core::str;
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
use std::sync::RwLock;
|
||||||
use tokio::io::AsyncReadExt;
|
use tokio::io::AsyncReadExt;
|
||||||
use tokio::io::AsyncWriteExt;
|
use tokio::io::AsyncWriteExt;
|
||||||
use tokio::sync::{Mutex, oneshot};
|
use tokio::sync::{oneshot, Mutex};
|
||||||
|
|
||||||
use std::sync::atomic::{AtomicU64, Ordering};
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
|
|
||||||
@@ -11,20 +12,19 @@ use crate::cmd::Cmd;
|
|||||||
use crate::error::DBError;
|
use crate::error::DBError;
|
||||||
use crate::options;
|
use crate::options;
|
||||||
use crate::protocol::Protocol;
|
use crate::protocol::Protocol;
|
||||||
|
use crate::storage::Storage;
|
||||||
|
use crate::storage_sled::SledStorage;
|
||||||
use crate::storage_trait::StorageBackend;
|
use crate::storage_trait::StorageBackend;
|
||||||
use crate::admin_meta;
|
use crate::tantivy_search::TantivySearch;
|
||||||
|
|
||||||
#[derive(Clone)]
|
#[derive(Clone)]
|
||||||
pub struct Server {
|
pub struct Server {
|
||||||
pub db_cache: std::sync::Arc<std::sync::RwLock<HashMap<u64, Arc<dyn StorageBackend>>>>,
|
pub db_cache: Arc<RwLock<HashMap<u64, Arc<dyn StorageBackend>>>>,
|
||||||
|
pub search_indexes: Arc<RwLock<HashMap<String, Arc<TantivySearch>>>>,
|
||||||
pub option: options::DBOption,
|
pub option: options::DBOption,
|
||||||
pub client_name: Option<String>,
|
pub client_name: Option<String>,
|
||||||
pub selected_db: u64, // Changed from usize to u64
|
pub selected_db: u64, // Changed from usize to u64
|
||||||
pub queued_cmd: Option<Vec<(Cmd, Protocol)>>,
|
pub queued_cmd: Option<Vec<(Cmd, Protocol)>>,
|
||||||
pub current_permissions: Option<crate::rpc::Permissions>,
|
|
||||||
|
|
||||||
// In-memory registry of Tantivy search indexes for this server
|
|
||||||
pub search_indexes: Arc<std::sync::RwLock<HashMap<String, Arc<crate::tantivy_search::TantivySearch>>>>,
|
|
||||||
|
|
||||||
// BLPOP waiter registry: per (db_index, key) FIFO of waiters
|
// BLPOP waiter registry: per (db_index, key) FIFO of waiters
|
||||||
pub list_waiters: Arc<Mutex<HashMap<u64, HashMap<String, Vec<Waiter>>>>>,
|
pub list_waiters: Arc<Mutex<HashMap<u64, HashMap<String, Vec<Waiter>>>>>,
|
||||||
@@ -46,31 +46,18 @@ pub enum PopSide {
|
|||||||
impl Server {
|
impl Server {
|
||||||
pub async fn new(option: options::DBOption) -> Self {
|
pub async fn new(option: options::DBOption) -> Self {
|
||||||
Server {
|
Server {
|
||||||
db_cache: Arc::new(std::sync::RwLock::new(HashMap::new())),
|
db_cache: Arc::new(RwLock::new(HashMap::new())),
|
||||||
|
search_indexes: Arc::new(RwLock::new(HashMap::new())),
|
||||||
option,
|
option,
|
||||||
client_name: None,
|
client_name: None,
|
||||||
selected_db: 0,
|
selected_db: 0,
|
||||||
queued_cmd: None,
|
queued_cmd: None,
|
||||||
current_permissions: None,
|
|
||||||
|
|
||||||
search_indexes: Arc::new(std::sync::RwLock::new(HashMap::new())),
|
|
||||||
list_waiters: Arc::new(Mutex::new(HashMap::new())),
|
list_waiters: Arc::new(Mutex::new(HashMap::new())),
|
||||||
waiter_seq: Arc::new(AtomicU64::new(1)),
|
waiter_seq: Arc::new(AtomicU64::new(1)),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Path where search indexes are stored, namespaced per selected DB:
|
|
||||||
// <base_dir>/search_indexes/<db_id>
|
|
||||||
pub fn search_index_path(&self) -> std::path::PathBuf {
|
|
||||||
let base = std::path::PathBuf::from(&self.option.dir)
|
|
||||||
.join("search_indexes")
|
|
||||||
.join(self.selected_db.to_string());
|
|
||||||
if !base.exists() {
|
|
||||||
let _ = std::fs::create_dir_all(&base);
|
|
||||||
}
|
|
||||||
base
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn current_storage(&self) -> Result<Arc<dyn StorageBackend>, DBError> {
|
pub fn current_storage(&self) -> Result<Arc<dyn StorageBackend>, DBError> {
|
||||||
let mut cache = self.db_cache.write().unwrap();
|
let mut cache = self.db_cache.write().unwrap();
|
||||||
|
|
||||||
@@ -78,70 +65,58 @@ impl Server {
|
|||||||
return Ok(storage.clone());
|
return Ok(storage.clone());
|
||||||
}
|
}
|
||||||
|
|
||||||
// Use process-wide shared handles to avoid sled/reDB double-open lock contention.
|
// Create new database file
|
||||||
let storage = if self.selected_db == 0 {
|
let db_file_path = std::path::PathBuf::from(self.option.dir.clone())
|
||||||
// Admin DB 0: always via singleton
|
.join(format!("{}.db", self.selected_db));
|
||||||
admin_meta::open_admin_storage(
|
|
||||||
&self.option.dir,
|
// Ensure the directory exists before creating the database file
|
||||||
self.option.backend.clone(),
|
if let Some(parent_dir) = db_file_path.parent() {
|
||||||
&self.option.admin_secret,
|
std::fs::create_dir_all(parent_dir).map_err(|e| {
|
||||||
)?
|
DBError(format!(
|
||||||
} else {
|
"Failed to create directory {}: {}",
|
||||||
// Data DBs: via global registry keyed by id
|
parent_dir.display(),
|
||||||
admin_meta::open_data_storage(
|
e
|
||||||
&self.option.dir,
|
))
|
||||||
self.option.backend.clone(),
|
})?;
|
||||||
&self.option.admin_secret,
|
}
|
||||||
self.selected_db,
|
|
||||||
)?
|
println!("Creating new db file: {}", db_file_path.display());
|
||||||
|
|
||||||
|
let storage: Arc<dyn StorageBackend> = match self.option.backend {
|
||||||
|
options::BackendType::Redb => Arc::new(Storage::new(
|
||||||
|
db_file_path,
|
||||||
|
self.should_encrypt_db(self.selected_db),
|
||||||
|
self.option.encryption_key.as_deref(),
|
||||||
|
)?),
|
||||||
|
options::BackendType::Sled => Arc::new(SledStorage::new(
|
||||||
|
db_file_path,
|
||||||
|
self.should_encrypt_db(self.selected_db),
|
||||||
|
self.option.encryption_key.as_deref(),
|
||||||
|
)?),
|
||||||
};
|
};
|
||||||
|
|
||||||
cache.insert(self.selected_db, storage.clone());
|
cache.insert(self.selected_db, storage.clone());
|
||||||
Ok(storage)
|
Ok(storage)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn should_encrypt_db(&self, db_index: u64) -> bool {
|
||||||
/// Check if current permissions allow read operations
|
// DB 0-9 are non-encrypted, DB 10+ are encrypted
|
||||||
pub fn has_read_permission(&self) -> bool {
|
self.option.encrypt && db_index >= 10
|
||||||
// If an explicit permission is set for this connection, honor it.
|
|
||||||
if let Some(perms) = self.current_permissions.as_ref() {
|
|
||||||
return matches!(*perms, crate::rpc::Permissions::Read | crate::rpc::Permissions::ReadWrite);
|
|
||||||
}
|
|
||||||
// Fallback ONLY when no explicit permission context (e.g., JSON-RPC flows without SELECT).
|
|
||||||
match crate::admin_meta::verify_access(
|
|
||||||
&self.option.dir,
|
|
||||||
self.option.backend.clone(),
|
|
||||||
&self.option.admin_secret,
|
|
||||||
self.selected_db,
|
|
||||||
None,
|
|
||||||
) {
|
|
||||||
Ok(Some(crate::rpc::Permissions::Read)) | Ok(Some(crate::rpc::Permissions::ReadWrite)) => true,
|
|
||||||
_ => false,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Check if current permissions allow write operations
|
// Add method to get search index path
|
||||||
pub fn has_write_permission(&self) -> bool {
|
pub fn search_index_path(&self) -> std::path::PathBuf {
|
||||||
// If an explicit permission is set for this connection, honor it.
|
std::path::PathBuf::from(&self.option.dir).join("search_indexes")
|
||||||
if let Some(perms) = self.current_permissions.as_ref() {
|
|
||||||
return matches!(*perms, crate::rpc::Permissions::ReadWrite);
|
|
||||||
}
|
|
||||||
// Fallback ONLY when no explicit permission context (e.g., JSON-RPC flows without SELECT).
|
|
||||||
match crate::admin_meta::verify_access(
|
|
||||||
&self.option.dir,
|
|
||||||
self.option.backend.clone(),
|
|
||||||
&self.option.admin_secret,
|
|
||||||
self.selected_db,
|
|
||||||
None,
|
|
||||||
) {
|
|
||||||
Ok(Some(crate::rpc::Permissions::ReadWrite)) => true,
|
|
||||||
_ => false,
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ----- BLPOP waiter helpers -----
|
// ----- BLPOP waiter helpers -----
|
||||||
|
|
||||||
pub async fn register_waiter(&self, db_index: u64, key: &str, side: PopSide) -> (u64, oneshot::Receiver<(String, String)>) {
|
pub async fn register_waiter(
|
||||||
|
&self,
|
||||||
|
db_index: u64,
|
||||||
|
key: &str,
|
||||||
|
side: PopSide,
|
||||||
|
) -> (u64, oneshot::Receiver<(String, String)>) {
|
||||||
let id = self.waiter_seq.fetch_add(1, Ordering::Relaxed);
|
let id = self.waiter_seq.fetch_add(1, Ordering::Relaxed);
|
||||||
let (tx, rx) = oneshot::channel::<(String, String)>();
|
let (tx, rx) = oneshot::channel::<(String, String)>();
|
||||||
|
|
||||||
@@ -217,10 +192,7 @@ impl Server {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
pub async fn handle(
|
pub async fn handle(&mut self, mut stream: tokio::net::TcpStream) -> Result<(), DBError> {
|
||||||
&mut self,
|
|
||||||
mut stream: tokio::net::TcpStream,
|
|
||||||
) -> Result<(), DBError> {
|
|
||||||
// Accumulate incoming bytes to handle partial RESP frames
|
// Accumulate incoming bytes to handle partial RESP frames
|
||||||
let mut acc = String::new();
|
let mut acc = String::new();
|
||||||
let mut buf = vec![0u8; 8192];
|
let mut buf = vec![0u8; 8192];
|
||||||
@@ -257,7 +229,10 @@ impl Server {
|
|||||||
acc = remaining.to_string();
|
acc = remaining.to_string();
|
||||||
|
|
||||||
if self.option.debug {
|
if self.option.debug {
|
||||||
println!("\x1b[34;1mgot command: {:?}, protocol: {:?}\x1b[0m", cmd, protocol);
|
println!(
|
||||||
|
"\x1b[34;1mgot command: {:?}, protocol: {:?}\x1b[0m",
|
||||||
|
cmd, protocol
|
||||||
|
);
|
||||||
} else {
|
} else {
|
||||||
println!("got command: {:?}, protocol: {:?}", cmd, protocol);
|
println!("got command: {:?}, protocol: {:?}", cmd, protocol);
|
||||||
}
|
}
|
||||||
|
@@ -12,9 +12,9 @@ use crate::error::DBError;
|
|||||||
|
|
||||||
// Re-export modules
|
// Re-export modules
|
||||||
mod storage_basic;
|
mod storage_basic;
|
||||||
|
mod storage_extra;
|
||||||
mod storage_hset;
|
mod storage_hset;
|
||||||
mod storage_lists;
|
mod storage_lists;
|
||||||
mod storage_extra;
|
|
||||||
|
|
||||||
// Re-export implementations
|
// Re-export implementations
|
||||||
// Note: These imports are used by the impl blocks in the submodules
|
// Note: These imports are used by the impl blocks in the submodules
|
||||||
@@ -28,7 +28,8 @@ const STRINGS_TABLE: TableDefinition<&str, &[u8]> = TableDefinition::new("string
|
|||||||
const HASHES_TABLE: TableDefinition<(&str, &str), &[u8]> = TableDefinition::new("hashes");
|
const HASHES_TABLE: TableDefinition<(&str, &str), &[u8]> = TableDefinition::new("hashes");
|
||||||
const LISTS_TABLE: TableDefinition<&str, &[u8]> = TableDefinition::new("lists");
|
const LISTS_TABLE: TableDefinition<&str, &[u8]> = TableDefinition::new("lists");
|
||||||
const STREAMS_META_TABLE: TableDefinition<&str, &[u8]> = TableDefinition::new("streams_meta");
|
const STREAMS_META_TABLE: TableDefinition<&str, &[u8]> = TableDefinition::new("streams_meta");
|
||||||
const STREAMS_DATA_TABLE: TableDefinition<(&str, &str), &[u8]> = TableDefinition::new("streams_data");
|
const STREAMS_DATA_TABLE: TableDefinition<(&str, &str), &[u8]> =
|
||||||
|
TableDefinition::new("streams_data");
|
||||||
const ENCRYPTED_TABLE: TableDefinition<&str, u8> = TableDefinition::new("encrypted");
|
const ENCRYPTED_TABLE: TableDefinition<&str, u8> = TableDefinition::new("encrypted");
|
||||||
const EXPIRATION_TABLE: TableDefinition<&str, u64> = TableDefinition::new("expiration");
|
const EXPIRATION_TABLE: TableDefinition<&str, u64> = TableDefinition::new("expiration");
|
||||||
|
|
||||||
@@ -55,7 +56,11 @@ pub struct Storage {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl Storage {
|
impl Storage {
|
||||||
pub fn new(path: impl AsRef<Path>, should_encrypt: bool, master_key: Option<&str>) -> Result<Self, DBError> {
|
pub fn new(
|
||||||
|
path: impl AsRef<Path>,
|
||||||
|
should_encrypt: bool,
|
||||||
|
master_key: Option<&str>,
|
||||||
|
) -> Result<Self, DBError> {
|
||||||
let db = Database::create(path)?;
|
let db = Database::create(path)?;
|
||||||
|
|
||||||
// Create tables if they don't exist
|
// Create tables if they don't exist
|
||||||
@@ -75,14 +80,19 @@ impl Storage {
|
|||||||
// Check if database was previously encrypted
|
// Check if database was previously encrypted
|
||||||
let read_txn = db.begin_read()?;
|
let read_txn = db.begin_read()?;
|
||||||
let encrypted_table = read_txn.open_table(ENCRYPTED_TABLE)?;
|
let encrypted_table = read_txn.open_table(ENCRYPTED_TABLE)?;
|
||||||
let was_encrypted = encrypted_table.get("encrypted")?.map(|v| v.value() == 1).unwrap_or(false);
|
let was_encrypted = encrypted_table
|
||||||
|
.get("encrypted")?
|
||||||
|
.map(|v| v.value() == 1)
|
||||||
|
.unwrap_or(false);
|
||||||
drop(read_txn);
|
drop(read_txn);
|
||||||
|
|
||||||
let crypto = if should_encrypt || was_encrypted {
|
let crypto = if should_encrypt || was_encrypted {
|
||||||
if let Some(key) = master_key {
|
if let Some(key) = master_key {
|
||||||
Some(CryptoFactory::new(key.as_bytes()))
|
Some(CryptoFactory::new(key.as_bytes()))
|
||||||
} else {
|
} else {
|
||||||
return Err(DBError("Encryption requested but no master key provided".to_string()));
|
return Err(DBError(
|
||||||
|
"Encryption requested but no master key provided".to_string(),
|
||||||
|
));
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
@@ -98,10 +108,7 @@ impl Storage {
|
|||||||
write_txn.commit()?;
|
write_txn.commit()?;
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(Storage {
|
Ok(Storage { db, crypto })
|
||||||
db,
|
|
||||||
crypto,
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn is_encrypted(&self) -> bool {
|
pub fn is_encrypted(&self) -> bool {
|
||||||
@@ -165,11 +172,22 @@ impl StorageBackend for Storage {
|
|||||||
self.get_key_type(key)
|
self.get_key_type(key)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn scan(&self, cursor: u64, pattern: Option<&str>, count: Option<u64>) -> Result<(u64, Vec<(String, String)>), DBError> {
|
fn scan(
|
||||||
|
&self,
|
||||||
|
cursor: u64,
|
||||||
|
pattern: Option<&str>,
|
||||||
|
count: Option<u64>,
|
||||||
|
) -> Result<(u64, Vec<(String, String)>), DBError> {
|
||||||
self.scan(cursor, pattern, count)
|
self.scan(cursor, pattern, count)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn hscan(&self, key: &str, cursor: u64, pattern: Option<&str>, count: Option<u64>) -> Result<(u64, Vec<(String, String)>), DBError> {
|
fn hscan(
|
||||||
|
&self,
|
||||||
|
key: &str,
|
||||||
|
cursor: u64,
|
||||||
|
pattern: Option<&str>,
|
||||||
|
count: Option<u64>,
|
||||||
|
) -> Result<(u64, Vec<(String, String)>), DBError> {
|
||||||
self.hscan(key, cursor, pattern, count)
|
self.hscan(key, cursor, pattern, count)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@@ -1,6 +1,6 @@
|
|||||||
use redb::{ReadableTable};
|
|
||||||
use crate::error::DBError;
|
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use crate::error::DBError;
|
||||||
|
use redb::ReadableTable;
|
||||||
|
|
||||||
impl Storage {
|
impl Storage {
|
||||||
pub fn flushdb(&self) -> Result<(), DBError> {
|
pub fn flushdb(&self) -> Result<(), DBError> {
|
||||||
@@ -15,11 +15,17 @@ impl Storage {
|
|||||||
let mut expiration_table = write_txn.open_table(EXPIRATION_TABLE)?;
|
let mut expiration_table = write_txn.open_table(EXPIRATION_TABLE)?;
|
||||||
|
|
||||||
// inefficient, but there is no other way
|
// inefficient, but there is no other way
|
||||||
let keys: Vec<String> = types_table.iter()?.map(|item| item.unwrap().0.value().to_string()).collect();
|
let keys: Vec<String> = types_table
|
||||||
|
.iter()?
|
||||||
|
.map(|item| item.unwrap().0.value().to_string())
|
||||||
|
.collect();
|
||||||
for key in keys {
|
for key in keys {
|
||||||
types_table.remove(key.as_str())?;
|
types_table.remove(key.as_str())?;
|
||||||
}
|
}
|
||||||
let keys: Vec<String> = strings_table.iter()?.map(|item| item.unwrap().0.value().to_string()).collect();
|
let keys: Vec<String> = strings_table
|
||||||
|
.iter()?
|
||||||
|
.map(|item| item.unwrap().0.value().to_string())
|
||||||
|
.collect();
|
||||||
for key in keys {
|
for key in keys {
|
||||||
strings_table.remove(key.as_str())?;
|
strings_table.remove(key.as_str())?;
|
||||||
}
|
}
|
||||||
@@ -34,23 +40,35 @@ impl Storage {
|
|||||||
for (key, field) in keys {
|
for (key, field) in keys {
|
||||||
hashes_table.remove((key.as_str(), field.as_str()))?;
|
hashes_table.remove((key.as_str(), field.as_str()))?;
|
||||||
}
|
}
|
||||||
let keys: Vec<String> = lists_table.iter()?.map(|item| item.unwrap().0.value().to_string()).collect();
|
let keys: Vec<String> = lists_table
|
||||||
|
.iter()?
|
||||||
|
.map(|item| item.unwrap().0.value().to_string())
|
||||||
|
.collect();
|
||||||
for key in keys {
|
for key in keys {
|
||||||
lists_table.remove(key.as_str())?;
|
lists_table.remove(key.as_str())?;
|
||||||
}
|
}
|
||||||
let keys: Vec<String> = streams_meta_table.iter()?.map(|item| item.unwrap().0.value().to_string()).collect();
|
let keys: Vec<String> = streams_meta_table
|
||||||
|
.iter()?
|
||||||
|
.map(|item| item.unwrap().0.value().to_string())
|
||||||
|
.collect();
|
||||||
for key in keys {
|
for key in keys {
|
||||||
streams_meta_table.remove(key.as_str())?;
|
streams_meta_table.remove(key.as_str())?;
|
||||||
}
|
}
|
||||||
let keys: Vec<(String,String)> = streams_data_table.iter()?.map(|item| {
|
let keys: Vec<(String, String)> = streams_data_table
|
||||||
|
.iter()?
|
||||||
|
.map(|item| {
|
||||||
let binding = item.unwrap();
|
let binding = item.unwrap();
|
||||||
let (key, field) = binding.0.value();
|
let (key, field) = binding.0.value();
|
||||||
(key.to_string(), field.to_string())
|
(key.to_string(), field.to_string())
|
||||||
}).collect();
|
})
|
||||||
|
.collect();
|
||||||
for (key, field) in keys {
|
for (key, field) in keys {
|
||||||
streams_data_table.remove((key.as_str(), field.as_str()))?;
|
streams_data_table.remove((key.as_str(), field.as_str()))?;
|
||||||
}
|
}
|
||||||
let keys: Vec<String> = expiration_table.iter()?.map(|item| item.unwrap().0.value().to_string()).collect();
|
let keys: Vec<String> = expiration_table
|
||||||
|
.iter()?
|
||||||
|
.map(|item| item.unwrap().0.value().to_string())
|
||||||
|
.collect();
|
||||||
for key in keys {
|
for key in keys {
|
||||||
expiration_table.remove(key.as_str())?;
|
expiration_table.remove(key.as_str())?;
|
||||||
}
|
}
|
||||||
@@ -163,7 +181,8 @@ impl Storage {
|
|||||||
{
|
{
|
||||||
let mut types_table = write_txn.open_table(TYPES_TABLE)?;
|
let mut types_table = write_txn.open_table(TYPES_TABLE)?;
|
||||||
let mut strings_table = write_txn.open_table(STRINGS_TABLE)?;
|
let mut strings_table = write_txn.open_table(STRINGS_TABLE)?;
|
||||||
let mut hashes_table: redb::Table<(&str, &str), &[u8]> = write_txn.open_table(HASHES_TABLE)?;
|
let mut hashes_table: redb::Table<(&str, &str), &[u8]> =
|
||||||
|
write_txn.open_table(HASHES_TABLE)?;
|
||||||
let mut lists_table = write_txn.open_table(LISTS_TABLE)?;
|
let mut lists_table = write_txn.open_table(LISTS_TABLE)?;
|
||||||
|
|
||||||
// Remove from type table
|
// Remove from type table
|
||||||
|
@@ -1,10 +1,15 @@
|
|||||||
use redb::{ReadableTable};
|
|
||||||
use crate::error::DBError;
|
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use crate::error::DBError;
|
||||||
|
use redb::ReadableTable;
|
||||||
|
|
||||||
impl Storage {
|
impl Storage {
|
||||||
// ✅ ENCRYPTION APPLIED: Values are decrypted after retrieval
|
// ✅ ENCRYPTION APPLIED: Values are decrypted after retrieval
|
||||||
pub fn scan(&self, cursor: u64, pattern: Option<&str>, count: Option<u64>) -> Result<(u64, Vec<(String, String)>), DBError> {
|
pub fn scan(
|
||||||
|
&self,
|
||||||
|
cursor: u64,
|
||||||
|
pattern: Option<&str>,
|
||||||
|
count: Option<u64>,
|
||||||
|
) -> Result<(u64, Vec<(String, String)>), DBError> {
|
||||||
let read_txn = self.db.begin_read()?;
|
let read_txn = self.db.begin_read()?;
|
||||||
let types_table = read_txn.open_table(TYPES_TABLE)?;
|
let types_table = read_txn.open_table(TYPES_TABLE)?;
|
||||||
let strings_table = read_txn.open_table(STRINGS_TABLE)?;
|
let strings_table = read_txn.open_table(STRINGS_TABLE)?;
|
||||||
@@ -50,7 +55,11 @@ impl Storage {
|
|||||||
current_cursor += 1;
|
current_cursor += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
let next_cursor = if result.len() < limit { 0 } else { current_cursor };
|
let next_cursor = if result.len() < limit {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
current_cursor
|
||||||
|
};
|
||||||
Ok((next_cursor, result))
|
Ok((next_cursor, result))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -178,8 +187,12 @@ impl Storage {
|
|||||||
.unwrap_or(false);
|
.unwrap_or(false);
|
||||||
if is_string {
|
if is_string {
|
||||||
let mut expiration_table = write_txn.open_table(EXPIRATION_TABLE)?;
|
let mut expiration_table = write_txn.open_table(EXPIRATION_TABLE)?;
|
||||||
let expires_at_ms: u128 = if ts_secs <= 0 { 0 } else { (ts_secs as u128) * 1000 };
|
let expires_at_ms: u128 = if ts_secs <= 0 {
|
||||||
expiration_table.insert(key, &((expires_at_ms as u64)))?;
|
0
|
||||||
|
} else {
|
||||||
|
(ts_secs as u128) * 1000
|
||||||
|
};
|
||||||
|
expiration_table.insert(key, &(expires_at_ms as u64))?;
|
||||||
applied = true;
|
applied = true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -201,7 +214,7 @@ impl Storage {
|
|||||||
if is_string {
|
if is_string {
|
||||||
let mut expiration_table = write_txn.open_table(EXPIRATION_TABLE)?;
|
let mut expiration_table = write_txn.open_table(EXPIRATION_TABLE)?;
|
||||||
let expires_at_ms: u128 = if ts_ms <= 0 { 0 } else { ts_ms as u128 };
|
let expires_at_ms: u128 = if ts_ms <= 0 { 0 } else { ts_ms as u128 };
|
||||||
expiration_table.insert(key, &((expires_at_ms as u64)))?;
|
expiration_table.insert(key, &(expires_at_ms as u64))?;
|
||||||
applied = true;
|
applied = true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@@ -1,6 +1,6 @@
|
|||||||
use redb::{ReadableTable};
|
|
||||||
use crate::error::DBError;
|
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use crate::error::DBError;
|
||||||
|
use redb::ReadableTable;
|
||||||
|
|
||||||
impl Storage {
|
impl Storage {
|
||||||
// ✅ ENCRYPTION APPLIED: Values are encrypted before storage
|
// ✅ ENCRYPTION APPLIED: Values are encrypted before storage
|
||||||
@@ -18,7 +18,8 @@ impl Storage {
|
|||||||
};
|
};
|
||||||
|
|
||||||
match key_type.as_deref() {
|
match key_type.as_deref() {
|
||||||
Some("hash") | None => { // Proceed if hash or new key
|
Some("hash") | None => {
|
||||||
|
// Proceed if hash or new key
|
||||||
// Set the type to hash (only if new key or existing hash)
|
// Set the type to hash (only if new key or existing hash)
|
||||||
types_table.insert(key, "hash")?;
|
types_table.insert(key, "hash")?;
|
||||||
|
|
||||||
@@ -35,7 +36,12 @@ impl Storage {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Some(_) => return Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => {
|
||||||
|
return Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value"
|
||||||
|
.to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -62,7 +68,9 @@ impl Storage {
|
|||||||
None => Ok(None),
|
None => Ok(None),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Some(_) => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
)),
|
||||||
None => Ok(None),
|
None => Ok(None),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -94,7 +102,9 @@ impl Storage {
|
|||||||
|
|
||||||
Ok(result)
|
Ok(result)
|
||||||
}
|
}
|
||||||
Some(_) => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
)),
|
||||||
None => Ok(Vec::new()),
|
None => Ok(Vec::new()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -138,7 +148,11 @@ impl Storage {
|
|||||||
types_table.remove(key)?;
|
types_table.remove(key)?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Some(_) => return Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => {
|
||||||
|
return Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
None => {} // Key does not exist, nothing to delete, return 0 deleted
|
None => {} // Key does not exist, nothing to delete, return 0 deleted
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -159,7 +173,9 @@ impl Storage {
|
|||||||
let hashes_table = read_txn.open_table(HASHES_TABLE)?;
|
let hashes_table = read_txn.open_table(HASHES_TABLE)?;
|
||||||
Ok(hashes_table.get((key, field))?.is_some())
|
Ok(hashes_table.get((key, field))?.is_some())
|
||||||
}
|
}
|
||||||
Some(_) => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
)),
|
||||||
None => Ok(false),
|
None => Ok(false),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -188,7 +204,9 @@ impl Storage {
|
|||||||
|
|
||||||
Ok(result)
|
Ok(result)
|
||||||
}
|
}
|
||||||
Some(_) => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
)),
|
||||||
None => Ok(Vec::new()),
|
None => Ok(Vec::new()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -220,7 +238,9 @@ impl Storage {
|
|||||||
|
|
||||||
Ok(result)
|
Ok(result)
|
||||||
}
|
}
|
||||||
Some(_) => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
)),
|
||||||
None => Ok(Vec::new()),
|
None => Ok(Vec::new()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -249,7 +269,9 @@ impl Storage {
|
|||||||
|
|
||||||
Ok(count)
|
Ok(count)
|
||||||
}
|
}
|
||||||
Some(_) => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
)),
|
||||||
None => Ok(0),
|
None => Ok(0),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -281,7 +303,9 @@ impl Storage {
|
|||||||
|
|
||||||
Ok(result)
|
Ok(result)
|
||||||
}
|
}
|
||||||
Some(_) => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
)),
|
||||||
None => Ok(fields.into_iter().map(|_| None).collect()),
|
None => Ok(fields.into_iter().map(|_| None).collect()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -301,7 +325,8 @@ impl Storage {
|
|||||||
};
|
};
|
||||||
|
|
||||||
match key_type.as_deref() {
|
match key_type.as_deref() {
|
||||||
Some("hash") | None => { // Proceed if hash or new key
|
Some("hash") | None => {
|
||||||
|
// Proceed if hash or new key
|
||||||
// Check if field already exists
|
// Check if field already exists
|
||||||
if hashes_table.get((key, field))?.is_none() {
|
if hashes_table.get((key, field))?.is_none() {
|
||||||
// Set the type to hash (only if new key or existing hash)
|
// Set the type to hash (only if new key or existing hash)
|
||||||
@@ -313,7 +338,12 @@ impl Storage {
|
|||||||
result = true;
|
result = true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Some(_) => return Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => {
|
||||||
|
return Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value"
|
||||||
|
.to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -322,7 +352,13 @@ impl Storage {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ✅ ENCRYPTION APPLIED: Values are decrypted after retrieval
|
// ✅ ENCRYPTION APPLIED: Values are decrypted after retrieval
|
||||||
pub fn hscan(&self, key: &str, cursor: u64, pattern: Option<&str>, count: Option<u64>) -> Result<(u64, Vec<(String, String)>), DBError> {
|
pub fn hscan(
|
||||||
|
&self,
|
||||||
|
key: &str,
|
||||||
|
cursor: u64,
|
||||||
|
pattern: Option<&str>,
|
||||||
|
count: Option<u64>,
|
||||||
|
) -> Result<(u64, Vec<(String, String)>), DBError> {
|
||||||
let read_txn = self.db.begin_read()?;
|
let read_txn = self.db.begin_read()?;
|
||||||
let types_table = read_txn.open_table(TYPES_TABLE)?;
|
let types_table = read_txn.open_table(TYPES_TABLE)?;
|
||||||
let key_type = {
|
let key_type = {
|
||||||
@@ -367,10 +403,16 @@ impl Storage {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let next_cursor = if result.len() < limit { 0 } else { current_cursor };
|
let next_cursor = if result.len() < limit {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
current_cursor
|
||||||
|
};
|
||||||
Ok((next_cursor, result))
|
Ok((next_cursor, result))
|
||||||
}
|
}
|
||||||
Some(_) => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
Some(_) => Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
)),
|
||||||
None => Ok((0, Vec::new())),
|
None => Ok((0, Vec::new())),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@@ -1,6 +1,6 @@
|
|||||||
use redb::{ReadableTable};
|
|
||||||
use crate::error::DBError;
|
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use crate::error::DBError;
|
||||||
|
use redb::ReadableTable;
|
||||||
|
|
||||||
impl Storage {
|
impl Storage {
|
||||||
// ✅ ENCRYPTION APPLIED: Elements are encrypted before storage
|
// ✅ ENCRYPTION APPLIED: Elements are encrypted before storage
|
||||||
@@ -248,8 +248,16 @@ impl Storage {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let len = list.len() as i64;
|
let len = list.len() as i64;
|
||||||
let start_idx = if start < 0 { std::cmp::max(0, len + start) } else { std::cmp::min(start, len) };
|
let start_idx = if start < 0 {
|
||||||
let stop_idx = if stop < 0 { std::cmp::max(-1, len + stop) } else { std::cmp::min(stop, len - 1) };
|
std::cmp::max(0, len + start)
|
||||||
|
} else {
|
||||||
|
std::cmp::min(start, len)
|
||||||
|
};
|
||||||
|
let stop_idx = if stop < 0 {
|
||||||
|
std::cmp::max(-1, len + stop)
|
||||||
|
} else {
|
||||||
|
std::cmp::min(stop, len - 1)
|
||||||
|
};
|
||||||
|
|
||||||
if start_idx > stop_idx || start_idx >= len {
|
if start_idx > stop_idx || start_idx >= len {
|
||||||
return Ok(Vec::new());
|
return Ok(Vec::new());
|
||||||
@@ -298,8 +306,16 @@ impl Storage {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let len = list.len() as i64;
|
let len = list.len() as i64;
|
||||||
let start_idx = if start < 0 { std::cmp::max(0, len + start) } else { std::cmp::min(start, len) };
|
let start_idx = if start < 0 {
|
||||||
let stop_idx = if stop < 0 { std::cmp::max(-1, len + stop) } else { std::cmp::min(stop, len - 1) };
|
std::cmp::max(0, len + start)
|
||||||
|
} else {
|
||||||
|
std::cmp::min(start, len)
|
||||||
|
};
|
||||||
|
let stop_idx = if stop < 0 {
|
||||||
|
std::cmp::max(-1, len + stop)
|
||||||
|
} else {
|
||||||
|
std::cmp::min(stop, len - 1)
|
||||||
|
};
|
||||||
|
|
||||||
let mut lists_table = write_txn.open_table(LISTS_TABLE)?;
|
let mut lists_table = write_txn.open_table(LISTS_TABLE)?;
|
||||||
if start_idx > stop_idx || start_idx >= len {
|
if start_idx > stop_idx || start_idx >= len {
|
||||||
|
@@ -1,12 +1,12 @@
|
|||||||
// src/storage_sled/mod.rs
|
// src/storage_sled/mod.rs
|
||||||
use std::path::Path;
|
use crate::crypto::CryptoFactory;
|
||||||
use std::sync::Arc;
|
|
||||||
use std::collections::HashMap;
|
|
||||||
use std::time::{SystemTime, UNIX_EPOCH};
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
use crate::error::DBError;
|
use crate::error::DBError;
|
||||||
use crate::storage_trait::StorageBackend;
|
use crate::storage_trait::StorageBackend;
|
||||||
use crate::crypto::CryptoFactory;
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::{SystemTime, UNIX_EPOCH};
|
||||||
|
|
||||||
#[derive(Serialize, Deserialize, Debug, Clone)]
|
#[derive(Serialize, Deserialize, Debug, Clone)]
|
||||||
enum ValueType {
|
enum ValueType {
|
||||||
@@ -28,13 +28,22 @@ pub struct SledStorage {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl SledStorage {
|
impl SledStorage {
|
||||||
pub fn new(path: impl AsRef<Path>, should_encrypt: bool, master_key: Option<&str>) -> Result<Self, DBError> {
|
pub fn new(
|
||||||
|
path: impl AsRef<Path>,
|
||||||
|
should_encrypt: bool,
|
||||||
|
master_key: Option<&str>,
|
||||||
|
) -> Result<Self, DBError> {
|
||||||
let db = sled::open(path).map_err(|e| DBError(format!("Failed to open sled: {}", e)))?;
|
let db = sled::open(path).map_err(|e| DBError(format!("Failed to open sled: {}", e)))?;
|
||||||
let types = db.open_tree("types").map_err(|e| DBError(format!("Failed to open types tree: {}", e)))?;
|
let types = db
|
||||||
|
.open_tree("types")
|
||||||
|
.map_err(|e| DBError(format!("Failed to open types tree: {}", e)))?;
|
||||||
|
|
||||||
// Check if database was previously encrypted
|
// Check if database was previously encrypted
|
||||||
let encrypted_tree = db.open_tree("encrypted").map_err(|e| DBError(e.to_string()))?;
|
let encrypted_tree = db
|
||||||
let was_encrypted = encrypted_tree.get("encrypted")
|
.open_tree("encrypted")
|
||||||
|
.map_err(|e| DBError(e.to_string()))?;
|
||||||
|
let was_encrypted = encrypted_tree
|
||||||
|
.get("encrypted")
|
||||||
.map_err(|e| DBError(e.to_string()))?
|
.map_err(|e| DBError(e.to_string()))?
|
||||||
.map(|v| v[0] == 1)
|
.map(|v| v[0] == 1)
|
||||||
.unwrap_or(false);
|
.unwrap_or(false);
|
||||||
@@ -43,7 +52,9 @@ impl SledStorage {
|
|||||||
if let Some(key) = master_key {
|
if let Some(key) = master_key {
|
||||||
Some(CryptoFactory::new(key.as_bytes()))
|
Some(CryptoFactory::new(key.as_bytes()))
|
||||||
} else {
|
} else {
|
||||||
return Err(DBError("Encryption requested but no master key provided".to_string()));
|
return Err(DBError(
|
||||||
|
"Encryption requested but no master key provided".to_string(),
|
||||||
|
));
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
@@ -51,7 +62,8 @@ impl SledStorage {
|
|||||||
|
|
||||||
// Mark database as encrypted if enabling encryption
|
// Mark database as encrypted if enabling encryption
|
||||||
if should_encrypt && !was_encrypted {
|
if should_encrypt && !was_encrypted {
|
||||||
encrypted_tree.insert("encrypted", &[1u8])
|
encrypted_tree
|
||||||
|
.insert("encrypted", &[1u8])
|
||||||
.map_err(|e| DBError(e.to_string()))?;
|
.map_err(|e| DBError(e.to_string()))?;
|
||||||
encrypted_tree.flush().map_err(|e| DBError(e.to_string()))?;
|
encrypted_tree.flush().map_err(|e| DBError(e.to_string()))?;
|
||||||
}
|
}
|
||||||
@@ -101,7 +113,7 @@ impl SledStorage {
|
|||||||
|
|
||||||
Ok(Some(storage_val))
|
Ok(Some(storage_val))
|
||||||
}
|
}
|
||||||
None => Ok(None)
|
None => Ok(None),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -109,7 +121,9 @@ impl SledStorage {
|
|||||||
let data = bincode::serialize(&storage_val)
|
let data = bincode::serialize(&storage_val)
|
||||||
.map_err(|e| DBError(format!("Serialization error: {}", e)))?;
|
.map_err(|e| DBError(format!("Serialization error: {}", e)))?;
|
||||||
let encrypted = self.encrypt_if_needed(&data)?;
|
let encrypted = self.encrypt_if_needed(&data)?;
|
||||||
self.db.insert(key, encrypted).map_err(|e| DBError(e.to_string()))?;
|
self.db
|
||||||
|
.insert(key, encrypted)
|
||||||
|
.map_err(|e| DBError(e.to_string()))?;
|
||||||
|
|
||||||
// Store type info (unencrypted for efficiency)
|
// Store type info (unencrypted for efficiency)
|
||||||
let type_str = match &storage_val.value {
|
let type_str = match &storage_val.value {
|
||||||
@@ -117,7 +131,9 @@ impl SledStorage {
|
|||||||
ValueType::Hash(_) => "hash",
|
ValueType::Hash(_) => "hash",
|
||||||
ValueType::List(_) => "list",
|
ValueType::List(_) => "list",
|
||||||
};
|
};
|
||||||
self.types.insert(key, type_str.as_bytes()).map_err(|e| DBError(e.to_string()))?;
|
self.types
|
||||||
|
.insert(key, type_str.as_bytes())
|
||||||
|
.map_err(|e| DBError(e.to_string()))?;
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -168,9 +184,9 @@ impl StorageBackend for SledStorage {
|
|||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::String(s) => Ok(Some(s)),
|
ValueType::String(s) => Ok(Some(s)),
|
||||||
_ => Ok(None)
|
_ => Ok(None),
|
||||||
}
|
},
|
||||||
None => Ok(None)
|
None => Ok(None),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -196,7 +212,9 @@ impl StorageBackend for SledStorage {
|
|||||||
|
|
||||||
fn del(&self, key: String) -> Result<(), DBError> {
|
fn del(&self, key: String) -> Result<(), DBError> {
|
||||||
self.db.remove(&key).map_err(|e| DBError(e.to_string()))?;
|
self.db.remove(&key).map_err(|e| DBError(e.to_string()))?;
|
||||||
self.types.remove(&key).map_err(|e| DBError(e.to_string()))?;
|
self.types
|
||||||
|
.remove(&key)
|
||||||
|
.map_err(|e| DBError(e.to_string()))?;
|
||||||
self.db.flush().map_err(|e| DBError(e.to_string()))?;
|
self.db.flush().map_err(|e| DBError(e.to_string()))?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -222,7 +240,12 @@ impl StorageBackend for SledStorage {
|
|||||||
Ok(keys)
|
Ok(keys)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn scan(&self, cursor: u64, pattern: Option<&str>, count: Option<u64>) -> Result<(u64, Vec<(String, String)>), DBError> {
|
fn scan(
|
||||||
|
&self,
|
||||||
|
cursor: u64,
|
||||||
|
pattern: Option<&str>,
|
||||||
|
count: Option<u64>,
|
||||||
|
) -> Result<(u64, Vec<(String, String)>), DBError> {
|
||||||
let mut result = Vec::new();
|
let mut result = Vec::new();
|
||||||
let mut current_cursor = 0u64;
|
let mut current_cursor = 0u64;
|
||||||
let limit = count.unwrap_or(10) as usize;
|
let limit = count.unwrap_or(10) as usize;
|
||||||
@@ -258,7 +281,11 @@ impl StorageBackend for SledStorage {
|
|||||||
current_cursor += 1;
|
current_cursor += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
let next_cursor = if result.len() < limit { 0 } else { current_cursor };
|
let next_cursor = if result.len() < limit {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
current_cursor
|
||||||
|
};
|
||||||
Ok((next_cursor, result))
|
Ok((next_cursor, result))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -286,7 +313,7 @@ impl StorageBackend for SledStorage {
|
|||||||
if self.get_storage_value(key)?.is_some() {
|
if self.get_storage_value(key)?.is_some() {
|
||||||
match self.types.get(key).map_err(|e| DBError(e.to_string()))? {
|
match self.types.get(key).map_err(|e| DBError(e.to_string()))? {
|
||||||
Some(data) => Ok(Some(String::from_utf8_lossy(&data).to_string())),
|
Some(data) => Ok(Some(String::from_utf8_lossy(&data).to_string())),
|
||||||
None => Ok(None)
|
None => Ok(None),
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
Ok(None)
|
Ok(None)
|
||||||
@@ -302,7 +329,11 @@ impl StorageBackend for SledStorage {
|
|||||||
|
|
||||||
let hash = match &mut storage_val.value {
|
let hash = match &mut storage_val.value {
|
||||||
ValueType::Hash(h) => h,
|
ValueType::Hash(h) => h,
|
||||||
_ => return Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
_ => {
|
||||||
|
return Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut new_fields = 0i64;
|
let mut new_fields = 0i64;
|
||||||
@@ -322,9 +353,9 @@ impl StorageBackend for SledStorage {
|
|||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::Hash(h) => Ok(h.get(field).cloned()),
|
ValueType::Hash(h) => Ok(h.get(field).cloned()),
|
||||||
_ => Ok(None)
|
_ => Ok(None),
|
||||||
}
|
},
|
||||||
None => Ok(None)
|
None => Ok(None),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -332,13 +363,19 @@ impl StorageBackend for SledStorage {
|
|||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::Hash(h) => Ok(h.into_iter().collect()),
|
ValueType::Hash(h) => Ok(h.into_iter().collect()),
|
||||||
_ => Ok(Vec::new())
|
_ => Ok(Vec::new()),
|
||||||
}
|
},
|
||||||
None => Ok(Vec::new())
|
None => Ok(Vec::new()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn hscan(&self, key: &str, cursor: u64, pattern: Option<&str>, count: Option<u64>) -> Result<(u64, Vec<(String, String)>), DBError> {
|
fn hscan(
|
||||||
|
&self,
|
||||||
|
key: &str,
|
||||||
|
cursor: u64,
|
||||||
|
pattern: Option<&str>,
|
||||||
|
count: Option<u64>,
|
||||||
|
) -> Result<(u64, Vec<(String, String)>), DBError> {
|
||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::Hash(h) => {
|
ValueType::Hash(h) => {
|
||||||
@@ -365,24 +402,30 @@ impl StorageBackend for SledStorage {
|
|||||||
current_cursor += 1;
|
current_cursor += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
let next_cursor = if result.len() < limit { 0 } else { current_cursor };
|
let next_cursor = if result.len() < limit {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
current_cursor
|
||||||
|
};
|
||||||
Ok((next_cursor, result))
|
Ok((next_cursor, result))
|
||||||
}
|
}
|
||||||
_ => Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string()))
|
_ => Err(DBError(
|
||||||
}
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
None => Ok((0, Vec::new()))
|
)),
|
||||||
|
},
|
||||||
|
None => Ok((0, Vec::new())),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn hdel(&self, key: &str, fields: Vec<String>) -> Result<i64, DBError> {
|
fn hdel(&self, key: &str, fields: Vec<String>) -> Result<i64, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(0)
|
None => return Ok(0),
|
||||||
};
|
};
|
||||||
|
|
||||||
let hash = match &mut storage_val.value {
|
let hash = match &mut storage_val.value {
|
||||||
ValueType::Hash(h) => h,
|
ValueType::Hash(h) => h,
|
||||||
_ => return Ok(0)
|
_ => return Ok(0),
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut deleted = 0i64;
|
let mut deleted = 0i64;
|
||||||
@@ -406,9 +449,9 @@ impl StorageBackend for SledStorage {
|
|||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::Hash(h) => Ok(h.contains_key(field)),
|
ValueType::Hash(h) => Ok(h.contains_key(field)),
|
||||||
_ => Ok(false)
|
_ => Ok(false),
|
||||||
}
|
},
|
||||||
None => Ok(false)
|
None => Ok(false),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -416,9 +459,9 @@ impl StorageBackend for SledStorage {
|
|||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::Hash(h) => Ok(h.keys().cloned().collect()),
|
ValueType::Hash(h) => Ok(h.keys().cloned().collect()),
|
||||||
_ => Ok(Vec::new())
|
_ => Ok(Vec::new()),
|
||||||
}
|
},
|
||||||
None => Ok(Vec::new())
|
None => Ok(Vec::new()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -426,9 +469,9 @@ impl StorageBackend for SledStorage {
|
|||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::Hash(h) => Ok(h.values().cloned().collect()),
|
ValueType::Hash(h) => Ok(h.values().cloned().collect()),
|
||||||
_ => Ok(Vec::new())
|
_ => Ok(Vec::new()),
|
||||||
}
|
},
|
||||||
None => Ok(Vec::new())
|
None => Ok(Vec::new()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -436,21 +479,19 @@ impl StorageBackend for SledStorage {
|
|||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::Hash(h) => Ok(h.len() as i64),
|
ValueType::Hash(h) => Ok(h.len() as i64),
|
||||||
_ => Ok(0)
|
_ => Ok(0),
|
||||||
}
|
},
|
||||||
None => Ok(0)
|
None => Ok(0),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn hmget(&self, key: &str, fields: Vec<String>) -> Result<Vec<Option<String>>, DBError> {
|
fn hmget(&self, key: &str, fields: Vec<String>) -> Result<Vec<Option<String>>, DBError> {
|
||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::Hash(h) => {
|
ValueType::Hash(h) => Ok(fields.into_iter().map(|f| h.get(&f).cloned()).collect()),
|
||||||
Ok(fields.into_iter().map(|f| h.get(&f).cloned()).collect())
|
_ => Ok(fields.into_iter().map(|_| None).collect()),
|
||||||
}
|
},
|
||||||
_ => Ok(fields.into_iter().map(|_| None).collect())
|
None => Ok(fields.into_iter().map(|_| None).collect()),
|
||||||
}
|
|
||||||
None => Ok(fields.into_iter().map(|_| None).collect())
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -462,7 +503,11 @@ impl StorageBackend for SledStorage {
|
|||||||
|
|
||||||
let hash = match &mut storage_val.value {
|
let hash = match &mut storage_val.value {
|
||||||
ValueType::Hash(h) => h,
|
ValueType::Hash(h) => h,
|
||||||
_ => return Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
_ => {
|
||||||
|
return Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
if hash.contains_key(field) {
|
if hash.contains_key(field) {
|
||||||
@@ -484,7 +529,11 @@ impl StorageBackend for SledStorage {
|
|||||||
|
|
||||||
let list = match &mut storage_val.value {
|
let list = match &mut storage_val.value {
|
||||||
ValueType::List(l) => l,
|
ValueType::List(l) => l,
|
||||||
_ => return Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
_ => {
|
||||||
|
return Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
for element in elements.into_iter().rev() {
|
for element in elements.into_iter().rev() {
|
||||||
@@ -505,7 +554,11 @@ impl StorageBackend for SledStorage {
|
|||||||
|
|
||||||
let list = match &mut storage_val.value {
|
let list = match &mut storage_val.value {
|
||||||
ValueType::List(l) => l,
|
ValueType::List(l) => l,
|
||||||
_ => return Err(DBError("WRONGTYPE Operation against a key holding the wrong kind of value".to_string())),
|
_ => {
|
||||||
|
return Err(DBError(
|
||||||
|
"WRONGTYPE Operation against a key holding the wrong kind of value".to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
list.extend(elements);
|
list.extend(elements);
|
||||||
@@ -518,12 +571,12 @@ impl StorageBackend for SledStorage {
|
|||||||
fn lpop(&self, key: &str, count: u64) -> Result<Vec<String>, DBError> {
|
fn lpop(&self, key: &str, count: u64) -> Result<Vec<String>, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(Vec::new())
|
None => return Ok(Vec::new()),
|
||||||
};
|
};
|
||||||
|
|
||||||
let list = match &mut storage_val.value {
|
let list = match &mut storage_val.value {
|
||||||
ValueType::List(l) => l,
|
ValueType::List(l) => l,
|
||||||
_ => return Ok(Vec::new())
|
_ => return Ok(Vec::new()),
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut result = Vec::new();
|
let mut result = Vec::new();
|
||||||
@@ -547,12 +600,12 @@ impl StorageBackend for SledStorage {
|
|||||||
fn rpop(&self, key: &str, count: u64) -> Result<Vec<String>, DBError> {
|
fn rpop(&self, key: &str, count: u64) -> Result<Vec<String>, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(Vec::new())
|
None => return Ok(Vec::new()),
|
||||||
};
|
};
|
||||||
|
|
||||||
let list = match &mut storage_val.value {
|
let list = match &mut storage_val.value {
|
||||||
ValueType::List(l) => l,
|
ValueType::List(l) => l,
|
||||||
_ => return Ok(Vec::new())
|
_ => return Ok(Vec::new()),
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut result = Vec::new();
|
let mut result = Vec::new();
|
||||||
@@ -576,9 +629,9 @@ impl StorageBackend for SledStorage {
|
|||||||
match self.get_storage_value(key)? {
|
match self.get_storage_value(key)? {
|
||||||
Some(storage_val) => match storage_val.value {
|
Some(storage_val) => match storage_val.value {
|
||||||
ValueType::List(l) => Ok(l.len() as i64),
|
ValueType::List(l) => Ok(l.len() as i64),
|
||||||
_ => Ok(0)
|
_ => Ok(0),
|
||||||
}
|
},
|
||||||
None => Ok(0)
|
None => Ok(0),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -598,9 +651,9 @@ impl StorageBackend for SledStorage {
|
|||||||
Ok(None)
|
Ok(None)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
_ => Ok(None)
|
_ => Ok(None),
|
||||||
}
|
},
|
||||||
None => Ok(None)
|
None => Ok(None),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -633,21 +686,21 @@ impl StorageBackend for SledStorage {
|
|||||||
|
|
||||||
Ok(list[start_usize..std::cmp::min(stop_usize, list.len())].to_vec())
|
Ok(list[start_usize..std::cmp::min(stop_usize, list.len())].to_vec())
|
||||||
}
|
}
|
||||||
_ => Ok(Vec::new())
|
_ => Ok(Vec::new()),
|
||||||
}
|
},
|
||||||
None => Ok(Vec::new())
|
None => Ok(Vec::new()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn ltrim(&self, key: &str, start: i64, stop: i64) -> Result<(), DBError> {
|
fn ltrim(&self, key: &str, start: i64, stop: i64) -> Result<(), DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(())
|
None => return Ok(()),
|
||||||
};
|
};
|
||||||
|
|
||||||
let list = match &mut storage_val.value {
|
let list = match &mut storage_val.value {
|
||||||
ValueType::List(l) => l,
|
ValueType::List(l) => l,
|
||||||
_ => return Ok(())
|
_ => return Ok(()),
|
||||||
};
|
};
|
||||||
|
|
||||||
if list.is_empty() {
|
if list.is_empty() {
|
||||||
@@ -687,12 +740,12 @@ impl StorageBackend for SledStorage {
|
|||||||
fn lrem(&self, key: &str, count: i64, element: &str) -> Result<i64, DBError> {
|
fn lrem(&self, key: &str, count: i64, element: &str) -> Result<i64, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(0)
|
None => return Ok(0),
|
||||||
};
|
};
|
||||||
|
|
||||||
let list = match &mut storage_val.value {
|
let list = match &mut storage_val.value {
|
||||||
ValueType::List(l) => l,
|
ValueType::List(l) => l,
|
||||||
_ => return Ok(0)
|
_ => return Ok(0),
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut removed = 0i64;
|
let mut removed = 0i64;
|
||||||
@@ -751,14 +804,14 @@ impl StorageBackend for SledStorage {
|
|||||||
Ok(-1) // Key exists but has no expiration
|
Ok(-1) // Key exists but has no expiration
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
None => Ok(-2) // Key does not exist
|
None => Ok(-2), // Key does not exist
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn expire_seconds(&self, key: &str, secs: u64) -> Result<bool, DBError> {
|
fn expire_seconds(&self, key: &str, secs: u64) -> Result<bool, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(false)
|
None => return Ok(false),
|
||||||
};
|
};
|
||||||
|
|
||||||
storage_val.expires_at = Some(Self::now_millis() + (secs as u128) * 1000);
|
storage_val.expires_at = Some(Self::now_millis() + (secs as u128) * 1000);
|
||||||
@@ -770,7 +823,7 @@ impl StorageBackend for SledStorage {
|
|||||||
fn pexpire_millis(&self, key: &str, ms: u128) -> Result<bool, DBError> {
|
fn pexpire_millis(&self, key: &str, ms: u128) -> Result<bool, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(false)
|
None => return Ok(false),
|
||||||
};
|
};
|
||||||
|
|
||||||
storage_val.expires_at = Some(Self::now_millis() + ms);
|
storage_val.expires_at = Some(Self::now_millis() + ms);
|
||||||
@@ -782,7 +835,7 @@ impl StorageBackend for SledStorage {
|
|||||||
fn persist(&self, key: &str) -> Result<bool, DBError> {
|
fn persist(&self, key: &str) -> Result<bool, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(false)
|
None => return Ok(false),
|
||||||
};
|
};
|
||||||
|
|
||||||
if storage_val.expires_at.is_some() {
|
if storage_val.expires_at.is_some() {
|
||||||
@@ -798,10 +851,14 @@ impl StorageBackend for SledStorage {
|
|||||||
fn expire_at_seconds(&self, key: &str, ts_secs: i64) -> Result<bool, DBError> {
|
fn expire_at_seconds(&self, key: &str, ts_secs: i64) -> Result<bool, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(false)
|
None => return Ok(false),
|
||||||
};
|
};
|
||||||
|
|
||||||
let expires_at_ms: u128 = if ts_secs <= 0 { 0 } else { (ts_secs as u128) * 1000 };
|
let expires_at_ms: u128 = if ts_secs <= 0 {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
(ts_secs as u128) * 1000
|
||||||
|
};
|
||||||
storage_val.expires_at = Some(expires_at_ms);
|
storage_val.expires_at = Some(expires_at_ms);
|
||||||
self.set_storage_value(key, storage_val)?;
|
self.set_storage_value(key, storage_val)?;
|
||||||
self.db.flush().map_err(|e| DBError(e.to_string()))?;
|
self.db.flush().map_err(|e| DBError(e.to_string()))?;
|
||||||
@@ -811,7 +868,7 @@ impl StorageBackend for SledStorage {
|
|||||||
fn pexpire_at_millis(&self, key: &str, ts_ms: i64) -> Result<bool, DBError> {
|
fn pexpire_at_millis(&self, key: &str, ts_ms: i64) -> Result<bool, DBError> {
|
||||||
let mut storage_val = match self.get_storage_value(key)? {
|
let mut storage_val = match self.get_storage_value(key)? {
|
||||||
Some(sv) => sv,
|
Some(sv) => sv,
|
||||||
None => return Ok(false)
|
None => return Ok(false),
|
||||||
};
|
};
|
||||||
|
|
||||||
let expires_at_ms: u128 = if ts_ms <= 0 { 0 } else { ts_ms as u128 };
|
let expires_at_ms: u128 = if ts_ms <= 0 { 0 } else { ts_ms as u128 };
|
||||||
|
@@ -15,8 +15,19 @@ pub trait StorageBackend: Send + Sync {
|
|||||||
fn get_key_type(&self, key: &str) -> Result<Option<String>, DBError>;
|
fn get_key_type(&self, key: &str) -> Result<Option<String>, DBError>;
|
||||||
|
|
||||||
// Scanning
|
// Scanning
|
||||||
fn scan(&self, cursor: u64, pattern: Option<&str>, count: Option<u64>) -> Result<(u64, Vec<(String, String)>), DBError>;
|
fn scan(
|
||||||
fn hscan(&self, key: &str, cursor: u64, pattern: Option<&str>, count: Option<u64>) -> Result<(u64, Vec<(String, String)>), DBError>;
|
&self,
|
||||||
|
cursor: u64,
|
||||||
|
pattern: Option<&str>,
|
||||||
|
count: Option<u64>,
|
||||||
|
) -> Result<(u64, Vec<(String, String)>), DBError>;
|
||||||
|
fn hscan(
|
||||||
|
&self,
|
||||||
|
key: &str,
|
||||||
|
cursor: u64,
|
||||||
|
pattern: Option<&str>,
|
||||||
|
count: Option<u64>,
|
||||||
|
) -> Result<(u64, Vec<(String, String)>), DBError>;
|
||||||
|
|
||||||
// Hash operations
|
// Hash operations
|
||||||
fn hset(&self, key: &str, pairs: Vec<(String, String)>) -> Result<i64, DBError>;
|
fn hset(&self, key: &str, pairs: Vec<(String, String)>) -> Result<i64, DBError>;
|
||||||
|
123
src/sym.rs
123
src/sym.rs
@@ -1,123 +0,0 @@
|
|||||||
//! sym.rs — Stateless symmetric encryption (Phase 1)
|
|
||||||
//!
|
|
||||||
//! Commands implemented (RESP):
|
|
||||||
//! - SYM KEYGEN
|
|
||||||
//! - SYM ENCRYPT <key_b64> <message>
|
|
||||||
//! - SYM DECRYPT <key_b64> <ciphertext_b64>
|
|
||||||
//!
|
|
||||||
//! Notes:
|
|
||||||
//! - Raw key: exactly 32 bytes, provided as Base64 in commands.
|
|
||||||
//! - Cipher: XChaCha20-Poly1305 (AEAD) without AAD in Phase 1
|
|
||||||
//! - Ciphertext binary layout: [version:1][nonce:24][ciphertext||tag]
|
|
||||||
//! - Encoding for wire I/O: Base64
|
|
||||||
|
|
||||||
use base64::{engine::general_purpose::STANDARD as B64, Engine as _};
|
|
||||||
use chacha20poly1305::{
|
|
||||||
aead::{Aead, KeyInit, OsRng},
|
|
||||||
XChaCha20Poly1305, XNonce,
|
|
||||||
};
|
|
||||||
use rand::RngCore;
|
|
||||||
|
|
||||||
use crate::protocol::Protocol;
|
|
||||||
|
|
||||||
const VERSION: u8 = 1;
|
|
||||||
const NONCE_LEN: usize = 24;
|
|
||||||
const TAG_LEN: usize = 16;
|
|
||||||
|
|
||||||
#[derive(Debug)]
|
|
||||||
pub enum SymWireError {
|
|
||||||
InvalidKey,
|
|
||||||
BadEncoding,
|
|
||||||
BadFormat,
|
|
||||||
BadVersion(u8),
|
|
||||||
Crypto,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl SymWireError {
|
|
||||||
fn to_protocol(self) -> Protocol {
|
|
||||||
match self {
|
|
||||||
SymWireError::InvalidKey => Protocol::err("ERR sym: invalid key"),
|
|
||||||
SymWireError::BadEncoding => Protocol::err("ERR sym: bad encoding"),
|
|
||||||
SymWireError::BadFormat => Protocol::err("ERR sym: bad format"),
|
|
||||||
SymWireError::BadVersion(v) => Protocol::err(&format!("ERR sym: unsupported version {}", v)),
|
|
||||||
SymWireError::Crypto => Protocol::err("ERR sym: auth failed"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn decode_key_b64(s: &str) -> Result<chacha20poly1305::Key, SymWireError> {
|
|
||||||
let bytes = B64.decode(s.as_bytes()).map_err(|_| SymWireError::BadEncoding)?;
|
|
||||||
if bytes.len() != 32 {
|
|
||||||
return Err(SymWireError::InvalidKey);
|
|
||||||
}
|
|
||||||
Ok(chacha20poly1305::Key::from_slice(&bytes).to_owned())
|
|
||||||
}
|
|
||||||
|
|
||||||
fn encrypt_blob(key: &chacha20poly1305::Key, plaintext: &[u8]) -> Result<Vec<u8>, SymWireError> {
|
|
||||||
let cipher = XChaCha20Poly1305::new(key);
|
|
||||||
|
|
||||||
let mut nonce_bytes = [0u8; NONCE_LEN];
|
|
||||||
OsRng.fill_bytes(&mut nonce_bytes);
|
|
||||||
let nonce = XNonce::from_slice(&nonce_bytes);
|
|
||||||
|
|
||||||
let mut out = Vec::with_capacity(1 + NONCE_LEN + plaintext.len() + TAG_LEN);
|
|
||||||
out.push(VERSION);
|
|
||||||
out.extend_from_slice(&nonce_bytes);
|
|
||||||
|
|
||||||
let ct = cipher.encrypt(nonce, plaintext).map_err(|_| SymWireError::Crypto)?;
|
|
||||||
out.extend_from_slice(&ct);
|
|
||||||
Ok(out)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn decrypt_blob(key: &chacha20poly1305::Key, blob: &[u8]) -> Result<Vec<u8>, SymWireError> {
|
|
||||||
if blob.len() < 1 + NONCE_LEN + TAG_LEN {
|
|
||||||
return Err(SymWireError::BadFormat);
|
|
||||||
}
|
|
||||||
let ver = blob[0];
|
|
||||||
if ver != VERSION {
|
|
||||||
return Err(SymWireError::BadVersion(ver));
|
|
||||||
}
|
|
||||||
let nonce = XNonce::from_slice(&blob[1..1 + NONCE_LEN]);
|
|
||||||
let ct = &blob[1 + NONCE_LEN..];
|
|
||||||
|
|
||||||
let cipher = XChaCha20Poly1305::new(key);
|
|
||||||
cipher.decrypt(nonce, ct).map_err(|_| SymWireError::Crypto)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------- Command handlers (RESP) ----------
|
|
||||||
|
|
||||||
pub async fn cmd_sym_keygen() -> Protocol {
|
|
||||||
let mut key_bytes = [0u8; 32];
|
|
||||||
OsRng.fill_bytes(&mut key_bytes);
|
|
||||||
let key_b64 = B64.encode(key_bytes);
|
|
||||||
Protocol::BulkString(key_b64)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub async fn cmd_sym_encrypt(key_b64: &str, message: &str) -> Protocol {
|
|
||||||
let key = match decode_key_b64(key_b64) {
|
|
||||||
Ok(k) => k,
|
|
||||||
Err(e) => return e.to_protocol(),
|
|
||||||
};
|
|
||||||
match encrypt_blob(&key, message.as_bytes()) {
|
|
||||||
Ok(blob) => Protocol::BulkString(B64.encode(blob)),
|
|
||||||
Err(e) => e.to_protocol(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pub async fn cmd_sym_decrypt(key_b64: &str, ct_b64: &str) -> Protocol {
|
|
||||||
let key = match decode_key_b64(key_b64) {
|
|
||||||
Ok(k) => k,
|
|
||||||
Err(e) => return e.to_protocol(),
|
|
||||||
};
|
|
||||||
let blob = match B64.decode(ct_b64.as_bytes()) {
|
|
||||||
Ok(b) => b,
|
|
||||||
Err(_) => return SymWireError::BadEncoding.to_protocol(),
|
|
||||||
};
|
|
||||||
match decrypt_blob(&key, &blob) {
|
|
||||||
Ok(pt) => match String::from_utf8(pt) {
|
|
||||||
Ok(s) => Protocol::BulkString(s),
|
|
||||||
Err(_) => Protocol::err("ERR sym: invalid UTF-8 plaintext"),
|
|
||||||
},
|
|
||||||
Err(e) => e.to_protocol(),
|
|
||||||
}
|
|
||||||
}
|
|
@@ -7,13 +7,10 @@ use tantivy::{
|
|||||||
collector::TopDocs,
|
collector::TopDocs,
|
||||||
directory::MmapDirectory,
|
directory::MmapDirectory,
|
||||||
query::{BooleanQuery, Occur, Query, QueryParser, TermQuery},
|
query::{BooleanQuery, Occur, Query, QueryParser, TermQuery},
|
||||||
schema::{
|
schema::{Field, Schema, TextFieldIndexing, TextOptions, Value, STORED, STRING},
|
||||||
DateOptions, Field, IndexRecordOption, NumericOptions, Schema, TextFieldIndexing, TextOptions, STORED, STRING,
|
|
||||||
},
|
|
||||||
tokenizer::TokenizerManager,
|
tokenizer::TokenizerManager,
|
||||||
DateTime, Index, IndexReader, IndexWriter, TantivyDocument, Term,
|
DateTime, Index, IndexReader, IndexWriter, ReloadPolicy, TantivyDocument, Term,
|
||||||
};
|
};
|
||||||
use tantivy::schema::Value;
|
|
||||||
|
|
||||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
pub enum FieldDef {
|
pub enum FieldDef {
|
||||||
@@ -124,20 +121,25 @@ impl TantivySearch {
|
|||||||
fast: _fast,
|
fast: _fast,
|
||||||
} => {
|
} => {
|
||||||
let mut text_options = TextOptions::default();
|
let mut text_options = TextOptions::default();
|
||||||
|
|
||||||
if *stored {
|
if *stored {
|
||||||
text_options = text_options.set_stored();
|
text_options = text_options.set_stored();
|
||||||
}
|
}
|
||||||
|
|
||||||
if *indexed {
|
if *indexed {
|
||||||
let indexing_options = if *tokenized {
|
let indexing_options = if *tokenized {
|
||||||
TextFieldIndexing::default()
|
TextFieldIndexing::default()
|
||||||
.set_tokenizer("default")
|
.set_tokenizer("default")
|
||||||
.set_index_option(IndexRecordOption::WithFreqsAndPositions)
|
.set_index_option(
|
||||||
|
tantivy::schema::IndexRecordOption::WithFreqsAndPositions,
|
||||||
|
)
|
||||||
} else {
|
} else {
|
||||||
TextFieldIndexing::default()
|
TextFieldIndexing::default()
|
||||||
.set_tokenizer("raw")
|
.set_tokenizer("raw")
|
||||||
.set_index_option(IndexRecordOption::Basic)
|
.set_index_option(tantivy::schema::IndexRecordOption::Basic)
|
||||||
};
|
};
|
||||||
text_options = text_options.set_indexing_options(indexing_options);
|
text_options = text_options.set_indexing_options(indexing_options);
|
||||||
|
|
||||||
let f = schema_builder.add_text_field(&field_name, text_options);
|
let f = schema_builder.add_text_field(&field_name, text_options);
|
||||||
if *tokenized {
|
if *tokenized {
|
||||||
default_search_fields.push(f);
|
default_search_fields.push(f);
|
||||||
@@ -154,7 +156,7 @@ impl TantivySearch {
|
|||||||
precision,
|
precision,
|
||||||
} => match precision {
|
} => match precision {
|
||||||
NumericType::I64 => {
|
NumericType::I64 => {
|
||||||
let mut opts = NumericOptions::default();
|
let mut opts = tantivy::schema::NumericOptions::default();
|
||||||
if *stored {
|
if *stored {
|
||||||
opts = opts.set_stored();
|
opts = opts.set_stored();
|
||||||
}
|
}
|
||||||
@@ -167,7 +169,7 @@ impl TantivySearch {
|
|||||||
schema_builder.add_i64_field(&field_name, opts)
|
schema_builder.add_i64_field(&field_name, opts)
|
||||||
}
|
}
|
||||||
NumericType::U64 => {
|
NumericType::U64 => {
|
||||||
let mut opts = NumericOptions::default();
|
let mut opts = tantivy::schema::NumericOptions::default();
|
||||||
if *stored {
|
if *stored {
|
||||||
opts = opts.set_stored();
|
opts = opts.set_stored();
|
||||||
}
|
}
|
||||||
@@ -180,7 +182,7 @@ impl TantivySearch {
|
|||||||
schema_builder.add_u64_field(&field_name, opts)
|
schema_builder.add_u64_field(&field_name, opts)
|
||||||
}
|
}
|
||||||
NumericType::F64 => {
|
NumericType::F64 => {
|
||||||
let mut opts = NumericOptions::default();
|
let mut opts = tantivy::schema::NumericOptions::default();
|
||||||
if *stored {
|
if *stored {
|
||||||
opts = opts.set_stored();
|
opts = opts.set_stored();
|
||||||
}
|
}
|
||||||
@@ -193,7 +195,7 @@ impl TantivySearch {
|
|||||||
schema_builder.add_f64_field(&field_name, opts)
|
schema_builder.add_f64_field(&field_name, opts)
|
||||||
}
|
}
|
||||||
NumericType::Date => {
|
NumericType::Date => {
|
||||||
let mut opts = DateOptions::default();
|
let mut opts = tantivy::schema::DateOptions::default();
|
||||||
if *stored {
|
if *stored {
|
||||||
opts = opts.set_stored();
|
opts = opts.set_stored();
|
||||||
}
|
}
|
||||||
@@ -218,21 +220,23 @@ impl TantivySearch {
|
|||||||
text_options = text_options.set_indexing_options(
|
text_options = text_options.set_indexing_options(
|
||||||
TextFieldIndexing::default()
|
TextFieldIndexing::default()
|
||||||
.set_tokenizer("raw")
|
.set_tokenizer("raw")
|
||||||
.set_index_option(IndexRecordOption::Basic),
|
.set_index_option(tantivy::schema::IndexRecordOption::Basic),
|
||||||
);
|
);
|
||||||
schema_builder.add_text_field(&field_name, text_options)
|
schema_builder.add_text_field(&field_name, text_options)
|
||||||
}
|
}
|
||||||
FieldDef::Geo { stored } => {
|
FieldDef::Geo { stored } => {
|
||||||
// For now, store as two f64 fields for lat/lon
|
// For now, store as two f64 fields for lat/lon
|
||||||
let mut opts = NumericOptions::default();
|
let mut opts = tantivy::schema::NumericOptions::default();
|
||||||
if *stored {
|
if *stored {
|
||||||
opts = opts.set_stored();
|
opts = opts.set_stored();
|
||||||
}
|
}
|
||||||
opts = opts.set_indexed().set_fast();
|
opts = opts.set_indexed().set_fast();
|
||||||
|
|
||||||
let lat_field =
|
let lat_field =
|
||||||
schema_builder.add_f64_field(&format!("{}_lat", field_name), opts.clone());
|
schema_builder.add_f64_field(&format!("{}_lat", field_name), opts.clone());
|
||||||
let lon_field =
|
let lon_field =
|
||||||
schema_builder.add_f64_field(&format!("{}_lon", field_name), opts);
|
schema_builder.add_f64_field(&format!("{}_lon", field_name), opts);
|
||||||
|
|
||||||
fields.insert(
|
fields.insert(
|
||||||
format!("{}_lat", field_name),
|
format!("{}_lat", field_name),
|
||||||
(
|
(
|
||||||
@@ -260,6 +264,7 @@ impl TantivySearch {
|
|||||||
continue; // Skip adding the geo field itself
|
continue; // Skip adding the geo field itself
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
fields.insert(field_name.clone(), (field, field_def));
|
fields.insert(field_name.clone(), (field, field_def));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -273,8 +278,9 @@ impl TantivySearch {
|
|||||||
// Create or open index
|
// Create or open index
|
||||||
let dir = MmapDirectory::open(&index_path)
|
let dir = MmapDirectory::open(&index_path)
|
||||||
.map_err(|e| DBError(format!("Failed to open index directory: {}", e)))?;
|
.map_err(|e| DBError(format!("Failed to open index directory: {}", e)))?;
|
||||||
let mut index =
|
|
||||||
Index::open_or_create(dir, schema).map_err(|e| DBError(format!("Failed to create index: {}", e)))?;
|
let mut index = Index::open_or_create(dir, schema)
|
||||||
|
.map_err(|e| DBError(format!("Failed to create index: {}", e)))?;
|
||||||
|
|
||||||
// Configure tokenizers
|
// Configure tokenizers
|
||||||
let tokenizer_manager = TokenizerManager::default();
|
let tokenizer_manager = TokenizerManager::default();
|
||||||
@@ -283,8 +289,11 @@ impl TantivySearch {
|
|||||||
let writer = index
|
let writer = index
|
||||||
.writer(15_000_000)
|
.writer(15_000_000)
|
||||||
.map_err(|e| DBError(format!("Failed to create index writer: {}", e)))?;
|
.map_err(|e| DBError(format!("Failed to create index writer: {}", e)))?;
|
||||||
|
|
||||||
let reader = index
|
let reader = index
|
||||||
.reader()
|
.reader_builder()
|
||||||
|
.reload_policy(ReloadPolicy::OnCommitWithDelay)
|
||||||
|
.try_into()
|
||||||
.map_err(|e| DBError(format!("Failed to create reader: {}", e)))?;
|
.map_err(|e| DBError(format!("Failed to create reader: {}", e)))?;
|
||||||
|
|
||||||
let config = config.unwrap_or_default();
|
let config = config.unwrap_or_default();
|
||||||
@@ -361,11 +370,14 @@ impl TantivySearch {
|
|||||||
} else {
|
} else {
|
||||||
field_value.clone()
|
field_value.clone()
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// Store tags as separate terms for efficient filtering
|
||||||
for tag in tags.split(separator.as_str()) {
|
for tag in tags.split(separator.as_str()) {
|
||||||
doc.add_text(*field, tag.trim());
|
doc.add_text(*field, tag.trim());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
FieldDef::Geo { .. } => {
|
FieldDef::Geo { .. } => {
|
||||||
|
// Parse "lat,lon" format
|
||||||
let parts: Vec<&str> = field_value.split(',').collect();
|
let parts: Vec<&str> = field_value.split(',').collect();
|
||||||
if parts.len() == 2 {
|
if parts.len() == 2 {
|
||||||
if let (Ok(lat), Ok(lon)) =
|
if let (Ok(lat), Ok(lon)) =
|
||||||
@@ -391,13 +403,11 @@ impl TantivySearch {
|
|||||||
writer
|
writer
|
||||||
.add_document(doc)
|
.add_document(doc)
|
||||||
.map_err(|e| DBError(format!("Failed to add document: {}", e)))?;
|
.map_err(|e| DBError(format!("Failed to add document: {}", e)))?;
|
||||||
|
|
||||||
writer
|
writer
|
||||||
.commit()
|
.commit()
|
||||||
.map_err(|e| DBError(format!("Failed to commit: {}", e)))?;
|
.map_err(|e| DBError(format!("Failed to commit: {}", e)))?;
|
||||||
// Make new documents visible to searches
|
|
||||||
self.reader
|
|
||||||
.reload()
|
|
||||||
.map_err(|e| DBError(format!("Failed to reload reader: {}", e)))?;
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -406,87 +416,51 @@ impl TantivySearch {
|
|||||||
query_str: &str,
|
query_str: &str,
|
||||||
options: SearchOptions,
|
options: SearchOptions,
|
||||||
) -> Result<SearchResults, DBError> {
|
) -> Result<SearchResults, DBError> {
|
||||||
// Ensure reader is up to date with latest commits
|
|
||||||
self.reader
|
|
||||||
.reload()
|
|
||||||
.map_err(|e| DBError(format!("Failed to reload reader: {}", e)))?;
|
|
||||||
let searcher = self.reader.searcher();
|
let searcher = self.reader.searcher();
|
||||||
|
|
||||||
// Ensure we have searchable fields
|
|
||||||
if self.index_schema.default_search_fields.is_empty() {
|
|
||||||
return Err(DBError("No searchable fields defined in schema".to_string()));
|
|
||||||
}
|
|
||||||
|
|
||||||
// Parse query based on search fields
|
// Parse query based on search fields
|
||||||
|
let query: Box<dyn Query> = if self.index_schema.default_search_fields.is_empty() {
|
||||||
|
return Err(DBError(
|
||||||
|
"No searchable fields defined in schema".to_string(),
|
||||||
|
));
|
||||||
|
} else {
|
||||||
let query_parser = QueryParser::for_index(
|
let query_parser = QueryParser::for_index(
|
||||||
&self.index,
|
&self.index,
|
||||||
self.index_schema.default_search_fields.clone(),
|
self.index_schema.default_search_fields.clone(),
|
||||||
);
|
);
|
||||||
let parsed_query = query_parser
|
|
||||||
|
Box::new(
|
||||||
|
query_parser
|
||||||
.parse_query(query_str)
|
.parse_query(query_str)
|
||||||
.map_err(|e| DBError(format!("Failed to parse query: {}", e)))?;
|
.map_err(|e| DBError(format!("Failed to parse query: {}", e)))?,
|
||||||
let mut clauses: Vec<(Occur, Box<dyn Query>)> = vec![(Occur::Must, parsed_query)];
|
)
|
||||||
|
};
|
||||||
|
|
||||||
// Apply filters if any
|
// Apply filters if any
|
||||||
|
let final_query = if !options.filters.is_empty() {
|
||||||
|
let mut clauses: Vec<(Occur, Box<dyn Query>)> = vec![(Occur::Must, query)];
|
||||||
|
|
||||||
|
// Add filters
|
||||||
for filter in options.filters {
|
for filter in options.filters {
|
||||||
if let Some((field, field_def)) = self.index_schema.fields.get(&filter.field) {
|
if let Some((field, _)) = self.index_schema.fields.get(&filter.field) {
|
||||||
match filter.filter_type {
|
match filter.filter_type {
|
||||||
FilterType::Equals(value) => {
|
FilterType::Equals(value) => {
|
||||||
match field_def {
|
let term_query = TermQuery::new(
|
||||||
FieldDef::Text { .. } | FieldDef::Tag { .. } => {
|
Term::from_field_text(*field, &value),
|
||||||
let term_query =
|
tantivy::schema::IndexRecordOption::Basic,
|
||||||
TermQuery::new(Term::from_field_text(*field, &value), IndexRecordOption::Basic);
|
);
|
||||||
clauses.push((Occur::Must, Box::new(term_query)));
|
clauses.push((Occur::Must, Box::new(term_query)));
|
||||||
}
|
}
|
||||||
FieldDef::Numeric { precision, .. } => {
|
FilterType::Range { min: _, max: _ } => {
|
||||||
// Equals on numeric fields: parse to the right numeric type and use term query
|
// Would need numeric field handling here
|
||||||
match precision {
|
// Simplified for now
|
||||||
NumericType::I64 => {
|
|
||||||
if let Ok(v) = value.parse::<i64>() {
|
|
||||||
let term = Term::from_field_i64(*field, v);
|
|
||||||
let tq = TermQuery::new(term, IndexRecordOption::Basic);
|
|
||||||
clauses.push((Occur::Must, Box::new(tq)));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
NumericType::U64 => {
|
|
||||||
if let Ok(v) = value.parse::<u64>() {
|
|
||||||
let term = Term::from_field_u64(*field, v);
|
|
||||||
let tq = TermQuery::new(term, IndexRecordOption::Basic);
|
|
||||||
clauses.push((Occur::Must, Box::new(tq)));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
NumericType::F64 => {
|
|
||||||
if let Ok(v) = value.parse::<f64>() {
|
|
||||||
let term = Term::from_field_f64(*field, v);
|
|
||||||
let tq = TermQuery::new(term, IndexRecordOption::Basic);
|
|
||||||
clauses.push((Occur::Must, Box::new(tq)));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
NumericType::Date => {
|
|
||||||
if let Ok(v) = value.parse::<i64>() {
|
|
||||||
let dt = DateTime::from_timestamp_millis(v);
|
|
||||||
let term = Term::from_field_date(*field, dt);
|
|
||||||
let tq = TermQuery::new(term, IndexRecordOption::Basic);
|
|
||||||
clauses.push((Occur::Must, Box::new(tq)));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
FieldDef::Geo { .. } => {
|
|
||||||
// Geo equals isn't supported in this simplified version
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
FilterType::Range { .. } => {
|
|
||||||
// TODO: Implement numeric range queries by building a RangeQuery per type
|
|
||||||
}
|
}
|
||||||
FilterType::InSet(values) => {
|
FilterType::InSet(values) => {
|
||||||
// OR across values
|
|
||||||
let mut sub_clauses: Vec<(Occur, Box<dyn Query>)> = vec![];
|
let mut sub_clauses: Vec<(Occur, Box<dyn Query>)> = vec![];
|
||||||
for value in values {
|
for value in values {
|
||||||
let term_query = TermQuery::new(
|
let term_query = TermQuery::new(
|
||||||
Term::from_field_text(*field, &value),
|
Term::from_field_text(*field, &value),
|
||||||
IndexRecordOption::Basic,
|
tantivy::schema::IndexRecordOption::Basic,
|
||||||
);
|
);
|
||||||
sub_clauses.push((Occur::Should, Box::new(term_query)));
|
sub_clauses.push((Occur::Should, Box::new(term_query)));
|
||||||
}
|
}
|
||||||
@@ -496,27 +470,30 @@ impl TantivySearch {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let final_query: Box<dyn Query> = if clauses.len() == 1 {
|
|
||||||
clauses.pop().unwrap().1
|
|
||||||
} else {
|
|
||||||
Box::new(BooleanQuery::new(clauses))
|
Box::new(BooleanQuery::new(clauses))
|
||||||
|
} else {
|
||||||
|
query
|
||||||
};
|
};
|
||||||
|
|
||||||
// Execute search
|
// Execute search
|
||||||
let top_docs = searcher
|
let top_docs = searcher
|
||||||
.search(&*final_query, &TopDocs::with_limit(options.limit + options.offset))
|
.search(
|
||||||
|
&*final_query,
|
||||||
|
&TopDocs::with_limit(options.limit + options.offset),
|
||||||
|
)
|
||||||
.map_err(|e| DBError(format!("Search failed: {}", e)))?;
|
.map_err(|e| DBError(format!("Search failed: {}", e)))?;
|
||||||
|
|
||||||
let total_hits = top_docs.len();
|
let total_hits = top_docs.len();
|
||||||
let mut documents = Vec::new();
|
let mut documents = Vec::new();
|
||||||
|
|
||||||
for (score, doc_address) in top_docs.into_iter().skip(options.offset).take(options.limit) {
|
for (score, doc_address) in top_docs.iter().skip(options.offset).take(options.limit) {
|
||||||
let retrieved_doc: TantivyDocument = searcher
|
let retrieved_doc: TantivyDocument = searcher
|
||||||
.doc(doc_address)
|
.doc(*doc_address)
|
||||||
.map_err(|e| DBError(format!("Failed to retrieve doc: {}", e)))?;
|
.map_err(|e| DBError(format!("Failed to retrieve doc: {}", e)))?;
|
||||||
|
|
||||||
let mut doc_fields = HashMap::new();
|
let mut doc_fields = HashMap::new();
|
||||||
|
|
||||||
// Extract stored fields (or synthesize)
|
// Extract all stored fields
|
||||||
for (field_name, (field, field_def)) in &self.index_schema.fields {
|
for (field_name, (field, field_def)) in &self.index_schema.fields {
|
||||||
match field_def {
|
match field_def {
|
||||||
FieldDef::Text { stored, .. } | FieldDef::Tag { stored, .. } => {
|
FieldDef::Text { stored, .. } | FieldDef::Tag { stored, .. } => {
|
||||||
@@ -550,6 +527,7 @@ impl TantivySearch {
|
|||||||
.and_then(|v| v.as_datetime())
|
.and_then(|v| v.as_datetime())
|
||||||
.map(|v| v.into_timestamp_millis().to_string()),
|
.map(|v| v.into_timestamp_millis().to_string()),
|
||||||
};
|
};
|
||||||
|
|
||||||
if let Some(v) = value_str {
|
if let Some(v) = value_str {
|
||||||
doc_fields.insert(field_name.clone(), v);
|
doc_fields.insert(field_name.clone(), v);
|
||||||
}
|
}
|
||||||
@@ -569,8 +547,10 @@ impl TantivySearch {
|
|||||||
.get(&format!("{}_lon", field_name))
|
.get(&format!("{}_lon", field_name))
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.0;
|
.0;
|
||||||
|
|
||||||
let lat = retrieved_doc.get_first(lat_field).and_then(|v| v.as_f64());
|
let lat = retrieved_doc.get_first(lat_field).and_then(|v| v.as_f64());
|
||||||
let lon = retrieved_doc.get_first(lon_field).and_then(|v| v.as_f64());
|
let lon = retrieved_doc.get_first(lon_field).and_then(|v| v.as_f64());
|
||||||
|
|
||||||
if let (Some(lat), Some(lon)) = (lat, lon) {
|
if let (Some(lat), Some(lon)) = (lat, lon) {
|
||||||
doc_fields.insert(field_name.clone(), format!("{},{}", lat, lon));
|
doc_fields.insert(field_name.clone(), format!("{},{}", lat, lon));
|
||||||
}
|
}
|
||||||
@@ -581,7 +561,7 @@ impl TantivySearch {
|
|||||||
|
|
||||||
documents.push(SearchDocument {
|
documents.push(SearchDocument {
|
||||||
fields: doc_fields,
|
fields: doc_fields,
|
||||||
score,
|
score: *score,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -594,6 +574,7 @@ impl TantivySearch {
|
|||||||
pub fn get_info(&self) -> Result<IndexInfo, DBError> {
|
pub fn get_info(&self) -> Result<IndexInfo, DBError> {
|
||||||
let searcher = self.reader.searcher();
|
let searcher = self.reader.searcher();
|
||||||
let num_docs = searcher.num_docs();
|
let num_docs = searcher.num_docs();
|
||||||
|
|
||||||
let fields_info: Vec<FieldInfo> = self
|
let fields_info: Vec<FieldInfo> = self
|
||||||
.index_schema
|
.index_schema
|
||||||
.fields
|
.fields
|
||||||
@@ -603,6 +584,7 @@ impl TantivySearch {
|
|||||||
field_type: format!("{:?}", def),
|
field_type: format!("{:?}", def),
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
Ok(IndexInfo {
|
Ok(IndexInfo {
|
||||||
name: self.name.clone(),
|
name: self.name.clone(),
|
||||||
num_docs,
|
num_docs,
|
||||||
@@ -610,40 +592,6 @@ impl TantivySearch {
|
|||||||
config: self.config.clone(),
|
config: self.config.clone(),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Delete a document by its _id term. Returns true if the document existed before deletion.
|
|
||||||
pub fn delete_document_by_id(&self, doc_id: &str) -> Result<bool, DBError> {
|
|
||||||
// Determine existence by running a tiny term query
|
|
||||||
let existed = if let Some((id_field, _)) = self.index_schema.fields.get("_id") {
|
|
||||||
let term = Term::from_field_text(*id_field, doc_id);
|
|
||||||
let searcher = self.reader.searcher();
|
|
||||||
let tq = TermQuery::new(term.clone(), IndexRecordOption::Basic);
|
|
||||||
let hits = searcher
|
|
||||||
.search(&tq, &TopDocs::with_limit(1))
|
|
||||||
.map_err(|e| DBError(format!("Failed to search for existing doc: {}", e)))?;
|
|
||||||
!hits.is_empty()
|
|
||||||
} else {
|
|
||||||
false
|
|
||||||
};
|
|
||||||
|
|
||||||
// Perform deletion and commit
|
|
||||||
let mut writer = self
|
|
||||||
.writer
|
|
||||||
.write()
|
|
||||||
.map_err(|e| DBError(format!("Failed to acquire writer lock: {}", e)))?;
|
|
||||||
if let Some((id_field, _)) = self.index_schema.fields.get("_id") {
|
|
||||||
writer.delete_term(Term::from_field_text(*id_field, doc_id));
|
|
||||||
}
|
|
||||||
writer
|
|
||||||
.commit()
|
|
||||||
.map_err(|e| DBError(format!("Failed to commit delete: {}", e)))?;
|
|
||||||
// Refresh reader to observe deletion
|
|
||||||
self.reader
|
|
||||||
.reload()
|
|
||||||
.map_err(|e| DBError(format!("Failed to reload reader: {}", e)))?;
|
|
||||||
|
|
||||||
Ok(existed)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
|
@@ -1,11 +1,10 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
set -euo pipefail
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
cd "$SCRIPT_DIR"
|
|
||||||
|
|
||||||
# Test script for HeroDB - Redis-compatible database with redb backend
|
# Test script for HeroDB - Redis-compatible database with redb backend
|
||||||
# This script starts the server and runs comprehensive tests
|
# This script starts the server and runs comprehensive tests
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
# Colors for output
|
# Colors for output
|
||||||
RED='\033[0;31m'
|
RED='\033[0;31m'
|
||||||
GREEN='\033[0;32m'
|
GREEN='\033[0;32m'
|
||||||
|
@@ -1,4 +1,4 @@
|
|||||||
use herodb::{server::Server, options::DBOption};
|
use herodb::{options::DBOption, server::Server};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||||
use tokio::net::TcpStream;
|
use tokio::net::TcpStream;
|
||||||
@@ -28,7 +28,6 @@ async fn debug_hset_simple() {
|
|||||||
encrypt: false,
|
encrypt: false,
|
||||||
encryption_key: None,
|
encryption_key: None,
|
||||||
backend: herodb::options::BackendType::Redb,
|
backend: herodb::options::BackendType::Redb,
|
||||||
admin_secret: "test-admin".to_string(),
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut server = Server::new(option).await;
|
let mut server = Server::new(option).await;
|
||||||
@@ -48,23 +47,31 @@ async fn debug_hset_simple() {
|
|||||||
|
|
||||||
sleep(Duration::from_millis(200)).await;
|
sleep(Duration::from_millis(200)).await;
|
||||||
|
|
||||||
let mut stream = TcpStream::connect(format!("127.0.0.1:{}", port)).await.unwrap();
|
let mut stream = TcpStream::connect(format!("127.0.0.1:{}", port))
|
||||||
// Acquire ReadWrite permissions on this connection
|
.await
|
||||||
let resp = send_command(
|
.unwrap();
|
||||||
&mut stream,
|
|
||||||
"*4\r\n$6\r\nSELECT\r\n$1\r\n0\r\n$3\r\nKEY\r\n$10\r\ntest-admin\r\n",
|
|
||||||
).await;
|
|
||||||
assert!(resp.contains("OK"), "Failed SELECT handshake: {}", resp);
|
|
||||||
|
|
||||||
// Test simple HSET
|
// Test simple HSET
|
||||||
println!("Testing HSET...");
|
println!("Testing HSET...");
|
||||||
let response = send_command(&mut stream, "*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
println!("HSET response: {}", response);
|
println!("HSET response: {}", response);
|
||||||
assert!(response.contains("1"), "Expected '1' but got: {}", response);
|
assert!(response.contains("1"), "Expected '1' but got: {}", response);
|
||||||
|
|
||||||
// Test HGET
|
// Test HGET
|
||||||
println!("Testing HGET...");
|
println!("Testing HGET...");
|
||||||
let response = send_command(&mut stream, "*3\r\n$4\r\nHGET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$4\r\nHGET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
println!("HGET response: {}", response);
|
println!("HGET response: {}", response);
|
||||||
assert!(response.contains("value1"), "Expected 'value1' but got: {}", response);
|
assert!(
|
||||||
|
response.contains("value1"),
|
||||||
|
"Expected 'value1' but got: {}",
|
||||||
|
response
|
||||||
|
);
|
||||||
}
|
}
|
@@ -1,4 +1,4 @@
|
|||||||
use herodb::{server::Server, options::DBOption};
|
use herodb::{options::DBOption, server::Server};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||||
use tokio::net::TcpStream;
|
use tokio::net::TcpStream;
|
||||||
@@ -19,7 +19,6 @@ async fn debug_hset_return_value() {
|
|||||||
encrypt: false,
|
encrypt: false,
|
||||||
encryption_key: None,
|
encryption_key: None,
|
||||||
backend: herodb::options::BackendType::Redb,
|
backend: herodb::options::BackendType::Redb,
|
||||||
admin_secret: "test-admin".to_string(),
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let mut server = Server::new(option).await;
|
let mut server = Server::new(option).await;
|
||||||
@@ -42,18 +41,11 @@ async fn debug_hset_return_value() {
|
|||||||
// Connect and test HSET
|
// Connect and test HSET
|
||||||
let mut stream = TcpStream::connect("127.0.0.1:16390").await.unwrap();
|
let mut stream = TcpStream::connect("127.0.0.1:16390").await.unwrap();
|
||||||
|
|
||||||
// Acquire ReadWrite permissions for this new connection
|
|
||||||
let handshake = "*4\r\n$6\r\nSELECT\r\n$1\r\n0\r\n$3\r\nKEY\r\n$10\r\ntest-admin\r\n";
|
|
||||||
stream.write_all(handshake.as_bytes()).await.unwrap();
|
|
||||||
let mut buffer = [0; 1024];
|
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
|
||||||
let resp = String::from_utf8_lossy(&buffer[..n]);
|
|
||||||
assert!(resp.contains("OK"), "Failed SELECT handshake: {}", resp);
|
|
||||||
|
|
||||||
// Send HSET command
|
// Send HSET command
|
||||||
let cmd = "*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n";
|
let cmd = "*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n";
|
||||||
stream.write_all(cmd.as_bytes()).await.unwrap();
|
stream.write_all(cmd.as_bytes()).await.unwrap();
|
||||||
|
|
||||||
|
let mut buffer = [0; 1024];
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
let response = String::from_utf8_lossy(&buffer[..n]);
|
let response = String::from_utf8_lossy(&buffer[..n]);
|
||||||
|
|
||||||
@@ -61,5 +53,9 @@ async fn debug_hset_return_value() {
|
|||||||
println!("Response bytes: {:?}", &buffer[..n]);
|
println!("Response bytes: {:?}", &buffer[..n]);
|
||||||
|
|
||||||
// Check if response contains "1"
|
// Check if response contains "1"
|
||||||
assert!(response.contains("1"), "Expected response to contain '1', got: {}", response);
|
assert!(
|
||||||
|
response.contains("1"),
|
||||||
|
"Expected response to contain '1', got: {}",
|
||||||
|
response
|
||||||
|
);
|
||||||
}
|
}
|
@@ -1,11 +1,14 @@
|
|||||||
use herodb::protocol::Protocol;
|
|
||||||
use herodb::cmd::Cmd;
|
use herodb::cmd::Cmd;
|
||||||
|
use herodb::protocol::Protocol;
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_protocol_parsing() {
|
fn test_protocol_parsing() {
|
||||||
// Test TYPE command parsing
|
// Test TYPE command parsing
|
||||||
let type_cmd = "*2\r\n$4\r\nTYPE\r\n$7\r\nnoexist\r\n";
|
let type_cmd = "*2\r\n$4\r\nTYPE\r\n$7\r\nnoexist\r\n";
|
||||||
println!("Parsing TYPE command: {}", type_cmd.replace("\r\n", "\\r\\n"));
|
println!(
|
||||||
|
"Parsing TYPE command: {}",
|
||||||
|
type_cmd.replace("\r\n", "\\r\\n")
|
||||||
|
);
|
||||||
|
|
||||||
match Protocol::from(type_cmd) {
|
match Protocol::from(type_cmd) {
|
||||||
Ok((protocol, _)) => {
|
Ok((protocol, _)) => {
|
||||||
@@ -20,7 +23,10 @@ fn test_protocol_parsing() {
|
|||||||
|
|
||||||
// Test HEXISTS command parsing
|
// Test HEXISTS command parsing
|
||||||
let hexists_cmd = "*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$7\r\nnoexist\r\n";
|
let hexists_cmd = "*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$7\r\nnoexist\r\n";
|
||||||
println!("\nParsing HEXISTS command: {}", hexists_cmd.replace("\r\n", "\\r\\n"));
|
println!(
|
||||||
|
"\nParsing HEXISTS command: {}",
|
||||||
|
hexists_cmd.replace("\r\n", "\\r\\n")
|
||||||
|
);
|
||||||
|
|
||||||
match Protocol::from(hexists_cmd) {
|
match Protocol::from(hexists_cmd) {
|
||||||
Ok((protocol, _)) => {
|
Ok((protocol, _)) => {
|
||||||
|
@@ -12,17 +12,9 @@ fn get_redis_connection(port: u16) -> Connection {
|
|||||||
match client.get_connection() {
|
match client.get_connection() {
|
||||||
Ok(mut conn) => {
|
Ok(mut conn) => {
|
||||||
if redis::cmd("PING").query::<String>(&mut conn).is_ok() {
|
if redis::cmd("PING").query::<String>(&mut conn).is_ok() {
|
||||||
// Acquire ReadWrite permissions on this connection
|
|
||||||
let sel: RedisResult<String> = redis::cmd("SELECT")
|
|
||||||
.arg(0)
|
|
||||||
.arg("KEY")
|
|
||||||
.arg("test-admin")
|
|
||||||
.query(&mut conn);
|
|
||||||
if sel.is_ok() {
|
|
||||||
return conn;
|
return conn;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
if attempts >= 120 {
|
if attempts >= 120 {
|
||||||
panic!(
|
panic!(
|
||||||
@@ -86,8 +78,6 @@ fn setup_server() -> (ServerProcessGuard, u16) {
|
|||||||
"--port",
|
"--port",
|
||||||
&port.to_string(),
|
&port.to_string(),
|
||||||
"--debug",
|
"--debug",
|
||||||
"--admin-secret",
|
|
||||||
"test-admin",
|
|
||||||
])
|
])
|
||||||
.spawn()
|
.spawn()
|
||||||
.expect("Failed to start server process");
|
.expect("Failed to start server process");
|
||||||
@@ -216,7 +206,9 @@ async fn test_expiration(conn: &mut Connection) {
|
|||||||
async fn test_scan_operations(conn: &mut Connection) {
|
async fn test_scan_operations(conn: &mut Connection) {
|
||||||
cleanup_keys(conn).await;
|
cleanup_keys(conn).await;
|
||||||
for i in 0..5 {
|
for i in 0..5 {
|
||||||
let _: () = conn.set(format!("key{}", i), format!("value{}", i)).unwrap();
|
let _: () = conn
|
||||||
|
.set(format!("key{}", i), format!("value{}", i))
|
||||||
|
.unwrap();
|
||||||
}
|
}
|
||||||
let result: (u64, Vec<String>) = redis::cmd("SCAN")
|
let result: (u64, Vec<String>) = redis::cmd("SCAN")
|
||||||
.arg(0)
|
.arg(0)
|
||||||
@@ -263,7 +255,9 @@ async fn test_scan_with_count(conn: &mut Connection) {
|
|||||||
async fn test_hscan_operations(conn: &mut Connection) {
|
async fn test_hscan_operations(conn: &mut Connection) {
|
||||||
cleanup_keys(conn).await;
|
cleanup_keys(conn).await;
|
||||||
for i in 0..3 {
|
for i in 0..3 {
|
||||||
let _: () = conn.hset("testhash", format!("field{}", i), format!("value{}", i)).unwrap();
|
let _: () = conn
|
||||||
|
.hset("testhash", format!("field{}", i), format!("value{}", i))
|
||||||
|
.unwrap();
|
||||||
}
|
}
|
||||||
let result: (u64, Vec<String>) = redis::cmd("HSCAN")
|
let result: (u64, Vec<String>) = redis::cmd("HSCAN")
|
||||||
.arg("testhash")
|
.arg("testhash")
|
||||||
@@ -283,8 +277,16 @@ async fn test_hscan_operations(conn: &mut Connection) {
|
|||||||
async fn test_transaction_operations(conn: &mut Connection) {
|
async fn test_transaction_operations(conn: &mut Connection) {
|
||||||
cleanup_keys(conn).await;
|
cleanup_keys(conn).await;
|
||||||
let _: () = redis::cmd("MULTI").query(conn).unwrap();
|
let _: () = redis::cmd("MULTI").query(conn).unwrap();
|
||||||
let _: () = redis::cmd("SET").arg("key1").arg("value1").query(conn).unwrap();
|
let _: () = redis::cmd("SET")
|
||||||
let _: () = redis::cmd("SET").arg("key2").arg("value2").query(conn).unwrap();
|
.arg("key1")
|
||||||
|
.arg("value1")
|
||||||
|
.query(conn)
|
||||||
|
.unwrap();
|
||||||
|
let _: () = redis::cmd("SET")
|
||||||
|
.arg("key2")
|
||||||
|
.arg("value2")
|
||||||
|
.query(conn)
|
||||||
|
.unwrap();
|
||||||
let _: Vec<String> = redis::cmd("EXEC").query(conn).unwrap();
|
let _: Vec<String> = redis::cmd("EXEC").query(conn).unwrap();
|
||||||
let result: String = conn.get("key1").unwrap();
|
let result: String = conn.get("key1").unwrap();
|
||||||
assert_eq!(result, "value1");
|
assert_eq!(result, "value1");
|
||||||
@@ -296,7 +298,11 @@ async fn test_transaction_operations(conn: &mut Connection) {
|
|||||||
async fn test_discard_transaction(conn: &mut Connection) {
|
async fn test_discard_transaction(conn: &mut Connection) {
|
||||||
cleanup_keys(conn).await;
|
cleanup_keys(conn).await;
|
||||||
let _: () = redis::cmd("MULTI").query(conn).unwrap();
|
let _: () = redis::cmd("MULTI").query(conn).unwrap();
|
||||||
let _: () = redis::cmd("SET").arg("discard").arg("value").query(conn).unwrap();
|
let _: () = redis::cmd("SET")
|
||||||
|
.arg("discard")
|
||||||
|
.arg("value")
|
||||||
|
.query(conn)
|
||||||
|
.unwrap();
|
||||||
let _: () = redis::cmd("DISCARD").query(conn).unwrap();
|
let _: () = redis::cmd("DISCARD").query(conn).unwrap();
|
||||||
let result: Option<String> = conn.get("discard").unwrap();
|
let result: Option<String> = conn.get("discard").unwrap();
|
||||||
assert_eq!(result, None);
|
assert_eq!(result, None);
|
||||||
@@ -316,7 +322,6 @@ async fn test_type_command(conn: &mut Connection) {
|
|||||||
cleanup_keys(conn).await;
|
cleanup_keys(conn).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
async fn test_info_command(conn: &mut Connection) {
|
async fn test_info_command(conn: &mut Connection) {
|
||||||
cleanup_keys(conn).await;
|
cleanup_keys(conn).await;
|
||||||
let result: String = redis::cmd("INFO").query(conn).unwrap();
|
let result: String = redis::cmd("INFO").query(conn).unwrap();
|
||||||
|
@@ -1,4 +1,4 @@
|
|||||||
use herodb::{server::Server, options::DBOption};
|
use herodb::{options::DBOption, server::Server};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||||
use tokio::net::TcpStream;
|
use tokio::net::TcpStream;
|
||||||
@@ -23,29 +23,18 @@ async fn start_test_server(test_name: &str) -> (Server, u16) {
|
|||||||
encrypt: false,
|
encrypt: false,
|
||||||
encryption_key: None,
|
encryption_key: None,
|
||||||
backend: herodb::options::BackendType::Redb,
|
backend: herodb::options::BackendType::Redb,
|
||||||
admin_secret: "test-admin".to_string(),
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let server = Server::new(option).await;
|
let server = Server::new(option).await;
|
||||||
(server, port)
|
(server, port)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Helper function to connect to the test server
|
// Helper function to connect to the test server
|
||||||
async fn connect_to_server(port: u16) -> TcpStream {
|
async fn connect_to_server(port: u16) -> TcpStream {
|
||||||
let mut attempts = 0;
|
let mut attempts = 0;
|
||||||
loop {
|
loop {
|
||||||
match TcpStream::connect(format!("127.0.0.1:{}", port)).await {
|
match TcpStream::connect(format!("127.0.0.1:{}", port)).await {
|
||||||
Ok(mut stream) => {
|
Ok(stream) => return stream,
|
||||||
// Obtain ReadWrite permissions for this connection by selecting DB 0 with admin key
|
|
||||||
let resp = send_command(
|
|
||||||
&mut stream,
|
|
||||||
"*4\r\n$6\r\nSELECT\r\n$1\r\n0\r\n$3\r\nKEY\r\n$10\r\ntest-admin\r\n",
|
|
||||||
).await;
|
|
||||||
if !resp.contains("OK") {
|
|
||||||
panic!("Failed to acquire write permissions via SELECT 0 KEY test-admin: {}", resp);
|
|
||||||
}
|
|
||||||
return stream;
|
|
||||||
}
|
|
||||||
Err(_) if attempts < 10 => {
|
Err(_) if attempts < 10 => {
|
||||||
attempts += 1;
|
attempts += 1;
|
||||||
sleep(Duration::from_millis(100)).await;
|
sleep(Duration::from_millis(100)).await;
|
||||||
@@ -110,7 +99,11 @@ async fn test_string_operations() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Test SET
|
// Test SET
|
||||||
let response = send_command(&mut stream, "*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("OK"));
|
assert!(response.contains("OK"));
|
||||||
|
|
||||||
// Test GET
|
// Test GET
|
||||||
@@ -159,7 +152,11 @@ async fn test_incr_operations() {
|
|||||||
assert!(response.contains("2"));
|
assert!(response.contains("2"));
|
||||||
|
|
||||||
// Test INCR on string value (should fail)
|
// Test INCR on string value (should fail)
|
||||||
send_command(&mut stream, "*3\r\n$3\r\nSET\r\n$6\r\nstring\r\n$5\r\nhello\r\n").await;
|
send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$3\r\nSET\r\n$6\r\nstring\r\n$5\r\nhello\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
let response = send_command(&mut stream, "*2\r\n$4\r\nINCR\r\n$6\r\nstring\r\n").await;
|
let response = send_command(&mut stream, "*2\r\n$4\r\nINCR\r\n$6\r\nstring\r\n").await;
|
||||||
assert!(response.contains("ERR"));
|
assert!(response.contains("ERR"));
|
||||||
}
|
}
|
||||||
@@ -185,11 +182,19 @@ async fn test_hash_operations() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Test HSET
|
// Test HSET
|
||||||
let response = send_command(&mut stream, "*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("1")); // 1 new field
|
assert!(response.contains("1")); // 1 new field
|
||||||
|
|
||||||
// Test HGET
|
// Test HGET
|
||||||
let response = send_command(&mut stream, "*3\r\n$4\r\nHGET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$4\r\nHGET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("value1"));
|
assert!(response.contains("value1"));
|
||||||
|
|
||||||
// Test HSET multiple fields
|
// Test HSET multiple fields
|
||||||
@@ -208,14 +213,26 @@ async fn test_hash_operations() {
|
|||||||
assert!(response.contains("3"));
|
assert!(response.contains("3"));
|
||||||
|
|
||||||
// Test HEXISTS
|
// Test HEXISTS
|
||||||
let response = send_command(&mut stream, "*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$6\r\nfield1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$6\r\nfield1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("1"));
|
assert!(response.contains("1"));
|
||||||
|
|
||||||
let response = send_command(&mut stream, "*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$7\r\nnoexist\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$7\r\nnoexist\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("0"));
|
assert!(response.contains("0"));
|
||||||
|
|
||||||
// Test HDEL
|
// Test HDEL
|
||||||
let response = send_command(&mut stream, "*3\r\n$4\r\nHDEL\r\n$4\r\nhash\r\n$6\r\nfield1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$4\r\nHDEL\r\n$4\r\nhash\r\n$6\r\nfield1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("1"));
|
assert!(response.contains("1"));
|
||||||
|
|
||||||
// Test HKEYS
|
// Test HKEYS
|
||||||
@@ -251,7 +268,11 @@ async fn test_expiration() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Test SETEX (expire in 1 second)
|
// Test SETEX (expire in 1 second)
|
||||||
let response = send_command(&mut stream, "*5\r\n$3\r\nSET\r\n$6\r\nexpkey\r\n$5\r\nvalue\r\n$2\r\nEX\r\n$1\r\n1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*5\r\n$3\r\nSET\r\n$6\r\nexpkey\r\n$5\r\nvalue\r\n$2\r\nEX\r\n$1\r\n1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("OK"));
|
assert!(response.contains("OK"));
|
||||||
|
|
||||||
// Test TTL
|
// Test TTL
|
||||||
@@ -305,7 +326,11 @@ async fn test_scan_operations() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Test SCAN
|
// Test SCAN
|
||||||
let response = send_command(&mut stream, "*6\r\n$4\r\nSCAN\r\n$1\r\n0\r\n$5\r\nMATCH\r\n$1\r\n*\r\n$5\r\nCOUNT\r\n$2\r\n10\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*6\r\n$4\r\nSCAN\r\n$1\r\n0\r\n$5\r\nMATCH\r\n$1\r\n*\r\n$5\r\nCOUNT\r\n$2\r\n10\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("key"));
|
assert!(response.contains("key"));
|
||||||
|
|
||||||
// Test KEYS
|
// Test KEYS
|
||||||
@@ -336,7 +361,10 @@ async fn test_hscan_operations() {
|
|||||||
|
|
||||||
// Set up hash data
|
// Set up hash data
|
||||||
for i in 0..3 {
|
for i in 0..3 {
|
||||||
let cmd = format!("*4\r\n$4\r\nHSET\r\n$8\r\ntesthash\r\n$6\r\nfield{}\r\n$6\r\nvalue{}\r\n", i, i);
|
let cmd = format!(
|
||||||
|
"*4\r\n$4\r\nHSET\r\n$8\r\ntesthash\r\n$6\r\nfield{}\r\n$6\r\nvalue{}\r\n",
|
||||||
|
i, i
|
||||||
|
);
|
||||||
send_command(&mut stream, &cmd).await;
|
send_command(&mut stream, &cmd).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -371,10 +399,18 @@ async fn test_transaction_operations() {
|
|||||||
assert!(response.contains("OK"));
|
assert!(response.contains("OK"));
|
||||||
|
|
||||||
// Test queued commands
|
// Test queued commands
|
||||||
let response = send_command(&mut stream, "*3\r\n$3\r\nSET\r\n$4\r\nkey1\r\n$6\r\nvalue1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$3\r\nSET\r\n$4\r\nkey1\r\n$6\r\nvalue1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("QUEUED"));
|
assert!(response.contains("QUEUED"));
|
||||||
|
|
||||||
let response = send_command(&mut stream, "*3\r\n$3\r\nSET\r\n$4\r\nkey2\r\n$6\r\nvalue2\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$3\r\nSET\r\n$4\r\nkey2\r\n$6\r\nvalue2\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("QUEUED"));
|
assert!(response.contains("QUEUED"));
|
||||||
|
|
||||||
// Test EXEC
|
// Test EXEC
|
||||||
@@ -414,7 +450,11 @@ async fn test_discard_transaction() {
|
|||||||
assert!(response.contains("OK"));
|
assert!(response.contains("OK"));
|
||||||
|
|
||||||
// Test queued command
|
// Test queued command
|
||||||
let response = send_command(&mut stream, "*3\r\n$3\r\nSET\r\n$7\r\ndiscard\r\n$5\r\nvalue\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$3\r\nSET\r\n$7\r\ndiscard\r\n$5\r\nvalue\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("QUEUED"));
|
assert!(response.contains("QUEUED"));
|
||||||
|
|
||||||
// Test DISCARD
|
// Test DISCARD
|
||||||
@@ -447,12 +487,20 @@ async fn test_type_command() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Test string type
|
// Test string type
|
||||||
send_command(&mut stream, "*3\r\n$3\r\nSET\r\n$6\r\nstring\r\n$5\r\nvalue\r\n").await;
|
send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$3\r\nSET\r\n$6\r\nstring\r\n$5\r\nvalue\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$6\r\nstring\r\n").await;
|
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$6\r\nstring\r\n").await;
|
||||||
assert!(response.contains("string"));
|
assert!(response.contains("string"));
|
||||||
|
|
||||||
// Test hash type
|
// Test hash type
|
||||||
send_command(&mut stream, "*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$5\r\nfield\r\n$5\r\nvalue\r\n").await;
|
send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$5\r\nfield\r\n$5\r\nvalue\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$4\r\nhash\r\n").await;
|
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$4\r\nhash\r\n").await;
|
||||||
assert!(response.contains("hash"));
|
assert!(response.contains("hash"));
|
||||||
|
|
||||||
@@ -482,12 +530,20 @@ async fn test_config_commands() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Test CONFIG GET databases
|
// Test CONFIG GET databases
|
||||||
let response = send_command(&mut stream, "*3\r\n$6\r\nCONFIG\r\n$3\r\nGET\r\n$9\r\ndatabases\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$6\r\nCONFIG\r\n$3\r\nGET\r\n$9\r\ndatabases\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("databases"));
|
assert!(response.contains("databases"));
|
||||||
assert!(response.contains("16"));
|
assert!(response.contains("16"));
|
||||||
|
|
||||||
// Test CONFIG GET dir
|
// Test CONFIG GET dir
|
||||||
let response = send_command(&mut stream, "*3\r\n$6\r\nCONFIG\r\n$3\r\nGET\r\n$3\r\ndir\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$6\r\nCONFIG\r\n$3\r\nGET\r\n$3\r\ndir\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("dir"));
|
assert!(response.contains("dir"));
|
||||||
assert!(response.contains("/tmp/herodb_test_config"));
|
assert!(response.contains("/tmp/herodb_test_config"));
|
||||||
}
|
}
|
||||||
@@ -542,8 +598,16 @@ async fn test_error_handling() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Test WRONGTYPE error - try to use hash command on string
|
// Test WRONGTYPE error - try to use hash command on string
|
||||||
send_command(&mut stream, "*3\r\n$3\r\nSET\r\n$6\r\nstring\r\n$5\r\nvalue\r\n").await;
|
send_command(
|
||||||
let response = send_command(&mut stream, "*3\r\n$4\r\nHGET\r\n$6\r\nstring\r\n$5\r\nfield\r\n").await;
|
&mut stream,
|
||||||
|
"*3\r\n$3\r\nSET\r\n$6\r\nstring\r\n$5\r\nvalue\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$4\r\nHGET\r\n$6\r\nstring\r\n$5\r\nfield\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("WRONGTYPE"));
|
assert!(response.contains("WRONGTYPE"));
|
||||||
|
|
||||||
// Test unknown command
|
// Test unknown command
|
||||||
@@ -580,11 +644,19 @@ async fn test_list_operations() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Test LPUSH
|
// Test LPUSH
|
||||||
let response = send_command(&mut stream, "*4\r\n$5\r\nLPUSH\r\n$4\r\nlist\r\n$1\r\na\r\n$1\r\nb\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$5\r\nLPUSH\r\n$4\r\nlist\r\n$1\r\na\r\n$1\r\nb\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("2")); // 2 elements
|
assert!(response.contains("2")); // 2 elements
|
||||||
|
|
||||||
// Test RPUSH
|
// Test RPUSH
|
||||||
let response = send_command(&mut stream, "*4\r\n$5\r\nRPUSH\r\n$4\r\nlist\r\n$1\r\nc\r\n$1\r\nd\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$5\r\nRPUSH\r\n$4\r\nlist\r\n$1\r\nc\r\n$1\r\nd\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("4")); // 4 elements
|
assert!(response.contains("4")); // 4 elements
|
||||||
|
|
||||||
// Test LLEN
|
// Test LLEN
|
||||||
@@ -592,11 +664,22 @@ async fn test_list_operations() {
|
|||||||
assert!(response.contains("4"));
|
assert!(response.contains("4"));
|
||||||
|
|
||||||
// Test LRANGE
|
// Test LRANGE
|
||||||
let response = send_command(&mut stream, "*4\r\n$6\r\nLRANGE\r\n$4\r\nlist\r\n$1\r\n0\r\n$2\r\n-1\r\n").await;
|
let response = send_command(
|
||||||
assert_eq!(response, "*4\r\n$1\r\nb\r\n$1\r\na\r\n$1\r\nc\r\n$1\r\nd\r\n");
|
&mut stream,
|
||||||
|
"*4\r\n$6\r\nLRANGE\r\n$4\r\nlist\r\n$1\r\n0\r\n$2\r\n-1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert_eq!(
|
||||||
|
response,
|
||||||
|
"*4\r\n$1\r\nb\r\n$1\r\na\r\n$1\r\nc\r\n$1\r\nd\r\n"
|
||||||
|
);
|
||||||
|
|
||||||
// Test LINDEX
|
// Test LINDEX
|
||||||
let response = send_command(&mut stream, "*3\r\n$6\r\nLINDEX\r\n$4\r\nlist\r\n$1\r\n0\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$6\r\nLINDEX\r\n$4\r\nlist\r\n$1\r\n0\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert_eq!(response, "$1\r\nb\r\n");
|
assert_eq!(response, "$1\r\nb\r\n");
|
||||||
|
|
||||||
// Test LPOP
|
// Test LPOP
|
||||||
@@ -608,12 +691,24 @@ async fn test_list_operations() {
|
|||||||
assert_eq!(response, "$1\r\nd\r\n");
|
assert_eq!(response, "$1\r\nd\r\n");
|
||||||
|
|
||||||
// Test LREM
|
// Test LREM
|
||||||
send_command(&mut stream, "*3\r\n$5\r\nLPUSH\r\n$4\r\nlist\r\n$1\r\na\r\n").await; // list is now a, c, a
|
send_command(
|
||||||
let response = send_command(&mut stream, "*4\r\n$4\r\nLREM\r\n$4\r\nlist\r\n$1\r\n1\r\n$1\r\na\r\n").await;
|
&mut stream,
|
||||||
|
"*3\r\n$5\r\nLPUSH\r\n$4\r\nlist\r\n$1\r\na\r\n",
|
||||||
|
)
|
||||||
|
.await; // list is now a, c, a
|
||||||
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$4\r\nLREM\r\n$4\r\nlist\r\n$1\r\n1\r\n$1\r\na\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("1"));
|
assert!(response.contains("1"));
|
||||||
|
|
||||||
// Test LTRIM
|
// Test LTRIM
|
||||||
let response = send_command(&mut stream, "*4\r\n$5\r\nLTRIM\r\n$4\r\nlist\r\n$1\r\n0\r\n$1\r\n0\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$5\r\nLTRIM\r\n$4\r\nlist\r\n$1\r\n0\r\n$1\r\n0\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("OK"));
|
assert!(response.contains("OK"));
|
||||||
let response = send_command(&mut stream, "*2\r\n$4\r\nLLEN\r\n$4\r\nlist\r\n").await;
|
let response = send_command(&mut stream, "*2\r\n$4\r\nLLEN\r\n$4\r\nlist\r\n").await;
|
||||||
assert!(response.contains("1"));
|
assert!(response.contains("1"));
|
||||||
|
@@ -1,85 +0,0 @@
|
|||||||
use herodb::rpc::{BackendType, DatabaseConfig};
|
|
||||||
use herodb::admin_meta;
|
|
||||||
use herodb::options::BackendType as OptionsBackendType;
|
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn test_rpc_server_basic() {
|
|
||||||
// This test would require starting the RPC server in a separate thread
|
|
||||||
// For now, we'll just test that the types compile correctly
|
|
||||||
|
|
||||||
// Test serialization of types
|
|
||||||
let backend = BackendType::Redb;
|
|
||||||
let config = DatabaseConfig {
|
|
||||||
name: Some("test_db".to_string()),
|
|
||||||
storage_path: Some("/tmp/test".to_string()),
|
|
||||||
max_size: Some(1024 * 1024),
|
|
||||||
redis_version: Some("7.0".to_string()),
|
|
||||||
};
|
|
||||||
|
|
||||||
let backend_json = serde_json::to_string(&backend).unwrap();
|
|
||||||
let config_json = serde_json::to_string(&config).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(backend_json, "\"Redb\"");
|
|
||||||
assert!(config_json.contains("test_db"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn test_database_config_serialization() {
|
|
||||||
let config = DatabaseConfig {
|
|
||||||
name: Some("my_db".to_string()),
|
|
||||||
storage_path: None,
|
|
||||||
max_size: Some(1000000),
|
|
||||||
redis_version: Some("7.0".to_string()),
|
|
||||||
};
|
|
||||||
|
|
||||||
let json = serde_json::to_value(&config).unwrap();
|
|
||||||
assert_eq!(json["name"], "my_db");
|
|
||||||
assert_eq!(json["max_size"], 1000000);
|
|
||||||
assert_eq!(json["redis_version"], "7.0");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn test_backend_type_serialization() {
|
|
||||||
// Test that both Redb and Sled backends serialize correctly
|
|
||||||
let redb_backend = BackendType::Redb;
|
|
||||||
let sled_backend = BackendType::Sled;
|
|
||||||
|
|
||||||
let redb_json = serde_json::to_string(&redb_backend).unwrap();
|
|
||||||
let sled_json = serde_json::to_string(&sled_backend).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(redb_json, "\"Redb\"");
|
|
||||||
assert_eq!(sled_json, "\"Sled\"");
|
|
||||||
|
|
||||||
// Test deserialization
|
|
||||||
let redb_deserialized: BackendType = serde_json::from_str(&redb_json).unwrap();
|
|
||||||
let sled_deserialized: BackendType = serde_json::from_str(&sled_json).unwrap();
|
|
||||||
|
|
||||||
assert!(matches!(redb_deserialized, BackendType::Redb));
|
|
||||||
assert!(matches!(sled_deserialized, BackendType::Sled));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn test_database_name_persistence() {
|
|
||||||
let base_dir = "/tmp/test_db_name_persistence";
|
|
||||||
let admin_secret = "test-admin-secret";
|
|
||||||
let backend = OptionsBackendType::Redb;
|
|
||||||
let db_id = 1;
|
|
||||||
let test_name = "test-database-name";
|
|
||||||
|
|
||||||
// Clean up any existing test data
|
|
||||||
let _ = std::fs::remove_dir_all(base_dir);
|
|
||||||
|
|
||||||
// Set the database name
|
|
||||||
admin_meta::set_database_name(base_dir, backend.clone(), admin_secret, db_id, test_name)
|
|
||||||
.expect("Failed to set database name");
|
|
||||||
|
|
||||||
// Retrieve the database name
|
|
||||||
let retrieved_name = admin_meta::get_database_name(base_dir, backend, admin_secret, db_id)
|
|
||||||
.expect("Failed to get database name");
|
|
||||||
|
|
||||||
// Verify the name matches
|
|
||||||
assert_eq!(retrieved_name, Some(test_name.to_string()));
|
|
||||||
|
|
||||||
// Clean up
|
|
||||||
let _ = std::fs::remove_dir_all(base_dir);
|
|
||||||
}
|
|
@@ -1,8 +1,8 @@
|
|||||||
use herodb::{server::Server, options::DBOption};
|
use herodb::{options::DBOption, server::Server};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
use tokio::time::sleep;
|
|
||||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||||
use tokio::net::TcpStream;
|
use tokio::net::TcpStream;
|
||||||
|
use tokio::time::sleep;
|
||||||
|
|
||||||
// Helper function to start a test server with clean data directory
|
// Helper function to start a test server with clean data directory
|
||||||
async fn start_test_server(test_name: &str) -> (Server, u16) {
|
async fn start_test_server(test_name: &str) -> (Server, u16) {
|
||||||
@@ -25,7 +25,6 @@ async fn start_test_server(test_name: &str) -> (Server, u16) {
|
|||||||
encrypt: false,
|
encrypt: false,
|
||||||
encryption_key: None,
|
encryption_key: None,
|
||||||
backend: herodb::options::BackendType::Redb,
|
backend: herodb::options::BackendType::Redb,
|
||||||
admin_secret: "test-admin".to_string(),
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let server = Server::new(option).await;
|
let server = Server::new(option).await;
|
||||||
@@ -34,17 +33,12 @@ async fn start_test_server(test_name: &str) -> (Server, u16) {
|
|||||||
|
|
||||||
// Helper function to send Redis command and get response
|
// Helper function to send Redis command and get response
|
||||||
async fn send_redis_command(port: u16, command: &str) -> String {
|
async fn send_redis_command(port: u16, command: &str) -> String {
|
||||||
let mut stream = TcpStream::connect(format!("127.0.0.1:{}", port)).await.unwrap();
|
let mut stream = TcpStream::connect(format!("127.0.0.1:{}", port))
|
||||||
|
.await
|
||||||
// Acquire ReadWrite permissions on this new connection
|
.unwrap();
|
||||||
let handshake = "*4\r\n$6\r\nSELECT\r\n$1\r\n0\r\n$3\r\nKEY\r\n$10\r\ntest-admin\r\n";
|
|
||||||
stream.write_all(handshake.as_bytes()).await.unwrap();
|
|
||||||
let mut buffer = [0; 1024];
|
|
||||||
let _ = stream.read(&mut buffer).await.unwrap(); // Read and ignore the OK for handshake
|
|
||||||
|
|
||||||
// Now send the intended command
|
|
||||||
stream.write_all(command.as_bytes()).await.unwrap();
|
stream.write_all(command.as_bytes()).await.unwrap();
|
||||||
|
|
||||||
|
let mut buffer = [0; 1024];
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
String::from_utf8_lossy(&buffer[..n]).to_string()
|
String::from_utf8_lossy(&buffer[..n]).to_string()
|
||||||
}
|
}
|
||||||
@@ -74,7 +68,8 @@ async fn test_basic_redis_functionality() {
|
|||||||
assert!(response.contains("PONG"));
|
assert!(response.contains("PONG"));
|
||||||
|
|
||||||
// Test SET
|
// Test SET
|
||||||
let response = send_redis_command(port, "*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n").await;
|
let response =
|
||||||
|
send_redis_command(port, "*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n").await;
|
||||||
assert!(response.contains("OK"));
|
assert!(response.contains("OK"));
|
||||||
|
|
||||||
// Test GET
|
// Test GET
|
||||||
@@ -82,11 +77,16 @@ async fn test_basic_redis_functionality() {
|
|||||||
assert!(response.contains("value"));
|
assert!(response.contains("value"));
|
||||||
|
|
||||||
// Test HSET
|
// Test HSET
|
||||||
let response = send_redis_command(port, "*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$5\r\nfield\r\n$5\r\nvalue\r\n").await;
|
let response = send_redis_command(
|
||||||
|
port,
|
||||||
|
"*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$5\r\nfield\r\n$5\r\nvalue\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("1"));
|
assert!(response.contains("1"));
|
||||||
|
|
||||||
// Test HGET
|
// Test HGET
|
||||||
let response = send_redis_command(port, "*3\r\n$4\r\nHGET\r\n$4\r\nhash\r\n$5\r\nfield\r\n").await;
|
let response =
|
||||||
|
send_redis_command(port, "*3\r\n$4\r\nHGET\r\n$4\r\nhash\r\n$5\r\nfield\r\n").await;
|
||||||
assert!(response.contains("value"));
|
assert!(response.contains("value"));
|
||||||
|
|
||||||
// Test EXISTS
|
// Test EXISTS
|
||||||
@@ -102,8 +102,13 @@ async fn test_basic_redis_functionality() {
|
|||||||
assert!(response.contains("string"));
|
assert!(response.contains("string"));
|
||||||
|
|
||||||
// Test QUIT to close connection gracefully
|
// Test QUIT to close connection gracefully
|
||||||
let mut stream = TcpStream::connect(format!("127.0.0.1:{}", port)).await.unwrap();
|
let mut stream = TcpStream::connect(format!("127.0.0.1:{}", port))
|
||||||
stream.write_all("*1\r\n$4\r\nQUIT\r\n".as_bytes()).await.unwrap();
|
.await
|
||||||
|
.unwrap();
|
||||||
|
stream
|
||||||
|
.write_all("*1\r\n$4\r\nQUIT\r\n".as_bytes())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
let mut buffer = [0; 1024];
|
let mut buffer = [0; 1024];
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
let response = String::from_utf8_lossy(&buffer[..n]);
|
let response = String::from_utf8_lossy(&buffer[..n]);
|
||||||
@@ -150,7 +155,11 @@ async fn test_hash_operations() {
|
|||||||
assert!(response.contains("value2"));
|
assert!(response.contains("value2"));
|
||||||
|
|
||||||
// Test HEXISTS
|
// Test HEXISTS
|
||||||
let response = send_redis_command(port, "*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$6\r\nfield1\r\n").await;
|
let response = send_redis_command(
|
||||||
|
port,
|
||||||
|
"*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$6\r\nfield1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert!(response.contains("1"));
|
assert!(response.contains("1"));
|
||||||
|
|
||||||
// Test HLEN
|
// Test HLEN
|
||||||
@@ -193,46 +202,59 @@ async fn test_transaction_operations() {
|
|||||||
sleep(Duration::from_millis(100)).await;
|
sleep(Duration::from_millis(100)).await;
|
||||||
|
|
||||||
// Use a single connection for the transaction
|
// Use a single connection for the transaction
|
||||||
let mut stream = TcpStream::connect(format!("127.0.0.1:{}", port)).await.unwrap();
|
let mut stream = TcpStream::connect(format!("127.0.0.1:{}", port))
|
||||||
|
.await
|
||||||
// Acquire write permissions for this connection
|
.unwrap();
|
||||||
let handshake = "*4\r\n$6\r\nSELECT\r\n$1\r\n0\r\n$3\r\nKEY\r\n$10\r\ntest-admin\r\n";
|
|
||||||
stream.write_all(handshake.as_bytes()).await.unwrap();
|
|
||||||
let mut buffer = [0; 1024];
|
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
|
||||||
let resp = String::from_utf8_lossy(&buffer[..n]);
|
|
||||||
assert!(resp.contains("OK"));
|
|
||||||
|
|
||||||
// Test MULTI
|
// Test MULTI
|
||||||
stream.write_all("*1\r\n$5\r\nMULTI\r\n".as_bytes()).await.unwrap();
|
stream
|
||||||
|
.write_all("*1\r\n$5\r\nMULTI\r\n".as_bytes())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let mut buffer = [0; 1024];
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
let response = String::from_utf8_lossy(&buffer[..n]);
|
let response = String::from_utf8_lossy(&buffer[..n]);
|
||||||
assert!(response.contains("OK"));
|
assert!(response.contains("OK"));
|
||||||
|
|
||||||
// Test queued commands
|
// Test queued commands
|
||||||
stream.write_all("*3\r\n$3\r\nSET\r\n$4\r\nkey1\r\n$6\r\nvalue1\r\n".as_bytes()).await.unwrap();
|
stream
|
||||||
|
.write_all("*3\r\n$3\r\nSET\r\n$4\r\nkey1\r\n$6\r\nvalue1\r\n".as_bytes())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
let response = String::from_utf8_lossy(&buffer[..n]);
|
let response = String::from_utf8_lossy(&buffer[..n]);
|
||||||
assert!(response.contains("QUEUED"));
|
assert!(response.contains("QUEUED"));
|
||||||
|
|
||||||
stream.write_all("*3\r\n$3\r\nSET\r\n$4\r\nkey2\r\n$6\r\nvalue2\r\n".as_bytes()).await.unwrap();
|
stream
|
||||||
|
.write_all("*3\r\n$3\r\nSET\r\n$4\r\nkey2\r\n$6\r\nvalue2\r\n".as_bytes())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
let response = String::from_utf8_lossy(&buffer[..n]);
|
let response = String::from_utf8_lossy(&buffer[..n]);
|
||||||
assert!(response.contains("QUEUED"));
|
assert!(response.contains("QUEUED"));
|
||||||
|
|
||||||
// Test EXEC
|
// Test EXEC
|
||||||
stream.write_all("*1\r\n$4\r\nEXEC\r\n".as_bytes()).await.unwrap();
|
stream
|
||||||
|
.write_all("*1\r\n$4\r\nEXEC\r\n".as_bytes())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
let response = String::from_utf8_lossy(&buffer[..n]);
|
let response = String::from_utf8_lossy(&buffer[..n]);
|
||||||
assert!(response.contains("OK")); // Should contain array of OK responses
|
assert!(response.contains("OK")); // Should contain array of OK responses
|
||||||
|
|
||||||
// Verify commands were executed
|
// Verify commands were executed
|
||||||
stream.write_all("*2\r\n$3\r\nGET\r\n$4\r\nkey1\r\n".as_bytes()).await.unwrap();
|
stream
|
||||||
|
.write_all("*2\r\n$3\r\nGET\r\n$4\r\nkey1\r\n".as_bytes())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
let response = String::from_utf8_lossy(&buffer[..n]);
|
let response = String::from_utf8_lossy(&buffer[..n]);
|
||||||
assert!(response.contains("value1"));
|
assert!(response.contains("value1"));
|
||||||
|
|
||||||
stream.write_all("*2\r\n$3\r\nGET\r\n$4\r\nkey2\r\n".as_bytes()).await.unwrap();
|
stream
|
||||||
|
.write_all("*2\r\n$3\r\nGET\r\n$4\r\nkey2\r\n".as_bytes())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
let n = stream.read(&mut buffer).await.unwrap();
|
let n = stream.read(&mut buffer).await.unwrap();
|
||||||
let response = String::from_utf8_lossy(&buffer[..n]);
|
let response = String::from_utf8_lossy(&buffer[..n]);
|
||||||
assert!(response.contains("value2"));
|
assert!(response.contains("value2"));
|
||||||
|
@@ -1,4 +1,4 @@
|
|||||||
use herodb::{server::Server, options::DBOption};
|
use herodb::{options::DBOption, server::Server};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||||
use tokio::net::TcpStream;
|
use tokio::net::TcpStream;
|
||||||
@@ -23,7 +23,6 @@ async fn start_test_server(test_name: &str) -> (Server, u16) {
|
|||||||
encrypt: false,
|
encrypt: false,
|
||||||
encryption_key: None,
|
encryption_key: None,
|
||||||
backend: herodb::options::BackendType::Redb,
|
backend: herodb::options::BackendType::Redb,
|
||||||
admin_secret: "test-admin".to_string(),
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let server = Server::new(option).await;
|
let server = Server::new(option).await;
|
||||||
@@ -39,22 +38,12 @@ async fn send_command(stream: &mut TcpStream, command: &str) -> String {
|
|||||||
String::from_utf8_lossy(&buffer[..n]).to_string()
|
String::from_utf8_lossy(&buffer[..n]).to_string()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Helper function to connect to the test server
|
// Helper function to connect to the test server
|
||||||
async fn connect_to_server(port: u16) -> TcpStream {
|
async fn connect_to_server(port: u16) -> TcpStream {
|
||||||
let mut attempts = 0;
|
let mut attempts = 0;
|
||||||
loop {
|
loop {
|
||||||
match TcpStream::connect(format!("127.0.0.1:{}", port)).await {
|
match TcpStream::connect(format!("127.0.0.1:{}", port)).await {
|
||||||
Ok(mut stream) => {
|
Ok(stream) => return stream,
|
||||||
// Acquire ReadWrite permissions for this connection
|
|
||||||
let resp = send_command(
|
|
||||||
&mut stream,
|
|
||||||
"*4\r\n$6\r\nSELECT\r\n$1\r\n0\r\n$3\r\nKEY\r\n$10\r\ntest-admin\r\n",
|
|
||||||
).await;
|
|
||||||
if !resp.contains("OK") {
|
|
||||||
panic!("Failed to acquire write permissions via SELECT 0 KEY test-admin: {}", resp);
|
|
||||||
}
|
|
||||||
return stream;
|
|
||||||
}
|
|
||||||
Err(_) if attempts < 10 => {
|
Err(_) if attempts < 10 => {
|
||||||
attempts += 1;
|
attempts += 1;
|
||||||
sleep(Duration::from_millis(100)).await;
|
sleep(Duration::from_millis(100)).await;
|
||||||
@@ -109,20 +98,25 @@ async fn test_hset_clean_db() {
|
|||||||
|
|
||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Ensure clean DB state (admin DB 0 may be shared due to global singleton)
|
// Test HSET - should return 1 for new field
|
||||||
let flush = send_command(&mut stream, "*1\r\n$7\r\nFLUSHDB\r\n").await;
|
let response = send_command(
|
||||||
assert!(flush.contains("OK"), "Failed to FLUSHDB: {}", flush);
|
&mut stream,
|
||||||
|
"*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n",
|
||||||
// Test HSET - should return 1 for new field (use a unique key name to avoid collisions)
|
)
|
||||||
let key = "hash_clean";
|
.await;
|
||||||
let hset_cmd = format!("*4\r\n$4\r\nHSET\r\n${}\r\n{}\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n", key.len(), key);
|
|
||||||
let response = send_command(&mut stream, &hset_cmd).await;
|
|
||||||
println!("HSET response: {}", response);
|
println!("HSET response: {}", response);
|
||||||
assert!(response.contains("1"), "Expected HSET to return 1, got: {}", response);
|
assert!(
|
||||||
|
response.contains("1"),
|
||||||
|
"Expected HSET to return 1, got: {}",
|
||||||
|
response
|
||||||
|
);
|
||||||
|
|
||||||
// Test HGET
|
// Test HGET
|
||||||
let hget_cmd = format!("*3\r\n$4\r\nHGET\r\n${}\r\n{}\r\n$6\r\nfield1\r\n", key.len(), key);
|
let response = send_command(
|
||||||
let response = send_command(&mut stream, &hget_cmd).await;
|
&mut stream,
|
||||||
|
"*3\r\n$4\r\nHGET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
println!("HGET response: {}", response);
|
println!("HGET response: {}", response);
|
||||||
assert!(response.contains("value1"));
|
assert!(response.contains("value1"));
|
||||||
}
|
}
|
||||||
@@ -149,13 +143,21 @@ async fn test_type_command_simple() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Test string type
|
// Test string type
|
||||||
send_command(&mut stream, "*3\r\n$3\r\nSET\r\n$6\r\nstring\r\n$5\r\nvalue\r\n").await;
|
send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$3\r\nSET\r\n$6\r\nstring\r\n$5\r\nvalue\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$6\r\nstring\r\n").await;
|
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$6\r\nstring\r\n").await;
|
||||||
println!("TYPE string response: {}", response);
|
println!("TYPE string response: {}", response);
|
||||||
assert!(response.contains("string"));
|
assert!(response.contains("string"));
|
||||||
|
|
||||||
// Test hash type
|
// Test hash type
|
||||||
send_command(&mut stream, "*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$5\r\nfield\r\n$5\r\nvalue\r\n").await;
|
send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$5\r\nfield\r\n$5\r\nvalue\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$4\r\nhash\r\n").await;
|
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$4\r\nhash\r\n").await;
|
||||||
println!("TYPE hash response: {}", response);
|
println!("TYPE hash response: {}", response);
|
||||||
assert!(response.contains("hash"));
|
assert!(response.contains("hash"));
|
||||||
@@ -163,7 +165,11 @@ async fn test_type_command_simple() {
|
|||||||
// Test non-existent key
|
// Test non-existent key
|
||||||
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$7\r\nnoexist\r\n").await;
|
let response = send_command(&mut stream, "*2\r\n$4\r\nTYPE\r\n$7\r\nnoexist\r\n").await;
|
||||||
println!("TYPE noexist response: {}", response);
|
println!("TYPE noexist response: {}", response);
|
||||||
assert!(response.contains("none"), "Expected 'none' for non-existent key, got: {}", response);
|
assert!(
|
||||||
|
response.contains("none"),
|
||||||
|
"Expected 'none' for non-existent key, got: {}",
|
||||||
|
response
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
@@ -188,15 +194,31 @@ async fn test_hexists_simple() {
|
|||||||
let mut stream = connect_to_server(port).await;
|
let mut stream = connect_to_server(port).await;
|
||||||
|
|
||||||
// Set up hash
|
// Set up hash
|
||||||
send_command(&mut stream, "*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n").await;
|
send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*4\r\n$4\r\nHSET\r\n$4\r\nhash\r\n$6\r\nfield1\r\n$6\r\nvalue1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
// Test HEXISTS for existing field
|
// Test HEXISTS for existing field
|
||||||
let response = send_command(&mut stream, "*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$6\r\nfield1\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$6\r\nfield1\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
println!("HEXISTS existing field response: {}", response);
|
println!("HEXISTS existing field response: {}", response);
|
||||||
assert!(response.contains("1"));
|
assert!(response.contains("1"));
|
||||||
|
|
||||||
// Test HEXISTS for non-existent field
|
// Test HEXISTS for non-existent field
|
||||||
let response = send_command(&mut stream, "*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$7\r\nnoexist\r\n").await;
|
let response = send_command(
|
||||||
|
&mut stream,
|
||||||
|
"*3\r\n$7\r\nHEXISTS\r\n$4\r\nhash\r\n$7\r\nnoexist\r\n",
|
||||||
|
)
|
||||||
|
.await;
|
||||||
println!("HEXISTS non-existent field response: {}", response);
|
println!("HEXISTS non-existent field response: {}", response);
|
||||||
assert!(response.contains("0"), "Expected HEXISTS to return 0 for non-existent field, got: {}", response);
|
assert!(
|
||||||
|
response.contains("0"),
|
||||||
|
"Expected HEXISTS to return 0 for non-existent field, got: {}",
|
||||||
|
response
|
||||||
|
);
|
||||||
}
|
}
|
@@ -1,294 +0,0 @@
|
|||||||
use redis::{Client, Connection, RedisResult};
|
|
||||||
use std::process::{Child, Command};
|
|
||||||
use std::time::Duration;
|
|
||||||
use jsonrpsee::http_client::{HttpClientBuilder, HttpClient};
|
|
||||||
use herodb::rpc::{RpcClient, BackendType, DatabaseConfig};
|
|
||||||
|
|
||||||
// Helper function to get Redis connection, retrying until successful
|
|
||||||
fn get_redis_connection(port: u16) -> Connection {
|
|
||||||
let connection_info = format!("redis://127.0.0.1:{}", port);
|
|
||||||
let client = Client::open(connection_info).unwrap();
|
|
||||||
let mut attempts = 0;
|
|
||||||
loop {
|
|
||||||
match client.get_connection() {
|
|
||||||
Ok(mut conn) => {
|
|
||||||
if redis::cmd("PING").query::<String>(&mut conn).is_ok() {
|
|
||||||
return conn;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
if attempts >= 120 {
|
|
||||||
panic!(
|
|
||||||
"Failed to connect to Redis server after 120 attempts: {}",
|
|
||||||
e
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
attempts += 1;
|
|
||||||
std::thread::sleep(Duration::from_millis(100));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Helper function to get RPC client
|
|
||||||
async fn get_rpc_client(port: u16) -> HttpClient {
|
|
||||||
let url = format!("http://127.0.0.1:{}", port + 1); // RPC port is Redis port + 1
|
|
||||||
let client = HttpClientBuilder::default().build(url).unwrap();
|
|
||||||
client
|
|
||||||
}
|
|
||||||
|
|
||||||
// A guard to ensure the server process is killed when it goes out of scope
|
|
||||||
struct ServerProcessGuard {
|
|
||||||
process: Child,
|
|
||||||
test_dir: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Drop for ServerProcessGuard {
|
|
||||||
fn drop(&mut self) {
|
|
||||||
println!("Killing server process (pid: {})...", self.process.id());
|
|
||||||
if let Err(e) = self.process.kill() {
|
|
||||||
eprintln!("Failed to kill server process: {}", e);
|
|
||||||
}
|
|
||||||
match self.process.wait() {
|
|
||||||
Ok(status) => println!("Server process exited with: {}", status),
|
|
||||||
Err(e) => eprintln!("Failed to wait on server process: {}", e),
|
|
||||||
}
|
|
||||||
|
|
||||||
// Clean up the specific test directory
|
|
||||||
println!("Cleaning up test directory: {}", self.test_dir);
|
|
||||||
if let Err(e) = std::fs::remove_dir_all(&self.test_dir) {
|
|
||||||
eprintln!("Failed to clean up test directory: {}", e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Helper to set up the server and return connections
|
|
||||||
async fn setup_server() -> (ServerProcessGuard, u16, Connection, HttpClient) {
|
|
||||||
use std::sync::atomic::{AtomicU16, Ordering};
|
|
||||||
static PORT_COUNTER: AtomicU16 = AtomicU16::new(16500);
|
|
||||||
let port = PORT_COUNTER.fetch_add(1, Ordering::SeqCst);
|
|
||||||
|
|
||||||
let test_dir = format!("/tmp/herodb_tantivy_test_{}", port);
|
|
||||||
|
|
||||||
// Clean up previous test data
|
|
||||||
if std::path::Path::new(&test_dir).exists() {
|
|
||||||
let _ = std::fs::remove_dir_all(&test_dir);
|
|
||||||
}
|
|
||||||
std::fs::create_dir_all(&test_dir).unwrap();
|
|
||||||
|
|
||||||
// Start the server in a subprocess
|
|
||||||
let child = Command::new("cargo")
|
|
||||||
.args(&[
|
|
||||||
"run",
|
|
||||||
"--",
|
|
||||||
"--dir",
|
|
||||||
&test_dir,
|
|
||||||
"--port",
|
|
||||||
&port.to_string(),
|
|
||||||
"--rpc-port",
|
|
||||||
&(port + 1).to_string(),
|
|
||||||
"--enable-rpc",
|
|
||||||
"--debug",
|
|
||||||
"--admin-secret",
|
|
||||||
"test-admin",
|
|
||||||
])
|
|
||||||
.spawn()
|
|
||||||
.expect("Failed to start server process");
|
|
||||||
|
|
||||||
// Create a new guard that also owns the test directory path
|
|
||||||
let guard = ServerProcessGuard {
|
|
||||||
process: child,
|
|
||||||
test_dir,
|
|
||||||
};
|
|
||||||
|
|
||||||
// Give the server time to build and start (cargo run may compile first)
|
|
||||||
std::thread::sleep(Duration::from_millis(3000));
|
|
||||||
|
|
||||||
let conn = get_redis_connection(port);
|
|
||||||
let rpc_client = get_rpc_client(port).await;
|
|
||||||
|
|
||||||
(guard, port, conn, rpc_client)
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn test_tantivy_full_text_search() {
|
|
||||||
let (_server_guard, _port, mut conn, rpc_client) = setup_server().await;
|
|
||||||
|
|
||||||
// Create a Tantivy database via RPC
|
|
||||||
let db_config = DatabaseConfig {
|
|
||||||
name: Some("test_tantivy_db".to_string()),
|
|
||||||
storage_path: None,
|
|
||||||
max_size: None,
|
|
||||||
redis_version: None,
|
|
||||||
};
|
|
||||||
|
|
||||||
let db_id = rpc_client.create_database(BackendType::Tantivy, db_config, None).await.unwrap();
|
|
||||||
assert_eq!(db_id, 1);
|
|
||||||
|
|
||||||
// Add readwrite access key
|
|
||||||
let _ = rpc_client.add_access_key(db_id, "readwrite_key".to_string(), "readwrite".to_string()).await.unwrap();
|
|
||||||
|
|
||||||
// Add read-only access key
|
|
||||||
let _ = rpc_client.add_access_key(db_id, "read_key".to_string(), "read".to_string()).await.unwrap();
|
|
||||||
|
|
||||||
// Test with readwrite permissions
|
|
||||||
test_tantivy_with_readwrite_permissions(&mut conn, db_id).await;
|
|
||||||
|
|
||||||
// Test with read-only permissions
|
|
||||||
test_tantivy_with_read_permissions(&mut conn, db_id).await;
|
|
||||||
|
|
||||||
// Test access denied for invalid key
|
|
||||||
test_tantivy_access_denied(&mut conn, db_id).await;
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn test_tantivy_with_readwrite_permissions(conn: &mut Connection, db_id: u64) {
|
|
||||||
// Select database with readwrite key
|
|
||||||
let result: RedisResult<String> = redis::cmd("SELECT")
|
|
||||||
.arg(db_id)
|
|
||||||
.arg("KEY")
|
|
||||||
.arg("readwrite_key")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
assert_eq!(result.unwrap(), "OK");
|
|
||||||
|
|
||||||
// Test FT.CREATE
|
|
||||||
let result: RedisResult<String> = redis::cmd("FT.CREATE")
|
|
||||||
.arg("test_index")
|
|
||||||
.arg("SCHEMA")
|
|
||||||
.arg("title")
|
|
||||||
.arg("TEXT")
|
|
||||||
.arg("content")
|
|
||||||
.arg("TEXT")
|
|
||||||
.arg("tags")
|
|
||||||
.arg("TAG")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
assert_eq!(result.unwrap(), "OK");
|
|
||||||
|
|
||||||
// Test FT.ADD
|
|
||||||
let result: RedisResult<String> = redis::cmd("FT.ADD")
|
|
||||||
.arg("test_index")
|
|
||||||
.arg("doc1")
|
|
||||||
.arg("1.0")
|
|
||||||
.arg("title")
|
|
||||||
.arg("Hello World")
|
|
||||||
.arg("content")
|
|
||||||
.arg("This is a test document")
|
|
||||||
.arg("tags")
|
|
||||||
.arg("test,example")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
assert_eq!(result.unwrap(), "OK");
|
|
||||||
|
|
||||||
// Add another document
|
|
||||||
let result: RedisResult<String> = redis::cmd("FT.ADD")
|
|
||||||
.arg("test_index")
|
|
||||||
.arg("doc2")
|
|
||||||
.arg("1.0")
|
|
||||||
.arg("title")
|
|
||||||
.arg("Goodbye World")
|
|
||||||
.arg("content")
|
|
||||||
.arg("Another test document")
|
|
||||||
.arg("tags")
|
|
||||||
.arg("test,another")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
assert_eq!(result.unwrap(), "OK");
|
|
||||||
|
|
||||||
// Test FT.SEARCH
|
|
||||||
let result: RedisResult<Vec<String>> = redis::cmd("FT.SEARCH")
|
|
||||||
.arg("test_index")
|
|
||||||
.arg("test")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
let results = result.unwrap();
|
|
||||||
assert!(results.len() >= 3); // At least total count + 2 documents
|
|
||||||
assert_eq!(results[0], "2"); // Total matches
|
|
||||||
|
|
||||||
// Test FT.INFO
|
|
||||||
let result: RedisResult<Vec<String>> = redis::cmd("FT.INFO")
|
|
||||||
.arg("test_index")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
let info = result.unwrap();
|
|
||||||
assert!(info.contains(&"index_name".to_string()));
|
|
||||||
assert!(info.contains(&"test_index".to_string()));
|
|
||||||
|
|
||||||
// Test FT.DEL
|
|
||||||
let result: RedisResult<String> = redis::cmd("FT.DEL")
|
|
||||||
.arg("test_index")
|
|
||||||
.arg("doc1")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
assert_eq!(result.unwrap(), "1");
|
|
||||||
|
|
||||||
// Verify document was deleted
|
|
||||||
let result: RedisResult<Vec<String>> = redis::cmd("FT.SEARCH")
|
|
||||||
.arg("test_index")
|
|
||||||
.arg("Hello")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
let results = result.unwrap();
|
|
||||||
assert_eq!(results[0], "0"); // No matches
|
|
||||||
|
|
||||||
// Test FT.DROP
|
|
||||||
let result: RedisResult<String> = redis::cmd("FT.DROP")
|
|
||||||
.arg("test_index")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
assert_eq!(result.unwrap(), "OK");
|
|
||||||
|
|
||||||
// Verify index was dropped
|
|
||||||
let result: RedisResult<String> = redis::cmd("FT.INFO")
|
|
||||||
.arg("test_index")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_err()); // Should fail
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn test_tantivy_with_read_permissions(conn: &mut Connection, db_id: u64) {
|
|
||||||
// Select database with read-only key
|
|
||||||
let result: RedisResult<String> = redis::cmd("SELECT")
|
|
||||||
.arg(db_id)
|
|
||||||
.arg("KEY")
|
|
||||||
.arg("read_key")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_ok());
|
|
||||||
assert_eq!(result.unwrap(), "OK");
|
|
||||||
|
|
||||||
// Recreate index for testing
|
|
||||||
let result: RedisResult<String> = redis::cmd("FT.CREATE")
|
|
||||||
.arg("test_index_read")
|
|
||||||
.arg("SCHEMA")
|
|
||||||
.arg("title")
|
|
||||||
.arg("TEXT")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_err()); // Should fail due to read-only permissions
|
|
||||||
assert!(result.unwrap_err().to_string().contains("write permission denied"));
|
|
||||||
|
|
||||||
// Add document should fail
|
|
||||||
let result: RedisResult<String> = redis::cmd("FT.ADD")
|
|
||||||
.arg("test_index_read")
|
|
||||||
.arg("doc1")
|
|
||||||
.arg("1.0")
|
|
||||||
.arg("title")
|
|
||||||
.arg("Test")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(result.unwrap_err().to_string().contains("write permission denied"));
|
|
||||||
|
|
||||||
// But search should work (if index exists)
|
|
||||||
// First create index with write permissions, then switch to read
|
|
||||||
// For this test, we'll assume the index doesn't exist, so search fails differently
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn test_tantivy_access_denied(conn: &mut Connection, db_id: u64) {
|
|
||||||
// Try to select with invalid key
|
|
||||||
let result: RedisResult<String> = redis::cmd("SELECT")
|
|
||||||
.arg(db_id)
|
|
||||||
.arg("KEY")
|
|
||||||
.arg("invalid_key")
|
|
||||||
.query(conn);
|
|
||||||
assert!(result.is_err());
|
|
||||||
assert!(result.unwrap_err().to_string().contains("invalid access key"));
|
|
||||||
}
|
|
@@ -23,7 +23,6 @@ async fn start_test_server(test_name: &str) -> (Server, u16) {
|
|||||||
encrypt: false,
|
encrypt: false,
|
||||||
encryption_key: None,
|
encryption_key: None,
|
||||||
backend: herodb::options::BackendType::Redb,
|
backend: herodb::options::BackendType::Redb,
|
||||||
admin_secret: "test-admin".to_string(),
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let server = Server::new(option).await;
|
let server = Server::new(option).await;
|
||||||
@@ -62,17 +61,7 @@ async fn connect(port: u16) -> TcpStream {
|
|||||||
let mut attempts = 0;
|
let mut attempts = 0;
|
||||||
loop {
|
loop {
|
||||||
match TcpStream::connect(format!("127.0.0.1:{}", port)).await {
|
match TcpStream::connect(format!("127.0.0.1:{}", port)).await {
|
||||||
Ok(mut s) => {
|
Ok(s) => return s,
|
||||||
// Acquire ReadWrite permissions for this connection using admin DB 0
|
|
||||||
let resp = send_cmd(&mut s, &["SELECT", "0", "KEY", "test-admin"]).await;
|
|
||||||
assert_contains(&resp, "OK", "SELECT 0 KEY test-admin handshake");
|
|
||||||
|
|
||||||
// Ensure clean slate per test on DB 0
|
|
||||||
let fl = send_cmd(&mut s, &["FLUSHDB"]).await;
|
|
||||||
assert_contains(&fl, "OK", "FLUSHDB after handshake");
|
|
||||||
|
|
||||||
return s;
|
|
||||||
}
|
|
||||||
Err(_) if attempts < 30 => {
|
Err(_) if attempts < 30 => {
|
||||||
attempts += 1;
|
attempts += 1;
|
||||||
sleep(Duration::from_millis(100)).await;
|
sleep(Duration::from_millis(100)).await;
|
||||||
@@ -257,9 +246,9 @@ async fn test_01_connection_and_info() {
|
|||||||
let getname = send_cmd(&mut s, &["CLIENT", "GETNAME"]).await;
|
let getname = send_cmd(&mut s, &["CLIENT", "GETNAME"]).await;
|
||||||
assert_contains(&getname, "myapp", "CLIENT GETNAME");
|
assert_contains(&getname, "myapp", "CLIENT GETNAME");
|
||||||
|
|
||||||
// SELECT db (requires key on DB 0)
|
// SELECT db
|
||||||
let sel = send_cmd(&mut s, &["SELECT", "0", "KEY", "test-admin"]).await;
|
let sel = send_cmd(&mut s, &["SELECT", "0"]).await;
|
||||||
assert_contains(&sel, "OK", "SELECT 0 with key");
|
assert_contains(&sel, "OK", "SELECT 0");
|
||||||
|
|
||||||
// QUIT should close connection after sending OK
|
// QUIT should close connection after sending OK
|
||||||
let quit = send_cmd(&mut s, &["QUIT"]).await;
|
let quit = send_cmd(&mut s, &["QUIT"]).await;
|
||||||
@@ -291,10 +280,6 @@ async fn test_02_strings_and_expiry() {
|
|||||||
let ex0 = send_cmd(&mut s, &["EXISTS", "user:1"]).await;
|
let ex0 = send_cmd(&mut s, &["EXISTS", "user:1"]).await;
|
||||||
assert_contains(&ex0, "0", "EXISTS after DEL");
|
assert_contains(&ex0, "0", "EXISTS after DEL");
|
||||||
|
|
||||||
// DEL non-existent should return 0
|
|
||||||
let del0 = send_cmd(&mut s, &["DEL", "user:1"]).await;
|
|
||||||
assert_contains(&del0, "0", "DEL user:1 when not exists -> 0");
|
|
||||||
|
|
||||||
// INCR behavior
|
// INCR behavior
|
||||||
let i1 = send_cmd(&mut s, &["INCR", "count"]).await;
|
let i1 = send_cmd(&mut s, &["INCR", "count"]).await;
|
||||||
assert_contains(&i1, "1", "INCR new key -> 1");
|
assert_contains(&i1, "1", "INCR new key -> 1");
|
||||||
@@ -340,7 +325,11 @@ async fn test_03_scan_and_keys() {
|
|||||||
let mut s = connect(port).await;
|
let mut s = connect(port).await;
|
||||||
|
|
||||||
for i in 0..5 {
|
for i in 0..5 {
|
||||||
let _ = send_cmd(&mut s, &["SET", &format!("key{}", i), &format!("value{}", i)]).await;
|
let _ = send_cmd(
|
||||||
|
&mut s,
|
||||||
|
&["SET", &format!("key{}", i), &format!("value{}", i)],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
}
|
}
|
||||||
|
|
||||||
let scan = send_cmd(&mut s, &["SCAN", "0", "MATCH", "key*", "COUNT", "10"]).await;
|
let scan = send_cmd(&mut s, &["SCAN", "0", "MATCH", "key*", "COUNT", "10"]).await;
|
||||||
@@ -373,7 +362,11 @@ async fn test_04_hashes_suite() {
|
|||||||
assert_contains(&h2, "2", "HSET added 2 new fields");
|
assert_contains(&h2, "2", "HSET added 2 new fields");
|
||||||
|
|
||||||
// HMGET
|
// HMGET
|
||||||
let hmg = send_cmd(&mut s, &["HMGET", "profile:1", "name", "age", "city", "nope"]).await;
|
let hmg = send_cmd(
|
||||||
|
&mut s,
|
||||||
|
&["HMGET", "profile:1", "name", "age", "city", "nope"],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert_contains(&hmg, "alice", "HMGET name");
|
assert_contains(&hmg, "alice", "HMGET name");
|
||||||
assert_contains(&hmg, "30", "HMGET age");
|
assert_contains(&hmg, "30", "HMGET age");
|
||||||
assert_contains(&hmg, "paris", "HMGET city");
|
assert_contains(&hmg, "paris", "HMGET city");
|
||||||
@@ -407,7 +400,11 @@ async fn test_04_hashes_suite() {
|
|||||||
assert_contains(&hnx1, "1", "HSETNX new field -> 1");
|
assert_contains(&hnx1, "1", "HSETNX new field -> 1");
|
||||||
|
|
||||||
// HSCAN
|
// HSCAN
|
||||||
let hscan = send_cmd(&mut s, &["HSCAN", "profile:1", "0", "MATCH", "n*", "COUNT", "10"]).await;
|
let hscan = send_cmd(
|
||||||
|
&mut s,
|
||||||
|
&["HSCAN", "profile:1", "0", "MATCH", "n*", "COUNT", "10"],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert_contains(&hscan, "name", "HSCAN matches fields starting with n");
|
assert_contains(&hscan, "name", "HSCAN matches fields starting with n");
|
||||||
assert_contains(&hscan, "nickname", "HSCAN nickname present");
|
assert_contains(&hscan, "nickname", "HSCAN nickname present");
|
||||||
|
|
||||||
@@ -439,13 +436,21 @@ async fn test_05_lists_suite_including_blpop() {
|
|||||||
assert_eq_resp(&lidx, "$1\r\nb\r\n", "LINDEX q:jobs 0 should be b");
|
assert_eq_resp(&lidx, "$1\r\nb\r\n", "LINDEX q:jobs 0 should be b");
|
||||||
|
|
||||||
let lr = send_cmd(&mut a, &["LRANGE", "q:jobs", "0", "-1"]).await;
|
let lr = send_cmd(&mut a, &["LRANGE", "q:jobs", "0", "-1"]).await;
|
||||||
assert_eq_resp(&lr, "*3\r\n$1\r\nb\r\n$1\r\na\r\n$1\r\nc\r\n", "LRANGE q:jobs 0 -1 should be [b,a,c]");
|
assert_eq_resp(
|
||||||
|
&lr,
|
||||||
|
"*3\r\n$1\r\nb\r\n$1\r\na\r\n$1\r\nc\r\n",
|
||||||
|
"LRANGE q:jobs 0 -1 should be [b,a,c]",
|
||||||
|
);
|
||||||
|
|
||||||
// LTRIM
|
// LTRIM
|
||||||
let ltrim = send_cmd(&mut a, &["LTRIM", "q:jobs", "0", "1"]).await;
|
let ltrim = send_cmd(&mut a, &["LTRIM", "q:jobs", "0", "1"]).await;
|
||||||
assert_contains(<rim, "OK", "LTRIM OK");
|
assert_contains(<rim, "OK", "LTRIM OK");
|
||||||
let lr_post = send_cmd(&mut a, &["LRANGE", "q:jobs", "0", "-1"]).await;
|
let lr_post = send_cmd(&mut a, &["LRANGE", "q:jobs", "0", "-1"]).await;
|
||||||
assert_eq_resp(&lr_post, "*2\r\n$1\r\nb\r\n$1\r\na\r\n", "After LTRIM, list [b,a]");
|
assert_eq_resp(
|
||||||
|
&lr_post,
|
||||||
|
"*2\r\n$1\r\nb\r\n$1\r\na\r\n",
|
||||||
|
"After LTRIM, list [b,a]",
|
||||||
|
);
|
||||||
|
|
||||||
// LREM remove first occurrence of b
|
// LREM remove first occurrence of b
|
||||||
let lrem = send_cmd(&mut a, &["LREM", "q:jobs", "1", "b"]).await;
|
let lrem = send_cmd(&mut a, &["LREM", "q:jobs", "1", "b"]).await;
|
||||||
@@ -459,7 +464,11 @@ async fn test_05_lists_suite_including_blpop() {
|
|||||||
|
|
||||||
// LPOP with count on empty -> []
|
// LPOP with count on empty -> []
|
||||||
let lpop0 = send_cmd(&mut a, &["LPOP", "q:jobs", "2"]).await;
|
let lpop0 = send_cmd(&mut a, &["LPOP", "q:jobs", "2"]).await;
|
||||||
assert_eq_resp(&lpop0, "*0\r\n", "LPOP with count on empty returns empty array");
|
assert_eq_resp(
|
||||||
|
&lpop0,
|
||||||
|
"*0\r\n",
|
||||||
|
"LPOP with count on empty returns empty array",
|
||||||
|
);
|
||||||
|
|
||||||
// BLPOP: block on one client, push from another
|
// BLPOP: block on one client, push from another
|
||||||
let c1 = connect(port).await;
|
let c1 = connect(port).await;
|
||||||
@@ -516,11 +525,11 @@ async fn test_07_age_stateless_suite() {
|
|||||||
let mut s = connect(port).await;
|
let mut s = connect(port).await;
|
||||||
|
|
||||||
// GENENC -> [recipient, identity]
|
// GENENC -> [recipient, identity]
|
||||||
let genenc = send_cmd(&mut s, &["AGE", "GENENC"]).await;
|
let gen = send_cmd(&mut s, &["AGE", "GENENC"]).await;
|
||||||
assert!(
|
assert!(
|
||||||
genenc.starts_with("*2\r\n$"),
|
gen.starts_with("*2\r\n$"),
|
||||||
"AGE GENENC should return array [recipient, identity], got:\n{}",
|
"AGE GENENC should return array [recipient, identity], got:\n{}",
|
||||||
genenc
|
gen
|
||||||
);
|
);
|
||||||
|
|
||||||
// Parse simple RESP array of two bulk strings to extract keys
|
// Parse simple RESP array of two bulk strings to extract keys
|
||||||
@@ -535,7 +544,7 @@ async fn test_07_age_stateless_suite() {
|
|||||||
let ident = lines.next().unwrap_or("").to_string();
|
let ident = lines.next().unwrap_or("").to_string();
|
||||||
(recip, ident)
|
(recip, ident)
|
||||||
}
|
}
|
||||||
let (recipient, identity) = parse_two_bulk_array(&genenc);
|
let (recipient, identity) = parse_two_bulk_array(&gen);
|
||||||
assert!(
|
assert!(
|
||||||
recipient.starts_with("age1") && identity.starts_with("AGE-SECRET-KEY-1"),
|
recipient.starts_with("age1") && identity.starts_with("AGE-SECRET-KEY-1"),
|
||||||
"Unexpected AGE key formats.\nrecipient: {}\nidentity: {}",
|
"Unexpected AGE key formats.\nrecipient: {}\nidentity: {}",
|
||||||
@@ -563,8 +572,16 @@ async fn test_07_age_stateless_suite() {
|
|||||||
let v_ok = send_cmd(&mut s, &["AGE", "VERIFY", &verify_pub, "msg", &sig_b64]).await;
|
let v_ok = send_cmd(&mut s, &["AGE", "VERIFY", &verify_pub, "msg", &sig_b64]).await;
|
||||||
assert_contains(&v_ok, "1", "VERIFY should be 1 for valid signature");
|
assert_contains(&v_ok, "1", "VERIFY should be 1 for valid signature");
|
||||||
|
|
||||||
let v_bad = send_cmd(&mut s, &["AGE", "VERIFY", &verify_pub, "tampered", &sig_b64]).await;
|
let v_bad = send_cmd(
|
||||||
assert_contains(&v_bad, "0", "VERIFY should be 0 for invalid message/signature");
|
&mut s,
|
||||||
|
&["AGE", "VERIFY", &verify_pub, "tampered", &sig_b64],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert_contains(
|
||||||
|
&v_bad,
|
||||||
|
"0",
|
||||||
|
"VERIFY should be 0 for invalid message/signature",
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
@@ -596,7 +613,7 @@ async fn test_08_age_persistent_named_suite() {
|
|||||||
skg
|
skg
|
||||||
);
|
);
|
||||||
|
|
||||||
let sig = send_cmd(&mut s, &["AGE", "SIGNNAME", "app1", "m"] ).await;
|
let sig = send_cmd(&mut s, &["AGE", "SIGNNAME", "app1", "m"]).await;
|
||||||
let sig_b64 = extract_bulk_payload(&sig).expect("Failed to parse bulk payload from SIGNNAME");
|
let sig_b64 = extract_bulk_payload(&sig).expect("Failed to parse bulk payload from SIGNNAME");
|
||||||
let v1 = send_cmd(&mut s, &["AGE", "VERIFYNAME", "app1", "m", &sig_b64]).await;
|
let v1 = send_cmd(&mut s, &["AGE", "VERIFYNAME", "app1", "m", &sig_b64]).await;
|
||||||
assert_contains(&v1, "1", "VERIFYNAME valid => 1");
|
assert_contains(&v1, "1", "VERIFYNAME valid => 1");
|
||||||
@@ -606,7 +623,7 @@ async fn test_08_age_persistent_named_suite() {
|
|||||||
|
|
||||||
// AGE LIST
|
// AGE LIST
|
||||||
let lst = send_cmd(&mut s, &["AGE", "LIST"]).await;
|
let lst = send_cmd(&mut s, &["AGE", "LIST"]).await;
|
||||||
// After flattening, LIST returns a flat array of managed key names
|
assert_contains(&lst, "encpub", "AGE LIST label encpub");
|
||||||
assert_contains(&lst, "app1", "AGE LIST includes app1");
|
assert_contains(&lst, "app1", "AGE LIST includes app1");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -655,7 +672,12 @@ async fn test_10_expire_pexpire_persist() {
|
|||||||
let _ = send_cmd(&mut s, &["EXPIRE", "exp:persist", "5"]).await;
|
let _ = send_cmd(&mut s, &["EXPIRE", "exp:persist", "5"]).await;
|
||||||
let ttl_pre = send_cmd(&mut s, &["TTL", "exp:persist"]).await;
|
let ttl_pre = send_cmd(&mut s, &["TTL", "exp:persist"]).await;
|
||||||
assert!(
|
assert!(
|
||||||
ttl_pre.contains("5") || ttl_pre.contains("4") || ttl_pre.contains("3") || ttl_pre.contains("2") || ttl_pre.contains("1") || ttl_pre.contains("0"),
|
ttl_pre.contains("5")
|
||||||
|
|| ttl_pre.contains("4")
|
||||||
|
|| ttl_pre.contains("3")
|
||||||
|
|| ttl_pre.contains("2")
|
||||||
|
|| ttl_pre.contains("1")
|
||||||
|
|| ttl_pre.contains("0"),
|
||||||
"TTL exp:persist should be >=0 before persist, got: {}",
|
"TTL exp:persist should be >=0 before persist, got: {}",
|
||||||
ttl_pre
|
ttl_pre
|
||||||
);
|
);
|
||||||
@@ -665,7 +687,11 @@ async fn test_10_expire_pexpire_persist() {
|
|||||||
assert_contains(&ttl_post, "-1", "TTL after PERSIST -> -1 (no expiration)");
|
assert_contains(&ttl_post, "-1", "TTL after PERSIST -> -1 (no expiration)");
|
||||||
// Second persist should return 0 (nothing to remove)
|
// Second persist should return 0 (nothing to remove)
|
||||||
let persist2 = send_cmd(&mut s, &["PERSIST", "exp:persist"]).await;
|
let persist2 = send_cmd(&mut s, &["PERSIST", "exp:persist"]).await;
|
||||||
assert_contains(&persist2, "0", "PERSIST again -> 0 (no expiration to remove)");
|
assert_contains(
|
||||||
|
&persist2,
|
||||||
|
"0",
|
||||||
|
"PERSIST again -> 0 (no expiration to remove)",
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
@@ -678,7 +704,11 @@ async fn test_11_set_with_options() {
|
|||||||
|
|
||||||
// SET with GET on non-existing key -> returns Null, sets value
|
// SET with GET on non-existing key -> returns Null, sets value
|
||||||
let set_get1 = send_cmd(&mut s, &["SET", "s1", "v1", "GET"]).await;
|
let set_get1 = send_cmd(&mut s, &["SET", "s1", "v1", "GET"]).await;
|
||||||
assert_contains(&set_get1, "$-1", "SET s1 v1 GET returns Null when key didn't exist");
|
assert_contains(
|
||||||
|
&set_get1,
|
||||||
|
"$-1",
|
||||||
|
"SET s1 v1 GET returns Null when key didn't exist",
|
||||||
|
);
|
||||||
let g1 = send_cmd(&mut s, &["GET", "s1"]).await;
|
let g1 = send_cmd(&mut s, &["GET", "s1"]).await;
|
||||||
assert_contains(&g1, "v1", "GET s1 after first SET");
|
assert_contains(&g1, "v1", "GET s1 after first SET");
|
||||||
|
|
||||||
@@ -877,9 +907,16 @@ async fn test_14_expireat_pexpireat() {
|
|||||||
let mut s = connect(port).await;
|
let mut s = connect(port).await;
|
||||||
|
|
||||||
// EXPIREAT: seconds since epoch
|
// EXPIREAT: seconds since epoch
|
||||||
let now_secs = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() as i64;
|
let now_secs = SystemTime::now()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.unwrap()
|
||||||
|
.as_secs() as i64;
|
||||||
let _ = send_cmd(&mut s, &["SET", "exp:at:s", "v"]).await;
|
let _ = send_cmd(&mut s, &["SET", "exp:at:s", "v"]).await;
|
||||||
let exat = send_cmd(&mut s, &["EXPIREAT", "exp:at:s", &format!("{}", now_secs + 1)]).await;
|
let exat = send_cmd(
|
||||||
|
&mut s,
|
||||||
|
&["EXPIREAT", "exp:at:s", &format!("{}", now_secs + 1)],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert_contains(&exat, "1", "EXPIREAT exp:at:s now+1s -> 1 (applied)");
|
assert_contains(&exat, "1", "EXPIREAT exp:at:s now+1s -> 1 (applied)");
|
||||||
let ttl1 = send_cmd(&mut s, &["TTL", "exp:at:s"]).await;
|
let ttl1 = send_cmd(&mut s, &["TTL", "exp:at:s"]).await;
|
||||||
assert!(
|
assert!(
|
||||||
@@ -889,12 +926,23 @@ async fn test_14_expireat_pexpireat() {
|
|||||||
);
|
);
|
||||||
sleep(Duration::from_millis(1200)).await;
|
sleep(Duration::from_millis(1200)).await;
|
||||||
let exists_after_exat = send_cmd(&mut s, &["EXISTS", "exp:at:s"]).await;
|
let exists_after_exat = send_cmd(&mut s, &["EXISTS", "exp:at:s"]).await;
|
||||||
assert_contains(&exists_after_exat, "0", "EXISTS exp:at:s after EXPIREAT expiry -> 0");
|
assert_contains(
|
||||||
|
&exists_after_exat,
|
||||||
|
"0",
|
||||||
|
"EXISTS exp:at:s after EXPIREAT expiry -> 0",
|
||||||
|
);
|
||||||
|
|
||||||
// PEXPIREAT: milliseconds since epoch
|
// PEXPIREAT: milliseconds since epoch
|
||||||
let now_ms = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_millis() as i64;
|
let now_ms = SystemTime::now()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.unwrap()
|
||||||
|
.as_millis() as i64;
|
||||||
let _ = send_cmd(&mut s, &["SET", "exp:at:ms", "v"]).await;
|
let _ = send_cmd(&mut s, &["SET", "exp:at:ms", "v"]).await;
|
||||||
let pexat = send_cmd(&mut s, &["PEXPIREAT", "exp:at:ms", &format!("{}", now_ms + 450)]).await;
|
let pexat = send_cmd(
|
||||||
|
&mut s,
|
||||||
|
&["PEXPIREAT", "exp:at:ms", &format!("{}", now_ms + 450)],
|
||||||
|
)
|
||||||
|
.await;
|
||||||
assert_contains(&pexat, "1", "PEXPIREAT exp:at:ms now+450ms -> 1 (applied)");
|
assert_contains(&pexat, "1", "PEXPIREAT exp:at:ms now+450ms -> 1 (applied)");
|
||||||
let ttl2 = send_cmd(&mut s, &["TTL", "exp:at:ms"]).await;
|
let ttl2 = send_cmd(&mut s, &["TTL", "exp:at:ms"]).await;
|
||||||
assert!(
|
assert!(
|
||||||
@@ -904,5 +952,9 @@ async fn test_14_expireat_pexpireat() {
|
|||||||
);
|
);
|
||||||
sleep(Duration::from_millis(600)).await;
|
sleep(Duration::from_millis(600)).await;
|
||||||
let exists_after_pexat = send_cmd(&mut s, &["EXISTS", "exp:at:ms"]).await;
|
let exists_after_pexat = send_cmd(&mut s, &["EXISTS", "exp:at:ms"]).await;
|
||||||
assert_contains(&exists_after_pexat, "0", "EXISTS exp:at:ms after PEXPIREAT expiry -> 0");
|
assert_contains(
|
||||||
|
&exists_after_pexat,
|
||||||
|
"0",
|
||||||
|
"EXISTS exp:at:ms after PEXPIREAT expiry -> 0",
|
||||||
|
);
|
||||||
}
|
}
|
Reference in New Issue
Block a user