update documentation about 0.db admin db + symmetric encryption + include RPC examples + asymmetric transpart named key instances for encryption and signatures
This commit is contained in:
		
							
								
								
									
										43
									
								
								README.md
									
									
									
									
									
								
							
							
						
						
									
										43
									
								
								README.md
									
									
									
									
									
								
							| @@ -17,6 +17,8 @@ The main purpose of HeroDB is to offer a lightweight, embeddable, and Redis-comp | ||||
| - **Expiration**: Time-to-live (TTL) functionality for keys. | ||||
| - **Scanning**: Cursor-based iteration for keys and hash fields (`SCAN`, `HSCAN`). | ||||
| - **AGE Cryptography Commands**: HeroDB-specific extensions for cryptographic operations. | ||||
| - **Symmetric Encryption**: Stateless symmetric encryption using XChaCha20-Poly1305. | ||||
| - **Admin Database 0**: Centralized control for database management, access control, and per-database encryption. | ||||
|  | ||||
| ## Quick Start | ||||
|  | ||||
| @@ -30,31 +32,14 @@ cargo build --release | ||||
|  | ||||
| ### Running HeroDB | ||||
|  | ||||
| You can start HeroDB with different backends and encryption options: | ||||
|  | ||||
| #### Default `redb` Backend | ||||
| Launch HeroDB with the required `--admin-secret` flag, which encrypts the admin database (DB 0) and authorizes admin access. Optional flags include `--dir` for the database directory, `--port` for the TCP port (default 6379), `--sled` for the sled backend, and `--enable-rpc` to start the JSON-RPC management server on port 8080. | ||||
|  | ||||
| Example: | ||||
| ```bash | ||||
| ./target/release/herodb --dir /tmp/herodb_redb --port 6379 | ||||
| ./target/release/herodb --dir /tmp/herodb --admin-secret myadminsecret --port 6379 --enable-rpc | ||||
| ``` | ||||
|  | ||||
| #### `sled` Backend | ||||
|  | ||||
| ```bash | ||||
| ./target/release/herodb --dir /tmp/herodb_sled --port 6379 --sled | ||||
| ``` | ||||
|  | ||||
| #### `redb` with Encryption | ||||
|  | ||||
| ```bash | ||||
| ./target/release/herodb --dir /tmp/herodb_encrypted --port 6379 --encrypt --encryption_key mysecretkey | ||||
| ``` | ||||
|  | ||||
| #### `sled` with Encryption | ||||
|  | ||||
| ```bash | ||||
| ./target/release/herodb --dir /tmp/herodb_sled_encrypted --port 6379 --sled --encrypt --encryption_key mysecretkey | ||||
| ``` | ||||
| For detailed launch options, see [Basics](docs/basics.md). | ||||
|  | ||||
| ## Usage with Redis Clients | ||||
|  | ||||
| @@ -76,10 +61,24 @@ redis-cli -p 6379 SCAN 0 MATCH user:* COUNT 10 | ||||
| #    2) 1) "user:1" | ||||
| ``` | ||||
|  | ||||
| ## Cryptography | ||||
|  | ||||
| HeroDB supports asymmetric encryption/signatures via AGE commands (X25519 for encryption, Ed25519 for signatures) in stateless or key-managed modes, and symmetric encryption via SYM commands. Keys are persisted in the admin database (DB 0) for managed modes. | ||||
|  | ||||
| For details, see [AGE Cryptography](docs/age.md) and [Basics](docs/basics.md). | ||||
|  | ||||
| ## Database Management | ||||
|  | ||||
| Databases are managed via JSON-RPC API, with metadata stored in the encrypted admin database (DB 0). Databases are public by default upon creation; use RPC to set them private, requiring access keys for SELECT operations (read or readwrite based on permissions). This includes per-database encryption keys, access control, and lifecycle management. | ||||
|  | ||||
| For examples, see [JSON-RPC Examples](docs/rpc_examples.md) and [Admin DB 0 Model](docs/admin.md). | ||||
|  | ||||
| ## Documentation | ||||
|  | ||||
| For more detailed information on commands, features, and advanced usage, please refer to the documentation: | ||||
|  | ||||
| - [Basics](docs/basics.md) | ||||
| - [Supported Commands](docs/cmds.md) | ||||
| - [AGE Cryptography](docs/age.md) | ||||
| - [AGE Cryptography](docs/age.md) | ||||
| - [Admin DB 0 Model (access control, per-db encryption)](docs/admin.md) | ||||
| - [JSON-RPC Examples (management API)](docs/rpc_examples.md) | ||||
							
								
								
									
										181
									
								
								docs/admin.md
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										181
									
								
								docs/admin.md
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,181 @@ | ||||
| # Admin Database 0 (`0.db`) | ||||
|  | ||||
| This page explains what the Admin Database `DB 0` is, why HeroDB uses it, and how to work with it as a developer and end-user. It’s a practical guide covering how databases are created, listed, secured with access keys, and encrypted using per-database secrets. | ||||
|  | ||||
| ## What is `DB 0`? | ||||
|  | ||||
| `DB 0` is the control-plane for a HeroDB instance. It stores metadata for all user databases (`db_id >= 1`) so the server can: | ||||
| - Know which databases exist (without scanning the filesystem) | ||||
| - Enforce access control (public/private with access keys) | ||||
| - Enforce per-database encryption (whether a given database must be opened encrypted and with which write-only key) | ||||
|  | ||||
| `DB 0` itself is always encrypted with the admin secret (the process-level secret provided at startup). | ||||
|  | ||||
| ## How `DB 0` is created and secured | ||||
|  | ||||
| - `DB 0` lives at `<base_dir>/0.db` | ||||
| - It is always encrypted using the `admin secret` provided at process startup (using the `--admin-secret <secret>` CLI flag) | ||||
| - Only clients that provide the correct admin secret can `SELECT 0` (see “`SELECT` + `KEY`” below) | ||||
|  | ||||
| At startup, the server bootstraps `DB 0` (initializes counters and structures) if it’s missing. | ||||
|  | ||||
| ## Metadata stored in `DB 0` | ||||
|  | ||||
| Keys in `DB 0` (internal layout, but useful to understand how things work): | ||||
|  | ||||
| - `admin:next_id` | ||||
|   - String counter holding the next id to allocate (initialized to `"1"`) | ||||
|  | ||||
| - `admin:dbs` | ||||
|   - A hash acting as a set of existing database ids | ||||
|   - field = id (as string), value = `"1"` | ||||
|  | ||||
| - `meta:db:<id>` | ||||
|   - A hash holding db-level metadata | ||||
|   - field `public` = `"true"` or `"false"` (defaults to `true` if missing) | ||||
|  | ||||
| - `meta:db:<id>:keys` | ||||
|   - A hash mapping access-key hashes to the string `Permission:created_at_seconds` | ||||
|   - Examples: `Read:1713456789` or `ReadWrite:1713456789` | ||||
|   - The plaintext access keys are never stored; only their `SHA-256` hashes are kept | ||||
|  | ||||
| - `meta:db:<id>:enc` | ||||
|    - A string holding the per-database encryption key used to open `<id>.db` encrypted | ||||
|    - This value is write-only from the perspective of the management APIs (it’s set at creation and never returned) | ||||
|  | ||||
| - `age:key:<name>` | ||||
|    - Base64-encoded X25519 recipient (public encryption key) for named AGE keys | ||||
| - `age:privkey:<name>` | ||||
|    - Base64-encoded X25519 identity (secret encryption key) for named AGE keys | ||||
| - `age:signpub:<name>` | ||||
|    - Base64-encoded Ed25519 verify public key for named AGE keys | ||||
| - `age:signpriv:<name>` | ||||
|    - Base64-encoded Ed25519 signing secret key for named AGE keys | ||||
|  | ||||
| > You don’t need to manipulate these keys directly; they’re listed to clarify the model. AGE keys are managed via AGE commands. | ||||
|  | ||||
| ## Database lifecycle | ||||
|  | ||||
| 1) Create a database (via JSON-RPC) | ||||
| - The server allocates an id from `admin:next_id`, registers it in `admin:dbs`, and defaults the database to `public=true` | ||||
| - If you pass an optional `encryption_key` during creation, the server persists it in `meta:db:<id>:enc`. That database will be opened in encrypted mode from then on | ||||
|  | ||||
| 2) Open and use a database | ||||
| - Clients select a database over RESP using `SELECT` | ||||
| - Authorization and encryption state are enforced using `DB 0` metadata | ||||
|  | ||||
| 3) Delete database files | ||||
| - Removing `<id>.db` removes the physical storage | ||||
| - `DB 0` remains the source of truth for existence and may be updated by future management methods as the system evolves | ||||
|  | ||||
| ## Access control model | ||||
|  | ||||
| - Public database (default) | ||||
|   - Anyone can `SELECT <id>` with no key, and will get `ReadWrite` permission | ||||
| - Private database | ||||
|   - You must provide an access key when selecting the database | ||||
|   - The server hashes the provided key with `SHA-256` and checks membership in `meta:db:<id>:keys` | ||||
|   - Permissions are `Read` or `ReadWrite` depending on how the key was added | ||||
| - Admin `DB 0` | ||||
|   - Requires the exact admin secret as the `KEY` argument to `SELECT 0` | ||||
|   - Permission is `ReadWrite` when the secret matches | ||||
|  | ||||
| ### How to select databases with optional `KEY` | ||||
|  | ||||
| - Public DB (no key required) | ||||
|   - `SELECT <id>` | ||||
|  | ||||
| - Private DB (access key required) | ||||
|   - `SELECT <id> KEY <plaintext_key>` | ||||
|  | ||||
| - Admin `DB 0` (admin secret required) | ||||
|   - `SELECT 0 KEY <admin_secret>` | ||||
|  | ||||
| Examples (using `redis-cli`): | ||||
| ```bash | ||||
| # Public database | ||||
| redis-cli -p $PORT SELECT 1 | ||||
| # → OK | ||||
|  | ||||
| # Private database | ||||
| redis-cli -p $PORT SELECT 2 KEY my-db2-access-key | ||||
| # → OK | ||||
|  | ||||
| # Admin DB 0 | ||||
| redis-cli -p $PORT SELECT 0 KEY my-admin-secret | ||||
| # → OK | ||||
| ``` | ||||
|  | ||||
| ## Per-database encryption | ||||
|  | ||||
| - At database creation, you can provide an optional per-db encryption key | ||||
| - If provided, the server persists that key in `DB 0` as `meta:db:<id>:enc` | ||||
| - When you later open the database, the engine checks whether `meta:db:<id>:enc` exists to decide if it must open `<id>.db` in encrypted mode | ||||
| - The per-db key is not returned by RPC—it is considered write-only configuration data | ||||
|  | ||||
| Operationally: | ||||
| - Create with encryption: pass a non-null `encryption_key` to the `createDatabase` RPC | ||||
| - Open later: simply `SELECT` the database; encryption is transparent to clients | ||||
|  | ||||
| ## Management via JSON-RPC | ||||
|  | ||||
| You can manage databases using the management RPC (namespaced `herodb.*`). Typical operations: | ||||
| - `createDatabase(backend, config, encryption_key?)` | ||||
|   - Allocates a new id, sets optional encryption key | ||||
| - `listDatabases()` | ||||
|   - Lists database ids and info (including whether storage is currently encrypted) | ||||
| - `getDatabaseInfo(db_id)` | ||||
|   - Returns details: backend, encrypted flag, size on disk, `key_count`, timestamps, etc. | ||||
| - `addAccessKey(db_id, key, permissions)` | ||||
|   - Adds a `Read` or `ReadWrite` access key (permissions = `"read"` | `"readwrite"`) | ||||
| - `listAccessKeys(db_id)` | ||||
|   - Returns hashes and permissions; you can use these hashes to delete keys | ||||
| - `deleteAccessKey(db_id, key_hash)` | ||||
|   - Removes a key by its hash | ||||
| - `setDatabasePublic(db_id, public)` | ||||
|   - Toggles public/private | ||||
|  | ||||
| Copyable JSON examples are provided in the [RPC examples documentation](./rpc_examples.md). | ||||
|  | ||||
| ## Typical flows | ||||
|  | ||||
| 1) Public, unencrypted database | ||||
| - Create a new database without an encryption key | ||||
| - Clients can immediately `SELECT <id>` without a key | ||||
| - You can later make it private and add keys if needed | ||||
|  | ||||
| 2) Private, encrypted database | ||||
| - Create passing an `encryption_key` | ||||
| - Mark it private (`setDatabasePublic false`) and add access keys | ||||
| - Clients must use `SELECT <id> KEY <plaintext_access_key>` | ||||
| - Storage opens in encrypted mode automatically | ||||
|  | ||||
| ## Security notes | ||||
|  | ||||
| - Only `SHA-256` hashes of access keys are stored in `DB 0`; keep plaintext keys safe on the client side | ||||
| - The per-db encryption key is never exposed via the API after it is set | ||||
| - The admin secret must be kept secure; anyone with it can `SELECT 0` and perform administrative actions | ||||
|  | ||||
| ## Troubleshooting | ||||
|  | ||||
| - `ERR invalid access key` when selecting a private db | ||||
|   - Ensure you passed the `KEY` argument: `SELECT <id> KEY <plaintext_key>` | ||||
|   - If you recently added the key, confirm the permissions and that you used the exact plaintext (hash must match) | ||||
|  | ||||
| - `Database X not found` | ||||
|   - The id isn’t registered in `DB 0` (`admin:dbs`). Use the management APIs to create or list databases | ||||
|  | ||||
| - Cannot `SELECT 0` | ||||
|   - The `KEY` must be the exact admin secret passed at server startup | ||||
|  | ||||
| ## Reference | ||||
|  | ||||
| - Admin metadata lives in `DB 0` (`0.db`) and controls: | ||||
|   - Existence: `admin:dbs` | ||||
|   - Access: `meta:db:<id>.public` and `meta:db:<id>:keys` | ||||
|   - Encryption: `meta:db:<id>:enc` | ||||
|  | ||||
| For command examples and management payloads: | ||||
| - RESP command basics: `docs/basics.md` | ||||
| - Supported commands: `docs/cmds.md` | ||||
| - JSON-RPC examples: `docs/rpc_examples.md` | ||||
							
								
								
									
										242
									
								
								docs/age.md
									
									
									
									
									
								
							
							
						
						
									
										242
									
								
								docs/age.md
									
									
									
									
									
								
							| @@ -1,188 +1,96 @@ | ||||
| # HeroDB AGE usage: Stateless vs Key‑Managed | ||||
| # HeroDB AGE Cryptography | ||||
|  | ||||
| This document explains how to use the AGE cryptography commands exposed by HeroDB over the Redis protocol in two modes: | ||||
| - Stateless (ephemeral keys; nothing stored on the server) | ||||
| - Key‑managed (server‑persisted, named keys) | ||||
| HeroDB provides AGE-based asymmetric encryption and digital signatures over the Redis protocol using X25519 for encryption and Ed25519 for signatures. Keys can be used in stateless (ephemeral) or key-managed (persistent, named) modes. | ||||
|  | ||||
| If you are new to the codebase, the exact tests that exercise these behaviors are: | ||||
| - [rust.test_07_age_stateless_suite()](herodb/tests/usage_suite.rs:495) | ||||
| - [rust.test_08_age_persistent_named_suite()](herodb/tests/usage_suite.rs:555) | ||||
| In key-managed mode, HeroDB uses a unified keypair concept: a single Ed25519 signing key is deterministically derived into X25519 keys for encryption, allowing one keypair to handle both encryption and signatures transparently. | ||||
|  | ||||
| Implementation entry points: | ||||
| - [herodb/src/age.rs](herodb/src/age.rs) | ||||
| - Dispatch from [herodb/src/cmd.rs](herodb/src/cmd.rs) | ||||
| ## Cryptographic Algorithms | ||||
|  | ||||
| Note: Database-at-rest encryption flags in the test harness are unrelated to AGE commands; those flags control storage-level encryption of DB files. See the harness near [rust.start_test_server()](herodb/tests/usage_suite.rs:10). | ||||
| ### X25519 (Encryption) | ||||
| - Elliptic-curve Diffie-Hellman key exchange for symmetric key derivation. | ||||
| - Used for encrypting/decrypting messages. | ||||
|  | ||||
| ## Quick start | ||||
| ### Ed25519 (Signatures) | ||||
| - EdDSA digital signatures for message authentication. | ||||
| - Used for signing/verifying messages. | ||||
|  | ||||
| Assuming the server is running on localhost on some $PORT: | ||||
| ### Key Derivation | ||||
| Ed25519 signing keys are deterministically converted to X25519 keys for encryption. This enables a single keypair to support both operations without additional keys. Derivation uses the Ed25519 secret scalar clamped for X25519. | ||||
|  | ||||
| In named keypairs, Ed25519 keys are stored, and X25519 keys are derived on-demand and cached. | ||||
|  | ||||
| ## Stateless Mode (Ephemeral Keys) | ||||
| No server-side storage; keys are provided with each command. | ||||
|  | ||||
| Available commands: | ||||
| - `AGE GENENC`: Generate ephemeral X25519 keypair. Returns `[recipient, identity]`. | ||||
| - `AGE GENSIGN`: Generate ephemeral Ed25519 keypair. Returns `[verify_pub, sign_secret]`. | ||||
| - `AGE ENCRYPT <recipient> <message>`: Encrypt message. Returns base64 ciphertext. | ||||
| - `AGE DECRYPT <identity> <ciphertext_b64>`: Decrypt ciphertext. Returns plaintext. | ||||
| - `AGE SIGN <sign_secret> <message>`: Sign message. Returns base64 signature. | ||||
| - `AGE VERIFY <verify_pub> <message> <signature_b64>`: Verify signature. Returns 1 (valid) or 0 (invalid). | ||||
|  | ||||
| Example: | ||||
| ```bash | ||||
| ~/code/git.ourworld.tf/herocode/herodb/herodb/build.sh | ||||
| ~/code/git.ourworld.tf/herocode/herodb/target/release/herodb --dir /tmp/data --debug --$PORT 6381 --encryption-key 1234 --encrypt | ||||
| ``` | ||||
| redis-cli AGE GENENC | ||||
| # → 1) "age1qz..."  # recipient (X25519 public) | ||||
| #    2) "AGE-SECRET-KEY-1..."  # identity (X25519 secret) | ||||
|  | ||||
| redis-cli AGE ENCRYPT "age1qz..." "hello" | ||||
| # → base64_ciphertext | ||||
|  | ||||
| ```bash | ||||
| export PORT=6381 | ||||
| # Generate an ephemeral keypair and encrypt/decrypt a message (stateless mode) | ||||
| redis-cli -p $PORT AGE GENENC | ||||
| # → returns an array: [recipient, identity] | ||||
|  | ||||
| redis-cli -p $PORT AGE ENCRYPT <recipient> "hello world" | ||||
| # → returns ciphertext (base64 in a bulk string) | ||||
|  | ||||
| redis-cli -p $PORT AGE DECRYPT <identity> <ciphertext_b64> | ||||
| # → returns "hello world" | ||||
| ``` | ||||
|  | ||||
| For key‑managed mode, generate a named key once and reference it by name afterwards: | ||||
|  | ||||
| ```bash | ||||
| redis-cli -p $PORT AGE KEYGEN app1 | ||||
| # → persists encryption keypair under name "app1" | ||||
|  | ||||
| redis-cli -p $PORT AGE ENCRYPTNAME app1 "hello" | ||||
| redis-cli -p $PORT AGE DECRYPTNAME app1 <ciphertext_b64> | ||||
| ``` | ||||
|  | ||||
| ## Stateless AGE (ephemeral) | ||||
|  | ||||
| Characteristics | ||||
|  | ||||
| - No server‑side storage of keys. | ||||
| - You pass the actual key material with every call. | ||||
| - Not listable via AGE LIST. | ||||
|  | ||||
| Commands and examples | ||||
|  | ||||
| 1) Ephemeral encryption keys | ||||
|  | ||||
| ```bash | ||||
| # Generate an ephemeral encryption keypair  | ||||
| redis-cli -p $PORT AGE GENENC | ||||
| # Example output (abridged): | ||||
| # 1) "age1qz..."          # recipient (public key) = can be used by others e.g. to verify what I sign | ||||
| # 2) "AGE-SECRET-KEY-1..." # identity (secret) = is like my private, cannot lose this one | ||||
|  | ||||
| # Encrypt with the recipient public key | ||||
| redis-cli -p $PORT AGE ENCRYPT "age1qz..." "hello world" | ||||
|  | ||||
| # → returns bulk string payload: base64 ciphertext (encrypted content) | ||||
|  | ||||
| # Decrypt with the identity (secret) in other words your private key | ||||
| redis-cli -p $PORT AGE DECRYPT "AGE-SECRET-KEY-1..." "<ciphertext_b64>" | ||||
| # → "hello world" | ||||
| ``` | ||||
|  | ||||
| 2) Ephemeral signing keys | ||||
|  | ||||
| > ? is this same as my private key | ||||
|  | ||||
| ```bash | ||||
|  | ||||
| # Generate an ephemeral signing keypair | ||||
| redis-cli -p $PORT AGE GENSIGN | ||||
| # Example output: | ||||
| # 1) "<verify_pub_b64>" | ||||
| # 2) "<sign_secret_b64>" | ||||
|  | ||||
| # Sign a message with the secret | ||||
| redis-cli -p $PORT AGE SIGN "<sign_secret_b64>" "msg" | ||||
| # → returns "<signature_b64>" | ||||
|  | ||||
| # Verify with the public key | ||||
| redis-cli -p $PORT AGE VERIFY "<verify_pub_b64>" "msg" "<signature_b64>" | ||||
| # → 1 (valid) or 0 (invalid) | ||||
| ``` | ||||
|  | ||||
| When to use | ||||
| - You do not want the server to store private keys. | ||||
| - You already manage key material on the client side. | ||||
| - You need ad‑hoc operations without persistence. | ||||
|  | ||||
| Reference test: [rust.test_07_age_stateless_suite()](herodb/tests/usage_suite.rs:495) | ||||
|  | ||||
| ## Key‑managed AGE (persistent, named) | ||||
|  | ||||
| Characteristics | ||||
| - Server generates and persists keypairs under a chosen name. | ||||
| - Clients refer to keys by name; raw secrets are not supplied on each call. | ||||
| - Keys are discoverable via AGE LIST. | ||||
|  | ||||
| Commands and examples | ||||
|  | ||||
| 1) Named encryption keys | ||||
|  | ||||
| ```bash | ||||
| # Create/persist a named encryption keypair | ||||
| redis-cli -p $PORT AGE KEYGEN app1 | ||||
| # → returns [recipient, identity] but also stores them under name "app1" | ||||
|  | ||||
| > TODO: should not return identity (security, but there can be separate function to export it e.g. AGE EXPORTKEY app1) | ||||
|  | ||||
| # Encrypt using the stored public key | ||||
| redis-cli -p $PORT AGE ENCRYPTNAME app1 "hello" | ||||
| # → returns bulk string payload: base64 ciphertext | ||||
|  | ||||
| # Decrypt using the stored secret | ||||
| redis-cli -p $PORT AGE DECRYPTNAME app1 "<ciphertext_b64>" | ||||
| redis-cli AGE DECRYPT "AGE-SECRET-KEY-1..." base64_ciphertext | ||||
| # → "hello" | ||||
| ``` | ||||
|  | ||||
| 2) Named signing keys | ||||
| ## Key-Managed Mode (Persistent Named Keys) | ||||
| Keys are stored server-side under names. Supports unified keypairs for both encryption and signatures. | ||||
|  | ||||
| Available commands: | ||||
| - `AGE KEYGEN <name>`: Generate and store unified keypair. Returns `[recipient, identity]` in age format. | ||||
| - `AGE SIGNKEYGEN <name>`: Generate and store Ed25519 signing keypair. Returns `[verify_pub, sign_secret]`. | ||||
| - `AGE ENCRYPTNAME <name> <message>`: Encrypt with named key. Returns base64 ciphertext. | ||||
| - `AGE DECRYPTNAME <name> <ciphertext_b64>`: Decrypt with named key. Returns plaintext. | ||||
| - `AGE SIGNNAME <name> <message>`: Sign with named key. Returns base64 signature. | ||||
| - `AGE VERIFYNAME <name> <message> <signature_b64>`: Verify with named key. Returns 1 or 0. | ||||
| - `AGE LIST`: List all stored key names. Returns sorted array of names. | ||||
|  | ||||
| ### AGE LIST Output | ||||
| Returns a flat, deduplicated, sorted array of key names (strings). Each name corresponds to a stored keypair, which may include encryption keys (X25519), signing keys (Ed25519), or both. | ||||
|  | ||||
| Output format: `["name1", "name2", ...]` | ||||
|  | ||||
| Example: | ||||
| ```bash | ||||
| # Create/persist a named signing keypair | ||||
| redis-cli -p $PORT AGE SIGNKEYGEN app1 | ||||
| # → returns [verify_pub_b64, sign_secret_b64] and stores under name "app1" | ||||
|  | ||||
| > TODO: should not return sign_secret_b64 (for security, but there can be separate function to export it e.g. AGE EXPORTSIGNKEY app1) | ||||
|  | ||||
| # Sign using the stored secret | ||||
| redis-cli -p $PORT AGE SIGNNAME app1 "msg" | ||||
| # → returns "<signature_b64>" | ||||
|  | ||||
| # Verify using the stored public key | ||||
| redis-cli -p $PORT AGE VERIFYNAME app1 "msg" "<signature_b64>" | ||||
| # → 1 (valid) or 0 (invalid) | ||||
| redis-cli AGE LIST | ||||
| # →  1) "<named_keypair_1>" | ||||
| #    2) "<named_keypair_2>" | ||||
| ``` | ||||
|  | ||||
| 3) List stored AGE keys | ||||
| For unified keypairs (from `AGE KEYGEN`), the name handles both encryption (derived X25519) and signatures (stored Ed25519) transparently. | ||||
|  | ||||
| Example with named keys: | ||||
| ```bash | ||||
| redis-cli -p $PORT AGE LIST | ||||
| # Example output includes labels such as "encpub" and your key names (e.g., "app1") | ||||
| redis-cli AGE KEYGEN app1 | ||||
| # →  1) "age1..."  # recipient | ||||
| #    2) "AGE-SECRET-KEY-1..."  # identity | ||||
|  | ||||
| redis-cli AGE ENCRYPTNAME app1 "secret message" | ||||
| # → base64_ciphertext | ||||
|  | ||||
| redis-cli AGE DECRYPTNAME app1 base64_ciphertext | ||||
| # → "secret message" | ||||
|  | ||||
| redis-cli AGE SIGNNAME app1 "message" | ||||
| # → base64_signature | ||||
|  | ||||
| redis-cli AGE VERIFYNAME app1 "message" base64_signature | ||||
| # → 1 | ||||
| ``` | ||||
|  | ||||
| When to use | ||||
| - You want centralized key storage/rotation and fewer secrets on the client. | ||||
| - You need names/labels for workflows and can trust the server with secrets. | ||||
| - You want discoverability (AGE LIST) and simpler client commands. | ||||
| ## Choosing a Mode | ||||
| - **Stateless**: For ad-hoc operations without persistence; client manages keys. | ||||
| - **Key-managed**: For centralized key lifecycle; server stores keys for convenience and discoverability. | ||||
|  | ||||
| Reference test: [rust.test_08_age_persistent_named_suite()](herodb/tests/usage_suite.rs:555) | ||||
|  | ||||
| ## Choosing a mode | ||||
|  | ||||
| - Prefer Stateless when: | ||||
|   - Minimizing server trust for secret material is the priority. | ||||
|   - Clients already have a secure mechanism to store/distribute keys. | ||||
| - Prefer Key‑managed when: | ||||
|   - Centralized lifecycle, naming, and discoverability are beneficial. | ||||
|   - You plan to integrate rotation, ACLs, or auditability on the server side. | ||||
|  | ||||
| ## Security notes | ||||
|  | ||||
| - Treat identities and signing secrets as sensitive; avoid logging them. | ||||
| - For key‑managed mode, ensure server storage (and backups) are protected. | ||||
| - AGE operations here are application‑level crypto and are distinct from database-at-rest encryption configured in the test harness. | ||||
|  | ||||
| ## Repository pointers | ||||
|  | ||||
| - Stateless examples in tests: [rust.test_07_age_stateless_suite()](herodb/tests/usage_suite.rs:495) | ||||
| - Key‑managed examples in tests: [rust.test_08_age_persistent_named_suite()](herodb/tests/usage_suite.rs:555) | ||||
| - AGE implementation: [herodb/src/age.rs](herodb/src/age.rs) | ||||
| - Command dispatch: [herodb/src/cmd.rs](herodb/src/cmd.rs) | ||||
| - Bash demo: [herodb/examples/age_bash_demo.sh](herodb/examples/age_bash_demo.sh) | ||||
| - Rust persistent demo: [herodb/examples/age_persist_demo.rs](herodb/examples/age_persist_demo.rs) | ||||
| - Additional notes: [herodb/instructions/encrypt.md](herodb/instructions/encrypt.md) | ||||
| Implementation: [herodb/src/age.rs](herodb/src/age.rs) <br>  | ||||
| Tests: [herodb/tests/usage_suite.rs](herodb/tests/usage_suite.rs) | ||||
							
								
								
									
										103
									
								
								docs/basics.md
									
									
									
									
									
								
							
							
						
						
									
										103
									
								
								docs/basics.md
									
									
									
									
									
								
							| @@ -1,4 +1,58 @@ | ||||
| Here's an expanded version of the cmds.md documentation to include the list commands: | ||||
| # HeroDB Basics | ||||
|  | ||||
| ## Launching HeroDB | ||||
|  | ||||
| To launch HeroDB, use the binary with required and optional flags. The `--admin-secret` flag is mandatory, encrypting the admin database (DB 0) and authorizing admin access. | ||||
|  | ||||
| ### Launch Flags | ||||
| - `--dir <path>`: Directory for database files (default: current directory). | ||||
| - `--port <port>`: TCP port for Redis protocol (default: 6379). | ||||
| - `--debug`: Enable debug logging. | ||||
| - `--sled`: Use Sled backend (default: Redb). | ||||
| - `--enable-rpc`: Start JSON-RPC management server on port 8080. | ||||
| - `--rpc-port <port>`: Custom RPC port (default: 8080). | ||||
| - `--admin-secret <secret>`: Required secret for DB 0 encryption and admin access. | ||||
|  | ||||
| Example: | ||||
| ```bash | ||||
| ./target/release/herodb --dir /tmp/herodb --admin-secret mysecret --port 6379 --enable-rpc | ||||
| ``` | ||||
|  | ||||
| Deprecated flags (`--encrypt`, `--encryption-key`) are ignored for data DBs; per-database encryption is managed via RPC. | ||||
|  | ||||
| ## Admin Database (DB 0) | ||||
|  | ||||
| DB 0 acts as the administrative database instance, storing metadata for all user databases (IDs >= 1). It controls existence, access control, and per-database encryption. DB 0 is always encrypted with the `--admin-secret`. | ||||
|  | ||||
| When creating a new database, DB 0 allocates an ID, registers it, and optionally stores a per-database encryption key (write-only). Databases are public by default; use RPC to set them private, requiring access keys for SELECT (read or readwrite based on permissions). Keys are persisted in DB 0 for managed AGE operations. | ||||
|  | ||||
| Access DB 0 with `SELECT 0 KEY <admin-secret>`. | ||||
|  | ||||
| ## Symmetric Encryption | ||||
|  | ||||
| HeroDB supports stateless symmetric encryption via SYM commands, using XChaCha20-Poly1305 AEAD. | ||||
|  | ||||
| Commands: | ||||
| - `SYM KEYGEN`: Generate 32-byte key. Returns base64-encoded key. | ||||
| - `SYM ENCRYPT <key_b64> <message>`: Encrypt message. Returns base64 ciphertext. | ||||
| - `SYM DECRYPT <key_b64> <ciphertext_b64>`: Decrypt. Returns plaintext. | ||||
|  | ||||
| Example: | ||||
| ```bash | ||||
| redis-cli SYM KEYGEN | ||||
| # → base64_key | ||||
|  | ||||
| redis-cli SYM ENCRYPT base64_key "secret" | ||||
| # → base64_ciphertext | ||||
|  | ||||
| redis-cli SYM DECRYPT base64_key base64_ciphertext | ||||
| # → "secret" | ||||
| ``` | ||||
|  | ||||
| ## RPC Options | ||||
|  | ||||
| Enable the JSON-RPC server with `--enable-rpc` for database management. Methods include creating databases, managing access keys, and setting encryption. See [JSON-RPC Examples](./rpc_examples.md) for payloads. | ||||
|  | ||||
| # HeroDB Commands | ||||
|  | ||||
| HeroDB implements a subset of Redis commands over the Redis protocol. This document describes the available commands and their usage. | ||||
| @@ -575,6 +629,29 @@ redis-cli -p $PORT AGE LIST | ||||
| #    2) "keyname2" | ||||
| ``` | ||||
|  | ||||
| ## SYM Commands | ||||
|  | ||||
| ### SYM KEYGEN | ||||
| Generate a symmetric encryption key. | ||||
| ```bash | ||||
| redis-cli -p $PORT SYM KEYGEN | ||||
| # → base64_encoded_32byte_key | ||||
| ``` | ||||
|  | ||||
| ### SYM ENCRYPT | ||||
| Encrypt a message with a symmetric key. | ||||
| ```bash | ||||
| redis-cli -p $PORT SYM ENCRYPT <key_b64> "message" | ||||
| # → base64_encoded_ciphertext | ||||
| ``` | ||||
|  | ||||
| ### SYM DECRYPT | ||||
| Decrypt a ciphertext with a symmetric key. | ||||
| ```bash | ||||
| redis-cli -p $PORT SYM DECRYPT <key_b64> <ciphertext_b64> | ||||
| # → decrypted_message | ||||
| ``` | ||||
|  | ||||
| ## Server Information Commands | ||||
|  | ||||
| ### INFO | ||||
| @@ -621,3 +698,27 @@ This expanded documentation includes all the list commands that were implemented | ||||
| 10. LINDEX - get element by index | ||||
| 11. LRANGE - get range of elements | ||||
|  | ||||
|  | ||||
| ## Updated Database Selection and Access Keys | ||||
|  | ||||
| HeroDB uses an `Admin DB 0` to control database existence, access, and encryption. Access to data DBs can be public (no key) or private (requires a key). See detailed model in `docs/admin.md`. | ||||
|  | ||||
| Examples: | ||||
|  | ||||
| ```bash | ||||
| # Public database (no key required) | ||||
| redis-cli -p $PORT SELECT 1 | ||||
| # → OK | ||||
| ``` | ||||
|  | ||||
| ```bash | ||||
| # Private database (requires access key) | ||||
| redis-cli -p $PORT SELECT 2 KEY my-db2-access-key | ||||
| # → OK | ||||
| ``` | ||||
|  | ||||
| ```bash | ||||
| # Admin DB 0 (requires admin secret) | ||||
| redis-cli -p $PORT SELECT 0 KEY my-admin-secret | ||||
| # → OK | ||||
| ``` | ||||
|   | ||||
							
								
								
									
										23
									
								
								docs/cmds.md
									
									
									
									
									
								
							
							
						
						
									
										23
									
								
								docs/cmds.md
									
									
									
									
									
								
							| @@ -122,4 +122,27 @@ redis-cli -p 6379 --rdb dump.rdb | ||||
|  | ||||
| # Import to sled | ||||
| redis-cli -p 6381 --pipe < dump.rdb | ||||
| ``` | ||||
|  | ||||
| ## Authentication and Database Selection | ||||
|  | ||||
| HeroDB uses an `Admin DB 0` to govern database existence, access and per-db encryption. Access control is enforced via `Admin DB 0` metadata. See the full model in `docs/admin.md`. | ||||
|  | ||||
| Examples: | ||||
| ```bash | ||||
| # Public database (no key required) | ||||
| redis-cli -p $PORT SELECT 1 | ||||
| # → OK | ||||
| ``` | ||||
|  | ||||
| ```bash | ||||
| # Private database (requires access key) | ||||
| redis-cli -p $PORT SELECT 2 KEY my-db2-access-key | ||||
| # → OK | ||||
| ``` | ||||
|  | ||||
| ```bash | ||||
| # Admin DB 0 (requires admin secret) | ||||
| redis-cli -p $PORT SELECT 0 KEY my-admin-secret | ||||
| # → OK | ||||
| ``` | ||||
							
								
								
									
										141
									
								
								docs/rpc_examples.md
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										141
									
								
								docs/rpc_examples.md
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,141 @@ | ||||
| # HeroDB JSON-RPC Examples | ||||
|  | ||||
| These examples show full JSON-RPC 2.0 payloads for managing HeroDB via the RPC API (enable with `--enable-rpc`). Methods are named as `hero_<function>`. Params are positional arrays; enum values are strings (e.g., `"Redb"`). Copy-paste into Postman or similar clients. | ||||
|  | ||||
| ## Database Management | ||||
|  | ||||
| ### Create Database | ||||
| Creates a new database with optional per-database encryption key (stored write-only in Admin DB 0). | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 1, | ||||
|   "method": "hero_createDatabase", | ||||
|   "params": [ | ||||
|     "Redb", | ||||
|     { "name": null, "storage_path": null, "max_size": null, "redis_version": null }, | ||||
|     null | ||||
|   ] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| With encryption: | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 2, | ||||
|   "method": "hero_createDatabase", | ||||
|   "params": [ | ||||
|     "Sled", | ||||
|     { "name": "secure-db", "storage_path": null, "max_size": null, "redis_version": null }, | ||||
|     "my-per-db-encryption-key" | ||||
|   ] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ### List Databases | ||||
| Returns array of database infos (id, backend, encrypted status, size, etc.). | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 3, | ||||
|   "method": "hero_listDatabases", | ||||
|   "params": [] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ### Get Database Info | ||||
| Retrieves detailed info for a specific database. | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 4, | ||||
|   "method": "hero_getDatabaseInfo", | ||||
|   "params": [1] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ### Delete Database | ||||
| Removes physical database file; metadata remains in Admin DB 0. | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 5, | ||||
|   "method": "hero_deleteDatabase", | ||||
|   "params": [1] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ## Access Control | ||||
|  | ||||
| ### Add Access Key | ||||
| Adds a hashed access key for private databases. Permissions: `"read"` or `"readwrite"`. | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 6, | ||||
|   "method": "hero_addAccessKey", | ||||
|   "params": [2, "my-access-key", "readwrite"] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ### List Access Keys | ||||
| Returns array of key hashes, permissions, and creation timestamps. | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 7, | ||||
|   "method": "hero_listAccessKeys", | ||||
|   "params": [2] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ### Delete Access Key | ||||
| Removes key by its SHA-256 hash. | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 8, | ||||
|   "method": "hero_deleteAccessKey", | ||||
|   "params": [2, "0123abcd...keyhash..."] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ### Set Database Public/Private | ||||
| Toggles public access (default true). Private databases require access keys. | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 9, | ||||
|   "method": "hero_setDatabasePublic", | ||||
|   "params": [2, false] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ## Server Info | ||||
|  | ||||
| ### Get Server Stats | ||||
| Returns stats like total databases and uptime. | ||||
|  | ||||
| ```json | ||||
| { | ||||
|   "jsonrpc": "2.0", | ||||
|   "id": 10, | ||||
|   "method": "hero_getServerStats", | ||||
|   "params": [] | ||||
| } | ||||
| ``` | ||||
|  | ||||
| ## Notes | ||||
| - Per-database encryption keys are write-only; set at creation and used transparently. | ||||
| - Access keys are hashed (SHA-256) for storage; provide plaintext in requests. | ||||
| - Backend options: `"Redb"` (default) or `"Sled"`. | ||||
| - Config object fields (name, storage_path, etc.) are optional and currently ignored but positional. | ||||
		Reference in New Issue
	
	Block a user