- Adds a new mechanism to synchronize the database efficiently
by serializing updates using binary encoding. This improves
performance and reduces bandwidth usage compared to previous methods.
- Introduces `SyncRecord` struct to represent database updates
for easier handling and serialization.
- Implements `push_updates` to serialize database changes since a
given index, handling both initial and incremental syncs.
- Implements `sync_updates` to apply received serialized updates
to the database, robustly handling errors and deletions.
- Added a diagram explaining the architecture of the OurDB
syncer, clarifying the interaction between the HTTP server,
master, and worker nodes.
- Added a README file providing a comprehensive overview of
the OurDB syncer project, including its architecture,
features, prerequisites, installation instructions, and usage
examples.
- Removed outdated Mycelium_Streamer documentation as it's no
longer relevant to the current project structure.
- Created example scripts for running the database, master,
and worker components, simplifying the setup and execution of
the system.
- Added HTTP client and server documentation, clarifying their
functionalities and interaction with the OurDB system.
- Added a client and server for a simple key-value store.
- Improved documentation with client and server usage examples.
- Created client and server implementations using the V language.
- Adds a new lightweight key-value store server implemented in V.
- Includes basic CRUD operations (`set`, `get`, `delete`).
- Provides configurable host and operation restrictions for security.
- Offers middleware for logging and request validation.
- Supports incremental mode for automatic ID generation.
- Includes comprehensive documentation and example usage.
- Adds unit tests to ensure functionality and stability.
- Added a new documentation file explaining the Mycelium Streamer
example, covering setup, prerequisites, and execution.
- This provides users with clear instructions on how to use the
example project.
- Add worker registration to MyceliumStreamer: Allows for explicit
addition of workers, improving management and control.
- Simplify worker message handling: Streamlines message processing
for increased efficiency and readability. Removes unnecessary
logging and simplifies message routing.
- Remove redundant message handling: Eliminates duplicate code
paths for cleaner and more maintainable code.
- Improve worker data retrieval: Facilitates direct data retrieval
from workers, enhancing efficiency and reliability.
- Allow reading data directly from specific workers by specifying
their public key in `MyceliumStreamer.read()`. This improves
data retrieval flexibility and allows for distributed data access.
- Add master data reading to ensure data consistency and allow
comparison between master and worker data. This helps debug and
verify data replication.
- Implement JSON encoding/decoding for database transfer between
master and worker nodes. This enables efficient and structured
data exchange.
- Removed unnecessary test data from `deduped_mycelium_master.vsh`.
- Simplified `MyceliumStreamer.listen()` to efficiently handle
incoming messages, removing redundant code and improving readability.
- Enhanced error handling in `MyceliumStreamer.listen()` for more robust
operation.
- Added worker ID to master and worker configurations for improved
identification and management.
- Implemented worker registration and data synchronization mechanisms
to enable distributed data access.
- Added a read function to retrieve data from specific workers,
enhancing data access flexibility.
- Improved logging for better monitoring and debugging of the system.
- Added continuous data writing and verification to the master node
to ensure data persistence and integrity.
- Simplified worker update handling in the `listen` function for
better efficiency and error handling. The previous implementation
had unnecessary complexity and potential for hangs.
- Changed Mycelium worker port to avoid conflict with master.
- Added debug print statements to Mycelium client for better troubleshooting.
- Removed unnecessary `SyncData` struct, simplifying data handling.
- Updated data encoding/decoding to directly use base64 for efficiency.
- Clarified message topic names for better understanding.
- Refactor database streamer to support multiple workers.
- Add master node to manage and distribute data updates.
- Implement worker nodes to receive and apply updates.
- Remove unnecessary slave node.
- Improve error handling and logging.
- Use base64 encoding for JSON compatibility in data transfer.
- Renamed the topic for database synchronization messages from
'db_sync' to 'sync_db' for clarity.
- Updated the Mycelium slave to decode base64 payload before
processing and to log received messages and their source.
- Added logging to the Mycelium streamer to track sent messages.
- Added a new feature to retrieve and log the last index from the
worker after syncing updates. This improves monitoring and
debugging capabilities.
- Refactor data synchronization logic to use Mycelium messages for
efficient updates between master and worker nodes. This removes
the previous inefficient polling method and simplifies the code.
- Update the slave node to receive and apply updates from the
master, improving synchronization efficiency and robustness.
- Change the default slave port to 9000.
- Rename `db` variable to `worker` for clarity.
- Introduces `MyceliumStreamer` for synchronizing data across a
Mycelium network, enabling distributed data access.
- Allows adding multiple worker nodes to the streamer for data
replication and redundancy.
- Provides `write` and `read` methods for seamless data
management across nodes.
- Add `find_last_entry` function to efficiently determine the
highest used ID in the lookup table, improving performance
for non-incremental databases.
- Implement deleted record handling using a special marker,
allowing for efficient tracking and synchronization of
deleted entries.
- Enhance `get_last_index` to handle both incremental and
non-incremental modes correctly, providing a unified
interface for retrieving the last index.
- Modify `push_updates` to correctly handle initial syncs and
account for deleted records during synchronization.
- Update `sync_updates` to correctly handle empty update data,
indicating a record deletion.
- Add comprehensive tests for database synchronization, including
edge cases like empty updates, invalid data, and various
scenarios with deleted records.
- Fixed ID generation for files and directories in OurDBFS,
preventing collisions and improving data integrity. This
ensures that IDs are consistently and uniquely assigned.
- Updated save methods to correctly update the `metadata.id`
field across all FSEntry types (File, Directory, Symlink).
This change solves a previous issue where IDs weren't being
properly persisted.
- Added incremental mode to OurDB, improving performance for
large datasets. This allows for more efficient updates
instead of full overwrites.
