Commit Graph

498 Commits

Author SHA1 Message Date
timurgordon
90c81a1234 add path to fsentry metadata 2025-03-06 15:48:37 +01:00
timurgordon
fe934bba36 webdav fixes 2025-03-06 01:24:04 +01:00
timurgordon
b01e40da40 fix vfsimplemetation, add path metadata 2025-03-06 01:23:17 +01:00
Mahmoud Emad
ae7e7ecb84 feat: Add OurDB key-value store server
- Adds a new lightweight key-value store server implemented in V.
- Includes basic CRUD operations (`set`, `get`, `delete`).
- Provides configurable host and operation restrictions for security.
- Offers middleware for logging and request validation.
- Supports incremental mode for automatic ID generation.
- Includes comprehensive documentation and example usage.
- Adds unit tests to ensure functionality and stability.
2025-03-05 23:02:35 +02:00
Mahmoud Emad
fdf540cbd0 docs: Add documentation for Mycelium Streamer example
- Added a new documentation file explaining the Mycelium Streamer
  example, covering setup, prerequisites, and execution.
- This provides users with clear instructions on how to use the
  example project.
2025-03-04 21:34:02 +02:00
timurgordon
7cb26e5569 fix webdav lock handling and xlm parsing 2025-03-04 00:50:40 +01:00
timurgordon
913b0cb790 fix vfs_db and add metadata_db 2025-03-04 00:49:57 +01:00
Mahmoud Emad
485b47d145 feat: Improve MyceliumStreamer worker communication
- Add worker registration to MyceliumStreamer:  Allows for explicit
  addition of workers, improving management and control.
- Simplify worker message handling: Streamlines message processing
  for increased efficiency and readability.  Removes unnecessary
  logging and simplifies message routing.
- Remove redundant message handling: Eliminates duplicate code
  paths for cleaner and more maintainable code.
- Improve worker data retrieval: Facilitates direct data retrieval
  from workers, enhancing efficiency and reliability.
2025-03-04 00:40:05 +02:00
Mahmoud Emad
5b69f935a5 feat: Improve MyceliumStreamer to read from workers
- Allow reading data directly from specific workers by specifying
  their public key in `MyceliumStreamer.read()`. This improves
  data retrieval flexibility and allows for distributed data access.
- Add master data reading to ensure data consistency and allow
  comparison between master and worker data.  This helps debug and
  verify data replication.
- Implement JSON encoding/decoding for database transfer between
  master and worker nodes. This enables efficient and structured
  data exchange.
2025-03-03 22:44:53 +02:00
Mahmoud Emad
9dbc36d634 feat: Enhance Deduplicated Mycelium example with multiple workers
- Updated the master node to support multiple workers, allowing for
  increased scalability and redundancy.
- Modified the worker node to simplify initialization and connection
  to the master.
- Added logging statements for better monitoring and debugging.
2025-03-03 17:34:41 +02:00
Mahmoud Emad
42b0c4d48f refactor: Improve MyceliumStreamer's message handling
- Removed unnecessary test data from `deduped_mycelium_master.vsh`.
- Simplified `MyceliumStreamer.listen()` to efficiently handle
  incoming messages, removing redundant code and improving readability.
- Enhanced error handling in `MyceliumStreamer.listen()` for more robust
  operation.
2025-03-03 16:58:13 +02:00
Mahmoud Emad
5343d9bff6 feat: Add unique ID to MyceliumStreamer
- Added a unique ID to the MyceliumStreamer struct for better
  identification of individual streamer instances.
2025-03-03 14:21:25 +02:00
Mahmoud Emad
eb47c8490d feat: Enhance Mycelium streamer with worker support
- Added worker ID to master and worker configurations for improved
  identification and management.
- Implemented worker registration and data synchronization mechanisms
  to enable distributed data access.
- Added a read function to retrieve data from specific workers,
  enhancing data access flexibility.
- Improved logging for better monitoring and debugging of the system.
2025-03-03 14:18:55 +02:00
Mahmoud Emad
1de9be2a8a feat: Improve Mycelium Streamer functionality
- Added continuous data writing and verification to the master node
  to ensure data persistence and integrity.
- Simplified worker update handling in the `listen` function for
  better efficiency and error handling.  The previous implementation
  had unnecessary complexity and potential for hangs.
2025-03-03 13:48:55 +02:00
Mahmoud Emad
d852ecc5b1 feat: Improve Mycelium client and streamer
- Changed Mycelium worker port to avoid conflict with master.
- Added debug print statements to Mycelium client for better troubleshooting.
- Removed unnecessary `SyncData` struct, simplifying data handling.
- Updated data encoding/decoding to directly use base64 for efficiency.
- Clarified message topic names for better understanding.
2025-03-03 12:50:59 +02:00
Mahmoud Emad
368edcd93a feat: Implement distributed database using Mycelium
- Refactor database streamer to support multiple workers.
- Add master node to manage and distribute data updates.
- Implement worker nodes to receive and apply updates.
- Remove unnecessary slave node.
- Improve error handling and logging.
- Use base64 encoding for JSON compatibility in data transfer.
2025-03-03 12:19:03 +02:00
Mahmoud Emad
71906fd891 feat: Improve Mycelium slave and streamer communication
- Renamed the topic for database synchronization messages from
  'db_sync' to 'sync_db' for clarity.
- Updated the Mycelium slave to decode base64 payload before
  processing and to log received messages and their source.
- Added logging to the Mycelium streamer to track sent messages.
- Added a new feature to retrieve and log the last index from the
  worker after syncing updates.  This improves monitoring and
  debugging capabilities.
2025-03-02 22:01:44 +02:00
Mahmoud Emad
dbd187a017 feat: Streamline mycelium example and add server port config
- Remove unnecessary code from the example to improve clarity.
- Add a configurable server port to the `MyceliumStreamer`.
2025-03-02 21:47:12 +02:00
Mahmoud Emad
52a3546325 feat: Improve Mycelium-based data synchronization
- Refactor data synchronization logic to use Mycelium messages for
  efficient updates between master and worker nodes.  This removes
  the previous inefficient polling method and simplifies the code.
- Update the slave node to receive and apply updates from the
  master, improving synchronization efficiency and robustness.
- Change the default slave port to 9000.
- Rename `db` variable to `worker` for clarity.
2025-03-02 21:40:37 +02:00
Mahmoud Emad
a23eb0ff0b feat: Implement MyceliumStreamer for distributed data synchronization
- Introduces `MyceliumStreamer` for synchronizing data across a
  Mycelium network, enabling distributed data access.
- Allows adding multiple worker nodes to the streamer for data
  replication and redundancy.
- Provides `write` and `read` methods for seamless data
  management across nodes.
2025-03-02 20:25:29 +02:00
Mahmoud Emad
b40c366335 feat: Update slave communication in mycelium example
- Rename `get_last_id` to `get_last_index` for clarity.
- Correctly send sync message to the slave instead of master.
2025-03-02 15:17:00 +02:00
Mahmoud Emad
d2ad18e8ec feat: Implement two-way communication between master and slave nodes
- Added message reply functionality to both master and slave
  nodes to enable a two-way communication flow for database
  synchronization. This improves the robustness and reliability
  of the database synchronization process.
- Enhanced the database synchronization process by allowing the
  slave node to send the last inserted record ID to the master
  node.  This provides better tracking of data changes.
2025-03-02 15:14:51 +02:00
Mahmoud Emad
9b737c9280 feat: Improve Mycelium example with bidirectional communication
- Remove unnecessary public key printing in master node.
- Use variable for slave public key in master node.
- Add message receiving functionality to master node.
- Remove redundant sending logic from slave node.
2025-03-02 14:57:33 +02:00
timurgordon
171e54a68c added vfs_db examples with ourdb & dedupstore 2025-02-28 03:16:46 +03:00
timurgordon
a690f98cc1 fine tuned vfs_db to work with dedupe db 2025-02-28 03:15:47 +03:00
timurgordon
400ea6e80e moved webdav to dav 2025-02-28 03:15:05 +03:00
timurgordon
84c19ca9a4 implemented updates for radix tree 2025-02-28 03:14:44 +03:00
timurgordon
3b7ec028f9 created vscode extension for ourdb decoded viewing 2025-02-28 03:14:08 +03:00
Mahmoud Emad
a0b53126ca feat: Add example scripts for Mycelium inter-node communication
- Added `deduped_mycelium_master.vsh` to demonstrate a master node
  sending data.
- Added `deduped_mycelium_slave.vsh` to demonstrate a slave node
  receiving data.  These scripts showcase basic inter-node
  communication using the Mycelium library.
2025-02-27 14:46:57 +02:00
Mahmoud Emad
38cd933d41 feat: Update import path for mycelium installer
- Correct the import path for the mycelium installer module.
2025-02-27 12:22:47 +02:00
Mahmoud Emad
2da014c407 Merge branch 'development_ourdb_new' of https://github.com/freeflowuniverse/herolib into development_ourdb_new 2025-02-27 12:09:50 +02:00
Mahmoud Emad
9d1752f4ed feat: Improve database synchronization and add deleted record handling
- Add `find_last_entry` function to efficiently determine the
  highest used ID in the lookup table, improving performance
  for non-incremental databases.
- Implement deleted record handling using a special marker,
  allowing for efficient tracking and synchronization of
  deleted entries.
- Enhance `get_last_index` to handle both incremental and
  non-incremental modes correctly, providing a unified
  interface for retrieving the last index.
- Modify `push_updates` to correctly handle initial syncs and
  account for deleted records during synchronization.
- Update `sync_updates` to correctly handle empty update data,
  indicating a record deletion.
- Add comprehensive tests for database synchronization, including
  edge cases like empty updates, invalid data, and various
  scenarios with deleted records.
2025-02-27 12:09:33 +02:00
timurgordon
d06a806184 isolate vfs's and improve documentation 2025-02-27 11:42:46 +03:00
timurgordon
fe1becabaf fix and refactor vfs modules to merge ourdb implementations into generic vfs_db and separate vfs_local 2025-02-27 11:12:17 +03:00
timurgordon
972bb9f755 implement better reference tracking for deduped files 2025-02-26 22:24:40 +03:00
timurgordon
a798b2347f start implementing ourdb sync 2025-02-26 12:29:27 +03:00
timurgordon
17979b4fde start implementing vfs with dedupe ourdb driver 2025-02-26 03:12:09 +03:00
timurgordon
68d25d3622 move in radixtree and dedupstore 2025-02-26 02:38:38 +03:00
Mahmoud Emad
59efa18bce feat: Uncomment lib/vfs path in test_basic.vsh
- Uncommented the path `lib/vfs` in `test_basic.vsh`.
- This path was commented out, and is needed after merging PR #68.
2025-02-25 13:25:08 +02:00
Mahmoud Emad
4ed80481aa test: Add more comprehensive tests for vfsourdb
- Added tests to verify directory listing functionality after
  creating and moving directories.
- Improved test coverage for file operations within directories.
- Ensured tests accurately reflect the updated behavior of
  `dir_list` function.
2025-02-25 13:13:41 +02:00
Mahmoud Emad
1f58676278 Merge branch 'development' into development_vfs 2025-02-24 15:32:30 +02:00
Mahmoud Emad
b67db23e07 feat: Improve VFS rename and copy operations
- Return FSEntry from `rename` and `copy` operations in VFS
  to provide more information about the result.  This allows
  access to metadata after a successful rename or copy.

- Update `LocalVFS` and `NestedVFS` implementations to return
  the appropriate FSEntry objects after successful rename and
  copy operations.
2025-02-24 14:03:41 +02:00
Mahmoud Emad
4fe1e70881 feat: Improve directory copy functionality and add error handling
- Refactor `Directory.copy()` to use a struct for arguments,
  improving readability and maintainability.
- Add comprehensive error handling to `Directory.copy()`,
  preventing unexpected failures and providing informative error
  messages.  This includes handling cases where the source is not
  a directory, or a source and destination path are the same.
- Implement recursive copying of directory contents, including files
  and symlinks.
- Add unit tests to cover the new `copy` functionality and error
  handling.
- Update `OurDBVFS.copy()` to utilize the improved `Directory.copy()`
  method and add input validation.
2025-02-24 13:55:16 +02:00
Mahmoud Emad
988602f90f feat: Enhance VFS with file and directory manipulation
- Add `move`, `copy`, and `rename` methods to `Directory` and `File`
  for improved file system management.
- Refactor `move` operation in `Directory` for better error handling and
  support for recursive directory moves.  Improves robustness and
  clarity of the move operation.
- Implement a `MoveDirArgs` struct to improve the clarity and
  maintainability of the `move` function arguments.
- Remove unnecessary `save()` calls for improved performance.
- Add comprehensive tests for the new and improved file system
  operations.  Ensures reliability and correctness of the added
  functionality.
2025-02-24 13:10:34 +02:00
Mahmoud Emad
306de32de8 feat: Implement rename functionality for directories and files
- Added `rename` method to `Directory` struct to rename files and
  directories, updating metadata and timestamps.  This improves
  file management capabilities.
- Added `rename` method to `OurDBVFS` to provide a unified
  interface for renaming files and directories across the VFS. This
  allows for consistent file system operations.
- Added tests for the new rename functionality in `vfsourdb_test.v`
  to ensure correctness and robustness. This enhances confidence in
  the implementation.
2025-02-23 22:35:37 +02:00
Mahmoud Emad
c0b57e2a01 feat: Add file move operation
- Added `move` operation to `Directory` to rename files and
  directories within the same directory.  This improves
  file management capabilities.
- Updated `VFS` interface to include `move` function with
  FSEntry return type for consistency. This allows for
  retrieving metadata of the moved file/directory.
- Implemented `move` operation for `LocalVFS`, `OurDBVFS`, and
  `NestedVFS`.  This provides consistent file move
  functionality across different VFS implementations.
- Added tests for the new move functionality in
  `vfsourdb_test.v`. This ensures the correct behavior of the
  new feature.
2025-02-23 22:26:05 +02:00
Mahmoud Emad
aeeacc877b feat: Improve VFS handling and authentication middleware
- Remove unnecessary debug print statements in VFS and WebDAV
  middleware for cleaner code.
- Fix a bug in `OurDBVFS.exists` to correctly handle root and
  current directory paths.
- Enhance `OurDBVFS.get_entry` to handle '.' path correctly.
- Improve WebDAV authentication middleware to gracefully handle
  unauthenticated requests.
2025-02-23 14:33:18 +02:00
5263798b11 s 2025-02-22 19:03:45 +03:00
timurgordon
0d96c5fc65 fix webdav server implementation and logic 2025-02-22 01:40:46 +03:00
timurgordon
6b0cf48292 implement webdav server in veb 2025-02-20 19:04:07 +03:00