Commit Graph

185 Commits

Author SHA1 Message Date
a96903da0e ... 2025-03-09 21:17:32 +01:00
3dbcf00e9f vfs_basics working 2025-03-09 20:14:08 +01:00
26289bb00f Merge branch 'development_actions007' into development_ourdb_new
# Conflicts:
#	lib/data/dedupestor/dedupestor.v
#	lib/data/dedupestor/dedupestor_test.v
2025-03-09 19:07:12 +01:00
13f482fa12 Merge branch 'development_bizmodel' into development_grid_deploy 2025-03-09 17:38:23 +01:00
e5aa8bca09 Merge branch 'development_bizmodel' of github.com:freeflowuniverse/herolib into development_bizmodel 2025-03-09 17:25:22 +01:00
4dd3908ff7 ... 2025-03-09 17:24:44 +01:00
Scott Yeager
407f3f85bc Restore all needed for basic deployments, add vm example 2025-03-07 18:09:28 -08:00
Scott Yeager
0ccf317564 Update module paths 2025-03-07 16:36:01 -08:00
Scott Yeager
0c49e83a68 Update grid proxy module path 2025-03-07 16:00:15 -08:00
ffafef0c88 markdown code 2025-03-07 21:03:55 +01:00
ca3bac1d76 ... 2025-03-07 20:02:37 +01:00
Mahmoud Emad
d2c1be5396 feat: Add basic key-value store example
- Added a client and server for a simple key-value store.
- Improved documentation with client and server usage examples.
- Created client and server implementations using the V language.
2025-03-06 13:32:57 +02:00
Mahmoud Emad
ae7e7ecb84 feat: Add OurDB key-value store server
- Adds a new lightweight key-value store server implemented in V.
- Includes basic CRUD operations (`set`, `get`, `delete`).
- Provides configurable host and operation restrictions for security.
- Offers middleware for logging and request validation.
- Supports incremental mode for automatic ID generation.
- Includes comprehensive documentation and example usage.
- Adds unit tests to ensure functionality and stability.
2025-03-05 23:02:35 +02:00
Mahmoud Emad
485b47d145 feat: Improve MyceliumStreamer worker communication
- Add worker registration to MyceliumStreamer:  Allows for explicit
  addition of workers, improving management and control.
- Simplify worker message handling: Streamlines message processing
  for increased efficiency and readability.  Removes unnecessary
  logging and simplifies message routing.
- Remove redundant message handling: Eliminates duplicate code
  paths for cleaner and more maintainable code.
- Improve worker data retrieval: Facilitates direct data retrieval
  from workers, enhancing efficiency and reliability.
2025-03-04 00:40:05 +02:00
fb9c9b8070 commetbft 2025-03-03 17:32:28 -05:00
Mahmoud Emad
5b69f935a5 feat: Improve MyceliumStreamer to read from workers
- Allow reading data directly from specific workers by specifying
  their public key in `MyceliumStreamer.read()`. This improves
  data retrieval flexibility and allows for distributed data access.
- Add master data reading to ensure data consistency and allow
  comparison between master and worker data.  This helps debug and
  verify data replication.
- Implement JSON encoding/decoding for database transfer between
  master and worker nodes. This enables efficient and structured
  data exchange.
2025-03-03 22:44:53 +02:00
Mahmoud Emad
9dbc36d634 feat: Enhance Deduplicated Mycelium example with multiple workers
- Updated the master node to support multiple workers, allowing for
  increased scalability and redundancy.
- Modified the worker node to simplify initialization and connection
  to the master.
- Added logging statements for better monitoring and debugging.
2025-03-03 17:34:41 +02:00
Mahmoud Emad
42b0c4d48f refactor: Improve MyceliumStreamer's message handling
- Removed unnecessary test data from `deduped_mycelium_master.vsh`.
- Simplified `MyceliumStreamer.listen()` to efficiently handle
  incoming messages, removing redundant code and improving readability.
- Enhanced error handling in `MyceliumStreamer.listen()` for more robust
  operation.
2025-03-03 16:58:13 +02:00
Mahmoud Emad
eb47c8490d feat: Enhance Mycelium streamer with worker support
- Added worker ID to master and worker configurations for improved
  identification and management.
- Implemented worker registration and data synchronization mechanisms
  to enable distributed data access.
- Added a read function to retrieve data from specific workers,
  enhancing data access flexibility.
- Improved logging for better monitoring and debugging of the system.
2025-03-03 14:18:55 +02:00
Mahmoud Emad
1de9be2a8a feat: Improve Mycelium Streamer functionality
- Added continuous data writing and verification to the master node
  to ensure data persistence and integrity.
- Simplified worker update handling in the `listen` function for
  better efficiency and error handling.  The previous implementation
  had unnecessary complexity and potential for hangs.
2025-03-03 13:48:55 +02:00
Mahmoud Emad
d852ecc5b1 feat: Improve Mycelium client and streamer
- Changed Mycelium worker port to avoid conflict with master.
- Added debug print statements to Mycelium client for better troubleshooting.
- Removed unnecessary `SyncData` struct, simplifying data handling.
- Updated data encoding/decoding to directly use base64 for efficiency.
- Clarified message topic names for better understanding.
2025-03-03 12:50:59 +02:00
Mahmoud Emad
368edcd93a feat: Implement distributed database using Mycelium
- Refactor database streamer to support multiple workers.
- Add master node to manage and distribute data updates.
- Implement worker nodes to receive and apply updates.
- Remove unnecessary slave node.
- Improve error handling and logging.
- Use base64 encoding for JSON compatibility in data transfer.
2025-03-03 12:19:03 +02:00
Mahmoud Emad
71906fd891 feat: Improve Mycelium slave and streamer communication
- Renamed the topic for database synchronization messages from
  'db_sync' to 'sync_db' for clarity.
- Updated the Mycelium slave to decode base64 payload before
  processing and to log received messages and their source.
- Added logging to the Mycelium streamer to track sent messages.
- Added a new feature to retrieve and log the last index from the
  worker after syncing updates.  This improves monitoring and
  debugging capabilities.
2025-03-02 22:01:44 +02:00
Mahmoud Emad
dbd187a017 feat: Streamline mycelium example and add server port config
- Remove unnecessary code from the example to improve clarity.
- Add a configurable server port to the `MyceliumStreamer`.
2025-03-02 21:47:12 +02:00
Mahmoud Emad
52a3546325 feat: Improve Mycelium-based data synchronization
- Refactor data synchronization logic to use Mycelium messages for
  efficient updates between master and worker nodes.  This removes
  the previous inefficient polling method and simplifies the code.
- Update the slave node to receive and apply updates from the
  master, improving synchronization efficiency and robustness.
- Change the default slave port to 9000.
- Rename `db` variable to `worker` for clarity.
2025-03-02 21:40:37 +02:00
Mahmoud Emad
a23eb0ff0b feat: Implement MyceliumStreamer for distributed data synchronization
- Introduces `MyceliumStreamer` for synchronizing data across a
  Mycelium network, enabling distributed data access.
- Allows adding multiple worker nodes to the streamer for data
  replication and redundancy.
- Provides `write` and `read` methods for seamless data
  management across nodes.
2025-03-02 20:25:29 +02:00
Mahmoud Emad
b40c366335 feat: Update slave communication in mycelium example
- Rename `get_last_id` to `get_last_index` for clarity.
- Correctly send sync message to the slave instead of master.
2025-03-02 15:17:00 +02:00
Mahmoud Emad
d2ad18e8ec feat: Implement two-way communication between master and slave nodes
- Added message reply functionality to both master and slave
  nodes to enable a two-way communication flow for database
  synchronization. This improves the robustness and reliability
  of the database synchronization process.
- Enhanced the database synchronization process by allowing the
  slave node to send the last inserted record ID to the master
  node.  This provides better tracking of data changes.
2025-03-02 15:14:51 +02:00
Mahmoud Emad
9b737c9280 feat: Improve Mycelium example with bidirectional communication
- Remove unnecessary public key printing in master node.
- Use variable for slave public key in master node.
- Add message receiving functionality to master node.
- Remove redundant sending logic from slave node.
2025-03-02 14:57:33 +02:00
834f612bfe ..deployments 2025-02-28 07:40:10 -07:00
timurgordon
171e54a68c added vfs_db examples with ourdb & dedupstore 2025-02-28 03:16:46 +03:00
75255d8cd0 ... 2025-02-27 06:34:24 -07:00
Mahmoud Emad
a0b53126ca feat: Add example scripts for Mycelium inter-node communication
- Added `deduped_mycelium_master.vsh` to demonstrate a master node
  sending data.
- Added `deduped_mycelium_slave.vsh` to demonstrate a slave node
  receiving data.  These scripts showcase basic inter-node
  communication using the Mycelium library.
2025-02-27 14:46:57 +02:00
Mahmoud Emad
38cd933d41 feat: Update import path for mycelium installer
- Correct the import path for the mycelium installer module.
2025-02-27 12:22:47 +02:00
fff14183a4 .... 2025-02-24 06:34:38 -07:00
Mahmoud Emad
1f58676278 Merge branch 'development' into development_vfs 2025-02-24 15:32:30 +02:00
6a2e143b98 ... 2025-02-24 02:47:59 -07:00
6820a7e9c8 ... 2025-02-23 10:34:18 +00:00
1c7621f20a docusaurus 2025-02-23 07:42:16 +03:00
timurgordon
88fe8f503f rename example output dir 2025-02-20 08:15:49 +03:00
timurgordon
4719876feb get example to work with echarts 2025-02-20 06:43:26 +03:00
Mahmoud Emad
f08af0e2c5 refactor: Improve WebDAV VFS implementation
- Removed unnecessary dependencies and improved code structure in `webdav` module.
- Updated VFS configuration to use global VFS instance for WebDAV app.
- Renamed example VFS file to reflect WebDAV functionality.
- Removed redundant code and simplified app initialization.
- Used the vfs interface to interact with files and dirs.
2025-02-19 22:57:15 +02:00
timurgordon
814b61f25e Merge branch 'development_installers' into development_bizmodel 2025-02-19 13:30:34 +03:00
timurgordon
1c264815c3 update example 2025-02-19 13:28:38 +03:00
ff45beac09 ... 2025-02-19 09:46:41 +03:00
49e2146152 Merge branch 'development_bizmodel' into development_installers 2025-02-19 07:15:28 +03:00
0880823576 Merge branch 'development_installers' of github.com:freeflowuniverse/herolib into development_installers 2025-02-19 05:10:45 +03:00
d6108c9836 Merge branch 'development' into development_installers 2025-02-19 05:08:41 +03:00
Mahmoud Emad
7b453962ca feat: Enhance WebDAV server and add VFS encoder/decoder tests
- Add user authentication to the WebDAV server using a user
  database.
- Implement encoding and decoding functionality for directories,
  files, and symlinks in the OurDBFS VFS.
- Add comprehensive unit tests for the encoder and decoder
  functions.
- Improve the OurDBFS factory method to handle directory creation
  more robustly using pathlib.
- Add `delete` and `link_delete` methods to the `NestedVFS` and
  `OurDBVFS` implementations (though currently unimplemented).
- Improve WebDAV file handling to correctly determine and set the
  content type.  The previous implementation was incomplete and
  returned a dummy response.
- Update VFS test to actually test functionality.
- Remove unnecessary `root_dir` parameter from the WebDAV app.
2025-02-18 17:40:37 +02:00
timurgordon
6305cf159e small fixes on example 2025-02-18 05:22:38 +03:00