Commit Graph

904 Commits

Author SHA1 Message Date
237f9bd742 qdrant 2025-03-11 20:41:33 +01:00
Mahmoud Emad
cf27e7880e feat: Add classifier listing functionality
- Added a new function to list available classifiers.
- Extended the Jina client with `list_classifiers()` method.
- Added unit tests to verify the new functionality.
2025-03-11 21:38:06 +02:00
Mahmoud Emad
ad300c068f feat: Enhance Jina client with improved classification API
- Update `jina.vsh` example to showcase the new classification API
  with support for both text and image inputs. This improves
  the flexibility and usability of the client.
- Introduce new structs `TextDoc`, `ImageDoc`, `ClassificationInput`,
  `ClassificationOutput`, `ClassificationResult`, and `LabelScore`
  to represent data structures for classification requests and
  responses. This enhances code clarity and maintainability.
- Implement the `classify` function in `jina_client.v` to handle
  classification requests with support for text and image inputs,
  model selection, and label specification. This adds a crucial
  feature to the Jina client.
- Add comprehensive unit tests in `jina_client_test.v` to cover
  the new `classify` function's functionality. This ensures the
  correctness and robustness of the implemented feature.
- Remove redundant code related to old classification API and data
  structures from `model_embed.v`, `model_rank.v`, and
  `jina_client.v`. This streamlines the codebase and removes
  obsolete elements.
2025-03-11 21:11:04 +02:00
Mahmoud Emad
1a02dcaf0f feat: Add train functionality to Jina client
- Added a `train` method to the Jina client for training models.
- Added a test case to verify the `train` functionality.
2025-03-11 20:20:46 +02:00
Mahmoud Emad
9ecc2444aa feat: Add Jina client training and classification features
- Added `train` function to the Jina client for training
  classifiers.
- Added `ClassificationTrain` struct to define training
  parameters.
- Added `TrainingExample` struct to represent training data.
- Added `ClassificationTrainOutput` struct for the training
  response.
- Added a new `classification_api.v` module for classifier
  training functionalities.
- Added a new `classify` function to the Jina client for
  classification tasks (currently commented out).
2025-03-11 20:17:35 +02:00
Mahmoud Emad
0e1836c5d0 feat: Add reranking functionality to Jina client
- Added a new `rerank` function to the Jina client for reranking documents.
- Added a new `RerankParams` struct to define parameters for reranking.
- Added unit tests for the new `rerank` function.
- Updated the example script to demonstrate reranking.
- Improved error handling and added more comprehensive logging.
2025-03-11 19:27:01 +02:00
Mahmoud Emad
7965883744 feat: Enhance Jina client with additional embedding parameters
- Add `type_`, `truncate`, and `late_chunking` parameters to the
  `create_embeddings` function for finer control over embedding
  generation. This allows users to specify embedding type,
  truncation method, and whether to apply late chunking.
- Rename model parameter to `model` for clarity and consistency.
- Improve model enum naming for better readability and API consistency.
- Add unit tests for the `create_embeddings` function to ensure
  correct functionality and handle potential errors.
2025-03-11 17:45:55 +02:00
Mahmoud Emad
b006bb1e41 feat: Add create_embeddings function to Jina client
- Added a `create_embeddings` function to the Jina client to
  generate embeddings for given input texts.
- Improved the `create_embeddings` function input parameters
  for better flexibility and error handling.
- Updated `TextEmbeddingInput` struct to handle optional
  parameters for embedding type, truncation type, and late
  chunking.  This improves the flexibility of the embedding
  generation process.
2025-03-11 17:18:47 +02:00
Mahmoud Emad
27c9018c48 fix: Ensure the code compiles and add a test example
- Fixed compilation issues and ensured the code builds successfully
- Created an example to test the client functionality
- Started implementing additional endpoints
2025-03-11 16:49:39 +02:00
Mahmoud Emad
f1991d89b3 feat: Implement database synchronization using binary encoding
- Adds a new mechanism to synchronize the database efficiently
  by serializing updates using binary encoding.  This improves
  performance and reduces bandwidth usage compared to previous methods.

- Introduces `SyncRecord` struct to represent database updates
  for easier handling and serialization.

- Implements `push_updates` to serialize database changes since a
  given index, handling both initial and incremental syncs.

- Implements `sync_updates` to apply received serialized updates
  to the database, robustly handling errors and deletions.
2025-03-11 15:15:19 +02:00
9448ae85cf ... 2025-03-11 14:07:16 +01:00
timurgordon
a64e964d83 Add documentation and tests for model_property.v 2025-03-11 14:03:10 +01:00
Mahmoud Emad
5e321b6b0f feat: Add diagrams and README for OurDB syncer
- Added a diagram explaining the architecture of the OurDB
  syncer, clarifying the interaction between the HTTP server,
  master, and worker nodes.
- Added a README file providing a comprehensive overview of
  the OurDB syncer project, including its architecture,
  features, prerequisites, installation instructions, and usage
  examples.
- Removed outdated Mycelium_Streamer documentation as it's no
  longer relevant to the current project structure.
- Created example scripts for running the database, master,
  and worker components, simplifying the setup and execution of
  the system.
- Added HTTP client and server documentation, clarifying their
  functionalities and interaction with the OurDB system.
2025-03-11 14:49:43 +02:00
b1453e3580 jina specs 2025-03-11 11:23:58 +01:00
3da895083b circle core objects work again 2025-03-11 08:49:47 +01:00
ac583741a4 ... 2025-03-11 08:27:46 +01:00
4358ba6471 ... 2025-03-10 14:31:26 +01:00
46afb63f31 radix tree has now prefix 2025-03-10 09:05:57 +01:00
f773ce168e names 2025-03-10 01:29:15 +01:00
aa79df1fcd models 2025-03-10 01:07:23 +01:00
420c9cb9e5 ... 2025-03-10 00:50:08 +01:00
83d935930f .... 2025-03-10 00:31:31 +01:00
2e2c94e897 ... 2025-03-09 22:56:33 +01:00
a96903da0e ... 2025-03-09 21:17:32 +01:00
3dbcf00e9f vfs_basics working 2025-03-09 20:14:08 +01:00
708147435e vfs working 2025-03-09 20:01:00 +01:00
26289bb00f Merge branch 'development_actions007' into development_ourdb_new
# Conflicts:
#	lib/data/dedupestor/dedupestor.v
#	lib/data/dedupestor/dedupestor_test.v
2025-03-09 19:07:12 +01:00
1489b9f44f ... 2025-03-09 18:34:58 +01:00
e4045ef179 Merge branch 'development_grid_deploy' into development_actions007 2025-03-09 17:40:20 +01:00
13f482fa12 Merge branch 'development_bizmodel' into development_grid_deploy 2025-03-09 17:38:23 +01:00
3e2013576f ... 2025-03-09 17:37:37 +01:00
e5aa8bca09 Merge branch 'development_bizmodel' of github.com:freeflowuniverse/herolib into development_bizmodel 2025-03-09 17:25:22 +01:00
4dd3908ff7 ... 2025-03-09 17:24:44 +01:00
09f388e2ff ... 2025-03-09 15:59:57 +01:00
baecb9bbe4 Merge branch 'development_grid_deploy' into development_generator_docusaurus
* development_grid_deploy:
  Restore all needed for basic deployments, add vm example
  Update griddriver to use prebuilt binary
  Return Deployer and update references
  Update module paths
  Update grid proxy module path
  commetbft
  cleanup client for grid
2025-03-08 10:56:16 +01:00
2c5986295e Merge branch 'development_bizmodel' into development_generator_docusaurus
* development_bizmodel: (93 commits)
  s
  Revert "test: add cmdline parser tests"
  test: add cmdline parser tests
  markdown code
  ...
  revert
  ...
  ..deployments
  ...
  bump version to 1.0.21
  ...
  bump version to 1.0.20
  ...
  fix tests and example
  bump version to 1.0.19
  bump version to 1.0.18
  bump version to 1.0.17
  ...
  ...
  bump version to 1.0.16
  ...

# Conflicts:
#	lib/web/docusaurus/config.v
2025-03-08 10:55:58 +01:00
126f23dfa2 s 2025-03-08 08:56:14 +01:00
Scott Yeager
407f3f85bc Restore all needed for basic deployments, add vm example 2025-03-07 18:09:28 -08:00
timurgordon
94da4fff5e fix delete data decoding 2025-03-08 01:55:10 +01:00
Scott Yeager
0fa54f1354 Update griddriver to use prebuilt binary 2025-03-07 16:54:44 -08:00
timurgordon
80677f7177 lock support fixes 2025-03-08 01:53:39 +01:00
timurgordon
01cac0f741 fixes to chunked file data processing 2025-03-08 01:52:53 +01:00
Scott Yeager
30546a34f9 Return Deployer and update references 2025-03-07 16:39:52 -08:00
Scott Yeager
0ccf317564 Update module paths 2025-03-07 16:36:01 -08:00
Scott Yeager
0c49e83a68 Update grid proxy module path 2025-03-07 16:00:15 -08:00
51db8257f5 Revert "test: add cmdline parser tests"
This reverts commit ecc2977581.
2025-03-07 21:56:33 +01:00
ecc2977581 test: add cmdline parser tests 2025-03-07 21:54:04 +01:00
ffafef0c88 markdown code 2025-03-07 21:03:55 +01:00
ca3bac1d76 ... 2025-03-07 20:02:37 +01:00
43f7bc7943 revert 2025-03-07 18:36:45 +01:00