tfgrid/circle_engineering
24
0
Fork 0
Code Issues 67 Pull Requests Packages Projects Releases Wiki Activity

AI Hub: AI Agent Dashboard Minimum Viable Product on the TFGrid #171

New Issue
Open
opened 2025-01-29 20:40:59 +00:00 by mik-tf · 13 comments
mik-tf commented 2025-01-29 20:40:59 +00:00
Owner
Copy Link

Overview

  • In the first part, we explain the basic for a minimum viable product (AI Agent Dashboard)
  • In the second part, we provide a greater narrative around this AI Agent Dashboard

Minimum Viable Product

We present a minimum viable product for an AI Agent Dashboard running on the grid, offering geo-aware and open-source AI agents.

Situation

  • As discussed within the whole company in the latest meetings and days, the DeepSeek situation is very interesting in many aspects
    • Without dwelling in all details, as a product perspective, we can see a need/problem and product/solution:
      • There is a great product out there, it is FOSS (DeepSeek and others AI models)
      • There are many ways to deploy such models
      • Some of the ways to deploy such models can get attacked
        • e.g. currently we can't use deepseek chat nor API (deepseek.com attacked), but we can use HuggingFace's DeepSeek API!
    • Problem: AI models are hosted on vulnerable and centralized sites
    • Solution: ThreeFold geo-aware AI agent infrastructure across the world
      • specs:
        • in 60+ countries
        • geo-aware + deterministic = Safe AI: you know what model you deploy, you know where it is deployed, end-to-end encrypted

Proposition:

  • Product: AI Agent Dashboard
  • Specs: the simplest dashboard possible to interact with AI agents
    • You just need tokens in your TF Connect app wallet and scan the QR code to talk to an AI agent, on any web browser.

UX

  • I go to ai.threefold.io
  • I scan the QR code and connect my TF Connect App wallet
  • I choose any FOSS AI models (Optional, By Default we set optimal DeepSeek model)
    • e.g. list: DeepSeek and variations
  • I can ask anything to the AI agent

If I don't have tokens, I can scan a QR code for airdrop to get started, and I can buy tokens on the TF connect app (credit/debit card, crypto).

Context

We provide context for the MVP.

Background

  • We have almost everything at ThreeFold to deploy a minimum viable product where anyone can deploy the basic DeepSeek and other models on the grid and get a smooth UX experience
  • We can use OpenWeb UI as a basic and link the wallet to pay with tokens

GPU Nodes

  • There are current talks of setting GPU nodes on the grid
  • For now we are limited

Long Term Vision

We present a world where everyone can benefit from AI revolution and become smarter and be more creative without risking attacks or vulnerabilities.

  • As long term, our concept of having different Internets/grids (the Internet of internets concept) become highly attractive: say one grid has an attack, another grid can be still running and people can redirect their workloads

TF Connect Update

There are upcoming updates to TF Connect app that will make the whole UX very smooth with the AI agent.

  • Roadmap: With TF Connect you can buy tokens with credit/debit card and crypto, you can swap and bridge TFT+INCA to other chains (Solana, Stellar, ETH, BSC, TFChain), you can send tokens to anyone using their simple and unique ThreeFold ID (e.g. @StanUniqueID, you can get airdop tokens by scanning QR codes and participating to the TF ecosystem

Air Drop

  • People can get free tokens to deploy on the grid, they just need to scan the TF Connect app wallet
    • This will fit perfectly with INCA token distribution
# Overview - In the first part, we explain the basic for a minimum viable product (AI Agent Dashboard) - In the second part, we provide a greater narrative around this AI Agent Dashboard # Minimum Viable Product We present a minimum viable product for an AI Agent Dashboard running on the grid, offering geo-aware and open-source AI agents. ## Situation - As discussed within the whole company in the latest meetings and days, the DeepSeek situation is very interesting in many aspects - Without dwelling in all details, as a product perspective, we can see a need/problem and product/solution: - There is a great product out there, it is FOSS (DeepSeek and others AI models) - There are many ways to deploy such models - Some of the ways to deploy such models can get attacked - e.g. currently we can't use deepseek chat nor API (deepseek.com attacked), but we can use HuggingFace's DeepSeek API! - Problem: AI models are hosted on vulnerable and centralized sites - Solution: ThreeFold geo-aware AI agent infrastructure across the world - specs: - in 60+ countries - geo-aware + deterministic = Safe AI: you know what model you deploy, you know where it is deployed, end-to-end encrypted ## Proposition: - Product: AI Agent Dashboard - Specs: the simplest dashboard possible to interact with AI agents - You just need tokens in your TF Connect app wallet and scan the QR code to talk to an AI agent, on any web browser. ### UX - I go to ai.threefold.io - I scan the QR code and connect my TF Connect App wallet - I choose any FOSS AI models (Optional, By Default we set optimal DeepSeek model) - e.g. list: DeepSeek and variations - I can ask anything to the AI agent If I don't have tokens, I can scan a QR code for airdrop to get started, and I can buy tokens on the TF connect app (credit/debit card, crypto). # Context We provide context for the MVP. ## Background - We have almost everything at ThreeFold to deploy a minimum viable product where anyone can deploy the basic DeepSeek and other models on the grid and get a smooth UX experience - We can use OpenWeb UI as a basic and link the wallet to pay with tokens - We already provide docs in the manual: https://manual.grid.tf/documentation/system_administrators/advanced/ai_ml_workloads/cpu_and_llama.html - TF did 2 demos on youtube for OpenWeb UI: CPU and GPU demos - Jan and Kristof made an internal demo with DeepSeek and OpenWeb UI ## GPU Nodes - There are current talks of setting GPU nodes on the grid - For now we are limited ## Long Term Vision We present a world where everyone can benefit from AI revolution and become smarter and be more creative without risking attacks or vulnerabilities. - As long term, our concept of having different Internets/grids (the Internet of internets concept) become highly attractive: say one grid has an attack, another grid can be still running and people can redirect their workloads ## TF Connect Update There are upcoming updates to TF Connect app that will make the whole UX very smooth with the AI agent. - Roadmap: With TF Connect you can buy tokens with credit/debit card and crypto, you can swap and bridge TFT+INCA to other chains (Solana, Stellar, ETH, BSC, TFChain), you can send tokens to anyone using their simple and unique ThreeFold ID (e.g. @StanUniqueID, you can get airdop tokens by scanning QR codes and participating to the TF ecosystem ## Air Drop - People can get free tokens to deploy on the grid, they just need to scan the TF Connect app wallet - This will fit perfectly with INCA token distribution
mik-tf added the
Story
 label 2025-01-29 20:40:59 +00:00
mik-tf added this to the tfgrid_3_17 project 2025-01-29 20:40:59 +00:00
Florian commented 2025-01-30 17:48:31 +00:00
Member
Copy Link

This could be the release for ETH Denver - what is needed to prove our cost advantage/security?

This could be the release for ETH Denver - what is needed to prove our cost advantage/security?
mik-tf commented 2025-02-03 15:49:59 +00:00
Author
Owner
Copy Link

Update

  • Had a great discussion on the feasibility of this project with @thabeta
    • Consensus: We will start with a basic proof-of-concept on the Dashboard (first CPU, then GPU):
      • V1: AI Solution on the Dashboard: one-click ollama+openwebUI deployment with CPU
      • V2: AI Solution on the Dashboard: one-click ollama+openwebUI deployment with GPU
  • Once we have this, we can check how to expand on the concept provided above.

Notes

  • The solutions on the dashboard would then be: one user deploying on one node
    • The initial story is many users using one main website, and the website distributed compute and GPU resources in the backend on different nodes. So this is more technically advanced.
  • Goal: We show a basic one-click AI solution on the Dashboard using CPU and/or GPU, we can start from this and expand.

Dashboard Solution Specs

  • POC v1 (CPU and GPU) UX
    • go to dashboard
    • click deploy webAI Solution
    • deploy the vm with gpu or just cpu
    • In the background:
      • start ollama
      • start openwebui
      • give url to access it
    • User click on link and access the AI chat

Note: user pays for ressources including GPU if it applies

Pros

  • This method, the user knows exactly what they are deploying, which models, and on which nodes. It is deterministic and geo-aware.
  • It is reasonable to have this ready for ETH Denver, but we can't promise anything.

Cons

  • Can't scale the deployment, not distributed.
# Update - Had a great discussion on the feasibility of this project with @thabeta - Consensus: We will start with a basic proof-of-concept on the Dashboard (first CPU, then GPU): - V1: AI Solution on the Dashboard: one-click ollama+openwebUI deployment with CPU - V2: AI Solution on the Dashboard: one-click ollama+openwebUI deployment with GPU - Once we have this, we can check how to expand on the concept provided above. # Notes - The solutions on the dashboard would then be: one user deploying on one node - The initial story is many users using one main website, and the website distributed compute and GPU resources in the backend on different nodes. So this is more technically advanced. - Goal: We show a basic one-click AI solution on the Dashboard using CPU and/or GPU, we can start from this and expand. # Dashboard Solution Specs - POC v1 (CPU and GPU) UX - go to dashboard - click deploy webAI Solution - deploy the vm with gpu or just cpu - In the background: - start ollama - start openwebui - give url to access it - User click on link and access the AI chat > Note: user pays for ressources including GPU if it applies ## Pros - This method, the user knows exactly what they are deploying, which models, and on which nodes. It is deterministic and geo-aware. - It is reasonable to have this ready for ETH Denver, but we can't promise anything. ## Cons - Can't scale the deployment, not distributed.
Florian commented 2025-02-03 16:31:59 +00:00
Member
Copy Link

Why we cannot scale the deployment, this is not distributed? @mik-tf

Why we cannot scale the deployment, this is not distributed? @mik-tf
mik-tf commented 2025-02-03 17:18:52 +00:00
Author
Owner
Copy Link

For the first iteration of the AI chat, we will work on a solution on the dashboard, where a user deploys on one specific node. So this is not distributed. Just like Jan and Kristof showed, they had the deepseek model running on one node, and they accessed this node to interact with the AI chat.

When this is done, we can check how to make it distributed. But this takes more dev resources and careful planning + architecture.

I think I can get the first iteration described above quickly.

The idea is: we get the ball rolling with a dashboard solution: user deploys the solution (AI chat) on one VM and they access it. It is very secure, provides deterministic deployment and geo-awareness (geo-fencing). Then when we show this is possible, we can see how to make this distributed. It would be a different architecture overall. It's all about lean/agile workflow: get a minimum viable product out quickly, and build on it.

Lmk if you have other questions or if this wasn't clear. @Florian

For the first iteration of the AI chat, we will work on a solution on the dashboard, where a user deploys on one specific node. So this is not distributed. Just like Jan and Kristof showed, they had the deepseek model running on one node, and they accessed this node to interact with the AI chat. When this is done, we can check how to make it distributed. But this takes more dev resources and careful planning + architecture. I think I can get the first iteration described above quickly. The idea is: we get the ball rolling with a dashboard solution: user deploys the solution (AI chat) on one VM and they access it. It is very secure, provides deterministic deployment and geo-awareness (geo-fencing). Then when we show this is possible, we can see how to make this distributed. It would be a different architecture overall. It's all about lean/agile workflow: get a minimum viable product out quickly, and build on it. Lmk if you have other questions or if this wasn't clear. @Florian
mik-tf commented 2025-02-05 01:41:42 +00:00
Author
Owner
Copy Link

Update

Docker Version

  • This one uses watchtower, as recommended by @scott
    • This makes the whole flist auto-updatable (the docker openwebui will update constantly)
    • For this reason, we will go with this one

Next Step

  • Make a Dashboard solution for the Open WebUI Flist
  • Then make a PR to the devs
    • Devs can check and make sure everything is OK
  • Then we can release on next Dashboard release
# Update - We have so far 2 working flists for open webui and ollama - the first uses docker to install the 2: https://github.com/mik-tf/openwebui-flist-docker - the second uses non-docker method (uses python for openweb uiand ollama directly): https://github.com/mik-tf/openwebui-flist ## Docker Version - This one uses watchtower, as recommended by @scott - This makes the whole flist auto-updatable (the docker openwebui will update constantly) - For this reason, we will go with this one # Next Step - Make a Dashboard solution for the Open WebUI Flist - Then make a PR to the devs - Devs can check and make sure everything is OK - Then we can release on next Dashboard release
mik-tf commented 2025-02-05 18:45:14 +00:00
Author
Owner
Copy Link

Update

  • There is ongoing work for openwebui to be deployed on the grid using the tfgrid3deployer:
    https://github.com/freeflowuniverse/herolib/issues/49
  • This dashboard openwebui flist will be simpler and we will focus on making sure it works with the dashboard and stays compatible
  • This means that the two different projects are compatible
  • We note it here so there is clear direction between organizaion (tfgrid-sdk-ts, and freeflow's herolib)
# Update - There is ongoing work for openwebui to be deployed on the grid using the tfgrid3deployer: https://github.com/freeflowuniverse/herolib/issues/49 - This dashboard openwebui flist will be simpler and we will focus on making sure it works with the dashboard and stays compatible - This means that the two different projects are compatible - We note it here so there is clear direction between organizaion (tfgrid-sdk-ts, and freeflow's herolib)
mik-tf commented 2025-02-06 16:26:40 +00:00
Author
Owner
Copy Link

Update

Next Steps

  • We can add GPU support on the app
    • But we need GPU nodes on the grid to test
    • I am checking with @delandtj to get input on what setup to get at my place so I can test GPU nodes efficiently.
  • We can check how to integrate a more distributed approach but this is a whole next level.
# Update - The open webui app is done - tfgrid-sdk-ts PR: https://github.com/threefoldtech/tfgrid-sdk-ts/pull/3869 - info_grid PR: https://github.com/threefoldtech/info_grid/pull/739 - tf-images PR: https://github.com/threefoldtech/tf-images/pull/328 # Next Steps - We can add GPU support on the app - But we need GPU nodes on the grid to test - I am checking with @delandtj to get input on what setup to get at my place so I can test GPU nodes efficiently. - We can check how to integrate a more distributed approach but this is a whole next level.
mik-tf commented 2025-02-09 04:26:47 +00:00
Author
Owner
Copy Link

Update

  • GPU Support
    • Started to work on this: we have now a bash script that sets up everything on a 3node for a gpu nvidia:
    • TODO:
      • add this script to the open webui flist with some parameters so it only runs when the user wants to use a GPU nvidia
      • add gpu logic to dashboard app openwebui
      • write an equivalent script for AMD and test it, then same process as above but for AMD
# Update - GPU Support - Started to work on this: we have now a bash script that sets up everything on a 3node for a gpu nvidia: - https://github.com/mik-tf/ubundia - test: tested on a gpu node on the grid, works 100% - TODO: - add this script to the open webui flist with some parameters so it only runs when the user wants to use a GPU nvidia - add gpu logic to dashboard app openwebui - write an equivalent script for AMD and test it, then same process as above but for AMD
mik-tf commented 2025-02-13 14:21:48 +00:00
Author
Owner
Copy Link

Update

  • CPU version for micro VM has been approved on Dashboard tfgrid-sdk-ts
  • The GPU version might not be able to run on micro VM, Scott is checking if it is feasible
  • Working on the GPU version on a full VM in parallel so we have different avenues
# Update - CPU version for micro VM has been approved on Dashboard tfgrid-sdk-ts - The GPU version might not be able to run on micro VM, Scott is checking if it is feasible - Working on the GPU version on a full VM in parallel so we have different avenues
mik-tf commented 2025-02-17 16:57:46 +00:00
Author
Owner
Copy Link

Update

# Update - Working on GPU full vm version - Flist full vm gpu version is done - link: https://github.com/threefoldtech/tf-images/pull/332 - current deployment: full vm on grid, add gateway to port 8080, instant access - TODO - finalize dashboard app version - link: https://github.com/threefoldtech/tfgrid-sdk-ts/pull/3869
mik-tf commented 2025-02-18 01:23:57 +00:00
Author
Owner
Copy Link

Update

Next Step

  • Deploy Nvidia 4090 gpu node with Philippe and Dimo
  • Deploy Open WebUI on it
  • Showcase at ETH Denver
# Update - GPU version done, works very well with nvidia - PR: https://github.com/threefoldtech/tfgrid-sdk-ts/pull/3869 - Even if PR is not accepted/merged to development, we can showcase the app at ETHDenver and deploy on a locally run dashboard with development_ai branch # Next Step - Deploy Nvidia 4090 gpu node with Philippe and Dimo - Deploy Open WebUI on it - Showcase at ETH Denver
❤️ 1
gosam commented 2025-02-19 07:53:12 +00:00
Owner
Copy Link

Amazing work! We would also like to do a recorded demo.

Need to see if it's @mik-tf and @scott or @despiegk who will do it.

Amazing work! We would also like to do a recorded demo. Need to see if it's @mik-tf and @scott or @despiegk who will do it.
mik-tf commented 2025-02-19 13:59:15 +00:00
Author
Owner
Copy Link

Update

  • GPU farmer has deployed the node, it is being registered on TFChain
  • Once it is available online, bert (ops) will reserve it and test
  • Then we can deploy open webui and do the demo
# Update - GPU farmer has deployed the node, it is being registered on TFChain - Once it is available online, bert (ops) will reserve it and test - Then we can deploy open webui and do the demo
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
  
Issue

   
Question

   
Roadmap

   
Story

   
Task

   
Urgent
No Label
Issue
Question
Roadmap
Story
Task
Urgent
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
tfgrid_3_17
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: tfgrid/circle_engineering#171
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?