lhumina_code/home
22
0
Fork 0
Code Issues 14 Pull requests Projects Releases Packages Wiki Activity Actions

Bring all Hero services to 4-pillar standard (OpenRPC + MCP + Health + Socket) #34

New issue
Open
opened 2026-03-18 14:14:33 +00:00 by mik-tf · 7 comments
mik-tf commented 2026-03-18 14:14:33 +00:00
Owner
Copy link

Goal

Bring all user-facing Hero services to the full Hero OS service standard (the "4 pillars"), then wire them to the AI Assistant. hero_books is the reference implementation — it has all 4. Every other service must match.

The 4 Pillars (hero_books = gold standard)

Pillar What Why
OpenRPC rpc.discover returns full API spec Inspector discovery, documentation, client generation
MCP POST /mcp with curated AI-friendly tools AI Assistant integration, context-aware operations
Health GET /health or rpc.health Monitoring, liveness, deployment verification
Unix Socket ~/hero/var/sockets/<service>.sock Auto-discovery, zero-config, context-scoped routing

1. OpenRPC — the service contract

Every service implements rpc.discover, returning a full OpenRPC 1.3 specification. This is the machine-readable contract that describes every method, its parameters, return types, and errors. hero_inspector auto-discovers services by probing sockets for rpc.discover — if a service doesn't respond, it's invisible to the platform. OpenRPC also powers documentation generation, client SDK generation, and the inspector's MCP gateway. A service without OpenRPC is undocumented and undiscoverable.

2. MCP — the AI interface

Every service exposes POST /mcp with curated, hand-crafted tool definitions optimized for LLM consumption. Tools are not auto-generated from OpenRPC — they are a deliberately curated subset with AI-friendly descriptions, sensible defaults, and clear input schemas. Context-scoped services (hero_osis, hero_foundry) accept an optional context parameter on every tool, defaulting to the user's active Hero OS context. Global services (hero_auth, hero_books, hero_inspector, hero_indexer) ignore context. Each service owns its MCP interface independently — no routing through gateways or proxies. A stdio↔HTTP bridge script per service connects it to the AI Assistant via mcp_servers.json.

3. Health — operational visibility

Every service responds to GET /health (HTTP services) or rpc.health (raw socket services) with a structured status. This is what zinit monitors, what the inspector checks for liveness, and what deployment automation uses to verify a service came up correctly. Health endpoints should include version, uptime, and any service-specific readiness indicators (e.g., hero_books reports library and embedder status).

4. Unix Socket — the universal transport

Every service binds to ~/hero/var/sockets/<service>.sock. Unix sockets are the standard transport in Hero OS — no TCP ports except for external-facing proxies. Sockets enable inspector auto-discovery (scans the socket directory), zero-config inter-service communication, and clean filesystem-based access control. Context-scoped services create per-context sockets: ~/hero/var/sockets/{context}/<service>.sock. Socket symlinks provide stable aliases for services with multiple names.

Verified Audit (live on herodev, 2026-03-18)

All services running and healthy. Inspector discovers 42 sockets.

Service OpenRPC MCP Health Socket Inspector Status
hero_books ✅ rpc.discover ✅ 6 tools, wired ✅ hero_books_server.sock ✅ 19 methods Gold standard
hero_auth ❌ ✅ 10 tools, not wired ✅ hero_auth_ui.sock ⚠️ web only Needs bridge + OpenRPC
hero_foundry_server ✅ embedded spec at /api/rpc ❌ ✅ hero_foundry_server.sock ❌ path mismatch Needs MCP
hero_foundry_admin ✅ proxies to server ❌ (GET page only) ✅ hero_foundry_admin.sock ✅ 32 methods UI layer only
hero_inspector ✅ own spec ⚠️ Gateway only /mcp/:id ✅ hero_inspector_server.sock ✅ 18 methods Needs self-MCP
hero_indexer_ui ✅ proxies rpc.discover to backend ❌ ✅ hero_indexer_ui.sock ✅ 19 methods Needs MCP only
hero_osis_ui ❌ doesn't proxy rpc.discover ❌ ✅ hero_osis_ui.sock ⚠️ web only Needs MCP + OpenRPC
hero_osis_server ✅ via raw socket ❌ ✅ {ctx}/hero_osis_server.sock ❌ raw protocol Per-context backend

Key findings from live testing:

  • hero_auth binds to hero_auth_ui.sock (process: hero_auth_server --bind unix:...hero_auth_ui.sock)
  • hero_indexer_ui already proxies rpc.discover — inspector sees 19 methods. Only MCP is missing.
  • hero_foundry has two binaries: hero_foundry_server (backend, OpenRPC at /api/rpc) and hero_foundry_admin (UI dashboard, proxies via SDK). MCP goes on the server.
  • hero_osis_server responds to rpc.discover via raw socket (confirmed with socat). Returns full merged per-domain spec. Inspector can't see it because it only probes HTTP.
  • hero_foundry_server invisible to inspector because it serves at /api/rpc not /rpc

Context Architecture

Hero OS contexts (Geomind, Incubaid, ThreeFold, Default) are the fundamental data scoping mechanism. A context is a workspace that isolates all user data — contacts, files, calendar, media, projects. The context selector in Hero OS (top-left) determines what every app shows. Context-awareness is built into the MCP pillar: context-scoped services route tool calls to the right data based on the user's active context.

Service Context-aware? Mechanism
hero_osis YES Per-context sockets: ~/hero/var/sockets/{ctx}/hero_osis_server.sock
hero_foundry YES Per-context WebDAV: ~/hero/var/hero_foundry/webdav/{ctx}/
hero_books NO Global library
hero_auth NO Global auth
hero_inspector NO Global discovery
hero_indexer NO Global search

The AI Island already passes context_name to the chat function. Today hero_shrimp doesn't use it because no context-aware MCP tools exist yet. As part of this work, context flows end-to-end: Hero OS → AI Island → hero_shrimp → HERO_CONTEXT env → bridge script → service endpoint.

Known Blockers

  • AI error on some queries — "Help me set up my AI provider keys" returns error. hero_shrimp logs empty. Needs deeper investigation (try direct curl to shrimp /api/chat, check hero_aibroker).
  • hero_indexer build broken — hero_rpc commit 5094617 has use hero_service::ZinitLifecycle (should be HeroLifecycle). Fix on WIP branch development_mik_34, update Cargo.lock.

Execution Plan — Learn Then Replicate

Phase 0: Fix blockers

  • Debug AI error on herodev
  • Fix hero_rpc import on development_mik_34 branch, update hero_indexer Cargo.lock

Phase 1: hero_auth — validate bridge pattern (bridge only, MCP exists)

  • Create mcp_auth bridge script → hero_auth_ui.sock at /mcp
  • Add mcp_auth entry to mcp_servers.json
  • Deploy, test: "list all users", "who has admin access?"
  • Learn: confirms bridge pipeline end-to-end

Phase 2: hero_foundry — first MCP addition (has OpenRPC)

  • Add MCP handler to hero_foundry_server (POST /mcp)
  • Curated tools: repo_list_files, repo_read_file, repo_write_file, repo_find, server_list_repositories
  • Context-aware: tools accept optional context param for WebDAV scoping
  • Create mcp_foundry bridge + config
  • Deploy, test: "show my files", "read the README"
  • Learn: establishes the template for adding MCP

Phase 3: hero_inspector — self-MCP

  • Add POST /mcp (self) alongside existing /mcp/:id (gateway)
  • Tools: services_list, service_detail, scan, status
  • Create mcp_inspector bridge + config
  • Deploy, test: "what services are running?"

Phase 4: hero_indexer — MCP for search

  • Add POST /mcp to hero_indexer_ui
  • Tools: search, index_status, collections_list
  • Proxy to backend via existing proxy_to_socket()
  • Create mcp_indexer bridge + config
  • Deploy, test: "find documents about ThreeFold"

Phase 5: hero_osis — MCP for personal data (largest, context-critical)

  • Add POST /mcp to hero_osis_ui
  • All tools accept optional context parameter (default: session context)
  • Priority domains (~20-25 tools): identity, calendar, communication, files, projects
  • Create mcp_osis bridge + config
  • Deploy, test: "show my contacts", "create a calendar event"

Phase 6: Context flow through to MCP

  • Pass context_name through hero_shrimp's /api/chat (currently ignored)
  • hero_shrimp injects HERO_CONTEXT into MCP bridge process env
  • Bridge scripts use HERO_CONTEXT for context-aware service routing

Phase 7: OpenRPC gaps

  • hero_auth: add rpc.discover
  • hero_osis_ui: proxy rpc.discover to backend
  • hero_foundry_server: fix inspector path mismatch (/api/rpc vs /rpc)

Phase 8: Build, deploy, full integration test

  • All MCP bridges working on herodev.gent04.grid.tf
  • AI can operate in each context

Acceptance Criteria

  • All 6 services have POST /mcp with hand-crafted tool definitions
  • All wired to AI Assistant via bridge scripts + mcp_servers.json
  • AI can: list users, browse files, search documents, manage contacts/calendar, check service status
  • Context flows from Hero OS → AI → MCP tools for context-aware services
  • All deployed and verified on herodev
  • OpenRPC gaps addressed

Related

  • Follows #31 (Self-Documenting Platform) — hero_books established the 4-pillar standard
  • #35 — Context creation broken from UI (separate, non-blocking)
## Goal Bring all user-facing Hero services to the full Hero OS service standard (the "4 pillars"), then wire them to the AI Assistant. **hero_books is the reference implementation** — it has all 4. Every other service must match. ## The 4 Pillars (hero_books = gold standard) | Pillar | What | Why | |--------|------|-----| | **OpenRPC** | `rpc.discover` returns full API spec | Inspector discovery, documentation, client generation | | **MCP** | `POST /mcp` with curated AI-friendly tools | AI Assistant integration, context-aware operations | | **Health** | `GET /health` or `rpc.health` | Monitoring, liveness, deployment verification | | **Unix Socket** | `~/hero/var/sockets/<service>.sock` | Auto-discovery, zero-config, context-scoped routing | ### 1. OpenRPC — the service contract Every service implements `rpc.discover`, returning a full OpenRPC 1.3 specification. This is the machine-readable contract that describes every method, its parameters, return types, and errors. hero_inspector auto-discovers services by probing sockets for `rpc.discover` — if a service doesn't respond, it's invisible to the platform. OpenRPC also powers documentation generation, client SDK generation, and the inspector's MCP gateway. A service without OpenRPC is undocumented and undiscoverable. ### 2. MCP — the AI interface Every service exposes `POST /mcp` with curated, hand-crafted tool definitions optimized for LLM consumption. Tools are not auto-generated from OpenRPC — they are a deliberately curated subset with AI-friendly descriptions, sensible defaults, and clear input schemas. Context-scoped services (hero_osis, hero_foundry) accept an optional `context` parameter on every tool, defaulting to the user's active Hero OS context. Global services (hero_auth, hero_books, hero_inspector, hero_indexer) ignore context. Each service owns its MCP interface independently — no routing through gateways or proxies. A stdio↔HTTP bridge script per service connects it to the AI Assistant via `mcp_servers.json`. ### 3. Health — operational visibility Every service responds to `GET /health` (HTTP services) or `rpc.health` (raw socket services) with a structured status. This is what zinit monitors, what the inspector checks for liveness, and what deployment automation uses to verify a service came up correctly. Health endpoints should include version, uptime, and any service-specific readiness indicators (e.g., hero_books reports library and embedder status). ### 4. Unix Socket — the universal transport Every service binds to `~/hero/var/sockets/<service>.sock`. Unix sockets are the standard transport in Hero OS — no TCP ports except for external-facing proxies. Sockets enable inspector auto-discovery (scans the socket directory), zero-config inter-service communication, and clean filesystem-based access control. Context-scoped services create per-context sockets: `~/hero/var/sockets/{context}/<service>.sock`. Socket symlinks provide stable aliases for services with multiple names. ## Verified Audit (live on herodev, 2026-03-18) All services running and healthy. Inspector discovers 42 sockets. | Service | OpenRPC | MCP | Health | Socket | Inspector | Status | |---------|---------|-----|--------|--------|-----------|--------| | **hero_books** | ✅ `rpc.discover` | ✅ 6 tools, wired | ✅ | hero_books_server.sock | ✅ 19 methods | **Gold standard** | | **hero_auth** | ❌ | ✅ 10 tools, not wired | ✅ | hero_auth_ui.sock | ⚠️ web only | Needs bridge + OpenRPC | | **hero_foundry_server** | ✅ embedded spec at `/api/rpc` | ❌ | ✅ | hero_foundry_server.sock | ❌ path mismatch | Needs MCP | | **hero_foundry_admin** | ✅ proxies to server | ❌ (GET page only) | ✅ | hero_foundry_admin.sock | ✅ 32 methods | UI layer only | | **hero_inspector** | ✅ own spec | ⚠️ Gateway only `/mcp/:id` | ✅ | hero_inspector_server.sock | ✅ 18 methods | Needs self-MCP | | **hero_indexer_ui** | ✅ proxies `rpc.discover` to backend | ❌ | ✅ | hero_indexer_ui.sock | ✅ 19 methods | Needs MCP only | | **hero_osis_ui** | ❌ doesn't proxy `rpc.discover` | ❌ | ✅ | hero_osis_ui.sock | ⚠️ web only | Needs MCP + OpenRPC | | **hero_osis_server** | ✅ via raw socket | ❌ | ✅ | {ctx}/hero_osis_server.sock | ❌ raw protocol | Per-context backend | ### Key findings from live testing: - **hero_auth** binds to `hero_auth_ui.sock` (process: `hero_auth_server --bind unix:...hero_auth_ui.sock`) - **hero_indexer_ui** already proxies `rpc.discover` — inspector sees 19 methods. Only MCP is missing. - **hero_foundry** has two binaries: `hero_foundry_server` (backend, OpenRPC at `/api/rpc`) and `hero_foundry_admin` (UI dashboard, proxies via SDK). MCP goes on the server. - **hero_osis_server** responds to `rpc.discover` via raw socket (confirmed with socat). Returns full merged per-domain spec. Inspector can't see it because it only probes HTTP. - **hero_foundry_server** invisible to inspector because it serves at `/api/rpc` not `/rpc` ## Context Architecture Hero OS contexts (Geomind, Incubaid, ThreeFold, Default) are the fundamental data scoping mechanism. A context is a workspace that isolates all user data — contacts, files, calendar, media, projects. The context selector in Hero OS (top-left) determines what every app shows. Context-awareness is built into the MCP pillar: context-scoped services route tool calls to the right data based on the user's active context. | Service | Context-aware? | Mechanism | |---------|---------------|-----------| | hero_osis | YES | Per-context sockets: `~/hero/var/sockets/{ctx}/hero_osis_server.sock` | | hero_foundry | YES | Per-context WebDAV: `~/hero/var/hero_foundry/webdav/{ctx}/` | | hero_books | NO | Global library | | hero_auth | NO | Global auth | | hero_inspector | NO | Global discovery | | hero_indexer | NO | Global search | The AI Island already passes `context_name` to the chat function. Today hero_shrimp doesn't use it because no context-aware MCP tools exist yet. As part of this work, context flows end-to-end: Hero OS → AI Island → hero_shrimp → `HERO_CONTEXT` env → bridge script → service endpoint. ## Known Blockers - [ ] **AI error on some queries** — "Help me set up my AI provider keys" returns error. hero_shrimp logs empty. Needs deeper investigation (try direct curl to shrimp /api/chat, check hero_aibroker). - [ ] **hero_indexer build broken** — hero_rpc commit `5094617` has `use hero_service::ZinitLifecycle` (should be `HeroLifecycle`). Fix on WIP branch `development_mik_34`, update Cargo.lock. ## Execution Plan — Learn Then Replicate ### Phase 0: Fix blockers - [ ] Debug AI error on herodev - [ ] Fix hero_rpc import on `development_mik_34` branch, update hero_indexer Cargo.lock ### Phase 1: hero_auth — validate bridge pattern (bridge only, MCP exists) - [ ] Create `mcp_auth` bridge script → `hero_auth_ui.sock` at `/mcp` - [ ] Add `mcp_auth` entry to `mcp_servers.json` - [ ] Deploy, test: "list all users", "who has admin access?" - **Learn**: confirms bridge pipeline end-to-end ### Phase 2: hero_foundry — first MCP addition (has OpenRPC) - [ ] Add MCP handler to `hero_foundry_server` (`POST /mcp`) - [ ] Curated tools: `repo_list_files`, `repo_read_file`, `repo_write_file`, `repo_find`, `server_list_repositories` - [ ] Context-aware: tools accept optional `context` param for WebDAV scoping - [ ] Create `mcp_foundry` bridge + config - [ ] Deploy, test: "show my files", "read the README" - **Learn**: establishes the template for adding MCP ### Phase 3: hero_inspector — self-MCP - [ ] Add `POST /mcp` (self) alongside existing `/mcp/:id` (gateway) - [ ] Tools: `services_list`, `service_detail`, `scan`, `status` - [ ] Create `mcp_inspector` bridge + config - [ ] Deploy, test: "what services are running?" ### Phase 4: hero_indexer — MCP for search - [ ] Add `POST /mcp` to hero_indexer_ui - [ ] Tools: `search`, `index_status`, `collections_list` - [ ] Proxy to backend via existing `proxy_to_socket()` - [ ] Create `mcp_indexer` bridge + config - [ ] Deploy, test: "find documents about ThreeFold" ### Phase 5: hero_osis — MCP for personal data (largest, context-critical) - [ ] Add `POST /mcp` to hero_osis_ui - [ ] All tools accept optional `context` parameter (default: session context) - [ ] Priority domains (~20-25 tools): identity, calendar, communication, files, projects - [ ] Create `mcp_osis` bridge + config - [ ] Deploy, test: "show my contacts", "create a calendar event" ### Phase 6: Context flow through to MCP - [ ] Pass `context_name` through hero_shrimp's `/api/chat` (currently ignored) - [ ] hero_shrimp injects `HERO_CONTEXT` into MCP bridge process env - [ ] Bridge scripts use `HERO_CONTEXT` for context-aware service routing ### Phase 7: OpenRPC gaps - [ ] hero_auth: add `rpc.discover` - [ ] hero_osis_ui: proxy `rpc.discover` to backend - [ ] hero_foundry_server: fix inspector path mismatch (`/api/rpc` vs `/rpc`) ### Phase 8: Build, deploy, full integration test - [ ] All MCP bridges working on herodev.gent04.grid.tf - [ ] AI can operate in each context ## Acceptance Criteria - [ ] All 6 services have `POST /mcp` with hand-crafted tool definitions - [ ] All wired to AI Assistant via bridge scripts + `mcp_servers.json` - [ ] AI can: list users, browse files, search documents, manage contacts/calendar, check service status - [ ] Context flows from Hero OS → AI → MCP tools for context-aware services - [ ] All deployed and verified on herodev - [ ] OpenRPC gaps addressed ## Related - Follows #31 (Self-Documenting Platform) — hero_books established the 4-pillar standard - #35 — Context creation broken from UI (separate, non-blocking)
mik-tf changed title from Wire all Hero services to AI Assistant via MCP to Bring all Hero services to 4-pillar standard (OpenRPC + MCP + Health + Socket) 2026-03-18 15:47:59 +00:00
mik-tf commented 2026-03-18 16:33:49 +00:00
Author
Owner
Copy link

Session 15 Progress — Phase 0b: hero_rpc fix ✅

Problem

hero_rpc commit 5094617 had a broken import in crates/server/src/server/server.rs:38:

use hero_service::ZinitLifecycle; // ZinitLifecycle doesn't exist in hero_service

hero_service exports HeroLifecycle (the renamed type). This broke compilation of hero_rpc_server for all downstream services — cargo compiles the entire crate including OServer even if consumers only use ZinitLifecycle via the type alias.

Fix

Branch: development_mik_34 on hero_rpc (commit 54d0258)

  • Changed use hero_service::ZinitLifecycle → use hero_service::HeroLifecycle in OServer
  • Updated all references within that file (type signature + doc comments)
  • The pub type ZinitLifecycle = HeroLifecycle alias in lib.rs preserves backwards compat for all 25+ downstream services

Verified

  • hero_indexer: branch development_mik_34 created, Cargo.toml updated to branch = "development_mik_34", Docker release build succeeded ✅
  • Two cosmetic deprecation warnings (ZinitLifecycle deprecated in favor of HeroLifecycle) — non-blocking, can be addressed in a separate cleanup

Assessment of my_init / hero_init

  • my_init (geomind_code) = evolution/rebrand of zinit itself, same service supervisor
  • hero_init (geomind_research) = cloud-init style provisioning tool, different purpose
  • Neither replaces the ZinitLifecycle/HeroLifecycle pattern — our 1-line fix is correct

Next: Phase 1 — hero_auth bridge wiring

## Session 15 Progress — Phase 0b: hero_rpc fix ✅ ### Problem hero_rpc commit `5094617` had a broken import in `crates/server/src/server/server.rs:38`: ```rust use hero_service::ZinitLifecycle; // ZinitLifecycle doesn't exist in hero_service ``` `hero_service` exports `HeroLifecycle` (the renamed type). This broke compilation of `hero_rpc_server` for **all downstream services** — cargo compiles the entire crate including OServer even if consumers only use `ZinitLifecycle` via the type alias. ### Fix Branch: `development_mik_34` on hero_rpc (commit `54d0258`) - Changed `use hero_service::ZinitLifecycle` → `use hero_service::HeroLifecycle` in OServer - Updated all references within that file (type signature + doc comments) - The `pub type ZinitLifecycle = HeroLifecycle` alias in `lib.rs` preserves backwards compat for all 25+ downstream services ### Verified - hero_indexer: branch `development_mik_34` created, Cargo.toml updated to `branch = "development_mik_34"`, **Docker release build succeeded** ✅ - Two cosmetic deprecation warnings (`ZinitLifecycle` deprecated in favor of `HeroLifecycle`) — non-blocking, can be addressed in a separate cleanup ### Assessment of my_init / hero_init - `my_init` (geomind_code) = evolution/rebrand of zinit itself, same service supervisor - `hero_init` (geomind_research) = cloud-init style provisioning tool, different purpose - Neither replaces the ZinitLifecycle/HeroLifecycle pattern — our 1-line fix is correct ### Next: Phase 1 — hero_auth bridge wiring
mik-tf commented 2026-03-18 16:57:08 +00:00
Author
Owner
Copy link

Session 15 Progress — Phases 1-6 Complete

Phase 1: hero_auth bridge ✅

  • Created mcp_auth bridge script
  • Added to mcp_servers.json
  • Socket: hero_auth_ui.sock (verified live)

Phase 2: hero_foundry MCP ✅

  • New src/http/mcp.rs with 5 curated tools (repo_list_files, repo_read_file, repo_write_file, repo_find, server_list_repositories)
  • Added /mcp route + /rpc alias (fixes inspector discovery)
  • mcp_foundry bridge script

Phase 3: hero_inspector self-MCP ✅

  • Self-registers as ServiceEntry in own cache → existing MCP gateway handles it automatically
  • No new handler code — reuses openrpc_to_mcp_tools() conversion
  • mcp_inspector bridge routes to /mcp/hero_inspector

Phase 4: hero_indexer MCP ✅

  • MCP handler in main.rs with 3 tools (search, index_status, collections_list)
  • Proxies to backend via existing socket mechanism
  • mcp_indexer bridge script

Phase 5: hero_osis MCP ✅

  • Context-aware MCP with 14 tools across 5 domains (identity, calendar, communication, files, projects)
  • All tools accept optional context param, defaults to HERO_CONTEXT env → "default"
  • Refactored rpc_proxy_inner into reusable rpc_call_inner for MCP dispatch
  • mcp_osis bridge script

Phase 6: Context plumbing ✅

  • hero_archipelagos: context_name now serialized in ChatRequest to shrimp
  • hero_shrimp: context flows admin.ts → contract.ts → agent.ts → ToolContext
  • LLM system prompt includes context hint so it passes context param to MCP tools

Build infrastructure

  • build-local.sh updated to copy MCP bridge scripts to dist/bin/
  • mcp_servers.json auto-transformed to mcp.json (Claude Desktop format) for shrimp

All commits (local, not pushed yet)

Repo Branch Changes
hero_rpc development_mik_34 ZinitLifecycle → HeroLifecycle fix
hero_foundry development_mik_34 MCP endpoint + /rpc alias
hero_inspector development_mik_34 Self-registration for MCP
hero_indexer development_mik_34 MCP endpoint (Cargo.toml + hero_rpc fix)
hero_indexer_ui development_mik_34 MCP endpoint
hero_osis development_mik_34 Context-aware MCP endpoint
hero_services development_mik_34 5 bridge scripts + build fixes
hero_archipelagos development_mik_34 Context plumbing
hero_aibroker development_mik_6_1 mcp_servers.json (7 service entries)
hero_shrimp development_mik_6_1 Context plumbing (5 files)

Related issues filed

  • Comment on #27: proposed MCP as auto-generated endpoint on HeroRpcServer
  • #36: Clean MCP architecture (shrimp direct spawning, aibroker as pure LLM proxy)

Next: Phase 8 — Build in Docker, deploy to herodev, test end-to-end

## Session 15 Progress — Phases 1-6 Complete ### Phase 1: hero_auth bridge ✅ - Created `mcp_auth` bridge script - Added to `mcp_servers.json` - Socket: `hero_auth_ui.sock` (verified live) ### Phase 2: hero_foundry MCP ✅ - New `src/http/mcp.rs` with 5 curated tools (repo_list_files, repo_read_file, repo_write_file, repo_find, server_list_repositories) - Added `/mcp` route + `/rpc` alias (fixes inspector discovery) - `mcp_foundry` bridge script ### Phase 3: hero_inspector self-MCP ✅ - Self-registers as ServiceEntry in own cache → existing MCP gateway handles it automatically - No new handler code — reuses `openrpc_to_mcp_tools()` conversion - `mcp_inspector` bridge routes to `/mcp/hero_inspector` ### Phase 4: hero_indexer MCP ✅ - MCP handler in `main.rs` with 3 tools (search, index_status, collections_list) - Proxies to backend via existing socket mechanism - `mcp_indexer` bridge script ### Phase 5: hero_osis MCP ✅ - Context-aware MCP with 14 tools across 5 domains (identity, calendar, communication, files, projects) - All tools accept optional `context` param, defaults to `HERO_CONTEXT` env → "default" - Refactored `rpc_proxy_inner` into reusable `rpc_call_inner` for MCP dispatch - `mcp_osis` bridge script ### Phase 6: Context plumbing ✅ - hero_archipelagos: `context_name` now serialized in ChatRequest to shrimp - hero_shrimp: context flows admin.ts → contract.ts → agent.ts → ToolContext - LLM system prompt includes context hint so it passes `context` param to MCP tools ### Build infrastructure - `build-local.sh` updated to copy MCP bridge scripts to `dist/bin/` - `mcp_servers.json` auto-transformed to `mcp.json` (Claude Desktop format) for shrimp ### All commits (local, not pushed yet) | Repo | Branch | Changes | |------|--------|---------| | hero_rpc | `development_mik_34` | ZinitLifecycle → HeroLifecycle fix | | hero_foundry | `development_mik_34` | MCP endpoint + /rpc alias | | hero_inspector | `development_mik_34` | Self-registration for MCP | | hero_indexer | `development_mik_34` | MCP endpoint (Cargo.toml + hero_rpc fix) | | hero_indexer_ui | `development_mik_34` | MCP endpoint | | hero_osis | `development_mik_34` | Context-aware MCP endpoint | | hero_services | `development_mik_34` | 5 bridge scripts + build fixes | | hero_archipelagos | `development_mik_34` | Context plumbing | | hero_aibroker | `development_mik_6_1` | mcp_servers.json (7 service entries) | | hero_shrimp | `development_mik_6_1` | Context plumbing (5 files) | ### Related issues filed - Comment on #27: proposed MCP as auto-generated endpoint on HeroRpcServer - #36: Clean MCP architecture (shrimp direct spawning, aibroker as pure LLM proxy) ### Next: Phase 8 — Build in Docker, deploy to herodev, test end-to-end
mik-tf commented 2026-03-18 17:07:03 +00:00
Author
Owner
Copy link

Session 15 — Phase 8: Deploy & Test Results

Deployed to herodev ✅

All binaries rebuilt in Docker, image packed and pushed as :herodev, container restarted.

MCP Endpoint Test Results

Service MCP Status Tools Notes
hero_auth ✅ Working 9 tools health, stats, users_list, users_add, users_delete, users_update_scope, login, token_create, token_validate
hero_foundry ✅ Working 5 tools repo_list_files, repo_read_file, repo_write_file, repo_find, server_list_repositories
hero_foundry /rpc alias ✅ Working — Inspector now discovers 32 methods (was 0 due to path mismatch)
hero_indexer ✅ Working 3 tools search, index_status, collections_list
hero_osis ✅ Working 13 tools contacts, events, messages, channels, files, projects (all context-aware)
hero_inspector self-MCP ⚠️ Gateway lookup issue 0 tools Self-registration works (appears in service list with 18 methods), but MCP gateway can't proxy to itself — needs source format fix. Non-blocking: inspector's own methods accessible via /rpc.

What's working

  • 4 out of 5 new MCP endpoints fully operational
  • 30+ AI-accessible tools across services
  • hero_foundry now visible to inspector (was invisible due to /rpc vs /api/rpc mismatch)
  • All bridge scripts deployed to /root/hero/bin/ in container

Remaining items

  • Inspector self-MCP gateway lookup (minor — service available via RPC)
  • WASM rebuild needed for context plumbing to flow from UI → shrimp
  • End-to-end AI Assistant test (requires WASM rebuild for context, but MCP tools work independently)

Related

  • Comment posted on #27 proposing MCP as auto-generated HeroRpcServer endpoint
  • #36 filed for clean MCP architecture (shrimp direct, aibroker LLM-only)
## Session 15 — Phase 8: Deploy & Test Results ### Deployed to herodev ✅ All binaries rebuilt in Docker, image packed and pushed as `:herodev`, container restarted. ### MCP Endpoint Test Results | Service | MCP Status | Tools | Notes | |---------|-----------|-------|-------| | hero_auth | ✅ **Working** | 9 tools | health, stats, users_list, users_add, users_delete, users_update_scope, login, token_create, token_validate | | hero_foundry | ✅ **Working** | 5 tools | repo_list_files, repo_read_file, repo_write_file, repo_find, server_list_repositories | | hero_foundry `/rpc` alias | ✅ **Working** | — | Inspector now discovers 32 methods (was 0 due to path mismatch) | | hero_indexer | ✅ **Working** | 3 tools | search, index_status, collections_list | | hero_osis | ✅ **Working** | 13 tools | contacts, events, messages, channels, files, projects (all context-aware) | | hero_inspector self-MCP | ⚠️ Gateway lookup issue | 0 tools | Self-registration works (appears in service list with 18 methods), but MCP gateway can't proxy to itself — needs source format fix. Non-blocking: inspector's own methods accessible via `/rpc`. | ### What's working - 4 out of 5 new MCP endpoints fully operational - 30+ AI-accessible tools across services - hero_foundry now visible to inspector (was invisible due to `/rpc` vs `/api/rpc` mismatch) - All bridge scripts deployed to `/root/hero/bin/` in container ### Remaining items - Inspector self-MCP gateway lookup (minor — service available via RPC) - WASM rebuild needed for context plumbing to flow from UI → shrimp - End-to-end AI Assistant test (requires WASM rebuild for context, but MCP tools work independently) ### Related - Comment posted on #27 proposing MCP as auto-generated HeroRpcServer endpoint - #36 filed for clean MCP architecture (shrimp direct, aibroker LLM-only)
mik-tf commented 2026-03-18 17:10:30 +00:00
Author
Owner
Copy link

Session 15 — Comprehensive Status & Roadmap

What's done (all on WIP branches, NOT merged to development)

Commits by repo

Repo Branch Commits Summary
hero_rpc development_mik_34 54d0258 Fix ZinitLifecycle → HeroLifecycle in OServer
hero_foundry development_mik_34 9e07628 MCP endpoint (5 tools) + /rpc alias
hero_inspector development_mik_34 253472e Self-registers in cache for MCP gateway
hero_indexer development_mik_34 50f712d, 9f40a7d MCP endpoint (3 tools) + hero_rpc branch update
hero_indexer_ui development_mik_34 d7ec91f MCP endpoint (3 tools)
hero_osis development_mik_34 709ef6d Context-aware MCP (13 tools, 5 domains)
hero_services development_mik_34 ddb7160, 3cea7f7 6 bridge scripts + build-local.sh MCP distribution
hero_archipelagos development_mik_34 99e3009 Wire context_name to ChatRequest
hero_aibroker development_mik_6_1 469edae mcp_servers.json with 7 service entries
hero_shrimp development_mik_6_1 cb9557f Context plumbing through agent pipeline

Live verification on herodev

Service MCP Tools Status
hero_auth 9 tools (users, tokens, health) ✅ Working
hero_foundry 5 tools (files, repos) + 32 OpenRPC methods now discoverable ✅ Working
hero_indexer 3 tools (search, status, collections) ✅ Working
hero_osis 13 tools (contacts, calendar, messages, files, projects) ✅ Working
hero_inspector Self-registered (18 methods), gateway lookup issue ⚠️ Minor fix needed
hero_books 6 tools (existing, reference impl) ✅ Already working

What remains for 100%

  1. Inspector self-MCP gateway fix — self-registration works but MCP gateway can't proxy to its own socket. Need to fix source format matching in gateway lookup.
  2. WASM rebuild — hero_archipelagos AI Island needs dx build for context_name to flow from UI dropdown → shrimp. MCP tools work independently without this.
  3. hero_shrimp Bun rebuild — needs bun build --compile for context plumbing changes to take effect in the compiled binary deployed to herodev.
  4. End-to-end AI test — verify "show my contacts" / "search for deployment guides" / "what services are running?" work through the full chain.

Ecosystem Roadmap — How all issues fit together

Current state (development):
  Services use ZinitLifecycle, fragmented server patterns,
  hand-written MCP endpoints, two MCP config formats

#34 (this issue) → Merge WIP branches:
  All services have MCP endpoints, bridge scripts,
  AI Assistant can use 30+ tools across services,
  context flows from UI to tools

#27 (Timur) → Merge development_home27:
  All services use HeroRpcServer/HeroUiServer,
  hero_init replaces zinit, OSIS single-socket,
  /mcp auto-generated from OpenRPC (proposed)
  → Small adaptation: OSIS MCP switches to _context param

#36 (new) → Clean MCP architecture:
  Shrimp owns MCP directly (Claude Desktop pattern),
  aibroker becomes pure LLM proxy,
  single mcp.json config, remove duplicate paths

#30 → Deprecate OServer:
  Remove legacy code after #27 migration complete

#29 → Standard tests:
  E2E test harness for all services

End state:
  OpenRPC = single source of truth
  MCP/SDKs/docs/discovery = all derived
  Clean separation: aibroker=LLM, shrimp=agent+tools
  Every service: HeroRpcServer + auto-MCP + health + socket

Design principle established

OpenRPC is the single source of truth — same philosophy as OSIS generating from OTML schemas. MCP tools, client SDKs, documentation, and service discovery are all auto-generated from the OpenRPC spec. Hand-written MCP handlers from #34 are temporary scaffolding until HeroRpcServer auto-generates /mcp (proposed extension of #27).

## Session 15 — Comprehensive Status & Roadmap ### What's done (all on WIP branches, NOT merged to development) #### Commits by repo | Repo | Branch | Commits | Summary | |------|--------|---------|--------| | hero_rpc | `development_mik_34` | `54d0258` | Fix ZinitLifecycle → HeroLifecycle in OServer | | hero_foundry | `development_mik_34` | `9e07628` | MCP endpoint (5 tools) + `/rpc` alias | | hero_inspector | `development_mik_34` | `253472e` | Self-registers in cache for MCP gateway | | hero_indexer | `development_mik_34` | `50f712d`, `9f40a7d` | MCP endpoint (3 tools) + hero_rpc branch update | | hero_indexer_ui | `development_mik_34` | `d7ec91f` | MCP endpoint (3 tools) | | hero_osis | `development_mik_34` | `709ef6d` | Context-aware MCP (13 tools, 5 domains) | | hero_services | `development_mik_34` | `ddb7160`, `3cea7f7` | 6 bridge scripts + build-local.sh MCP distribution | | hero_archipelagos | `development_mik_34` | `99e3009` | Wire context_name to ChatRequest | | hero_aibroker | `development_mik_6_1` | `469edae` | mcp_servers.json with 7 service entries | | hero_shrimp | `development_mik_6_1` | `cb9557f` | Context plumbing through agent pipeline | #### Live verification on herodev | Service | MCP Tools | Status | |---------|----------|--------| | hero_auth | 9 tools (users, tokens, health) | ✅ Working | | hero_foundry | 5 tools (files, repos) + 32 OpenRPC methods now discoverable | ✅ Working | | hero_indexer | 3 tools (search, status, collections) | ✅ Working | | hero_osis | 13 tools (contacts, calendar, messages, files, projects) | ✅ Working | | hero_inspector | Self-registered (18 methods), gateway lookup issue | ⚠️ Minor fix needed | | hero_books | 6 tools (existing, reference impl) | ✅ Already working | ### What remains for 100% 1. **Inspector self-MCP gateway fix** — self-registration works but MCP gateway can't proxy to its own socket. Need to fix source format matching in gateway lookup. 2. **WASM rebuild** — hero_archipelagos AI Island needs `dx build` for context_name to flow from UI dropdown → shrimp. MCP tools work independently without this. 3. **hero_shrimp Bun rebuild** — needs `bun build --compile` for context plumbing changes to take effect in the compiled binary deployed to herodev. 4. **End-to-end AI test** — verify "show my contacts" / "search for deployment guides" / "what services are running?" work through the full chain. ### Ecosystem Roadmap — How all issues fit together ``` Current state (development): Services use ZinitLifecycle, fragmented server patterns, hand-written MCP endpoints, two MCP config formats #34 (this issue) → Merge WIP branches: All services have MCP endpoints, bridge scripts, AI Assistant can use 30+ tools across services, context flows from UI to tools #27 (Timur) → Merge development_home27: All services use HeroRpcServer/HeroUiServer, hero_init replaces zinit, OSIS single-socket, /mcp auto-generated from OpenRPC (proposed) → Small adaptation: OSIS MCP switches to _context param #36 (new) → Clean MCP architecture: Shrimp owns MCP directly (Claude Desktop pattern), aibroker becomes pure LLM proxy, single mcp.json config, remove duplicate paths #30 → Deprecate OServer: Remove legacy code after #27 migration complete #29 → Standard tests: E2E test harness for all services End state: OpenRPC = single source of truth MCP/SDKs/docs/discovery = all derived Clean separation: aibroker=LLM, shrimp=agent+tools Every service: HeroRpcServer + auto-MCP + health + socket ``` ### Design principle established OpenRPC is the single source of truth — same philosophy as OSIS generating from OTML schemas. MCP tools, client SDKs, documentation, and service discovery are all auto-generated from the OpenRPC spec. Hand-written MCP handlers from #34 are temporary scaffolding until HeroRpcServer auto-generates `/mcp` (proposed extension of #27).
mik-tf commented 2026-03-18 17:35:30 +00:00
Author
Owner
Copy link

Session 15 — Final Verification ✅

All 6 MCP endpoints verified on herodev

Service tools/list tools/call Tools
hero_auth ✅ ✅ users_list returns data 9: health, stats, users_list, users_add, users_delete, users_update_scope, login, token_create, token_validate
hero_books ✅ (existing, proven) 6: libraries_list, books_list, book_pages, page_read, search, ask
hero_foundry ✅ ✅ server_list_repositories returns data 5: repo_list_files, repo_read_file, repo_write_file, repo_find, server_list_repositories
hero_indexer ✅ (backend-dependent) 3: search, index_status, collections_list
hero_osis ✅ ✅ contacts_list(context=root) returns data 13: contacts CRUD, events CRUD, messages, channels, files, projects
hero_inspector ✅ (via gateway) 18: all inspector methods auto-converted from OpenRPC

Total: 54 MCP tools across 6 services

Fixes applied

  • Inspector MCP gateway proxy_to_service() now sends to /rpc instead of / (was causing 404 for all services' tools/call)
  • Inspector bridge script connects to hero_inspector_ui.sock (MCP gateway) not hero_inspector_server.sock (raw RPC)
  • hero_foundry /rpc alias enables inspector to discover 32 OpenRPC methods (was 0)

Remaining

  • WASM rebuild for context flow from UI (building now)
  • Full end-to-end AI chat test

Master roadmap filed as #38

Includes #18 (AI broker code generation), #27 (HeroRpcServer), #36 (clean MCP arch), and all other issues in dependency order.

## Session 15 — Final Verification ✅ ### All 6 MCP endpoints verified on herodev | Service | tools/list | tools/call | Tools | |---------|-----------|-----------|-------| | hero_auth | ✅ | ✅ users_list returns data | 9: health, stats, users_list, users_add, users_delete, users_update_scope, login, token_create, token_validate | | hero_books | ✅ | (existing, proven) | 6: libraries_list, books_list, book_pages, page_read, search, ask | | hero_foundry | ✅ | ✅ server_list_repositories returns data | 5: repo_list_files, repo_read_file, repo_write_file, repo_find, server_list_repositories | | hero_indexer | ✅ | (backend-dependent) | 3: search, index_status, collections_list | | hero_osis | ✅ | ✅ contacts_list(context=root) returns data | 13: contacts CRUD, events CRUD, messages, channels, files, projects | | hero_inspector | ✅ | (via gateway) | 18: all inspector methods auto-converted from OpenRPC | **Total: 54 MCP tools across 6 services** ### Fixes applied - Inspector MCP gateway `proxy_to_service()` now sends to `/rpc` instead of `/` (was causing 404 for all services' `tools/call`) - Inspector bridge script connects to `hero_inspector_ui.sock` (MCP gateway) not `hero_inspector_server.sock` (raw RPC) - hero_foundry `/rpc` alias enables inspector to discover 32 OpenRPC methods (was 0) ### Remaining - WASM rebuild for context flow from UI (building now) - Full end-to-end AI chat test ### Master roadmap filed as #38 Includes #18 (AI broker code generation), #27 (HeroRpcServer), #36 (clean MCP arch), and all other issues in dependency order.
mik-tf commented 2026-03-18 18:17:05 +00:00
Author
Owner
Copy link

Session 15 — Final Status

Done (verified on herodev from development branch)

  • 54 MCP tools across 6 services (auth 9, books 6, foundry 5, indexer 3, osis 13, inspector 18)
  • All tools/list and tools/call verified via direct curl
  • 6 bridge scripts + mcp.json deployed
  • 39 WASM islands rebuilt (including AI context fix)
  • Inspector MCP gateway proxy fixed (/ to /rpc)
  • 8 repos merged to development
  • Issues #36, #37, #38 filed

Remaining: shrimp-to-bridge stdio integration

Shrimp spawns bridge scripts but MCP initialize handshake times out. Direct curl to endpoints works perfectly. Issue is in the compiled Bun binary stdio pipe to shell scripts. Next session: debug with shrimp from source, add logging to mcp_manager.ts.

## Session 15 — Final Status ### Done (verified on herodev from development branch) - 54 MCP tools across 6 services (auth 9, books 6, foundry 5, indexer 3, osis 13, inspector 18) - All tools/list and tools/call verified via direct curl - 6 bridge scripts + mcp.json deployed - 39 WASM islands rebuilt (including AI context fix) - Inspector MCP gateway proxy fixed (/ to /rpc) - 8 repos merged to development - Issues #36, #37, #38 filed ### Remaining: shrimp-to-bridge stdio integration Shrimp spawns bridge scripts but MCP initialize handshake times out. Direct curl to endpoints works perfectly. Issue is in the compiled Bun binary stdio pipe to shell scripts. Next session: debug with shrimp from source, add logging to mcp_manager.ts.
mik-tf commented 2026-03-18 19:34:37 +00:00
Author
Owner
Copy link

Session 15 Handoff — Ready for Session 16

Completed

  • 54 MCP tools across 6 services, verified via direct curl on herodev
  • 8 repos merged to development, deployed to herodev
  • 39 WASM islands rebuilt with context plumbing
  • Inspector MCP gateway proxy fix + bridge timeout fixes
  • Issues #36 (clean MCP arch), #37 (preflight), #38 (master roadmap) filed

Blocker for next session

Shrimp cannot load bridge scripts via stdio MCP protocol. Bridge scripts work perfectly via direct curl. The issue is in the Bun compiled binary spawning shell scripts and doing JSON-RPC over stdin/stdout pipes. Shrimp starts, spawns bridges, sends initialize, but never receives a response — bridges timeout after 15s.

Next session priorities

  1. Debug shrimp→bridge stdio pipe (run from source for logs)
  2. Fix pipe issue and verify all 54 tools load in shrimp
  3. End-to-end AI test through Hero OS chat UI
  4. Continue #38 roadmap

Branch state

  • development: all MCP endpoints merged (8 repos)
  • development_mik_6_1: hero_aibroker + hero_shrimp (context plumbing + MCP config, not merged)
## Session 15 Handoff — Ready for Session 16 ### Completed - 54 MCP tools across 6 services, verified via direct curl on herodev - 8 repos merged to `development`, deployed to herodev - 39 WASM islands rebuilt with context plumbing - Inspector MCP gateway proxy fix + bridge timeout fixes - Issues #36 (clean MCP arch), #37 (preflight), #38 (master roadmap) filed ### Blocker for next session **Shrimp cannot load bridge scripts via stdio MCP protocol.** Bridge scripts work perfectly via direct curl. The issue is in the Bun compiled binary spawning shell scripts and doing JSON-RPC over stdin/stdout pipes. Shrimp starts, spawns bridges, sends `initialize`, but never receives a response — bridges timeout after 15s. ### Next session priorities 1. Debug shrimp→bridge stdio pipe (run from source for logs) 2. Fix pipe issue and verify all 54 tools load in shrimp 3. End-to-end AI test through Hero OS chat UI 4. Continue #38 roadmap ### Branch state - `development`: all MCP endpoints merged (8 repos) - `development_mik_6_1`: hero_aibroker + hero_shrimp (context plumbing + MCP config, not merged)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
development_home27
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/home#34
No description provided.