tfgrid/zosbuilder
11
0
Fork 1
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge commit '10025f9fa5503865918cbae2af5366afe7fd7c54' as 'components/mycelium'

Browse Source
This commit is contained in:
Jan De Landtsheer
2025-08-16 21:12:34 +02:00
parent 9293ee45cb 10025f9fa5
commit 3d6caf7d91
132 changed files with 50951 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

990
components/mycelium/docs/api.yaml Normal file
View File

@@ -0,0 +1,990 @@
openapi: 3.0.2
info:
version: "1.0.0"
title: Mycelium management
contact:
url: "https://github.com/threefoldtech/mycelium"
license:
name: Apache 2.0
url: "https://github.com/threefoldtech/mycelium/blob/master/LICENSE"
description: |
This is the specification of the **mycelium** management API. It is used to perform admin tasks on the system, and
to perform administrative duties.
externalDocs:
description: For full documentation, check out the mycelium github repo.
url: "https://github.com/threefoldtech/mycelium"
tags:
- name: Admin
description: Administrative operations
- name: Peer
description: Operations related to peer management
- name: Route
description: Operations related to network routes
- name: Message
description: Operations on the embedded message subsystem
- name: Topic
description: Operations related to message topic configuration
servers:
- url: "http://localhost:8989"
paths:
"/api/v1/admin":
get:
tags:
- Admin
summary: Get general info about the node
description: |
Get general info about the node, which is not related to other more specific functionality
operationId: getInfo
responses:
"200":
description: Success
content:
application/json:
schema:
$ref: "#/components/schemas/Info"
"/api/v1/admin/peers":
get:
tags:
- Admin
- Peer
summary: List known peers
description: |
List all peers known in the system, and info about their connection.
This includes the endpoint, how we know about the peer, the connection state, and if the connection is alive the amount
of bytes we've sent to and received from the peer.
operationId: getPeers
responses:
"200":
description: Success
content:
application/json:
schema:
type: array
items:
$ref: "#/components/schemas/PeerStats"
post:
tags:
- Admin
- Peer
summary: Add a new peer
description: |
Add a new peer identified by the provided endpoint.
The peer is added to the list of known peers. It will eventually be connected
to by the standard connection loop of the peer manager. This means that a peer
which can't be connected to will stay in the system, as it might be reachable
later on.
operationId: addPeer
responses:
"204":
description: Peer added
"400":
description: Malformed endpoint
content:
text/plain:
schema:
type: string
description: Details about why the endpoint is not valid
"409":
description: Peer already exists
content:
text/plain:
schema:
type: string
description: message saying we already know this peer
"/api/v1/admin/peers/{endpoint}":
delete:
tags:
- Admin
- Peer
summary: Remove an existing peer
description: |
Remove an existing peer identified by the provided endpoint.
The peer is removed from the list of known peers. If a connection to it
is currently active, it will be closed.
operationId: deletePeer
responses:
"204":
description: Peer removed
"400":
description: Malformed endpoint
content:
text/plain:
schema:
type: string
description: Details about why the endpoint is not valid
"404":
description: Peer doesn't exist
content:
text/plain:
schema:
type: string
description: message saying we don't know this peer
"/api/v1/admin/routes/selected":
get:
tags:
- Admin
- Route
summary: List all selected routes
description: |
List all selected routes in the system, and their next hop identifier, metric and sequence number.
It is possible for a route to be selected and have an infinite metric. This route will however not forward packets.
operationId: getSelectedRoutes
responses:
"200":
description: Success
content:
application/json:
schema:
type: array
items:
$ref: "#/components/schemas/Route"
"/api/v1/admin/routes/fallback":
get:
tags:
- Admin
- Route
summary: List all active fallback routes
description: |
List all fallback routes in the system, and their next hop identifier, metric and sequence number.
These routes are available to be selected in case the selected route for a destination suddenly fails, or gets retracted.
operationId: getFallbackRoutes
responses:
"200":
description: Success
content:
application/json:
schema:
type: array
items:
$ref: "#/components/schemas/Route"
"/api/v1/admin/routes/queried":
get:
tags:
- Admin
- Route
summary: List all currently queried subnets
description: |
List all currently queried subnets in the system, and the amount of seconds until the query expires.
These subnets are actively being probed in the network. If no route to them is discovered before the query expires,
they will be marked as not reachable temporarily.
operationId: getQueriedSubnets
responses:
"200":
description: Success
content:
application/json:
schema:
type: array
items:
$ref: "#/components/schemas/QueriedSubnet"
"/api/v1/admin/routes/no_route":
get:
tags:
- Admin
- Route
summary: List all subnets which are explicitly marked as no route
description: |
List all subnets in the system which are marked no route, and the amount of seconds until the query expires.
These subnets have recently been probed in the network, and no route for them was discovered in time. No more
route requests will be send for these subnets until the entry expires.
operationId: getNoRouteEntries
responses:
"200":
description: Success
content:
application/json:
schema:
type: array
items:
$ref: "#/components/schemas/NoRouteSubnet"
"/api/v1/messages":
get:
tags:
- Message
summary: Get a message from the inbound message queue
description: |
Get a message from the inbound message queue. By default, the message is removed from the queue and won't be shown again.
If the peek query parameter is set to true, the message will be peeked, and the next call to this endpoint will show the same message.
This method returns immediately by default: a message is returned if one is ready, and if there isn't nothing is returned. If the timeout
query parameter is set, this call won't return for the given amount of seconds, unless a message is received
operationId: popMessage
parameters:
- in: query
name: peek
required: false
schema:
type: boolean
description: Whether to peek the message or not. If this is true, the message won't be removed from the inbound queue when it is read
example: true
- in: query
name: timeout
required: false
schema:
type: integer
format: int64
minimum: 0
description: |
Amount of seconds to wait for a message to arrive if one is not available. Setting this to 0 is valid and will return
a message if present, or return immediately if there isn't
example: 60
- in: query
name: topic
required: false
schema:
type: string
format: byte
minLength: 0
maxLength: 340
description: |
Optional filter for loading messages. If set, the system checks if the message has the given string at the start. This way
a topic can be encoded.
example: example.topic
responses:
"200":
description: Message retrieved
content:
application/json:
schema:
$ref: "#/components/schemas/InboundMessage"
"204":
description: No message ready
post:
tags:
- Message
summary: Submit a new message to the system.
description: |
Push a new message to the systems outbound message queue. The system will continuously attempt to send the message until
it is either fully transmitted, or the send deadline is expired.
operationId: pushMessage
parameters:
- in: query
name: reply_timeout
required: false
schema:
type: integer
format: int64
minimum: 0
description: |
Amount of seconds to wait for a reply to this message to come in. If not set, the system won't wait for a reply and return
the ID of the message, which can be used later. If set, the system will wait for at most the given amount of seconds for a reply
to come in. If a reply arrives, it is returned to the client. If not, the message ID is returned for later use.
example: 120
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/PushMessageBody"
responses:
"200":
description: We received a reply within the specified timeout
content:
application/json:
schema:
$ref: "#/components/schemas/InboundMessage"
"201":
description: Message pushed successfully, and not waiting for a reply
content:
application/json:
schema:
$ref: "#/components/schemas/PushMessageResponseId"
"408":
description: The system timed out waiting for a reply to the message
content:
application/json:
schema:
$ref: "#/components/schemas/PushMessageResponseId"
"/api/v1/messages/reply/{id}":
post:
tags:
- Message
summary: Reply to a message with the given ID
description: |
Submits a reply message to the system, where ID is an id of a previously received message. If the sender is waiting
for a reply, it will bypass the queue of open messages.
operationId: pushMessageReply
parameters:
- in: path
name: id
required: true
schema:
type: string
format: hex
minLength: 16
maxLength: 16
example: abcdef0123456789
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/PushMessageBody"
responses:
"204":
description: successfully submitted the reply
"/api/v1/messages/status/{id}":
get:
tags:
- Message
summary: Get the status of an outbound message
description: |
Get information about the current state of an outbound message. This can be used to check the transmission
state, size and destination of the message.
operationId: getMessageInfo
parameters:
- in: path
name: id
required: true
schema:
type: string
format: hex
minLength: 16
maxLength: 16
example: abcdef0123456789
responses:
"200":
description: Success
content:
application/json:
schema:
$ref: "#/components/schemas/MessageStatusResponse"
"404":
description: Message not found
"/api/v1/messages/topics/default":
get:
tags:
- Message
- Topic
summary: Get the default topic action
description: |
Get the default action for topics that are not explicitly configured (accept or reject).
operationId: getDefaultTopicAction
responses:
"200":
description: Success
content:
application/json:
schema:
$ref: "#/components/schemas/DefaultTopicActionResponse"
put:
tags:
- Message
- Topic
summary: Set the default topic action
description: |
Set the default action for topics that are not explicitly configured (accept or reject).
operationId: setDefaultTopicAction
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/DefaultTopicActionRequest"
responses:
"204":
description: Default topic action set successfully
"/api/v1/messages/topics":
get:
tags:
- Message
- Topic
summary: Get all configured topics
description: |
Get all topics that are explicitly configured in the system.
operationId: getTopics
responses:
"200":
description: Success
content:
application/json:
schema:
type: array
items:
type: string
format: byte
description: Base64 encoded topic identifier
post:
tags:
- Message
- Topic
summary: Add a new topic
description: |
Add a new topic to the system's whitelist.
operationId: addTopic
requestBody:
content:
application/json:
schema:
type: string
format: byte
description: The topic to add
responses:
"201":
description: Topic added successfully
"/api/v1/messages/topics/{topic}":
delete:
tags:
- Message
- Topic
summary: Remove a topic
description: |
Remove a topic from the system's whitelist.
operationId: removeTopic
parameters:
- in: path
name: topic
required: true
schema:
type: string
format: byte
description: The topic to remove (base64 encoded)
responses:
"204":
description: Topic removed successfully
"400":
description: Invalid topic format
"/api/v1/messages/topics/{topic}/sources":
get:
tags:
- Message
- Topic
summary: Get sources for a topic
description: |
Get all sources (subnets) that are allowed to send messages for a specific topic.
operationId: getTopicSources
parameters:
- in: path
name: topic
required: true
schema:
type: string
format: byte
description: The topic to get sources for (base64 encoded)
responses:
"200":
description: Success
content:
application/json:
schema:
type: array
items:
type: string
description: Subnet in CIDR notation
"400":
description: Invalid topic format
post:
tags:
- Message
- Topic
summary: Add a source to a topic
description: |
Add a source (subnet) that is allowed to send messages for a specific topic.
operationId: addTopicSource
parameters:
- in: path
name: topic
required: true
schema:
type: string
format: byte
description: The topic to add a source to (base64 encoded)
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/TopicSourceRequest"
responses:
"204":
description: Source added successfully
"400":
description: Invalid topic format or subnet
"/api/v1/messages/topics/{topic}/sources/{subnet}":
delete:
tags:
- Message
- Topic
summary: Remove a source from a topic
description: |
Remove a source (subnet) that is allowed to send messages for a specific topic.
operationId: removeTopicSource
parameters:
- in: path
name: topic
required: true
schema:
type: string
format: byte
description: The topic to remove a source from (base64 encoded)
- in: path
name: subnet
required: true
schema:
type: string
description: The subnet to remove as a source
responses:
"204":
description: Source removed successfully
"400":
description: Invalid topic format or subnet
"/api/v1/messages/topics/{topic}/forward":
get:
tags:
- Message
- Topic
summary: Get the forward socket for a topic
description: |
Get the socket path where messages for a specific topic are forwarded to.
operationId: getTopicForwardSocket
parameters:
- in: path
name: topic
required: true
schema:
type: string
format: byte
description: The topic to get the forward socket for (base64 encoded)
responses:
"200":
description: Success
content:
application/json:
schema:
type: string
nullable: true
description: The socket path where messages are forwarded to
"400":
description: Invalid topic format
put:
tags:
- Message
- Topic
summary: Set the forward socket for a topic
description: |
Set the socket path where messages for a specific topic should be forwarded to.
operationId: setTopicForwardSocket
parameters:
- in: path
name: topic
required: true
schema:
type: string
format: byte
description: The topic to set the forward socket for (base64 encoded)
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/TopicForwardSocketRequest"
responses:
"204":
description: Forward socket set successfully
"400":
description: Invalid topic format
delete:
tags:
- Message
- Topic
summary: Remove the forward socket for a topic
description: |
Remove the socket path where messages for a specific topic are forwarded to.
operationId: removeTopicForwardSocket
parameters:
- in: path
name: topic
required: true
schema:
type: string
format: byte
description: The topic to remove the forward socket for (base64 encoded)
responses:
"204":
description: Forward socket removed successfully
"400":
description: Invalid topic format
"/api/v1/pubkey/{mycelium_ip}":
get:
summary: Get the pubkey from node ip
description: |
Get the node's public key from it's IP address.
operationId: getPublicKeyFromIp
parameters:
- in: path
name: mycelium_ip
required: true
schema:
type: string
format: ipv6
example: 5fd:7636:b80:9ad0::1
responses:
"200":
description: Success
content:
application/json:
schema:
$ref: "#/components/schemas/PublicKeyResponse"
"404":
description: Public key not found
components:
schemas:
Info:
description: General information about a node
type: object
properties:
nodeSubnet:
description: The subnet owned by the node and advertised to peers
type: string
example: 54f:b680:ba6e:7ced::/64
nodePubkey:
description: The public key of the node
type: string
format: hex
minLength: 64
maxLength: 64
example: 02468ace13579bdf02468ace13579bdf02468ace13579bdf02468ace13579bdf
Endpoint:
description: Identification to connect to a peer
type: object
properties:
proto:
description: Protocol used
type: string
enum:
- "tcp"
- "quic"
example: tcp
socketAddr:
description: The socket address used
type: string
example: 192.0.2.6:9651
PeerStats:
description: Info about a peer
type: object
properties:
endpoint:
$ref: "#/components/schemas/Endpoint"
type:
description: How we know about this peer
type: string
enum:
- "static"
- "inbound"
- "linkLocalDiscovery"
example: static
connectionState:
description: The current state of the connection to the peer
type: string
enum:
- "alive"
- "connecting"
- "dead"
example: alive
txBytes:
description: The amount of bytes transmitted to this peer
type: integer
format: int64
minimum: 0
example: 464531564
rxBytes:
description: The amount of bytes received from this peer
type: integer
format: int64
minimum: 0
example: 64645089
Route:
description: Information about a route
type: object
properties:
subnet:
description: The overlay subnet for which this is the route
type: string
example: 469:1348:ab0c:a1d8::/64
nextHop:
description: A way to identify the next hop of the route, where forwarded packets will be sent
type: string
example: TCP 203.0.113.2:60128 <-> 198.51.100.27:9651
metric:
description: The metric of the route, an estimation of how long the packet will take to arrive at its final destination
oneOf:
- description: A finite metric value
type: integer
format: int32
minimum: 0
maximum: 65534
example: 13
- description: An infinite (unreachable) metric. This is always `infinite`
type: string
example: infinite
seqno:
description: the sequence number advertised with this route by the source
type: integer
format: int32
minimum: 0
maximum: 65535
example: 1
QueriedSubnet:
description: Information about a subnet currently being queried
type: object
properties:
subnet:
description: The overlay subnet which we are currently querying
type: string
example: 503:5478:df06:d79a::/64
expiration:
description: The amount of seconds until the query expires
type: string
example: "37"
NoRouteSubnet:
description: Information about a subnet which is marked as no route
type: object
properties:
subnet:
description: The overlay subnet which is marked
type: string
example: 503:5478:df06:d79a::/64
expiration:
description: The amount of seconds until the entry expires
type: string
example: "37"
InboundMessage:
description: A message received by the system
type: object
properties:
id:
description: Id of the message, hex encoded
type: string
format: hex
minLength: 16
maxLength: 16
example: 0123456789abcdef
srcIp:
description: Sender overlay IP address
type: string
format: ipv6
example: 449:abcd:0123:defa::1
srcPk:
description: Sender public key, hex encoded
type: string
format: hex
minLength: 64
maxLength: 64
example: fedbca9876543210fedbca9876543210fedbca9876543210fedbca9876543210
dstIp:
description: Receiver overlay IP address
type: string
format: ipv6
example: 34f:b680:ba6e:7ced:355f:346f:d97b:eecb
dstPk:
description: Receiver public key, hex encoded. This is the public key of the system
type: string
format: hex
minLength: 64
maxLength: 64
example: 02468ace13579bdf02468ace13579bdf02468ace13579bdf02468ace13579bdf
topic:
description: An optional message topic
type: string
format: byte
minLength: 0
maxLength: 340
example: hpV+
payload:
description: The message payload, encoded in standard alphabet base64
type: string
format: byte
example: xuV+
PushMessageBody:
description: A message to send to a given receiver
type: object
properties:
dst:
$ref: "#/components/schemas/MessageDestination"
topic:
description: An optional message topic
type: string
format: byte
minLength: 0
maxLength: 340
example: hpV+
payload:
description: The message to send, base64 encoded
type: string
format: byte
example: xuV+
MessageDestination:
oneOf:
- description: An IP in the subnet of the receiver node
type: object
properties:
ip:
description: The target IP of the message
format: ipv6
example: 449:abcd:0123:defa::1
- description: The hex encoded public key of the receiver node
type: object
properties:
pk:
description: The hex encoded public key of the target node
type: string
minLength: 64
maxLength: 64
example: bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32
PushMessageResponseId:
description: The ID generated for a message after pushing it to the system
type: object
properties:
id:
description: Id of the message, hex encoded
type: string
format: hex
minLength: 16
maxLength: 16
example: 0123456789abcdef
MessageStatusResponse:
description: Information about an outbound message
type: object
properties:
dst:
description: IP address of the receiving node
type: string
format: ipv6
example: 449:abcd:0123:defa::1
state:
$ref: "#/components/schemas/TransmissionState"
created:
description: Unix timestamp of when this message was created
type: integer
format: int64
example: 1649512789
deadline:
description: Unix timestamp of when this message will expire. If the message is not received before this, the system will give up
type: integer
format: int64
example: 1649513089
msgLen:
description: Length of the message in bytes
type: integer
minimum: 0
example: 27
TransmissionState:
description: The state of an outbound message in it's lifetime
oneOf:
- type: string
enum: ["pending", "received", "read", "aborted"]
example: "received"
- type: object
properties:
sending:
type: object
properties:
pending:
type: integer
minimum: 0
example: 5
sent:
type: integer
minimum: 0
example: 17
acked:
type: integer
minimum: 0
example: 3
example: "received"
PublicKeyResponse:
description: Public key requested based on a node's IP
type: object
properties:
NodePubKey:
type: string
format: hex
minLength: 64
maxLength: 64
example: 02468ace13579bdf02468ace13579bdf02468ace13579bdf02468ace13579bdf
DefaultTopicActionResponse:
description: Response for the default topic action
type: object
properties:
accept:
description: Whether unconfigured topics are accepted by default
type: boolean
example: true
DefaultTopicActionRequest:
description: Request to set the default topic action
type: object
properties:
accept:
description: Whether to accept unconfigured topics by default
type: boolean
example: true
TopicInfo:
description: Information about a configured topic
type: object
properties:
topic:
description: The topic identifier (base64 encoded)
type: string
format: byte
example: "ZXhhbXBsZS50b3BpYw=="
sources:
description: List of subnets that are allowed to send messages for this topic
type: array
items:
type: string
example: "503:5478:df06:d79a::/64"
forward_socket:
description: Optional socket path where messages for this topic are forwarded to
type: string
nullable: true
example: "/var/run/mycelium/topic_socket"
TopicSourceRequest:
description: Request to add a source to a topic whitelist
type: object
properties:
subnet:
description: The subnet to add as a source in CIDR notation
type: string
example: "503:5478:df06:d79a::/64"
TopicForwardSocketRequest:
description: Request to set a forward socket for a topic
type: object
properties:
socket_path:
description: The socket path where messages should be forwarded to
type: string
example: "/var/run/mycelium/topic_socket"

54
components/mycelium/docs/data_packet.md Normal file
View File

@@ -0,0 +1,54 @@
# Data packet
A `data packet` contains user specified data. This can be any data, as long as the sender and receiver
both understand what it is, without further help. Intermediate hops, which route the data have sufficient
information with the header to know where to forward the packet. In practice, the data will be encrypted
to avoid eavesdropping by intermediate hops.
## Packet header
The packet header has a fixed size of 36 bytes, with the following layout:
```
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Length | Hop Limit |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ Source IP +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ Destination IP +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```
The first 8 bits are reserved and must be set to 0.
The next 16 bits are used to specify the length of the body. It is expected that
the actual length of a packet does not exceed 65K right now, and overhead related
to encryption should be handled by the client before sending the packet.
The next byte is the hop-limit. Every node decrements this value by 1 before sending
the packet. If a node decrements this value to 0, the packet is discarded.
The next 16 bytes contain the sender IP address.
The final 16 bytes contain the destination IP address.
## Body
Following the header is a variable length body. The protocol does not have any requirements for the
body, and the only requirement imposed is that the body is as long as specified in the header length
field. It is technically legal according to the protocol to transmit a data packet without a body,
i.e. a body length of 0. This is useless however, as there will not be any data to interpret.

161
components/mycelium/docs/message.md Normal file
View File

@@ -0,0 +1,161 @@
# Message subsystem
The message subsystem can be used to send arbitrary length messages to receivers. A receiver is any
other node in the network. It can be identified both by its public key, or an IP address in its announced
range. The message subsystem can be interacted with both via the HTTP API, which is
[documented here](./api.yaml), or via the `mycelium` binary. By default, the messages do not interpret
the data in any way. When using the binary, the message is slightly modified to include an optional
topic at the start of the message. Note that in the HTTP API, all messages are encoded in base64. This
might make it difficult to consume these messages without additional tooling.
Messages can be categorized by topics, which can be configured with whitelisted subnets and socket forwarding paths.
For detailed information on how to configure topics, see the [Topic Configuration Guide](./topic_configuration.md).
## JSON-RPC API Examples
These examples assume you have at least 2 nodes running, and that they are both part of the same network.
Send a message on node1, waiting up to 2 minutes for a possible reply:
```json
{
"jsonrpc": "2.0",
"method": "pushMessage",
"params": [
{
"dst": {"pk": "bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32"},
"payload": "xuV+"
},
120
],
"id": 1
}
```
Using curl:
```bash
curl -X POST http://localhost:8990/rpc \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"method": "pushMessage",
"params": [
{
"dst": {"pk": "bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32"},
"payload": "xuV+"
},
120
],
"id": 1
}'
```
Listen for a message on node2. Note that messages received while nothing is listening are added to
a queue for later consumption. Wait for up to 1 minute.
```json
{
"jsonrpc": "2.0",
"method": "popMessage",
"params": [false, 60, null],
"id": 1
}
```
Using curl:
```bash
curl -X POST http://localhost:8990/rpc \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"method": "popMessage",
"params": [false, 60, null],
"id": 1
}'
```
The system will (immediately) receive our previously sent message:
```json
{"id":"e47b25063912f4a9","srcIp":"34f:b680:ba6e:7ced:355f:346f:d97b:eecb","srcPk":"955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b","dstIp":"2e4:9ace:9252:630:beee:e405:74c0:d876","dstPk":"bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32","payload":"xuV+"}
```
To send a reply, we can post a message on the reply path, with the received message `id` (still on
node2):
```json
{
"jsonrpc": "2.0",
"method": "pushMessageReply",
"params": [
"e47b25063912f4a9",
{
"dst": {"pk":"955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b"},
"payload": "xuC+"
}
],
"id": 1
}
```
Using curl:
```bash
curl -X POST http://localhost:8990/rpc \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"method": "pushMessageReply",
"params": [
"e47b25063912f4a9",
{
"dst": {"pk":"955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b"},
"payload": "xuC+"
}
],
"id": 1
}'
```
If you did this fast enough, the initial sender (node1) will now receive the reply.
## Mycelium binary examples
As explained above, while using the binary the message is slightly modified to insert the optional
topic. As such, when using the binary to send messages, it is suggested to make sure the receiver is
also using the binary to listen for messages. The options discussed here are not covering all possibilities,
use the `--help` flag (`mycelium message send --help` and `mycelium message receive --help`) for a
full overview.
Once again, send a message. This time using a topic (example.topic). Note that there are no constraints
on what a valid topic is, other than that it is valid UTF-8, and at most 255 bytes in size. The `--wait`
flag can be used to indicate that we are waiting for a reply. If it is set, we can also use an additional
`--timeout` flag to govern exactly how long (in seconds) to wait for. The default is to wait forever.
```bash
mycelium message send 2e4:9ace:9252:630:beee:e405:74c0:d876 'this is a message' -t example.topic --wait
```
On the second node, listen for messages with this topic. If a different topic is used, the previous
message won't be received. If no topic is set, all messages are received. An optional timeout flag
can be specified, which indicates how long to wait for. Absence of this flag will cause the binary
to wait forever.
```bash
mycelium message receive -t example.topic
```
Again, if the previous command was executed a message will be received immediately:
```json
{"id":"4a6c956e8d36381f","topic":"example.topic","srcIp":"34f:b680:ba6e:7ced:355f:346f:d97b:eecb","srcPk":"955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b","dstIp":"2e4:9ace:9252:630:beee:e405:74c0:d876","dstPk":"bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32","payload":"this is a message"}
```
And once again, we can use the ID from this message to reply to the original sender, who might be waiting
for this reply (notice we used the hex encoded public key to identify the receiver here, rather than an IP):
```bash
mycelium message send 955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b "this is a reply" --reply-to 4a6c956e8d36381f
```

1190
components/mycelium/docs/openrpc.json Normal file
View File

File diff suppressed because it is too large Load Diff

22
components/mycelium/docs/packet.md Normal file
View File

@@ -0,0 +1,22 @@
# Packet
A `Packet` is the largest communication object between established `peers`. All communication is done
via these `packets`. The `packet` itself consists of a fixed size header, and a variable size body.
The body contains a more specific type of data.
## Packet header
The packet header has a fixed size of 4 bytes, with the following layout:
```
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version | Type | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```
The first byte is used to indicate the version of the protocol. Currently, only version 1 is supported
(0x01). The next byte is used to indicate the type of the body. `0x00` indicates a data packet, while
`0x01` indicates a control packet. The remaining 16 bits are currently reserved, and should be set to
all 0.

35
components/mycelium/docs/private_network.md Normal file
View File

@@ -0,0 +1,35 @@
# Private network
> Private network functionality is currently in an experimental stage
While traffic is end-to-end encrypted in mycelium, any node in the network learns
every available connected subnet (and can derive the associated default address
in that subnet). As a result, running a mycelium node adds what is effectively
a public interface to your computer, so everyone can send traffic to it. On top
of this, the routing table consumes memory in relation to the amount of nodes in
the network. To remedy this, people can opt to run a "private network". By configuring
a pre shared key (and network name), only nodes which know the key associated to
the name can connect to your network.
## Implementation
Private networks are implemented entirely in the connection layer (no specific
protocol logic is implemented to support this). This relies on the pre shared key
functionality of TLS 1.3. As such, you need both a `network name` (an `identity`),
and the `PSK` itself. Next to the limitations in the protocol, we currently further
limit the network name and PSK as follows:
- Network name must be a UTF-8 encoded string of 2 to 64 bytes.
- PSK must be exactly 32 bytes.
Not all cipher suites supported in TLS1.3 are supported. At present, _at least_
`TLS_AES_128_GCM_SHA256` and `TLS_CHACHA20_POLY1305_SHA256` are supported.
## Enable private network
In order to use the private network implementation of `mycelium`, a separate `mycelium-private`
binary is available. Private network functionality can be enabled by setting both
the `network-name` and `network-key-file` flags on the command line. All nodes who
wish to join the network must use the same values for both flags.
> ⚠️ Network name is public, do not put any confidential data here.

211
components/mycelium/docs/topic_configuration.md Normal file
View File

@@ -0,0 +1,211 @@
# Topic Configuration Guide
This document explains how to configure message topics in Mycelium, including how to add new topics, configure socket forwarding paths, and manage whitelisted subnets.
## Overview
Mycelium's messaging system uses topics to categorize and route messages. Each topic can be configured with:
- **Whitelisted subnets**: IP subnets that are allowed to send messages to this topic
- **Forward socket**: A Unix domain socket path where messages for this topic will be forwarded
When a message is received with a topic that has a configured socket path, the content of the message is pushed to the socket, and the system waits for a reply from the socket, which is then sent back to the original sender.
## Configuration Using JSON-RPC API
The JSON-RPC API provides a comprehensive set of methods for managing topics, socket forwarding paths, and whitelisted subnets.
## Adding a New Topic
### Using the JSON-RPC API
```json
{
"jsonrpc": "2.0",
"method": "addTopic",
"params": ["dGVzdC10b3BpYw=="], // base64 encoding of "test-topic"
"id": 1
}
```
Example using curl:
```bash
curl -X POST http://localhost:8990/rpc \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"method": "addTopic",
"params": ["dGVzdC10b3BpYw=="],
"id": 1
}'
```
## Configuring a Socket Forwarding Path
When a topic is configured with a socket forwarding path, messages for that topic will be forwarded to the specified Unix domain socket instead of being pushed to the message queue.
### Using the JSON-RPC API
```json
{
"jsonrpc": "2.0",
"method": "setTopicForwardSocket",
"params": ["dGVzdC10b3BpYw==", "/path/to/socket"],
"id": 1
}
```
Example using curl:
```bash
curl -X POST http://localhost:8990/rpc \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"method": "setTopicForwardSocket",
"params": ["dGVzdC10b3BpYw==", "/path/to/socket"],
"id": 1
}'
```
```
## Adding a Whitelisted Subnet
Whitelisted subnets control which IP addresses are allowed to send messages to a specific topic. If a message is received from an IP that is not in the whitelist, it will be dropped.
### Using the JSON-RPC API
```json
{
"jsonrpc": "2.0",
"method": "addTopicSource",
"params": ["dGVzdC10b3BpYw==", "192.168.1.0/24"],
"id": 1
}
```
Example using curl:
```bash
curl -X POST http://localhost:8990/rpc \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"method": "addTopicSource",
"params": ["dGVzdC10b3BpYw==", "192.168.1.0/24"],
"id": 1
}'
```
```
## Setting the Default Topic Action
You can configure the default action to take for topics that don't have explicit whitelist configurations:
### Using the JSON-RPC API
```json
{
"jsonrpc": "2.0",
"method": "setDefaultTopicAction",
"params": [true],
"id": 1
}
```
Example using curl:
```bash
curl -X POST http://localhost:8990/rpc \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"method": "setDefaultTopicAction",
"params": [true],
"id": 1
}'
```
```
## Socket Protocol
When a message is forwarded to a socket, the raw message data is sent to the socket. The socket is expected to process the message and send a reply, which will be forwarded back to the original sender.
The socket protocol is simple:
1. The message data is written to the socket
2. The system waits for a reply from the socket (with a configurable timeout)
3. The reply data is read from the socket and sent back to the original sender
## Example: Creating a Socket Server
Here's an example of a simple socket server that echoes back the received data:
```rust
use std::{
io::{Read, Write},
os::unix::net::UnixListener,
path::Path,
thread,
};
fn main() {
let socket_path = "/tmp/mycelium-socket";
// Remove the socket file if it already exists
if Path::new(socket_path).exists() {
std::fs::remove_file(socket_path).unwrap();
}
// Create the Unix domain socket
let listener = UnixListener::bind(socket_path).unwrap();
println!("Socket server listening on {}", socket_path);
// Accept connections in a loop
for stream in listener.incoming() {
match stream {
Ok(mut stream) => {
// Spawn a thread to handle the connection
thread::spawn(move || {
// Read the data
let mut buffer = Vec::new();
stream.read_to_end(&mut buffer).unwrap();
println!("Received {} bytes", buffer.len());
// Process the data (in this case, just echo it back)
// In a real application, you would parse and process the message here
// Send the reply
stream.write_all(&buffer).unwrap();
println!("Sent reply");
});
}
Err(e) => {
eprintln!("Error accepting connection: {}", e);
}
}
}
}
```
## Troubleshooting
### Message Not Being Forwarded to Socket
1. Check that the topic is correctly configured with a socket path
2. Verify that the socket server is running and the socket file exists
3. Ensure that the sender's IP is in the whitelisted subnets for the topic
4. Check the logs for any socket connection or timeout errors
### Socket Server Not Receiving Messages
1. Verify that the socket path is correct and accessible
2. Check that the socket server has the necessary permissions to read/write to the socket
3. Ensure that the socket server is properly handling the connection
### Reply Not Being Sent Back
1. Verify that the socket server is sending a reply
2. Check for any timeout errors in the logs
3. Ensure that the original sender is still connected and able to receive the reply