manual removed files

This commit is contained in:
2024-04-15 17:49:09 +00:00
parent a567404ef3
commit c19931fd32
1763 changed files with 0 additions and 51340 deletions

View File

@@ -1,431 +0,0 @@
<h1>API</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [File Example](#file-example)
***
## Introduction
We provide an example of a YAML API file.
## File Example
```
openapi: 3.0.2
info:
version: '1.0.0'
title: Mycelium management
contact:
url: 'https://github.com/threefoldtech/mycelium'
license:
name: Apache 2.0
url: 'https://github.com/threefoldtech/mycelium/blob/master/LICENSE'
description: |
This is the specification of the **mycelium** management API. It is used to perform admin tasks on the system, and
to perform administrative duties.
externalDocs:
description: For full documentation, check out the mycelium github repo.
url: 'https://github.com/threefoldtech/mycelium'
tags:
- name: Admin
description: Administrative operations
- name: Peer
description: Operations related to peer management
- name: Message
description: Operations on the embedded message subsystem
servers:
- url: 'http://localhost:8989'
paths:
'/api/v1/peers':
get:
tags:
- Admin
- Peer
summary: List known peers
description: |
List all peers known in the system, and info about their connection.
This includes the endpoint, how we know about the peer, the connection state, and if the connection is alive the amount
of bytes we've sent to and received from the peer.
operationId: getPeers
responses:
'200':
description: Succes
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/PeerStats'
'/api/v1/messages':
get:
tags:
- Message
summary: Get a message from the inbound message queue
description: |
Get a message from the inbound message queue. By default, the message is removed from the queue and won't be shown again.
If the peek query parameter is set to true, the message will be peeked, and the next call to this endpoint will show the same message.
This method returns immediately by default: a message is returned if one is ready, and if there isn't nothing is returned. If the timeout
query parameter is set, this call won't return for the given amount of seconds, unless a message is received
operationId: popMessage
parameters:
- in: query
name: peek
required: false
schema:
type: boolean
description: Whether to peek the message or not. If this is true, the message won't be removed from the inbound queue when it is read
example: true
- in: query
name: timeout
required: false
schema:
type: integer
format: int64
minimum: 0
description: |
Amount of seconds to wait for a message to arrive if one is not available. Setting this to 0 is valid and will return
a message if present, or return immediately if there isn't
example: 60
- in: query
name: topic
required: false
schema:
type: string
format: byte
minLength: 0
maxLength: 340
description: |
Optional filter for loading messages. If set, the system checks if the message has the given string at the start. This way
a topic can be encoded.
example: example.topic
responses:
'200':
description: Message retrieved
content:
application/json:
schema:
$ref: '#/components/schemas/InboundMessage'
'204':
description: No message ready
post:
tags:
- Message
summary: Submit a new message to the system.
description: |
Push a new message to the systems outbound message queue. The system will continuously attempt to send the message until
it is either fully transmitted, or the send deadline is expired.
operationId: pushMessage
parameters:
- in: query
name: reply_timeout
required: false
schema:
type: integer
format: int64
minimum: 0
description: |
Amount of seconds to wait for a reply to this message to come in. If not set, the system won't wait for a reply and return
the ID of the message, which can be used later. If set, the system will wait for at most the given amount of seconds for a reply
to come in. If a reply arrives, it is returned to the client. If not, the message ID is returned for later use.
example: 120
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/PushMessageBody'
responses:
'200':
description: We received a reply within the specified timeout
content:
application/json:
schema:
$ref: '#/components/schemas/InboundMessage'
'201':
description: Message pushed successfully, and not waiting for a reply
content:
application/json:
schema:
$ref: '#/components/schemas/PushMessageResponseId'
'408':
description: The system timed out waiting for a reply to the message
content:
application/json:
schema:
$ref: '#/components/schemas/PushMessageResponseId'
'/api/v1/messsages/reply/{id}':
post:
tags:
- Message
summary: Reply to a message with the given ID
description: |
Submits a reply message to the system, where ID is an id of a previously received message. If the sender is waiting
for a reply, it will bypass the queue of open messages.
operationId: pushMessageReply
parameters:
- in: path
name: id
required: true
schema:
type: string
format: hex
minLength: 16
maxLength: 16
example: abcdef0123456789
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/PushMessageBody'
responses:
'204':
description: successfully submitted the reply
'/api/v1/messages/status/{id}':
get:
tags:
- Message
summary: Get the status of an outbound message
description: |
Get information about the current state of an outbound message. This can be used to check the transmission
state, size and destination of the message.
operationId: getMessageInfo
parameters:
- in: path
name: id
required: true
schema:
type: string
format: hex
minLength: 16
maxLength: 16
example: abcdef0123456789
responses:
'200':
description: Success
content:
application/json:
schema:
$ref: '#/components/schemas/MessageStatusResponse'
'404':
description: Message not found
components:
schemas:
Endpoint:
description: Identification to connect to a peer
type: object
properties:
proto:
description: Protocol used
type: string
enum:
- 'tcp'
- 'quic'
example: tcp
socketAddr:
description: The socket address used
type: string
example: 192.0.2.6:9651
PeerStats:
description: Info about a peer
type: object
properties:
endpoint:
$ref: '#/components/schemas/Endpoint'
type:
description: How we know about this peer
type: string
enum:
- 'static'
- 'inbound'
- 'linkLocalDiscovery'
example: static
connectionState:
description: The current state of the connection to the peer
type: string
enum:
- 'alive'
- 'connecting'
- 'dead'
example: alive
connectionTxBytes:
description: The amount of bytes transmitted on the current connection
type: integer
format: int64
minimum: 0
example: 464531564
connectionRxBytes:
description: The amount of bytes received on the current connection
type: integer
format: int64
minimum: 0
example: 64645089
InboundMessage:
description: A message received by the system
type: object
properties:
id:
description: Id of the message, hex encoded
type: string
format: hex
minLength: 16
maxLength: 16
example: 0123456789abcdef
srcIp:
description: Sender overlay IP address
type: string
format: ipv6
example: 249:abcd:0123:defa::1
srcPk:
description: Sender public key, hex encoded
type: string
format: hex
minLength: 64
maxLength: 64
example: fedbca9876543210fedbca9876543210fedbca9876543210fedbca9876543210
dstIp:
description: Receiver overlay IP address
type: string
format: ipv6
example: 34f:b680:ba6e:7ced:355f:346f:d97b:eecb
dstPk:
description: Receiver public key, hex encoded. This is the public key of the system
type: string
format: hex
minLength: 64
maxLength: 64
example: 02468ace13579bdf02468ace13579bdf02468ace13579bdf02468ace13579bdf
topic:
description: An optional message topic
type: string
format: byte
minLength: 0
maxLength: 340
example: hpV+
payload:
description: The message payload, encoded in standard alphabet base64
type: string
format: byte
example: xuV+
PushMessageBody:
description: A message to send to a given receiver
type: object
properties:
dst:
$ref: '#/components/schemas/MessageDestination'
topic:
description: An optional message topic
type: string
format: byte
minLength: 0
maxLength: 340
example: hpV+
payload:
description: The message to send, base64 encoded
type: string
format: byte
example: xuV+
MessageDestination:
oneOf:
- description: An IP in the subnet of the receiver node
type: object
properties:
ip:
description: The target IP of the message
format: ipv6
example: 249:abcd:0123:defa::1
- description: The hex encoded public key of the receiver node
type: object
properties:
pk:
description: The hex encoded public key of the target node
type: string
minLength: 64
maxLength: 64
example: bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32
PushMessageResponseId:
description: The ID generated for a message after pushing it to the system
type: object
properties:
id:
description: Id of the message, hex encoded
type: string
format: hex
minLength: 16
maxLength: 16
example: 0123456789abcdef
MessageStatusResponse:
description: Information about an outobund message
type: object
properties:
dst:
description: Ip address of the receiving node
type: string
format: ipv6
example: 249:abcd:0123:defa::1
state:
$ref: '#/components/schemas/TransmissionState'
created:
description: Unix timestamp of when this message was created
type: integer
format: int64
example: 1649512789
deadline:
description: Unix timestamp of when this message will expire. If the message is not received before this, the system will give up
type: integer
format: int64
example: 1649513089
msgLen:
description: Length of the message in bytes
type: integer
minimum: 0
example: 27
TransmissionState:
description: The state of an outbound message in it's lifetime
oneOf:
- type: string
enum: ['pending', 'received', 'read', 'aborted']
example: 'received'
- type: object
properties:
sending:
type: object
properties:
pending:
type: integer
minimum: 0
example: 5
sent:
type: integer
minimum: 0
example: 17
acked:
type: integer
minimum: 0
example: 3
example: 'received'
```

View File

@@ -1,66 +0,0 @@
<h1>Data Packet</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Packet Header](#packet-header)
- [Body](#body)
***
## Introduction
A `data packet` contains user specified data. This can be any data, as long as the sender and receiver
both understand what it is, without further help. Intermediate hops, which route the data have sufficient
information with the header to know where to forward the packet. In practice, the data will be encrypted
to avoid eavesdropping by intermediate hops.
## Packet Header
The packet header has a fixed size of 36 bytes, with the following layout:
```
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Reserved | Length | Hop Limit |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ Source IP +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ Destination IP +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```
The first 5 bits are reserved and must be set to 0.
The next 19 bits are used to specify the length of the body. It is expected that
the actual length of a packet does not exceed 256K right now, so the 19th bit is
only needed because we have to account for some overhead related to the encryption.
The next byte is the hop-limit. Every node decrements this value by 1 before sending
the packet. If a node decrements this value to 0, the packet is discarded.
The next 16 bytes contain the sender IP address.
The final 16 bytes contain the destination IP address.
## Body
Following the header is a variable length body. The protocol does not have any requirements for the
body, and the only requirement imposed is that the body is as long as specified in the header length
field. It is technically legal according to the protocol to transmit a data packet without a body,
i.e. a body length of 0. This is useless however, as there will not be any data to interpret.

View File

@@ -1,139 +0,0 @@
<h1>Additional Information</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Connect to Other Nodes](#connect-to-other-nodes)
- [Possible Peers](#possible-peers)
- [Default Port](#default-port)
- [Check Network Information](#check-network-information)
- [Test the Network](#test-the-network)
- [Key Pair](#key-pair)
- [Running without TUN interface](#running-without-tun-interface)
- [API](#api)
- [Message System](#message-system)
- [Inspecting Node Keys](#inspecting-node-keys)
***
## Introduction
We provide additional information concerning Mycelium and how to properly use it.
## Connect to Other Nodes
If you want to connect to other nodes, you can specify their listening address as
part of the command (combined with the protocol they are listening on, usually TCP);
```sh
mycelium --peers tcp://83.231.240.31:9651 quic://185.206.122.71:9651
```
If you are using other tun inferface, e.g. utun3 (default), you can set a different utun inferface
```sh
mycelium --peers tcp://83.231.240.31:9651 quic://185.206.122.71:9651 --tun-name utun9
```
## Possible Peers
Here are some possible peers.
```
tcp://146.185.93.83:9651
quic://83.231.240.31:9651
quic://185.206.122.71:9651
tcp://[2a04:f340:c0:71:28cc:b2ff:fe63:dd1c]:9651
tcp://[2001:728:1000:402:78d3:cdff:fe63:e07e]:9651
quic://[2a10:b600:1:0:ec4:7aff:fe30:8235]:9651
```
## Default Port
By default, the node will listen on port `9651`, though this can be overwritten with the `-p` flag.
## Check Network Information
You can check your Mycelium network information by running the following line:
```bash
mycelium inspect --json
```
Where a typical output would be:
```
{
"publicKey": "abd16194646defe7ad2318a0f0a69eb2e3fe939c3b0b51cf0bb88bb8028ecd1d",
"address": "3c4:c176:bf44:b2ab:5e7e:f6a:b7e2:11ca"
}
```
## Test the Network
You can easily test that the network works by pinging to anyone in the network.
```
ping6 3c4:c176:bf44:b2ab:5e7e:f6a:b7e2:11ca
```
## Key Pair
The node uses a `x25519` key pair from which its identity is derived. The private key of this key pair
is saved in a local file (32 bytes in binary format). You can specify the path to this file with the
`-k` flag. By default, the file is saved in the current working directory as `priv_key.bin`.
## Running without TUN interface
It is possible to run the system without creating a TUN interface, by starting with the `--no-tun` flag.
Obviously, this means that your node won't be able to send or receive L3 traffic. There is no interface
to send packets on, and consequently no interface to send received packets out of. From the point of
other nodes, your node will simply drop all incoming L3 traffic destined for it. The node **will still
route traffic** as normal. It takes part in routing, exchanges route info, and forwards packets not
intended for itself.
The node also still allows access to the [message subsystem](#message-system).
## API
The node starts an HTTP API, which by default listens on `localhost:8989`. A different listening address
can be specified on the CLI when starting the system through the `--api-server-addr` flag. The API
allows access to [send and receive messages](#message-system), and will later be expanded to allow
admin functionality on the system. Note that message are sent using the identity of the node, and a
future admin API can be used to change the system behavior. As such, care should be taken that this
API is not accessible to unauthorized users.
## Message System
A message system is provided which allows users to send a message, which is essentially just "some data"
to a remote. Since the system is end-to-end encrypted, a receiver of a message is sure of the authenticity
and confidentiality of the content. The system does not interpret the data in any way and handles it
as an opaque block of bytes. Messages are sent with a deadline. This means the system continuously
tries to send (part of) the message, until it either succeeds, or the deadline expires. This happens
similar to the way TCP handles data. Messages are transmitted in chunks, which are embedded in the
same data stream used by L3 packets. As such, intermediate nodes can't distinguish between regular L3
and message data.
The primary way to interact with the message system is through [the API](#API). The message API is
documented in [here](./api_yaml.md). For some more info about how to
use the message system, see [the Message section](./message.md).
## Inspecting Node Keys
Using the `inspect` subcommand, you can view the address associated with a public key. If no public key is provided, the node will show
its own public key. In either case, the derived address is also printed. You can specify the path to the private key with the `-k` flag.
If the file does not exist, a new private key will be generated. The optional `--json` flag can be used to print the information in json
format.
```sh
mycelium inspect a47c1d6f2a15b2c670d3a88fbe0aeb301ced12f7bcb4c8e3aa877b20f8559c02
```
Where the output could be something like this:
```sh
Public key: a47c1d6f2a15b2c670d3a88fbe0aeb301ced12f7bcb4c8e3aa877b20f8559c02
Address: 27f:b2c5:a944:4dad:9cb1:da4:8bf7:7e65
```

View File

@@ -1,48 +0,0 @@
<h1>Installation</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Full VM Example](#full-vm-example)
***
## Introduction
In this section, we cover how to install Mycelium. For this guide, we will show the steps on a full VM running on the TFGrid.
Currently, Linux, macOS and Windows are supported. On Windows, you must have `wintun.dll` in the same directory you are executing the binary from.
## Full VM Example
- Deploy a Full VM with Planetary network and SSH into the VM
- Update the system
```
apt update
```
- Download the latest Mycelium release: [https://github.com/threefoldtech/mycelium/releases/latest](https://github.com/threefoldtech/mycelium/releases/latest)
```
wget https://github.com/threefoldtech/mycelium/releases/download/v0.4.0/mycelium-x86_64-unknown-linux-musl.tar.gz
```
- Extract Mycelium
```
tar -xvf mycelium-x86_64-unknown-linux-musl.tar.gz
```
- Move Mycelium to your path
```
mv mycelium /usr/local/bin
```
- Start Mycelium
```
mycelium --peers tcp://83.231.240.31:9651 quic://185.206.122.71:9651 --tun-name utun2
```
- Open another terminal
- Check the Mycelium connection information (address: ...)
```
mycelium inspect --json
```
- Ping the VM from another machine with IPv6
```
ping6 mycelium_address
```

View File

@@ -1,90 +0,0 @@
<h1> Message Subsystem</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Curl Examples](#curl-examples)
- [Mycelium Binary Examples](#mycelium-binary-examples)
***
## Introduction
The message subsystem can be used to send arbitrary length messages to receivers. A receiver is any
other node in the network. It can be identified both by its public key, or an IP address in its announced
range. The message subsystem can be interacted with both via the HTTP API, which is
[documented here](./api_yaml.md), or via the `mycelium` binary. By default, the messages do not interpret
the data in any way. When using the binary, the message is slightly modified to include an optional
topic at the start of the message. Note that in the HTTP API, all messages are encoded in base64. This
might make it difficult to consume these messages without additional tooling.
## Curl Examples
These examples assume you have at least 2 nodes running, and that they are both part of the same network.
Send a message on node1, waiting up to 2 minutes for a possible reply:
```bash
curl -v -H 'Content-Type: application/json' -d '{"dst": {"pk": "bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32"}, "payload": "xuV+"}' http://localhost:8989/api/v1/messages\?reply_timeout\=120
```
Listen for a message on node2. Note that messages received while nothing is listening are added to
a queue for later consumption. Wait for up to 1 minute.
```bash
curl -v http://localhost:8989/api/v1/messages\?timeout\=60\
```
The system will (immediately) receive our previously sent message:
```json
{"id":"e47b25063912f4a9","srcIp":"34f:b680:ba6e:7ced:355f:346f:d97b:eecb","srcPk":"955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b","dstIp":"2e4:9ace:9252:630:beee:e405:74c0:d876","dstPk":"bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32","payload":"xuV+"}
```
To send a reply, we can post a message on the reply path, with the received message `id` (still on
node2):
```bash
curl -H 'Content-Type: application/json' -d '{"dst": {"pk":"955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b"}, "payload": "xuC+"}' http://localhost:8989/api/v1/messages/reply/e47b25063912f4a9
```
If you did this fast enough, the initial sender (node1) will now receive the reply.
## Mycelium Binary Examples
As explained above, while using the binary the message is slightly modified to insert the optional
topic. As such, when using the binary to send messages, it is suggested to make sure the receiver is
also using the binary to listen for messages. The options discussed here are not covering all possibilities,
use the `--help` flag (`mycelium message send --help` and `mycelium message receive --help`) for a
full overview.
Once again, send a message. This time using a topic (example.topic). Note that there are no constraints
on what a valid topic is, other than that it is valid UTF-8, and at most 255 bytes in size. The `--wait`
flag can be used to indicate that we are waiting for a reply. If it is set, we can also use an additional
`--timeout` flag to govern exactly how long (in seconds) to wait for. The default is to wait forever.
```bash
mycelium message send 2e4:9ace:9252:630:beee:e405:74c0:d876 'this is a message' -t example.topic --wait
```
On the second node, listen for messages with this topic. If a different topic is used, the previous
message won't be received. If no topic is set, all messages are received. An optional timeout flag
can be specified, which indicates how long to wait for. Absence of this flag will cause the binary
to wait forever.
```bash
mycelium message receive -t example.topic
```
Again, if the previous command was executed a message will be received immediately:
```json
{"id":"4a6c956e8d36381f","topic":"example.topic","srcIp":"34f:b680:ba6e:7ced:355f:346f:d97b:eecb","srcPk":"955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b","dstIp":"2e4:9ace:9252:630:beee:e405:74c0:d876","dstPk":"bb39b4a3a4efd70f3e05e37887677e02efbda14681d0acd3882bc0f754792c32","payload":"this is a message"}
```
And once again, we can use the ID from this message to reply to the original sender, who might be waiting
for this reply (notice we used the hex encoded public key to identify the receiver here, rather than an IP):
```bash
mycelium message send 955bf6bea5e1150fd8e270c12e5b2fc08f08f7c5f3799d10550096cc137d671b "this is a reply" --reply-to 4a6c956e8d36381f
```

View File

@@ -1,14 +0,0 @@
<h1>Mycelium</h1>
In this section, we present [Mycelium](https://github.com/threefoldtech/mycelium), an end-to-end encrypted IPv6 overlay network.
<h2>Table of Contents</h2>
- [Overview](./overview.md)
- [Installation](./installation.md)
- [Additional Information](./information.md)
- [Message](./message.md)
- [Packet](./packet.md)
- [Data Packet](./data_packet.md)
- [API YAML](./api_yaml.md)

View File

@@ -1,33 +0,0 @@
<h1>Overview</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Features](#features)
- [Testing](#testing)
***
## Introduction
Mycelium is an end-2-end encrypted IPv6 overlay network written in Rust where each node that joins the overlay network will receive an overlay network IP in the 400::/7 range.
The overlay network uses some of the core principles of the [Babel routing protocol](https://www.irif.fr/~jch/software/babel).
## Features
- Mycelium, is locality aware, it will look for the shortest path between nodes
- All traffic between the nodes is end-2-end encrypted
- Traffic can be routed over nodes of friends, location aware
- If a physical link goes down Mycelium will automatically reroute your traffic
- The IP address is IPV6 and linked to private key
- A simple reliable messagebus is implemented on top of Mycelium
- Mycelium has multiple ways how to communicate quic, tcp, ... and we are working on holepunching for Quick which means P2P traffic without middlemen for NATted networks e.g. most homes
- Scalability is very important for us, we tried many overlay networks before and got stuck on all of them, we are trying to design a network which scales to a planetary level
- You can run mycelium without TUN and only use it as reliable message bus.
## Testing
We are looking for lots of testers to push the system. Visit the [Mycelium repository](https://github.com/threefoldtech/mycelium) to contribute.

View File

@@ -1,33 +0,0 @@
<h1>Packet</h1>
<h2>Table of Contents</h2>
- [Introduction](#introduction)
- [Packet Header](#packet-header)
***
## Introduction
A `Packet` is the largest communication object between established `peers`. All communication is done
via these `packets`. The `packet` itself consists of a fixed size header, and a variable size body.
The body contains a more specific type of data.
## Packet Header
The packet header has a fixed size of 4 bytes, with the following layout:
```
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version | Type | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```
The first byte is used to indicate the version of the protocol. Currently, only version 1 is supported
(0x01). The next byte is used to indicate the type of the body. `0x00` indicates a data packet, while
`0x01` indicates a control packet. The remaining 16 bits are currently reserved, and should be set to
all 0.