282 lines
7.0 KiB
Markdown
282 lines
7.0 KiB
Markdown
# Mycelium CNI Plugin
|
|
|
|
A Container Network Interface (CNI) plugin that enables Kubernetes containers to connect to the Mycelium network.
|
|
|
|
## Overview
|
|
|
|
This CNI plugin integrates with the Mycelium overlay network to provide IPv6 connectivity for Kubernetes containers. It creates veth pairs and assigns IPv6 addresses from the host's Mycelium /64 block to containers.
|
|
|
|
## Prerequisites
|
|
|
|
- Mycelium daemon running on the host
|
|
- Go 1.21+
|
|
- Standard CNI plugins (for loopback and other basic functionality)
|
|
- Root privileges for installation
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
# 1. Install standard CNI plugins (required for loopback)
|
|
CNI_VERSION="v1.3.0"
|
|
wget https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
|
|
sudo mkdir -p /opt/cni/bin
|
|
sudo tar -xzf cni-plugins-linux-amd64-${CNI_VERSION}.tgz -C /opt/cni/bin/
|
|
|
|
# 2. Download dependencies and build the plugin
|
|
make build
|
|
|
|
# 3. Install plugin and configuration
|
|
make install
|
|
```
|
|
|
|
> **Note**: The first build will download Go dependencies automatically via `go mod tidy`.
|
|
|
|
## Configuration
|
|
|
|
The plugin uses a CNI configuration file (`10-mycelium.conflist`) that includes the loopback plugin and specifies the Mycelium interface name:
|
|
|
|
```json
|
|
{
|
|
"cniVersion": "1.0.0",
|
|
"name": "mycelium-network",
|
|
"plugins": [
|
|
{
|
|
"type": "loopback"
|
|
},
|
|
{
|
|
"type": "mycelium-cni",
|
|
"myceliumInterface": "mycelium"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## How it Works
|
|
|
|
1. **ADD Operation**: Creates a veth pair, moves one end to the container namespace, assigns an IPv6 address from the Mycelium prefix, and sets up routing.
|
|
|
|
2. **DEL Operation**: Cleans up the host-side veth interface when containers are destroyed.
|
|
|
|
## Usage with Kubernetes
|
|
|
|
### For k3s
|
|
|
|
k3s requires special setup since it uses Flannel CNI by default:
|
|
|
|
```bash
|
|
# 1. Install k3s without default CNI
|
|
curl -sfL https://get.k3s.io | sh -s - --flannel-backend=none --disable-network-policy
|
|
|
|
# OR modify existing k3s installation
|
|
sudo systemctl edit k3s
|
|
# Add these lines:
|
|
# [Service]
|
|
# ExecStart=
|
|
# ExecStart=/usr/local/bin/k3s server --flannel-backend=none --disable-network-policy
|
|
|
|
# 2. Install CNI plugins and Mycelium CNI plugin (follow installation steps above)
|
|
|
|
# 3. Copy CNI config to k3s location
|
|
sudo cp /etc/cni/net.d/10-mycelium.conflist /var/lib/rancher/k3s/agent/etc/cni/net.d/
|
|
|
|
# 4. Restart k3s
|
|
sudo systemctl restart k3s
|
|
```
|
|
|
|
### For standard Kubernetes
|
|
|
|
### 1. Setup Mycelium on all nodes
|
|
|
|
First, install Mycelium on all Kubernetes nodes:
|
|
|
|
```bash
|
|
# Download and install Mycelium
|
|
MYCELIUM_VERSION="v0.5.6"
|
|
wget https://github.com/threefoldtech/mycelium/releases/download/${MYCELIUM_VERSION}/mycelium-x86_64-unknown-linux-musl.tar.gz
|
|
tar xf mycelium-x86_64-unknown-linux-musl.tar.gz
|
|
sudo cp mycelium /usr/local/bin/
|
|
|
|
# Create systemd service
|
|
sudo tee /etc/systemd/system/mycelium.service > /dev/null <<EOF
|
|
[Unit]
|
|
Description=Mycelium Network
|
|
After=network.target
|
|
|
|
[Service]
|
|
ExecStart=/usr/local/bin/mycelium --peers tcp://188.40.132.242:9651 tcp://136.243.47.186:9651 tcp://185.69.166.7:9651 tcp://65.21.231.58:9651 tcp://209.159.146.190:9651
|
|
Restart=always
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
EOF
|
|
|
|
# Enable IPv6 forwarding
|
|
echo 'net.ipv6.conf.all.forwarding=1' | sudo tee -a /etc/sysctl.conf
|
|
sudo sysctl -p
|
|
|
|
# Start Mycelium
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now mycelium
|
|
```
|
|
|
|
### 2. Install the CNI plugin
|
|
|
|
On each Kubernetes node:
|
|
|
|
```bash
|
|
# Install standard CNI plugins first (if not already done)
|
|
CNI_VERSION="v1.3.0"
|
|
wget https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
|
|
sudo mkdir -p /opt/cni/bin
|
|
sudo tar -xzf cni-plugins-linux-amd64-${CNI_VERSION}.tgz -C /opt/cni/bin/
|
|
|
|
# Download dependencies, build and install the plugin
|
|
make build
|
|
sudo make install
|
|
|
|
# Verify installation
|
|
ls -la /opt/cni/bin/mycelium-cni
|
|
ls -la /opt/cni/bin/loopback
|
|
ls -la /etc/cni/net.d/10-mycelium.conflist
|
|
```
|
|
|
|
### 3. Configure Kubernetes
|
|
|
|
For **kubeadm** clusters, the CNI configuration is automatically picked up. For other setups, ensure the kubelet is configured to use CNI:
|
|
|
|
```bash
|
|
# Add to kubelet configuration
|
|
--network-plugin=cni
|
|
--cni-conf-dir=/etc/cni/net.d
|
|
--cni-bin-dir=/opt/cni/bin
|
|
```
|
|
|
|
### 4. Test with a pod
|
|
|
|
Create a test pod to verify Mycelium connectivity:
|
|
|
|
```yaml
|
|
# test-pod.yaml
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: mycelium-test
|
|
spec:
|
|
containers:
|
|
- name: test
|
|
image: ubuntu:22.04
|
|
command: ["/bin/sleep", "3600"]
|
|
```
|
|
|
|
Deploy and test:
|
|
|
|
```bash
|
|
kubectl apply -f test-pod.yaml
|
|
|
|
# Check pod IP (should be IPv6 from Mycelium range)
|
|
kubectl get pod mycelium-test -o wide
|
|
|
|
# Test connectivity from inside the pod
|
|
kubectl exec -it mycelium-test -- ip -6 addr show
|
|
kubectl exec -it mycelium-test -- ping6 -c 3 400::1 # Test Mycelium network access
|
|
```
|
|
|
|
### 5. Multi-node connectivity
|
|
|
|
Test communication between pods on different nodes:
|
|
|
|
```yaml
|
|
# multi-node-test.yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: mycelium-deployment
|
|
spec:
|
|
replicas: 3
|
|
selector:
|
|
matchLabels:
|
|
app: mycelium-test
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: mycelium-test
|
|
spec:
|
|
containers:
|
|
- name: test
|
|
image: ubuntu:22.04
|
|
command: ["/bin/sleep", "3600"]
|
|
```
|
|
|
|
```bash
|
|
kubectl apply -f multi-node-test.yaml
|
|
|
|
# Get pod IPs and test inter-pod communication
|
|
kubectl get pods -o wide
|
|
kubectl exec -it <pod1> -- ping6 -c 3 <pod2-ipv6>
|
|
```
|
|
|
|
## Testing
|
|
|
|
### Manual CNI Testing
|
|
|
|
You can test the plugin directly using CNI testing tools:
|
|
|
|
```bash
|
|
# Install CNI plugins for testing
|
|
go install github.com/containernetworking/cni/cnitool@latest
|
|
|
|
# Set environment
|
|
export CNI_PATH=/opt/cni/bin
|
|
export NETCONFPATH=/etc/cni/net.d
|
|
|
|
# Test ADD operation
|
|
echo '{}' | sudo cnitool add mycelium-network /var/run/netns/test
|
|
|
|
# Test DEL operation
|
|
echo '{}' | sudo cnitool del mycelium-network /var/run/netns/test
|
|
```
|
|
|
|
### Troubleshooting
|
|
|
|
Check common issues:
|
|
|
|
```bash
|
|
# Verify Mycelium is running
|
|
sudo systemctl status mycelium
|
|
ip -6 addr show mycelium
|
|
|
|
# Check CNI logs (kubelet for standard k8s, k3s for k3s)
|
|
journalctl -u kubelet | grep -i cni
|
|
# OR for k3s:
|
|
journalctl -u k3s | grep -i cni
|
|
|
|
# Verify CNI plugins are installed
|
|
ls -la /opt/cni/bin/loopback
|
|
ls -la /opt/cni/bin/mycelium-cni
|
|
|
|
# Check CNI configuration location
|
|
ls -la /etc/cni/net.d/10-mycelium.conflist
|
|
# OR for k3s:
|
|
ls -la /var/lib/rancher/k3s/agent/etc/cni/net.d/10-mycelium.conflist
|
|
|
|
# Verify network namespaces
|
|
sudo ip netns list
|
|
|
|
# Check container interfaces
|
|
kubectl exec -it <pod> -- ip link show
|
|
kubectl exec -it <pod> -- ip -6 route show
|
|
```
|
|
|
|
**Common errors and solutions:**
|
|
- `failed to find plugin "loopback"`: Install standard CNI plugins (see installation section)
|
|
- `failed to find interface mycelium`: Mycelium daemon not running or interface not created
|
|
- `no global IPv6 address found`: Mycelium not connected to network peers
|
|
|
|
## Architecture
|
|
|
|
Based on the docker-demo.sh script, this plugin:
|
|
- Uses IPv6 addressing from Mycelium's /64 block
|
|
- Creates veth pairs for container connectivity
|
|
- Sets up routing for Mycelium network (400::/7)
|
|
- Enables IPv6 forwarding on the host
|