Mycelium CNI Plugin

A Container Network Interface (CNI) plugin that enables Kubernetes containers to connect to the Mycelium network.

Overview

This CNI plugin integrates with the Mycelium overlay network to provide IPv6 connectivity for Kubernetes containers. It creates veth pairs and assigns IPv6 addresses from the host's Mycelium /64 block to containers.

Prerequisites

• Mycelium daemon running on the host
• Go 1.21+
• Standard CNI plugins (for loopback and other basic functionality)
• Root privileges for installation

Installation

# 1. Install standard CNI plugins (required for loopback)
CNI_VERSION="v1.3.0"
wget https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
sudo mkdir -p /opt/cni/bin
sudo tar -xzf cni-plugins-linux-amd64-${CNI_VERSION}.tgz -C /opt/cni/bin/

# 2. Download dependencies and build the plugin
make build

# 3. Install plugin and configuration
make install

Note

: The first build will download Go dependencies automatically via go mod tidy.

Configuration

The plugin uses a CNI configuration file (10-mycelium.conflist) that includes the loopback plugin and specifies the Mycelium interface name:

{
  "cniVersion": "1.0.0",
  "name": "mycelium-network",
  "plugins": [
    {
      "type": "loopback"
    },
    {
      "type": "mycelium-cni",
      "myceliumInterface": "mycelium"
    }
  ]
}

How it Works

1. ADD Operation: Creates a veth pair, moves one end to the container namespace, assigns an IPv6 address from the Mycelium prefix, and sets up routing.

2. DEL Operation: Cleans up the host-side veth interface when containers are destroyed.

Usage with Kubernetes

For k3s

k3s requires special setup since it uses Flannel CNI by default:

# 1. Install k3s without default CNI
curl -sfL https://get.k3s.io | sh -s - --flannel-backend=none --disable-network-policy

# OR modify existing k3s installation
sudo systemctl edit k3s
# Add these lines:
# [Service]
# ExecStart=
# ExecStart=/usr/local/bin/k3s server --flannel-backend=none --disable-network-policy

# 2. Install CNI plugins and Mycelium CNI plugin (follow installation steps above)

# 3. Copy CNI config to k3s location
sudo cp /etc/cni/net.d/10-mycelium.conflist /var/lib/rancher/k3s/agent/etc/cni/net.d/

# 4. Restart k3s
sudo systemctl restart k3s

For standard Kubernetes

1. Setup Mycelium on all nodes

First, install Mycelium on all Kubernetes nodes:

# Download and install Mycelium
MYCELIUM_VERSION="v0.5.6"
wget https://github.com/threefoldtech/mycelium/releases/download/${MYCELIUM_VERSION}/mycelium-x86_64-unknown-linux-musl.tar.gz
tar xf mycelium-x86_64-unknown-linux-musl.tar.gz
sudo cp mycelium /usr/local/bin/

# Create systemd service
sudo tee /etc/systemd/system/mycelium.service > /dev/null <<EOF
[Unit]
Description=Mycelium Network
After=network.target

[Service]
ExecStart=/usr/local/bin/mycelium --peers tcp://188.40.132.242:9651 tcp://136.243.47.186:9651 tcp://185.69.166.7:9651 tcp://65.21.231.58:9651 tcp://209.159.146.190:9651
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Enable IPv6 forwarding
echo 'net.ipv6.conf.all.forwarding=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Start Mycelium
sudo systemctl daemon-reload
sudo systemctl enable --now mycelium

2. Install the CNI plugin

On each Kubernetes node:

# Install standard CNI plugins first (if not already done)
CNI_VERSION="v1.3.0"
wget https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
sudo mkdir -p /opt/cni/bin
sudo tar -xzf cni-plugins-linux-amd64-${CNI_VERSION}.tgz -C /opt/cni/bin/

# Download dependencies, build and install the plugin
make build
sudo make install

# Verify installation
ls -la /opt/cni/bin/mycelium-cni
ls -la /opt/cni/bin/loopback
ls -la /etc/cni/net.d/10-mycelium.conflist

3. Configure Kubernetes

For kubeadm clusters, the CNI configuration is automatically picked up. For other setups, ensure the kubelet is configured to use CNI:

# Add to kubelet configuration
--network-plugin=cni
--cni-conf-dir=/etc/cni/net.d
--cni-bin-dir=/opt/cni/bin

4. Test with a pod

Create a test pod to verify Mycelium connectivity:

# test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mycelium-test
spec:
  containers:
  - name: test
    image: ubuntu:22.04
    command: ["/bin/sleep", "3600"]

Deploy and test:

kubectl apply -f test-pod.yaml

# Check pod IP (should be IPv6 from Mycelium range)
kubectl get pod mycelium-test -o wide

# Test connectivity from inside the pod
kubectl exec -it mycelium-test -- ip -6 addr show
kubectl exec -it mycelium-test -- ping6 -c 3 400::1  # Test Mycelium network access

5. Multi-node connectivity

Test communication between pods on different nodes:

# multi-node-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mycelium-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mycelium-test
  template:
    metadata:
      labels:
        app: mycelium-test
    spec:
      containers:
      - name: test
        image: ubuntu:22.04
        command: ["/bin/sleep", "3600"]

kubectl apply -f multi-node-test.yaml

# Get pod IPs and test inter-pod communication
kubectl get pods -o wide
kubectl exec -it <pod1> -- ping6 -c 3 <pod2-ipv6>

Testing

Manual CNI Testing

You can test the plugin directly using CNI testing tools:

# Install CNI plugins for testing
go install github.com/containernetworking/cni/cnitool@latest

# Set environment
export CNI_PATH=/opt/cni/bin
export NETCONFPATH=/etc/cni/net.d

# Test ADD operation
echo '{}' | sudo cnitool add mycelium-network /var/run/netns/test

# Test DEL operation  
echo '{}' | sudo cnitool del mycelium-network /var/run/netns/test

Troubleshooting

Check common issues:

# Verify Mycelium is running
sudo systemctl status mycelium
ip -6 addr show mycelium

# Check CNI logs (kubelet for standard k8s, k3s for k3s)
journalctl -u kubelet | grep -i cni
# OR for k3s:
journalctl -u k3s | grep -i cni

# Verify CNI plugins are installed
ls -la /opt/cni/bin/loopback
ls -la /opt/cni/bin/mycelium-cni

# Check CNI configuration location
ls -la /etc/cni/net.d/10-mycelium.conflist
# OR for k3s:
ls -la /var/lib/rancher/k3s/agent/etc/cni/net.d/10-mycelium.conflist

# Verify network namespaces
sudo ip netns list

# Check container interfaces
kubectl exec -it <pod> -- ip link show
kubectl exec -it <pod> -- ip -6 route show

Common errors and solutions:

• failed to find plugin "loopback": Install standard CNI plugins (see installation section)
• failed to find interface mycelium: Mycelium daemon not running or interface not created
• no global IPv6 address found: Mycelium not connected to network peers

Architecture

Based on the docker-demo.sh script, this plugin:

• Uses IPv6 addressing from Mycelium's /64 block
• Creates veth pairs for container connectivity
• Sets up routing for Mycelium network (400::/7)
• Enables IPv6 forwarding on the host