Mycelium CNI Plugin

A Container Network Interface (CNI) plugin that enables Kubernetes containers to connect to the Mycelium network.

Overview

This CNI plugin integrates with the Mycelium overlay network to provide IPv6 connectivity for Kubernetes containers. It creates veth pairs and assigns IPv6 addresses from the host's Mycelium /64 block to containers.

Prerequisites

• Mycelium daemon running on the host
• Go 1.21+
• Root privileges for installation

Installation

# Build the plugin
make build

# Install plugin and configuration
make install

Configuration

The plugin uses a CNI configuration file (10-mycelium.conflist) that specifies the Mycelium interface name:

{
  "cniVersion": "1.0.0",
  "name": "mycelium-network",
  "plugins": [
    {
      "type": "mycelium-cni",
      "myceliumInterface": "mycelium"
    }
  ]
}

How it Works

1. ADD Operation: Creates a veth pair, moves one end to the container namespace, assigns an IPv6 address from the Mycelium prefix, and sets up routing.

2. DEL Operation: Cleans up the host-side veth interface when containers are destroyed.

Usage with Kubernetes

1. Setup Mycelium on all nodes

First, install Mycelium on all Kubernetes nodes:

# Download and install Mycelium
MYCELIUM_VERSION="v0.5.6"
wget https://github.com/threefoldtech/mycelium/releases/download/${MYCELIUM_VERSION}/mycelium-x86_64-unknown-linux-musl.tar.gz
tar xf mycelium-x86_64-unknown-linux-musl.tar.gz
sudo cp mycelium /usr/local/bin/

# Create systemd service
sudo tee /etc/systemd/system/mycelium.service > /dev/null <<EOF
[Unit]
Description=Mycelium Network
After=network.target

[Service]
ExecStart=/usr/local/bin/mycelium --peers tcp://188.40.132.242:9651 tcp://136.243.47.186:9651 tcp://185.69.166.7:9651 tcp://65.21.231.58:9651 tcp://209.159.146.190:9651
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Enable IPv6 forwarding
echo 'net.ipv6.conf.all.forwarding=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Start Mycelium
sudo systemctl daemon-reload
sudo systemctl enable --now mycelium

2. Install the CNI plugin

On each Kubernetes node:

# Build and install the plugin
make build
sudo make install

# Verify installation
ls -la /opt/cni/bin/mycelium-cni
ls -la /etc/cni/net.d/10-mycelium.conflist

3. Configure Kubernetes

For kubeadm clusters, the CNI configuration is automatically picked up. For other setups, ensure the kubelet is configured to use CNI:

# Add to kubelet configuration
--network-plugin=cni
--cni-conf-dir=/etc/cni/net.d
--cni-bin-dir=/opt/cni/bin

4. Test with a pod

Create a test pod to verify Mycelium connectivity:

# test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mycelium-test
spec:
  containers:
  - name: test
    image: ubuntu:22.04
    command: ["/bin/sleep", "3600"]

Deploy and test:

kubectl apply -f test-pod.yaml

# Check pod IP (should be IPv6 from Mycelium range)
kubectl get pod mycelium-test -o wide

# Test connectivity from inside the pod
kubectl exec -it mycelium-test -- ip -6 addr show
kubectl exec -it mycelium-test -- ping6 -c 3 400::1  # Test Mycelium network access

5. Multi-node connectivity

Test communication between pods on different nodes:

# multi-node-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mycelium-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mycelium-test
  template:
    metadata:
      labels:
        app: mycelium-test
    spec:
      containers:
      - name: test
        image: ubuntu:22.04
        command: ["/bin/sleep", "3600"]

kubectl apply -f multi-node-test.yaml

# Get pod IPs and test inter-pod communication
kubectl get pods -o wide
kubectl exec -it <pod1> -- ping6 -c 3 <pod2-ipv6>

Testing

Manual CNI Testing

You can test the plugin directly using CNI testing tools:

# Install CNI plugins for testing
go install github.com/containernetworking/cni/cnitool@latest

# Set environment
export CNI_PATH=/opt/cni/bin
export NETCONFPATH=/etc/cni/net.d

# Test ADD operation
echo '{}' | sudo cnitool add mycelium-network /var/run/netns/test

# Test DEL operation  
echo '{}' | sudo cnitool del mycelium-network /var/run/netns/test

Troubleshooting

Check common issues:

# Verify Mycelium is running
sudo systemctl status mycelium
ip -6 addr show mycelium

# Check CNI logs
journalctl -u kubelet | grep -i cni

# Verify network namespaces
sudo ip netns list

# Check container interfaces
kubectl exec -it <pod> -- ip link show
kubectl exec -it <pod> -- ip -6 route show

Architecture

Based on the docker-demo.sh script, this plugin:

• Uses IPv6 addressing from Mycelium's /64 block
• Creates veth pairs for container connectivity
• Sets up routing for Mycelium network (400::/7)
• Enables IPv6 forwarding on the host