# Mycelium CNI Plugin

A Container Network Interface (CNI) plugin that enables Kubernetes containers to connect to the Mycelium network.

## Overview

This CNI plugin integrates with the Mycelium overlay network to provide IPv6 connectivity for Kubernetes containers. It creates veth pairs and assigns IPv6 addresses from the host's Mycelium /64 block to containers.

## Prerequisites

- Mycelium daemon running on the host
- Go 1.21+
- Standard CNI plugins (for loopback and other basic functionality)
- Root privileges for installation

## Installation

```bash
# 1. Install standard CNI plugins (required for loopback)
CNI_VERSION="v1.3.0"
wget https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
sudo mkdir -p /opt/cni/bin
sudo tar -xzf cni-plugins-linux-amd64-${CNI_VERSION}.tgz -C /opt/cni/bin/

# 2. Download dependencies and build the plugin
make build

# 3. Install plugin and configuration
make install
```

> **Note**: The first build will download Go dependencies automatically via `go mod tidy`.

## Configuration

The plugin uses a CNI configuration file (`10-mycelium.conflist`) that includes the loopback plugin and specifies the Mycelium interface name:

```json
{
  "cniVersion": "1.0.0",
  "name": "mycelium-network",
  "plugins": [
    {
      "type": "loopback"
    },
    {
      "type": "mycelium-cni",
      "myceliumInterface": "mycelium"
    }
  ]
}
```

## How it Works

1. **ADD Operation**: Creates a veth pair, moves one end to the container namespace, assigns an IPv6 address from the Mycelium prefix, and sets up routing.

2. **DEL Operation**: Cleans up the host-side veth interface when containers are destroyed.

## Usage with Kubernetes

### For k3s

k3s requires special setup since it uses Flannel CNI by default:

```bash
# 1. Install k3s without default CNI
curl -sfL https://get.k3s.io | sh -s - --flannel-backend=none --disable-network-policy

# OR modify existing k3s installation
sudo systemctl edit k3s
# Add these lines:
# [Service]
# ExecStart=
# ExecStart=/usr/local/bin/k3s server --flannel-backend=none --disable-network-policy

# 2. Install CNI plugins and Mycelium CNI plugin (follow installation steps above)

# 3. Copy CNI config to k3s location
sudo cp /etc/cni/net.d/10-mycelium.conflist /var/lib/rancher/k3s/agent/etc/cni/net.d/

# 4. Restart k3s
sudo systemctl restart k3s
```

### For standard Kubernetes

### 1. Setup Mycelium on all nodes

First, install Mycelium on all Kubernetes nodes:

```bash
# Download and install Mycelium
MYCELIUM_VERSION="v0.5.6"
wget https://github.com/threefoldtech/mycelium/releases/download/${MYCELIUM_VERSION}/mycelium-x86_64-unknown-linux-musl.tar.gz
tar xf mycelium-x86_64-unknown-linux-musl.tar.gz
sudo cp mycelium /usr/local/bin/

# Create systemd service
sudo tee /etc/systemd/system/mycelium.service > /dev/null <<EOF
[Unit]
Description=Mycelium Network
After=network.target

[Service]
ExecStart=/usr/local/bin/mycelium --peers tcp://188.40.132.242:9651 tcp://136.243.47.186:9651 tcp://185.69.166.7:9651 tcp://65.21.231.58:9651 tcp://209.159.146.190:9651
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Enable IPv6 forwarding
echo 'net.ipv6.conf.all.forwarding=1' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Start Mycelium
sudo systemctl daemon-reload
sudo systemctl enable --now mycelium
```

### 2. Install the CNI plugin

On each Kubernetes node:

```bash
# Install standard CNI plugins first (if not already done)
CNI_VERSION="v1.3.0"
wget https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
sudo mkdir -p /opt/cni/bin
sudo tar -xzf cni-plugins-linux-amd64-${CNI_VERSION}.tgz -C /opt/cni/bin/

# Download dependencies, build and install the plugin
make build
sudo make install

# Verify installation
ls -la /opt/cni/bin/mycelium-cni
ls -la /opt/cni/bin/loopback
ls -la /etc/cni/net.d/10-mycelium.conflist
```

### 3. Configure Kubernetes

For **kubeadm** clusters, the CNI configuration is automatically picked up. For other setups, ensure the kubelet is configured to use CNI:

```bash
# Add to kubelet configuration
--network-plugin=cni
--cni-conf-dir=/etc/cni/net.d
--cni-bin-dir=/opt/cni/bin
```

### 4. Test with a pod

Create a test pod to verify Mycelium connectivity:

```yaml
# test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mycelium-test
spec:
  containers:
  - name: test
    image: ubuntu:22.04
    command: ["/bin/sleep", "3600"]
```

Deploy and test:

```bash
kubectl apply -f test-pod.yaml

# Check pod IP (should be IPv6 from Mycelium range)
kubectl get pod mycelium-test -o wide

# Test connectivity from inside the pod
kubectl exec -it mycelium-test -- ip -6 addr show
kubectl exec -it mycelium-test -- ping6 -c 3 400::1  # Test Mycelium network access
```

### 5. Multi-node connectivity

Test communication between pods on different nodes:

```yaml
# multi-node-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mycelium-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mycelium-test
  template:
    metadata:
      labels:
        app: mycelium-test
    spec:
      containers:
      - name: test
        image: ubuntu:22.04
        command: ["/bin/sleep", "3600"]
```

```bash
kubectl apply -f multi-node-test.yaml

# Get pod IPs and test inter-pod communication
kubectl get pods -o wide
kubectl exec -it <pod1> -- ping6 -c 3 <pod2-ipv6>
```

## Testing

### Manual CNI Testing

You can test the plugin directly using CNI testing tools:

```bash
# Install CNI plugins for testing
go install github.com/containernetworking/cni/cnitool@latest

# Set environment
export CNI_PATH=/opt/cni/bin
export NETCONFPATH=/etc/cni/net.d

# Test ADD operation
echo '{}' | sudo cnitool add mycelium-network /var/run/netns/test

# Test DEL operation  
echo '{}' | sudo cnitool del mycelium-network /var/run/netns/test
```

### Troubleshooting

Check common issues:

```bash
# Verify Mycelium is running
sudo systemctl status mycelium
ip -6 addr show mycelium

# Check CNI logs (kubelet for standard k8s, k3s for k3s)
journalctl -u kubelet | grep -i cni
# OR for k3s:
journalctl -u k3s | grep -i cni

# Verify CNI plugins are installed
ls -la /opt/cni/bin/loopback
ls -la /opt/cni/bin/mycelium-cni

# Check CNI configuration location
ls -la /etc/cni/net.d/10-mycelium.conflist
# OR for k3s:
ls -la /var/lib/rancher/k3s/agent/etc/cni/net.d/10-mycelium.conflist

# Verify network namespaces
sudo ip netns list

# Check container interfaces
kubectl exec -it <pod> -- ip link show
kubectl exec -it <pod> -- ip -6 route show
```

**Common errors and solutions:**
- `failed to find plugin "loopback"`: Install standard CNI plugins (see installation section)
- `failed to find interface mycelium`: Mycelium daemon not running or interface not created
- `no global IPv6 address found`: Mycelium not connected to network peers

## Architecture

Based on the docker-demo.sh script, this plugin:
- Uses IPv6 addressing from Mycelium's /64 block
- Creates veth pairs for container connectivity  
- Sets up routing for Mycelium network (400::/7)
- Enables IPv6 forwarding on the host