b9b8e7ab75
Implemented a production-ready K3s Kubernetes installer with full lifecycle support including installation, startup management, and cleanup. Key features: - Install first master (cluster init), join additional masters (HA), and workers - Systemd service management via StartupManager abstraction - IPv6 support with Mycelium interface auto-detection - Robust destroy/cleanup with proper ordering to prevent hanging - Complete removal of services, processes, network interfaces, and data
225 lines
6.0 KiB
Markdown
225 lines
6.0 KiB
Markdown
# K3s Installer
|
|
|
|
Complete K3s cluster installer with multi-master HA support, worker nodes, and Mycelium IPv6 networking.
|
|
|
|
## Features
|
|
|
|
- **Multi-Master HA**: Install multiple master nodes with `--cluster-init`
|
|
- **Worker Nodes**: Add worker nodes to the cluster
|
|
- **Mycelium IPv6**: Automatic detection of Mycelium IPv6 addresses from the 400::/7 range
|
|
- **Lifecycle Management**: Start, stop, restart K3s via startupmanager (systemd/zinit/screen)
|
|
- **Join Scripts**: Auto-generate heroscripts for joining additional nodes
|
|
- **Complete Cleanup**: Destroy removes all K3s components, network interfaces, and data
|
|
|
|
## Quick Start
|
|
|
|
### Install First Master
|
|
|
|
```v
|
|
import incubaid.herolib.installers.virt.kubernetes_installer
|
|
|
|
heroscript := "
|
|
!!kubernetes_installer.configure
|
|
name:'k3s_master_1'
|
|
k3s_version:'v1.33.1'
|
|
node_name:'master-1'
|
|
mycelium_interface:'mycelium0'
|
|
|
|
!!kubernetes_installer.install_master name:'k3s_master_1'
|
|
!!kubernetes_installer.start name:'k3s_master_1'
|
|
"
|
|
|
|
kubernetes_installer.play(heroscript: heroscript)!
|
|
```
|
|
|
|
### Join Additional Master (HA)
|
|
|
|
```v
|
|
heroscript := "
|
|
!!kubernetes_installer.configure
|
|
name:'k3s_master_2'
|
|
node_name:'master-2'
|
|
token:'<TOKEN_FROM_FIRST_MASTER>'
|
|
master_url:'https://[<MASTER_IPV6>]:6443'
|
|
|
|
!!kubernetes_installer.join_master name:'k3s_master_2'
|
|
!!kubernetes_installer.start name:'k3s_master_2'
|
|
"
|
|
|
|
kubernetes_installer.play(heroscript: heroscript)!
|
|
```
|
|
|
|
### Install Worker Node
|
|
|
|
```v
|
|
heroscript := "
|
|
!!kubernetes_installer.configure
|
|
name:'k3s_worker_1'
|
|
node_name:'worker-1'
|
|
token:'<TOKEN_FROM_FIRST_MASTER>'
|
|
master_url:'https://[<MASTER_IPV6>]:6443'
|
|
|
|
!!kubernetes_installer.install_worker name:'k3s_worker_1'
|
|
!!kubernetes_installer.start name:'k3s_worker_1'
|
|
"
|
|
|
|
kubernetes_installer.play(heroscript: heroscript)!
|
|
```
|
|
|
|
## Configuration Options
|
|
|
|
| Field | Type | Default | Description |
|
|
|-------|------|---------|-------------|
|
|
| `name` | string | 'default' | Instance name |
|
|
| `k3s_version` | string | 'v1.33.1' | K3s version to install |
|
|
| `data_dir` | string | '~/hero/var/k3s' | Data directory for K3s |
|
|
| `node_name` | string | hostname | Unique node identifier |
|
|
| `mycelium_interface` | string | auto-detected | Mycelium interface name (auto-detected from 400::/7 route) |
|
|
| `token` | string | auto-generated | Cluster authentication token |
|
|
| `master_url` | string | - | Master URL for joining (e.g., 'https://[ipv6]:6443') |
|
|
| `node_ip` | string | auto-detected | Node IPv6 (auto-detected from Mycelium) |
|
|
|
|
## Actions
|
|
|
|
### Installation Actions
|
|
|
|
- `install_master` - Install first master node (generates token, uses --cluster-init)
|
|
- `join_master` - Join as additional master (requires token + master_url)
|
|
- `install_worker` - Install worker node (requires token + master_url)
|
|
|
|
### Lifecycle Actions
|
|
|
|
- `start` - Start K3s via startupmanager
|
|
- `stop` - Stop K3s
|
|
- `restart` - Restart K3s
|
|
- `destroy` - Complete cleanup (removes all K3s components)
|
|
|
|
### Utility Actions
|
|
|
|
- `get_kubeconfig` - Get kubeconfig content
|
|
- `generate_join_script` - Generate heroscript for joining nodes
|
|
|
|
## Requirements
|
|
|
|
- **OS**: Ubuntu (installer checks and fails on non-Ubuntu systems)
|
|
- **Mycelium**: Must be installed and running with interface in 400::/7 range
|
|
- **Root Access**: Required for installing system packages and managing network
|
|
|
|
## How It Works
|
|
|
|
### Mycelium IPv6 Detection
|
|
|
|
The installer automatically detects your Mycelium IPv6 address by:
|
|
|
|
1. Finding the 400::/7 route via the Mycelium interface
|
|
2. Extracting the next-hop IPv6 and getting the prefix (first 4 segments)
|
|
3. Matching global IPv6 addresses on the interface with the same prefix
|
|
4. Using the matched IPv6 for K3s `--node-ip`
|
|
|
|
This ensures K3s binds to the correct Mycelium IPv6 even if the server has other IPv6 addresses.
|
|
|
|
### Cluster Setup
|
|
|
|
**First Master:**
|
|
|
|
- Uses `--cluster-init` flag
|
|
- Auto-generates secure token
|
|
- Configures IPv6 CIDRs: cluster=2001:cafe:42::/56, service=2001:cafe:43::/112
|
|
- Generates join script for other nodes
|
|
|
|
**Additional Masters:**
|
|
|
|
- Joins with `--server <master_url>`
|
|
- Requires token and master_url from first master
|
|
- Provides HA for control plane
|
|
|
|
**Workers:**
|
|
|
|
- Joins as agent with `--server <master_url>`
|
|
- Requires token and master_url from first master
|
|
|
|
### Cleanup
|
|
|
|
The `destroy` action performs complete cleanup:
|
|
|
|
- Stops K3s process
|
|
- Removes network interfaces (cni0, flannel.*, etc.)
|
|
- Unmounts kubelet mounts
|
|
- Removes data directory
|
|
- Cleans up iptables/ip6tables rules
|
|
- Removes CNI namespaces
|
|
|
|
## Example Workflow
|
|
|
|
1. **Install first master on server1:**
|
|
|
|
```bash
|
|
hero run templates/examples.heroscript
|
|
# Note the token and IPv6 address displayed
|
|
```
|
|
|
|
2. **Join additional master on server2:**
|
|
|
|
```bash
|
|
# Edit examples.heroscript Section 2 with token and master_url
|
|
hero run templates/examples.heroscript
|
|
```
|
|
|
|
3. **Add worker on server3:**
|
|
|
|
```bash
|
|
# Edit examples.heroscript Section 3 with token and master_url
|
|
hero run templates/examples.heroscript
|
|
```
|
|
|
|
4. **Verify cluster:**
|
|
|
|
```bash
|
|
kubectl get nodes
|
|
kubectl get pods --all-namespaces
|
|
```
|
|
|
|
## Kubeconfig
|
|
|
|
The kubeconfig is located at: `<data_dir>/server/cred/admin.kubeconfig`
|
|
|
|
To use kubectl:
|
|
|
|
```bash
|
|
export KUBECONFIG=~/hero/var/k3s/server/cred/admin.kubeconfig
|
|
kubectl get nodes
|
|
```
|
|
|
|
Or copy to default location:
|
|
|
|
```bash
|
|
mkdir -p ~/.kube
|
|
cp ~/hero/var/k3s/server/cred/admin.kubeconfig ~/.kube/config
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
**K3s won't start:**
|
|
|
|
- Check if Mycelium is running: `ip -6 addr show mycelium0`
|
|
- Verify 400::/7 route exists: `ip -6 route | grep 400::/7`
|
|
- Check logs: `journalctl -u k3s_* -f`
|
|
|
|
**Can't join cluster:**
|
|
|
|
- Verify token matches first master
|
|
- Ensure master_url uses correct IPv6 in brackets: `https://[ipv6]:6443`
|
|
- Check network connectivity over Mycelium: `ping6 <master_ipv6>`
|
|
|
|
**Cleanup issues:**
|
|
|
|
- Run destroy with sudo if needed
|
|
- Manually check for remaining processes: `pgrep -f k3s`
|
|
- Check for remaining mounts: `mount | grep k3s`
|
|
|
|
## See Also
|
|
|
|
- [K3s Documentation](https://docs.k3s.io/)
|
|
- [Mycelium Documentation](https://github.com/threefoldtech/mycelium)
|
|
- [Example Heroscript](templates/examples.heroscript)
|