diff --git a/examples/nginx-nodeport/IPV6_FIX.md b/examples/nginx-nodeport/IPV6_FIX.md new file mode 100644 index 0000000..108fddb --- /dev/null +++ b/examples/nginx-nodeport/IPV6_FIX.md @@ -0,0 +1,242 @@ +# ๐Ÿšจ CRITICAL: IPv6 Configuration Fix Required + +## The Problem + +When initially deploying NodePort services on Mycelium Cloud, you may encounter a situation where: +- โœ… `ping -6` works (ICMP reaches the node) +- โŒ `curl -6 http://[ipv6]:30091/` fails (HTTP service unreachable) + +## Root Cause + +**IPv4-only service configuration!** + +By default, Kubernetes services created without explicit IP family configuration are often IPv4-only, even when nodes have IPv6 addresses. + +### Symptom +```bash +# Service shows IPv4-only +$ kubectl get svc nginx-nodeport-service -o json | grep ipFamily +"ipFamilyPolicy": "SingleStack", +"ipFamilies": ["IPv4"], +``` + +### Result +- Services only listen on IPv4 addresses +- IPv6 Mycelium addresses are "reachable" (ping works) but HTTP fails +- nginx itself is fine (listens on dual-stack) +- **kube-proxy won't forward IPv6 traffic to the service** + +--- + +## The Fix โญ + +Update the Service YAML to explicitly support dual-stack: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: nginx-nodeport-service +spec: + type: NodePort + externalTrafficPolicy: Local + ipFamilies: # โ† ADD THESE 3 LINES + - IPv4 + - IPv6 + ipFamilyPolicy: RequireDualStack # โ† ADD THIS + selector: + app: nginx-nodeport + ports: + - name: http + port: 8080 + targetPort: 8080 + nodePort: 30091 + protocol: TCP +``` + +### Apply the fix: +```bash +kubectl apply -f nginx-nodeport-service.yaml +``` + +### Verify the fix: +```bash +# Should now show dual-stack +$ kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.ipFamilies}' +["IPv4","IPv6"] + +$ kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.ipFamilyPolicy}' +RequireDualStack +``` + +--- + +## Why This Happens + +1. **Legacy default**: Many Kubernetes setups default to IPv4-only +2. **Service doesn't inherit node IPs**: Service has its own ClusterIP +3. **kube-proxy needs explicit instruction**: Must know to create IPv6 rules +4. **IP family policy**: Controls whether service listens on IPv4, IPv6, or both + +### Without dual-stack: +``` +User โ†’ Mycelium IPv6 โ†’ Node port 30091 + โ†“ + kube-proxy + โ†“ + IPv4 service โ† BOTTLENECK + โ†“ + Pod responds +``` + +### With dual-stack: +``` +User โ†’ Mycelium IPv6 โ†’ Node port 30091 + โ†“ + kube-proxy + โ†“ + IPv6 + IPv4 service โ† WORKS! + โ†“ + Pod responds +``` + +--- + +## Testing the Fix + +### Before fix: +```bash +$ curl -6 "http://[552:5984:2d97:72dc:ff0f:39ef:6ec:a48c]:30091/" +curl: (7) Failed to connect to ... Connection refused +``` + +### After fix: +```bash +$ curl -6 "http://[552:5984:2d97:72dc:ff0f:39ef:6ec:a48c]:30091/" + + + + Mycelium Cloud - Nginx NodePort Website +... +``` + +### Automated test: +```bash +cd myceliumcloud-examples/examples/nginx-nodeport +./test-nodeport-ipv6.sh + +# Should now show: +# โœ… External Mycelium IPv6 connectivity is working! +``` + +--- + +## Other Affected Variants + +This IP family configuration is critical for: + +- โœ… **NodePort** - This guide +- โœ… **LoadBalancer** - Will need same configuration (future example) +- โœ… **Ingress** - Will need same configuration (future example) +- โŒ **hostNetwork** - NOT affected (directly uses host network stack) + +--- + +## Kubernetes Version Compatibility + +### Kubernetes 1.20+ (Recommended) +- Full dual-stack support +- `RequireDualStack` policy available +- Complete IPv4/IPv6 feature set + +### Kubernetes 1.19 and earlier +- Limited dual-stack support +- May need `ipFamily: IPv6` instead of `RequireDualStack` +- Some features may not work + +**Check your version:** +```bash +kubectl version --short +``` + +--- + +## Quick Reference + +### Service configuration checklist: +- [ ] `type: NodePort` โœ“ +- [ ] `nodePort: 30091` โœ“ +- [ ] `externalTrafficPolicy: Local` โœ“ +- [ ] `ipFamilies: [IPv4, IPv6]` โญ Critical for Mycelium +- [ ] `ipFamilyPolicy: RequireDualStack` โญ Critical for Mycelium + +### Test sequence: +```bash +# 1. Deploy +kubectl apply -f nginx-nodeport-configmaps.yaml +kubectl apply -f nginx-nodeport-deployment.yaml +kubectl apply -f nginx-nodeport-service.yaml + +# 2. Wait +kubectl wait --for=condition=ready pod -l app=nginx-nodeport --timeout=60s + +# 3. Check IP families +kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.ipFamilies}' + +# 4. Test +NODE_IPV6=$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}{end}' | grep ':' | head -1) +curl -6 "http://[$NODE_IPV6]:30091/" +``` + +--- + +## Troubleshooting + +### Still not working after dual-stack config? + +1. **Check kube-proxy logs:** + ```bash + kubectl logs -n kube-system -l k8s-app=kube-proxy | grep -i error + ``` + +2. **Verify nginx listens on IPv6:** + ```bash + POD_NAME=$(kubectl get pods -l app=nginx-nodeport -o name | head -1) + kubectl exec $POD_NAME -- netstat -tuln | grep 8080 + # Should show: + # tcp 0 0 0.0.0.0:8080 :::* LISTEN + # tcp 0 0 :::8080 :::* LISTEN โ† IPv6 + ``` + +3. **Check service endpoints:** + ```bash + kubectl get endpoints nginx-nodeport-service + ``` + +4. **Test from within cluster:** + ```bash + kubectl run test --rm -it --image=curlimages/curl -- sh -c "curl http://$SERVICE_IP:8080" + ``` + +--- + +## Summary + +**For Mycelium Cloud NodePort deployments, ALWAYS include:** + +```yaml +spec: + ipFamilies: + - IPv4 + - IPv6 + ipFamilyPolicy: RequireDualStack +``` + +Without this, services won't be accessible via Mycelium IPv6 addresses, even though the nodes are reachable via ping. + +--- + +**Impact:** This is a **deployment-blocking issue** for anyone trying to use NodePort with Mycelium IPv6. + +**Status:** โœ… Fixed in updated `nginx-nodeport-service.yaml` +**Related:** [TROUBLESHOOTING.md](TROUBLESHOOTING.md) \ No newline at end of file diff --git a/examples/nginx-nodeport/debug-nodeport.sh b/examples/nginx-nodeport/debug-nodeport.sh new file mode 100755 index 0000000..45b5a10 --- /dev/null +++ b/examples/nginx-nodeport/debug-nodeport.sh @@ -0,0 +1,157 @@ +#!/bin/bash + +# NodePort Connectivity Debug Script +# Diagnoses why NodePort 30091 is not accessible despite successful ping + +set -e + +echo "๐Ÿ” NodePort Connectivity Diagnosis" +echo "====================================" +echo "" + +# Colors +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +RED='\033[0;31m' +NC='\033[0m' # No Color + +# Get pod info +POD_NAME=$(kubectl get pods -l app=nginx-nodeport -o name | head -1) +POD_NODE=$(kubectl get pods -l app=nginx-nodeport -o jsonpath='{.items[0].spec.nodeName}') +NODE_IPV6=$(kubectl get node "$POD_NODE" -o jsonpath='{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}' | grep -E '^[0-9a-f]+:[0-9a-f]+:' | head -1) + +echo "๐Ÿ“Š Pod Information:" +echo " Pod: $POD_NAME" +echo " Node: $POD_NODE" +echo " Node IPv6: $NODE_IPV6" +echo "" + +# Check service +echo "๐Ÿ” Service Configuration:" +kubectl get svc nginx-nodeport-service -o yaml | grep -E "type:|nodePort:|externalTrafficPolicy:|ipFamilies:" | sed 's/^/ /' +echo "" + +# Check endpoints +echo "๐Ÿ” Service Endpoints:" +kubectl get endpoints nginx-nodeport-service -o wide +echo "" + +# Test from within pod +echo "๐Ÿงช Test 1: Internal pod access (localhost)" +if kubectl exec $POD_NAME -- curl -s -m 5 http://localhost:8080/health > /dev/null 2>&1; then + echo -e " ${GREEN}โœ… SUCCESS${NC} - nginx responding on localhost:8080" +else + echo -e " ${RED}โŒ FAILED${NC} - nginx not responding internally" + exit 1 +fi +echo "" + +# Test service ClusterIP +echo "๐Ÿงช Test 2: Service ClusterIP access" +SERVICE_IP=$(kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.clusterIP}') +echo " Service ClusterIP: $SERVICE_IP" + +# Create a test pod to check from +echo " Creating test pod..." +kubectl run test-curl --image=curlimages/curl:latest --rm -i --restart=Never --command -- sh -c "curl -s -m 5 http://$SERVICE_IP:8080/health" > /tmp/test-result.txt 2>&1 || true + +if grep -q "healthy" /tmp/test-result.txt 2>/dev/null; then + echo -e " ${GREEN}โœ… SUCCESS${NC} - Service ClusterIP accessible" +else + echo -e " ${YELLOW}โš ๏ธ FAILED${NC} - Service ClusterIP not accessible" + cat /tmp/test-result.txt 2>/dev/null || true +fi +rm -f /tmp/test-result.txt +echo "" + +# Check if kube-proxy is running +echo "๐Ÿ” kube-proxy Status:" +KUBE_PROXY_PODS=$(kubectl get pods -n kube-system -l k8s-app=kube-proxy --no-headers 2>/dev/null | wc -l) +if [ "$KUBE_PROXY_PODS" -gt 0 ]; then + echo -e " ${GREEN}โœ… kube-proxy running${NC} ($KUBE_PROXY_PODS pods)" +else + echo -e " ${RED}โŒ kube-proxy NOT running${NC}" +fi +echo "" + +# Check node ports +echo "๐Ÿ” NodePort Listener Check:" +echo " Checking if port 30091 is being listened on..." +echo "" +echo " Note: On the worker node ($POD_NODE), check with:" +echo " ssh root@$POD_NODE 'netstat -tuln | grep 30091'" +echo " ssh root@$POD_NODE 'ss -tuln | grep 30091'" +echo "" + +# Try to access NodePort from inside cluster +echo "๐Ÿงช Test 3: NodePort access from test pod" +echo " Testing: http://[$NODE_IPV6]:30091/" + +kubectl run test-nodeport --image=curlimages/curl:latest --rm -i --restart=Never --command -- sh -c "curl -6 -s -m 10 http://[$NODE_IPV6]:30091/health" > /tmp/nodeport-test.txt 2>&1 || true + +if grep -q "healthy" /tmp/nodeport-test.txt 2>/dev/null; then + echo -e " ${GREEN}โœ… SUCCESS${NC} - NodePort accessible from within cluster" +else + echo -e " ${RED}โŒ FAILED${NC} - NodePort not accessible from within cluster" + echo " Output:" + cat /tmp/nodeport-test.txt 2>/dev/null | sed 's/^/ /' || true +fi +rm -f /tmp/nodeport-test.txt +echo "" + +# Check iptables rules (if we can access the node) +echo "๐Ÿ” Potential Issues:" +echo "" +echo " 1. IPv6 Support in kube-proxy:" +echo " kube-proxy might not be configured for IPv6" +echo " Check with: kubectl logs -n kube-system -l k8s-app=kube-proxy | grep -i ipv6" +echo "" +echo " 2. Firewall on worker node:" +echo " Port 30091 might be blocked by firewall" +echo " On node: sudo iptables -L -n | grep 30091" +echo " On node: sudo ip6tables -L -n | grep 30091" +echo "" +echo " 3. externalTrafficPolicy: Local:" +echo " Service only accessible on the node where pod is running" +echo " Pod is on: $POD_NODE" +echo " You must use that node's IPv6: $NODE_IPV6" +echo "" +echo " 4. IPv6 iptables rules:" +echo " kube-proxy might not have created IPv6 rules" +echo " On node: sudo ip6tables -t nat -L -n | grep 30091" +echo "" + +# Get all node IPs to show which one to use +echo "๐Ÿ“‹ Summary:" +echo "" +echo " Pod running on node: $POD_NODE" +echo " MUST use this IPv6: $NODE_IPV6" +echo "" +echo " Try accessing:" +echo " curl -6 \"http://[$NODE_IPV6]:30091/\"" +echo "" +echo " All worker node IPs (for reference):" +kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}{end}' | grep -E '^[0-9a-f]+:[0-9a-f]+:' | nl -v 1 -w 5 -s '. ' | sed 's/^/ /' +echo "" + +# Suggest checking k3s/k8s configuration +echo "๐Ÿ”ง Recommended Checks:" +echo "" +echo "1. Check if this is a K3s cluster:" +echo " kubectl version --short" +echo "" +echo "2. Check kube-proxy mode:" +echo " kubectl logs -n kube-system -l k8s-app=kube-proxy | grep -i mode" +echo "" +echo "3. Check if IPv6 is enabled in cluster:" +echo " kubectl get nodes -o jsonpath='{.items[0].status.addresses}' | grep ':'" +echo "" +echo "4. Check service IP families:" +echo " kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.ipFamilies}'" +echo "" +echo "5. Try changing externalTrafficPolicy to Cluster:" +echo " kubectl patch svc nginx-nodeport-service -p '{\"spec\":{\"externalTrafficPolicy\":\"Cluster\"}}'" +echo "" + +echo "======================================" +echo "Diagnosis complete!" \ No newline at end of file diff --git a/examples/nginx-nodeport/nginx-nodeport-service.yaml b/examples/nginx-nodeport/nginx-nodeport-service.yaml index 408ee37..6acf1bd 100644 --- a/examples/nginx-nodeport/nginx-nodeport-service.yaml +++ b/examples/nginx-nodeport/nginx-nodeport-service.yaml @@ -9,6 +9,10 @@ metadata: spec: type: NodePort externalTrafficPolicy: Local + ipFamilies: + - IPv4 + - IPv6 + ipFamilyPolicy: RequireDualStack selector: app: nginx-nodeport ports: diff --git a/examples/nginx-nodeport/nginx-nodeport.md b/examples/nginx-nodeport/nginx-nodeport.md index d571f4c..a2dd8c0 100644 --- a/examples/nginx-nodeport/nginx-nodeport.md +++ b/examples/nginx-nodeport/nginx-nodeport.md @@ -2,6 +2,22 @@ A complete, production-ready example for deploying a secure, globally accessible website on Mycelium Cloud using Kubernetes **NodePort** services with enhanced network isolation. This demonstrates **security-first IPv6 web hosting** with standard Kubernetes patterns. +## ๐Ÿšจ CRITICAL: IPv6 Configuration Required + +**IMPORTANT:** This example requires **dual-stack service configuration** to work with Mycelium IPv6 addresses. Without it, you'll be able to ping IPv6 addresses but not access the service. + +**The Fix:** The `nginx-nodeport-service.yaml` includes: +```yaml +ipFamilies: + - IPv4 + - IPv6 +ipFamilyPolicy: RequireDualStack +``` + +**Why:** Kubernetes services default to IPv4-only. Mycelium uses IPv6, so we must explicitly enable dual-stack. + +**Status:** โœ… Included in provided YAML files. See [IPV6_FIX.md](IPV6_FIX.md) for details. + ## ๐Ÿ”‘ Key Concept: NodePort Access Pattern **Important:** With NodePort, you access the service via: