tfgrid/myceliumcloud-examples
11
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b195cd01713c0a17c65c065a55447f2b186fdb5a
myceliumcloud-examples/examples/nginx-nodeport/TROUBLESHOOTING.md

9.1 KiB
Raw Blame History

nginx-nodeport Troubleshooting Guide

Common Issues and Solutions

Issue: External IPv6 Connectivity Test Failed

Symptoms:

⚠️  External Mycelium IPv6 connectivity test failed
ℹ️  This might be due to:
ℹ️    • Firewall or network policies blocking port 30091
ℹ️    • Pod not running on this specific node
ℹ️    • Mycelium network not properly configured

Root Causes & Solutions:

1. Mycelium Not Running on Your Machine ⭐ Most Common

Problem: You're trying to access from a machine without Mycelium installed/running.

Solution: Install and start Mycelium on your local machine:

# Download Mycelium
wget https://github.com/threefoldtech/mycelium/releases/latest/download/mycelium-private-x86_64-unknown-linux-musl.tar.gz
tar -xzf mycelium-private-x86_64-unknown-linux-musl.tar.gz
sudo chmod +x mycelium-private
sudo mv mycelium-private /usr/local/bin/mycelium

# Start Mycelium with peers
sudo mycelium --peers tcp://188.40.132.242:9651 tcp://136.243.47.186:9651 tcp://185.69.166.7:9651

After starting Mycelium, test connectivity:

# Your machine should now have a Mycelium IPv6 address
ip addr show | grep mycelium

# Test connection
NODE_IPV6="552:5984:2d97:72dc:ff0f:39ef:6ec:a48c"  # Replace with your node's IP
curl -6 "http://[$NODE_IPV6]:30091/"

2. Testing from Wrong Location

Problem: Testing from within cluster or wrong network segment.

Solution: Test from a machine WITH Mycelium running:

# ✅ CORRECT: Test from machine with Mycelium
your-laptop$ curl -6 "http://[node-mycelium-ipv6]:30091/"

# ❌ WRONG: Test from within cluster (use localhost instead)
kubectl exec pod/nginx-nodeport-xxx -- curl localhost:8080

3. Firewall Blocking NodePort

Problem: Firewall rules blocking port 30091.

Check firewall:

# On the worker node
sudo iptables -L -n | grep 30091
sudo firewall-cmd --list-ports  # if using firewalld

Solution: Open port 30091 on worker nodes:

# Using iptables
sudo iptables -I INPUT -p tcp --dport 30091 -j ACCEPT

# Using firewalld
sudo firewall-cmd --permanent --add-port=30091/tcp
sudo firewall-cmd --reload

4. externalTrafficPolicy: Local Issue

Problem: With externalTrafficPolicy: Local, service is only accessible on nodes where pods are running.

Understanding:

  • ✅ Pod on node A → Access via node A's IP works
  • ❌ Pod on node A → Access via node B's IP fails

Solution: Always use the IP of the node where your pod is running:

# Find which node has your pod
POD_NODE=$(kubectl get pods -l app=nginx-nodeport -o jsonpath='{.items[0].spec.nodeName}')
echo "Pod is on: $POD_NODE"

# Get THAT node's Mycelium IPv6
NODE_IPV6=$(kubectl get node "$POD_NODE" -o jsonpath='{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}' | grep ':' | head -1)
echo "Use this IP: $NODE_IPV6"

# Test
curl -6 "http://[$NODE_IPV6]:30091/"

5. IPv6 vs IPv4 Confusion

Problem: Getting both IPv4 and IPv6 addresses, causing malformed URLs.

Symptoms:

curl: (3) bad range in URL position 9:
http://[10.20.2.2 51d:3596:6cc3:81e7:ff0f:d546:3737:4c8c]:30091/

Solution: Filter for IPv6 only:

# Correct way to get IPv6 only
kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}{end}' | grep -E '^[0-9a-f]+:[0-9a-f]+:'

# NOT this (gets both IPv4 and IPv6)
kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'

Issue: Pod Not Accessible

Check pod status:

kubectl get pods -l app=nginx-nodeport
kubectl describe pod -l app=nginx-nodeport
kubectl logs -l app=nginx-nodeport

Test internal connectivity:

POD_NAME=$(kubectl get pods -l app=nginx-nodeport -o name | head -1)

# Test from within pod
kubectl exec $POD_NAME -- curl -s http://localhost:8080/

# Test health endpoint
kubectl exec $POD_NAME -- curl -s http://localhost:8080/health

Issue: Service Not Working

Check service configuration:

# Verify service type
kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.type}'
# Should output: NodePort

# Check NodePort assignment
kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.ports[0].nodePort}'
# Should output: 30091

# Check endpoints
kubectl get endpoints nginx-nodeport-service
# Should show pod IP and port

Issue: Wrong Content Displayed

Update ConfigMap:

# Regenerate content with current IPs
./update-content.sh

# Restart deployment to pick up changes
kubectl rollout restart deployment/nginx-nodeport

# Wait for rollout
kubectl rollout status deployment/nginx-nodeport

Testing Checklist

Run through this checklist to verify everything:

  • Mycelium Running: ip addr show | grep mycelium
  • Pod Running: kubectl get pods -l app=nginx-nodeport
  • Service Exists: kubectl get svc nginx-nodeport-service
  • NodePort 30091: kubectl get svc nginx-nodeport-service -o jsonpath='{.spec.ports[0].nodePort}'
  • Pod Node Located: kubectl get pods -l app=nginx-nodeport -o wide
  • Node IPv6 Found: kubectl get node <node-name> -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}' | grep ':'
  • Internal Access: kubectl exec <pod> -- curl -s http://localhost:8080/
  • External Access: curl -6 "http://[node-ipv6]:30091/"

Quick Validation Commands

# Complete test sequence
cd myceliumcloud-examples/examples/nginx-nodeport

# 1. Run automated tests
./test-nodeport-ipv6.sh

# 2. Find your pod's node
POD_NODE=$(kubectl get pods -l app=nginx-nodeport -o jsonpath='{.items[0].spec.nodeName}')
echo "Pod is on node: $POD_NODE"

# 3. Get that node's Mycelium IPv6
NODE_IPV6=$(kubectl get node "$POD_NODE" -o jsonpath='{range .status.addresses[?(@.type=="InternalIP")]}{.address}{"\n"}{end}' | grep -E '^[0-9a-f]+:[0-9a-f]+:' | head -1)
echo "Node Mycelium IPv6: $NODE_IPV6"

# 4. Test from machine with Mycelium
curl -6 "http://[$NODE_IPV6]:30091/"

Expected Behavior

✅ Working Correctly

When everything is working, you should see:

  1. Test Script Output:
✅ nginx-nodeport pods are ready
✅ nginx configuration is valid
✅ Health endpoint is working
✅ NodePort service is configured correctly
✅ NodePort is correctly set to 30091
  1. External Access:
$ curl -6 "http://[552:5984:2d97:72dc:ff0f:39ef:6ec:a48c]:30091/"
<!DOCTYPE html>
<html lang="en">
<head>
    <title>Mycelium Cloud - Nginx NodePort Website</title>
...
  1. Browser Access:
    • Navigate to: http://[node-mycelium-ipv6]:30091/
    • Should see: Styled website with "NODEPORT SECURE" badge

Advanced Debugging

Check kube-proxy

# On worker node
sudo iptables-save | grep 30091

View Service Routing

# Check service details
kubectl describe svc nginx-nodeport-service

# Check endpoints
kubectl get endpoints nginx-nodeport-service -o yaml

Test from Another Pod

# Create test pod
kubectl run test-pod --image=nginx:alpine --rm -it -- sh

# From inside test pod
wget -O- http://nginx-nodeport-service:8080/
# Should work (internal cluster access)

# Try NodePort (won't work from inside cluster)
wget -O- http://[node-ip]:30091/
# May not work depending on network policies

Still Having Issues?

If none of the above solutions work:

  1. Check Logs:

    kubectl logs -l app=nginx-nodeport --tail=50

  2. Describe Resources:

    kubectl describe deployment nginx-nodeport
kubectl describe service nginx-nodeport-service
kubectl describe pod -l app=nginx-nodeport

  3. Verify Network:

    # Check pod can reach internet
kubectl exec <pod> -- ping -c 3 8.8.8.8

# Check pod networking
kubectl exec <pod> -- ip addr
kubectl exec <pod> -- ip route

  4. Compare with hostNetwork:

    • The nginx-mycelium example uses hostNetwork
    • If that works but NodePort doesn't, it's likely a service/networking issue
    • If neither works, check Mycelium connectivity

Common Questions

Q: Why can't I access from any node's IP? A: With externalTrafficPolicy: Local, you can only access via the node where the pod is running. This preserves source IP but limits accessibility.

Q: Should I change to externalTrafficPolicy: Cluster? A: You can, but you'll lose source IP preservation. Decision depends on your logging/security needs.

Q: How is this different from hostNetwork? A: NodePort provides network isolation and standard Kubernetes patterns. HostNetwork gives direct host access (less secure). See compare-approaches.md.

Q: Why port 30091 and not 80? A: Kubernetes NodePorts use range 30000-32767. Port 80 requires LoadBalancer or Ingress type services.

Last Updated: 2025-01-07 Related Docs: