myceliumcloud-examples/examples/nginx-nodeport/deployment-troubleshooting.md

Deployment Troubleshooting Guide for nginx-nodeport

This guide helps resolve common connectivity and deployment issues with the nginx-nodeport implementation.

🚨 Common Connection Issues

Issue 1: API Server Timeout

Error: dial tcp [IPv6]:6443: i/o timeout

Cause: Kubernetes API server is slow to respond or network connectivity issues.

Solutions:

# Option 1: Disable API validation (faster deployment)
kubectl apply -f nginx-nodeport-configmaps.yaml --validate=false
kubectl apply -f nginx-nodeport-deployment.yaml --validate=false
kubectl apply -f nginx-nodeport-service.yaml --validate=false

# Option 2: Use client-side dry-run to verify YAML syntax
kubectl apply -f nginx-nodeport-configmaps.yaml --dry-run=client
kubectl apply -f nginx-nodeport-deployment.yaml --dry-run=client
kubectl apply -f nginx-nodeport-service.yaml --dry-run=client

# Option 3: Increase timeout
kubectl apply -f nginx-nodeport-configmaps.yaml --timeout=5m

Issue 2: Network Connectivity Problems

Error: Unable to connect to the server

Diagnose and Fix:

# Check cluster connectivity
kubectl cluster-info

# Check node status
kubectl get nodes

# Verify kubeconfig
kubectl config view

# Test basic connectivity
kubectl get pods --all-namespaces

Issue 3: Slow API Responses

Error: Operations take very long or timeout

Performance Optimizations:

# Use smaller output formats
kubectl get pods -l app=nginx-nodeport -o wide

# Disable unnecessary features
kubectl apply -f nginx-nodeport-configmaps.yaml --v=1

# Use specific resource targeting
kubectl apply -f nginx-nodeport-configmaps.yaml --namespace=default

🔍 Pre-Deployment Validation

YAML Syntax Validation (No Cluster Required)

# Validate all files without cluster connection
for file in *.yaml; do
    echo "Validating $file..."
    kubectl create -f "$file" --dry-run=client --validate=false
done

File-by-File Validation

# Test each component individually
echo "=== Testing ConfigMaps ==="
kubectl apply -f nginx-nodeport-configmaps.yaml --dry-run=client

echo "=== Testing Deployment ==="
kubectl apply -f nginx-nodeport-deployment.yaml --dry-run=client

echo "=== Testing Service ==="
kubectl apply -f nginx-nodeport-service.yaml --dry-run=client

🚀 Alternative Deployment Methods

Method 1: Manual Resource Creation

If kubectl apply fails, create resources manually:

# 1. Create ConfigMaps
kubectl create configmap nginx-nodeport-content --from-literal=index.html="<h1>Hello NodePort</h1>"
kubectl create configmap nginx-nodeport-nginx-config --from-literal=default.conf="server { listen 8080; }"

# 2. Create Deployment
kubectl create deployment nginx-nodeport --image=nginx:alpine --replicas=1

# 3. Expose Service
kubectl expose deployment nginx-nodeport --type=NodePort --port=8080 --target-port=8080

Method 2: Using Helm (Advanced)

Create a Helm chart for easier deployment:

# Install Helm if not present
# Then package and deploy
helm package .
helm install nginx-nodeport ./nginx-nodeport-0.1.0.tgz

Method 3: Scripted Deployment

Use the provided deployment script:

# Make script executable
chmod +x deploy-nginx-nodeport.sh

# Run automated deployment
./deploy-nginx-nodeport.sh

🛠️ Creating deployment-nginx-nodeport.sh

Let me create a robust deployment script that handles connectivity issues:

#!/bin/bash

# nginx-nodeport Deployment Script
# Handles connectivity issues and provides fallback options

set -e

echo "🚀 Deploying nginx-nodeport to Mycelium Cloud"
echo "=============================================="

# Colors
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
RED='\033[0;31m'
NC='\033[0m'

print_success() { echo -e "${GREEN}✅ $1${NC}"; }
print_warning() { echo -e "${YELLOW}⚠️  $1${NC}"; }
print_error() { echo -e "${RED}❌ $1${NC}"; }

# Function to deploy with fallback options
deploy_with_fallback() {
    local file=$1
    local name=$2
    
    echo "📋 Deploying $name..."
    
    # Try normal deployment
    if kubectl apply -f "$file" --validate=false 2>/dev/null; then
        print_success "$name deployed successfully"
        return 0
    fi
    
    # Try with client validation only
    print_warning "Normal deployment failed, trying client-side validation..."
    if kubectl apply -f "$file" --dry-run=client --validate=false 2>/dev/null; then
        print_warning "$name validated (client-side only) - cluster may be unavailable"
        return 1
    fi
    
    # Try manual creation for specific resource types
    print_error "$name deployment failed completely"
    return 2
}

# Check cluster connectivity
echo "🔍 Checking cluster connectivity..."
if kubectl cluster-info &>/dev/null; then
    print_success "Cluster is accessible"
    CLUSTER_AVAILABLE=true
else
    print_warning "Cluster is not accessible - will use client-side validation only"
    CLUSTER_AVAILABLE=false
fi

# Deploy ConfigMaps
if [ "$CLUSTER_AVAILABLE" = true ]; then
    deploy_with_fallback "nginx-nodeport-configmaps.yaml" "ConfigMaps"
else
    echo "🔍 Validating ConfigMaps (client-side)..."
    if kubectl create -f nginx-nodeport-configmaps.yaml --dry-run=client --validate=false &>/dev/null; then
        print_success "ConfigMaps validated (client-side)"
    else
        print_error "ConfigMaps validation failed"
        exit 1
    fi
fi

# Deploy Deployment
if [ "$CLUSTER_AVAILABLE" = true ]; then
    deploy_with_fallback "nginx-nodeport-deployment.yaml" "Deployment"
else
    echo "🔍 Validating Deployment (client-side)..."
    if kubectl create -f nginx-nodeport-deployment.yaml --dry-run=client --validate=false &>/dev/null; then
        print_success "Deployment validated (client-side)"
    else
        print_error "Deployment validation failed"
        exit 1
    fi
fi

# Deploy Service
if [ "$CLUSTER_AVAILABLE" = true ]; then
    deploy_with_fallback "nginx-nodeport-service.yaml" "Service"
else
    echo "🔍 Validating Service (client-side)..."
    if kubectl create -f nginx-nodeport-service.yaml --dry-run=client --validate=false &>/dev/null; then
        print_success "Service validated (client-side)"
    else
        print_error "Service validation failed"
        exit 1
    fi
fi

# Final status
if [ "$CLUSTER_AVAILABLE" = true ]; then
    echo ""
    echo "🎉 Deployment Complete!"
    echo "======================"
    echo "📊 Checking deployment status..."
    kubectl get all -l app=nginx-nodeport
    
    # Get access information
    NODE_IPV6=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}' 2>/dev/null || echo "YOUR-NODE-IPV6")
    echo ""
    echo "🌐 Access your website at: http://[$NODE_IPV6]:30090"
    echo "🧪 Run health check: curl -6 http://[$NODE_IPV6]:30090/health"
    echo "📋 Test script: ./test-nodeport-ipv6.sh"
else
    echo ""
    print_warning "Deployment files are valid but cluster is not accessible"
    echo "📝 When cluster becomes available, run:"
    echo "   kubectl apply -f nginx-nodeport-configmaps.yaml --validate=false"
    echo "   kubectl apply -f nginx-nodeport-deployment.yaml --validate=false"
    echo "   kubectl apply -f nginx-nodeport-service.yaml --validate=false"
fi

echo ""
echo "📚 Next Steps:"
echo "   • Check documentation: nginx-nodeport.md"
echo "   • Compare approaches: compare-approaches.md"
echo "   • Run validation: deployment-validation.md"

🧪 Testing Without Cluster

Syntax Validation

# Test all YAML files for syntax errors
for file in *.yaml; do
    echo "Testing $file syntax..."
    python3 -c "
import yaml
import sys
try:
    with open('$file') as f:
        yaml.safe_load_all(f)
    print('✅ $file: Valid YAML')
except Exception as e:
    print('❌ $file: Invalid -', str(e))
    sys.exit(1)
"
done

Lint Kubernetes YAML

# Install kube-linter if not present
# wget https://github.com/stackrox/kube-linter/releases/download/v0.6.0/kube-linter_Linux_x86_64.tar.gz
# tar xzf kube-linter_Linux_x86_64.tar.gz
# sudo mv kube-linter /usr/local/bin/

# Lint all files
kube-linter lint *.yaml

🔧 Cluster-Specific Issues

Mycelium Cloud Specific

# Check Mycelium-specific connectivity
kubectl get nodes -o wide
kubectl describe node | grep -A 5 -B 5 "Addresses"

# Test IPv6 connectivity
ping6 -c 3 $(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')

# Check Mycelium network plugin
kubectl get pods -n kube-system | grep mycelium

Local Development Clusters

# For minikube
minikube status
minikube start --driver=docker

# For kind
kind get clusters
kind create cluster --name mycelium-test

# For k3s
sudo systemctl status k3s

📊 Performance Optimization

Faster Operations

# Use shorter timeouts for faster failure
kubectl get pods --timeout=30s

# Reduce verbosity
kubectl get pods -l app=nginx-nodeport -o json

# Use specific namespaces
kubectl config set-context --current --namespace=default

Resource-Afficient Queries

# Lightweight status checks
kubectl get pods -l app=nginx-nodeport --no-headers

# Minimal output
kubectl get svc nginx-nodeport-service -o name

# Batch operations
kubectl get deployment,svc,configmap -l app=nginx-nodeport

🆘 Emergency Deployment

If all else fails, here's a minimal emergency deployment:

# Emergency nginx deployment
kubectl create deployment emergency-nginx --image=nginx:alpine
kubectl expose deployment emergency-nginx --type=NodePort --port=80
kubectl patch service emergency-nginx -p '{"spec":{"ports":[{"port":80,"targetPort":80,"nodePort":30090}]}}'

# Get access
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
echo "Emergency site: http://$NODE_IP:30090"

This comprehensive troubleshooting guide should help resolve most deployment issues and provide multiple fallback options for different scenarios.