Kubernetes Operations¶

Prerequisites¶

• K3s installed and running
• kubectl available

Procedure: Install K3s¶

When to use: Setting up a single-node Kubernetes cluster on a fresh VM using the K3s lightweight distribution.

Steps:

1. Create pre-configuration files before running the installer. K3s reads both at startup and will not pick them up if created afterwards:

   • /var/lib/rancher/k3s/server/manifests/traefik-config.yaml — overrides Traefik's default HTTP port from 80 to 8080 so it does not conflict with Apache:

     apiVersion: helm.cattle.io/v1
     kind: HelmChartConfig
     metadata:
       name: traefik
       namespace: kube-system
     spec:
       valuesContent: |-
         ports:
           web:
             exposedPort: 8080
           websecure:
             exposedPort: 8443
     

   • /etc/rancher/k3s/config.yaml — sets the kubeconfig permissions to world-readable and adds the VM's external IP as a valid TLS subject alternative name (required for kubectl to connect from outside the VM):

     write-kubeconfig-mode: "0644"
     tls-san:
       - "<your-vm-external-IP>"
     cluster-init: true
     

2. Run the installer:

   curl -sfL https://get.k3s.io | sh -
   

3. Verify the service started and the node is ready:

   systemctl status k3s
   kubectl get nodes
   

4. Open port 6443/tcp in firewalld and your cloud security group — this is the Kubernetes API port used by kubectl and the scoring server. You have done this before with other services.

5. Add the K3s pod and service networks to the trusted firewall zone so that the Traefik ingress pod can reach application pods:

   firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16  # pods
   firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16  # services
   firewall-cmd --reload
   

Notes:

• kubectl is bundled with K3s — the installer creates a symlink at /usr/local/bin/kubectl. No separate installation is required.
• The kubeconfig file at /etc/rancher/k3s/k3s.yaml is set to world-readable by write-kubeconfig-mode: "0644", so kubectl works for any user without setting a KUBECONFIG environment variable.
• If kubectl gives "command not found", /usr/local/bin is not in your PATH. Either use the full path or run export PATH=$PATH:/usr/local/bin.

Troubleshooting:

• Traefik still listening on port 80: The traefik-config.yaml was not in place before installation. Reinstall or patch the HelmChartConfig resource.
• kubectl certificate error: The VM's external IP was not in tls-san. Add it and restart k3s: systemctl restart k3s.
• K3s fails to start — check journalctl -u k3s -n 50 for the specific error.

Procedure: Test Pod Connectivity at Each Layer¶

When to use: Verifying that a new deployment works at each level of the stack before testing the full Ingress path.

Steps:

1. Get pod IPs:

   kubectl get pods -n <namespace> -l app=<app-label> -o wide
   

2. Test a pod directly (no Service, no Ingress — raw pod IP):

   curl http://<pod-ip>:<container-port>/
   

   If this fails, the application itself has a problem. Check kubectl logs <pod-name> -n <namespace>.

3. Get the Service ClusterIP:

   kubectl get svc <service-name> -n <namespace>
   

4. Test via the Service (stable IP, load-balanced across pods):

   curl http://<cluster-ip>:<service-port>/
   

   If pod works but Service doesn't, check that the Service selector matches the pod's labels.

5. Test via Traefik Ingress (uses hostname routing):

   curl --resolve <hostname>:8080:<vm-ip> http://<hostname>:8080/
   

   If Service works but Ingress doesn't, check that the Ingress host matches exactly and that ingressClassName: traefik is set.

Troubleshooting:

• Pod IP is unreachable from the host: ensure the pod/service CIDRs are in the trusted firewall zone (see Install K3s procedure).
• Service returns no endpoints: kubectl describe svc <name> -n <namespace> — check the Endpoints field.
• Ingress returns 404: kubectl get ingress -n <namespace> and check the host and path match what you're requesting.

Procedure: Apply a Manifest¶

When to use: Creating or updating Kubernetes resources (Pods, Deployments, Services).

Steps:

1. Apply a YAML file:

   kubectl apply -f manifest.yaml
   

2. Apply all files in a directory:

   kubectl apply -f ./k8s/
   

Troubleshooting:

• "The connection to the server was refused": Check if K3s/K8s is running (systemctl status k3s) or if KUBECONFIG is set.

Procedure: Check Pod Status¶

When to use: Monitoring deployment rollout and health.

Steps:

1. List all pods in default namespace:

   kubectl get pods
   

2. List pods in all namespaces:

   kubectl get pods -A
   

3. Watch for changes (live update):

   kubectl get pods -w
   

Troubleshooting:

• "ImagePullBackOff": Check image name and registry credentials. If using a local image, ensure imagePullPolicy is set to Never or IfNotPresent and the image exists on the node.
• "CrashLoopBackOff": Application is crashing. Check logs. Use kubectl logs <pod> --previous to see the logs of the last crashed instance.
• "Pending": The pod cannot be scheduled. Run kubectl describe pod <name> to check for resource limits (CPU/Memory) or Node Selectors/Taints.

Procedure: View Pod Logs¶

When to use: Debugging application errors.

Steps:

1. View logs:

   kubectl logs <pod_name>
   

2. View logs of a specific container (in multi-container pod):

   kubectl logs <pod_name> -c <container_name>
   

3. Follow logs:

   kubectl logs -f <pod_name>
   

Troubleshooting:

• "Pod not found": Check if you are in the correct namespace (-n <namespace>).

Procedure: Create a Namespace¶

When to use: Isolating resources for different environments or projects.

Steps:

1. Create namespace:

   kubectl create namespace my-app
   

2. Switch context to use it by default:

   kubectl config set-context --current --namespace=my-app
   

Troubleshooting:

• "Already exists": No action needed, or check spelling.

Procedure: Expose a Service via NodePort¶

When to use: Making an internal application accessible from outside the cluster.

Steps:

1. Expose a deployment:

   kubectl expose deployment my-dep --type=NodePort --port=80 --name=my-svc
   

2. Find the assigned NodePort:

   kubectl get svc my-svc
   

Troubleshooting:

• Cannot access port: Ensure firewall allows traffic on the assigned NodePort (30000-32767).

Procedure: Create an Ingress¶

When to use: Exposing HTTP/HTTPS services via a domain name.

Steps:

1. Create ingress.yaml:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: my-ingress
   spec:
     rules:
     - host: app.example.com
       http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: my-svc
               port:
                 number: 80
   

2. Apply it:

   kubectl apply -f ingress.yaml
   

Troubleshooting:

• 404 Not Found: Ensure the Ingress Controller (Traefik/Nginx) is running and DNS points to the cluster IP.

Procedure: Debug a Failing Pod¶

When to use: Detailed investigation of why a pod is pending or failing.

Steps:

1. Describe the pod events and status:

   kubectl describe pod <pod_name>
   

   Look at the "Events" section at the bottom.

2. Exec into the pod (if running):

   kubectl exec -it <pod_name> -- /bin/sh
   

Troubleshooting:

• "SchedulingFailed": Insufficient CPU/Memory resources on nodes.

Quick Reference¶

Related Documentation¶

• Technologies: Kubernetes
• Concepts: Container Orchestration