I'm continuing to interview for Staff DevOps Engineer which is typically working with k8s. I wanted to share some of the interview questions I've seen lately.

Q: In regards to running Kubernetes in a highly secure/compliant environment, best practices state to avoid containers running as the root user. What are some examples of times when would you NOT want to follow this recommendation?

A: Running monitoring agent, or generally collecting host level metrics.

Q: You deploy a helm chart to your cluster but your pods are failing to start. Walk me through the commands you would use to investigate this issue.

A: Start with listing all pods across all namespaces using `kubectl get pods -A`, looking for issues related to the helm chart but also other controller pods that may be having issues. Describe any pods that look interesting with `kubectl describe <pod_name>`. Start investigating pods that are trying to start using `kubectl logs <pod_name> -c <container_name>` (walking through each container in the pod). Exec into any containers to confirm any connection related hypothesis that may have formed using `kubectl exec -it <pod_name> -c <container_name> bash`. If the problem was related to storage, start describing the Storage Class, PV, PVCs with `kubectl describe`.

Q: When running a multi-tenant k8s cluster, explain the pros/cons of using namespaces vs virtual clusters.

A: Namespaces are easy to implement, they provide some isolation for multi-tenant applications, but the resources are by default sharing the underlying host infrastructure (nodes, NICs, etc.). Virtual clusters are more work but allow you to run k8s (k3s) within k8s that enables true isolation using virtual nodes and other resources for sensitive tenants wanting to co-exist on the same cluster.

Q: Your production k8s cluster runs 3 services from 3 different business units in AWS EKS. You know the running costs for the entire cluster. You are asked to identify the costs per service. Explain how you would accomplish this.

A: AWS EKS supports kubecost which can monitor costs by k8s resources.

Q: Consider an enterprise-level cloud-based k8s environment with appropriate IAM access control (AWS, GKE, or Azure). How does RBAC work in this environment?

A: This one has a lot to it and is easily found on Google.

Q: What are some challenges running k8s in a hybrid cloud environment where some nodes are on-prem and others are in the cloud?

A: Networking, latency. (Thank you @Taran_preet_Singh)

Q: What are some known security vulnerabilities or risks associated with running k8s? What are some hardening practices?

A: Ensure the Cluster API is only accessible from a private subnet, avoid running containers as root user, by default encryption isn't enabled in many places, network segmentation, supply chain, container scanning, etc.

Q: What are some cost optimization strategies for running k8s in AWS EKS or similar?

A: Invoke pod resource limits, using right-sized nodes, using Karpenter for dynamic node provisioning/auto-scaling nodes, consider Fargate for appropriate workloads needing to scale up/down frequently, basically trying to ensure resource utilization remains high to avoid wasted costs.

Q: Some developers came back from an AWS conference and want to move everything into AWS EKS Fargate. How would you approach an upcoming meeting to discuss this idea? What are some of the questions you would ask?

A: My goal for approaching this meeting is to understand whether there are true benefits to migrating to EKS/Fargate. Too often people think of k8s as this silver bullet that will solve all problems, or just blindly want to migrate to it so they can add it to their resume. I think it's been shown that just about anything can run on k8s but that doesn't always mean that there will be benefits to justify the migration work. The greater benefit often is in (proper) containerization itself, and that isn't synonymous with migrating to k8s. My questions would include: What problems are you hoping to solve by migrating to k8s? Is the app already containerized? Which components of the app need to scale independently? Are there stateful or legacy applications that have special requirements? Any other requirements related to security/compliance, networking, storage, etc.? Who on the team has the necessary skills to work with k8s and follow best practices? Have you considered how this will work with current/future plans for CI/CD, monitoring/logging, configuration management, and integrating with other infrastructure? Is there a timeline? - There are many more questions that should be addressed. Essentially, I want to understand the motive, expectations, and timeline. If it has support I would want to move forward with a POC and ideally let the data influence the decision as much as possible.

Q: Your AWS EKS cluster is designed to use 3 private subnets across 3 AZs. You notice that your 6 pod service has 3 pods running in AZ1, 2 running in AZ2, and 1 running in AZ3. How would you accomplish ensuring the pods are spread evenly across each AZ?

A: Define topology spread constraints and ideally use Karpenter with a different instance types. Too often I've seen a specific instance type be unavailable in a certain AZ due to high demand. Providing Karpenter with a few options [m5.xlarge, m5.2xlarge, m6i.large, m6i.2xlarge] reduces the likelihood of this happening.

Q: What is the most challenging problem you've faced related to k8s and how did you work through it? Be as detailed as possible.

A: This one should be personal from your own experience.

Please share some of the memorable questions you've encountered lately!

Edit: Added answers. Formatting could be better.