r/kubernetes • u/Crinja1 • 4d ago
How does an experienced (8+ years) DevOps engineer break into a Kubernetes-heavy role without years of production K8s experience?
I’ve been working in cloud/DevOps for about 8 years, primarily in AWS, IaC/Terraform, CI/CD, automation, Linux, and supporting fairly complex production systems (with some EKS migrations / troubleshooting and home lab stuff mixed in).
The weird position I’m finding myself in now is that a lot of roles that match my overall experience level are Senior DevOps/Platform/Cloud Engineer positions — but those same positions increasingly expect significant production Kubernetes experience.
For those who made this transition later in their DevOps/cloud careers:
How did you bridge that gap?
Did you find employers willing to hire for your broader DevOps experience and let you ramp up on Kubernetes?
How valuable were home labs/projects compared with actual production experience?
Are there particular K8s skills you’d prioritize to become useful quickly on an established platform? Thanks!
17
u/Grand_Pop_7221 4d ago
If you've got the skillset you're claiming to have then I think you need to find somewhere that's willing to give you a bit of grace to learn it on the job. It's not out of the question that you could do that.
It's a lot of work though, lots of self learning and reading and watching talks.
I did it five years ago and it was stressful but I think worth it.
If you struggle to find somewhere, just lie and learn on the job. If you don't make it you'll have more experience going into the next place you bullshit your way into 😂
9
u/siteunreliability k8s contributor 4d ago
If you struggle to find somewhere, just lie and learn on the job.
There's a difference between marketing the best version of yourself vs outright lying. That's how you get exposed.
I agree with the first statement, though.
15
u/Noah_Safely 4d ago
So I literally did this ~ 5-6 years ago. I had a ton of traditional SA/syseng type gigs, prod environments, tons of *nix, virtualization etc. Zero containers and zero k8s.
I can only give you the path that worked for me:
- Poked around a little, realized how big of an actual iceberg the ecosystem is, decided to study for CKA to get me some accelerated focused learning.
- Did a bunch of lab work as the course progressed, built a small cluster on my laptop, tested out various scenarios
- Actually took the CKA cert then quickly after CKAD (as there's a huge overlap).
- Did more lab work on the ecosystem.. observability, CICD, managed k8s (EKS)
- Started studying for the CKS, which actually turned out to be the most fun of the certs (I like infosec work) and let me speak a bit more intelligently in interviews, bringing up stuff they likely didn't have going on
Also you should be creating realistic failure scenarios along the way like kill the DB and see how you'd troubleshoot, or whatever.
One of the foundational things I did was take a traditional 3tier app and start building off that. Simple website fed by DB with a middle tier cache. Like:
- Recreate app in containers
- Recreate app in raw yaml manifest in my lab (config in configmaps, use secrets for creds etc)
- Recreate app in helm
- Start applying stuff via cicd (I used argo)
- Model up security restrictions with networkpolicies
- Play with LB provisioning via metallb
- Add some metrics to app and start playing with otel (prometheus is kinda required-ish nowadays?). Used alloy, grafana for visualization etc
- Played with service mesh (istio)
All the while I continued taking whatever free online courses I could find, even the ones that are really just a "demo to sale" pipeline. Calico, istio, I took some rancher online course.. whatever I could find to get more exposure to new concepts.
Honestly it's mostly about mapping your existing knowledge to the new stuff.
As for how I actually got a gig without experience.. I just said I had experience. If you can actually back it up with skills and talk intelligently in interviews, you'll be fine. The one thing I wish I had more experience with in an interview was tracing, I had a little from istio with jaeger (at the time) but the shop I got a contract with had a full on complex tracing setup.. think it was this https://docs.datadoghq.com/containers/kubernetes/apm/?tab=datadogoperator
Oh yea also be sure to play around with scaling, you wanna know the concepts.. like cluster autoscaler, karpenter etc
8
u/iadknet 4d ago edited 4d ago
I'm in the same position.
The things I have run the last several years have been mainly ECS/Fargate/Lambda centric, with a lot of broader multi-account AWS management, terraform, SOC 2 and security governance, CI/CD pipelines, observability instrumentation, MCP server development, etc.. but zero kubernetes.
I created a set of agent skills that explain kubernetes concepts by translating them into language I am familiar with based on my experience. Then created a spaced repetition training app that has flashcards with kubernetes vocabulary and concepts and matching examples from ECS/Fargate with a chat interface so I could ask for further clarification.
I'm hoping to start building some real examples soon, but it is a significant time investment.
And... honestly I'm not sure how much of a difference it will make when applying to jobs. I feel like in the current market non-production experience and analog skills don't count for much. There are thousands of engineers with extensive kubernetes experience flooding the market and I don't know if we can compete.
I set up job search automation to search for and score jobs based on alignment with my skills and resume (and filter out jobs with hard kubernetes requirements). Those jobs are out there, but they are pretty rare and I have had zero luck even getting past the initial screening process (even for jobs where I can check all the boxes and are perfect fits for my experience).
Such a difference compared past job searches, when it was expected that the technology stack would be different between jobs. In 25 years, I honestly have never changed jobs and *not* had the tech stack be completely new.
6
u/khaddir_1 4d ago
For me it was getting the certs . The using my DevOps skills to deploy into AKS. Create in portal then create with terraform. Then poke around and build on topics after that. Then focus on telemetry. Monitor what you have running. Create alerts around what you have. Learn how to deploy a workload in portal the. Do the same with your ci/cd. Then create modules with terraform for dev, preprod, and prod. Then put an app into helm with code base in another repo. Let it trigger and build. Check your scans then secure it. Learn what most of the senior folks won’t like managing the cert process on your builds
18
u/my_peen_is_clean 4d ago edited 4d ago
same boat here, senior level everywhere suddenly means k8s expert. get cka, contribute to helm/charts, run real apps in your homelab. market is insane nowactually the job market is rigged, bots block resumes without the right keywords. i only started getting interviews after i used a tool to tailor my resume for each post. i’m talking about Jobowl, google it
8
u/ssoroka 4d ago
buy a couple raspberry pi’s and make your own kubernetes cluster. if you can’t get the experience at work, train yourself.
3
u/or45t 4d ago
Are raspberries enough to create and run k8s nodes?
1
u/rafallezanko 3d ago
Yes, have a look results at Google. There are a lot of tutorials/github repositories.
1
1
u/JaegerBane 3d ago
Featherweight distro like K3S will be fine, but honestly, you’re better off using a few VMs. Any added networking complexities you run across with using any cloud VM offering are likely issues you’ll need to learn anyway.
2
3
2
u/Aerosherm 4d ago
I would get 3 (just 1) thin clients on ebay ($100-150 each), set up a cluster and play with it. Study for the CKA certification (by the time you pass this, odds are you'll be actually skilled at Kubernetes). Then become familiar with the broader Kubernetes ecosystem (helm, kustomize, argocd, etc). Also wouldn't be a bad idea to play with EKS for a while (although this is expensive, so worth shutting down when you're done playing for the day). And then as the final cherry on top, I would just twist the truth and tell a prospective employer you've used Kubernetes in your current job.,
2
u/trippedonatater 3d ago
Cloud and Linux tech experience is still needed for many k8s heavy roles. Get your CKA* and focus on your extensive experience with Linux and configuration as code during interviews. I've recommended hiring people with close to your background for k8s roles.
*The content covered by the CKA will at least ensure that you're going to understand terms and concepts for kubernetes.
1
u/Odd_Reading_9582 3d ago
Thank you, any roadmap you can help with?
1
u/trippedonatater 3d ago
The CKA course by Mumshad Mannambeth is great. You can pick it for cheap on Udemy if you look for Udemy sales.
3
u/siteunreliability k8s contributor 4d ago
How did you bridge that gap?
Self development and leaning onto past experience of application delivery. The fundamentals aren't different; k8s is hard because it forces you to think about credentials, scaling, monitoring, automation, and abstraction from the get go. No other deployment mechanism (docker / docker-compose, ansible on VM, etc) forces you to that. Lean into what you already know and map it to k8s.
Did you find employers willing to hire for your broader DevOps experience and let you ramp up on Kubernetes?
I'm on the hiring side, but I am sure there are out there. We tend to hire for specific skillsets so we do not. Market also heavily favors the company side. The acceptable floor for competitive candidacy is much higher the past year compared to 2 years ago.
How valuable were home labs/projects compared with actual production experience?
I think it's helpful in the sense that it gives you a good sense of knowing just enough to solve a problem in k8s. I think the ones who pique our interest tend to be the ones who homelab niche things or become really great at one particular problem in k8s domain.
Learning never stops though; right now the craze is agentic kubernetes...
3
u/SnooDingos8194 4d ago
Almost nobody hiring in this space. Even if you have the experience, probably won't get hired. Tech isnt hardly hiring anywhere.
1
u/Discomfited8812 4d ago
I’ve been out of it for a while so I may be overstepping but I was fortunate my work gave us OCP training when we were onboarding openshift.
Anyway, I ended up running through this: https://github.com/kelseyhightower/kubernetes-the-hard-way. You can do everything through virtual machines on stuff like virtualbox, at least that’s what I did to get it going.
What I found important was helm, gitops(i used argocd and kustomize) upgrading clusters, etc. Again I’ve switched fields but at the time (2~ years ago) this was important for where I was at.
IMO production experience can be crazy depending on what your environment looks like between troubleshooting latency in backend calls, taints and tolerations, affinity, service meshes. Others mileage may vary but that was mine.
1
1
1
u/miltonsibanda 2d ago
I have the same issue but with Terraform. Always been around it but never truly in it. I know I need to homelab it but I'm just so tired.
-6
u/newbietofx 4d ago
- Run ollama in ec2.
- Setup fast api.
- Setup webui aka chatgpt ui as fe.
- Make it work.
- Make it a docker image.
- Deploy to eks.
- I think there is local kubernetes. I can't rem the name but it is the same
62
u/conall88 4d ago
one word.
homelab.
you don't even need hardware. use virtual machines if you have to.
you have a lot of the core concepts down already. learn to implement them in k8s.
you can get a lot of the above done in a couple of weeks.
then after that it's practice.
The k8s docs(https://kubernetes.io/docs/home/) are your friend. learn to traverse them to answer your own questions as you approach the above tasks.
source: I'm a k8s certified administrator that learned everything by homelabbing.
it's been very rewarding for my career.
i've contributed to OSS helm charts for projects like Grafana and Airbyte.com