r/devops • u/FaithlessnessEqual44 • 1d ago
AI content Anyone else seeing AI make DevOps/infra the bottleneck?
I'm curious if other DevOps/platform/SRE teams are running into the same thing my team is.
We're a fairly large environment, mostly EKS, and essentially 100% IaC/Terraform. We also support multiple companies/business units, so while I'd argue our infrastructure is fairly well organized, there's inherently a lot of it and a lot of architecture and context to understand.
Over the last year, our devs have sped up dramatically with AI. The company has leaned heavily into AI-assisted development, reduced developer headcount, and is now pushing toward developers being more "full stack with AI," including having them contribute more of their own infrastructure changes.
In theory, I'm completely in favor of that. I've always wanted developers to be able to own more of the infrastructure surrounding their applications.
In practice, though, it has been kind of a disaster.
We're getting flooded with infrastructure PRs largely written by Claude/other AI tools from developers who don't really understand the infrastructure they're modifying. The Terraform might look plausible, but once you understand the larger system there are frequently significant problems with it.
So instead of reducing the workload on DevOps, it feels like AI has massively increased it.
A huge percentage of our time is now spent reviewing AI-generated Terraform, finding problems, explaining why something won't work, explaining how AWS/EKS/networking/IAM/CI/CD/etc. fit together in our environment, and then going through another iteration of an AI-generated PR.
There's an interesting asymmetry I've noticed too. Our DevOps team is mostly made up of former software developers who moved toward infrastructure, automation, and pipelines. Most of us can jump into application code and be productive pretty quickly, especially with AI helping us. Like,...I feel like (and have some evidence to support) that our small DevOps team could largely take over all of the dev's tasks, but they are falling on their faces trying to deal with ours.
AI seems extremely good at helping someone who understands software write more software. It seems much less capable of allowing someone without infrastructure experience to suddenly understand a large production environment.
The complaint we're increasingly hearing is basically: "We can't successfully do full-stack development with AI because the infrastructure is too complicated."
And maybe they're right, but before AI, I would have said that this company is the most organized and best architected I've ever been at. I mean....100% IaC has never been something I've experienced, and it's very rare that we hit a use case brought up by one of the several companies where we don't already have a set of generalized modules that can't support it.
Our environment is complex, but a lot of that complexity isn't accidental. We have a large organization, multiple companies we deploy for, Kubernetes, networking, security requirements, IAM, CI/CD, observability, etc. You can't abstract away the fact that these things exist. And we're already 100% Terraform/IaC, which I would have thought would make this considerably easier for AI to reason about than an environment full of manually configured infrastructure.
The strangest part is the staffing effect.
AI allowed the organization to reduce software engineering headcount because individual developers became more productive. But now those remaining developers can generate changes so quickly that our DevOps team is completely overwhelmed trying to support and review them.
It genuinely feels like we could double the size of the DevOps team right now and still have plenty of work. We are working on an AI assistant that can help the devs deploy to our environment more effectively, but we're having a hard time finding time to work on it because we're constantly helping the devs.
I'm starting to wonder whether this is going to be a broader consequence of AI-assisted development: AI increases the rate at which software can be produced much faster than it increases the rate at which infrastructure/platform teams can safely absorb changes.
For those of you working in DevOps/platform/SRE at companies heavily adopting AI:
Are you seeing this too?
And if you are, how are you handling it?
Have you increased platform/DevOps staffing? Built better abstractions or internal developer platforms? Given developers more direct infrastructure ownership? Put stricter boundaries around what application teams can modify? Found ways of giving AI enough context about your infrastructure that it actually produces good changes?
Or has AI actually reduced your infrastructure workload, and we're doing something wrong?
20
u/twistdafterdark DevOps 1d ago edited 23h ago
Do their agents have access to accurate and sufficient context regarding your entire infrastructure? If not then the agents are likely making all the wrong assumptions for the PRs.
edit: fixed typos
2
u/InvisibleTextArea 22h ago
A read only MCP server into Terraform would seem like the best way to solve this problem?
1
u/HokageNiya 3h ago
I think this is the key distinction. A good CLAUDE.md gives the agent instructions, but that’s not the same as giving it a current model of the environment. Ownership, dependencies, deployment state, policies, permissions, incidents, approved workflows etc. are all changing context that’s hard to keep in static files. I've seen that Port has been calling this a Context Lake, which is basically a shared engineering context layer that agents can query rather than trying to stuff the whole organization into the prompt, which sounded very interesting, since it feels like infra is one of the clearest places where repo context alone just isn’t enough.
1
u/FaithlessnessEqual44 1d ago
Yeah, we have really great claude files that keep a load of context. I've tried to get the devs to be less specific about how they want something done, and really just tell it what the end goal is. But most of them don't know enough to ask correctly. But maybe looking into how to make the context better.....or at least handle overly specific, but incorrect requests, better is a route we should go down.
10
u/HeligKo 1d ago
AI effectiveness is ultimately about context. For most programming and software development tasks there is a ton that it could have been trained on, and the problems being solved often only differ in context. Now here comes infrastructure, and even though the engineering principles are largely the same, the actual implementations vary wildly. If the model doesn't have a way to understand the big picture and the details of your environment, then it will do it's best based on the code it is working with. As a platform engineer, I call this job security. AI makes my job easier, but it needs me.
38
u/SuperQue 1d ago
We're a fairly large environment, mostly EKS, and essentially 100% IaC/Terraform
Seems ok.
And we're already 100% Terraform/IaC
I think there might be your problem. Over the past few years we've been massively shrinking our TF footprint.
Our primary developer infrastructure interface is Kubernetes. So we've been replacing anything where a dev needs to do TF with a Kubernetes CRD. We've wrapped the infra in such a way that there's only a few knobs to turn.
This allows things to be a lot more self-service. Infra doesn't even get PRs for a lot of this and we spend more of our time improving the underlying self-service nature of the system.
7
u/FaithlessnessEqual44 1d ago
I'll look into that more. We have attempted to make the creation of resources easier for the devs by creating our custom modules that only take the input that they'd need. But you're right, and that's an extra step they wouldn't need to take if we did this. Thanks for the input.
9
2
u/Klafka612 1d ago
Are you using the operators to deploy resources related to the cluster or is this more cluster as control plane like crossplane gcp cloud config connector?
I was having this thought earlier today about when operators vs other abstractions make sense for interrelated kube/cloud resource/dep management.
1
u/SuperQue 1d ago
It's a bit of both. Some crossplane, some custom operators.
For example we wrote an opinionated ServiceAutoScaler operator that wraps Keda ScaledObjects so that the team doesn't have to deal directly with all the various knobs, Prometheus queries, Kafka topic depth, etc. They just put a "I would like an auto-scale, kthxbye" in their service deploy config and the automation does the rest. There are always escape hatches for more advanced needs, so they could directly add a Keda config if they wanted. But we try and solve the 90% case with the easy-to-use objects.
At the end of the day, what matters is that the individual service teams get a simple declarative interface.
The end goal is that to the dev team it feels more like a FaaS for "effortless deploy", but we can scale much more for a lot less cost.
1
u/Klafka612 1d ago
Oh keda is interesting. We've done some similar things with sveltos as well. Some food for thought.
1
u/dariusbiggs 1d ago
How are/would you approach it where the infrastructure needs to outlive the Kubernetes workloads.
Things with 7 year retention and governance requirements.
For prod you have those requirements, but for dev, stage, and test, you don't.
1
u/SuperQue 1d ago
Backups?
If the service is gone, all you have is the at-rest data. Back it up to object storage with an object policy.
1
u/dariusbiggs 23h ago
The resource itself (the object store) itself is the thing that cannot be destroyed for 7 years due to the governance locks and retention requirements. So if the object store was created with the workload it would die with the removal of the workload, which would be prevented due to the controls and thus outlive the k8s workload.
So it would need to be managed and created outside of the workload that uses it. Which makes it more complex to manage with a delf service approach.
3
u/SuperQue 23h ago
You make that sound like it's an impossible requirement.
- You can design controllers to allow abandoned resources.
- You can design resources to allow migration of adoption.
- You can design backup systems such that the storage claim is owned by the backup controller / governance and not the service. The service only owns the access control claim to write backups.
It's not really that complicated. It's not some "gotcha" that you're trying to insinuate.
1
1
u/nomadProgrammer 11h ago
have you considered Crossplane?
I also 100% believe IaC shouldn't manage most k8s resource, otherwise it gets extremely slow
17
u/killz111 1d ago
It's called slop. I'll bet even when the changes are decent the developers and Claude just fail to account for dependencies on apply so you get random weird failures.
Dont get me started on resources in the cloud that never delete so a recreate needs an import.
I'm dealing with the same thing where a senior software engineer is proactive in creating tf but then constantly asks for help on shit they wrote that doesn't apply properly.
You have some choices. If the organisation wants more slop, you have to implement some basic guarails that protect your stateful resources. Scanning tf output with AI then blocking on state full resource delete is an example.
Everything else just let it through but make sure to assign cause of any breakages and track those. I call it blame because what else can you do when people throw shit at environments that they dont understand?
10
u/vacant_gonzo 1d ago
but the answer must be more AI, surely? AI speeding up X means more AI use in Y to cope. Means more AI use in Z to cope…
3
u/killz111 23h ago
Yeah i dont think there's an alternative. Until something seriously breaks and they institute mandatory human reviews like Amazon did.
3
u/vacant_gonzo 23h ago
I’m fortunately at a place that requires human review, would hate to lose that
2
u/killz111 22h ago
I think the scary parts is a lot of companies have implicit requirement for human review but due to the sheer volume and size of PRs people just wave them through.
2
u/forever-butlerian Solaris 8 Enjoyer 6h ago
Of course. The way to deal with the negative side effects of crystal meth is more crystal meth.
3
u/DevWorkflowBuilder 1d ago
we stopped reviewing the terraform hunks. PR has to drop a blast-radius.txt: created/changed/destroyed, who pages if it dies, terraform plan attached. over 40 resources changed means a second reviewer. how many of those plausible PRs would have died on a missing plan vs still needing someone who knows the cluster?
3
u/rofolo_189 23h ago
Absolutly, DevOps and Infra is our current bottleneck and it was that before, but now it's just extreme. That tells you something: DevOps is important and pretty safe.
1
u/Cpt_Chaos_ 23h ago
Same here. AI boosted the number of PRs being created, now the bottleneck is CI and reviews. That was a problem already in the past, but now it's much worse.
2
u/aragossa 20h ago
the CRD/self-service route people are describing is the right long-term call. it's also a multi-quarter platform build, so it doesn't do anything for this week's PR pile. a policy-as-code gate in CI, checkov or tfsec or OPA/conftest against the plan JSON, catches a lot of the structural garbage on its own, wildcard IAM roles or something sitting there unencrypted that shouldn't be, without needing any of the environment context a human reviewer or the AI writing the PR would actually have to have. it won't get you the architectural correctness DevWorkflowBuilder's blast-radius check is going after, but I'd guess a decent chunk of what's flooding review right now just fails one of those checks automatically and never makes it to a person.
2
u/Lachtheblock 19h ago
See your problem is that you care about quality. Maintainability and uptime are what are holding you back. Once you decide to just be bad at your job, things will be much easier... for a while...
I'm an application developer, and the slop can roll in, but only when the authors also don't care. I can pretty much guarantee that the slop infra PRs are of the same quality as their application PRs. It's just that to unwind and bug fix infra is far more painful.
1
u/forever-butlerian Solaris 8 Enjoyer 6h ago edited 5h ago
One of the differences between app developers and operations is a care-free attitude to quality gets developers promoted and operations fired.
Also you, generally speaking, cannot fix borken infrastructure with more infrastructure.
2
u/AminAstaneh 14h ago
I actually did a talk about this very subject:
https://www.youtube.com/watch?v=eGsBawFwH90
Thesis: writing code used to be the constraint in delivering software. Thanks to LLMs, the constraint has moved to its operation.
1
u/FaithlessnessEqual44 6h ago
Excellent talk and a lot of great points, thanks for sharing. I've sent it to a few team members as well.
2
1
u/scrambledhelix making sashimi of your vpc 23h ago
Thanks for the post, I feel like I haven't seen this specific point addressed too much, and it was definitely a trend starting at my last SRE gig.
1
u/Pretend_Listen 23h ago
Yes, your describing our infra org. Trying to make all the things devs touch dummy proof. Any complex tf requires review.
Been creating several CRD interfaces to help with this
1
u/iotmos 21h ago
I share the same experience , Dev Team velocity exploded , but the main issue is defining boundaries and paved roads for infrastructure so LLM generates code within these boundaries , Infrastructure's context is difficult to put in every repository of every app , but building starters , infrastructure schemas that can be adopted directly by dev teams help tremendously.
1
1
u/RevolutionaryElk7446 18h ago
We had deterministic automation and it ran well
We introduced LLMs and it went bad, for 5 years we tried to meet or exceed the prior setup.
We moved back to primarily deterministic automation. CEO still wants us using AI so it look at things like Change Requests now.
1
u/Solitairee 18h ago
Our engineers manage the infrastructure themselves. We've trained them up to be full stack and we provide the tools for them to understand the infra
1
u/CartoonistStriking62 17h ago
Are you using a framework around Terraform, or mostly vanilla Terraform? Do you have a comprehensive CLAUDE.md that defines your conventions, guardrails, and parameters, along with opinionated SKILL.md files for common workflows?
We use Atmos as an orchestration layer for Terraform, Helm, and Packer, and it’s been working pretty well for us. We’ve standardized the workflows enough that developers can self-service a lot of infrastructure changes without needing the infrastructure team to walk them through every step.
1
u/derprondo 17h ago
Like others, we solved this many moons ago by segregating every project into its own separate cloud provider account, and forcing the project owners to operate in a self-service manner, ie they own their own terraform and maintain it themselves. We provide the platform and the pipelines for them to use, as well as the security governance to keep them from making egregious mistakes.
1
u/ArtitusDev 16h ago
Use ai to write iac
2
u/forever-butlerian Solaris 8 Enjoyer 6h ago
The only successful way to do IaC is to have as little of it as possible. The "AI" approach is just to jam a chopstick in your ear, stir up your brain, and open the slophose.
1
u/SamfromLucidSoftware 15h ago
One thing you can do in the short term is limit how much damage a bad PR can do instead of trying to keep up with review volume. Things like tighter module boundaries, stricter automated policy checks that catch AI generated mistakes, being clear with devs about which parts of the infrastructure they can touch and which require DevOps involvement. Basically, the goal is filtering the queue before it reaches the team.
But yes, the bottleneck is shifting and in your case, the PR review bottleneck is a predictable outcome of the headcount. If you reduce engineering headcount because of individual output increases, but that output still has to flow through the same review and validation, then you’ve just shifted it downstream and concentrated it there. The AI assistant you’re building sounds like the long-term answer here.
1
u/Fantastic-Mr-Default 14h ago
The flood is not a headcount problem. It is an interface problem.
Stop reviewing raw Terraform from agents. Ship a paved path for the 90% case (module, CRD, or schema) and reject anything outside it with policy-as-code before a human opens the PR.
Office hours and a plan-summary bot for the escape hatch. Reviewers should not be the parser.
1
1
u/DolGuldurWraith 13h ago
Ideally devops/infra should also speed up with the use of AI if ur Setup was planned better before the whole AI thing.
If ur setup is planned in such a way to it scales easily then AI helps u bring up infra much faster and match speed with Development.
e.g. in our case we creating a new env with all infra in 2 days compared to 2 weeks earlier.
Claude create all the files needed in 2-3 hrs and then its all just terraform apply with 1 hr per env and we have 4 env with eks, msk and opensearch taking most time to get deployed
1
u/bendem 12h ago
I'm battling this with a set of standards enforced with specific modules. You have this stack, this module deploys it, if your stack doesn't fit, modify your stack.
We work closely with the library guy that produces the standard java libraries which is mostly just a bunch of spring boot starters. Those spring boot starters contain everything to preconfigure SSO, logging, API security, etc so applications are already standardized when they arrive our way.
I'm also writing a bunch of skill currently so when a dev wants to write some infra config, the skill takes the wheel instead of them asking the most convoluted things.
1
u/Purple_Mo 12h ago
Not DevOps But other areas we can use ai - others not by policy The dynamic is crazy
1
u/Fentrax 11h ago
Just make the same seam between dev and infra in agentic and general ai use. Your assistant DOES know that nuance, and theirs doesn't need to. Your side will counsel on best practices for infrastructure, etc etc. register compatible requests and process them, educate and offer alternatives that do conform instead to their bot, who in turn educates the originator.
You people are all making this transition too hard. If your business already has departments, then agents should be specialized within the department. Business as a whole owns agent to agent policy and protocols, just like it does with business process and humans.
Then, you can start deploying specialty moe variants for cross department workflows... RAG, documentation files, memory, attribution, etc is all part of it. AI doesn't magically change everything without an intentional use driving it that way.
Definitely bake in executive layers too. Eventually, the board just queries the CEO bot, and the entire swarm of bots coordinates underneath that at diminishing levels of scope.. some poor sod will have 23% of their weekly/daily token allowance, and have to figure out how much they will do by hand vs their bot...
1
u/codemochi 11h ago
I've seen this too. Part of the problem is that AI enables developers to be more productive by supercharging them with general knowledge, while infra review runs on context that's built up after managing a number of deployments and getting a feel for what to expect and which changes are typical churn versus big deals. When you read through the HCL files you have to rebuild "how does this change fit with our environment?" in your memory. This can get crazy when managing many stacks though.
What helped was generating a terraform plan file and then comparing it to older copies of that plan for that stack. Then you still are building up muscle memory that "Stack A is always noisier because of reasons X,Y,Z" and "Stack B is quiet because it is smaller and not used much" but at least there's a more quantitative lens you can apply.
You can do this cheaply by posting add/changes/destroys/replaces summary in the PR commit and add auto-labels if numbers are above a fixed value.
1
u/See-9 11h ago
Devs and AI basically require a platform capable of a Containers as a Service model. And even then, you still have some issues, unless you really set your permissions up front for where the app registrations go - but yeah. I’d you’re doing VMs or managing stuff outside GitOps - good fuckin luck.
1
u/djbp 8h ago
Yeah, I'm seeing this too. AI speeds up the planning. Or it generates a ton of new ideas for infra changes. But then the actual execution still hits the same manual steps. Or the same approval processes. That's where things slow down. We're building https://snapflow.online/ for this exact problem. It helps teams automate those changes safely. You plan it, it runs in a container, then verifies it. And you still get human approval for the risky stuff. It really helps keep things moving without breaking anything. Anyone interested piloting SnapFlow drop me a line.
1
u/modern_medicine_isnt 4h ago
Create an AI agent with knowledge of the entire infrastructure (just tell it to read it and create context). Then "teach" it to review the prs. Tell them they have to pass it's review before you review it.
1
u/Exotic_eminence 23h ago
I’m a platform SRE and I been out of work since AI came out- is yall hiring or na?
0
-3
u/no_ragrats 22h ago
I'm an app dev by trade, but also dabble in the devops, design, architecture and have focused on resolving bottlenecks at times. Take this with a grain of salt, but here's what my strategy would look like.
- Steal back your time - most of your time should not act as a help desk.
- Ensure all work hits a prioritized queue and reduce direct message interruptions. If everything has to originate from a proper queued channel, then you'll have more control over your schedule. Prioritize the easy stuff and reserve a regular 'office-hours' time to help with the complex stuff.
- Make sure your team allocates time towards solving the problem rather than simply trying to keep up with the increased load. Ensure a base 20% of your teams effective work hours are spent on increasing effeciency and reducing overhead - ideally one or more people working on it full time rather than having everyone doing an 80/20 split for instance.
- Start gathering some data on how much time you are spending on answering the same questions. Rank them by quantity and time spent.
- Create some developer-oriented documentation. A "Getting Started" guide with a working template, a general FAQ document, and handful of 'how to' articles that address some of the more advanced/complex topics that you have found to be reoccurring time sinks based on the data you accumulated in the previous step. Now when devs start complaining about the time it takes their issue to get addressed, you can point them to your new self help documentation hub while they are waiting for their spot in the queue to pop-up.
- Add some extra requirements to your PRs: Every infra PR must attach 'terraform plan' output and a written "what this changes and why." Auto-bounce PRs missing either. Provide an example of an adequate description and if they dont provide an adequate one, ask them to update the description with a brief summary of why its not adequate. This alone will filter a lot of slop and forces the dev to read the plan and understand what was generated.
Ensure a couple basic guardrails to ensure disaster avoidance, reducing time spent on disaster recovery. Ie. prevent_destroy lifecycle to all stateful resources (RDS, S3, EBS volumes, stateful sets) that requires a PR with a well defined reason and appropriate manager approval.
Add additional reviewer Time-savers, ie:
Create/Use a plan summary bot to summarize potentially major changes, including plan diff + descriptive flags ("ALERT: this replaces a database", "ALERT: creates IAM roles")
Require approval above a cost threshold delta
Add additional CI guardrails that prevent a large portion of common AI/user pitfalls, ie:
Reject: public S3, 0.0.0.0/0 SGs, wildcard IAM, unpinned providers, resources outside an allowlist (Policy as code)
hard fail all delete/replace on a stateful resource unless a platform member adds an explicit label
Drift detection (ie. someone manually changed something which isnt defined in code) requiring individual determination/resolution before current changes are promoted
Create a streamlined interface and build platform for common requests. Rather than exposing raw Terraform and the full cloud API, give developers a small, validated service spec that describes what they need (a service, a database, a queue, an autoscaling range, a route) and let the platform own how it's built underneath. Result: Drastic reduction in PRs, more defined constraints, and less variability.
Start with the business case. Categorize all PRs in the past couple months into a few key topics: ie. new service, add managed datastore (RDS/Elasticache), change resources/autoscaling, add IAM permission or ingress route. The top few will likely account for the majority of time spent; pick out the few that will account for ~80% of your time spent.
Calculate the time spent on those topics (review-hours per infra PR, iterations-to-merge, infra lead time, incidents caused by dev infra changes).
Write a 1 pager business case: "Each AI infra PR costs us X senior-hours. Y weeks of protected capacity builds the self-service layer that removes ~80%, which can ensure time is quickly available for the complex ~20%, increasing the rate at which developers can optimize infrastructure and reduce errors caused by a lack of architectural domain knowledge." The object of the business case here is to carve out and prioritize the time your team needs to create/test/promote the new streamlined interface.
Either Kubernetes CRDs + operators / Crossplane compositions (keeps Terraform out of the dev loop entirely, or a schema-validated spec that a platform-owned generator turns into Terraform
Keep the option for the standard review process for the remaining edge cases that dont really match well for a streamlined interface.
Implement it starting with a couple competent dev teams to iron out kinks before pushing for major adoption.
Enjoy the time savings to work on more important stuff
5
u/aspitzer 22h ago
Thanks AI.
1
u/no_ragrats 21h ago
Helped make a couple of the lists/examples more thorough, but otherwise no. I do wish 30 minutes of my time wasnt wasted to simply get disregarded as AI, though.
I'll leave it in case its helpful nonetheless.
1
u/FaithlessnessEqual44 5h ago
Note I also had at least one person accuse my original post as AI, when I wrote every bit of it. But I'm very skeptical of everyone now. Who's real? Are any of us?
0
128
u/azjunglist05 1d ago
We fortunately solved this before LLMs came into the equation where I work. We accomplished it through opinionated, self-service IaC. Everything our developers use to build infrastructure is wrapped up in a well defined interface that platform teams developed with very strict rules and design philosophies that ensures truly declarative IaC.
Our developers have been deploying infra entirely on their own for a couple of years now. We only get involved to troubleshoot what are most commonly user errors. Once the developers had a Swiss Army knife for all infra they were more compelled to make it better than build their own solutions