trying to find a ref architecture that is widely recommended for deploying windows runners on k8s. From what I read ARC doesn't support it still? if not what are my options?

Hey everyone,

When I learnt kubernetes and completed CKA, I didn't instantly get a chance to work on a real project. Later, i got a job and learnt a lot, but I felt that there was a need for a tool where we can simulate real world scenarios, we can learn what kind of scenarios and incidents come in real environments.

For this purpose, i built a tool which does exactly the same. You can simulate real world scenarios, use kubernetes tools like opencost, keda, grafana, argocd, traefik. Learn how you can use it and find out different commands to help you learn.

Do check this out if you are interested in this.

I recently set up a small 3-node Kubernetes homelab using Dell OptiPlex Micro PCs running Talos Linux.

The main goal was to have a local cluster where I can learn, experiment, break things, and test tools without relying on cloud infrastructure every time. I’m planning to use it for Kubernetes networking, storage, observability, GitOps, security, upgrades, and general experimentation.

Small setup, but already a very useful playground.

Curious what others here are running for their Kubernetes homelabs.

After I first set up a deployment pipeline, I would simply drop DB passwords and API keys into GitHub Secrets and feel completely safe because they are encrypted. I recently went through a security breakdown on GitHub Actions that showed me that it could be a mistake to think that way.

The main issue is that an encrypted secret is still a static, long-lived target. Once the workflow finishes, that credential stays active indefinitely. In the breakdown, I saw a few default behaviors attackers look for, like teams forgetting to revoke access after a job runs or lacking the audit logs to even know when a key was used.

The proposed fix is shifting to dynamic orchestration where the pipeline generates a short-lived token at runtime and revokes it the second the deployment finishes.

If you're writing deployment workflows, how do you handle this, do you just use GitHub's default storage, or are you injecting temporarily credentials to avoid leaving static keys exposed?

i'm working on a decision-making problem where an agent gets a possible leaked api key and has to decide whether it's actually a prod credential and then either ignore, verify or remove/rotate it.

right now i'm using repo context (file path, variable name, surrounding code), key format, git history, whether the code is active/reachable, deployment context, and last_used_at from the provider's admin api.

the problem i'm running into is that most of these signals can tell me "this probably is a real secret", but don't tell me much about whether it's still live or already revoked.

for people who've dealt with secret leaks in actual devops workflows, what do you normally check before deciding what to do with a flagged key?

i'm especially interested in evidence you can get without actually authenticating with the discovered key.

disclosure: this is my own personal project, not a commercial/product promotion. just looking for technical feedback.

I have someone pushing a colleague to an Openshift virtualization position. We do multi cloud deployments, GHA heavily, ephemeral integration environments, etc. Not seeing the intersection with this guys skills or desires. Am I missing something?

I recently completed the full Kubestronaut certification path after roughly three months of focused preparation.

After completing it, quite a few people reached out asking about the order I followed, the resources I used, and how I prepared for each exam, so I decided to put everything together in one detailed blog rather than answering the same questions separately.

For KCNA and KCSA, my preparation was fairly straightforward and mainly consisted of the KodeKloud courses, KodeKloud notes, and practice tests, while CKAD, CKA, and especially CKS required much more hands-on practice with labs, mock exams, and repeated work on weaker areas.

I’ve also included links to my dedicated CKAD, CKA, and CKS exam-experience posts for anyone who wants a deeper breakdown of those exams.

Full 3-month Kubestronaut journey:
https://medium.com/@prateekjain.dev/my-3-month-journey-to-becoming-a-kubestronaut-c722c4a7cf75?sk=eb78b3ef703262f787f746cc6969d8f1

Hopefully this helps anyone currently working towards the Kubestronaut path. Happy to answer questions about the preparation or any of the five exams.

Say I want to establish a company or an organization's representation on the web quickly. The 'stack' involves domain, email in/out, public-facing website, (optionally) social media presence.

Are there any products out there that unify the 'stack' and allow provisioning from minimal amount of steps? Or do I need to set up each item through separate providers?

Ideal if the solution has an MCP or cli.

Thanks ❤️

I'm curious if other DevOps/platform/SRE teams are running into the same thing my team is.

We're a fairly large environment, mostly EKS, and essentially 100% IaC/Terraform. We also support multiple companies/business units, so while I'd argue our infrastructure is fairly well organized, there's inherently a lot of it and a lot of architecture and context to understand.

Over the last year, our devs have sped up dramatically with AI. The company has leaned heavily into AI-assisted development, reduced developer headcount, and is now pushing toward developers being more "full stack with AI," including having them contribute more of their own infrastructure changes.

In theory, I'm completely in favor of that. I've always wanted developers to be able to own more of the infrastructure surrounding their applications.

In practice, though, it has been kind of a disaster.

We're getting flooded with infrastructure PRs largely written by Claude/other AI tools from developers who don't really understand the infrastructure they're modifying. The Terraform might look plausible, but once you understand the larger system there are frequently significant problems with it.

So instead of reducing the workload on DevOps, it feels like AI has massively increased it.

A huge percentage of our time is now spent reviewing AI-generated Terraform, finding problems, explaining why something won't work, explaining how AWS/EKS/networking/IAM/CI/CD/etc. fit together in our environment, and then going through another iteration of an AI-generated PR.

There's an interesting asymmetry I've noticed too. Our DevOps team is mostly made up of former software developers who moved toward infrastructure, automation, and pipelines. Most of us can jump into application code and be productive pretty quickly, especially with AI helping us. Like,...I feel like (and have some evidence to support) that our small DevOps team could largely take over all of the dev's tasks, but they are falling on their faces trying to deal with ours.

AI seems extremely good at helping someone who understands software write more software. It seems much less capable of allowing someone without infrastructure experience to suddenly understand a large production environment.

The complaint we're increasingly hearing is basically: "We can't successfully do full-stack development with AI because the infrastructure is too complicated."

And maybe they're right, but before AI, I would have said that this company is the most organized and best architected I've ever been at. I mean....100% IaC has never been something I've experienced, and it's very rare that we hit a use case brought up by one of the several companies where we don't already have a set of generalized modules that can't support it.

Our environment is complex, but a lot of that complexity isn't accidental. We have a large organization, multiple companies we deploy for, Kubernetes, networking, security requirements, IAM, CI/CD, observability, etc. You can't abstract away the fact that these things exist. And we're already 100% Terraform/IaC, which I would have thought would make this considerably easier for AI to reason about than an environment full of manually configured infrastructure.

The strangest part is the staffing effect.

AI allowed the organization to reduce software engineering headcount because individual developers became more productive. But now those remaining developers can generate changes so quickly that our DevOps team is completely overwhelmed trying to support and review them.

It genuinely feels like we could double the size of the DevOps team right now and still have plenty of work. We are working on an AI assistant that can help the devs deploy to our environment more effectively, but we're having a hard time finding time to work on it because we're constantly helping the devs.

I'm starting to wonder whether this is going to be a broader consequence of AI-assisted development: AI increases the rate at which software can be produced much faster than it increases the rate at which infrastructure/platform teams can safely absorb changes.

For those of you working in DevOps/platform/SRE at companies heavily adopting AI:

Are you seeing this too?

And if you are, how are you handling it?

Have you increased platform/DevOps staffing? Built better abstractions or internal developer platforms? Given developers more direct infrastructure ownership? Put stricter boundaries around what application teams can modify? Found ways of giving AI enough context about your infrastructure that it actually produces good changes?

Or has AI actually reduced your infrastructure workload, and we're doing something wrong?

I genuinely think DevOps or platform engineering is the area of software I enjoy the most.

I like CI/CD, Terraform, cloud infrastructure, debugging weird deployment problems, trying to understand why systems fail, automating repetitive things, and generally having ownership instead of just implementing another CRUD endpoint.

I’m currently a working student in an SRE/platform team in a big company in Germany. I’ve already worked on things like services from Cloud Build, GitHub Actions, Terraform, Cloud Run, deployment alerts, state migrations and fixing random infrastructure problems that come up along the way.

And the funny thing is that the more I learn, the more I like it.

But looking for a junior position is becoming pretty frustrating:

A lot of “Junior DevOps” jobs seem to expect Kubernetes production experience, several cloud providers, Terraform, Ansible, networking, Linux, CI/CD, monitoring, security and somehow 2–3 years of professional experience with all of them.

Then there are actual entry level positions, but many of them seem to basically be IT support with “cloud” or “DevOps” in the title.

I know I still have a huge amount to learn. I don’t expect someone to give me a production Kubernetes cluster on day one and say good luck. I actually want to be around experienced engineers, get challenged, make mistakes and slowly become someone who can be trusted with serious systems.

My goal isn’t to job hop every six months either. I would genuinely like to find a team where I can stay for several years and become really good at this.

But sometimes I wonder how exactly companies expect junior DevOps engineers to become experienced DevOps engineers if almost everyone wants the experience before giving you the opportunity to get it.

For people who are already working in DevOps/SRE/platform engineering: how did you actually get your first proper role?

Did you already know most of the stack, or did somebody simply take a chance on you and let you learn?

Edit: I worded the sysadmin part badly. I don’t think sysadmin work is beneath me at all. I just want to move toward automation, infrastructure and software rather than mostly ticket-based support.

Edit 2: I respect that some of you suggest starting in help desk or sysadmin. But my long term goal is to move into an SRE role, ideally something closer to how Google approaches SRE. I read the SRE book and really liked the idea of treating operations as a software engineering problem, with automation, reliability, monitoring and reducing repetitive manual work. That is the direction I want to build toward.

were on 50+ microservices on gcp, still doing our own deploy tooling. keep hearing gitops is the way and honestly cant tell if thats real or just the current hype cycle.

not looking for a sales pitch, more curious what broke for you after the demo phase. drift detection, secrets, rollback under load, whatever. did it actually reduce incidents or just move the pain somewhere else

what would you tell yourself before adopting it

Hey everyone,
Is anyone here currently working in AWS Data Center Operations (DCO) in the Frankfurt region (or Germany) or familiar with their technical screening process?

I recently completed a 60-minute technical phone screen with an engineer for an AWS DCO / IT Support role in Germany. During the technical portion, I answered all hardware, networking, and cabling troubleshooting questions without getting stuck. At the end, the interviewer explicitly told me: "The technical part was pretty good, that's what I can tell you."

For the behavioral part, I answered four questions using the STAR method. In the closing feedback, the interviewer noted that I sounded a bit nervous and strictly said the literal STAR words out loud ("The situation was...", "My task was...", "The action was...", "The result was..."), though he mentioned that if my recruiter instructed me to use that exact structure, it doesn't matter.

For anyone who has been through this specific Frankfurt/EMEA pipeline or interviews for AWS DCO: does sounding nervous or explicitly vocalizing STAR labels carry a negative impact if the technical answers and core story data points were solid? What are the realistic odds of moving forward to the onsite loop from here?

Would really appreciate any insights from anyone with AWS DCO experience in Frankfurt or Germany. Thanks!

AI tools like Claude and Codex have made it much faster to write and modify code.
But I'm curious about what teams are seeing on the back-end side of that.
More code potentially means more commits and ultimately more deployments.
For teams where AI-assisted development is already heavily used:
How has this changed your CI/CD workload?
Are you:

running significantly more pipelines?
increasing runner capacity or parallelism?
changing how tests are triggered?
batching changes differently?
deploying more frequently?
seeing CI or testing become a new bottleneck?

The question I'm trying to understand is:
If AI dramatically increases how fast we produce code, how are teams scaling the infrastructure required to validate and deploy it?

Would be interested in hearing what people are actually seeing in production, especially from teams with relatively high commit or deployment volume.

An interesting article with Go examples on how to proxy and intercept database queries. Multiple use cases can come out of that.

I am seeking information regarding parameters for GPU monitoring for an internal tool. Specifically, I would like to identify the key metrics for monitoring GPUs and their associated agents, including relevant queries and performance indicators. Additionally, I am interested in exploring any open-source tools designed for this specific purpose.

[Not self-promo - genuinely looking for input/feedback here]

I think this is something a lot of companies deal with, not just the big ones. Outages in Azure and AWS happen regularly. Say your blob storage in a specific region goes down and your team isn't around, do you have anything automated to spin up a replacement resource in another region, or even another cloud provider if it's a platform-wide issue?

I know provisioning the resource is only part of the problem (some stateless resources can just pull their definitions from a registry and redeploy, but let's keep the scope to that for now, data replication is a whole separate can of worms).

You can automate a good chunk of this with Azure Monitor, for example, but then your actual infrastructure drifts from what's in your IaC repo, and you're back to a two-source-of-truth problem. (Happy to hear from anyone with real experience doing that.)

Another thing I keep thinking about: adjusting resource attributes (SKU, instance size, etc.) based on logs/events, like a massive traffic spike on an App Service, or the opposite: nobody's using it and you're paying for nothing.

So here's something I've been thinking about: a GitHub Action where you define condition-action rules (including recovery conditions, if you want, to roll back once things return to normal) directly in your Terraform IaC repo. You write your rules, run the action on a schedule (every 5 min, or whatever), and it checks each rule, a resource being down, or a KQL query against a Log Analytics workspace returning something you defined as a violation. If a rule matches, it modifies your Terraform code accordingly, either opens a PR or pushes directly to main (which I suspect most teams would never actually want, for good reason). Either way, your existing apply pipeline picks it up and runs like normal, no separate deploy mechanism, no new secrets to manage centrally, no SaaS.

Full transparency: I haven't worked at a company with the scale/complexity that actually needs this kind of multi-region, multi-cloud resilience, so I'd genuinely like to hear from people who have.

Curious what you all think:

• Is this solving a real problem for you, or are native tools (Autoscale, Resource Health alerts, etc.) already good enough for your use case?
• Would you ever trust automated infra changes without a PR review, or is that a hard no for you? (Sounds like a stupid question at first, but keep in mind you'd define the exact changes yourself, there's no AI/magic auto-generation involved. Maybe you'd let small, low-risk changes apply automatically but require review for anything bigger?)
• Anyone tried something similar and hit a wall I should know about?

Thanks!

Hi, I'm trying to setup a basic ha postgres cluster using the Spilos images in a docker swarm setup, but every few days the thing crumbles down with DNS resolution issues, timeouts, wal and etcd records corruption, I don't have the money to rely on an hosted solution rn, so has anybody run into these problems and solved them?. This makes me understand why the DBA role exists, but it is so frustrating and absurd that it is not a solved problem for something that feels so relatively trivial to setup in mariadb with galera.

I've heard the "engineering doesn't care about cost" problem come up quite a bit, but I'm starting to wonder if that's actually the right diagnosis.

If an engineer can see latency, errors, CPU, memory and availability for their service, but has no idea whether that service costs $500 or $50,000 a month, can we really expect cost to factor into their decisions?

What have teams done to give engineers enough cost visibility to make better decisions without turning every engineer into a FinOps specialist?

It is called Microsoft Certified: Cloud and AI Security Engineer Associate

For those who have done it, what do you think? Did you enjoy it? Was there something you disliked about it?

I'm currently working as an SRE while continuing my studies, and I'm still at the early stage of my career.

I live and work in an Asian country, and I'm trying to learn more about how DevOps/SRE work is done in different countries and companies. Rather than just learning from courses and documentation, I'd really like to connect with people who are already working in the field and learn from their real experiences.

I'm mainly hoping to build a network of people in the DevOps/SRE community and have conversations about things like:

• How did you start your DevOps/SRE career?
• What does your day-to-day work look like?
• What skills or areas did you improve the most as you gained experience?
• What do you wish you had learned earlier in your career?
• How do DevOps/SRE practices differ between companies or countries?
• What technologies or practices are becoming more important in your work?
• How important are communication and teamwork in your day-to-day role?
• What advice would you give to someone who is still early in their career?

I'm not looking for job offers or referrals. My main goal is networking, learning from other engineers, and understanding where I can improve.

I'd be happy to connect with people from different countries and backgrounds, whether you're an experienced engineer or you're also early in your career.

If you're open to a casual chat about DevOps/SRE, technology, career experiences, or even just exchanging ideas, feel free to comment or message me.

I'd really like to build some genuine connections in the DevOps/SRE community and learn together.

Thanks!

Note: Sorry about the title/heading. I can't change it after posting. By “abroad,” I mean countries outside my home country in Asia. I’m mainly interested in connecting with people from different countries and learning from their DevOps/SRE experiences.

Posting this both to ask if anyone else in **me-riyadh-1** is hitting it, and so the next person googling "OCI curl timeout Riyadh" finds something useful.

**Setup:** Ubuntu compute instance, public subnet, default route → Internet Gateway, stateful egress allow-all, VCN resolver for DNS. Running a Laravel app in Docker. Nothing exotic.

**Symptoms**

- `curl https://api.anthropic.com` → times out every single time (TCP connect, ~10s). Claude Code installer obviously fails too.

- `curl https://deb.debian.org` → works maybe half the time, times out the other half. So `apt update` inside Docker builds randomly fails, then `install-php-extensions` blows up with "Unable to locate package libfreetype6 / libpq5 / libzip" (not actually missing, just failed index fetch).

- `curl https://github.com` and `https://pecl.php.net\` → fine, every time.

- DNS is fine. Resolution works for everything. This is not a DNS problem.

- IPv6 is a separate thing: DNS returns AAAA but the instance has no global v6 (never enabled at VCN/subnet/VNIC level), so anything preferring v6 stalls. Fixed that on my side with `precedence ::ffff:0:0/96 100` in `/etc/gai.conf`. Not the main issue.

**The mtr evidence** (TCP mode, port 443, 100 probes each)

Anthropic — path just dies after the upstream handoff:

```

1. 84.8.76.x (OCI edge) 0.0%

2. 87.109.47.100.0%

3. ??? 100.0%

```

That's it. Nothing at any TTL beyond hop 2, no SYN-ACK from the destination, ever.

Debian (Fastly) — real loss starting at hop 3, carried all the way to the destination:

```

1. 84.8.76.x (OCI edge) 0.0%

2. 87.109.47.100.0%

3. 87.101.252.6154.0%

4. 10.0.28.1347.0%

5. fra1.decixfra.fastly.net51.0% ~80ms

6. 146.75.118.132 (destination) 54.0%

```

(Also getting routed to Fastly in **Frankfurt** from Riyadh, which seems wrong on its own.)

GitHub control trace completes fine and real traffic works, so the path itself isn't globally broken — it's destination-specific.

**Why I'm confident it's not my config:** hops 1 and 2 (OCI's edge + the upstream handoff) are 0% on every trace. My route table / security lists / IGW are passing everything. The loss and the blackhole both begin at the same boundary (87.109.47.10 → 87.101.252.61), i.e. regional transit/peering. I've opened an SR with all three reports.

**Questions**

1. Anyone else in me-riyadh-1 seeing Anthropic (AS399358, 160.79.104.0/21) unreachable, or Fastly flaky?

2. Has anyone gotten OCI to actually escalate a transit/peering issue in a newer region? How long did it take?

3. Any recommended Debian mirror that has a clean path from Riyadh? I'm currently shopping for one with curl to keep Docker builds working while this is open.

Interim plan: apt mirror swap for builds, and if the Anthropic blackhole drags on, a WireGuard egress through another region for Claude Code only. Would much rather OCI just fix the route.

I started learning CI/CD using github actions after containerising my application and I have created CI pipeline for django app that runs test, builds and pushes image to github container registry.
I am sharing my yaml file for CI pipeline. Please do share your thoughts and where can i improve.

name: Test Pipeline 
on: 
  push:
jobs:
  test-backend:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:14
        ports:
          - 5432:5432
        env: 
          POSTGRES_USER: test_user
          POSTGRES_DB: erp
          POSTGRES_PASSWORD: 123456

    steps:
      - name: Checkout repo
        uses: actions/checkout@v4

      - name: setup python
        uses: actions/setup-python@v5
        with: 
          python-version: "3.13.5"

      - name: install dependencies
        run: pip install -r Backend/requirement.txt

      - name: run tests
        env: 
          DATABASE_URL: postgresql://test_user:123456@localhost:5432/erp
          DEBUG: 'True'
          ALLOWED_HOST: '*'
        run: |
          cd Backend 
          python manage.py test

  build-and-push-image:
    needs: test-backend
    permissions:
      contents: read
      packages: write
    runs-on: ubuntu-latest
    steps:
      - name: login to ghcr
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}


      - name: checkout repo
        uses: actions/checkout@v4


      - name: build image
        run: docker build -t ghcr.io/namespace/erp:${{ github.sha }} ./Backend


      - name: push image
        run: docker push ghcr.io/namespace/erp:${{ github.sha }}

I am a contributor to both terraform-provider-aws and Ansible Core repos, as well as Ansible Community AWS repo. I am on the lookout for additional projects to contribute to, ideally ones that have plenty of issues and where reviews are done quickly. It should also be quick and easy to compile from source. Looking for Golang or Python projects for code base programming language. Any ideas here?

We launched our PaaS prematurely, but it's now running around 50-60 apps from our own SaaS offering and our partners' apps, so I thought I'd share what we're building and get some opinions from people who actually work with infrastructure.

The platform is Kubernetes-based and can deploy static sites, backend apps, databases, and cache. Tenants can choose between shared or dedicated app deployment. Shared apps run on the platform's shared worker nodes. For dedicated deployment, the platform provisions a dedicated worker node for the tenant, and the tenant can deploy their apps on it.

One of the things we're currently working on is app scheduling/parking. The idea is that tenants can deploy as many apps as they want, but decide which apps are actually scheduled based on the resources available on their worker node.

For example, a tenant could have 20 apps deployed but only have 5 scheduled and running on a worker node. The other applications can be parked and scheduled when they're needed. We're trying to make this more of a first-class feature instead of requiring tenants to manually manage workloads or infrastructure.

We're inspired by Coolify and have used it for self-hosting for years. Coolify has worked really well for us, but scaling isn't really a first-class feature, which I understand is intentional. There are workarounds, but we wanted to build something where Kubernetes scheduling, scaling, and resource allocation are part of the platform itself without requiring the tenant to deal directly with Kubernetes.

We're also working on making static site deployment free with some limitations. The current idea is 50MB maximum static output for free apps. If the final build exceeds 50MB, the app would require a $2/month upgrade, with the paid limit currently at 100MB. We're also planning 10GB of bandwidth per tenant per month for free, then $0.026/GB for additional bandwidth.

We're still working on some of the edge cases around this, such as deploy-time blocking/confirmation when an app exceeds the limit and what happens when a paid app is downgraded and goes back under the free limit.

We're still early and there are a lot of things we need to improve, but the platform is already running our own production SaaS applications and partner applications.

I'd particularly like feedback from people who actually operate Kubernetes environments.

Does this deployment/scheduling model make sense from an infrastructure perspective? Are there any problems or operational issues you would expect with the shared/dedicated worker model or the app parking approach?

Also interested in hearing what you think we're missing compared to existing PaaS platforms.