Anyone else hit this?

Users created with 2.14.1 / 2.14.2 / 2.15.0 cannot login #5537

https://github.com/rancher/rancher/issues/55378

The primary thing I use rancher for is to allow for local access, and you can't log in with accounts created in recent versions of rancher. I don't feel like downgrading k8s to a version the pre-bug version of rancher stable works on.

I have a prior working experience in k8s but when it comes to interview, I am not getting the confidence as I had my legin many boats.

It is boring to go through any book or course which gives me lots of basic info so I am looking for a way through i could brushup my skills and also practice/learn the things and master myself from interview & working perspective.

Looking for your valuable inputs,,

Thanks

One of the key tools for achieving graceful shutdown in Unix-based systems is the concept of signals, which are software interrupts sent to a program to indicate that an important event has occurred.

Hi all!

Along 32 other providers, you can now declare application secrets using a single toml and also use kubernetes among them.

It's OSS project we've been working on for a year with 10 SDKs supported.

Let us know, is has helped many people sort out their secrets!

A CRD upgrade can pass schema validation while a rolling deployment still leaves old controller pods reading or writing the previous shape. The risky cases seem to be renamed fields, changed defaults, status fields with different meaning, and a downgrade after some objects have already been rewritten. What compatibility contract do you use across that window? I would expect additive fields first, explicit conversion where versions differ, checks around storedVersions, fixtures that exercise both controller versions, and a rollback plan that does not assume newly stored objects are readable by the old binary. Which parts do you test before rollout, and when is a conversion webhook worth the operational cost instead of keeping one storage version and doing a staged migration?

Figured this would be useful for anyone here doing disconnected/air-gapped delivery or trying to find a way to package up artifacts. Disclosure: I am one of the maintainers.

If you haven't heard of it, Hauler is FOSS (Apache-2.0) tooling for packaging up OCI artifacts (images, charts, files) into a portable, cryptographically verifiable "store" you can carry across a boundary and serve or push into a registry on the other side.

We just cut 2.1.0, and it's a big one: pulls now run concurrently by default, which makes syncs 3-5x faster than previous versions, plus audit logging, store integrity checks, and store-to-manifest generation.

We're always very open to feedback and collaboration, so check it out!

Release notes here: https://github.com/hauler-dev/hauler/releases/tag/v2.1.0

And a quick rundown of the update here by another maintainer: https://www.youtube.com/watch?v=kUfjh7m-pYg

When HPA scales application or database pods up or down, existing long-lived connections may remain attached to the original pods, leaving new pods underutilized.

Does HPA provide a native lifecycle event or hook that applications can use to invalidate or gradually rebalance their connection pools after scaling?

If not, what is the recommended Kubernetes pattern - pod lifecycle hooks, endpoint watching, finite connection lifetimes, or a database-aware proxy?

fqdn-network-policy — egress control by hostname, not CIDR

Lets you write Kubernetes egress rules against FQDNs. The controller

resolves them against 4 public resolvers (Cloudflare, Google, Quad9,

OpenDNS), unions the results to catch CDN anycast divergence, and keeps

a standard NetworkPolicy in sync automatically.

No CNI replacement. Works on top of whatever you already run.

What makes it different from Cilium/Calico FQDN policy:

- Outputs standard networking.k8s.io/v1 NetworkPolicy (portable, auditable)

- kubectl plugin: `kubectl fqdn-policy preview policy.yaml` shows resolved

IPs, CNAME chains, and the generated policy before you touch the cluster

- DNS rebinding protection on by default (private IPs blocked)

- ClusterFQDNNetworkPolicy for cluster-wide rules

Built this after a CDN IP rotation took down prod egress at 2am.

https://github.com/kunaldevxxx/fqdn-network-policy

Running Java on Kubernetes? CPU and memory limits affect much more than scheduling.

A 4 GiB heap requires a container larger than 4 GiB. Fractional CPU limits can also change the processor count that HotSpot sees.

In our new article, you will learn:

• Why JVM heap size and container memory are different boundaries
• How CPU limits affect garbage collection, worker pools, and application performance
• Which JVM and container metrics to collect when validating resource settings under load

The article also includes practical experiments and an interactive configuration calculator.

Read: https://learnkube.com/java-jvm-kubernetes-requests-limits

I have a working experience in k8s and I want to start writing CRD.

I have good handon exp in python and I can learn any other lang like Go if reqd.

Pls provide me a roadmap or a easy path to start with.

Hey everyone,

I'm looking for a cheap and reliable cloud provider for a non-production Kubernetes environment (mainly dev/testing/staging).

I don't need anything super powerful, just something reasonably priced and easy to manage.

What are you guys using for your non-prod environments? Any recommendations for something cheap with predictable pricing?

Would also be great to know roughly how much you're paying per month and what kind of workloads you're running.

Thanks!

Infra/DevOps folks — when your cloud cost or capacity spikes unexpectedly, how do you actually trace it back to the cause?

1. Are you running Kubernetes (or similar) across cloud, on-prem, or both?
   
2. When infra cost/capacity spikes unexpectedly, how do you currently trace it back to the cause — check deploy logs, ask around, or not really investigate?
   
3. Roughly how long does that take when it happens?
   
4. Would you want something that automatically says "this spike traces back to deploy X / this config change / this scaling event," regardless of whether it's cloud-billed or on-prem capacity?
   
5. Do you currently use any tool for cost/capacity visibility (Kubecost, Vantage, CloudHealth, internal scripts, none)?

Trying to get a better handle on our gpu spend.

right now we can see the overall infrastructure cost, but its hard to tell which workloads are actually responsible for it.

ideally want to see cost by namespace / workload / job and understand how much gpu capacity is just sitting idle.

any tools doing this well?

Suppose you’re running your entire application stack on Kubernetes. The database is running as a Pod on one node, and the web server is running as another Pod on that same node. You expose the web server to the internet through a LoadBalancer Service, and clients connect to it over HTTPS.

Isn’t this inherently less secure because the same physical machine that is directly handling internet-facing traffic is also hosting the database? Without Kubernetes, you might isolate the web server and database onto separate machines, so compromising the web server wouldn’t directly put the database on the same machine. How does Kubernetes address this security concern?

I’ve been working in cloud/DevOps for about 8 years, primarily in AWS, IaC/Terraform, CI/CD, automation, Linux, and supporting fairly complex production systems (with some EKS migrations / troubleshooting and home lab stuff mixed in).
The weird position I’m finding myself in now is that a lot of roles that match my overall experience level are Senior DevOps/Platform/Cloud Engineer positions — but those same positions increasingly expect significant production Kubernetes experience.

For those who made this transition later in their DevOps/cloud careers:

How did you bridge that gap?

Did you find employers willing to hire for your broader DevOps experience and let you ramp up on Kubernetes?

How valuable were home labs/projects compared with actual production experience?

Are there particular K8s skills you’d prioritize to become useful quickly on an established platform? Thanks!

Reading this and chuckling, because .. wtf?!! hahah

Sometimes I think AI is pretty stupid, sometimes they just come with something like this, almost as if they were human and cared for my health, feelings, etc. lmao

Anyway.
I've spent the day [and the night, apparently, since it's almost 5 a.m. now] trying to understand how OpenBao works, how to integrate it into the K8s cluster I built in my homelab, and so on.

Now, I've come across some tools that are really nice to work with [e.g., Cilium CNI, Flux, etc.], but Rook|Ceph [thank Nature I'm past that now], and OpenBao — well, I'm either too stupid, or these things are extremely complicated, and integrating them is no walk in the park. The certificate stuff was particularly hard to grasp, and I still have a lot of notes to go over since I'm still not sure about certain things🤔.

I mean, it took me a while to even understand the structure of OpenBao docs so I could find the stuff that applied to my use case, and which didn't. On the other hand, OpenBao is supposed to work really nice with Flux, so .. that's a nice point.

Anyway. The AI said I should take a rest, so ..😂😂

project structure shot💀

Keep learning, folks.

How do you decide the numbers, and how do you keep them right over time?

Before I did any research: low requests → performance problems, high requests → huge cost. Felt like a lose-lose.

After digging into my own monitoring data, here's where I landed (tell me if this is wrong):

What I'm running today, and what I now think is off about it:

• CPU request too high (200–500m). My services barely use 20m steady-state. This is pure waste and it's what's driving my node count.
• CPU limit only a bit above request (800m–1000m). Causing two problems: (1) Java services are slow / crash-loop at startup, because the JVM needs a big CPU burst to boot (JIT + class loading + Spring context); (2) occasional slow requests from CPU throttling.
• Memory request is fine (600Mi–1000Mi). Matches what monitoring shows.
• Memory limit higher than request — and I now think this is a mistake. Unlike CPU, memory isn't reversible: once a pod grabs memory above its request it keeps it (a JVM especially never gives heap back). So the burst headroom just risks OOMKills for other pods on the node, with no real upside.

Am I thinking about this correctly now?

If yes, my plan:

• CPU: drop request low (30–50m) and set the limit very high (3–4 cores) or remove it entirely — should fix both the throttling and the slow startup.
• Memory: set request to real usage and keep the limit ~equal to the request.

My one worry: if I remove the CPU limit, won't a pod grab all the node's CPU during a spike and starve everything else?

Would love to hear how people actually run this in prod — especially anyone running Java with no CPU limit. Does this hold up, or am I about to learn something the hard way?

Hi, all, a bit later than usual, but my usual roundup of new Kubernetes security features is finally published:

• https://www.sysdig.com/blog/kubernetes-1-37-new-security-features

Hope you find it useful.

What may break things

• #1710 Speed up recursive SELinux label change
• #5343 Make nftables the default kube-proxy backend
• #140226 Kubelet: Static Pods can no longer reference Secrets or ConfigMaps

Net new enhancements

• #4939 Support TLS credentials in gRPC probe
• #5502 Add stickyBit support for emptydir volumes
• #5823 Pod-level checkpoint/restore
• #5855 Add bind mount options (noexec, nodev, nosuid) support on volumeMounts
• #5936 Add user fields to atomic write volumes
• #5943 Topology for volume snapshots
• #6060 API server authentication to webhooks

Enabled by default

• #2033 Kubelet-in-userns, aka rootless mode
• #5541 Report last used time on a PVC
• #5793 Manifest-based admission control config

Changes in other existing features

• #4412 Projected service account tokens for Kubelet image credential providers
• #3257 ClusterTrustBundles (previously Trust Anchor Sets)
• #4317 Pod certificates
• #4762 Allows setting any FQDN as the pod's hostname
• #5295 KYAML

Hello all!

I am a devops engineer, and now I want to take my kubernetes knowledge to the next level.

I want to write a "temu" cillium, for learning purposes only.

I am very familiar with kubernetes, have linux and programming(not go, but c++) knowledge.

If you did anything like this, do you have any advice?

I read kustomize a fews days ago. Looks like it's going to help us with organizing deployments in argocd

Here's an example I found:

my-update-project:
stage: gitdeploy
image: alpine:latest
script:
- apk add --no-cache git curl kustomize

# Clone GitOps repo
- git clone https://deploy-bot:${DEPLOY_TOKEN}@gitlab.com/myorg/gitops-repo.git
- cd gitops-repo/my-app/overlays/${DEPLOY_ENV}

# Update image tag
- kustomize edit set image myregistry/my-app:$CI_COMMIT_SHORT_SHA

# Commit & push
- git config user.email "ci-bot@myorg.com"
- git config user.name "CI Bot"
- git commit -am "Deploy ${DEPLOY_ENV} my-app:$CI_COMMIT_SHORT_SHA [skip ci]"
- git push

However, I believe this can encounter an issue like several devs making updates and pipeline almost running at the same time. How did you solve this?