Changelog for Kubernetes 1.35

Versions

The deployed Kubernetes patch version varies based on when your cluster is deployed or upgraded. We strive to use the latest versions available.

Current release leverages Kubernetes 1.35. Official release blogpost found here with corresponding official changelog.

Optional addons

• cert-manager is provided with version v1.21.2
• Envoy Gateway is provided with version v1.9.0

Major changes

• The kubeconfig is replaced with a revocable one. Please download the new kubeconfig after the upgrade, see below.
• Our support for ingress-nginx ends on 31 December 2026, see below.
• CoreDNS is now kept up to date together with the other addons. Before, it stayed on the version the cluster was created with.
• DNS lookups from pods for names outside the cluster now go to Elastx’s own DNS resolvers in our data centers instead of public resolvers, which makes them faster and keeps them within Elastx.
• New worker nodes are added to your load balancers once they have an address. Before, a new node could be missed until another node joined or left. With many load balancers, adding a node to all of them can take a while.
• The cluster autoscaler now matches the cluster’s Kubernetes version.

Introducing a revocable kubeconfig

The kubeconfig in your cluster’s object storage container is now a token that we can revoke, instead of a client certificate that cannot be revoked.

• The upgrade replaces the kubeconfig in object storage. Copies of the old one keep working until their certificate expires, at most one year later, but they cannot be revoked or rotated. We recommend that you switch to the new one.
• If someone leaves your team or a kubeconfig leaks, contact support to have it rotated. Every kubeconfig we issued for the cluster is revoked, and a new one is placed in object storage shortly after.
• The kubeconfig is valid for one year by default and is renewed automatically before it expires. The lifetime can be changed on request.
• If a tool requires a client certificate, we can issue a certificate-based kubeconfig instead. Switching between the two revokes the kubeconfig you had before.
• The user and context names are the same as before, so scripts that select a context keep working.

Introducing nodegroups on a shared network

A nodegroup can be attached, on request, to a network that another of your OpenStack projects owns and shares with the cluster’s project. Your virtual machines in that project and your cluster can then talk privately, without a floating IP per machine. Available from Kubernetes 1.35.

• Each nodegroup can use one shared network, and reaches only its own subnet on it.
• The shared subnet must not overlap the cluster’s node, pod or service ranges, and needs room for the nodegroup’s maximum size plus one address per availability zone.
• Give the shared network the same MTU as the cluster network. With a smaller MTU, pods on those nodes get a smaller MTU too, and large UDP packets to them are dropped.
• A load balancer that should only be reachable on the shared network needs these annotations when the Service is created. They are only read at creation, so a Service that lacks them must be recreated. With the Envoy Gateway addon, put them on your Gateway under spec.infrastructure.annotations.

metadata:
  annotations:
    service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
    loadbalancer.openstack.org/subnet-id: <subnet on the shared network>
    loadbalancer.openstack.org/member-subnet-id: <the cluster's subnet>

Introducing optional cluster settings

• Service range: a new cluster can be created with a Service range of your choice, between /12 and /24, for example to avoid overlapping a network you want to share with the cluster. It cannot be changed once the cluster exists.
• Encryption of Secrets at rest: Secrets stored in etcd can be encrypted, on request. Turning it on for a running cluster replaces the control plane nodes twice, without downtime for your workloads. Once turned on, it cannot be turned off. Existing Secrets are encrypted the next time they are written.

Kubelet configurations provided by Elastx, not configurable by client

• podPidsLimit: 4096 → no limit
  • The per-pod limit on process IDs introduced in Kubernetes 1.33 is removed. Clusters on 1.34 and older keep the limit of 4096. You are responsible for containing runaway processes in your own workloads.

End of support for ingress-nginx

The ingress-nginx project has been retired upstream, and our support for it ends on 31 December 2026.

• We will not remove ingress-nginx from any cluster, and it keeps running as it does today.
• No newer version of ingress-nginx is planned. Until 31 December 2026 we still help with problems in the ingress-nginx we installed. After that date, keeping it maintained is in your hands.
• We recommend Gateway API with our Envoy Gateway addon, which we support and keep up to date. Moving your ingresses to Gateway API is done by you, as announced in the Kubernetes 1.34 changelog.

Noteworthy changes in upcoming versions

Announcement of changes in future versions.

Scheduled for upcoming releases:

• We’ll remove the legacy nodelocaldns where still deployed. Relevant only if the cluster was created before v1.26.

Is downtime expected

The cluster control plane should remain available during the upgrade; however, pods will be restarted when workloads are migrated to new nodes. Plan for short pod restarts during the upgrade.

Known issues

Custom node taints and labels lost during upgrade.

Custom taints and labels on worker and control-plane nodes may be lost during the upgrade. We recommend auditing and reapplying any critical custom taints/labels via automation (e.g., cluster bootstrap, configuration management, or a post-upgrade job).

There is a label that is persistent across upgrades that can be used to direct workload to particular nodegroups. Example on how to use it:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodegroup.node.cluster.x-k8s.io
                operator: In
                values:
                - worker1

Snapshots are not working.

There is currently a limitation in the snapshot controller: it is not topology-aware. As a result, snapshot behavior may be unreliable for topology-sensitive volumes. Avoid depending on snapshots for cross-zone/region recovery until a topology-aware snapshot controller is available or confirm your storage driver’s snapshot semantics.