Upgrading to K8s 1.22 Requires Emissary-ingress/AES 2.X: Why, How, and What You Need to Do

Emissary-ingress (formerly Ambassador API Gateway) and Ambassador Edge Stack 1.y support Kubernetes 1.11 through 1.21, but will not work on Kubernetes 1.22. Emissary and AES 2.y support Kubernetes 1.16 and later. This means Emissary and AES users wishing to upgrade to Kubernetes 1.22 or later need to first upgrade to Emissary/AES 2.y.

Given that Kubernetes 1.21 is End of Life on June 28th, we advise users to upgrade both their Kubernetes clusters to 1.22 and Ambassador Edge Stack to the 2.y series. The 2.y series has significant enhancements over the 1.y series to allow Ambassador Edge Stack to more gracefully handle larger installations, reduce global configuration to better handle multitenant or multiorganizational installations, reduce memory footprint, and improve performance. The 2.y series is our current series, with 3 series coming this June (at which point 2.y becomes our ‘maintenance series’ and 3.y is promoted to ‘current’).

Why the need to upgrade to Emissary/AES 2.X?

The reason for the incompatibility is that deprecation of certain features in the Kubernetes `CustomResourceDefinitions` (or CRDs) API has made it is fundamentally not possible for a CRD-using application (such as Emissary and AES) to be compatible with both Kubernetes 1.15 and 1.22 at the same time. This is a big part of why we did a “2.0” major version bump; we had to drop support for Kubernetes 1.11 through 1.15 in order to add support for Kubernetes 1.22

CRDs are the mechanism by which Ambassador Edge Stack and Emissary-ingress (formerly Ambassador API Gateway) program the Kubernetes apiserver to allow users to configure Emissary and AES through the apiserver; they are what allow users to use the usual Kubernetes tools such as `kubectl` and `helm` to configure Emissary and AES.

The 1.y line of Emissary and AES has always used the `v1beta1` version of the Kubernetes CRD API (apiextensions.k8s.io/v1beta1). Before 1.0.0, this wasn’t always the case; Ambassador API Gateway’s creation in March 2017 actually predates the introduction of CRDs in Kubernetes 1.7.0 in June 2017. At that point in time, we were unable to count on users being on a new enough version of Kubernetes to have support for CRDs. However, since Ambassador API Gateway 0.70.0 in May 2019, `v1beta1` of the Kubernetes CRD API has been used up until the introduction of the 2.y series.

When Kubernetes 1.16 introduced the final `v1` version of the CRD API in September 2019, we were eager to upgrade Ambassador to it, but we knew at that time that many of our users were still on older versions of Kubernetes. We care deeply about not breaking our users, which is why the entire 1.y line of Emissary and AES have maintained support for all Kubernetes versions down to 1.11 (which was released in June 2018, and has been EOL since December 2019). We have taken great care to avoid taking a stance that says “those versions of Kubernetes are EOL, and Ambassador simply doesn’t work on them,” and we instead chose to do everything we could to not leave our users out in the cold.

How changes to the CRD spec impacted Emissary/AES

Kubernetes 1.22 dropped support for the CRD `v1beta1` API, and as of Kubernetes 1.22 you must now use the final `v1` API. This means that a CRD-using application (such as Emissary and AES) cannot be compatible with both Kubernetes 1.15 and 1.22 at the same time. This was a driving factor behind why we did a “2.0” major version bump; we had to drop support for Kubernetes 1.11 through 1.15 in order to add support for Kubernetes 1.22 and later.

One of the big changes from the `v1beta1` CRD API to `v1` is that `v1` requires what Kubernetes calls “structural schemas.” This was one of the major engineering challenges that went into Emissary and AES 2.0. The configuration language of Emissary predates the concept of structural schemas, and many aspects of the language were fundamentally non-structural. We had to re-think these aspects of the language when we were crafting Emissary 2.0’s new `getambassador.io/v3alpha1` configuration language. This isn’t pointless churn being forced by Kubernetes. Users do ultimately benefit from this: the structural schemas allow the Kubernetes apiserver and the `kubectl` command line to provide feedback to users much earlier. When something is wrong with the YAML, users can now find out about it straight from `kubectl` when they try to apply it; rather than having to go read the Emissary or AES logs when something doesn’t work.

Now, the astute reader may be wondering, doesn’t Emissary and AES 2.1 and later also support that older non-structural `getambassador.io/v2` configuration language? How do they do that on Kubernetes 1.22, which supposedly requires structural schemas? At first we didn’t think it was possible; that’s why 2.0 didn’t support `getambassador.io/v2`. For version 2.1 we figured out some clever tricks to allow specific parts of the language to be non-structural without the apiserver rejecting it.

So, if we decided that we were OK with the 1.y line dropping support for Kubernetes 1.15 and older, could we use those same tricks to make Emissary and AES 1.y be compatible with Kubernetes 1.22? Yes, theoretically this is possible. It’s a whole lot of engineering and testing work, but that’s exactly the work which we have delivered to you with “Emissary 2.1.0” and “AES 2.1.0.” Besides the changes to support Kubernetes 1.22+, the 2.y series has significant enhancements over the 1.y series to allow Ambassador Edge Stack to more gracefully handle larger installations, reduce global configuration to better handle multitenant or multiorganizational installations, reduce memory footprint, and improve performance.

For more information, see Kubernetes API and Feature Removals In 1.22: Here’s What You Need To Know. For the changelog detailing these beta API removals, please see the Kubernetes changelog.

What you need to do to upgrade to Emissary/AES 2.X

If you are using a hosted version of Kubernetes such as AWS EKS, you may have extended support from your provider. For example, AWS will support Kubernetes 1.21 until February 28 2023. In the event that you use hosted Kubernetes, we recommend you check maintenance dates with your provider in order to plan your Ambassador Edge Stack upgrade.

We advise users who are using upstream Kubernetes to upgrade their Ambassador Edge Stack installations to the 2.y series. A no-downtime migration path is available and our technical support team is happy to assist.

Thanks to Luke Shumaker and Alice Wasko for technical review.