NSA and CISA: Here’s how to improve your Kubernetes cluster security

The National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA) have published updated guidance about how to harden Kubernetes for managing container applications. 

Kubernetes is an open-source system that automates deployment, scaling, and management of applications run in containers.

The updated guidance refreshes the two agencies’ first Cybersecurity Technical Report regarding Kubernetes hardening guidance from August 2021. CISA says the update contains additional details and explanations based on feedback from industry, including more detailed info on logging and threat detection in addition to other clarifications. 

Some of the updates are subtle but important for those who protect Kubernetes clusters. NSA and CISA do not list what the changes are in the updated guidance, but the initial recommendations weren’t met with universal approval. 

For example NCC Group noted that advice about Kubernetes authentication was “largely incorrect when it states that Kubernetes does not provide an authentication method by default”, whereas most customer implementations NCCGroup had reviewed “support both token and certification authentication, both of which are supported natively.” NCCGroup advised against both for production loads because Kubernetes does not support certificate revocation, which can be a problem if an attacker has gained access to a certificate issued to privileged accounts. The updated guidance now says that “several user authentication mechanisms are supported but not enabled by default.”

Otherwise, key points of the original document appear to be unchanged. It looks at hardening within the context of typical Kubernetes cluster designs that include the control plane, worker nodes (for running containerized apps for the cluster), and pods for containers that are hosted upon these nodes. These clusters are often hosted in the cloud and often across multiple clouds in AWS, Azure, Google and elsewhere.   

The agencies maintain that Kubernetes is commonly targeted for data theft, computational power theft, or denial of service. Historically, flaws in Kubernetes and various dependencies as well as misconfigurations have been used to deploy cryptominers on victim’s infrastructure.    

It also maintains that Kubernetes is exposed to significant supply chain risks because clusters often have software and hardware dependences built by third-party developers. 

For example, security analysts last year warned of attacks against Kubernetes clusters via misconfigured Argo Workflows container workflow engine for K8s clusters.  

Besides supply chain risks, other key actors in the agencies’ threat model include malicious outsiders and insider threats. These help define its hardening recommendations.

For example, there is a common cloud case where workloads that aren’t managed by a given Kubernetes cluster share the same physical network. In that instance, a workload may have access to the kubelet and to control plane components, such as the API server. So, the agencies recommend network level isolation.   

The agencies provide advice on how to ensure strict workload isolation between pods running on in same node in a cluster, given that Kubernetes doesn’t by default guarantee this separation.  

Announcing the updated guidance, the NSA says: “Primary actions include the scanning of containers and pods for vulnerabilities or misconfigurations, running containers and pods with the least privileges possible, and using network separation, firewalls, strong authentication, and log auditing.”

The agencies also recommend periodic reviews of Kubernetes settings and vulnerability scans to ensure appropriate risks are account for and security patches are applied. 

But patching is not easy in the context of Kubernetes. CISA regularly publishes alerts about new Kubernetes related vulnerabilities. In February for example it warned of a critical (severity score 8.8 out of 10) privilege escalation flaw, CVE-2022-23652, which affected the capsule-proxy reverse proxy for Capsule Operator. 

But as NCCGroup points out: “patching everything is hard”, partly because of the pressure to avoid downtime but also because relevant vulnerabilities span Kubernetes, Containerd, runc, the Linux kernel and more.

“This is something that Kubernetes can help with, as the whole concept of orchestration is intended to keep services running even as nodes go on and offline. Despite this, we still regularly see customers running nodes that haven’t had patches applied in several months, or even years. (As a tip, server uptime isn’t a badge of honour as much as it used to be; it’s more likely indicative that you’re running an outdated kernel),” NCCGroup noted.