Addressing the Need for Egress Traffic Control in Kubernetes with Istio

When I was serving in the Air Force, Airspace planning and management was a very important process. The proper management of ingress and egress corridors to exercise areas and airdromes was of high importance to avoid deadly conflicts. The same is true in sea faring. A ship’s «κυβερνήτης» (kubernḗtēs, Greek for «helmsman”) has to follow certain ingress and egress routes in busy ports to avoid causing devastating accidents.

In the cyber world (hint: cyber comes from the Greek «κυβερνώ», to govern), Kubernetes has accelerated the speed with which developers can deploy new applications and their update releases because they have minimized complexity in resource allocation and standardized the deployment configurations. To achieve that, pods are communicating extensively. Having visibility and securing these incoming and outgoing data streams is important for their security and the overall operation of K8s.

The importance of egress control in Kubernetes

As StackRox mentions in a related blog post, “It becomes increasingly important to know and control what third-party services are being accessed, for auditing purposes, for data security, and to limit the potential for damage if any malicious code somehow finds its way into a running container in the service deployment.”

Securing the outgoing Kubernetes communications is part of the process of defining network policies. Network policies are used in Kubernetes to specify how pods communicate with each other and with external endpoints.

• Ingress traffic is traffic to a pod from an external endpoint.
• Egress traffic is traffic from a pod to an external endpoint.

It is interesting to note that traffic controls are bidirectional – traffic from pod A to pod B is allowed if and only if egress is allowed from A to B and ingress is allowed to B from A.

In addition to Kubernetes network policies, additional ingress and egress policies can be maintained by using a service mesh, such as Istio. Although controlling and securing ingress traffic is important for a service mesh to prevent attackers from penetrating through ingress APIs, securing egress traffic is equally important.

Once attackers penetrate an application in a cluster, they can leverage its communications to further attack other corporate services, such legacy on-premises systems, external web services and databases. In addition, they may steal or compromise sensitive data. Finally, the external attacker may use that cluster as a hub to launch further attacks and disrupt business services. Egress traffic may also be used by insiders, such as disgruntled DevOps staff, seeking to cause harm or for financial reasons.

While the possibilities of leveraging insecure, uncontrolled egress traffic seem endless, you can reduce them by controlling your K8s outgoing traffic.

Traffic control is also a regulatory requirement. For example, PCI DSS requires that inbound and outbound traffic must be restricted to what is necessary:

1.2.1 Restrict inbound and outbound traffic to that which is necessary for the cardholder data environment, and specifically deny all other traffic.

The same standard regarding outbound traffic mandates:

1.3.4 Do not allow unauthorized outbound traffic from the cardholder data environment to the Internet… Connections should be inspected to restrict traffic to only authorized communications.

What are the requirements for secure egress control?

Secure control of egress traffic is about monitoring the traffic and enforcing appropriate security policies. Continuous monitoring of egress traffic enables you to analyze the associate data streams and detect attacks that went under your detection radar. On the other hand, security policies should be based on the need-to-know principle limiting outgoing communications to the necessary for the applications to function.

IBM and Istio have developed a list of security practices to control egress traffic. They are:

• Only TLS encrypted egress traffic should be supported, meaning that either the application or Istio should perform TLS origination.
• Monitor Server Name Indication (SNI) and the source of all egress traffic to identify suspicious traffic and prevent attacks.
• Define and enforce policies per cluster stating which external services any pod in the cluster can access. All other unspecified access must be blocked. The purpose is to allow access only to the external services that the applications in the cluster need to access and to block access to all other services, reducing the attack surface.
• Define and enforce policies per source, using a Kubernetes-aware approach. All other access must be blocked. If policy enforcement is not Kubernetes-native, you must identify the source of traffic by the IP of the pod, which is not convenient, especially since these IPs are not static (to support the ephemeral nature of pods).
• Prevent tampering. In case an application pod is compromised, you should prevent the compromised pod from bypassing monitoring and violating the egress policies.
• Traffic control is transparent to the applications without changing the environment of the containers.

Egress control with Istio

While there are many alternates to control egress traffic, the most effective and robust solution is to use Istio. Istio blocks traffic to all endpoints outside the service mesh by default, requiring any valid external services to be explicitly defined on an allowlist.

StackRox mentions that further control of egress traffic can be facilitated “by deploying an Istio egress gateway and routing outgoing traffic through that. When combined with Kubernetes Network Policies that only allow Istio’s egress gateway deployment to make connections outside the cluster, these measures can also help prevent any rogue container or process in the Kubernetes cluster from bypassing the Envoy proxy and making direct outgoing connections on its own.”

Going back to the Airspace management topic, controlling your inbound and outbound air traffic is of utter importance for safe and reliable flights, not only for the military but also for the civilian aviation. With so many flights coming and going from busy airports, failure to control the ingress and egress traffic would result in delays and – most important – deadly accidents. Such is the case with K8s. Lack of secure egress traffic control of all your Kubernetes pods will leave your application workloads open to risks and challenges threatening your regulatory compliance and productivity.