In this article we’ll take a look at how to proxy external services through the Kubernetes Gateway API. There are of course more lightweight methods to proxy services, but once you already have the proverbial hammer, why not treat everything like a nail?
We’ll first proxy an external service that doesn’t provide its own TLS-certificate using an HTTPRoute. In the process we’ll also create a valid certificate for the service.
Next we’ll proxy an application that provides its own TLS certificate using a TLSRoute, assuming the application provides a valid certificate, or we decide to trust what it provides.
If you’re more keen on using the Ingress machinery instead, I can recommend Kris’ article on Using Kubernetes Service for Proxying to External Services. A third option is rolling your own DNS service like Pi-Hole, AdGuard Home, or Technitium, but that would only work when you’re connected to it.
This article relies on Cilium’s Gateway API implementation, but it should be easy to adapt to other Gateway API implementations.
Overview#
I’ll assume you have some familiarity with the Gateway API for internal services, but if not, — and you want ot learn more about it, I’ve written an article on how to get started with Gateway API with Cilium and Cert-manager.
As an example, we’ll take a look at how to proxy a Home Assistant OS instance running on a separate machine in the same network and attach a certificate using Cert-manager.
If you’re trying to proxy Home Assistant, make sure to edit the Configuration.yaml file to add the IP of the Gateway service as a trusted proxy.
Some applications, such as Proxmox VE, refuse to do anything but HTTPS. In these cases we can let the application handle the certificate and just pass the connection through to the correct hostname.
For applications like this we can’t rely on Cert-manager, and instead have to get our hands dirty reading the documentation on how to configure valid certificates, or if we’re lucky find an article that explains how to do it, like this one for Proxmox VE by Derek Seaman.
A rough sketch of the configuration we need to conjure is an endpoint (EndpointSlice/Endpoints) for the external application which is then routed to a Service. Next we connect the Service to a Route (HTTPRoute/TLSRoute) that gets picked up by a Gateway and exposed to our browser.
Service without selectors#
To get Kubernetes to talk with external services we can create a Service without selectors, which is exactly what it sounds like
| |
Since the Service doesn’t have any selectors, the corresponding EndpointSlice and (legacy) Endpoints resources won’t be automatically created.
Depending on the Gateway Controller you’re using, you need to craft either the EndpointSlice- or Endpoints-resource yourself. I know that Cilium works with EndpointSlices, but Traefik needs an Endpoint-resource till this GitHub issue gets resolved, your mileage may vary.
By convention, the EndpointSlice name is prefixed with the corresponding Service name,
but you’re free to choose whatever name as long as it’s namespace-unique for that resource type.
To link the EndpointSlice to the Service you can set the kubernetes.io/service-name label equal the Service name
(line 7).
| |
The EndpointSlice should point to the external service we want to reach.
In this case we’ve set the addressType type IPv4 and supplied an IP-address
— preferably static, on line 12.
In the case of Home Assistant we could’ve also set the addressTtype
to FQDN (Fully Qualified Domain Name) and
use homeassistant.local as the address.
Last we supply the port used by the external application (line 18) and give it a name matching the port in the Service.
If you use Argo CD you can add line 14-15 to avoid a sync-loop while someone figures out this GitHub issue.
HTTPRoute#
Up until now everything is similar to how you would expose an external service using an Ingress-resource. To use the Gateway API we instead create an HTTPRoute.
| |
Line 8–9 refers to a Gateway resource which we’ll create shortly. On line 11 we specify the hostname which we want to reach our external resource on. Similar to an Ingress resource we need to reference which Service and port the HTTPRoute should route to (line 18–19).
Gateway#
A key difference between the Ingress model and the Gateway API is separation of concerns. While routes are meant to be configured by application developers, Gateways are meant to be prepared by cluster operators.
To aid in this separation we can create our Gateway in a separate namespace to be used by different applications. In this example we’ve opened up for routes from all namespaces (line 23-24), though it’s possible to use a LabelSelector for more fine-grained control.
| |
To hook up the exposed with a TLS-certificate through a configured Cert-manager instance we simply give the Gateway the correct annotation (line 7) and a TLS-certificate reference (line 20-21). I’ve detailed how to do this in Gateway API with Cilium and Cert-manager.
Next we can assign an IP to the Service spawned by the Gateway by giving it an infrastructure annotation (line 12) assuming you’ve set up L2 announcements similar to what’s done in this article. Other possibilities would be to use Metal LB.
For our HTTPRoute to be picked up by the Gateway it has to match a listener, in this case it should match the hostname given on line 17.
To check the status of the Gateway and attached routes you can run
kubectl -n <GATEWAY-NAMESPACE> get gateway <GATEWAY-NAME> -o json | jq '.status'
TLS Passthrough#
In reverse this time, we’ll start with the Gateway
| |
Here we’ve added a listener with the TLS protocol (line 13) and put the tls-setting in Passthrough mode (line 17). Make sure the hostname (line 15) matches the TLSRoute we’re about to create next
| |
Here we’ve matched the TLSRoute with our Gateway by name and namespace (line 8-9) and hostname (line 11). Similar to the HTTPRoute we now have to create a matching headless Service
| |
where we make sure to match the target port (line 10) of our EndpointSlice
| |
Unless you’ve run into some issues you should now have TLS passthrough via the Gateway API!
Caveats#
At the time of writing I haven’t gotten Gateways to play nice when combining HTTPS-protocol listeners and TLS-protocol listeners in TLS Termination mode.
When I tried, I was met with this error in the event log of the Gateway
Skipped a listener block: [
spec.listeners[1].tls.certificateRef:
Required value: listener has no certificateRefs,
spec.listeners[1].tls.mode:
Unsupported value: "Passthrough":
supported values: "Terminate"]
and all attached Gateway routes appears to be non-responsive.
I was eventually able to track down the message as coming from Cert-manager. This is caused by the annotation
cert-manager.io/issuer: cloudflare-issuer
which instructs Cert-manager to create a certificates for the Gateway listener hostnames. I’ve opened this issue with them as I think this is not the intended behaviour.
I’ve also had issues with TLSRoutes apparently attaching to HTTPS listeners with a matching hostname, even though the only supported kind for HTTPS-protocol listeners is supposed to be HTTPRoute. When this happens all the HTTPRoutes stop working and I have to delete the TLSRoute and recreate the Gateway for the HTTPRoutes to start working again.
I’ve enquired about this behaviour with the helpful folks at Cilium in this comment.
Until these issues are fixed — or someone can explain that I’m doing something wrong and I fix it, the workaround appears to be two separate Gateways. This is not ideal though, as the attached Services for the two Gateways necessarily get different IPs. This means you can’t route to just one IP without getting creative with a third Gateway, or some other proxy mechanism.
Summary#
The resources for this article can be found in this GitLab repo.
If you’re using Kustomize — or possibly Argo CD with Kustomize + Helm, the above configuration can be summarised as
❯ tree
.
├── kustomization.yaml
├── gateway
│ ├── gw-https.yaml
│ ├── gw-tls-passthrough.yaml
│ └── kustomization.yaml
├── home-assistant
│ ├── endpoint-slice.yaml
│ ├── http-route.yaml
│ ├── kustomization.yaml
│ └── svc.yaml
└── proxmox
├── endpoint-slice.yaml
├── kustomization.yaml
├── svc.yaml
└── tls-route.yaml
# kustomization.yaml
resources:
- gateway
- home-assistant
- proxmoxGateway#
Take a look Gateway API with Cilium and Cert-manager for how to configure Cert-manager to provision TLS-certificates for Gateway resources.
# gateway/kustomization.yaml
resources:
- gw-https.yaml
- gw-tls-passthrough.yaml# gateway/gw-https.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: https-gateway
namespace: gateway
annotations:
cert-manager.io/issuer: cloudflare-issuer
spec:
gatewayClassName: cilium
infrastructure:
annotations:
io.cilium/lb-ipam-ips: 192.168.1.221
listeners:
- name: https-gateway
protocol: HTTPS
port: 443
hostname: "*.<DOMAIN>"
tls:
certificateRefs:
- kind: Secret
name: cert-stonegarden
allowedRoutes:
namespaces:
from: All# gateway/gw-tls-passthrough.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: tls-passthrough-gateway
namespace: gateway
spec:
gatewayClassName: cilium
infrastructure:
annotations:
io.cilium/lb-ipam-ips: 192.168.1.222
listeners:
- name: proxmox-tls-passthrough
protocol: TLS
port: 443
hostname: "proxmox.<DOMAIN>"
tls:
mode: Passthrough
allowedRoutes:
namespaces:
from: AllHome Assistant OS HTTP-passthrough#
# home-assistant/kustomization.yaml
resources:
- endpoint-slice.yaml
- svc.yaml
- http-route.yaml# home-assistant/endpoint-slice.yaml
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: home-assistant
namespace: home-assistant
labels:
kubernetes.io/service-name: home-assistant
endpointslice.kubernetes.io/managed-by: cluster-admins
addressType: IPv4
endpoints:
- addresses:
- 192.168.1.81
# https://github.com/argoproj/argo-cd/issues/15554
conditions:
ready: true
ports:
- name: http
port: 8123# home-assistant/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: home-assistant
namespace: home-assistant
spec:
ports:
- name: http
port: 80
targetPort: 8123# home-assistant/http-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: home-assistant
namespace: home-assistant
spec:
parentRefs:
- name: https-gateway
namespace: gateway
hostnames:
- "home-assistant.<DOMAIN>"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: home-assistant
port: 80Proxmox TLS Passthrough#
# proxmox/kustomization.yaml
resources:
- endpoint-slice.yaml
- svc.yaml
- tls-route.yaml# proxmox/endpoint-slice.yaml
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: proxmox
namespace: proxmox
labels:
kubernetes.io/service-name: proxmox
endpointslice.kubernetes.io/managed-by: cluster-admins
addressType: IPv4
endpoints:
- addresses:
- 192.168.1.42
# https://github.com/argoproj/argo-cd/issues/15554
conditions:
ready: true
ports:
- name: https
port: 8006# proxmox/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: proxmox
namespace: proxmox
spec:
ports:
- name: https
port: 443
targetPort: 8006# proxmox/tls-route.yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
name: proxmox
namespace: proxmox
spec:
parentRefs:
- name: tls-passthrough-gateway
namespace: gateway
hostnames:
- "proxmox.<DOMAIN>"
rules:
- backendRefs:
- name: proxmox
port: 443