Skip to main content
  1. Articles/

External services with Gateway API

··2056 words·10 mins·
Vegard S. Hagen
Author
Vegard S. Hagen
Pondering post-physicists
Table of Contents

In this article we’ll take a look at how to proxy external services through the Kubernetes Gateway API. There are of course more lightweight methods to proxy services, but once you already have the proverbial hammer, why not treat everything like a nail?

Gateway API logo
The logo for Gateway API (full size)

We’ll first proxy an external service that doesn’t provide its own TLS-certificate using an HTTPRoute. In the process we’ll also create a valid certificate for the service.

Next we’ll proxy an application that provides its own TLS certificate using a TLSRoute, assuming the application provides a valid certificate, or we decide to trust what it provides.

If you’re more keen on using the Ingress machinery instead, I can recommend Kris’ article on Using Kubernetes Service for Proxying to External Services. A third option is rolling your own DNS service like Pi-Hole, AdGuard Home, or Technitium, but that would only work when you’re connected to it.

This article relies on Cilium’s Gateway API implementation, but it should be easy to adapt to other Gateway API implementations.

Overview
#

I’ll assume you have some familiarity with the Gateway API for internal services, but if not, — and you want ot learn more about it, I’ve written an article on how to get started with Gateway API with Cilium and Cert-manager.

As an example, we’ll take a look at how to proxy a Home Assistant OS instance running on a separate machine in the same network and attach a certificate using Cert-manager.

If you’re trying to proxy Home Assistant, make sure to edit the Configuration.yaml file to add the IP of the Gateway service as a trusted proxy.

Some applications, such as Proxmox VE, refuse to do anything but HTTPS. In these cases we can let the application handle the certificate and just pass the connection through to the correct hostname.

For applications like this we can’t rely on Cert-manager, and instead have to get our hands dirty reading the documentation on how to configure valid certificates, or if we’re lucky find an article that explains how to do it, like this one for Proxmox VE by Derek Seaman.

flowchart LR subgraph Kubernetes Endpoint --> Service --> Route --> Gateway end Application -.-> Endpoint Gateway -.-> Browser

A rough sketch of the configuration we need to conjure is an endpoint (EndpointSlice/Endpoints) for the external application which is then routed to a Service. Next we connect the Service to a Route (HTTPRoute/TLSRoute) that gets picked up by a Gateway and exposed to our browser.

Service without selectors
#

To get Kubernetes to talk with external services we can create a Service without selectors, which is exactly what it sounds like

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
apiVersion: v1
kind: Service
metadata:
  name: home-assistant
  namespace: home-assistant
spec:
  ports:
    - name: http
      port: 80
      targetPort: 8123

Since the Service doesn’t have any selectors, the corresponding EndpointSlice and (legacy) Endpoints resources won’t be automatically created.

Depending on the Gateway Controller you’re using, you need to craft either the EndpointSlice- or Endpoints-resource yourself. I know that Cilium works with EndpointSlices, but Traefik needs an Endpoint-resource till this GitHub issue gets resolved, your mileage may vary.

By convention, the EndpointSlice name is prefixed with the corresponding Service name, but you’re free to choose whatever name as long as it’s namespace-unique for that resource type. To link the EndpointSlice to the Service you can set the kubernetes.io/service-name label equal the Service name (line 7).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: home-assistant
  namespace: home-assistant
  labels:
    kubernetes.io/service-name: home-assistant
    endpointslice.kubernetes.io/managed-by: cluster-admins
addressType: IPv4
endpoints:
  - addresses:
      - 192.168.1.81
    # https://github.com/argoproj/argo-cd/issues/15554
    conditions:
      ready: true
ports:
  - name: http
    port: 8123

The EndpointSlice should point to the external service we want to reach. In this case we’ve set the addressType type IPv4 and supplied an IP-address — preferably static, on line 12. In the case of Home Assistant we could’ve also set the addressTtype to FQDN (Fully Qualified Domain Name) and use homeassistant.local as the address. Last we supply the port used by the external application (line 18) and give it a name matching the port in the Service.

If you use Argo CD you can add line 14-15 to avoid a sync-loop while someone figures out this GitHub issue.

HTTPRoute
#

Up until now everything is similar to how you would expose an external service using an Ingress-resource. To use the Gateway API we instead create an HTTPRoute.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: home-assistant
  namespace: home-assistant
spec:
  parentRefs:
    - name: https-gateway
      namespace: gateway
  hostnames:
    - "home-assistant.<DOMAIN>"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: home-assistant
          port: 80

Line 8–9 refers to a Gateway resource which we’ll create shortly. On line 11 we specify the hostname which we want to reach our external resource on. Similar to an Ingress resource we need to reference which Service and port the HTTPRoute should route to (line 18–19).

Gateway
#

A key difference between the Ingress model and the Gateway API is separation of concerns. While routes are meant to be configured by application developers, Gateways are meant to be prepared by cluster operators.

To aid in this separation we can create our Gateway in a separate namespace to be used by different applications. In this example we’ve opened up for routes from all namespaces (line 23-24), though it’s possible to use a LabelSelector for more fine-grained control.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: https-gateway
  namespace: gateway
  annotations:
    cert-manager.io/issuer: cloudflare-issuer
spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
      io.cilium/lb-ipam-ips: 192.168.1.221
  listeners:
    - name: https-gateway
      protocol: HTTPS
      port: 443
      hostname: "*.<DOMAIN>"
      tls:
        certificateRefs:
          - kind: Secret
            name: cert-stonegarden
      allowedRoutes:
        namespaces:
          from: All

To hook up the exposed with a TLS-certificate through a configured Cert-manager instance we simply give the Gateway the correct annotation (line 7) and a TLS-certificate reference (line 20-21). I’ve detailed how to do this in Gateway API with Cilium and Cert-manager.

Next we can assign an IP to the Service spawned by the Gateway by giving it an infrastructure annotation (line 12) assuming you’ve set up L2 announcements similar to what’s done in this article. Other possibilities would be to use Metal LB.

For our HTTPRoute to be picked up by the Gateway it has to match a listener, in this case it should match the hostname given on line 17.

To check the status of the Gateway and attached routes you can run

kubectl -n <GATEWAY-NAMESPACE> get gateway <GATEWAY-NAME> -o json | jq '.status'

TLS Passthrough
#

In reverse this time, we’ll start with the Gateway

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: tls-passthrough-gateway
  namespace: gateway
spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
      io.cilium/lb-ipam-ips: 192.168.1.222
  listeners:
    - name: proxmox-tls-passthrough
      protocol: TLS
      port: 443
      hostname: "proxmox.<DOMAIN>"
      tls:
        mode: Passthrough
      allowedRoutes:
        namespaces:
          from: All

Here we’ve added a listener with the TLS protocol (line 13) and put the tls-setting in Passthrough mode (line 17). Make sure the hostname (line 15) matches the TLSRoute we’re about to create next

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: proxmox
  namespace: proxmox
spec:
  parentRefs:
    - name: tls-passthrough-gateway
      namespace: gateway
  hostnames:
    - "proxmox.<DOMAIN>"
  rules:
    - backendRefs:
        - name: proxmox
          port: 443

Here we’ve matched the TLSRoute with our Gateway by name and namespace (line 8-9) and hostname (line 11). Similar to the HTTPRoute we now have to create a matching headless Service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
apiVersion: v1
kind: Service
metadata:
  name: proxmox
  namespace: proxmox
spec:
  ports:
    - name: https
      port: 443
      targetPort: 8006

where we make sure to match the target port (line 10) of our EndpointSlice

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: proxmox
  namespace: proxmox
  labels:
    kubernetes.io/service-name: proxmox
    endpointslice.kubernetes.io/managed-by: cluster-admins
addressType: IPv4
endpoints:
  - addresses:
      - 192.168.1.42
    # https://github.com/argoproj/argo-cd/issues/15554
    conditions:
      ready: true
ports:
  - name: https
    port: 8006

Unless you’ve run into some issues you should now have TLS passthrough via the Gateway API!

Caveats
#

At the time of writing I haven’t gotten Gateways to play nice when combining HTTPS-protocol listeners and TLS-protocol listeners in TLS Termination mode.

When I tried, I was met with this error in the event log of the Gateway

Skipped a listener block: [
spec.listeners[1].tls.certificateRef:
  Required value: listener has no certificateRefs,
spec.listeners[1].tls.mode: 
  Unsupported value: "Passthrough":
    supported values: "Terminate"]

and all attached Gateway routes appears to be non-responsive.

I was eventually able to track down the message as coming from Cert-manager. This is caused by the annotation

cert-manager.io/issuer: cloudflare-issuer

which instructs Cert-manager to create a certificates for the Gateway listener hostnames. I’ve opened this issue with them as I think this is not the intended behaviour.

I’ve also had issues with TLSRoutes apparently attaching to HTTPS listeners with a matching hostname, even though the only supported kind for HTTPS-protocol listeners is supposed to be HTTPRoute. When this happens all the HTTPRoutes stop working and I have to delete the TLSRoute and recreate the Gateway for the HTTPRoutes to start working again.

I’ve enquired about this behaviour with the helpful folks at Cilium in this comment.

Until these issues are fixed — or someone can explain that I’m doing something wrong and I fix it, the workaround appears to be two separate Gateways. This is not ideal though, as the attached Services for the two Gateways necessarily get different IPs. This means you can’t route to just one IP without getting creative with a third Gateway, or some other proxy mechanism.

Summary
#

The resources for this article can be found in this GitLab repo.

If you’re using Kustomize — or possibly Argo CD with Kustomize + Helm, the above configuration can be summarised as

❯ tree
.
├── kustomization.yaml
├── gateway
│   ├── gw-https.yaml
│   ├── gw-tls-passthrough.yaml
│   └── kustomization.yaml
├── home-assistant
│   ├── endpoint-slice.yaml
│   ├── http-route.yaml
│   ├── kustomization.yaml
│   └── svc.yaml
└── proxmox
    ├── endpoint-slice.yaml
    ├── kustomization.yaml
    ├── svc.yaml
    └── tls-route.yaml
# kustomization.yaml
resources:
  - gateway
  - home-assistant
  - proxmox

Gateway
#

Take a look Gateway API with Cilium and Cert-manager for how to configure Cert-manager to provision TLS-certificates for Gateway resources.

# gateway/kustomization.yaml
resources:
  - gw-https.yaml
  - gw-tls-passthrough.yaml
# gateway/gw-https.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: https-gateway
  namespace: gateway
  annotations:
    cert-manager.io/issuer: cloudflare-issuer
spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
      io.cilium/lb-ipam-ips: 192.168.1.221
  listeners:
    - name: https-gateway
      protocol: HTTPS
      port: 443
      hostname: "*.<DOMAIN>"
      tls:
        certificateRefs:
          - kind: Secret
            name: cert-stonegarden
      allowedRoutes:
        namespaces:
          from: All
# gateway/gw-tls-passthrough.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: tls-passthrough-gateway
  namespace: gateway
spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
      io.cilium/lb-ipam-ips: 192.168.1.222
  listeners:
    - name: proxmox-tls-passthrough
      protocol: TLS
      port: 443
      hostname: "proxmox.<DOMAIN>"
      tls:
        mode: Passthrough
      allowedRoutes:
        namespaces:
          from: All

Home Assistant OS HTTP-passthrough
#

# home-assistant/kustomization.yaml
resources:
  - endpoint-slice.yaml
  - svc.yaml
  - http-route.yaml
# home-assistant/endpoint-slice.yaml
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: home-assistant
  namespace: home-assistant
  labels:
    kubernetes.io/service-name: home-assistant
    endpointslice.kubernetes.io/managed-by: cluster-admins
addressType: IPv4
endpoints:
  - addresses:
      - 192.168.1.81
    # https://github.com/argoproj/argo-cd/issues/15554
    conditions:
      ready: true
ports:
  - name: http
    port: 8123
# home-assistant/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: home-assistant
  namespace: home-assistant
spec:
  ports:
    - name: http
      port: 80
      targetPort: 8123
# home-assistant/http-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: home-assistant
  namespace: home-assistant
spec:
  parentRefs:
    - name: https-gateway
      namespace: gateway
  hostnames:
    - "home-assistant.<DOMAIN>"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: home-assistant
          port: 80

Proxmox TLS Passthrough
#

# proxmox/kustomization.yaml
resources:
  - endpoint-slice.yaml
  - svc.yaml
  - tls-route.yaml
# proxmox/endpoint-slice.yaml
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: proxmox
  namespace: proxmox
  labels:
    kubernetes.io/service-name: proxmox
    endpointslice.kubernetes.io/managed-by: cluster-admins
addressType: IPv4
endpoints:
  - addresses:
      - 192.168.1.42
    # https://github.com/argoproj/argo-cd/issues/15554
    conditions:
      ready: true
ports:
  - name: https
    port: 8006
# proxmox/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: proxmox
  namespace: proxmox
spec:
  ports:
    - name: https
      port: 443
      targetPort: 8006
# proxmox/tls-route.yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: proxmox
  namespace: proxmox
spec:
  parentRefs:
    - name: tls-passthrough-gateway
      namespace: gateway
  hostnames:
    - "proxmox.<DOMAIN>"
  rules:
    - backendRefs:
        - name: proxmox
          port: 443