Looking for recommendations for a LB with K3s for our env

[davehouser1]

Our env:

• 4 nodes.
• Nodes are on different subnets.
• Nodes are all on prem, not cloud based.
• We have no control over the network, so BGP configurations are not an option.
• I have no pool of ips I can use, would need to use host ips if possible.

Requirements:

• Deploy pods listening on port 6514, and have a load balancer in front to distribute load across nodes.
• If running an external load balancer want to run it on the master node as a single LB in a systemd service (like haproxy)

I have researched the following:

• service-lb (Klipper?): Out of the box this does not work, as we are on prem, and on different subnets. Not sure this can be configured to poke a hole into the host's network to listen on a port. Tired testing with it does not seem to work.
• Metal-lb: We want to use this, but for not its not an option. L2 is out, because of different subnets. and BGP is not an option as we dont have control over the network.
• External-lb (haproxy, nginx, etc.): Not sure I understand how this works. Would service-lb be disabled? I read though this documentation. I have deployed service-lb to use the host's IPs for the EXTERNAL-IP, but I dont see any of the hosts listening on the port. So its not clear what the backend server configuration should be for haproxy. Is this happening because I didn't deploy the agents or master nodes with K3S_TOKEN=lb-cluster-gd?

I know our env is limited. I also know the real answer is "Get Metal-Lb set up properly", but for now is there a way to use haproxy or have service-lb be able to expose ports on all the nodes to pass data to from haproxy?

[brandond]

I have no pool of ips I can use, would need to use host ips if possible.
service-lb (Klipper?): Out of the box this does not work, as we are on prem, and on different subnets.

Why doesn't it work? All it does is use the host IPs as loadbalancer addresses, as you asked for. What does this has to do with being on different subnets?

You need to either use the host IPs, OR use a virtual IP. You can't do both. Any "real" loadbalancer will use a Virtual IP, or an IP that floats between dedicated loadbalancer hosts. If you want to use a virtual/floating IP, your nodes or loadbalancers need to all be on the same subnet, or you need to be able to integrate with upstream routers via BGP or some other routing protocol.

You have a networking problem, not a Kubernetes problem.

[davehouser1]

Thanks @brandond for the details.
So will service-lb work with nodes on different subnets with our BGP integration or even setting up static routes?

[brandond]

without, yes.

Note that its not a real loadbalancer though. It works like one as far as Kubernetes is concerned, in that it will allow loadbalancer services to be provisioned, and will set all the correct status fields, but you still have to figure out how get clients connected to your node IPs. But that's a problem you would also need to solve with a loadbalancer if you were dynamically provisioning VIPs.

[davehouser1]

@brandond, yes, I understand it wont be a true load balancer, but for our testing that is ok.
To be clear, I am a novice at all this (if it isn't complete obvious already) so I appreciate your patience and time to help us out.
We were able to deploy our pods, and attach to a service-lb instance. Here is the output I see for the services:

NAMESPACE     NAME             TYPE           CLUSTER-IP      EXTERNAL-IP                                                   PORT(S)                      AGE
default       kubernetes       ClusterIP      10.43.0.1       <none>                                                        443/TCP                      7d
filebeat      filebeat         LoadBalancer   10.43.236.73    192.168.117.22,192.168.117.26,192.168.199.27,192.168.199.29   6514:31378/TCP               18s
kube-system   kube-dns         ClusterIP      10.43.0.10      <none>                                                        53/UDP,53/TCP,9153/TCP       7d
kube-system   metrics-server   ClusterIP      10.43.222.200   <none>                                                        443/TCP                      7d
kube-system   traefik          LoadBalancer   10.43.255.247   192.168.117.22,192.168.117.26,192.168.199.27,192.168.199.29   80:31032/TCP,443:31245/TCP   7d

But when I check all the nodes, none of them have the port exposed for 6514. This I believe is expected. The ports are only exposed on the cluster-IP. All of those EXTERNAL-IPs are each of the node's host ips.
Is there a way to allow the service-lb to listen on 6514 on one of the host nodes and have it LB to the the internal pod network to pass to all our pods on different nodes? Would I need an IngressClassName to accomplish this?

Note: subnets of nodes are 192.168.117.0/24 and 192.168.199.0/24.