Port forwarding to VIP in guest network

[bradh352]

My network setup is such that all Instances are isolated within a VPC with micro-segments (network tiers) per service offering.

My setup is pretty basic in the fact that I'm not using any external hardware integration (e.g. firewalls or load balancers), and strictly relying on the services provided by the virtual router.

Each segment will have multiple Instances of a given service for high availability, but there are cases where the load balancer provided by cloudstack's virtual router isn't sufficient. Examples include instances where services need to be accessed internally (tier2tier) as well as from public. Or when a service isn't supported by the virtual router load balancer (e.g. DNS via UDP).

So the use case is I assign another private ip address as a virtual IP in the same subnet as the instances providing the HA service. This may be something as simple as a floating virtual ip across those machines using something like keepalived, or it could be something more complex like a load balancer doing LVS Direct Routing (DR).

My question is, how do I forward ports to an ip address not directly assigned to an instance?

As far as I can tell, the UI can't do anything like that. The best I've found is in the API documentation addIpToNic to add a secondary ip address to one of my VMs. Then call createPortForwardingRule and specify the secondary ip address as the vmguestip.

I found where in the UI it says the secondary ip isn't automatically associated with the VM (but interestingly it doesn't say that in the API docs):

So this in theory could work for my use case if there aren't restrictions on other Instances also using this IP. But even so, this creates an issue with this secondary ip being associated with just one instance, such as what happens if this instance needs to be rebuilt? Wouldn't that effectively disable the port forward since the secondary ip address is no longer assigned anywhere even though there are other active instances that are providing this?

What do other people do here? This doesn't sound like that an unusual of a thing that people would normally do. Maybe I'm just missing something.

I'm using the terraform cloudstack provider, but can handle using the API or extending the provider if needed.

[luganofer]

Hi @bradh352...

The use of floating IP addresses or VIPs (such as those used in the case of Haproxy+Keepalived) is a very typical scenario in cloud deployment, but in CloudStack it is not covered ‘out of the box’. The workaround used is as you mentioned: with a secondary IP on one of the VMs that will use the floating IP and directing traffic to this IP. If the VM with the floating IP goes out of service (or any other problem occurs), Keepalived takes care of moving the floating IP to your other VM(s) and the connection flow will continue to function correctly (of course, this is not solved by CloudStack but by the software layer in the VM, which in this example is Keepalived).
I have many customers who use this scheme successfully.

[bradh352]

I guess my concern is what happens if I need to respin the Instance where the secondary ip is associated. By associated, I'm talking about associated in cloudstack for its rules. If that Instance goes away, so does the port forwarding, so even though internally that ip still exists, the forwarding rule will be purged. Is there a solution for that?

[weizhouapache]

I guess my concern is what happens if I need to respin the Instance where the secondary ip is associated. By associated, I'm talking about associated in cloudstack for its rules. If that Instance goes away, so does the port forwarding, so even though internally that ip still exists, the forwarding rule will be purged. Is there a solution for that?

Not a real solution:

You can create a dummy vm, stop it and enable delete protection (introduced in ACS 4.20)

[luganofer]

I suppose it depends on what exactly you mean by ‘respin the instance’. If you need to reinstall the VM from a template, you can use the virtualmachine restore API (https://cloudstack.apache.org/api/apidocs-4.19/apis/restoreVirtualMachine.html), which allows you to keep everything related to PF rules, LBs rules, and Static NAT intact without any modifications throughout the process. If you need to delete the VM that has the secondary IP that functions as a floating IP before reinstalling it, I understand that you should first manually modify the haproxy settings so that the connection flow is maintained on the other VMs. Then reinstall the VM and configure the secondary IP and PF, LBs or Static NAT rules again in the same way you did the first time.

[bradh352]

@luganofer in this case by respin, lets say I'm going from Ubuntu 22.04 to Ubuntu 24.04, so it needs to use a different template. When I make this change in terraform, it destroys the existing Instance and spawns a new one. Which means it also destroys the secondary ip assigned to the instance and the port forwarding rule, thus causing an outage even though the floating VIP is actually active, there's no port forward from the outside world anymore.

@weizhouapache did provide a hack here, unfortunately the terraform provider doesn't give me access to enable the protection flag.

I think maybe its a better idea to open an issue with this concern to come up with a proper solution. I'm not sure what might be accepted.