VR DHCP lease file not persistent

[DagSonsteboSB]

ISSUE TYPE

• Bug Report - BLOCKER
• Improvement request

COMPONENT NAME

• VR dnsmasq DHCP

CLOUDSTACK VERSION

• ACS 4.11.2
• ACS 4.12 not tested but assumed issues are still present looking at code changes
• Regressions from / previously present in ACS 4.9 / 4.10 / 4.11

REFERENCES

1. PR2427 - CLOUDSTACK-10252: Delete dnsmasq leases file on restart / CLOUDSTACK-10252: Delete dnsmasq leases file on restart #2427
2. Old Jira issue: VR DNSmasq lease file not persistent across DHCP actions
   / https://issues.apache.org/jira/browse/CLOUDSTACK-10204
3. Old Jira issue: New VR after network restart with cleanup is missing DNS and DHCP entries
   / https://issues.apache.org/jira/browse/CLOUDSTACK-10167
4. Finite lease introduction in Fixes for VirtualRouters in Basic Networking, especially with mutliple ranges in VLANs #1547
5. Also check related issue VR DHCP VPC multihomed VM broken #3273

CONFIGURATION

• ACS4.11.2
• Isolated guest networks and VPC networks

OS / ENVIRONMENT

• Any

SUMMARY

1. Leases file is not persistent across network operations:

• Refer to ref fix rpm/deb build error #2 above.
• Leases continues to be not persistent across dnsmasq restarts / additions - i.e. leases file is wiped and leases are not remembered.
• This causes the guest VM dhcp client to get an "NAK" DHCP deny on DHCP renewal, and have to go through the normal procedure again of DISCOVER>OFFER>REQUEST>ACK. For some guest OS versions (RedHat has shown this) where static routes are used the initial "NAK" will break these static routes, leading to problems in the guest OS, i.e. the VM will get a new lease but will have lost it's static routes.

---

2. Improvement request: Lease time for DHCP options is still hard coded to randint(700,760) hours:
   https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/cs/CsDhcp.py#L145

• Since this ends up in the packaged systemvm.iso file dynamically managing this might be a problem
• When looking at this issue could 5 mins be spent considering if the lease time could be put in as a global setting

---

STEPS TO REPRODUCE

1. Leases file is not persistent across network operations:

• Create VM
• Check contents of /var/lib/misc/dnsmasq.leases
• Create second VM
• Observe contents of /var/lib/misc/dnsmasq.leases is wiped
• Now wipe the lease file on the first VM and restart network
• Observe the /var/log/dnsmasq.log shows entries for a full DISCOVER>OFFER>REQUEST>ACK process rather than a renew

---

EXPECTED RESULTS

• Leases persistent throughout lifetime of VM

ACTUAL RESULTS

• Leases not persistent causing static route problems on guest VMs

[rohityadavcloud]

Interesting discussion https://serverfault.com/questions/723292/dnsmasq-doesnt-automatically-reload-when-entry-is-added-to-etc-hosts

[gildegoma]

FYI: I did some validation tests against CloudStack 4.11.3.0-RC1, and can confirm that this issue is now elegantly resolved 👍

❤️ ❤️ ❤️

[andrijapanicsb]

Not really... (partially solved i.e. we don't wipe any more the dnsmasq.leases file - we just remove the specific IP/MAC settings from the CONFIGURATION files for dnsmasq, BUT at the wrong point in time - causing other issues). This is solved now in master - i.e. we delete specific IP/MAC CONFIGURATION only when expunging the VM (i.e. not while stopping the VM - which is the current impemenation in 4.11.3 and 4.13)

Outstanding issues: restarting Network with CleanUP - inside the new VR that is created during network restart, dnsmasq.leases file is absent - this will cause clients to receive DHCPNAK when they try to renew the lease with DHCPREQUEST. Client's will then fallback to the "initialisation process" with DHCPDISCOVER and get's it's IP address - no downtime from IP connectivity point of view - i.e. the client will NOT lose it's IP address at any point in time.

BUT, on some RHEL system, this DHCPNAK will cause RHEL system to drop its routes... thus the problem. This is an edge case.

We yet need to solve that the new VR (that is created during network restarts with CleanUp) to also get the dnsmasq.leases file regenerated somehow (if possible)....
Current 4.11.3 and 4.13 are causing issues as following:

• VMware DRS - sometimes on busy clusters, the powerON report (from vSphere to ACS) will come a bit too late after the previous PowerOFF report (those are generated while LIVE migrating VMs via vSphere as opposed to via ACS) - and then ACS will say "ok, VM is off - let me remove it's IP/MAC addresses from the related CONFIGURATION files for dnsmasq) - and this will again lead to VM being unable to renew its IP address eventually.

Hope that clarifies the current situation with 4.11.3/4.13 - it's good to be documented somewhere, thus my looooooong comment here.