VR DHCP VPC multihomed VM broken

[DagSonsteboSB]

ISSUE TYPE

• Bug Report - BLOCKER

COMPONENT NAME

• VR dnsmasq DHCP

CLOUDSTACK VERSION

• ACS 4.11.2
• ACS 4.12 not tested but assumed issues are still present looking at code changes
• Regressions from / previously present in ACS 4.9 / 4.10 / 4.11

REFERENCES

1. PR2427 - CLOUDSTACK-10252: Delete dnsmasq leases file on restart / CLOUDSTACK-10252: Delete dnsmasq leases file on restart #2427
2. Old Jira issue: VR DNSmasq lease file not persistent across DHCP actions
   / https://issues.apache.org/jira/browse/CLOUDSTACK-10204
3. Old Jira issue: New VR after network restart with cleanup is missing DNS and DHCP entries
   / https://issues.apache.org/jira/browse/CLOUDSTACK-10167
4. Finite lease introduction in Fixes for VirtualRouters in Basic Networking, especially with mutliple ranges in VLANs #1547
5. Also check related issue VR DHCP lease file not persistent #3272

CONFIGURATION

• ACS4.11.2
• Isolated guest networks and VPC networks

OS / ENVIRONMENT

• Any

SUMMARY

VPC multihomed VMs DHCP partially broken:

What still works:

• VM with isolated network primary NIC and shared secondary NIC seems to work OK, VM gets both DHCP leases and Default GW appears on primary NIC only as expected. Checking /etc/dhcphosts.txt on secondary NIC shared network VR shows the DHCP address is tagged (set) and DHCP options set as per /etc/dhcpops.txt.
• VM with two isolated networks attached works the same way - both primary NIC and secondary NIC get DHCP leases, secondary NIC DHCP (on separate VR) sets the right tags and ensures no DG and no DHCP options 3/6/15.
• VM with primary VPC tier and secondary NIC on isolated network - works as above.
• VM with primary VPC tier and secondary NIC on shared network - works as above
• VM with primary isolated network and secondary NIC on VPC tier - works as above.
• VM with primary shared network and secondary NIC on VPC tier - works
• VMs with primary and secondary NIC on two different VPCs (actual VPCs, not tiers) - also works OK, secondary NIC again not set up with DG etc.

What is very broken:

• VM with multiple NICs attached to different VPC tiers in the same VPC fail to get DHCP address on it's primary NIC.
• General file / setting management around this also not right - e.g. the cloud.conf file ends up with duplicate entries

STEPS TO REPRODUCE

VPC VR tier VMs not getting DHCP leases:

• VPC created: super CIDR is 10.21.0.0/16, tier1 is 10.21.1.0/24, tier2 is 10.21.2.0/24

• VM created and attached to one tier1 only.

• VM at this point gets DHCP address OK on eth0

• VR files
  root@r-339-VM:# cat /etc/dhcphosts.txt
  02:00:2c:78:00:01,10.21.1.33,CROSSTIER1,723h
  root@r-339-VM:# cat /var/lib/misc/dnsmasq.leases
  1557669293 02:00:2c:78:00:01 10.21.1.33 CROSSTIER1

• Second VPC tier2 added to VM, all lease files removed and VM restarted

• VM comes online with no DHCP lease for eth0 but dhcp lease for eth1.

• /etc/dhcphosts is found to be empty:
  root@r-339-VM:~# cat /etc/dnsmasq.d/dhcphosts.txt

• Leases file show only the eth1 lease - BUT note this is not marked ("set") as being a secondary NIC:
  root@r-339-VM:~# cat /var/lib/misc/dnsmasq.leases
  1557806022 02:00:5e:96:00:03 10.21.2.238 CROSSTIER1

• Note /etc/dhcpopts.txt is populated as it should be when a NIC is secondary - although there is no tag matching that will make this active:
  root@r-339-VM:~# cat /etc/dhcpopts.txt
  10_21_2_238,3
  10_21_2_238,6
  10_21_2_238,15

• Probably unrelated (unknown what this file is used for) but the /etc/dnsmasq.d/dhcphosts.txt file is also empty:
  root@r-339-VM:~# cat /etc/dnsmasq.d/dhcphosts.txt

• Checking the /etc/dnsmasq.d/cloud.conf file which contains the defaults for each tier it is however found this is now double populated for each of the two tiers:
  root@r-339-VM:~# cat /etc/dnsmasq.d/cloud.conf
  dhcp-hostsfile=/etc/dhcphosts.txt
  dhcp-range=set:interface-eth2-0,10.21.2.1,static
  dhcp-option=tag:interface-eth2-0,15,cs2cloud.internal
  dhcp-option=tag:interface-eth2-0,6,10.21.2.1,8.8.8.8,8.8.4.4
  dhcp-option=tag:interface-eth2-0,3,10.21.2.1
  dhcp-option=tag:interface-eth2-0,1,255.255.255.0
  dhcp-range=set:interface-eth3-1,10.21.1.1,static
  dhcp-option=tag:interface-eth3-1,15,cs2cloud.internal
  dhcp-option=tag:interface-eth3-1,6,10.21.1.1,8.8.8.8,8.8.4.4
  dhcp-option=tag:interface-eth3-1,3,10.21.1.1
  dhcp-option=tag:interface-eth3-1,1,255.255.255.0
  dhcp-range=set:interface-eth3-0,10.21.1.1,static
  dhcp-option=tag:interface-eth3-0,15,cs2cloud.internal
  dhcp-option=tag:interface-eth3-0,6,10.21.1.1,8.8.8.8,8.8.4.4
  dhcp-option=tag:interface-eth3-0,3,10.21.1.1
  dhcp-option=tag:interface-eth3-0,1,255.255.255.0
  dhcp-range=set:interface-eth2-1,10.21.2.1,static
  dhcp-option=tag:interface-eth2-1,15,cs2cloud.internal
  dhcp-option=tag:interface-eth2-1,6,10.21.2.1,8.8.8.8,8.8.4.4
  dhcp-option=tag:interface-eth2-1,3,10.21.2.1
  dhcp-option=tag:interface-eth2-1,1,255.255.255.0

• Logs show eth1 not being present in host file:
  Apr 12 11:52:41 dnsmasq-dhcp[5302]: DHCPDISCOVER(eth3) 02:00:2c:78:00:01 no address available
  Apr 12 11:52:44 dnsmasq-dhcp[5302]: DHCPDISCOVER(eth3) 02:00:2c:78:00:01 no address available
  Apr 12 11:52:52 dnsmasq-dhcp[5302]: DHCPDISCOVER(eth3) 02:00:2c:78:00:01 no address available
  Apr 12 11:53:07 dnsmasq-dhcp[5302]: DHCPDISCOVER(eth3) 02:00:2c:78:00:01 no address available
  Apr 12 11:53:24 dnsmasq-dhcp[5302]: DHCPDISCOVER(eth3) 02:00:2c:78:00:01 no address available
  Apr 12 11:53:42 dnsmasq-dhcp[5302]: DHCPDISCOVER(eth2) 02:00:5e:96:00:03
  Apr 12 11:53:42 dnsmasq-dhcp[5302]: DHCPOFFER(eth2) 10.21.2.238 02:00:5e:96:00:03
  Apr 12 11:53:42 dnsmasq-dhcp[5302]: DHCPREQUEST(eth2) 10.21.2.238 02:00:5e:96:00:03
  Apr 12 11:53:42 dnsmasq-dhcp[5302]: DHCPACK(eth2) 10.21.2.238 02:00:5e:96:00:03 CROSSTIER1

EXPECTED RESULTS

• VM getting DHCP lease on all interfaces, with secondary NIC not getting DG etc.

ACTUAL RESULTS

• VM getting DHCP lease on eth1 only, not on eth0

[rohityadavcloud]

Reproduced on latest 4.11

[rohityadavcloud]

Fixed and merged in #3373