speedup iptables setup

[kollyma]

PR against 4.7 as discussed with Remi Bergsma. This will speed up the iptables creation on the virtual router.

Testing showed the following:
with current code:
root@kvm704:~# time /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vr_cfg.sh 169.254.1.176 -c /var/cache/cloud/VR-12f28879-de7e-44d2-8dbe-b93a04bd3ba4.cfg
real 2m56.401s
user 0m0.012s
sys 0m0.012s

modified version:
root@kvm704:~# time /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vr_cfg.sh 169.254.1.176 -c /var/cache/cloud/VR-12f28879-de7e-44d2-8dbe-b93a04bd3ba4.cfg
real 1m35.762s
user 0m0.020s
sys 0m0.004s

[cristofolini]

@kollyma I like what you did here. Besides apparently speeding up the iptables setup, the code looks more streamlined this way.
But it seems your build timed out. You might want to do another push later.

[kollyma]

@cristofolini : thanks! indeed the "Apache CloudStack Plugin - Quota Service" timed out. After reviewing the PR and the Jenkins output, I can not see any relations. Should I open another pull request with the same code and check if it works? Did I understand this correctly?

[swill]

@kollyma, maybe just add a comment at the top to mention something like "pre-fetch the variables so they are not fetched multiple times inline" or something like that and this should prompt another build without having to create a new PR. Since we can't close a PR without a merge, this is probably the simplest way to handle it. Good comments are rarely a bad thing.

[kollyma]

The third time is the charm,all checks have passed now! the first time Jenkins failed and strangely enough the second time Travis failed.

[milamberspace]

Good job. Thanks.
Works fine in my 4.8 test installation (real platform)
Several tests on RV and RVR routers with network restart with/without clean up
LGTM

[swill]

I am pretty comfortable with this PR given the scope of the change. Thanks @milamberspace for testing. Should this be committed against 4.7 and then forward merged? I am not entirely sure what the guidelines are for the types of changes that should be merged into previous releases. I know features should be against master, but I am not sure the details beyond that. @remibergsma do you have any guidance on this topic? Obviously the branch a PR it is opened against suggests a course of action, but I would like to have a clearer understanding of this.

[DaanHoogland]

@swill it is our responsibility as committers to make sure that it is merged against the right branch. if it is not a new feature and at the moment 4.7 is the branch to go for. merging forward to 4.8. and master/4.9 is the right procedure.

[resmo]

LGTM! please merge

[remibergsma]

@kollyma Please squash the commits.
@swill Let's run integration tests on this before merge.

[swill]

@remibergsma The test have already been started. :)

[swill]

CI Results

84/85 SUCCESSFUL

1 Failure (this and another similar test seem to fail consistently in my CI currently)

FAIL: Test redundant router internals
-------------------------------------
Traceback (most recent call last):
  File "/data/git/cs1/cloudstack/test/integration/smoke/test_routers_network_ops.py", line 483, in test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false
    "Attempt to retrieve google.com index page should be successful once rule is added!"
AssertionError: Attempt to retrieve google.com index page should be successful once rule is added!

Ran the following tests...

echo "Running tests with required_hardware=true"
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
smoke/test_password_server.py \
smoke/test_vpc_redundant.py \
smoke/test_routers_iptables_default_policy.py \
smoke/test_routers_network_ops.py \
smoke/test_vpc_router_nics.py \
smoke/test_router_dhcphosts.py \
smoke/test_loadbalance.py \
smoke/test_internal_lb.py \
smoke/test_ssvm.py \
smoke/test_vpc_vpn.py \
smoke/test_privategw_acl.py \
smoke/test_network.py

echo "Running tests with required_hardware=false"
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \
smoke/test_routers.py \
smoke/test_network_acl.py \
smoke/test_reset_vm_on_reboot.py \
smoke/test_vm_life_cycle.py \
smoke/test_service_offerings.py \
smoke/test_network.py \
component/test_vpc_offerings.py \
component/test_vpc_routers.py

Associated Uploads

test_network_1WIKLA:

• results.txt

test_vpc_routers_8ZWKPC:

• results.txt

Uploads will be available until 2016-06-02 00:00:00 +0000 GMT

Comment created by upr comment.

[swill]

The debut of upr, so let me know if you want me to change the format. Also if you want me to add more tests in future runs, please let me know...

Given these results and the fact that the one failure consistently fails in my environment, I think we are ready to merge...

[remibergsma]

@swill In the logs I see this:

{returns: [u'--2016-04-01 21:41:30--  http://www.google.com/', u'Resolving www.google.com... 2607:f8b0:4006:807::2004, 172.217.0.36', u'Connecting to www.google.com|2607:f8b0:4006:807::2004|:80... failed: Network is unreachable.', u'Connecting to www.google.com|172.217.0.36|:80... connected.', u'HTTP request sent, awaiting response... 302 Found', u'Location: http://www.google.ca/?gfe_rd=cr&ei=Cuv-Vo3NLuih8wfM-b6oAw [following]', u'--2016-04-01 21:41:30--  http://www.google.ca/?gfe_rd=cr&ei=Cuv-Vo3NLuih8wfM-b6oAw', u'Resolving www.google.ca... failed: Connection timed out.', u"wget: unable to resolve host address 'www.google.ca'"]}
root: DEBUG: Result from SSH into the Virtual Machine: [u'--2016-04-01 21:41:30--  http://www.google.com/', u'Resolving www.google.com... 2607:f8b0:4006:807::2004, 172.217.0.36', u'Connecting to www.google.com|2607:f8b0:4006:807::2004|:80... failed: Network is unreachable.', u'Connecting to www.google.com|172.217.0.36|:80... connected.', u'HTTP request sent, awaiting response... 302 Found', u'Location: http://www.google.ca/?gfe_rd=cr&ei=Cuv-Vo3NLuih8wfM-b6oAw [following]', u'--2016-04-01 21:41:30--  http://www.google.ca/?gfe_rd=cr&ei=Cuv-Vo3NLuih8wfM-b6oAw', u'Resolving www.google.ca... failed: Connection timed out.', u"wget: unable to resolve host address 'www.google.ca'"]

It resolved google.com but then gets redirected to www.google.ca which it cannot resolve due to a timeout. Looks like a local issue, indeed. In a newer version of the test we are testing the public gateway instead but that hasn't been merged / sent as PR I guess.

[swill]

@kollyma Can you squash the commits for me and I will then merge this in. Thanks...

[kollyma]

@swill: tried to squash the commits shortly from holidays. Did not worked as expected, messed up my PR. I need to debug this when I am back next week...

[kollyma]

During commit squashing something went wrong.. i could not revert. Here a new Pull request with exactly the same changes. Sorry for any inconvenience caused.. and thanks for your "LGTM"
#1487

[DaanHoogland]

@kollyma can you close this one if #1487 is indeed the same?