-
Notifications
You must be signed in to change notification settings - Fork 1
Micro service location services documentation
Eureka is a service used for discovering application service instances which are then used for load balancing and fail-over. Eureka is made up of 2 key components, a client and a core server implementation. The server implementation uses the client to manage the connections to other instances of the eureka service. There are many differences between the networking configuration of SoftLayer and the Amazon EC2 cloud, and we aimed to exploit the strengths of the SoftLayer network infrastructure.
During startup of the eureka server in the EC2 infrastructure, the available Elastic IP addresses (EIP) are discovered using the Amazon API, and compared to registered eureka services to discover which EIPs have yet to be bound.
Today SoftLayer supports programmatically updating a 'GlobalIP' address (one that is routed between datacenters). We avoided the use of this feature, as it can introduce an extra point of failure in this application scenario. We instead chose to use a feature called a 'portable subnet'. This is a defined IP block, which can be used by multiple servers on a VLAN.
We defined a portable subnet, for a VLAN in each datacenter. Then for each server instance we create a network script to register the server with an IP address allocated from the portable subnet.
[root@host ~] cat /etc/sysconfig/network-scripts/ifcfg-eth0-range1
IPADDR_START=10.80.194.122
IPADDR_END=10.80.194.122
CLONENUM_START=1
NETMASK=255.255.255.248
GATEWAY=10.80.194.121
This configuration gives us an eureka server running at a well known IP address, which can then be found through DNS.
We created a DNS entry for each eureka server portable subnet IP address, with the convention eureka-dddd-v-i where dddd is the datacenter, v is the VLAN, and i is an instance within the VLAN.
We then created a DNS entry for each datacenter. Neither of the two DNS providers we had access to, allowed configuration of a DNS entry with multiple TXT entries. We instead used a space delimited list, which meant the Java DNS context resolver enclosed the string in double quotes. This required a change to DiscoveryClient to remove double quotes from the string. As double quotes and spaces can not be part of a URL, the change is a safe change to pull into the base Netflix implementation, if necessary.
Finally, we created a DNS entry for a region, which listed the datacenter within that region.
At this point we could use the DNS Text record approach to high availability documented in the Eureka configuration wiki (See Configuring EIPs using DNS).
As an example, here is what we stored in DNS for our region, one of our availability zones and one of the eureka servers.
$ORIGIN us-1.flyacmeair.net.
txt 60 IN TXT dal05.flyacmeair.net sjc01.flyacmeair.net
$ORIGIN dal05.flyacmeair.net.
txt 480 IN TXT eureka-dal05-1-1.flyacmeair.net eureka-dal05-1-2.flyacmeair.net
$ORIGIN eureka-dal05-1-1.flyacmeair.net.
@ 86400 IN A 10.80.194.122
Instances within the SoftLayer cloud can have two network interfaces. One is privately accessible by machines within the same VLAN and other VLANs associated with the account. The other is a publicly accessible interface and IP. The hostname and domain associated with an instance at creation are only resolvable within that instance. A eureka client, tries to register it's hostname as the target for a service and therefore doesn't work within SoftLayer. We have seen similar issues on Amazon EC2 instances when they are allocated domU DNS hostnames.
We therefore implemented a change to AbstractInstanceInfo and PeerAwareInstanceRegistry to always use the IP address of the eth0 interface. Later we may make this more configurable. Finally, we added logic to compare the ip addresses as well as hostnames in order to detect if an instance is a local instance.
Eureka only recognizes three types of datacenters, and has special behavior defined for the Amazon type. It is not necessary (yet) to introduce a 4th type but any future enhancements for integration with SoftLayer will probably require it. We don't believe this is a major change.