Gain Central Visibility Across All Your AWS Cloud VMs With IPAM Repository

One of the main promises of the cloud native principle is elasticity. Workload should be able to benefit from a flexible infrastructure and scale whenever necessary. The primary and easiest component to scale up and down is the virtual machine, as most enterprises are still using servers. But with network teams simultaneously creating, modifying and destroying hundreds of VMS across multiple clouds in multiple locations, network administrators are quickly left with outdated status visibility and unmanageable management complexity. The obvious solution is to maintain a central repository offering global visibility.

Within this elastic infrastructure world, keeping a central repository like the IPAM up to date requires it to be able to adapt itself to each micro change occurring in the cloud. EfficientIP SOLIDserver provides this, thanks to its automation engine and performant open API, allowing it to react in real time to infrastructure changes.

Connecting SOLIDserver DDI solution to Amazon Web Services EC2 server virtualization engine is a simple and smart project, bringing immediate value to I&O teams. Every system connected to the DDI will always have accurate information from the IPAM and related records in the DNS. With SOLIDserver, we can also add valuable information in the Device Manager such as all the metadata associated with the EC2 instance. As well as easing understanding of the virtual servers running, this helps simplify initiatives like inventory or cloud cost analysis.

How does it work?

In order to enable the AWS EC2 event notification to the SOLIDserver, two main components are required in the design to: 1) Catch EC2 events and make them available 2) Collect events from the DDI and use them to keep up-to-date the IP “golden records” and related information. Both these parts are presented on the diagram below:

On the AWS side:

1. All events occurring on VMs hosted in the EC2 service are collected and pushed to the CloudWatch service (the internal event collector of AWS). Every event related to state change of virtual machines is collected. The events contains a virtual machine identifier which is unique and enables actions to potentially be performed on the VM, later in the process.
2. Within CloudWatch, a filter is applied to obtain specific states of the VM. This allows us to get information like the creation and the destruction of instances, as well as stop and resume.
3. CloudWatch pushes all filtered events towards a small piece of software hosted in the Lambda FaaS system.
4. The Lambda function is invoked – this software analyzes the event from EC2 in order to prepare it to be handled by the SOLIDserver.
5. Since the event does not carry all useful information, the Lambda function enriches the event message with information attached to the virtual machine, directly from the EC2 engine. This information will be seen as metadata in the DDI solution.
6. The Lambda function pushes the rich message related to the virtual machine state change towards a push-pull queue in the SQS service. This asynchronous method of exchanging information between AWS and SOLIDserver guarantees security and scalability. The queue can keep messages for some time in order to allow elasticity in the handling process. In this example, we have configured it to retain messages for one day.

On the SOLIDserver side:

7. The SOLIDserver function, named here AWS updater, asynchronously pulls the enriched events from the SQS message queue.
8. AWS updater uses the SOLIDserver APIs to update the IP Golden Records based on the information in the message. Device Manager is updated with information on the virtual machine itself, the network interfaces are used to link the device to its IP network in the IPAM. In the meantime, networks used in the VPC (main and subnets) are updated in the IPAM.
9. Thanks to the automation engine in the SOLIDserver DDI, updates are propagated to the appropriate components of the IPAM, the Device Manager and the DNS. From here, we can also imagine starting some external automation towards other components of the EfficientIP ecosystem like security filtering rules in the firewalls, whitelisting in the DNS engines or updating the CMDB repository for the ITIL process to perform its specific tasks.

All virtual machines in Device Manager are registered with some metadata collected during the enrich phase in the AWS Lambda function. This part of the ontology is designed in a specific metadata definition with the following information:

• Instance identifier: could be used to perform specific actions through the AWS API on the server itself
• Architecture of the VM: mainly x86 and ARM in EC2
• Number of CPU cores
• AWS EC2 instance type: like t2.micro, this is really useful for the cost tracking part
• Availability zone: in addition to the region where the virtual machine is running, this information provides the sub-region, which is useful for tracking redundancy of a service split across multiple servers that should be located in different AZs
• State of the virtual machine (can also be not running)
• Launch time: can be used to track costs or how horizontal elasticity is working for this server

How complex is it to set up?

Although it appears to be complex to set up, in reality it is fairly simple. Most of the components are provided by the AWS cloud solution and the SOLIDserver APIs and automation engine. Only a few lines of codes are required to implement this use case:

• Around 70 Python lines for the Lambda in AWS
• Around 325 Python lines for the AWS updater with updates on IPAM for networks and IP, and Device Manager, it uses the SOLIDserver python library with the advanced module

On the infrastructure part of the configuration for all the AWS components and related security, we used Hashicorp Terraform format with around 200 lines of HCL.

What is the value?

The kind of approach we have demonstrated here for AWS is table to be replicated to all the cloud providers used by I&O teams in order to bring:

• Near real-time update of the DDI on cloud compute engine changes
• Complete visibility of the cloud hosted virtual machines in the IPAM Device Manager
• A rich set of associated metadata on each component
• Visibility of all the virtual private networks used to connect the virtual machines (e.g. AWS VPC) and their IP addresses in the IPAM
• Ability to develop automation workflows based on the rich information collected and stored as IP Golden Records (inventory, whitelisting, filtering, billing…)

Demonstration video of above scenario