VXLAN allows us to create a logical topologies in the Data Center for our virtual machines. With VxLAN we are able to create a layer 2 network on top of layer 3 infrastructure. In the Data Centers VMs needs to be deployed in a specific segment of the network. These VMs can be moved from one host to another for maybe because of capacity problem or maybe host failure. In that case, It is clear that VM will keep its ip configuration, means it needs to be part of same broadcast domain even after it moved to another host which is probably connected to another access switch.
There are lot of answer for the question why not Vlan. 2 important answer is, you will need to take care of Spanning Tree issues with Vlans and the other answer maybe the limitation on Vlan count. Limitations of the number of the VLANs supported in the network is 4094. Scalability is an issue. VXLAN address this problem by increasing traditional VLAN limits from 4094 to 16 million. It uses a 24-bit segment identifier to scale beyond the 4096 limitations of VLANs. These are the main reasons VxLAN became a de facto standard for data center networks.
Actually Overlays have been used in data center environments only since 2008. Overlays in data center environments are referred to as network virtualization overlays.
A primary feature that all these network virtualization overlays exhibit is related to the concept of location and identity. Identity identifies an end host and could be its IP address, its MAC address, and so on. Location identifies the tunnel edge device that is responsible for encapsulating and decapsulating tunnel traffic for that end host. The end hosts could be virtual machines, bare-metal servers, containers, or any other workload. The outer header of the overlay references the source and destination locations, and the inner header references the source and destination end host identities.
With Overlay services defined along with the different data-plane encapsulations, a method of transport to move the data across the physical network is required. This method of transport is typically an underlay transport network.
Key Benefits of VxLan Overlay
- It does not depend on STP to converge the topology. You will not need to worry about or think about STP. Instead Layer 3 routing protocols are used as an underlay.
- No links within the fabric are blocked. All links are active and can carry traffic. In Spanning tree you will not have this option.
- The fabric can load balance traffic across all active links, ensuring no bandwidth is sitting idle.
- In the underlay you will have redundancy, high bandwidth etc. And if anything changes in your underlay, it will not affect your overlay.
In defining the underlay, we need to know the OSI layer where tunnel encapsulation occurs. In some sense, the overlay header type dictates the type of transport network. For example, with VXLAN, the underlying transport network (underlay) would be a Layer 3 network, which would transport the VXLAN-encapsulated packets between the source and destination tunnel edge devices. (VTEP: Virtual Tunnel Endpoint) Thus, the underlay provides reachability between the various tunnel/overlay edge devices.
In the typical VxLAN based Data center networks we have;
- Underlay with Layer 3 interface peering. In the underlay we can use any IGP or BGP. All the connections are based on Layer 3 and that’s why we are not going to be upset about spanning tree protocol.
- Edge devices which will initiate and terminate the overlay. Actually overlay edge devices doesn’t have to be network devices. We have also option to initiate overlay from servers which is called Host Overlay, but in my example I will go with network overlay.
- Endpoints which may be virtual or bare-metal servers.
Fundamentally, VXLAN provides mechanisms to tunnel multiple layer 2 networks across a Layer 3 IP based infrastructure. The VXLAN base use case is to connect two or more layer three network domains and make them look like a common layer two domain. In the Data center environment this allows virtual machines on different networks to communicate as if they were in the same layer 2 subnet.
Edge devices are the most important part of VxLAN overlay. Edge devices are connected to underlay with and ip interface. They responsible from VxLAN encapsulation and decapsulation. VxLAN tunnels doesn’t have state information in the core. Only the edge devices will be aware of these tunnels.
VxLAN Encapsulation
VxLAN is basically MAC in UDP encapsulation.
You may find all the details in RFC but most important part of VxLAN header is VxLan Network Identifier (VNI) field. This is a 24-bit value used to designate the individual VXLAN overlay. Devices or VMs in different VXLAN overlay networks cannot communicate with each other.
The UDP destination port number is 4789, and the source port number is obtained after the inner Ethernet packet header is hashed.
In the IP packet header, the source IP address and the destination IP address are the IP address of the local and remote VTEP of the VXLAN tunnel.
In the topology, we have 2 VTEPs and some VMs are connected these VTEPs. Note that this is very popular topology with 2 layers, Spine and Leaf. Between Spine and Leafs, the connections are IP based and IP connectivity between VTEPs can be provided by IGP or BGP.
According to this topology, assuming we have VxLAN tunnel between VTEPs, all VMs which are part of same VNI will be able to communicate.
