VMware ESXi Networking Concepts
For connecting physical servers and computers to a network, you need physical network adapters, switches, and routers. With virtual machines, virtual network concepts are used for communication between the different components of an infrastructure. The proper configuration of ESXi networking on a host is critically important to the configuration of any ESXi environment.
Generally, ESXi host networks include storage, vMotion, VM, and management networks. These are generally the basic networks configured in order to have a functioning ESXi cluster that is integrated with the vCenter Server.
In this post, we take a look at the VMware networking concepts and components used to configure networks in VMware vSphere.
Virtual Switches (vSwitches)
VMware vSwitch is the key component enabling communication and data transfers between VMs in a VMware environment. A vSwitch is exactly what the name says: a virtual switch that is a software analog of a physical switch. Though not a hardware device, a virtual switch passes traffic using the same concepts used in the OSI networking model as physical switches. In other words, layer 2 frames (OSI data link layer) enter and exit the vSwitch the same way they would a physical switch.
The difference between vSwitch and physical switch networking in the context of VMware is the absence of a table of MAC addresses for devices that the vSwitch does not directly know about, unlike physical switches. vSwitch only knows and cares about the MAC addresses for VMs and VMkernel port devices that are directly connected to the virtual switch. The vSwitch will not flood unknown frames out of all ports. Unknown frames are simply dropped by the vSwitch. Unknown MAC address destinations originating from a VM connected to the vSwitch are sent to the vSwitch physical uplink.
One virtual switch (vSwitch0) is created by default when you install ESXi.
A virtual switch is a piece of software emulating an L2 network device. Virtual switches are used for connecting virtual machines to the network and communicating with each other, with physical networks, and with external hosts. Next, let’s go over the elements used together with a virtual switch for VMware networking.
vSwitch types
In vSphere, there are two types of virtual switches that can be provisioned, the vSphere Standard Switch (vSS) and Distributed Switch (vDS). The vDS requires Enterprise Plus licensing to be available and is configured at the vCenter level. For the purposes of this post, we use vSS since they span the full range of licensed offerings from VMware and they don’t require vCenter Server.
Virtual Ports
Virtual ports on the vSwitch act like the ports on a physical switch. Each VM that is configured with a virtual network adapter uses one of these virtual ports on the vSwitch. Also, there are VMkernel ports, which are a special kind of ports that talk directly to the ESXi host’s VMkernel and are required to make storage, vMotion, and management network traffic possible.
VMkernel
VMkernel is a specialized POSIX-like (Portable Operating System Interface) operating system in VMware ESXi that provides a layer of abstraction between the virtualized hardware and the physical hardware. We can say that VMkernel is an OS inside ESXi. Virtual machines use VMkernel to communicate with an ESXi host.
VMkernel is responsible for providing various services to virtual machines (VMs) running on the VMware ESXi hypervisor, including memory management, process scheduling, and I/O (input/output) management. It also provides network and storage connectivity to the VMs.
One of the key features of VMkernel is its ability to virtualize physical hardware resources, such as CPU, memory, and storage. This allows multiple VMs to share the same physical resources, enabling higher utilization rates and greater flexibility in resource allocation. VMkernel is required for Distributed Resource Scheduler, which is a load balancing feature in vSphere.
Physical Uplinks
Virtual Switch traffic is uplinked on each ESXi host by the physical network cards in the ESXi host that are bound to that vSwitch in VMware. This provides the traffic path for traffic entering and exiting the vSwitch. The real physical interface on the ESXi host is mapped to a vmnic in ESXi. These mappings can be seen when managing the networking adapters on the ESXi host.
Note the physical adapters assigned to the vSwitches (see below).
If VMs reside on different ESXi hosts, the virtual switches must be connected to a physical network using physical adapters (uplinks) and the network traffic must pass physical network components between the ESXi hosts. You can connect one or more physical network adapters of an ESXi server to a virtual switch.
With two or more physical network interface controllers (NICs), also known as network adapters, you can use NIC teaming (link aggregation) to increase network bandwidth when connecting from multiple network devices to an ESXi host and VMs residing on this host. The Link Aggregation Control Protocol (LACP) can be used.
VMware Port Groups
Port groups in VMware environments allow you to logically carve up virtual ports that are available on a particular vSwitch. We can apply traffic policy rules at the port group level: security rules and traffic shaping. Port groups are where you can also assign VLANs to your traffic. Below, take note of the configured port group called TestPortGroup.
A virtual switch can contain multiple port groups.
The two types of port group in VMware ESXi can be classified as follows:
- Virtual machine port groups – for connecting virtual machines to a virtual switch.
- VMkernel port groups – for connecting a VMkernel interface to a virtual switch.
VLANs
VLANs are virtual networks based on the IEEE 802.1Q standard that logically separate traffic over the same physical infrastructure. VLANs allow for segmenting networks into different pieces without having to physically separate networks. This is made possible by an additional VLAN tag being sent within the Ethernet frame header, which identifies which VLAN the traffic belongs to. This is often done for security and performance reasons. As the network becomes larger, broadcast traffic can grow exponentially. VLANs segment broadcast domains so that unwanted broadcast traffic is contained within a logical network.
In terms of security, if you want to make sure that certain traffic is inaccessible from other parts of the network, then you can assign that traffic to its own VLAN. This isolates it from other VLANs on the production network allowing you to segment sensitive or private traffic from regular production traffic.
Going back to our vSwitch, there are three ways we can set VLAN tags:
- Virtual switch tagging (VST) is setting VLAN tags at the port group level. This means setting the VLAN tag at the vSwitch level on the port group. This way the vSwitch is managing VLAN tags. This is perhaps the easiest and preferred way to introduce VLANs into a virtual environment.
- External switch tagging (EST) is setting VLAN tags on a physical switch. The physical switch is responsible for managing VLAN tags. In many VMware environments, if you have a dedicated storage switch, the ports will be set in untagged mode. This means that the physical switch is doing all of the tagging operations, and the virtual switch is not aware of this.
- Virtual guest tagging (VGT) allows the virtual guest VM to tag traffic and pass this along to the vSwitch, which then passes the VLAN-tagged frame up to the physical switch. This is the least common use case.
vCenter Server
VMware vCenter is the core product required to turn on the enterprise features. This includes, but isn’t limited to, the following:
- Centralized management, including Single Sign On (SSO) authentication
- Centralized permissions management
- Centralized logging of tasks and events
- Resource management
- Central location to perform management and administrative tasks across ESXi hosts
- Distributed virtual switch
- High Availability
- Distributed Resource Scheduling
- Fault Tolerance
- Centralized updating via the update manager
Note: vCenter Server is also needed to unlock additional modules that one may wish to take advantage of in a VMware ecosystem, such as VMware NSX, VSAN, and others. VMware NSX provides more advanced networking options for virtual VMware networking than for standalone ESXi hosts and ESXi hosts managed by vCenter.
ESXi Networking Best Practices
Knowing the main VMware networking concepts, it would be useful to take into account the VMware networking best practices when designing and configuring the networking components in a VMware ESXi environment.
Plan your network architecture. Before deploying an ESXi host, it’s important to plan your network architecture. Determine how many physical network adapters you need, how you will segment your network, and how you will handle network redundancy and failover.
Use redundant physical network adapters. ESXi hosts should be equipped with at least two physical network adapters to provide redundancy and failover. Ideally, these adapters should be connected to separate physical switches to provide redundancy at the switch level as well.
Use VLANs for network segmentation. VLANs allow you to segment your network and isolate traffic for security or performance reasons. It’s best to use separate VLANs for management, VM traffic, and storage traffic. Read also how VXLAN can be useful for virtual networks.
Configure network redundancy. VMware vSphere offers several options for network redundancy, including NIC teaming and link aggregation. These technologies can help ensure that your network remains available even if a network adapter or switch fails.
Use distributed virtual switches (DVS). DVS is a VMware vSphere feature that allows you to manage network configuration across multiple ESXi hosts from a central location. This makes it easier to manage and configure networking in large virtual environments.
Use Network I/O Control (NIOC). NIOC is a feature that allows you to prioritize network traffic and allocate bandwidth to different types of traffic. This is especially useful for environments with heavy network traffic or for ensuring that critical traffic (such as storage traffic) is not impacted by other traffic.
Use Jumbo frames for storage traffic. Jumbo frames (MTU=9000 bytes) can increase the efficiency of storage traffic by reducing the overhead associated with sending small packets. However, not all storage arrays support Jumbo frames, so check with your vendor before enabling this feature.
Monitor network performance. It’s important to monitor network performance to ensure that your virtual environment is operating efficiently. VMware provides several tools for monitoring network performance, including vRealize Operations Manager and vCenter Server Performance Charts.
By following these VMware networking best practices, you can ensure that your VMware ESXi networking is designed and configured for optimal performance, reliability, and scalability.
Conclusion
To configure virtual networks in VMware virtual environments, you should configure virtual switches, port groups, VMkernel adapters, physical uplinks, and, optionally, VLANs. The above terminology and ESXi networking concepts are crucial to understanding the basics when it comes to setting up an ESXi host in VMware network configuration. Next, you can go to the practical steps and configure ESXi VM networks, ESXi storage networks, and ESXi vMotion networks.
Keep in mind that it is essential to protect your VMware VMs and data by deploying an advanced data protection solution like NAKIVO Backup & Replication.
