ESXiArgs Ransomware Attacks: Risks and Data Protection
Usually, ransomware attacks physical computers and virtual machines running on Windows and less frequently on macOS and Linux. Unfortunately, cybercriminals continue to develop new versions of ransomware that can infect other operating systems via new-found vulnerabilities. A recent example was the ESXi ransomware attack that hit VMware hypervisors by exploiting ESXi vulnerabilities, leading to severe downtime in organizations across the world.
This blog post explains how dangerous ransomware attacks are, how ransomware can infect an ESXi host, and how to protect against ransomware, including against ESXi Args ransomware.
The Start of an ESXi Ransomware Campaign
The first cases of ESXi ransomware attacks were registered in October 2022 when VMware ended general support for ESXi 6.5 and 6.7. A high number of ESXi hosts were infected in France, Germany, the United States, Canada, the United Kingdom, and other countries. In February 2023, there were more than 3,000 infected ESXi hosts with encrypted data and no means for recovery. The names of ransomware that attacked ESXi hosts are ESXiArgs, Royal, and Cl0p. Additionally, new ESXi ransomware families appeared such as RansomEXX, Lockbit, BlackBasta, Cheerscrypt, Hive, RedAlert/N13V, and others.
ESXi hosts are attractive for attackers because server virtualization has become more popular, and many organizations use virtual machines for their production environments. As a result, ransomware creators have been focused on initiating VMware ESXi ransomware attacks with the hope of making huge profits. Infecting one ESXi host allows attackers to encrypt/destroy data of multiple virtual machines residing on that host. This approach can be more effective than infecting VMs, which may be running different operating systems. The ESXi Args ransomware attack was one of the largest ransomware attacks for non-Windows servers.
Which ESXi Versions Are Vulnerable to ESXi Ransomware?
ESXi ransomware exploits different vulnerabilities, including CVE-2022-31699, CVE-2021-21995, CVE-2021-21974, CVE-2020-3992, and CVE-2019-5544.
ESXi Args ransomware uses CVE-2021-21974 to infect an ESXi host. This is a vulnerability discovered in 2021 and can happen on ESXi hosts that are still unpatched/not updated. This is a heap overflow vulnerability in the OpenSLP service that is run on ESXi. A patch for CVE-2021-21974 was published on February 21, 2021.
The following ESXi versions are vulnerable to CVE-2021-21974:
- ESXi 7.x prior to ESXi70U1c-17325551
- ESXi 6.7.x prior to ESXi670-202102401-SG
- ESXi 6.5.x prior to ESXi650-202102101-SG
Why were there so many unpatched servers and why were they accessible from the internet? Patching requires ESXi server downtime, and with a large number of hosts, some administrators may not have the resources or time to patch them in time. Additionally, ESXi vulnerabilities were not widely exploited before the ESXi Args ransomware attack. It created the illusion that unpatched ESXi servers were not a threat, and servers were patched slowly.
ESXi servers running in hosting provider companies were also infected by ESXi Args ransomware. Hosting providers supplied the infrastructure for customers, and customers installed ESXi hypervisors to run virtual machines. These ESXi servers were exposed to the internet by customers, which allowed attackers to access them. The attackers wanted the victim to pay about $23,000 in bitcoins.
How Does ESXi Ransomware Work?
Cybercriminals find vulnerable ESXi hosts, especially those hosts exposed to the internet. The compromise method has been verified to exploit an OpenSLP vulnerability, which is possibly CVE-2021-21974. Port 427 (TCP/UDP) is used for OpenSLP. The logs indicate the involvement of the dcui user in this compromise process. This ESXi ransomware vulnerability allows attackers to exploit arbitrary code remotely.
The encryption process utilizes a public key deployed by the malware, located at /tmp/public.pem. Specifically, this encryption process is aimed at virtual machine files, including file types like “.vmdk,” “.vmx,” “.vmxf,” “.vmsd,” “.vmsn,” “.vswp,” “.vmss,” “.nvram,” and files with the “.vmem” extension. Note that “.vmdk” file is a virtual disk descriptor file and “-flat.vmdk” is a virtual disk file containing VM data. ESXi Args ransomware creates an “.args” file for each encrypted file with metadata (probably, ransomware creators presume that “.args” files may be needed for decryption).
Below is the sequence in detail:
- When a server is breached, the following files are placed in the /tmp directory:
- encrypt represents the executable encryptor in ELF format.
- encrypt.sh serves as the operational logic for the attack. This is a shell script that carries out various actions prior to running the encryptor, as outlined below.
- public.pem is a public RSA key employed for encrypting the key responsible for file encryption.
- motd is a textual ransom note that is duplicated to /etc/motd, ensuring its display upon login. The original file on the server will be preserved as /etc/motd1.
- index.html is an HTML version of the ransom note designed to replace VMware ESXi’s homepage. The initial server file should be backed up as “index1.html” within the same directory.
- The encryptor is initiated through a shell script, which invokes it with an array of command line parameters. These parameters encompass the public RSA key file, the target file for encryption, sections of data to remain unaltered, encryption block size, and the overall file size.
Usage: encrypt <public_key> <file_to_encrypt> [<enc_step>] [<enc_size>] [<file_size>]
Where:
- enc_step is the number of MB to skip while encrypting files
- enc_size is the number of MB in the encryption block
- file_size is the file size in bytes (for sparse files)
- The initiation of this encryptor takes place via the encrypt.sh shell script, which serves as the underlying logic for the attack. Upon initiation, the script proceeds with the following actions, explained briefly below.
- The ESXi ransomware script executes a command to modify the configuration files of ESXi virtual machines (.vmx) by replacing occurrences of “.vmdk” and “.vswp” with “1.vmdk” and “1.vswp“, respectively.
- Subsequently, the ransomware script forcefully terminates all currently active virtual machines by issuing a “kill -9” command to halt processes containing the word “vmx“.
- The ransomware attempts to shut down virtual machines by terminating the VMX process to release the locked files and modify them. However, this function doesn’t consistently perform as expected, resulting in some files remaining locked. After the modification of VM files, VMs become unusable.
- The script proceeds to retrieve a list of ESXi volumes with the command:
esxcli storage filesystem list | grep "/vmfs/volumes/" | awk -F' ' '{print $2}With this command, ransomware scans these volumes for files bearing specific file extensions.
- For each discovered file, the script generates a corresponding [file_name].args file within the same directory. This .args file contains computed parameters, including a step (commonly ‘1’) and the file size. For example, if the file is “VM01.vmx“, an associated “VM01.vmx.args” file is created. The malware generates an “argsfile” to store arguments passed to the encrypted binary, which includes information such as the number of megabytes to skip, the size of the encryption block, and the file size.
- The script then employs the “encrypt” executable to encrypt the files based on the calculated parameters.
- Following encryption, the script replaces the ESXi “index.html” file and the server’s “motd” file with the aforementioned ransom notes, as described previously.
There is no evidence of data being transferred externally (data exfiltration). In certain instances, the encryption of files may only partially succeed, enabling the victim to potentially recover some of the data.
When the infection and data modifying/encryption were completed successfully, a ransomware note like this was displayed on an HTML page:
“Security alert! We hacked your company successfully.
…
Send money within 3 days, otherwise we will expose some data and raise the price.”
- Subsequently, the script carries out cleanup tasks, including log deletion and the removal of a Python backdoor located at /store/packages/vmtools.py. It also eliminates specific lines from certain files:
- /bin/hostd-probe.sh
- /var/spool/cron/crontabs/root
- /etc/rc.local.d/local.sh
- /etc/vmware/rhttpproxy/endpoints.conf
A critical note is issued, urging administrators to check whether the “vmtools.py” file exists on the ESXi host. If found, immediate removal is advised.
- Lastly, the script executes “/sbin/auto-backup.sh” to update the configuration saved in the “/bootbank/state.tgz” file and activates SSH.
Also, it has come to light that this vulnerability, along with other vulnerabilities in ESXi, is actively exploited by ransomware groups besides ESXiArgs.
NOTE: The behavior of VMware ESXi ransomware can change with updated versions of ransomware and new ransomware releases.
How to Recover Data after ESXi Args Ransomware
There are no bugs in the cryptography mechanism that can allow you to decrypt the encrypted files. However, the US Cybersecurity and Infrastructure Security Agency (CISA) created a script that can help recover VMs. The good thing is that VM files, including the virtual disk descriptor (.vmdk), are encrypted, but the -flat.vmdk file (where VM data is stored) is not. This makes VM recovery possible.
CISA has introduced a recovery script designed to aid organizations impacted by the ESXi Args ransomware. This ransomware specifically targets ESXi servers, encrypting their configuration files, and potentially rendering virtual machines (VMs) inoperative. Although the tool was collaboratively developed with VMware, it is not directly supported by VMware. If customers encounter any issues while using this tool, they are encouraged to report problems on GitHub at this address: https://github.com/cisagov/ESXiArgs-Recover/issues.
CISA is committed to addressing and resolving concerns promptly. For further details about using the script, please refer to this link.
You can also download a PDF document with instructions.
If you have signs that an ESXi host has been infected with ESXi ransomware, consider the following measures:
- Disconnect the infected ESXi host from the network.
- Don’t pay the ransom. Paying the ransom rewards cybercriminals and incentivizes them to create more ransomware to get more money. If you pay the ransom, there is no guarantee that the files corrupted with VMware ESXi ransomware will be recovered.
- Report the ransomware attack. Reporting ransomware is important because it enables rapid response, legal compliance, data protection, trust, threat mitigation, and collective defense against cyberattacks. Reporting a ransomware attack is a fundamental part of cybersecurity incident management. It not only helps individual organizations recover from attacks but also contributes to the overall security and resilience of the digital ecosystem.
- Check whether recovery or decryption tools are available for the current version of ransomware.
- If there are no decryption tools, recover data from a backup (you must have a backup created before a ransomware attack to use this method). Sometimes it may be more effective to recover VMs from a backup. Consider a fresh installation of ESXi and copying the recovered VMs to that fresh ESXi host to ensure that there are no pieces of ransomware infection on the host where the recovered VMs are placed.
How to Protect ESXi from Ransomware
Follow the recommendations below to protect ESXi from ransomware:
- Patch your ESXi hosts that have vulnerabilities such as CVE-2021-21974, CVE-2022-31699, CVE-2021-21995, CVE-2020-3992, and CVE-2019-5544. If your ESXi version is not supported anymore, consider updating to a major version that is supported. If you cannot update ESXi, disable the OpenSLP (Service Location Protocol) service on ESXi. Disabling a service that has a vulnerability for the affected ESXi version also helps.
- Install security patches (updates) on the ESXi host regularly to protect against the latest threats.
- Don’t expose ESXi hosts to the internet. If the working process requires ESXi hosts to be accessible from the internet by workers and partners, configure a VPN server and firewall. Configure the firewall to allow access only from trusted IP addresses. Connecting to the network via VPN to access ESXi hosts is secure in this case.
- Disable SSH access if this access is not needed or set timeouts.
- Disable SMB v1.0 and other old versions of protocols, especially if they are not used.
- Use network segmentation for networks, including the ESXi management network.
- Use strong passwords with at least 8 characters, including lowercase, uppercase, digits, and special characters.
- Install and configure infrastructure monitoring to monitor network traffic and server loads. Monitoring allows you to detect suspicious or malicious activity in time.
- Educate users about ransomware protection and ensure that users know what to do if they suspect that a ransomware attack or attack attempt has happened.
- Configure anti-malware email protection because sending malicious links or files via email is a popular method to infect computers with ransomware. Disable active hyperlinks in email messages.
- Install antivirus on users’ computers and servers. Regularly update antivirus databases for antivirus software.
- Back up your virtual machines regularly and use the 3-2-1 backup strategy. Don’t forget to have an immutable backup or air-gapped backup to ensure that this backup is not affected in case of a ransomware attack. Having an ESXi backup and vCenter backup can help save time when it comes to recovering data and workloads.
- Prepare an incident response plan and let everyone know what to do in case of an ESXi ransomware attack.
- Create a disaster recovery plan to ensure that you can recover data and restore workloads in different scenarios. Backup testing and disaster recovery testing are important.
Conclusion
ESXi ransomware can be devastating because you can lose many VMs even if one ESXi host gets infected. Data backup is the most effective strategy to avoid data loss in case of ransomware attacks. Follow the recommendations on how to protect ESXi from ransomware that were covered above in this blog post. Use NAKIVO Backup & Replication to back up VMs residing on VMware ESXi hosts to immutable repositories. This way you can use these ransomware-resistant backups for swift recoveries of full VMs or application data.
