Fix VMware Error: Virtual Machine Disks Consolidation Needed

Virtual machines in VMware vSphere have many advantages. They work reliably with high performance if the appropriate hardware is used and the right software configuration is applied. Sometimes the virtual machine consolidation needed status is displayed in the VMware vSphere Client, and a system administrator has to fix this error. This blog post explains why VMware virtual machine disks consolidation is needed and how this error can be fixed.

What Is VMware Disk Consolidation?

Virtual disk consolidation is the process of merging virtual disk files created after taking VM snapshots. You can initiate disk consolidation on demand to avoid performance degradation and other issues. When you take a snapshot, delta disks are created and changes are written to the appropriate delta disk (the latest delta disk by default, until one of the previous snapshots is selected to restore the virtual machine). As a result, redundant data is present in the virtual machine storage because each delta disk contains all changes made as compared to the previous delta disk state, and the parent disk remains unchanged. When the chain of snapshots and related virtual disk files becomes large, performance degrades. When you perform VMware disk consolidation, data from delta disks is merged to a single disk, and no redundant data is left on the datastore where this VM’s files are located. Storage space is cleaned after you consolidate virtual machine disk files. Remember that VMware recommends using no more than 32 snapshots per VM.

Possible Reasons for the VMware Disk Consolidation Error

The VMware virtual machine disks consolidation is needed error message is displayed in the VM Summary tab in the VMware vSphere Client (screenshot below). One of the common reasons for this error message is incorrectly deleting a VM snapshot, which can leave VMDK files related to snapshots and log files on the datastore (for example, after you performed the Delete All or Delete operation for VM snapshots).

The most common causes for the virtual machine consolidation needed status:

• Not enough disk space on the VMFS datastore to consolidate VM snapshots and consolidate virtual disk files. If there is less than 1 GB of free space on the datastore, this error may occur.
• Low performance of a storage system attached to an ESXi host on which a VM is running. If the snapshot size is large or times out, snapshot consolidation can fail.
• Connection issues between an ESXi host and vCenter.
• Snapshot files are locked by a third-party application, for example, a backup application. This may occur if a backup application did not finish a backup job and locked the snapshot (backup applications can create a temporary snapshot of a virtual machine when running a backup). Backup applications are unable to quiesce disk without using snapshot techniques to make a consistent backup of a running VM.

Basically, you should restore the consistency of VM disk files.

How to Fix the Error?

To fix the virtual machine consolidation needed status, right click the VM name in the VMware vSphere Client and in the menu that opens, click Snapshots > Consolidate.

A confirmation message is displayed:

This operation consolidates all redundant redo logs on your virtual machine. Are you sure you want to continue?

Click Yes to confirm and consolidate virtual machine disk files.

Note: If there is a concern that some data can be lost during VMware disk consolidation, make a VMware VM backup, for example, copy all VM files to another folder on a datastore and then perform VMware disk consolidation.

Wait until the VMware disk consolidation is complete and the logs are cleared. You can see the task progress in the task panel at the bottom of the VMware vSphere Client interface. If the VM is running, VM performance can degrade during this operation. The time needed to finish this operation depends on the VM size, the number of snapshots, and VM load.

After the VMware disk consolidation task is finished, the warning that VMware virtual machine disks consolidation is needed should disappear.

Errors that May Occur

Sometimes, another error can occur during VMware disk consolidation and the following error message is displayed:

Unable to access file since it is locked. An error occurred while consolidating disks: Failed to lock the file. Consolidation failed for disk node ‘scsi0:0’.

Locks can be also created when backup applications use the hot-add disk option for a backup when using snapshots. The VM file lock is intended to avoid concurrent writes to the file by multiple ESXi hosts.

Try to perform the following actions to fix this error.

1. Restart management agents on the ESXi server on which the VM resides. Open ESXi shell or connect to the ESXi host via SSH and run the command as root:

services.sh restart

or

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

You can also open the ESXi direct console, go to Troubleshooting Options, and select Restart Management Agents (press Enter to activate the needed option, then press F11 to confirm).

2. Check the locked file with the command:

vmfsfilelockinfo -p /vmfs/volumes/vm_datastore/vm_name/vm_name.vmx

3. Check the hostd log file if you need to identify the path on the datastore where the VM files are located.

cat /var/log/hostd.log |grep -i vm_name

4. Go to the VM directory on the datastore when you know the VM files’ location:

cd /vmfs/volumes/datastore_name/vm_name

5. Run the command to list ESXi hosts that are locking the VM files:

for i in `ls`; do vmfsfilelockinfo -p $i ;done|grep ‘is locked in\|Host owning the lock\|Total time taken’ | sed ‘s|Host owning the lock on file is||g’|sed ‘s|Total time|—|g’ | awk ‘{print $1}’ |uniq

6. Restart ESXi management agents, including hostd and vpxa as explained above, or run these commands on the ESXi host locking the VM files:

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

Another method to unlock VM files

As an alternative, try these steps to fix an error occurred while consolidating disks: Failed to lock the file.

1. Shut down the virtual machine.
2. Take a new VM disk snapshot.
3. Delete all existing VM snapshots. Right click the VM name in vSphere Client, select Snapshots, and click Delete All Snapshots.

4. Clone the VM or migrate the VM to another ESXi host with vMotion if the error has not been fixed yet.
5. Try to consolidate snapshots once again, as explained above.

If these actions didn’t help, try to shut down the VM, open VM settings, select the VM options tab, then expand the Advanced section, and click Edit Configuration. In the Configuration parameters window, click Add Configuration Params. Add the asyncConsolidate.forceSync parameter and set this parameter to TRUE.

Sometimes the virtual machine consolidation needed status can be caused by inconsistent -ctk.vmdk files that are created for changed block tracking (by VM backup software, for example). Try to delete CTK files. They are not needed for snapshot consolidation. Don’t forget to back up VM files before deleting CTK files. Then try to consolidate virtual machine disk files again.

The Consolidate option is inactive

If the VMware consolidate snapshot option is greyed out (inactive), an active task can be running on the VM now. If you see the “detected an invalid snapshot configuration” error, some snapshots can be broken or there are split-chains of snapshots. In this case, try to do the following actions.

1. Connect to the ESXi server via SSH and go to the directory where the VM files are located on the datastore.

cd vmfs/volumes/DatastoreName/VMFolderName

You can check the VM files’ location in the VMware vSphere Client and find the location of virtual disks by viewing VM settings.

2. Identify what disk snapshots are being pointed:

cat VM-name.vmx | grep scsi*

See the output to check virtual disk files including virtual disk snapshot files. For example, the name of the virtual disk file is displayed in the line. This is not a snapshot:

scsi 0:0.fileName = “VM-Name.vmdk”

In the output line, the name of the virtual disk snapshot file is displayed, and you are pointing the snapshot:

scsi 0:0.fileName = “VM-Name-000002.vmdk”

A SCSI virtual disk controller is used in VM configuration in this example.

3. The next command checks what each snapshot is pointing to. Use this command to detect which snapshot is pointing to the next disk file (the snapshot vmdk file or a base disk vmdk file) in the snapshot chain.

cat VM-Name-000002.vmdk | grep parentFileNameHint

You may need to execute this command again for other vmdk files (like VM-Name-000001.vmdk in this example) until you find the needed base disk. This method helps you find the inconsistency.

4. If you have found the inconsistency, for example, VM-Name-000002.vmdk is pointing to VM-Name-000001.vmdk which is pointing to VM-Name.vmdk, run vmkfstools on the highest point of the disk snapshot chain (VM-Name-000002.vmdk in this case):

vmkfstools -q -v10 VM-Name-000002.vmdk

This command tries to open and close all vmdk files in the chain and notifies you where failure occurs.

Using PowerCLI

As an alternative to executing VMware disk consolidation in the VMware vSphere Client graphical user interface, you can use vSphere PowerCLI with the command line interface.

1. Connect to the vCenter Server that manages the ESXi host with the VM that has the virtual machine consolidation needed status. Enter credentials when prompted.

Connect-VIServer vcenter01.test.com

2. Get the list of VMs with the virtual machine consolidation needed status:

Get-VM | Where-Object {$ _. Extensiondata.Runtime.ConsolidationNeeded}

3. Perform VMware disk consolidation.

Get-VM | Where-Object {$ . Extensiondata.Runtime.ConsolidationNeeded} | foreach {$ . ExtensionData.ConsolidateVMDisks_Task ()}

Conclusion

This blog post has explained recommendations that can help you fix the virtual machine consolidation needed status. VMware virtual machine disks consolidation is usually needed when one of the snapshots is corrupted, a snapshot chain is inconsistent, or a snapshot is locked by another ESXi host. Regularly run a VM backup solution in your VMware vSphere environment. If you have data backup, you can restore the data and resume working if data is deleted, corrupted, encrypted by ransomware or if you see an error occurred during vCenter operation. The VMware vCenter Server should be backed up, just like other important VMs. NAKIVO Backup & Replication is a professional backup product that supports backing up virtual machines in VMware vSphere. Download NAKIVO Backup & Replication and back up your VMs to avoid data loss and downtime.