What Is Application-Aware VM Backup?
When it comes to backups, there is a wide range of options that must be considered such as backup type, frequency, source, destination, and many others. Back in the early days of traditional backups, most backup solutions simply captured files on disk. However, in today’s modern backup technology world, most servers are application centric. Flat file “inconsistent” backups certainly are not adequate. We have talked about “crash consistent” backups that are consistent for a volume that is backed up by utilizing Volume Shadow Copy Service.
However, there is a particular option for consistent backups that are able to make consistent backups of applications. Most modern VMware virtual machine backup solutions usually offer the ability to have a backup be application-aware.
What are Application-Aware (Application-Consistent) Backups?
Application consistent or Application aware backups have a special use case in that they take the crash-consistent backups a step further. While crash consistent backups create consistent backups of files on a volume by utilizing the Volume Shadow Copy Service, they are unaware of the application data that may be living in memory as well as pending I/O operations. Application-aware backups leverage special hooks into the Volume Shadow Copy Service and the application they are backing up. These special hooks are called VSS writers. VSS writers are special application specific components of Microsoft’s Volume Shadow Copy Service. They serve the special role of making sure that application data is properly flushed from memory, frozen long enough for a VSS snapshot to take place, and then unfrozen after the snapshot has taken place. The process typically only takes a few seconds.
This is an extremely important process that needs to happen when we think about an application that requires transactional consistency such as Microsoft SQL Server. The Microsoft VSS writers for SQL Server are able to flush data from memory, freeze SQL operations and then release the freeze after the snapshot has taken place. This ensures that data living in memory and pending data I/O operations are properly flushed and taken care of before the snapshot operation takes the VSS snapshot of the disk. This makes the backup operation “application consistent” as not only the disk but also the application is backed up in a state where it maintains transactional consistency. If we only use a crash consistent backup without the application aware features turned on, there is a good chance we will be left with an application that is not in a consistent state.
Application consistency is not only a concern at the time the backup operation takes place. It is and should be a concern also when a restore operation is completed. We can successfully restore an application server such as Microsoft Exchange or Microsoft SQL Server using a restore of a crash consistent backup, however, we have to use the process those applications call for to bringing the application up to a consistent state. This may include replaying logs, etc. So, important to note, the time to restore those particular services will include not only the time to restore the raw file resources, whether it be a VMware or Hyper-V virtual machine, but also the time it takes to bring the application up to a consistent state.
As already noted, we benefit from using virtual server backup software that enables us to create application-aware backups. The restored resources including the VMware or Hyper-V VM will already have the application data in a consistent state. Having this type of backup in place for critical application servers such as Microsoft SQL server or Microsoft Exchange Server can greatly benefit our recovery time objective (RTO) for business services.
Creating Application-Aware Backup with NAKIVO Backup & Replication
Below, we see an example of a backup job in Nakivo Backup & Replication v7 where in the Options of the job, we can select App-aware mode as well as how we want the job to proceed if it encounters errors with VSS.
If you hover your mouse over the “question mark” icon next to the app-ware combo box, we can see a detailed description of the app-ware process and the dependencies. Note the mention, in the case of VMware virtual machines, of VMware tools. VMware tools is used for guest OS quiescing for application data.
VSS Writers and Troubleshooting
As mentioned, VSS writers are the special purpose application specific components of Microsoft’s Volume Shadow Copy Service. These can either be Microsoft installed or third party components that are typically installed with the application itself. In the case of Microsoft VSS writer components, these can be seen installed on a specific application basis. Domain controllers will have the “NTDS” writer, SQL Server will have the “SqlServerwriter”, and Exchange Server will have the “Microsoft Exchange Writer”. When it comes to troubleshooting VSS writers if and when we have issues with application-aware backups, we have several utilities and resources available to us for troubleshooting.
A great command line utility to see/troubleshoot the specific VSS writers that are active, as well as the state of these writers, is the vssadmin command. If you open a command prompt and simply type vssadmin you can see the specific commands that are available with vssadmin.
Using the vssadmin list writers command, we can see the detailed list of the special VSS writers used for various applications. Notice below we ran the command on a Microsoft SQL Server. We have the SqlServerWriter listed. Also, note the State and the Last error notations as these are very helpful to see the current state and any errors that might be present on the particular VSS writer.
The next screenshot is taken from a Windows Server 2016 domain controller. Note for this server the vssadmin list writers command shows the NTDS writer which is specific to Active Directory Domain Services.
As noted above, VMware tools is an essential part of application-aware processing inside a virtual machine. By default, logging for VMware tools is not turned on. When troubleshooting issues with VSS on a virtual machine, we may need to enable logging for VMware tools inside the guest operating system. To do that, we need to edit or create the file tools.conf in our virtual machine.
Below, the file was not present in our Windows Server 2016 virtual machine. Navigating to the C:\Programdata\VMware\VMware Tools directory, we created the configuration file. Make sure to have your extensions displayed to correctly name the file.
After the file is created, we need to edit the file and add the following for the respective operating system. Note for Windows the double backslashes for the tools data path. You can also use one forward slash for Windows as well.
Windows guests
[logging]
log = true
vmtoolsd.level = debug
vmtoolsd.handler = file
vmtoolsd.data = c:\\windows\\temp\\vmtoolsd.log
Linux guests
vmtoolsd.data = /tmp/vmtoolsd.log
After creating/editing the file, simply restart the VMware tools service.
The resulting log file contains debug information including VSS operation information. Below is a snippet of a fairly clean debug log right after enabling the debug mode of VMware Tools.
Also, advisable when troubleshooting any VSS related error is looking in the Windows event logs, particularly the Application and System logs. Application log entries will appear as the source VSS and System log entries will appear under the source of volsnap. There are also specific VMware VSS troubleshooting steps to go through in looking at and troubleshooting VMware tools related VSS errors.
Thoughts
Utilizing application-aware VM backups for business-critical applications is essential to having a solid disaster recovery strategy when backing up applications that rely on transactional consistency. Application-aware backups allow for all the data living in memory as well as pending I/O operations to be flushed to disk before the consistent backup of the disk is taken. This is accomplished through special Volume Shadow Copy Service writers that speak to the specific application and properly freeze operations long enough for the application to be backed up with transactional consistency. For organizations wanting to both protect data and be able to restore application data as quickly as possible, restoring application-aware backups bypasses the need to have to restore the application data separately such as replaying logs, etc.
Modern backup solutions such as NAKIVO Backup & Replication truly provide great features and functionality. Knowing the available options such as application-aware backups and leveraging these for transactional consistency with applications, helps organizations have a consistent, reliable, and efficient backup solution. This ensures that organizations can deliver on both restore point and restore time objectives.
