Posted in ESXi, Storage

Virtual Machine Snapshots

   July 16, 2016  Leave a comment

General overview of VM snapshots, benefits, risks and how to troubleshoot

VMs can reside on a VMFS, NFS or VSAN datastores. VMFS is a clustered file system. To prevent simultaneous writes the file system is using locks. (VMware official KB article)

Different lock modes:

Mode 0 = no lock
Mode 1 = exclusive access for one host only (e.g. vmx file)
Mode 2 = read-only (vmdk base disk when having snapshots)
Mode 3 = multiwriter (for Fault Tolerance or shared vmdk for clustering between VMs)

When a VM is running on snapshots, every disk in the chain becomes read-only (Mode 2), except the last one which has exclusive lock (Mode 1).

 

1
2
3
4
5
6
7
[root@esxmgnt:/vmfs/volumes/571bcd07-a3e5ce9c-601d-f04da20a8dce/Postfix] vmkfstools -D Postfix.vmdk
 
Lock [type 10c00001 offset 151642112 v 244, hb offset 3342336
gen 45, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2172789
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 343, 100>, gen 226, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 469, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 4305, bs 8192

 

00000000-00000000-0000-000000000000 <<— The host’s UUID which was created when installing ESXi. Last part of the UUID looks similar to the MAC address (unless the lock is being hold by the same host you are running the command on or if the lock is being held by an external non-ESXi server) because during installation ESXi installer uses the MAC address of the first physical NIC detected on the PCI bus.

1
2
3
4
[root@esxmgnt:~] cat /etc/vmware/esx.conf |grep -i uuid
/system/uuid = "5724e075-ef3c-d598-5552-f04da20a8dce"
 
[root@esxmgnt:~] esxcli network nic list

 

1
2
3
4
5
6
7
8
9
10
11
Name PCI Device Driver  Admin Status  Link Status  Speed  Duplex  MAC Address      MTU  Description
 
------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------
 
vmnic0  0000:01:00.0  bnx2 Up         Up         1000  Full f0:4d:a2:0a:8d:ce  1500  Broadcom Corporation QLogic 5709 1000Base-T
 
vmnic1  0000:01:00.1  bnx2 Up         Up         1000  Full f0:4d:a2:0a:8d:d0  1500  Broadcom Corporation QLogic 5709 1000Base-T
 
vmnic2  0000:02:00.0  bnx2 Up         Up         1000  Full f0:4d:a2:0a:8d:d2  1500  Broadcom Corporation QLogic 5709 1000Base-T
 
vmnic3  0000:02:00.1  bnx2 Up         Up         1000  Full f0:4d:a2:0a:8d:d4  1500  Broadcom Corporation QLogic 5709 1000Base-T

 

Different options when taking a snapshot:

Snapshot with VM memory:

  • includes the virtual RAM and CPU execution state
  • only available if VM is powered on
  • .vmsn filename
  • file size is at least the size of the VM’s RAM (taking snapshot can take longer the more RAM the VM has and the more active the VM’s RAM is)

Quiesce guest file system:

  • needs VMware Tools installed
  • Microsoft VMs using VSS (Volume Shadow Copy service) which is a complex operation
  • basic support for Linux via sync / vmsync or IOCTL. Depending on the distro
  • only available if VM is powered on

 

Quiesce snapshots of Windows VM using VSS and 2 snapshot disks are created for 1 snapshot. One disk is presented to the ESXi host, the other to the guest OS. Data is being written to both. Windows detects the disk because it has the same UUID as the base disk. Disk is visible in Windows Disk Manager and Device Manager.

.vmname-ctk.vmdk = change block tracking file used by backup software

 

Troubleshooting common snapshot issues:

Grep for ‘fileName’ in vmx file to find disk the VM is currently running on:

1
2
3
4
5
[root@esxmgnt:/vmfs/volumes/571bcd07-a3e5ce9c-601d-f04da20a8dce/HPE_Explorer_01] grep fileName *.vmx
 
ide1:0.fileName = "/vmfs/volumes/56c4ff11-997cd43e-1361-f04da20a8dce/ISO/en_windows_server_2012_r2_with_update_x64_dvd_6052708.iso"
scsi0:0.fileName = "HPE_Explorer_01.vmdk"
scsi0:1.fileName = "HPE_Explorer_01_1.vmdk"

 

A snapshot delta file can be identified by the header starting with COWD (Copy On Write Disk):

1
2
3
4
5
6
[root@esxmgnt:/vmfs/volumes/571bcd07-a3e5ce9c-601d-f04da20a8dce/HPE_Explorer_01] hexdump -C HPE_Explorer_01-000001-delta.vmdk |less
 
00000000  43 4f 57 44 01 00 00 00  03 00 00 00 00 00 80 0c  |COWD............|
00000010  01 00 00 00 04 00 00 00  00 c8 00 00 d1 92 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*

 

1
<span style="font-family: 'Fira Sans', sans-serif; font-size: 14px; background-color: #ffffff;">Also important to know: Snapshot files grow in 16MB increments.</span>

 

Consolidating snapshots can take a long time because snapshots are not sequential and have a lot of metadata overhead. ESXi requires time to go through all the metadata and place the delta from the snapshots back into the base disk when consolidating. It’s not like simply copying a file.

This repro shows that the file itself might not change, but the absolute position on the storage will:

  1. Created new empty text file within Linux guest OS
  2. Created a Snapshot of the VM
  3. Opened text file with vi and added text ‘test123abc’ and saved the file
  4. Shut down the VM

 

1
2
3
[root@esxmgnt:/vmfs/volumes/56c502f4-84f9f7bc-90b8-f04da20a8dce/Ubuntu16Desktop] hexdump -C Ubuntu16Desktop-000001-delta.vmdk |grep test123abc
 
0651a200  74 65 73 74 31 32 33 61  62 63 0a 00 00 00 00 00  |test123abc......|

 

0651a200 is the position of the text in the delta disk = 106013184 bytes = 106MB

5. Removed snapshot and consolidated back to the base disk

1
2
3
4
[root@esxmgnt:/vmfs/volumes/56c502f4-84f9f7bc-90b8-f04da20a8dce/Ubuntu16Desktop] hexdump -C Ubuntu16Desktop-flat.vmdk |grep test123abc
 
 
<span style="font-weight: 400; color: #ff9900;">13bbd000</span><span style="font-weight: 400;"> &nbsp;74 65 73 74 31 32 33 61 &nbsp;62 63 0a 00 00 00 00 00 &nbsp;|test123abc......|</span>

The position of the file has now changed:
13bbd000 = 331075584 bytes = 331.07MB

 

Recover from a deleted snapshot file in the chain

If a file within the snapshot chain gets deleted by mistake or becomes corrupted there are really only two options.

Option 1: Restore from Backup
Option 2: Edit the vmx file and point to the base disk or any other snapshot that was taken before the one that was deleted. All data written after this, will be lost

 

CID (content identifier) chain broken:

The VM will not power on if the CID is not correct.

1
grep fileName vmname.vmx

1
vmkfstools -e vmname-000004.vmdk

1
This will check the ‘Content ID:’ and the only option is to manually change it.

 

In this example, this should be the chain:
UPSA109x64-VM11-000002.vmdk
UPSA109x64-VM11-000001.vmdk
UPSA109x64-VM11.vmdk
But ‘…000001.vmdk’ was deleted by mistake. The snapshot chain is now incomplete and will result in a ‘file lock’ error. Consolidating the Snapshot back to the base disk will fail.

The error in the vSphere Client or Web Client is pretty generic.
This is what the hostd.log is seeing (grepped for the opID):

2016-12-10T20:41:23.227Z info hostd[35B01B70] [Originator@6876 sub=Vimsvc.TaskManager opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Task Created : haTask-6787-vim.VirtualMachine.removeAllSnapshots-83579029
2016-12-10T20:41:23.227Z info hostd[35603B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] State Transition (VM_STATE_ON -> VM_STATE_REMOVEALL_SNAPSHOT)
2016-12-10T20:41:23.233Z warning hostd[35B01B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Failed to remove snapshot(s): vim.fault.GenericVmConfigFault

This is the error we see in vCenter.

2016-12-10T20:41:23.238Z info hostd[35B01B70] [Originator@6876 sub=Libs opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] OBJLIB-FILEBE : FileBEOpen: can’t open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’ : Could not find the file (393218).
2016-12-10T20:41:23.238Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-DSCPTR: DescriptorOpenInt: failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’: Could not find the file (60002)
2016-12-10T20:41:23.238Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LINK : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk” : failed to open (The system cannot find the file specified).
2016-12-10T20:41:23.238Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-CHAIN : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk” : failed to open (The system cannot find the file specified).
2016-12-10T20:41:23.238Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB : Failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’ with flags 0x110015 The system cannot find the file specified (25).
2016-12-10T20:41:23.238Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_MISC : DiskLibEnumExtents: failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’: The system cannot find the file specified (25).
2016-12-10T20:41:23.238Z error hostd[35B01B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Failed to enumerate disk [LocalDataStore] UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk, The system cannot find the file specified
2016-12-10T20:41:23.247Z info hostd[35B01B70] [Originator@6876 sub=Libs opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] OBJLIB-FILEBE : FileBEOpen: can’t open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’ : Could not find the file (393218).
2016-12-10T20:41:23.247Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-DSCPTR: DescriptorOpenInt: failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’: Could not find the file (60002)
2016-12-10T20:41:23.247Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LINK : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk” : failed to open (The system cannot find the file specified).
2016-12-10T20:41:23.247Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-CHAIN : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk” : failed to open (The system cannot find the file specified).
2016-12-10T20:41:23.247Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB : Failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’ with flags 0x110015 The system cannot find the file specified (25).
2016-12-10T20:41:23.247Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_MISC : DiskLibEnumExtents: failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’: The system cannot find the file specified (25).
2016-12-10T20:41:23.247Z error hostd[35B01B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Failed to enumerate disk [LocalDataStore] UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk, The system cannot find the file specified
2016-12-10T20:41:23.251Z info hostd[35B01B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Send config update invoked
2016-12-10T20:41:23.258Z info hostd[35B01B70] [Originator@6876 sub=Libs opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] OBJLIB-FILEBE : FileBEOpen: can’t open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’ : Could not find the file (393218).
2016-12-10T20:41:23.258Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-DSCPTR: DescriptorOpenInt: failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’: Could not find the file (60002)
2016-12-10T20:41:23.258Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LINK : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk” : failed to open (The system cannot find the file specified).
2016-12-10T20:41:23.258Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-CHAIN :”/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000002.vmdk”: Failed to open parent “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk”: The system cannot find the file specified.
2016-12-10T20:41:23.258Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-CHAIN : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk” : failed to open (The parent of this virtual disk could not be opened).
2016-12-10T20:41:23.258Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB : Failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000002.vmdk’ with flags 0x13 The parent of this virtual disk could not be opened (23).
2016-12-10T20:41:23.258Z warning hostd[35B01B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Load failed: cannot open /vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000002.vmdk (23): The parent of this virtual disk could not be opened
2016-12-10T20:41:23.297Z info hostd[35B01B70] [Originator@6876 sub=Libs opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] OBJLIB-FILEBE : FileBEOpen: can’t open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’ : Could not find the file (393218).
2016-12-10T20:41:23.297Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-DSCPTR: DescriptorOpenInt: failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’: Could not find the file (60002)
2016-12-10T20:41:23.297Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LINK : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk” : failed to open (The system cannot find the file specified).
2016-12-10T20:41:23.297Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-CHAIN : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk” : failed to open (The system cannot find the file specified).
2016-12-10T20:41:23.297Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB : Failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’ with flags 0x110015 The system cannot find the file specified (25).
2016-12-10T20:41:23.297Z info hostd[35B01B70] [Originator@6876 sub=DiskLib opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_MISC : DiskLibEnumExtents: failed to open ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’: The system cannot find the file specified (25).
2016-12-10T20:41:23.297Z error hostd[35B01B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Failed to enumerate disk [LocalDataStore] UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk, The system cannot find the file specified
2016-12-10T20:41:23.301Z warning hostd[35B01B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Failed operation
2016-12-10T20:41:23.302Z info hostd[35B01B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11.vmx opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] State Transition (VM_STATE_REMOVEALL_SNAPSHOT -> VM_STATE_ON)
2016-12-10T20:41:23.303Z info hostd[35B01B70] [Originator@6876 sub=Vimsvc.TaskManager opID=495d89d8-2d6f-4b78-9bd9-5c480164a966-29184-ngc-f8-29-d38b user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask-6787-vim.VirtualMachine.removeAllSnapshots-83579029 Status error

Creating another Snapshot of the VM will also fail.
This is vmware.log (The logfile of the VM itself) when taking another snapshot:

2016-12-10T20:54:08.441Z| vcpu-0| I120: DISK: Cannot open disk ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk’ : The system cannot find the file specified.
2016-12-10T20:54:08.441Z| vcpu-0| I120: DISKLIB-VMFS : “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000002-delta.vmdk” : closed.
2016-12-10T20:54:08.441Z| vcpu-0| I120: Msg_Post: Error
2016-12-10T20:54:08.441Z| vcpu-0| I120: [msg.disk.fileNotFound] VMware ESX cannot find the virtual disk “/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000001.vmdk”. Verify the path is valid and try again.
2016-12-10T20:54:08.441Z| vcpu-0| I120: [msg.disklib.FILENOTFOUND] The system cannot find the file specified
2016-12-10T20:54:08.441Z| vcpu-0| I120: [msg.disk.noBackEnd] Cannot open the disk ‘/vmfs/volumes/569a2490-21d8efd8-d9c8-f04da20a8dce/UPSA109x64-VM11/UPSA109x64-VM11-000002.vmdk’ or one of the snapshot disks it depends on.

This is correct because it depends on UPSA109x64-VM11-000001.vmdk which is deleted.
2016-12-10T20:54:08.441Z| vcpu-0| I120: [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.
2016-12-10T20:54:08.441Z| vcpu-0| I120: —————————————-
2016-12-10T20:54:08.444Z| vcpu-0| I120: Vigor_MessageRevoke: message ‘msg.checkpoint.continuesync.error’ (seq 191550719) is revoked
2016-12-10T20:54:08.444Z| vcpu-0| I120: SnapshotVMXTakeSnapshotCB: Failed to quiesce for snapshotting. (mode=1, error=0)
2016-12-10T20:54:08.444Z| vcpu-0| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot ‘VM Snapshot 12%252f10%252f2016 8:54 PM UTC+00:00’: 0
2016-12-10T20:54:08.444Z| vcpu-0| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Unable to save snapshot file (13).

If the VM is being powered off, you won’t be able to power it on again.

 

Unable to consolidate due to locked file:

An easy way to check if files of the VM are locked is to use the chmod command which will try to access the file’s metadata.

1
chmod *

If a file is locked it will return: chmod: invalid mode ‘filename‘

We can now run the command to check the owner of the lock: # vmkfstools -D filename

The heartbeat offset is shown in the output. We can hexdump the LUN where the file resides and output the area where the lock owner’s network MAC address is listed.
In this example, checking the MAC addresses of the host we are logged in to we see that it is the actual owner of the lock.

Option 1: Shut down the VM
Option 2: Shut down the host holding the lock
Option 3: Migrate (vMotion) the VM to another host
If the locks follows the VM, shut down the VM
If the lock still stays with the host, shut down the host
Option 4: use lsof and ps to try to find the process holding the lock and kill it. There might not be a visible process  accessing the file. In this case the host needs to be restarted


Unable to quiesce Windows VM:

Usually caused by Microsoft’s VSS.

Option 1: Activate verbose mode for vmware.log and generate vsstrace.log
Option 2: Consult Microsoft Support
Option 3: Reinstall VMware Tools with VSS disabled

 

Datastore ran out of space:

Option 1: Try to consolidate
Option 2: Power off the VM and clone the current snapshot into a new disk. Depending on the disk size and amount of snapshots in the chain, this might take a very long time (hours, even days). After cloning is complete, edit the VMX file or use the GUI to mount the disk.

1
vmkfstools -i vm-000001.vmdk /vmfs/volumes/datastoreXY/newdisk.vmdk

 

Snapshots not showing in GUI:

Option 1: Add a vCenter alarm to monitor for snapshots (KB 1018029)
Option 2: Search for snapshot files:

1
find /vmfs/volumes/ -name ‘*[0-9][0-9][0-9][0-9][0-9][0-9].vmdk’

 

Read performance degraded:

Write operations are not affected as ESXi only writes into the latest disk. However ESXi has to read through every single layer of snapshots for every read-operation. So it is expected that the performance is degraded when the VM is running on a snapshot chain.


Unable to snapshot VMs with RDM mapped:

This is working as intended as ESXi has no control over the RDM LUN. Workaround: Use RDM in virtual compatibility mode or vmdk.

 

VM becomes unresponsive during consolidation:

High stun times might lead to ping loss. Check vmware.log for stun times. If VM resides on NFS3 storage the issue can multiply due to missing SCSI reservation or ATS features of NFS. A lot of these issues are solved with NFS 4.1 though.

If VM is too I/O active the final stun process might fail.

 

Author: kimbo

Leave a Reply

Your email address will not be published. Required fields are marked *