Saturday, 22 August 2015

RESOLVING THE CID MISMATCH ERROR: THE PARENT VIRTUAL DISK HAS BEEN MODIFIED SINCE THE CHILD WAS CREATED (1007969)


Symptoms


  • You are unable to power on a virtual machine or consolidate snapshots if a parent file has been modified after the child has been created. This includes base disks and snapshot delta files.
  • Depending on the modification or caused mismatch, you see any of these errors:
    • Failed to open 'virtual machine disk' with flags 0xe (The parent virtual disk has been modified since the child was created)
    • Failed to open (The parent virtual disk has been modified since the child was created)
    • Failed to open 'virtual machine disk': The parent virtual disk has been modified since the child was created (18).
    • DISKLIB-LINK : Attach: Content ID mismatch (7b7644b2 != 4f5a6761)
    • DISKLIB-LINK : Attach: the capacity of each link is different (83886080 != 46399652).
  • You see this error in the vSphere Client:

    Cannot open the disk '/vmfs/volumes/4a365b5d-eceda1-19-439b-000cfc0086f3/examplevm/examplevm-000001.vmdk' or one of the snapshot disks it depends on.
    Reason: The parent virtual disk has been modified since the child was created.
  • The virtual machine log contains entries similar to:

    vmx| DISKLIB-LINK : Attach: Content ID mismatch (d0fdb25b != ef4854ee).
    vmx| DISKLIB-CHAIN : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm.vmdk" : failed to open (The parent virtual disk has been modified since the child was created).
    vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002-delta.vmdk" : closed.
    vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000001-delta.vmdk" : closed.
    vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-flat.vmdk" : closed.
    vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk' with flags 0xa (The parent virtual disk has been modified since the child was created).
    vmx| DISK: Cannot open disk "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk": The parent virtual disk has been modified since the child was created (18).
    vmx| Msg_Post: Error
    vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk' or one of the snapshot disks it depends on.
    vmx| [msg.disk.configureDiskError] Reason: The parent virtual disk has been modified since the child was created.----------------------------------------
    vmx| Module DiskEarly power on failed.
  • The virtual machine shuts down abruptly during snapshot removal, deletion, or commit.
  • The virtual machine shuts down abruptly during or immediately following a Storage vMotion or Migration.
  • You see this error when attempting to power on a virtual machine:

    A general system error occurred: Internal error.
  • Performing a snapshot removal or powering on the virtual machine generates the error:

    Content ID mismatch
  • If you grow an RDM mounted inside a virtual machine while the snapshot is being committed, you may see errors similar to:

    DISKLIB-LINK : DiskLinkIsAttachPossible: the capacity of each link is different (262144000 != 52428800)
    /vmfs/volumes/<datastore-name>/<vm-name>/<vm-name>.vmdk" : failed to open (The capacity of the parent virtual disk and the capacity of the child disk are different)
    DISKLIB-VMFS : "./<vm-name>-000002-delta.vmdk" : closed.
    DISKLIB-VMFS : "/vmfs/volumes/<datastore-name>/<vm-name>/<vm-name>-000001-delta.vmdk" : closed.
    DISKLIB-VMFS : "/vmfs/volumes/datastore/<vm-name>/<vm-name>-rdm.vmdk" : closed.
    DISKLIB-LIB : Failed to open '<vm-name>-000002.vmdk' with flags 0x17 The capacity of the parent virtual disk and the capacity of the child disk are different (67).
    Failed to open <vm-name>-000002.vmdk' : The capacity of the parent virtual disk and the capacity of the child disk are different (67)

Purpose


This article provides information and troubleshooting steps to diagnose Content ID mismatch conditions between two or more virtual disk files for a virtual machine.
Note: The procedure to correct parent CID and CID mismatch explained in this article is valid for both VMFSSPARSE and SESPARSE type delta disks.

Resolution


Overview

The Content ID (CID) value of a virtual machine disk descriptor file aids in the goal of ensuring content in a parent virtual disk file, such as a flat or base disk, is retained in a consistent state.
The child delta disks that derive from that base disk's snapshot contain all further writes and changes. These changes depend on the source disk to remain intact.
A virtual machine disk descriptor file details the basic geometry, format, or otherwise identification and handling for a virtual disk or virtual disk delta file. A CID resides in each virtual machine's disk descriptor file for integrity or state tracking.
Example descriptor file for a base disk:
examplevm.vmdk:
# Disk DescriptorFile
version=1
CID= 7b7644b2
parentCID=ffffffff
createType="vmfs"

# Extent description
RW 20971520 VMFS "examplevm-flat.vmdk"
# The Disk Data Base
#DDB
ddb.toolsVersion = "0"
ddb.adapterType = "lsilogic"
ddb.geometry.sectors = "63"
ddb.geometry.heads = "255"
ddb.geometry.cylinders = "1305"
ddb.uuid = "60 00 C2 9f ae de ba e9-95 4e a7 a6 4e 95 c1 c1"
ddb.virtualHWVersion = "4"
Example descriptor file for a delta disk
examplevm-000001.vmdk:
# Disk DescriptorFile
version=1
CID= 69a1c662
parentCID= 7b7644b2
createType="vmfsSparse"
parentFileNameHint="examplevm.vmdk"
# Extent description
RW 20971520 VMFSSPARSE "examplevm-000001-delta.vmdk"
# The Disk Data Base
#DDB
ddb.toolsVersion = "7302"
Note: examplevm-000001.vmdk refers to, and in another sense depends on, examplevm.vmdk.
When the virtual machine references a virtual disk, it cites either the base disk's descriptor file, or a snapshot delta's descriptor file. In this example, the virtual machine configuration file, or examplevm.vmx, refers to the delta disk descriptor file:
scsi0:0.present = "true"
scsi0:0.fileName = "examplevm-000001.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
Any time a virtual machine is powered on, the referenced base or delta disk descriptor file's CID value is changed (see CID printed in blue, above):
examplevm-000001.vmdk before power-on:
CID= 69a1c662
parentCID=7b7644b2		examplevm-000001.vmdk after power-on:
CID= 6aff3ba2
parentCID=7b7644b2

All of this details a virtual machine in good running condition. A mismatch can be found here, which prevents tasks from succeeding for this virtual machine:
examplevm.vmdk:
CID= 12a9ffab
parentCID=ffffffff		examplevm-000001.vmdk:
CID=69a1c662
parentCID= 7b7644b2
In effect, a CID mismatch ensures that deviance from the original disk state results in all dependent child delta content being invalidated. This protects stored data from further potential corruption.

Cause

Content ID mismatch conditions are triggered by interruptions to major virtual machine migrations such as Storage vMotion or Migration, VMware software error, or user action.
These Content IDs are only used for virtual machine disks with snapshots. For more information about snapshots, see Understanding virtual machine snapshots in VMware ESXi and ESX (1015180).
Some scenarios to avoid in particular include:

Troubleshooting

When there is a CID mismatch, the virtual machine name is provided in the error message, but you must identify:
  • what virtual machine disk or disks are affected
  • what specific disk descriptor files are affected
  • the cause of the mismatch, or what changes occurred

Identifying the virtual machine disk and descriptor files affected

There are several methods to log into an ESX host to review content of utilized datastores, depending on the version of ESX utilized. For more information, see Editing configuration files in VMware ESXi and ESX (1017022).
Notes:
  • The datastore browser provided in the VMware vSphere Client or VMware Infrastructure Client do not distinguish between virtual machine descriptor (1) and flat or delta files (2). They are collapsed into singular entities to make datastore browsing simpler. As you need to distinguish between the two files, use the additional access methods provided in the referenced article.
  • The web-based Datastore Browser, accessible via https://server-ip, is used to browse VMFS datastores and distinguish between the virtual machine descriptor files and their corresponding associated binary files. You can copy the relevant file from the Datastore Browser for editing. The edited file can be uploaded to the datastore again via the vSphere Client.
  • Due to the nature of the problem experienced, the quickest method for resolving the issue is with the Command Line Interface available on the individual ESX host. Utilize this method if you have sufficient background knowledge on command-line usage. Alternatively, you can use the VMware vSphere Command Line Interface (CLI) or VMware vSphere Management Assistant appliance (vMA) to obtain the virtual machine disk descriptor files for review.
  • If you are unable proceed, please file a support request with VMware Technical Support. For more information, see Filing a Support Request in My VMware (2006985).
After locating the virtual machine's files and directory:
  1. The virtual machine's vmware.log file contains information which identifies the specific disk chain affected. Review the logs and note the location and files affected.

    Example:

    vmx| DISKLIB-LINK : Attach: Content ID mismatch (d0fdb25b != ef4854ee).
    vmx| DISKLIB-CHAIN : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm.vmdk" : failed to open (The parent virtual disk has been modified since the child was created).
    vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002-delta.vmdk" : closed.
    vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000001-delta.vmdk" : closed.
    vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-flat.vmdk" : closed.
    vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk' with flags 0xa (The parent virtual disk has been modified since the child was created).
    vmx| DISK: Cannot open disk "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk": The parent virtual disk has been modified since the child was created (18).
    vmx| Msg_Post: Error
    vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk' or one of the snapshot disks it depends on.
    vmx| [msg.disk.configureDiskError] Reason: The parent virtual disk has been modified since the child was created.----------------------------------------
    vmx| Module DiskEarly power on failed.
    Note: This indicates that the file examplevm-000002.vmdk references its parent (which in itself references another parent file), one of which has been modified some time after examplevm-000002.vmdk was created. You must take corrective measures on any of these files: examplevm.vmdk, examplevm-000001.vmdk , and examplevm-000002.vmdk.
  2. With the problem point (or points) determined, make backup copies of the disk descriptor files that require corrections or editing. In the example above, backups of examplevm.vmdk, examplevm-000001.vmdk , and examplevm-000002.vmdk are required.
  3. Review the contents of each affected descriptor file and compare the mismatching values. For example:

    examplevm.vmdk:
    CID= 12a9ffab
    parentCID=ffffffff
    examplevm-000001.vmdk:
    CID=69a1c662
    parentCID= 7b7644b2
    examplevm-000002.vmdk:
    CID=59fab513
    parentCID=69a1c662

    Disk examplevm-000002.vmdk links to examplevm-000001.vmdk without error. However, the base disk examplevm.vmdk has been modified, causing the error.
Note: The number sequences of the VMDK files are irrelevant. For example:

File              CID    PID
XXX.vmdk          1      ---------
XXX-000002.vmdk   2      1
XXX-000001.vmdk   3      2
The VMDK files forms a valid chain, though the XXX-000002.vmdk file precedes the XXX-000001.vmdk file within the chain.

Correcting the Content ID mismatch

At this point, the problem point has been identified, the virtual machine's files have backups, and corrections must be applied.
To correct the Content ID mismatch:
Warnings:
  • The CID mechanism is in place to prevent data corruption. Depending on the changes incurred upon the parent file or files, the guest operating system may be unable to boot successfully even after making corrections. A backup recovery should be made available for such circumstances.
  • The steps outlined here are potentially hazardous for your environment if they are not followed exactly. If you are not comfortable performing these steps, contact VMware Technical Support and work with them to resolve the issue.
  1. Confirm that the disks reference each other:

    Example descriptor file for a base diskexamplevm.vmdk:

    # Disk DescriptorFile
    version=1
    CID= 12a9ffab
    parentCID=ffffffff
    createType="vmfs"

    # Extent description
    RW 20971520 VMFS " examplevm-flat.vmdk"

    # The Disk Data Base
    #DDB
    ddb.toolsVersion = "0"
    ddb.adapterType = "lsilogic"
    ddb.geometry.sectors = "63"
    ddb.geometry.heads = "255"
    ddb.geometry.cylinders = "1305"
    ddb.uuid = "60 00 C2 9f ae de ba e9-95 4e a7 a6 4e 95 c1 c1"
    ddb.virtualHWVersion = "4"
    Example descriptor file for delta diskexamplevm-000001.vmdk:

    # Disk DescriptorFile
    version=1
    CID= 69a1c662
    parentCID= 7b7644b2
    createType="vmfsSparse"
    parentFileNameHint= "examplevm.vmdk"
    # Extent description
    RW 20971520 VMFSSPARSE "examplevm-000001-delta.vmdk"
    # The Disk Data Base
    #DDB
    ddb.toolsVersion = "7302"
    Example descriptor file for child delta disk examplevm-000002.vmdk:
    # Disk DescriptorFile
    version=1
    CID=59fab513
    parentCID= 69a1c662
    createType="vmfsSparse"
    parentFileNameHint= "examplevm-000001.vmdk"
    # Extent description
    RW 20971520 VMFSSPARSE "examplevm-000002-delta.vmdk"
    # The Disk Data Base
    #DDB
    ddb.toolsVersion = "7302"

    Note: The linking or chain references are highlighted in blue. This example shows that examplevm-000002.vmdk as a child ofexamplevm-000001.vmdk, which in turn is a child of examplevm.vmdk. These three disk files make up a singular virtual disk from the perspective of the virtual machine and guest operating system.
  2. Using a text editor, correct the mismatch at either of the two problem points. For more information, see Preferred Editors in Editing configuration files in VMware ESXi and ESX (1017022). You may either correct examplevm.vmdk, per this example, or the examplevm-000001.vmdk disk file. In either circumstance, the parentCID and CID relationship must be valid.

    Example: examplevm.vmdk can have its CID changed to 7b7644b2, making it match examplevm-000001.vmdk's expected parentCIDvalue. Alternatively, examplevm-000001.vmdk's parentCID value can be changed to 12a9ffab to match examplevm.vmdk's base disk descriptor file's CID. You may also consider creating a CID value on your own.

    Note: The CID consists of eight (8) hexadecimal lower-case digits (00000000-ffffffff) with no delimiter characters.

Verifying the CID corrections

The corrections made to the virtual machine files are usually not immediately acknowledged in the remainder of the product. As such, subsequent power-on attempts may not succeed as yet.
To verify the CID corrections:
  1. Log into the VMware vSphere Client or VMware Infrastructure Client.
  2. Select the virtual machine in the Inventory and click the Summary tab.
  3. Under Resources, right-click the datastore that contains the virtual machine's configuration file and select Browse. The Datastore Browser opens.
  4. Locate the virtual machine's directory and files. You can minimize it for now, as it will be used in step 7.
  5. Right-click the virtual machine and choose Remove from inventory.
  6. When prompted, confirm your selection by clicking Yes. The virtual machine disappears from the inventory on the left.
  7. Restore the Datastore Browser from step 4, right-click the virtual machine's configuration file (for example, examplevm.vmx), and choose Add to Inventory.
  8. Follow the on-screen prompts to bring the virtual machine back into the inventory.

    Warning    : Do not power on the virtual machine unless you have a valid copy or backup of all of its files. When the virtual machine has been powered on, further irreversible changes are applied to the disk structure.
  9. When ready, power on the virtual machine and verify the guest operating system's status. A file system integrity check may be performed to seek out and repair any complications that arise from the disk chain changes that caused the initial CID mismatch.

You can also verify if the CID mismatch has been corrected by running this command against the highest level snapshot .vmdk :

vmkfstools -q snapshot_ xxxxxx#.vmdk -v10 | less

or

For ESXi 5.x:

vmkfstools -e snapshot_ xxxxxx #.vmdk

Note:
  • If there are failed messages in the results, the CID mismatch has not been corrected or there is still a mismatch in the snapshot hierarchy.
  • The virtual machine must be powered off to run -e option, otherwise it will return the following error -
    Failed to lock the file (16392)Disk chain is not consistent : Failed to lock the file (16392)

Alternative procedure

At this point, the virtual machine should start successfully if the changes incurred were minimal. The guest operating system can exhibit varying symptoms, depending on what occurred in the snapshot.
However, if the virtual machine is not in an acceptable state, you may be required to restore from a backup copy. If one is not available, you can consider starting the virtual machine on older disks.
Example: The virtual disk referenced for the provided example is examplevm-000002.vmdk . If necessary, the virtual machine's configuration file can be modified to boot from examplevm-000001.vmdk, permanently invalidating examplevm-000002.vmdk. This may allow the guest operating system to proceed as intended, but at the cost of losing all information contained in the examplevm-000002.vmdk delta file.
You can log into a VMware ESX or VMware Command Line Interface to edit the virtual machine's configuration file, and you can copy the file from the datastore for editing on another system using a preferred text editor.
For additional information on this topic, see Editing configuration files in VMware ESXi and ESX (1017022).
If you are unable proceed, please file a support request with VMware Technical Support. For more information, see Filing a Support Request in My VMware (2006985).
THE CONTENT OF THIS ARTICLE IS PROVIDED "AS-IS," AND TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, VMWARE DISCLAIMS ALL OTHER REPRESENTATIONS AND WARRANTIES, EXPRESS OR IMPLIED, REGARDING THIS CONTENT, INCLUDING THEIR FITNESS FOR A PARTICULAR PURPOSE, THEIR MERCHANTABILITY, OR THEIR NONINFRINGEMENT. VMWARE SHALL NOT BE LIABLE FOR ANY DAMAGES ARISING OUT OF OR IN CONNECTION WITH THE USE OF THIS CONTENT, INCLUDING DIRECT, INDIRECT, CONSEQUENTIAL DAMAGES, LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF VMWARE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Additional Information


  • If the virtual machine is powered off and an attempt is made to consolidate the snapshots, the vmware.log file will not contain the reason for the failures. In this case, the hostd.log file is located at /var/log/ on the ESXi host where the affected virtual machine is registered on.

    In ESXi 5.x, instead of reviewing the logs, use the vmkfstools -e command to quickly identify the problem:

    The error shows the parent and child are affected

    For example:

    # vmkfstools -e APU-ESB-01-000001.vmdk
    Disk link /vmfs/volumes/5303d209-b32fa2a0-edb9-0017a477c480/APU-ESB-01_1/APU-ESB-01-000001.vmdk successfully opened.
    Disk link /vmfs/volumes/5303d209-b32fa2a0-edb9-0017a477c480/APU-ESB-01_1/APU-ESB-01-000002.vmdk successfully opened.
    Disk link /vmfs/volumes/5303d209-b32fa2a0-edb9-0017a477c480/APU-ESB-01_1/APU-ESB-01.vmdk successfully opened.
    Cannot establish parent(/vmfs/volumes/5303d209-b32fa2a0-edb9-0017a477c480/APU-ESB-01_1/APU-ESB-01.vmdk)-child(/vmfs/volumes/5303d209-b32fa2a0-edb9-0017a477c480/APU-ESB-01_1/APU-ESB-01-000002.vmdk) relationship : The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18)
    Parent link description
    Name: /vmfs/volumes/5303d209-b32fa2a0-edb9-0017a477c480/APU-ESB-01_1/APU-ESB-01.vmdk
    Create Type: vmfs
    Capacity: 26843545600
    Content ID: 1894300628
    Parent Content ID: 4294967295
    Child link description
    Name: /vmfs/volumes/5303d209-b32fa2a0-edb9-0017a477c480/APU-ESB-01_1/APU-ESB-01-000002.vmdk
    Create Type: vmfsSparse
    Capacity: 26843545600
    Content ID: 1894300628
    Parent Content ID: 3919862208
    Disk chain is not consistent : The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18)
  • This error can also occur with virtual machines running on the VMware Workstation product. The methods described here are valid for that environment, but it is necessary to use the Workstation utility vmware-vdiskmanager.exe program on a Windows system in place of the vmkfstools command.
RefferenceArticle: 
http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1007969
Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)