VMware Remote Console Freeze or Black Screen

The latest version of VMware Remote Console is 10.0.2. There should be some functions changing in the release. You may see following symptoms after upgrading to the version.

  1. The virtual machine console screen is black.
  2. Console screen works properly if you just hit buttons on VMware Remote Console window.
  3. The console screen freeze once VMware Remote Console grabs mouse or keyboard in screen.

The cause is your  anti-virus software blocked the internal functions of grab mouse/keyboard behavior of VMware Remote Console. Try disable anti-virus temporarily.

How To Migrate Parent Disk on Hyper-V 2012

If you are using Microsoft Hyper-V 2012 and “Differencing Disk” you may get trouble when you want to move whole VMs to another location due to “Parent Disk” migration is not so easy. Following is the steps to move parent disk on Hyper-V server.

Preparation

I assume you want to move bunch of virtual machines. First of all you need to get disk list of virtual machines. Following is a script to grab all parent and differencing disks on a Hyper-V server.

$VMs = Get-VM 
Foreach ($VM in $VMs)
{
  $VHDs = Get-VHD -Path $VM.harddrives.path
  foreach ($VHD in $VHDs)
  {
     [pscustomobject]@{
         Name = $VM.name
         VHDType = $VHD.VhdType
         VHD = $VHD.Path
         ParentVHD = $VHD.ParentPath
     }
  }
}

Save it to “Get-vhdParent.ps1”. Launch PowerShell by administrator right. Run following command to get parent disk table.

.Get-vhdParent.ps1 | format-table -autosize

Now you have disk list in hand.

Move parent disks to new location

Parent disk moving is simple. Just copy the parent disk to new location. I suggest make multiple copies if you have large number of virtual machines linked to a parent disk. The reason is if the parent disk failed, at lease it’s not impact to all linked virtual machines. You can also distribute the duplicated parent disks to multiple location to avoid single location failure.

Re-configure parent disks for virtual machine

To be safe, I suggest grab parent disk information again by following command:

Get-VHD -Path VHDPath

Replace “VHDPath” with real differencing disk path of the virtual machine.

The output shows what’s the linked parent disk. Then run the command below to reconfigure parent disk to new location.

Set-VHD -Path VHDPath -ParentPath ParentVHDPath

You should get nothing return if it’s successfully.

If you manage Hyper-V virtual machines by System Center Virtual Machine Manager. The new parent disk is reflected after you right click the virtual machine and do a “Refresh” in System Center Virtual Machine Manager console.

 

 

 

Cannot Complete File Creation Operation When Storage vMotion

Just quick notes. I saw following error  when do storage vMotion.

Cannot Complete File Creation Operation.

When check /var/log/hostd.log. I saw following errors:

2017-11-28T02:51:04.476Z info hostd[76A80B70] [Originator@6876 sub=Vimsvc.TaskManager opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Task Created : haTask--vim.host.OperationCleanup
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] CopyFromEntry: Hostlog_Dump: Hostlog /vmfs/volumes/598700ee-ec
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] UUID: 28dbb1b5-a9d8-e311-1061-03300000002d
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] MigID: 1511837464286041
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] HLState: none
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] ToFrom: none
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] MigType: invalid
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] OpType: nfc
2017-11-28T02:51:04.476Z info hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] WorldID: 0
2017-11-28T02:51:04.478Z warning hostd[772C2B70] [Originator@6876 sub=Libs opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Hostlog_Flush: Failed to open hostlog /vmfs/volumes/598700e
2017-11-28T02:51:04.478Z warning hostd[772C2B70] [Originator@6876 sub=Vcsvc.OCM opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] PersistToDisk: failed to persist entry /vmfs/volumes/5
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Default opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] AdapterServer caught exception: vim.fault.CannotCreateFile
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Vimsvc.TaskManager opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Task Completed : haTask--vim.host.OperationClean
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Activation [N5Vmomi10ActivationE:0x75395c80] : Invoke do
2017-11-28T02:51:04.478Z verbose hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Arg entry:
--> (vim.host.OperationCleanupManager.OperationEntry) {
--> hlogFile = "/vmfs/volumes/598700ee-ec0f9918-5b56-000000000000/XXX-VM-01/XXX-VM-01-375f29ae.hlog",
--> opId = 1511837464286041,
--> opState = "running",
--> opActivity = "nfc",
--> curHostUuid = "28dbb1b5-a9d8-e311-1061-03300000002d",
--> }
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Throw vim.fault.CannotCreateFile
2017-11-28T02:51:04.478Z info hostd[772C2B70] [Originator@6876 sub=Solo.Vmomi opID=459515D4-000040D6-2d-cf-d4-7817 user=vpxuser:contosotestuser] Result:
--> (vim.fault.CannotCreateFile) {
--> faultCause = (vmodl.MethodFault) null,
--> file = "/vmfs/volumes/598700ee-ec0f9918-5b56-000000000000/XXX-VM-01/XXX-VM-01-375f29ae.hlog",
--> msg = ""
--> }

It indicates there is a file cannot be created during migration. Further check on VM configuration file (.vmx) I noticed following parameter existing but the file doesn’t existing.

migrate.hostlog = "XXX-VM-01-375f29ae.hlog"

You cannot create the file directly. Workaround is create a .hlog file with other name then rename it to the same name.

BTW, there is a bug on ESXi 6.0 U1 for similar issue, but I saw this problem  on  U2. Just for your reference below.

Storage migration of a virtual machine with a name beginning with core fails with the error: Relocate virtual machine coreXX Cannot complete the operation because the file or folder coreXX-XXXXX.hlog already exists

Virtual Machine Cloning Is Failed At 33%

I got two exactly same hardware, installed same ESXi version. Somehow cloning from  other ESXi to one server was working, but another one always failed at 33%.

It only impacts to existing VM cloning but not impact to new created virtual  machines. I spent lot time on the troubleshooting. We replaced cables, switch ports, reinstalled ESXi.

There was no  abnormal logs except following:

2017-11-15T05:47:36.023Z [FFF001A0 verbose 'NfcManager' opID=F39DF7E6-00002211-da-c-bf] [NfcClient] Closing NFC connection to server

2017-11-15T05:47:36.023Z [FFF001A0 warning 'Libs' opID=F39DF7E6-00002211-da-c-bf] SSL: Unknown SSL Error

2017-11-15T05:47:36.023Z [FFF001A0 info 'Libs' opID=F39DF7E6-00002211-da-c-bf] SSL Error: error:1409E10F:SSL routines:SSL3_WRITE_BYTES:bad length

VMware support eventually provided me following KB to workaround the problem. Looks like it’s a bug on ESXi 5.5 2068190.

Disabling SSL for NFC data traffic in vCenter Server

Cloning or deploying from a template takes longer time after upgrading to VMware vSphere 5.1 Update 2 and 5.5

How To Get Used Space By PowerShell

I searched internet but hard to find an easy way to get used space on Windows Server.

Following is two lines PowerShell command to get used space on  Windows 2012 R2 Server.

Get-WmiObject win32_logicaldisk | select deviceid,@{n="Size";e={[math]::Round(($_.size/1GB),2)}},@{n="Used Space";e={[math]::Round((($_.Size-$_.FreeSpace)/1GB),2)}}

Could Not Complete Network Copy For File During VM Cloning

This error may only appears on legacy ESXi hosts. The cloning of virtual machine throws error at 33% of the task.

 Clone virtual machine
Could not complete network copy for file 
/vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST/TEST.vmdk
Copying Virtual Machine files

You may see following logs on vpxa.log of source ESXi host.

2017-11-02T02:51:20.238Z [FFF43B70 info 'Libs' opID=812BF517-00000667-f0-34-64] SSL: syscall error 32: Broken pipe
2017-11-02T02:51:20.238Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NfcNetTcpWrite: bWritten: -1
2017-11-02T02:51:20.238Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NfcFile_SendMessage: data send failed:
2017-11-02T02:51:20.238Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NFC_NETWORK_ERROR
2017-11-02T02:51:20.239Z [FFF43B70 error 'NfcManager' opID=812BF517-00000667-f0-34-64] [NfcClient] File transfer [/vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST/TEST.vmdk -> /vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST1/TEST1.vmdk] failed: The operation experienced a network error
2017-11-02T02:51:20.239Z [FFF43B70 verbose 'NfcManager' opID=812BF517-00000667-f0-34-64] [NfcClient] Closing NFC connection to server
2017-11-02T02:51:20.239Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] SSL: Unknown SSL Error
2017-11-02T02:51:20.239Z [FFF43B70 info 'Libs' opID=812BF517-00000667-f0-34-64] SSL Error: error:1409E10F:SSL routines:SSL3_WRITE_BYTES:bad length
2017-11-02T02:51:20.239Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NfcNetTcpWrite: bWritten: -1
2017-11-02T02:51:20.239Z [FFF43B70 warning 'Libs' opID=812BF517-00000667-f0-34-64] [NFC ERROR] NfcSendMessage: send failed: NFC_NETWORK_ERROR
2017-11-02T02:51:20.239Z [FFF43B70 error 'NfcManager' opID=812BF517-00000667-f0-34-64] [NfcWorker] Error encountered while processing copy spec for file [ds:///vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST/TEST.vmdk -> ds:///vmfs/volumes/5xxxxxxe-4fb01111-3911-0ccxxxxxac38/TEST1/TEST1.vmdk]:
--> vim.fault.NetworkCopyFault
2017-11-02T02:51:20.239Z [FFF43B70 error 'NfcManager' opID=812BF517-00000667-f0-34-64] [NfcManagerImpl] Copy operation failed with error: vim.fault.NetworkCopyFault

You may see following logs in vpxa.log of destination ESXi host.

2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] SSL: Unknown SSL Error
2017-11-02T02:51:18.289Z [304EEB70 info 'Libs' opID=task-internal-2164-739c5f01] SSL Error: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcNetTcpRead: bRead: -1
2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcNet_Recv: requested 262144, recevied only 16384 bytes
2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcFile_RecvMessage: data recv failed. retval = 3, expected 262144
2017-11-02T02:51:18.289Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcFile_ContinueReceive: failed to Recv message
2017-11-02T02:51:18.446Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcProcessStreamMsg: failed to receive file data
2017-11-02T02:51:18.446Z [304EEB70 warning 'Libs' opID=task-internal-2164-739c5f01] [NFC ERROR] NfcServerLoop: NfcServer_HandleRead returned an error : NFC_NETWORK_ERROR
2017-11-02T02:51:18.446Z [304EEB70 error 'provisioningvpxNfcServer' opID=task-internal-2164-739c5f01] [VPXNFCSERVER] Nfc server failed with return value : NFC_NETWORK_ERROR
2017-11-02T02:51:18.446Z [304EEB70 verbose 'provisioningvpxNfcServer' opID=task-internal-2164-739c5f01] [VPXNFCSERVER] Closing NFC session

It indicates the VM may be created on an older ESXi host or VMware Workstation. Somehow it imported to current ESXi host. Solution is create a new VM on the ESXi host and attach only virtual disks of the problematic VM.

Disable PXE Boot for Individual vNIC on Virtual Machine

To achieve Auto Deploy I’d like to control PXE boot process. I want the vNICs of management network can do PXE boot only. That’s because DHCP server may learns incorrect MAC address of management network if ESXi host boots up by non-management network NICs.

The psychical servers are easy to disable PXE boot feature of individual network adapters in BIOS or server profile. Virtual machines is tricky. Following is how to do it. It’s useful for home lab.

  1. Make sure your ESXi VM uses E1000E vNIC. You can only disable PXE boot for all vNICs in one time if type is vmxnet3. And nested ESXi doesn’t support E1000 vNICs.
  2. Go to the VM folder and edit vmx file.
  3. You should see similar entries below. The vNIC name starts from  ethernet0 to ethernetn. It matches vmnic0 to vmnicx on ESXi.

    ethernet1.virtualDev = “e1000e”

  4. I have 4 vNICs. I want to keep PXE boot for ethernet0 and ethernet1. So I only disable it on rest of vNICs. Add following lines to vmx file.

    ethernet2.opromsize = “0”
    ethernet3.opromsize = “0”

  5. Save the vmx file and quit.
  6. Power on the VM.

Please make sure you power on the virtual machines by vSphere Client or vSphere Web Client. As you may know VM console may opened by VMware Workstation if workstation and vSphere Client both existing on your computer. Looks like sometimes the parameter doesn’t work if you power on the  VM by VMware Workstation.

VMware KB Disabling Network boot option from appearing in a virtual machine’s BIOS (1014906) talks about same thing but the value of parameters looks like incorrect for ESXi 6.5.

How To Find Non-tagged ESXi Hosts

There are plenty of scripts to find tagged ESXi hosts. But what if you want to find out all ESXi hosts not be tagged? Following is a simple script:

Compare-Object ((Get-VMHost | Get-TagAssignment).Entity | select -uniq) (Get-VMHost)

The output is similar like following:

InputObject      SideIndicator
-----------      -------------
esx1         =>  
esx2         =>  
esx3         =>

The => indicates ESXi hosts in InputObject are not tagged.

If return is nothing, it means all ESXi hosts are tagged.

Please refer to “Using the Compare-Object Cmdlet” for detail.

Unable to Upgrade to Windows Server 2012 R2

I searched internet but there is no more information about this specific error message.

When you upgrade to Windows Server 2012 R2 or 2016. You may see following error message:

Windows won’t install unless each of these things is taken care of. Close Windows Setup, take care of each one, and then restart Windows Setup to continue.

Upgrades to this build have been disabled.

The reason is there is a hidden parameter in the image disabled uprading. You can only re-install by the image but cannot do upgrading. You have to ask vendor provide you a right copy, or buy Microsoft official image to do upgrading. I cannot publish the parameter due to legal reason.

Most of hardware vendors sale Windows copy along with new hardware. This kind of Windows calls OEM version. There are several different versions of Windows:

OEM SLP – This key comes pre-installed in Windows, when it comes from the Factory. This key is geared to work with the OEM Bios Flag found only on that Manufacturer’s computer hardware. So when Windows was installed using the OEM SLP key (at the factory) Windows looks at the motherboard and sees the proper OEM Bios Flag (for that Manufacturer and that version of Windows) and Self-Activates. (that’s why you did not need to Activate your computer after you brought it home)

OEM COA SLP – This is the Product key that you see on the sticker on the side (or bottom) of your computer. It is a valid product key, but should only be used in limited situations. The key must be activated by Phone. Usually you don’t have to input key during Windows installation since it check your hardware to get key.

OEM COA NSLP – Similar to OEM COA SLP license. Only different is you need to input the key during Windows installation. You must follow EULA to stay the copy on same computer forever.

Retail – Product keys are what the customer gets when he buys a Full Packaged Product (FPP), commonly known as a “boxed copy”, of Windows from a retail merchant or purchases Windows online from the Microsoft Store.

KMS Client and Volume MAK – They are issued by organizations for use on client computers associated in some way with the organization. Volume license keys may not be transferred with the computer if the computer changes ownership. Consult your organization or the Volume Licensing Service Center for help with volume license keys.

Hardware vendors may don’t allow you upgrade Windows in certain licensing mode. So they may provide you a newer Windows image to request you  do re-install on the computers but not upgrading.

Please refer following links for license key details.

What is the difference between SLP and NSLP versions of Windows 7?

Windows License Types Explained

HPE OEM Microsoft Windows Server FAQ Series- Part 1: Licensing Overview

HPE OEM Microsoft Windows Server FAQ Series- Part 2: OEM Licensing Basics

HPE OEM Microsoft Windows Server FAQ Series- Part 3: Microsoft Certificate of Authenticity (COA)

HPE OEM Microsoft Windows Server FAQ Series- Part 4: Windows Server 2016 Basics

HPE OEM Microsoft Windows Server FAQ Series- Part 5: Core-Based Licensing

HPE OEM Microsoft Windows Server FAQ Series- Part 6: Reseller Option Kit

HPE OEM Microsoft Windows Server FAQ Series- Part 7: Client Access Licenses (CALs)

HPE OEM Microsoft Windows Server FAQ Series- Part 8: OEM License Support