Move/Replace vCloud Director NFS Transfer Server Storage

In a multi-cell vCloud Director installation, all cells need access to a shared spooling area, also known as NFS transfer server storage. When you need to move or replace the NFS transfer server storage because the current presented NFS share is too small or maybe it got lost because of a crash, you can simply provide a new share to the vCloud Director cells.

The following procedure shows you how to replace the NFS transfer server storage:

Create NFS share

First you need to export a share on the new NFS server. The procedure might be different depending on the type of NFS server you’re using. RedHat version 5.7 has been used in the following procedure.

1. Create a directory to export:

mkdir /nfs/vCD-Transfer

2. Export the NFS directory. Add the following line to /etc/exports:

/nfs/vCD-Transfer <accesslist>   (rw,no_root_squash)

Note: Replace <accesslist> with the ip addresses of your VCD cells or allow a specific network like 10.1.1.0/24

3. Restart NFS service:

service nfs restart

Mount NFS share on VCD cells

When the new NFS share is ready

1. Unmount the current NFS share:

umount /opt/vmware/vcloud-director/data/transfer

2. Modify /etc/fstab and make sure the following line is present and matching your NFS server IP and directory:

<nfsip>:/nfs/vCD-Transfer /opt/vmware/vcloud-director/data/transfer nfs rw,soft,_netdev 0 0

Note: Replace <nfsip> with the ip addresses of your NFS server

3. Mount the NFS share:

mount -a

4. Verify that the permissions on the transfer directory are set to 750 (drwxr-x—). If not change it:

chmod 750 /opt/vmware/vcloud-director/data/transfer

5. Verify that both the user and group on the transfer folder are set to vcloud. If not change it:

chown -R vcloud:vcloud /opt/vmware/vcloud-director/data/transfer

6. Restart VCD cell:

service vmware-vcd restart

Note 1: Be aware that (in a production environment) tasks could be active on the cell. Restarting the cell this way will break any running tasks. You might want to quiesce the cell first as described in http://kb.vmware.com/kb/2034994

Note 2: You’ll notice a warning in the cell.log log file indicating that the cell is unable to verify that the other cells share the same spooling area. This is normal at this stage as the other cells haven’t been updated with the new NFS share yet. Just make sure that you update all cells. When a VCD cell starts it writes a marker file onto the spooling area to verify that the spooling area is writable and checks all other marker files to verify that it’s sharing the spooling area with all other cells.

Replacing VMware vCenter 5.1 Certificates Made Easy

The release of vCenter 5.1 added more components and therefore more certificates into the mix. Using CA signed certificates increase security, but the process of updating these certificates is currently very tedious and error prone.

VMware announced the general availability of vCenter Certificate Automation Tool 1.0. This tool provides an automated mechanism to replace certificates in the following components of the vCenter management platform:

  1. vCenter Server
  2. vCenter Single Sign On
  3. vCenter Inventory Service
  4. vSphere Web Client
  5. vCenter Log Browser
  6. vCenter Orchestrator (VCO)
  7. vSphere Update Manager (VUM)

The corresponding KB article can be found at: http://kb.vmware.com/kb/2041600

This tool is fully supported by VMware as well.

Versioning and Renaming Elements in vCenter Orchestrator

During the development cycle of a workflow, your workflow can change drastically whenever you add new functionalities. Therefore a good practice is saving different versions of your workflow. One way to achieve this is by utilizing the built-in versioning functionality. As a best practice, always increase the version number of your workflow when making changes and don’t forget to write a decent comment describing the last changes. Although this enables you to revert back to the saved versions using the “version history”, it doesn’t allow you to have a peak at that version. Therefore I often find it more convenient keeping a copy of the workflow or action for reference and backup. This way you can easily open a saved version without impacting the current version and risking the loss of any changes.

Note: Pressing the “Revert” button in the editor or History Version Inspector, does NOT prompt you, but reverts to the last saved or the selected version state instantly.

Workflow Elements

Making a copy of the workflow or action for backup is easy. Right-click on the item and select “Duplicate …”. But be careful when duplicating workflows or actions; every element in vCenter Orchestrator is identified by an internal ID. This allows you to have multiple items with the same name, even in the same folder. Because of this ID, you can’t simply replace a workflow by putting a copy in the original folder using the same name.

When a workflow is called from within another workflow, the linked workflow is referenced by its ID. Renaming the linked workflow doesn’t break the link in the parent workflow. When you want to replace the linked workflow with another copy or version, you have to change the linked workflow element in the parent workflow to point to the replacement. Suppose that you have a main workflow called ‘wfMain’ referencing a sub-workflow ‘wfSub’. Figure 1 shows a graphical representation of this workflow.

Figure 1

When you want to create a duplicate of ‘wfSub’ to develop new functionality without affecting users using the current workflow, you create a duplicate and call that ‘wfSub_v2′. After adding new functionality you want your users to use this new version of the workflow. You rename ‘wfSub’ to ‘wfSub_v1′ and then rename ‘wfSub_v2′ to ‘wfSub’. When viewing the main workflow, you’ll see that the name of the referenced sub-workflow has changed to ‘wfSub_v1′ according to the rename operations. This is because the sub-workflow is linked by ID. What happened under the surface in vCO is that the new copy got a new internal ID and is not linked to the parent workflow ‘wfMain’. Figure 2 shows that the referenced workflow has been changed. Note that the name of the workflow element didn’t change.

Figure 2

When you have multiple linked workflows, replacing workflows can be a tedious task as there is no direct reference back or back-link to any parent workflow. The only option is using the search feature of the client to search for references as shown in Figure 3. Therefore avoid having a production version and a ‘development’ version of the same workflow or action on one server.

Figure 3

A best practice is running a separate development instance of vCO. This way you can develop new functionalities on the development server and, when finished, synchronize the new version to the production instance. By synchronizing content between vCO instances, the synchronized items on both the source and destination instance have the same internal ID. This saves you from replacing and relinking workflows. But be careful that you keep developing in the same ‘master’ workflow and only make duplicates for backup.

Tip: Make a duplicate of your workflow or action after every significant change. I always append the version number to the element’s name for easy reference.

Action Elements

Now let’s do the same operation on an action element. Let’s add an action element to the ‘wfMain’ workflow called ‘getMyObject’. Figure 4 shows the new workflow.

Figure 4

Now when you rename the action called ‘getMyObject’ to ‘getMyObject_v2′, you’ll notice that the reference of the action element didn’t changed. In fact the workflow is unable to run because it can’t find the referenced element ‘getMyObject’. When running a workflow validation, you’ll receive an error stating that a referenced element could not be found as shown in Figure 5.

Figure 5

This is because action elements are referenced by path (their location in the action library tree). Opening the action element and viewing the ‘Scripting’ tab as shown in Figure 6 can validate this.

Figure 6

Now create a new copy of the action element and give it the original name (‘getMyObject’). When you run the validation process again, everything is fixed.

Unlike workflow elements, there is no option to relink an action element to another action. So you’ll have to replace the action element with a new one. This also means that you have to bind all input and output parameters again as well as setting the exception binding if any.

Conclusion

So before you start renaming elements, think twice. As a best practice never rename elements. If you want to keep some versions of an element as a backup while working on a solution, remember that you create a backup of the current element and keep editing in the original version and not the copy. Also make sure that you update the version number after every change, because this version number is the only parameter being evaluated by vCO when determining the elements that need to be copied to the remote server during synchronization.

vCenter Orchestrator Configuration Element Attribute Values Missing After Import.

When importing a package on a vCenter Orchestrator (vCO) server in my lab I noticed that the values of the attributes inside the Configuration Elements (CE) were missing. At first I thought that it was because of the different version of vCO. (I exported the package as a backup from a vCO 4.2.1 and imported it into a 5.1 vCO server), but I witnessed the same behavior when importing the package to vCO 4.2.1.

When searching the web for this phenomenon I couldn’t find any information describing this behavior until I caught eye on a blog post on the VMware vCenter Orchestrator blog: http://blogs.vmware.com/orchestrator/2012/02/configuration-elements-revisited.html

Somewhat hidden in the bottom text of the article the difference between exporting a single Configuration Element and exporting a Configuration Element as part of a package is explained:

Nevertheless there is a small difference with exporting a single configuration element, the difference is that in that case the values of the attributes are not exported! In another words, if you import a package containing a configuration element into another vCO, the configuration element attribute values are not set.

I was not aware of this and it kind of makes sense if you use Configuration Elements solely to hold vCO server specific information. But in my case, Configuration Elements have been used to store global information about the environment.

Also I’ve been saving the exported package as an extra kind of backup, next to the database backup, but actually being useless for the CE’s as I’ve discovered now.

The Configuration Element Content

Let’s have a closer look at the exported packages. First have a look at the Configuration Element when exported as a single configuration item. The CE is exported as an XML file. From the XML you see that the exported CE is called ‘CE1’ with three attributes called ‘att0’, ‘att1’ and ‘att2’. The <value> tag contains the value of each of the attributes. For instance attribute ‘att0’ is of type ‘string’ and has a value of “This is a Test”.

<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"?>
<config-element id="828080808080808080808080808080808180808001359849118679aebf2a6a5a5"  version="0.0.0" >
<display-name><![CDATA[CE1]]></display-name>
<atts>
<att name='att0' type='string' read-only='false' ><value encoded='n'><![CDATA[This is a Test]]></value>
<description><![CDATA[att0_desc]]></description>
</att>
<att name='att1' type='boolean' read-only='false' ><description><![CDATA[att1_desc]]></description>
</att>
<att name='att2' type='number' read-only='false' ><value encoded='n'><![CDATA[123.0]]></value>
<description><![CDATA[att2_desc]]></description>
</att>
</atts>
</config-element>

When we look at the same Configuration Element when exported as part of an Orchestrator package, you see that the <value> tag is omitted from the XML content.

<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"?>
<config-element id="828080808080808080808080808080808180808001359849118679aebf2a6a5a5"  version="0.0.0" >
<display-name><![CDATA[CE1]]></display-name>
<atts>
<att name='att0' type='string' read-only='false' ><description><![CDATA[att0_desc]]></description>
</att>
<att name='att1' type='boolean' read-only='false' ><description><![CDATA[att1_desc]]></description>
</att>
<att name='att2' type='number' read-only='false' ><description><![CDATA[att2_desc]]></description>
</att>
</atts>
</config-element>

Work Arounds

To work around this behavior of vCenter Orchestrator, I found two options:

  1. Export all Configuration Elements independently
  2. Synchronize the package to the server using the vCO synchronization option.

Option1: Export

When exporting a single Configuration Element, the values are exported as shown earlier in this post. Export all Configuration Elements separately as a single exported configuration item. Use this option when you want to create a backup or when there’s no network connectivity to the destination vCO server.

Option2: Synchronize

Use this option when you have network connectivity to your destination vCO server. This is the easiest and recommended option to copy content from one server to the other.

PHD Virtual Backup 6.1

PHD Virtual has released a new version of their award-winning data protection software PHD Virtual Backup 6.1. This version adds some major new functionality and enhancements, which include:

  • Instant Recovery for Full/Incremental Backup Mode
  • Rollback Recovery
  • Reporting Enhancements
  • Job Copy

Instant Recovery for Full/Incremental Backup Mode

With PHD Virtual Backup version 6.1, PHD Virtual brings Instant recovery to Full/Incremental Backup Mode. This functionality has been introduced in version 6.0, but was only available with their Virtual Full backup mode. In version 6.1, PHD Virtual extends the power of PHD Instant Recovery to include Full/Incremental backup mode as well. Instant Recovery allows you to start a VM from its backup location and drastically reduce RTO. After the VM is started, users can either leverage VMware Storage vMotion or use PHD Motion to move the VM’s to production storage.

Rollback Recovery

I think that Rollback recovery is a major new functionality. It allows you to restore a backup of a VM very quickly. Instead of doing a full restore, rollback recovery allows you to only restore the changed blocks (using VMware Changed Block Tracking (CBT) information) over the existing VM, effectively rolling back your VM to a previous point in time. This will enhance your RTO because there is generally only about 1-5% of changes occurring in a VM each day.

You might think why you will need Rollback Recovery when you have Instant Recovery available and are able to start a VM from the backup location straight away without waiting for any restore of vmdk files at all. The major difference is that backup files are typically stored on cheaper disks and hence slower performing storage. This kind of storage might not meet your application’s requirements when using Instant Recovery. Using Rollback Recovery you are able to restore the VM to the application’s production storage right away with only minimal downtime (depending on the number of changes to the VM).

Reporting Enhancements

Reporting enhancement in version 6.1 include:

  • Export Job History
    This allows users to export the job history table from PHD Virtual Backup v6.1 UI to Excel in CSV format.
  • Data Protection Risk Management Report
    Gain visibility into your data protection integrity by viewing a list of all VM’s in the environment, along with information on their sizes, virtual disks and the date of the last successful backup.

Job Copy

Spend less time creating and scheduling backup jobs with Job Copy. While it might seem only a small enhancement, the ability to copy a job will greatly accelerate the backup job creation process. There’s nothing more annoying than having to create a job from scratch every time.

What’s New in Version 6.1 Video

Watch this video to see a demo of the new functionality in PHD Backup 6.1

To find out more about PHD Virtual Backup go to http://phdvirtual.com

vSphere Update Manager Network Connections

Recently I was struggling getting vSphere Update Manager to work properly across firewalls. Information for the required firewall ports that need to be opened can be obtained from the following VMware KB articles:

http://kb.vmware.com/kb/1004543
http://kb.vmware.com/kb/1012382

During troubleshooting, some information got me confused and as the saying goes, “a picture is worth a thousand words.”, I decided to craft a picture from all this information.

vCenter Update Manager

Table 1 – vSphere Update Manager network port requirements.

Port Protocol Source Target
80 TCP vSphere Update Manager www.vmware.com and xml.shavlik.com
80 TCP ESXi Host vSphere Update Manager
80 TCP vSphere Update Manager vCenter Server
80 TCP vSphere Update Manager ESXi Host
443 TCP vSphere Update Manager www.vmware.com and xml.shavlik.com
443 TCP ESXi Host vSphere Update Manager
443 TCP vCenter Server vSphere Update Manager
902 TCP vSphere Update Manager ESXi Host
1433 TCP vSphere Update Manager Microsoft SQL Server
1521 TCP vSphere Update Manager Oracle Database Server
8084 TCP vCenter Server vSphere Update Manager
9084 TCP ESXi Host vSphere Update Manager
9087 TCP vCenter Server vSphere Update Manager

Restoring the default vCenter Server alarms

I’ve been troubleshooting some vCenter alarms lately, which involved a lot of editing, removing and adding of alarms. After the troubleshooting I wanted to reset everything back to default and ran into VMware KB article 2009166, explaining how to restore the default vCenter Server alarms.

To restore the default vCenter Server alarms you first have to delete all existing alarms, including the custom created alarms. Be warned that the restore procedure from the KB article will restore only the default vCenter Server alarms and not the custom created ones.

Backing up Alarms

When you’ve created a lot of custom alarms then KB Article 1032660 provides a way to backup your alarms by copying the alarms to another (temporary) vCenter server. Using the same procedure you can restore your custom alarms again and guess what? It’s done using a PowerCLI script :-)

Deleting Alarms

As per KB article:
To delete all the existing alarm definitions in the vCenter Server:

  1. Log in to the vCenter Server using the vSphere Client.
  2. Click the Hosts and Clusters view.
  3. Click the Alarms tab.
  4. Click Definitions.Note: All custom alarms must be manually recreated after they are removed. Ensure to note the custom alarms before removing.
  5. Select all alarms, right-click, and  click Remove.

Note: ALL vCenter Alarms need to be removed on ALL objects, not just only the alarms on the vCenter object.

Deleting Alarms using PowerCLI

When you’ve created alarms on different objects, going through the GUI deleting all of them might be a tedious task. You can easily remove all vCenter Server alarms using PowerCLI with just a one-liner regardless of the object the alarm is defined on.

Get-AlarmDefinition | %{$_.ExtensionData.RemoveAlarm()}

Restoring the Default Alarms

As per KB article:
To restore the default vCenter Server alarms, change the alarm version in vCenter Server:
  1. Connect to vCenter Server using the vSphere Client.
  2. Click Administration > vCenter Server Settings.
  3. Click Advanced Settings.
  4. Set alarm.version to 0.
  5. Set alarms.upgraded to false.
  6. Restart the VMware VirtualCenter Server service.

Creating Custom Alarms

If you’re interested in creating custom alarms then the document available at http://communities.vmware.com/docs/DOC-12145 is a great read.

PowerCLI automation: Create vCenter Notification Email Alarm Action

I’ve received a request from a customer recently who wanted to configure all vCenter alarms to send a notification email. This instantly made me decide that PowerCLI was the best way to go rather than wasting a lot of valuable time in the vSphere client configuring the email notification alarms.

PowerCLI Automation Script:

I’ve created a quick and dirty script to do the job that I wanted to share with you. It’s probably not the fastest or the best optimized script, but it got the job done.

Get-AlarmDefinition | %{
   $_ | Set-AlarmDefinition -ActionRepeatMinutes (60 * 24);
   $_ | New-AlarmAction -Email -To "vcenteralarms@customer.corp" | %{
      $_ | New-AlarmActionTrigger -StartStatus "Green" -EndStatus "Yellow" -Repeat
      $_ | Get-AlarmActionTrigger | ?{$_.repeat -eq $false} | Remove-AlarmActionTrigger -Confirm:$false
      $_ | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Red" -Repeat
   }
}

Line 1: The script first retrieves all the alarms. You can easily adapt the script to update only a specific alarms by changing this line to include the name of the alarm you want to update like:

Get-AlarmDefinition "Datastore usage on disk"

or even all datastore alarms with:

Get-AlarmDefinition "Datastore*"

Line 2: The script then configures the alarm action repeat frequency to repeat the action every 24 hours. The value must be specified in minutes and the default value is to repeat every 5 minutes.

Line 3: On this line a new send notification email action is created to send an email to the email address vcenteralarms@customer.corp.

Line 4: This creates an alarm action trigger when the status changes from “Green” to “Yellow” or from “Normal” to “Warning”.

Line 5: During the creation of the send notification email alarm action, a default alarm action trigger is created. Because there’s currently no Set-AlarmActionTrigger cmdlet available, I decided that it’s easier to remove the default trigger rather than trying to change it from “Once” to “Repeat” by falling back to using the SDK methods. I warned you that it was quick and dirty ;-)

Line 6: finally this line creates an alarm action trigger when the status changes from “Yellow” to “Red” or from “Warning” to “Alert”.

Have Fun!

Top Bloggers Voting Time Again

Eric Siebert has opened up another top VMware & virtualization blogs poll over at vSphere-land.com.  Please take a moment and cast your vote  for your favourite 10 bloggers.  It’ll take only a couple of minutes of your time and you can win a copy of the Train Signal’s new vSphere 5 and View 5 video training courses.

So head on over to http://vote.vsphere-land.com to cast your vote and reward the best bloggers for their hard work and dedication by letting them know that you appreciate them. They deserve it!

PHD Virtual Backup and Replication 5.3

PHD Virtual Backup and Replication 5.3, earlier demonstrated at VMworld 2011 by PHD Virtual, is coming soon. According to PHD Virtual it will be offering even faster backup and restore of virtual machines and new technologies that offer more flexibility for disaster recovery of your virtual environment.

NEW FEATURES

  • New Job Processing Engine with I/O optimizations
  • Up to 8 Concurrent Data Streams per VBA (Enterprise Edition)
  • Virtual Machine Replication
  • Virtual Machine Test Mode
  • Mass Restore Virtual Machines
  • Open Export of VM’s from Backup Storage to standard OVF
  • Tape Friendly Backup Support

For more information have a look at the PHD5.3 Datasheet or watch the recently released demo video below.