ESXCLI the PowerCLI way

When trying to perform a Round Robin (RR) test on my vSphere 4.1 environment, I needed to know the number of paths available to the luns of the test servers. I also needed to change the Path Selection Policy (PSP) for the luns to Round Robin (VMW_PSP_RR). I could have done this easily using the VI client, but this must be done on a per device basis and remember the golden rule of scripting: “Whenever you need to do a task more than once, script it!” 

Changing the multipath policy on the command line is normally done using the esxcli command. Reading the release notes of PowerCLI 4.1.1 a while back, I noticed that there was a new cmdlet available called Get-EsxCli. This was the perfect opportunity to give that cmdlet a try and see what it’s capable of. 

Before you read any further, please notice that the Get-EsxCli cmdlet is experimental. Also notice that in order to retrieve an esxcli instance you have to be connected directly to an ESX host. 

Retrieving and using an esxcli instance

You can retrieve an esxcli instance using: 

$esxcli = Get-EsxCli

You now have access to the esxcli instance using the $esxcli variable. Normally on the command line you would use the command “esxcli nmp paths list” to list all the paths that are claimed by the Native Multipath Plugin (NMP). In PowerCLI this is similar: 

$esxcli.nmp.path.list()

If you want to list the paths of only a specific device, you can narrow down the output by specifying the device as a parameter to the list() method.

$esxcli.nmp.path.list("naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")

Reporting the number of available paths

Now back to my original goal, reporting the number of paths for my devices. This can easily be achieved by grouping the output of the $esxcli.nmp.path.list() command.

$esxcli.nmp.path.list() | group-Object –Property Device

Changing the path selection policy

Next step is changing the PSP, but I only want that for those devices that have 8 path available. First you need to filter out the appropriate objects. 

$esxcli.nmp.path.list() | group-Object –Property Device | `
    Where {$_.count –ge 8}

Notice that, in order to filter on the number of available paths, I used the Group-Object cmdlet. If you want to change the PSP for all SAN devices, you need to create a filter on the device identifier. This is necessary because all device are returned, including device connected to local IDE or SATA controllers. These local devices are identified using a Multi Path X (MPX) identifier. SAN device are normally identified using a Network Addressing Authority (NAA) identifier.

$esxcli.nmp.device.list() | Where {$_.device -like "naa.*"}

Now you need to change the PSP. This is achieved by calling the setpolicy() method. The syntax of this method is:

$esxcli.nmp.device.setpolicy(<default>, <device>, <psp>)

The following parameters are available:

<default> If set to $true, the policy for the device is set back to the default for the assigned Storage Array Type Plugin (SATP)
<device> a string representing the device that you want to change the PSP for
<psp> a string specifying the PSP you whish to assign to the device

Remember that you always need the provide a value for all method parameters. If you don’t want to pass a value, specify $null for that parameter. Let’s change the PSP for the previously selected devices.

$esxcli.nmp.path.list() | group-Object –Property Device | `
    Where {$_.count –ge 8} | %{
        $esxcli.nmp.device.setpolicy($null, $_.Name, "VMW_PSP_RR")
    }

The following default Path Selection Policies are available: 

VMW_PSP_MRU Most Recently Used Path Selection
VMW_PSP_FIXED Fixed Path Selection
VMW_PSP_FIXED_AP Fixed Path Selection with Array Preference
VMW_PSP_RR Round Robin Path Selection

You can list the active PSPs in your environment using:

$esxcli.nmp.device.list() | select PathSelectionPolicy -Unique

Changing the default selection policy

After you’ve set the appropriate PSP for your devices, new devices will still receive the policy that is set as the default for the assigned SATP. However, you can change the default PSP, to ensure that new devices receive the RoundRobin PSP when discovered. On the command line you would normally use the command “esxcli nmp satp setdefaultpsp -psp VMW_PSP_RR -satp VMW_SATP_SVC”. To change the default PSP in PowerCLI you use:

$esxcli.nmp.satp.setdefaultpsp("VMW_PSP_RR","VMW_SATP_SVC")

Notice that I’ve been using IBM SVC as array type. In your environment this is likely to be different. To list the SATPs in use in your environment issue the command:

$esxcli.nmp.device.list() | select StorageArrayType -Unique

Setting round robin policy details

When using the Round Robin PSP, by default it will switch paths every 1000 Input/Output operations (IOPS). This behavior can be adjusted. There are 2 different RR thresholds that can be used for switching paths:

  1. number of IOPS
  2. number of bytes

The default values are 1000 IOPS or 10485760 bytes (10 MB). You can retrieve the round robin policy details from a device using the getconfig() Method.

$esxcli.nmp.roundrobin.getconfig("naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxx")

Some storage vendors recommend to change the IOPS value to 1. To change this we first need to retrieve all devices that are using the round robin PSP. This can be achieved simply by filtering the devices on the PathSelectionPolicy property, using the code below:

$esxcli.nmp.device.list() | Where {
  $_.PathSelectionPolicy -eq "VMW_PSP_RR"
}

Now, to change the round robin policy details, you need to use the setconfig() method. This method accepts the following parameters:

bytes specifies the number of bytes at which a path change should occur. The device will switch to the next available path each time the specified number of bytes have been sent along the current path.
device specifies the device you whish to set the round robin settings for.
iops specifies the number of I/O operations at which a path change should occur. The device will switch to the next available path each time the specified number of I/O operations have been performed on the current path.
type specifies the type of round robin path switching that will be used for the device. Valid options are “bytes”, “iops” or “default”.
useANO specifies if non-optimized paths should be used for the device. If set to $true, non-optimized paths will be included in the set of available paths for round robin load balancing.

To change the number of IOPS to 1 we issue the command

$esxcli.nmp.roundrobin.setconfig `
    ($null,"naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",1,”iops”,$null)

To change this setting on all devices

$esxcli.nmp.device.list() | Where {
  $_.PathSelectionPolicy -eq "VMW_PSP_RR"} | %{
  $esxcli.nmp.roundrobin.setconfig($null, $_.device, 1, ”iops”, `
      $null)
}

Notice that $null was specified for all other parameters. This is mandatory. When you set the RR type to “iops”, the bytes parameter is ignored. Counter wise, when specifying “bytes” as the RR type, the iops parameter is ignored. Whenever you specify “default” as the RR type, both the iops- and bytes parameters are ignored and their values are reset to the default values.

Putting it all together for my cluster

Because you want to make sure that you change the PSP for your devices on all your hosts in your cluster, let’s first write some code to connect to each server of a cluster.

foreach ($esx in Get-Cluster "MyCluster" | Get-VMHost) {
    Connect-VIServer $esx -Credential $esxCred
}

Now that we have an active connection to each host, we can retrieve the esxcli instances using the Get-EsxCli cmdlet again. Notice that this cmdlet now returns an array of esxcli instances, as you are now connected to multiple hosts. 

To change the PSP for the devices on all hosts we need to iterate through this array of instances. The code below will change the PSP for all devices with 8 or more paths on all hosts in the cluster. 

#Define some variables
$vcServer = "vCenter01"
$cluster = "CL01"
$psp = "VMW_PSP_RR"
$satp = "VMW_SATP_SVC"
$esxCred = Get-Credential

#Connect to vCenter
Connect-VIServer $vcServer | Out-Null

#Connect to ESX hosts in cluster
foreach ($esx in Get-Cluster $cluster | Get-VMHost) {
	Connect-VIServer $esx -Credential $esxCred | Out-Null
}

#Retrieve the esxcli instances and loop through them
foreach($esxcli in Get-EsxCli) {

	#Change PSP for my devices with 8 or more paths
	$esxcli.nmp.path.list() | group-Object –Property Device | Where {$_.count –ge 8} | %{
		$esxcli.nmp.device.setpolicy($null, $_.Name, $psp) | Out-Null
		$esxcli.nmp.roundrobin.setconfig($null, $_.Name, 1, "iops", $null) | Out-Null
	}

	#Change the default PSP for my SATP
	$esxcli.nmp.satp.setdefaultpsp($psp,$satp) | Out-Null
}

#Disconnect from ESX hosts
foreach ($esx in Get-Cluster $cluster | Get-VMHost) {
	Disconnect-VIServer $esx.name -Confirm:$false
}

#Disconnect from vCenter
Disconnect-VIServer $vcServer -Confirm:$false | Out-Null

Related posts:

  1. PowerCLI: Match VM and Windows harddisks – Part 2 Tweet This is a follow up on a post I did a couple of weeks ago to create a mapping table between Windows- and VMware hard disks. In another previous...
  2. PowerCLI: Get WMI info from isolated guests Tweet A few weeks back I posted an article on matching Windows and VMware disks. Unfortunately this would work only if you could remotely query WMI information from that VM....
  3. PowerCLI: Disable/Enable HA and DRS Tweet Before upgrading my Virtual Center 2.5 server to vCenter Server 4.0, I decided to temporarily disable HA and DRS. This is just a precaution taken to avoid waiting for...
  4. Setting custom attributes with VMware PowerCLI Tweet Last week I wanted to extend my vCenter with some extra custom attributes on my VMs. This would extend the usability of the Export List feature for reporting purposes....
  5. Match VM and Windows harddisks using PowerCLI Tweet Today I was asked to extend a disk of a Windows virtual machine. Normally this is a standard procedure and finished within minutes. The hardest part of the procedure...

18 Comments on “ESXCLI the PowerCLI way”

  1. #1 Ben
    on Jan 4th, 2011 at 8:25 pm

    Thanks for the ‘iops’ example, that’s a tough one to figure out!

  2. #2 mvoss18
    on Jan 23rd, 2011 at 5:22 pm

    Thanks for this blog! It connect to my vCenter and lists the names of all the hosts in my cluster so I know it’s *close*. It’s just having some issue with the $esxCred = Get-Credential . Do you have any pointers to get this working? I’m a newbie with PowerCLI, so I may just be leaving out something. Here is the error I get:

    [vSphere PowerCLI] C:\Program Files\VMware\Infrastructure\vSphere PowerCLI> c:\temp\set_mpath_iops.ps1

    Connect-VIServer : The argument cannot be null or empty.

    At C:\temp\set_mpath_iops.ps1:13 char:35

    + Connect-VIServer $esx -Credential <<<< $esxCred | Out-Null

    Connect-VIServer : The argument cannot be null or empty.

    At C:\temp\set_mpath_iops.ps1:13 char:35

    + Connect-VIServer $esx -Credential <<<< $esxCred | Out-Null

    Connect-VIServer : The argument cannot be null or empty.

    At C:\temp\set_mpath_iops.ps1:13 char:35

    + Connect-VIServer $esx -Credential <<<< $esxCred | Out-Null

    Connect-VIServer : The argument cannot be null or empty.

    At C:\temp\set_mpath_iops.ps1:13 char:35

    + Connect-VIServer $esx -Credential <<<< $esxCred | Out-Null

    Get-EsxCli : 1/23/2011 11:13:44 AM Get-EsxCli The requested operation is only supported when connected directly to ESX host.

    At C:\temp\set_mpath_iops.ps1:17 char:30

    + foreach($esxcli in Get-EsxCli) <<<< {

    Disconnect-VIServer : 1/23/2011 11:13:44 AM Disconnect-VIServer Could not find VIServer with name 'ESX001'.

    At C:\temp\set_mpath_iops.ps1:31 char:21

    + Disconnect-VIServer <<<< $esx.name -Confirm:$false

    Disconnect-VIServer : 1/23/2011 11:13:44 AM Disconnect-VIServer Could not find any of the servers specified by name.

    At C:\temp\set_mpath_iops.ps1:31 char:21

    + Disconnect-VIServer <<<< $esx.name -Confirm:$false

    Disconnect-VIServer : 1/23/2011 11:13:44 AM Disconnect-VIServer Could not find VIServer with name 'ESX002'.

    At C:\temp\set_mpath_iops.ps1:31 char:21

    + Disconnect-VIServer <<<< $esx.name -Confirm:$false

    Disconnect-VIServer : 1/23/2011 11:13:44 AM Disconnect-VIServer Could not find any of the servers specified by name.

    At C:\temp\set_mpath_iops.ps1:31 char:21

    + Disconnect-VIServer <<<< $esx.name -Confirm:$false

    Disconnect-VIServer : 1/23/2011 11:13:44 AM Disconnect-VIServer Could not find VIServer with name 'ESX003'.

    At C:\temp\set_mpath_iops.ps1:31 char:21

    + Disconnect-VIServer <<<< $esx.name -Confirm:$false

    Disconnect-VIServer : 1/23/2011 11:13:44 AM Disconnect-VIServer Could not find any of the servers specified by name.

    At C:\temp\set_mpath_iops.ps1:31 char:21

    + Disconnect-VIServer <<<< $esx.name -Confirm:$false

    Disconnect-VIServer : 1/23/2011 11:13:44 AM Disconnect-VIServer Could not find VIServer with name 'ESX004'.

    At C:\temp\set_mpath_iops.ps1:31 char:21

    + Disconnect-VIServer <<<< $esx.name -Confirm:$false

    Disconnect-VIServer : 1/23/2011 11:13:44 AM Disconnect-VIServer Could not find any of the servers specified by name.

    At C:\temp\set_mpath_iops.ps1:31 char:21

  3. #3 Arnim van Lieshout
    on Jan 24th, 2011 at 4:51 pm

    The $esxCred variable should hold your ESX host’s login credentials. This can be done using $esxCred = Get-Credential
    This cmdlet pops up a login dialog box where you need to enter your host’s credentials.
    You can also use the -UserName and -Password parameters instead of the -Credential parameter, but then you have to provide your credentials in plane text.

  4. #4 mvoss18
    on Jan 25th, 2011 at 3:10 am

    Thanks Arnim. In some documentation I found from HP, they mention that you should run the following to set the default policy to round robin:

    esxcli corestorage claiming unclaim –type location
    esxcli nmp satp setdefaultpsp –satp VMW_SATP_DEFAULT_AA –psp VMW_PSP_RR
    esxcli corestorage claimrule load
    esxcli corestorage claimrule run

    But in your script, you’re only running the second statement with the nmp satp setdefaultpsp. In what I found on the corestorage claimrule commands, that really has to do with LUN Masking or is commonly used during an upgrade. I’m doing a new deployment and all the iSCSI LUN assignments are configured at the storage level, so you do you think these extra corestorage commands are required if I’m not doing masking at the ESX level?

  5. #5 Technology Short Take #11 - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers
    on Feb 18th, 2011 at 12:02 am

    [...] Arnim van Lieshout posted an interesting article on how to configure EsxCli using PowerCLI. [...]

  6. #6 Arnim van Lieshout
    on Mar 17th, 2011 at 10:11 pm

    I’m not doing LUN masking at the ESX level either. I think you should be fine without the corestorage commands.
    At least I haven’t run into any issues without them. When I find the time I will investigate further and when absolutely necessary I will post a follow up.

  7. #7 VCAP-DCA Study Notes – 1.3 Complex Multipathing and PSA plugins | www.vExperienced.co.uk
    on Apr 16th, 2011 at 10:56 am

    [...] With the release of vSphere 4.1.1 you can now set this using PowerCLI. See this post from Arnim Van Lieshout for [...]

  8. #8 VMJim » Blog Archive » PowerCLI for setting multipathing policies
    on Jun 10th, 2011 at 6:46 pm

    [...] You can find an article on this cmdlet here.  There is also a good blog post here. [...]

  9. #9 Alasdair Carnie
    on Jul 26th, 2011 at 10:05 am

    Hi Arnim,

    I’m trying to use Get-EsxCli to mask off some Luns at ESXi Host Level, but I can’t get the syntax to work.

    $esxcli = Get-esxcli $esxHost
    $esxcli.corestorage.claimrlu.add()

    I’ve tried every combination I can think of to add a mask, but it just errors. Can you please help.

    Best Regards,

    Alasdair………

  10. #10 Masking LUN paths using ESXCLI the PowerCLI way | Arnim van Lieshout
    on Jul 27th, 2011 at 12:48 pm

    [...] got a question on my previous post about the Get-EsxCli cmdlet, ESXCLI the PowerCLI way, from Alasdair Carnie who was having trouble using the Get-EsxCli cmdlet to mask luns at the ESXi [...]

  11. #11 Arnim van Lieshout
    on Jul 27th, 2011 at 12:54 pm

    Alasdair,

    I’ve tried to answer your question in a new post.
    Have a look at it.
    http://www.van-lieshout.com/2011/07/esxcli-masking-lun-paths-powercli/

    Regards,
    Arnim

  12. #12 NerdFrenzy
    on Nov 4th, 2011 at 4:17 am

    Excellent blog. Saved my a bucket load of time.

    Keep up the great work!

  13. #13 VNXe 3300 performance follow up (EFD and RR settings) « Henriwithani
    on Nov 21st, 2011 at 1:18 pm

    [...] van Lieshout has a really good post about configuring RR using PowerCLI and I used his examples for configuring the IO operation limit from 1000 to 1. If you are not [...]

  14. #14 Nigel Boulton
    on Dec 7th, 2011 at 10:31 am

    Nice post Arnim, many thanks. Saved a whole load of time-consuming research into how to do this :-)

  15. #15 Arnim van Lieshout
    on Dec 8th, 2011 at 9:48 am

    Thank you Nigel.
    Always great to get feedback when something is useful!

  16. #16 Use PowerShell to change the VMWare Path Selection Policy for a LUN « SweeneyOps
    on Dec 11th, 2011 at 5:17 pm

    [...] got this information from Arnim Van Lieshout’s blog, and it turned out to be incredibly useful.  I found myself in a position where I needed to be [...]

  17. #17 Todd Volz
    on Nov 15th, 2012 at 4:07 pm

    Here is an updated script for esxi5 to change an individual host that has EMC VNX storage disks and set the default for VNX to round robin.

    #use powershell_ise
    #load in powershell: Add-PSSnapin vmware.vimautomation.core
    Connect-VIServer -server

    #Sets esxcli object
    $esxcli=Get-esxcli

    #prints out the current connected host and the available commands
    $esxcli

    #prints out the available EMC volumes and the path selection configuration
    $esxcli.storage.nmp.device.list() | Where {$_.DeviceDisplayName -like “DGC iSCSI*” -or $_.DeviceDisplayName -like “EMC Disk*”} | select -Property Device,PathSelectionPolicy
    read-host

    #Sets the default Path Selection policy to Round Robin
    $esxcli.storage.nmp.satp.set($null, “VMW_PSP_RR”, “VMW_SATP_ALUA_CX”)

    #Prints the default Path Selection Policies
    $esxcli.storage.nmp.satp.list()

    #Changes the path selection policy for existing volumes to Round robin
    $esxcli.storage.nmp.device.list() | Where {$_.DeviceDisplayName -like “DGC iSCSI*” -or $_.DeviceDisplayName -like “EMC Disk*”} | select -Property Device,PathSelectionPolicy | %{
    $esxcli.storage.nmp.device.set($null, $_.Device, “VMW_PSP_RR”)
    }

    #prints out the available EMC volumes and the path selection configuration
    $esxcli.storage.nmp.device.list() | Where {$_.DeviceDisplayName -like “DGC iSCSI*” -or $_.DeviceDisplayName -like “EMC Disk*”} | select -Property Device,PathSelectionPolicy

    #clears out the esxcli object
    Clear-variable esxcli

    #disconnects the server specified
    disconnect-VIServer -server

  18. #18 Back To Basics – Configuring Storage: PowerCLI (Part 6 of 6) « Mike Laverick…
    on Feb 7th, 2014 at 9:38 am

    [...] van Lieshout post ESXCLI the PowerCLI way provide an invaluable starting point for this [...]

Leave a Comment