PowerShell, as most command line and scripting environments, execute commands sequentially. In other words, instructions run synchronously. This means that PowerShell waits for a command to finish execution before running the next one. While this allows to have more control over each command and its output, it also reduces efficiency because only one action or task is carried out at a time. However, there are several ways in which PowerShell can take advantage of parallel execution and run tasks asynchronously:
- PowerShell background jobs
- PowerShell thread jobs
- Runspaces
- Working with several PowerShell windows simultaneously
- Foreach-Object -Parallel (PowerShell 7)
We re going to discuss the first item from this list, Powershell Background Jobs (and briefly the last one as well). In a nutshell, each PowerShell job starts a new PowerShell process which executes the specified set of commands. For instance: today we will work on VM deployments, each new VM will be deployed by an individual PowerShell process. I will not go into details right now but we will go over most aspects relating to PS Jobs. For more in-depth information you can read this.
ForEach-Object -Parallel
Starting with v7 PowerShell supports ForEach-Object with the Parallel parameter. As of the writing of this post, the latest PowerCLI version is 12.2 and is the first one to officially support the Parallel switch parameter of the ForEach-Object cmdlet.
PowerCLI offers two different approaches to parallel execution with ForEach-Object -Parallel:
- Creating a vCenter connection first, storing the connection in a variable and then passing this variable to the cmdlets within the script block.
- Save the PowerCLI context in a variable and use it within the the script block.
You can read more about PowerCLI support for ForEach -Parallel here.
While, this is a great step towards multithreading (ForEach-Object -Parallel uses multiple threads instead of multiple processes), during my tests, when deploying multiple VMs at least one fails. Not to mention that it takes longer compared to how long it takes to do the same with background jobs. Therefore, I still consider background jobs my preferred approach to multitasking with PowerShell.
Parallel Execution of VMs Deployment with Powershell Jobs
One of the first things that we need to take into account is that a job will create a new scope. This is extremely important when creating and using variables, which will be our first step to deploy new VMs from a template using background jobs.
$NumVMs = 5 #Amount of VMs to be deployed
$VMNamePrefix = 'MyVM' #This will be the first part of the VM name, followed by a number from $Range
$Password = '123456'
$Username = 'domain\fcorrales'
$vCenterAddress = 'MyvCenter.domain.org'
$Template = 'MyTemplate'
$vHost = 'MyvHost'
$Datastore = 'MyDS'
$PortGroup = 'MyvDPG'
$Folder = 'MyFolder' #Folder name where the VMs will be deployed. Must be unique.
$WaitInterval = 30 #This will determine the amount of seconds to wait before checking the jobs status
$VerbosePreference = 'Continue' #Set to SilentlyContinue to hide status messages
#Range defines VMs start and end numbers. I.e. amount of VMs.
$Range = 1..$NumVMs
All these variables names are fairly intuitive and should give you a hint on what each one does. In addition, comments should provide you with clarification where needed. Anyway, it will be easier to understand their purpose as we start using them in subsequent parts of the code. We can replace all these variables with a Read-Host input request as well. For this example, I am going to hard-code them instead.
Foreach Loop, Script Block and Using Scope Modifier
The next step is the actual VM deployment. With a foreach loop, each number within the $Range variable is processed.
First, we add a $ScriptBlock variable which will store the code that each job will execute. Then, inside the script block, we change PowerCLI configuration to prevent warnings and prompts, remember we are dealing with background jobs, which means they don't display their output onscreen, which is why we need to make sure no interaction is required from the user.
Next, we create the connection to the vCenter. Note here we start adding the Using: scope modifier. When added to a variable, Using enables access to this variable from a different scope. Here we are setting the variables outside the script block, but the new PowerShell process (job) will be able to use them thanks to the aforementioned scope modifier. You can read more about scopes here.
Then, we get the port group and folder for the new VMs using the Get-Folder and Get-VDPortGroup cmdlets respectively.
Finally, we deploy the new VMs with the New-VM cmdlet. All new VMs will be deployed from the template defined in the $Template variable. The VMs' names will be 'MyVM' followed by its corresponding number. E.g. 'VyVM2'. Again, we add Using to all variables that belong to the main scope.
To properly end the session, we also disconnect from the vCenter at the end.
foreach ($Num in $Range) {
#The script block that is sent to the vCenter by each job
$ScriptBlock = {
#Supress most PowerCLI warnings and prompts
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -DisplayDeprecationWarnings: $false -ParticipateInCeip: $false -Confirm: $false -Scope Session
#Connect to vCenter
Connect-VIServer -Server $Using:vCenterAddress -User $Using:Username -Password $Using:Password
#Set VMs folder and port group
$Location = Get-Folder -Name $Using:Folder
$PortGroup = Get-VDPortgroup -Name $Using:PortGroup
#Deploy the VMs from a template
New-VM -Name "$Using:VMNamePrefix$Using:Num" -VMHost $Using:vHost -Datastore $Using:Datastore -Template $Using:Template -Portgroup $PortGroup -Location $Location -RunAsync
#Disconnect from vCenter
Disconnect-VIServer -Server $Using:vCenterAddress -Confirm: $false
}
}
Starting the Background Jobs
IMPORTANT: Running too many jobs at the same time might impact your system's performance. Jobs run locally and each one requires dedicated processing power. Make sure to know what is the limit you should not exceed when running background jobs.
Now that we have our script block defined, we will start the background jobs. Still inside the foreach loop and immediately after the Script Block, we need to add this line.
Start-Job -ScriptBlock $ScriptBlock -Name "Job$Num"
Start-Job will trigger the new PowerShell process which will execute the script block we define in the ScriptBlock argument. The Name parameter sets the Job Name to 'Job' and its corresponding number. E.g. 'Job2'.
At this point if you open Task Manager or run the Get-Process cmdlet from a different Powershell window, you will notice that new PowerShell processes will be listed. These processes are executing the background jobs.
Reviewing Job Status and Verifying Jobs Output
Once the jobs are running there are a couple of cmdlets we can use to check their status. The first one is Get-Job which shows the Job status along with its name and id. Then, we can use Receive-Job followed by the job id, which shows the output of the script block execution. When running Receive-Job, it is important to keep in mind that once we get the output it will be deleted and it won't be possible to review it again. To avoid this use the -Keep parameter.
If something goes wrong, you can run Stop-Job <job id> to terminate any background job.
Receiving the Output of the VM Deployment Jobs
We will now add to our script a block of code to check the output of our jobs. I use a Do/Until construct to achieve this. It is similar to Do/While which is more commonly used but instead the script block runs until the condition is true, not as long as the condition is true. This is just a personal preference, feel free to edit it as desired.
With Do/While we wait the amount of seconds from the variable $WaitInterval and then check if all jobs have a 'Completed' status. If so it retrieves the output of all the jobs. Otherwise, it waits the same amount of seconds again and so on.
Do {
$Jobs = Get-Job | Where-Object State -EQ 'Completed'
If ($Jobs.Count -eq $Range.Length) {
foreach ($Job in $Jobs) {
Receive-Job -Id $Job.Id -Keep
}
}
#Wait the amount of seconds defined in the value of $WaitInterval variable
Start-Sleep -Seconds $WaitInterval
Write-Verbose -Message "[$(Get-Date -Format 'MM/dd/yyyy - HH:mm')] Waiting $WaitInterval seconds for jobs to complete. $($Jobs.Count)/$($Range.Length) completed"
}
Until (
$Jobs.Count -eq $Range.Length
)
Verbose Output and Status Messages
When we set the variables at the beginning, there was a $VerbosePreference variable. This is a built-in variable that determines if PowerShell shows Verbose messages or not. By default, the $VerbosePreference value is 'SilentlyContinue'. By setting it to 'Continue' we are telling PowerShell to show messages from the Verbose stream. More about Verbose here.
So, if we set the $VerbosePreference variable to 'Continue' we will see something like this while waiting for the Jobs to finish execution.
In order to restore this variable to its default value, we run the following code at the end of the script (outside the foreach loop).
If ($VerbosePreference -ne 'SilentlyContinue') {
$VerbosePreference = 'SilentlyContinue'
}
Full Script Code
And here is the whole script code. I have included some comments as usual to help with readability.
############################# User-defined variables. Can be replaced by Read-Host #############################
$NumVMs = 5 #Amount of VMs to be deployed
$VMNamePrefix = 'MyVM' #This will be the first part of the VM name, followed by a number from $Range
$Password = '123456'
$Username = 'domain\fcorrales'
$vCenterAddress = 'MyvCenter.domain.org'
$Template = 'MyTemplate'
$vHost = 'MyvHost'
$Datastore = 'MyDS'
$PortGroup = 'MyvDPG'
$Folder = 'MyFolder'
$WaitInterval = 30 #This will determine the amount of seconds to wait before checking the jobs status
$VerbosePreference = 'Continue' #Set to SilentlyContinue to hide status messages
################################################################################################################
#Range defines VMs start and end numbers. I.e. amount of VMs.
$Range = 1..$NumVMs
foreach ($Num in $Range) {
#The script block that is sent to the vCenter by each job
$ScriptBlock = {
#Supress most PowerCLI warnings and prompts
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -DisplayDeprecationWarnings: $false -ParticipateInCeip: $false -Confirm: $false -Scope Session
#Connect to vCenter
Connect-VIServer -Server $Using:vCenterAddress -User $Using:Username -Password $Using:Password
#Set VMs folder and port group
$Location = Get-Folder -Name $Using:Folder
$PortGroup = Get-VDPortgroup -Name $Using:PortGroup
#Deploy the VMs from a template
New-VM -Name "$Using:VMNamePrefix$Using:Num" -VMHost $Using:vHost -Datastore $Using:Datastore -Template $Using:Template -Portgroup $PortGroup -Location $Location -RunAsync
#Disconnect from vCenter
Disconnect-VIServer -Server $Using:vCenterAddress -Confirm: $false
}
#Start the jobs, one for each VM
Start-Job -ScriptBlock $ScriptBlock -Name "Job$Num"
}
#Do / Until block to wait until all jobs are completed to show the output.
Do {
$Jobs = Get-Job | Where-Object State -EQ 'Completed'
If ($Jobs.Count -eq $Range.Length) {
foreach ($Job in $Jobs) {
Receive-Job -Id $Job.Id -Keep
}
}
#Wait the amount of seconds defined in the value of $WaitInterval variable
Start-Sleep -Seconds $WaitInterval
Write-Verbose -Message "[$(Get-Date -Format 'MM/dd/yyyy - HH:mm')] Waiting 30 seconds for jobs to complete. $($Jobs.Count)/$($Range.Length) completed"
}
Until (
$Jobs.Count -eq $Range.Length
)
#Return $VerbosePreference to its default value
If ($VerbosePreference -ne 'SilentlyContinue') {
$VerbosePreference = 'SilentlyContinue'
}
Feel free to leave comments and questions below.
The source code for this post is available in my GitHub repository.