PowerShell Pipeline
Multithread Your PowerShell Commands Using Runspaces with PoshRSJob
Get more done with this handy module.
If you have used PowerShell for a while now, you probably know that there are a few ways to give PowerShell more of a multithreaded feel by using PowerShell jobs in the form of the *-job cmdlets as well as using Workflows or doing a "fan-out" approach to other systems by using Invoke-Command. These are all great and accomplish the goal of running commands in the background while giving us back the console to do anything else that we need to do. There is another way using runspaces which gives us the ability to run commands in the background, but also provides more options such as throttling using runspacepools and also provides less overhead by staying within the PowerShell process and instead opens up another thread to run. We can also use synchronized variables if the requirement calls for it.
The downside to runspaces are that they are more complex to work with. We have to create the runspace and/or runspacepool, provide the necessary commands and configurations to it and then kick it off while then manually monitoring the runspace for when it completes so we can then pull the data back from the thread.
Enter a module that I wrote called PoshRSjob. This module is built on the concept of using runspaces, but in a more familiar PSJobs type of role to provide multithreading support and is available out on GitHub. If you are running PowerShell V5, you can install it running the following command:
Install-Module -Name PoshRSJob
You are probably asking me what does this module do that PSJobs is not already doing? To answer that I will demo a few tasks that you would find difficult to use with the PSJobs.
A quick look at the commands will seem pretty familiar to what exists with the *-Job cmdlets.
[Click on image for larger view.] Figure 1. List of PoshRSJob commands.
A major difference is the need not to use ForEach to iterate and add an item to the RSjobs. Much how you would take advantage of some cmdlets that have building iterations of objects that are piped to them, you can do the same thing here.
$Test = 'test'
$Something = 1..10
1..50|start-rsjob -Name {$_} -ScriptBlock {
[pscustomobject]@{
Result=($_*2)
Test=$Using:Test
Something=$Using:Something
}
} -Throttle 5
This code example shows how you pipe your objects into Start-RSJob and you can specify the Name of the job by using a scriptblock and using $_ to take advantage of the pipelined object to name it. The scriptblock itself where all of the commands will be run has no need for a Param() to declare variables which are being used as the pipelined object is declared as $_. As with PowerShell V3 jobs, there is support for the $Using: scope variable so that means that anything in the main console can be brought into the runspace job simply by prepending the $Using: scope behind it. Throttling can be set to anything from 1 job to, well, whatever size that you deem necessary. Of course, you want to use something reasonable to avoid issues with CPU and possibly memory/hard drive depending on what it is you are doing with the job.
[Click on image for larger view.] Figure 2. Viewing the RSJobs from Start-RSJob command.
We can see all of the jobs are being shown (just like what you would see when using PSJobs) so we can see the status of the RSJobs.
Now we can pull back the data from the jobs and see what they have.
Get-RSJob | Receive-RSJob
[Click on image for larger view.] Figure 3. Retrieving output from the RSJobs.
If we wanted to kick off a bunch of jobs which could take a while, yet we wanted to track their progress, we can make use of Wait-RSJob (just like Wait-Job) and also provide the optional use of a progress bar to better track the state of all of the jobs being monitored.
$Test = 'test'
$Something = 1..10
1..50|start-rsjob -Name {$_} -ScriptBlock {
Start-Sleep -Seconds (Get-Random $Using:Something)
[pscustomobject]@{
Result=($_*2)
Test=$Using:Test
Something=$Using:Something
}
} -Throttle 5 | Wait-RSJob -ShowProgress
[Click on image for larger view.] Figure 4. Tracking the progress of all jobs.
Eventually when all of the jobs have completed, they will all be outputted to the console.
If you are concerned about possible errors that occurred in the jobs or are using streams such as Verbose, Debug or even Progress, then you can view these streams in the RSJob object itself.
$Test = 'test'
$Something = 1..10
1..5|start-rsjob -Name {$_} -ScriptBlock {
$DebugPreference = 'Continue'
Write-Debug "Something: $($Using:Something)" -Debug
Write-Verbose "Incoming Object: $($_)" -Verbose
[pscustomobject]@{
Result=($_*2)
Test=$Using:Test
Something=$Using:Something
}
} -Throttle 5 | Wait-RSJob -ShowProgress
[Click on image for larger view.] Figure 5. Viewing the various streams in a RSJob object.
Now we have a great way to track the internals of a job if it is still running. Errors are also noted as well and show up in the RSJob object if any occur.
$Test = 'test'
$Something = 1..10
1..5|start-rsjob -Name {$_} -ScriptBlock {
$Random = Get-Random -InputObject $Using:Something
If ($Random -ge 5) {1/0}
$DebugPreference = 'Continue'
Write-Debug "Something: $($Using:Something)" -Debug
Write-Verbose "Incoming Object: $($_)" -Verbose
[pscustomobject]@{
Result=($_*2)
Test=$Using:Test
Something=$Using:Something
}
} -Throttle 5 | Wait-RSJob -ShowProgress
[Click on image for larger view.] Figure 6. Display errors that occur during a background RSJob.
We have covered a few examples that showcase using PoshRSJob as a new module to provide runspace-based jobs, yet provide the familiarity of PSJobs while providing throttling support. Another great feature, but a little more complex to demonstrate here is using synchronized variables within the RSjobs (you can find another blog article here showing an example). Another thing to note is that this module does support downward to PowerShell V2. At the end of the day, this module can be a great item in your PowerShell toolkit to provide a different means of using multithreading for commands.
This module is available on GitHub which also means that if you want to provide more support to it or fix a bug that you happened to find, you can easily fork the repository and update it and submit a Pull Request to integrate it into the main module.
About the Author
Boe Prox is a Microsoft MVP in Windows PowerShell and a Senior Windows System Administrator. He has worked in the IT field since 2003, and he supports a variety of different platforms. He is a contributing author in PowerShell Deep Dives with chapters about WSUS and TCP communication. He is a moderator on the Hey, Scripting Guy! forum, and he has been a judge for the Scripting Games. He has presented talks on the topics of WSUS and PowerShell as well as runspaces to PowerShell user groups. He is an Honorary Scripting Guy, and he has submitted a number of posts as a to Microsoft's Hey, Scripting Guy! He also has a number of open source projects available on Codeplex and GitHub. His personal blog is at http://learn-powershell.net.