PowerShell Pipeline

Multithread Your PowerShell Commands Using Runspaces with PoshRSJob

Get more done with this handy module.

• By Boe Prox
• 08/06/2015

If you have used PowerShell for a while now, you probably know that there are a few ways to give PowerShell more of a multithreaded feel by using PowerShell jobs in the form of the *-job cmdlets as well as using Workflows or doing a "fan-out" approach to other systems by using Invoke-Command. These are all great and accomplish the goal of running commands in the background while giving us back the console to do anything else that we need to do. There is another way using runspaces which gives us the ability to run commands in the background, but also provides more options such as throttling using runspacepools and also provides less overhead by staying within the PowerShell process and instead opens up another thread to run. We can also use synchronized variables if the requirement calls for it.

The downside to runspaces are that they are more complex to work with. We have to create the runspace and/or runspacepool, provide the necessary commands and configurations to it and then kick it off while then manually monitoring the runspace for when it completes so we can then pull the data back from the thread.

Enter a module that I wrote called PoshRSjob. This module is built on the concept of using runspaces, but in a more familiar PSJobs type of role to provide multithreading support and is available out on GitHub. If you are running PowerShell V5, you can install it running the following command:

Install-Module -Name PoshRSJob 

You are probably asking me what does this module do that PSJobs is not already doing? To answer that I will demo a few tasks that you would find difficult to use with the PSJobs.

A quick look at the commands will seem pretty familiar to what exists with the *-Job cmdlets.

A major difference is the need not to use ForEach to iterate and add an item to the RSjobs. Much how you would take advantage of some cmdlets that have building iterations of objects that are piped to them, you can do the same thing here.

$Test = 'test'

  $Something = 1..10

  1..50|start-rsjob -Name {$_} -ScriptBlock  {

  [pscustomobject]@{

  Result=($_*2)

  Test=$Using:Test

  Something=$Using:Something

  }

}  -Throttle 5 

This code example shows how you pipe your objects into Start-RSJob and you can specify the Name of the job by using a scriptblock and using $_ to take advantage of the pipelined object to name it. The scriptblock itself where all of the commands will be run has no need for a Param() to declare variables which are being used as the pipelined object is declared as $_.  As with PowerShell V3 jobs, there is support for the $Using: scope variable so that means that anything in the main console can be brought into the runspace job simply by prepending the $Using: scope behind it. Throttling can be set to anything from 1 job to, well, whatever size that you deem necessary. Of course, you want to use something reasonable to avoid issues with CPU and possibly memory/hard drive depending on what it is you are doing with the job.

We can see all of the jobs are being shown (just like what you would see when using PSJobs) so we can see the status of the RSJobs.

Now we can pull back the data from the jobs and see what they have.

Get-RSJob | Receive-RSJob 

If we wanted to kick off a bunch of jobs which could take a while, yet we wanted to track their progress, we can make use of Wait-RSJob (just like Wait-Job) and also provide the optional use of a progress bar to better track the state of all of the jobs being monitored.

$Test = 'test'

  $Something = 1..10

  1..50|start-rsjob -Name {$_} -ScriptBlock  {

  Start-Sleep  -Seconds (Get-Random  $Using:Something)

  [pscustomobject]@{

  Result=($_*2)

  Test=$Using:Test

  Something=$Using:Something

  }

}  -Throttle 5 | Wait-RSJob -ShowProgress 

Eventually when all of the jobs have completed, they will all be outputted to the console.

If you are concerned about possible errors that occurred in the jobs or are using streams such as Verbose, Debug or even Progress, then you can view these streams in the RSJob object itself.

$Test = 'test'

  $Something = 1..10

  1..5|start-rsjob -Name {$_} -ScriptBlock  {

  $DebugPreference  = 'Continue'

  Write-Debug  "Something: $($Using:Something)" -Debug

  Write-Verbose  "Incoming Object: $($_)" -Verbose

  [pscustomobject]@{

  Result=($_*2)

  Test=$Using:Test

  Something=$Using:Something

  }

}  -Throttle 5 | Wait-RSJob -ShowProgress 

Now we have a great way to track the internals of a job if it is still running. Errors are also noted as well and show up in the RSJob object if any occur.

$Test = 'test'

  $Something = 1..10

  1..5|start-rsjob -Name {$_} -ScriptBlock  {

  $Random  = Get-Random  -InputObject $Using:Something

  If ($Random -ge 5) {1/0}

  $DebugPreference  = 'Continue'

  Write-Debug  "Something: $($Using:Something)" -Debug

  Write-Verbose  "Incoming Object: $($_)" -Verbose

  [pscustomobject]@{

  Result=($_*2)

  Test=$Using:Test

  Something=$Using:Something

  }

}  -Throttle 5 | Wait-RSJob -ShowProgress

We have covered a few examples that showcase using PoshRSJob as a new module to provide runspace-based jobs, yet provide the familiarity of PSJobs while providing throttling support. Another great feature, but a little more complex to demonstrate here is using synchronized variables within the RSjobs (you can find another blog article here showing an example). Another thing to note is that this module does support downward to PowerShell V2. At the end of the day, this module can be a great item in your PowerShell toolkit to provide a different means of using multithreading for commands.

This module is available on GitHub which also means that if you want to provide more support to it or fix a bug that you happened to find, you can easily fork the repository and update it and submit a Pull Request to integrate it into the main module.