Make Your PowerShell For Loops 4x Faster

by Doug Finke on January 16, 2011

in PowerShell,PowerShell Pearls

Looping through information is a fundamental task. It is done many times a day by many people in many ways on computers. In PowerShell, one way to loop is using a For loop (see Slow Loop).

Hidden in this simple example is a performance hit where looping over 100,000 items has a 2 second response time rather than subsecond. That’s the difference of hitting enter and getting a prompt compared to counting 1 one-thousand, 2 one-thousand.

Slow Loop

$range = 1..100000            
            
For($i=0; $i -lt $range.Count; $i++) {            
    $i            
}

A Single Line Change

If you needed to do the same loop 5 times in your script, you could speed it up so the entire script runs faster than executing the first loop.

You do this by storing (caching) the array count in a variable and using it in the for loop.

4x Faster

$range = 1..100000            
            
$count = $range.Count            
For($i=0; $i -lt $count; $i++) {            
    $i            
}

Good Habits

Making this a habit is good practice. It eliminates waste and enables your scripts be faster when lots of information is thrown at it when you least expect it.

More Results

Test                Range TotalSeconds
----                ----- ------------
Test-WithoutCache      10     0.000339
Test-WithCache         10     0.000201
Test-WithoutCache     100     0.001526
Test-WithCache        100     0.000478
Test-WithoutCache    1000     0.017743
Test-WithCache       1000     0.003395
Test-WithoutCache   10000     0.139237
Test-WithCache      10000     0.033718
Test-WithoutCache  100000     1.410456
Test-WithCache     100000     0.358698
Test-WithoutCache 1000000    14.398003
Test-WithCache    1000000     3.835144

Test Harness

Run the tests and read the script. Notice how the functions are stored in an array using the custom function ql. Extending this harness to handle more tests or different ranges is very simple.

Each test is executed, timed, formatted and cast when the new PSObject property TotalSeconds is created.

TotalSeconds = [Double]("{0:#0.#####0}" -f            
    (Measure-Command { & $test (1..$range) }).TotalSeconds)

 

Function Test-WithoutCache ($data) {

$sum = 0 # antipattern # access the array count on each iteration for($idx = 0 ; $idx -lt $data.count; $idx++) { $sum+=$data[$idx] } $sum } Function Test-WithCache ($data) { # capture the count; cache it $count = $data.count $sum = 0 # use the $count variable in the # for loop and improve performance 4x for($idx = 0 ; $idx -lt $count; $idx++) { $sum+=$data[$idx] } $sum } function ql {$args} $tests = ql Test-WithoutCache Test-WithCache $ranges = ql 10 100 1000 10000 100000 1000000 $(ForEach($range in $ranges) { ForEach($test in $tests) { $msg = ("[{0}] Running {1} with {2} items" -f (Get-Date), $test, $range) Write-Host -ForegroundColor Green $msg New-Object PSObject -Property @{ TotalSeconds = [Double]("{0:#0.#####0}" -f (Measure-Command { & $test (1..$range) }).TotalSeconds) Range = $range Test = $test } | Select Test, Range, TotalSeconds } }) | Format-Table -AutoSize

{ 6 trackbacks }

Tweets that mention Make Your PowerShell For Loops 4x Faster -- Topsy.com
01.16.11 at 12:16 pm
Tweets that mention Make Your PowerShell For Loops 4x Faster -- Topsy.com
01.16.11 at 5:38 pm
Tweets that mention Make Your PowerShell For Loops 4x Faster -- Topsy.com
01.20.11 at 12:27 pm
My Week in Geek – January 21, 2011 - My Geek Finds
01.21.11 at 6:08 am
Hey, Scripting Guy! Blog
05.18.14 at 2:05 am
???PowerShell For?????? | PowerShell ????
05.23.14 at 8:21 am

{ 15 comments… read them below or add one }

Oisin G. 01.16.11 at 12:12 pm

Urgh… I am surprised powershell does not hoist the test outside of the loop; I always assumed it did. This is a common optimisation in compilers but I guess not so much for interpreters. Nice tip.

Doug Finke 01.16.11 at 12:23 pm

@Oisin, Thanks. Yeah, I’ve been going along on the same assumption. I read some performance tips about JavaScript and they make the same recommendation. So I decided to test it in PowerShell. Sure enough, it is needed.

Bartek Bielawski 01.16.11 at 5:32 pm

It’s really nice to know. :) But what about:
Measure-Command { foreach ($i in 1..100000) { $i }}
On my machine it’s even quicker. ;)

wullxz 01.16.11 at 6:37 pm

The loop is faster if you take the ForEach-loop instead.
I added a new function “Test-ForEach”:

Function Test-ForEach ($data) {
foreach ($idx in $data) {
$sum +=$data[$idx]
}
$sum
}

… and the function-name at the end of the following line:

$tests = ql Test-WithoutCache Test-WithCache Test-ForEach

… and got this results:


Test Range TotalSeconds
---- ----- ------------
Test-WithoutCache 10 0,000359
Test-WithCache 10 0,000294
Test-ForEach 10 0,000375
Test-WithoutCache 100 0,001304
Test-WithCache 100 0,000546
Test-ForEach 100 0,000494
Test-WithoutCache 1000 0,013240
Test-WithCache 1000 0,004862
Test-ForEach 1000 0,002350
Test-WithoutCache 10000 0,123377
Test-WithCache 10000 0,037268
Test-ForEach 10000 0,018932
Test-WithoutCache 100000 1,274186
Test-WithCache 100000 0,452459
Test-ForEach 100000 0,228077
Test-WithoutCache 1000000 13,071414
Test-WithCache 1000000 4,374640
Test-ForEach 1000000 2,530647

Doug Finke 01.16.11 at 8:26 pm

@wullxz and @Bartek you are correct. Here is my previous post PowerShell – Four For Loops and their timings

Tome Tanasovski 01.19.11 at 8:48 pm

I can’t believe this isn’t optimized! Great find. Thanks.

Grant Steinfeld 02.04.11 at 11:06 am

Great pattern, thank for the tip. Also your example is elegantly written in a very Functional way.

Ludovic Emo 02.23.11 at 7:45 am

this is a good thing that the condition if executed each time, it allows to use variables which are changing inside the loop block to handle complex ending conditions

and when you know that the condition will never be impacted by the block content, the tip of this article helps to save perf

Pete 04.27.11 at 2:56 pm

I feel your title is misleading, specifically the ratio “4x”. I agree this is a good habit, but only if you can make the assumption your array will not be modified as Ludovic mentioned. Your performance gain will be proportional to the time it takes to evaluate “$i -lt $data.count” vs the amount of work done inside your loop. For large amounts of work (something a little more complex than simply summing integers) this gain will be much less significant.

Doug Finke 04.29.11 at 4:35 pm

Thanks for the comment Pete.

I encourage good PowerShell habits. In the end, there are performance gains.
These are same issues encountered. for example, in JavaScript and back in the Visual Basic days.

In VB, developers would unknowingly access information in a loop like:

' Anti pattern
For index As Integer = 1 to Form1.txtBoxCount.Text
Next

' Better
Dim count As Integer
count = Form1.txtBoxCount.Text
For index As Integer = 1 to count
Next

Again, taking the time, forming good habits is the goal and it has excellent pay back.

Tom Chantler 12.02.11 at 10:52 am

This shouldn’t be necessary, but given that it is then thanks for taking the time to discover it and also for writing it up!

Joe Leibowitz 01.21.12 at 10:09 am

Doug,

Thanks for the excellent piece. I was puzzled how the ‘ql’ function works.

Instead, I just built a couple of arrays, as below, and it worked just fine. Is there any special reason to use the ‘ql’? And, again, why does ‘ql’ work at all?

$test_names = ‘FirstTest’,’SecondTest’
$test_data_sizes = 1000,2000,3000

Thanks, J_L

Doug Finke 01.21.12 at 12:30 pm

Thanks for the comment Joe.

function ql {$args}

Lets you streamline array building. ql stands for quote list. This is a Perl-ism.
You don’t need commas or quotes.
Saves me 5 keystrokes when defining $testNames. ;-)

$testNames = ql FirstTest SecondTest
$testDataSizes = ql 1000 2000 3000

Doug

G. Richard Bellamy 07.19.12 at 7:27 pm


function Test-WhileWithCache ($data) {
# capture the count, cache it
$count = $data.count
$sum = 0
$idx = 0
# use the $count variable in the
# while loop and improve performance
while($count -lt $idx) {
$sum+=$data[$idx]
$idx++
}
$sum
}

Mike 11.11.14 at 3:25 pm

Thanks Doug, good tip, I just sped up my powershell adventure game’s path-finding algorythm’s duration into about 1/3 – 1/4 of what it was previously.

Leave a Comment

You can use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

You are forbidden!