Story #257

Performance testing and cost estimation for ~50 runners

Estimated Hours: 12 (12 added, 4 remaining) 100 %

Tasks: 2 (1 remaining)

Tasks

Id Description Hours Hours Added Assigned To Completed Actions  
Estimated Added Remaining
#258

Remove eclipse-specific tooling and get all ATs passing in IntelliJ.

4 0 4 Nobody
0 %

#259

Run JMeter ATs at 50 runners

8 12 0

bodriscoll picture bodriscoll 2 hours

bodriscoll picture bodriscoll 2 hours

bodriscoll picture bodriscoll 4 hours

bodriscoll picture bodriscoll 3 hours

bodriscoll picture bodriscoll 1 hour

Nobody
150 %

bodriscoll picture

bodriscoll on Sat 10 Jan, 2015

Tested the following with performance.jmx against aperunner-test
50 runners
300 watchers
10 minutes runtime
60 second ramp
~600 uncollected donations
100 live donations during the run (actual 97)
100 donation collections (actual 79)
10 second donor interval
8 second runner interval
0 errors server or client side

This cost about $0.50, so an hour at this rate would cost about $3.00. Some of the latencies are a little poor, but these were probably the unrealistic load during ramp up before a lot of instances had spun up. The main donor and runner requests are okayish at 3.4s and 4.9s 90%.

sampler_label,aggregate_report_count,average,aggregate_report_median,aggregate_report_90%_line,aggregate_report_min,aggregate_report_max,aggregate_report_error%,aggregate_report_rate,aggregate_report_bandwidth
Set variables,350,4,5,6,4,30,0.0,5.832555659245434,0.23353006057525663
Log in donor,300,12522,12861,21101,228,32604,0.0,3.9332398096311936,85.57156273763324
Create runner,50,13593,13050,23041,204,28143,0.0,0.6905886577718847,0.17332156742907653
Get pin,50,13015,13184,19327,694,22231,0.0,0.5912122215390436,6.606681104561793
Get map t3,15697,1804,1021,3436,161,36529,0.0,23.019909427941247,141.58520883607352
Log in runner,50,9665,9480,14442,365,21964,0.0,0.5593904881241399,3.4376184334549076
Add runner,50,8182,6105,13995,2997,27224,0.0,0.5358999367638074,0.16585684371014245
Update runner location,2936,2375,1706,4970,206,29767,0.0,4.235784298166898,25.277404997370667
Create donation,97,840,570,1660,188,7289,0.0,0.1684878992685541,0.04014750724758515
Collect donation,79,1997,1730,3683,295,7500,0.0,0.1344216381403532,0.6110272349511573
TOTAL,19659,2112,1111,4574,4,36529,0.0,28.076785201667555,174.07227434197935

 
bodriscoll picture

bodriscoll on Sat 10 Jan, 2015

I’m a little wary of this cost estimate – the appengine console only shows datastore read operations as billable. I suspect the others are still under the free quotas. Still, I think the above estimate is the right ball park.

 

You do not have sufficient permissions to comment

Revision Author Commited Message
273 bodriscoll picture bodriscoll Tue 06 Jan, 2015 21:02:34 +0000

#257, #259
Added unique runners.

274 bodriscoll picture bodriscoll Wed 07 Jan, 2015 23:49:35 +0000

#259 #257
Proper test with specified number of collections and creations during the run. Blows up my modem at 50 runners and 300 donors.

275 bodriscoll picture bodriscoll Fri 09 Jan, 2015 23:45:19 +0000

#259 #257
Collect a new random donation every time by storing all donations at the start. Always set the ifRange after the first request.

3 years ago
bodriscoll picture
bodriscoll created Comment #155430 on Story #257
bodriscoll picture
bodriscoll updated Task #259

Completed: NoYes

bodriscoll picture
bodriscoll created 1.0 hours in Task #259
bodriscoll picture
bodriscoll created Comment #155429 on Story #257
bodriscoll picture
bodriscoll created 3.0 hours in Task #259
bodriscoll picture
bodriscoll created 4.0 hours in Task #259
onesoma picture
onesoma updated Story #257
bodriscoll picture
bodriscoll created 2.0 hours in Task #259
bodriscoll picture
bodriscoll created 2.0 hours in Task #259
bodriscoll picture
bodriscoll created Task #259

History View full history