criterion performance measurements
overview
want to understand this report?
ternary-01 | |
ternary-02 | |
ternary-03 | |
ternary-04 | |
4-ary-05 | |
4-ary-06 | |
4-ary-07 | |
4-ary-08 |
ternary-01/naive
21.5 ms 22.0 22.5 23.0 23.5 24.0 24.5 25.0
mean |
21.5 ms 22.0 22.5 23.0 23.5 24.0 24.5 25.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 21.57 ms | 21.61 ms | 21.75 ms |
Standard deviation | 120.6 μs | 346.5 μs | 774.8 μs |
Outlying measurements have slight (8.5%) effect on estimated standard deviation.
ternary-01/lefteigen
295 ms 300 305 310 315 320 325
mean |
295 ms 300 305 310 315 320 325
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 295.3 ms | 295.9 ms | 297.2 ms |
Standard deviation | 2.742 ms | 4.538 ms | 7.698 ms |
Outlying measurements have slight (8.5%) effect on estimated standard deviation.
ternary-01/companion
104 ms 106 108 110 112
mean |
104 ms 106 108 110 112
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 105.1 ms | 105.3 ms | 105.7 ms |
Standard deviation | 747.6 μs | 1.239 ms | 1.967 ms |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
ternary-01/power
205 ms 210 215 220 225 230 235
mean |
205 ms 210 215 220 225 230 235
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 207.5 ms | 208.0 ms | 209.1 ms |
Standard deviation | 1.452 ms | 3.442 ms | 6.082 ms |
Outlying measurements have slight (9.4%) effect on estimated standard deviation.
ternary-02/naive
6.6 ms 6.7 6.8 6.9 7.0 7.1 7.2
mean |
6.6 ms 6.7 6.8 6.9 7.0 7.1 7.2
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 6.679 ms | 6.688 ms | 6.708 ms |
Standard deviation | 39.15 μs | 66.95 μs | 128.4 μs |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
ternary-02/lefteigen
57.0 ms 58.0 59.0 60.0 61.0 62.0
mean |
57.0 ms 58.0 59.0 60.0 61.0 62.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 57.49 ms | 57.57 ms | 57.73 ms |
Standard deviation | 278.6 μs | 528.7 μs | 1.009 ms |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
ternary-02/companion
20 ms 25 30 35
mean |
20 ms 25 30 35
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 19.98 ms | 20.18 ms | 20.95 ms |
Standard deviation | 369.1 μs | 1.734 ms | 4.024 ms |
Outlying measurements have severe (73.8%) effect on estimated standard deviation.
ternary-02/power
38.5 ms 39.0 39.5 40.0 40.5 41.0 41.5 42.0 42.5
mean |
38.5 ms 39.0 39.5 40.0 40.5 41.0 41.5 42.0 42.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 38.96 ms | 39.01 ms | 39.16 ms |
Standard deviation | 196.0 μs | 423.7 μs | 874.0 μs |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
ternary-03/naive
8.10 ms 8.20 8.30 8.40 8.50 8.60
mean |
8.10 ms 8.20 8.30 8.40 8.50 8.60
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 8.147 ms | 8.161 ms | 8.182 ms |
Standard deviation | 65.87 μs | 87.66 μs | 119.1 μs |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
ternary-03/lefteigen
80 ms 85 90 95 100
mean |
80 ms 85 90 95 100
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 81.97 ms | 82.23 ms | 83.02 ms |
Standard deviation | 645.9 μs | 2.081 ms | 4.483 ms |
Outlying measurements have moderate (19.0%) effect on estimated standard deviation.
ternary-03/companion
30.0 ms 32.5 35.0 37.5 40.0 42.5
mean |
30.0 ms 32.5 35.0 37.5 40.0 42.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 29.26 ms | 29.47 ms | 30.02 ms |
Standard deviation | 713.6 μs | 1.600 ms | 3.246 ms |
Outlying measurements have severe (52.5%) effect on estimated standard deviation.
ternary-03/power
54 ms 55 56 57 58 59 60
mean |
54 ms 55 56 57 58 59 60
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 54.68 ms | 54.78 ms | 54.99 ms |
Standard deviation | 408.7 μs | 721.4 μs | 1.285 ms |
Outlying measurements have slight (6.6%) effect on estimated standard deviation.
ternary-04/naive
13.9 ms 14.0 14.1 14.2 14.3 14.4 14.5 14.6
mean |
13.9 ms 14.0 14.1 14.2 14.3 14.4 14.5 14.6
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 14.02 ms | 14.04 ms | 14.06 ms |
Standard deviation | 84.56 μs | 112.7 μs | 157.9 μs |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
ternary-04/lefteigen
152.5 ms 155.0 157.5 160.0 162.5 165.0 167.5
mean |
152.5 ms 155.0 157.5 160.0 162.5 165.0 167.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 153.0 ms | 153.2 ms | 153.8 ms |
Standard deviation | 829.8 μs | 1.755 ms | 3.599 ms |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
ternary-04/companion
50.0 ms 52.5 55.0 57.5 60.0 62.5 65.0
mean |
50.0 ms 52.5 55.0 57.5 60.0 62.5 65.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 51.20 ms | 51.45 ms | 52.04 ms |
Standard deviation | 777.9 μs | 1.869 ms | 3.309 ms |
Outlying measurements have moderate (32.6%) effect on estimated standard deviation.
ternary-04/power
87.5 ms 90.0 92.5 95.0 97.5 100.0 102.5
mean |
87.5 ms 90.0 92.5 95.0 97.5 100.0 102.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 89.13 ms | 89.39 ms | 89.98 ms |
Standard deviation | 916.6 μs | 1.916 ms | 3.423 ms |
Outlying measurements have moderate (14.2%) effect on estimated standard deviation.
4-ary-05/naive
650 μs 700 750 800 850 900 950
mean |
650 μs 700 750 800 850 900 950
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 685.5 μs | 691.7 μs | 703.5 μs |
Standard deviation | 26.69 μs | 42.60 μs | 67.45 μs |
Outlying measurements have severe (58.5%) effect on estimated standard deviation.
4-ary-05/lefteigen
11.0 ms 11.5 12.0 12.5
mean |
11.0 ms 11.5 12.0 12.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 10.99 ms | 11.03 ms | 11.10 ms |
Standard deviation | 154.4 μs | 253.9 μs | 395.5 μs |
Outlying measurements have moderate (16.1%) effect on estimated standard deviation.
4-ary-05/companion
2.5 ms 5.0 7.5 10.0 12.5 15.0
mean |
2.5 ms 5.0 7.5 10.0 12.5 15.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 2.406 ms | 2.534 ms | 3.114 ms |
Standard deviation | 134.8 μs | 1.199 ms | 2.844 ms |
Outlying measurements have severe (98.9%) effect on estimated standard deviation.
4-ary-05/power
6.30 ms 6.40 6.50 6.60 6.70
mean |
6.30 ms 6.40 6.50 6.60 6.70
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 6.335 ms | 6.347 ms | 6.363 ms |
Standard deviation | 55.71 μs | 70.43 μs | 96.59 μs |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
4-ary-06/naive
4.20 ms 4.30 4.40 4.50 4.60 4.70
mean |
4.20 ms 4.30 4.40 4.50 4.60 4.70
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 4.318 ms | 4.336 ms | 4.359 ms |
Standard deviation | 85.07 μs | 103.3 μs | 124.1 μs |
Outlying measurements have moderate (17.1%) effect on estimated standard deviation.
4-ary-06/lefteigen
44 ms 46 48 50 52
mean |
44 ms 46 48 50 52
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 43.61 ms | 43.71 ms | 44.08 ms |
Standard deviation | 246.7 μs | 901.1 μs | 2.063 ms |
Outlying measurements have moderate (13.3%) effect on estimated standard deviation.
4-ary-06/companion
15.0 ms 16.0 17.0 18.0 19.0 20.0
mean |
15.0 ms 16.0 17.0 18.0 19.0 20.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 15.27 ms | 15.35 ms | 15.55 ms |
Standard deviation | 84.51 μs | 574.2 μs | 1.051 ms |
Outlying measurements have moderate (33.6%) effect on estimated standard deviation.
4-ary-06/power
24 ms 25 26 27 28 29 30 31
mean |
24 ms 25 26 27 28 29 30 31
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 24.51 ms | 24.58 ms | 24.84 ms |
Standard deviation | 165.5 μs | 623.9 μs | 1.420 ms |
Outlying measurements have moderate (19.0%) effect on estimated standard deviation.
4-ary-07/naive
2.12 ms 2.14 2.16 2.18 2.20
mean |
2.12 ms 2.14 2.16 2.18 2.20
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 2.133 ms | 2.136 ms | 2.140 ms |
Standard deviation | 16.35 μs | 19.39 μs | 23.08 μs |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
4-ary-07/lefteigen
22.5 ms 25.0 27.5 30.0 32.5 35.0
mean |
22.5 ms 25.0 27.5 30.0 32.5 35.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 22.10 ms | 22.28 ms | 22.83 ms |
Standard deviation | 577.1 μs | 1.469 ms | 3.202 ms |
Outlying measurements have severe (61.6%) effect on estimated standard deviation.
4-ary-07/companion
6.5 ms 7.0 7.5 8.0 8.5 9.0 9.5 10.0
mean |
6.5 ms 7.0 7.5 8.0 8.5 9.0 9.5 10.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 6.922 ms | 6.978 ms | 7.094 ms |
Standard deviation | 236.8 μs | 399.6 μs | 708.1 μs |
Outlying measurements have severe (55.5%) effect on estimated standard deviation.
4-ary-07/power
15.0 ms 15.5 16.0 16.5 17.0 17.5 18.0
mean |
15.0 ms 15.5 16.0 16.5 17.0 17.5 18.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 15.37 ms | 15.42 ms | 15.53 ms |
Standard deviation | 193.4 μs | 384.9 μs | 643.9 μs |
Outlying measurements have moderate (19.0%) effect on estimated standard deviation.
4-ary-08/naive
26.5 ms 27.0 27.5 28.0 28.5
mean |
26.5 ms 27.0 27.5 28.0 28.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 27.00 ms | 27.06 ms | 27.14 ms |
Standard deviation | 256.9 μs | 340.7 μs | 452.4 μs |
Outlying measurements have slight (2.4%) effect on estimated standard deviation.
4-ary-08/lefteigen
2.04 s 2.05 2.06 2.07 2.08 2.09 2.10
mean |
2.04 s 2.05 2.06 2.07 2.08 2.09 2.10
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 2.055 s | 2.057 s | 2.060 s |
Standard deviation | 11.38 ms | 13.37 ms | 15.57 ms |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
4-ary-08/companion
980 ms 990 1000 1010 1020 1030 1040
mean |
980 ms 990 1000 1010 1020 1030 1040
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 990.4 ms | 992.0 ms | 994.3 ms |
Standard deviation | 6.840 ms | 9.574 ms | 14.03 ms |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
4-ary-08/power
1.00 s 1.01 1.02 1.03 1.04 1.05 1.06
mean |
1.00 s 1.01 1.02 1.03 1.04 1.05 1.06
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 1.013 s | 1.015 s | 1.017 s |
Standard deviation | 9.176 ms | 10.94 ms | 13.43 ms |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. In each section, we display two charts, each with an x axis that represents measured execution time. These charts are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. Measurements are displayed on the y axis in the order in which they occurred.
Under the charts is a small table displaying the mean and standard deviation of the measurements. We use a statistical technique called the bootstrap to provide confidence intervals on our estimates of these values. The bootstrap-derived upper and lower bounds on the mean and standard deviation let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.