criterion performance measurements

overview

want to understand this report?

ternary-01/naive

21.5 ms
22.0
22.5
23.0
23.5
24.0
24.5
25.0
ternary-01/naive time densities
mean
21.5 ms
22.0
22.5
23.0
23.5
24.0
24.5
25.0
ternary-01/naive times
lower bound estimate upper bound
Mean execution time 21.57 ms 21.61 ms 21.75 ms
Standard deviation 120.6 μs 346.5 μs 774.8 μs

Outlying measurements have slight (8.5%) effect on estimated standard deviation.

ternary-01/lefteigen

295 ms
300
305
310
315
320
325
ternary-01/lefteigen time densities
mean
295 ms
300
305
310
315
320
325
ternary-01/lefteigen times
lower bound estimate upper bound
Mean execution time 295.3 ms 295.9 ms 297.2 ms
Standard deviation 2.742 ms 4.538 ms 7.698 ms

Outlying measurements have slight (8.5%) effect on estimated standard deviation.

ternary-01/companion

104 ms
106
108
110
112
ternary-01/companion time densities
mean
104 ms
106
108
110
112
ternary-01/companion times
lower bound estimate upper bound
Mean execution time 105.1 ms 105.3 ms 105.7 ms
Standard deviation 747.6 μs 1.239 ms 1.967 ms

Outlying measurements have no (1.0%) effect on estimated standard deviation.

ternary-01/power

205 ms
210
215
220
225
230
235
ternary-01/power time densities
mean
205 ms
210
215
220
225
230
235
ternary-01/power times
lower bound estimate upper bound
Mean execution time 207.5 ms 208.0 ms 209.1 ms
Standard deviation 1.452 ms 3.442 ms 6.082 ms

Outlying measurements have slight (9.4%) effect on estimated standard deviation.

ternary-02/naive

6.6 ms
6.7
6.8
6.9
7.0
7.1
7.2
ternary-02/naive time densities
mean
6.6 ms
6.7
6.8
6.9
7.0
7.1
7.2
ternary-02/naive times
lower bound estimate upper bound
Mean execution time 6.679 ms 6.688 ms 6.708 ms
Standard deviation 39.15 μs 66.95 μs 128.4 μs

Outlying measurements have no (1.0%) effect on estimated standard deviation.

ternary-02/lefteigen

57.0 ms
58.0
59.0
60.0
61.0
62.0
ternary-02/lefteigen time densities
mean
57.0 ms
58.0
59.0
60.0
61.0
62.0
ternary-02/lefteigen times
lower bound estimate upper bound
Mean execution time 57.49 ms 57.57 ms 57.73 ms
Standard deviation 278.6 μs 528.7 μs 1.009 ms

Outlying measurements have no (1.0%) effect on estimated standard deviation.

ternary-02/companion

20 ms
25
30
35
ternary-02/companion time densities
mean
20 ms
25
30
35
ternary-02/companion times
lower bound estimate upper bound
Mean execution time 19.98 ms 20.18 ms 20.95 ms
Standard deviation 369.1 μs 1.734 ms 4.024 ms

Outlying measurements have severe (73.8%) effect on estimated standard deviation.

ternary-02/power

38.5 ms
39.0
39.5
40.0
40.5
41.0
41.5
42.0
42.5
ternary-02/power time densities
mean
38.5 ms
39.0
39.5
40.0
40.5
41.0
41.5
42.0
42.5
ternary-02/power times
lower bound estimate upper bound
Mean execution time 38.96 ms 39.01 ms 39.16 ms
Standard deviation 196.0 μs 423.7 μs 874.0 μs

Outlying measurements have no (1.0%) effect on estimated standard deviation.

ternary-03/naive

8.10 ms
8.20
8.30
8.40
8.50
8.60
ternary-03/naive time densities
mean
8.10 ms
8.20
8.30
8.40
8.50
8.60
ternary-03/naive times
lower bound estimate upper bound
Mean execution time 8.147 ms 8.161 ms 8.182 ms
Standard deviation 65.87 μs 87.66 μs 119.1 μs

Outlying measurements have no (1.0%) effect on estimated standard deviation.

ternary-03/lefteigen

80 ms
85
90
95
100
ternary-03/lefteigen time densities
mean
80 ms
85
90
95
100
ternary-03/lefteigen times
lower bound estimate upper bound
Mean execution time 81.97 ms 82.23 ms 83.02 ms
Standard deviation 645.9 μs 2.081 ms 4.483 ms

Outlying measurements have moderate (19.0%) effect on estimated standard deviation.

ternary-03/companion

30.0 ms
32.5
35.0
37.5
40.0
42.5
ternary-03/companion time densities
mean
30.0 ms
32.5
35.0
37.5
40.0
42.5
ternary-03/companion times
lower bound estimate upper bound
Mean execution time 29.26 ms 29.47 ms 30.02 ms
Standard deviation 713.6 μs 1.600 ms 3.246 ms

Outlying measurements have severe (52.5%) effect on estimated standard deviation.

ternary-03/power

54 ms
55
56
57
58
59
60
ternary-03/power time densities
mean
54 ms
55
56
57
58
59
60
ternary-03/power times
lower bound estimate upper bound
Mean execution time 54.68 ms 54.78 ms 54.99 ms
Standard deviation 408.7 μs 721.4 μs 1.285 ms

Outlying measurements have slight (6.6%) effect on estimated standard deviation.

ternary-04/naive

13.9 ms
14.0
14.1
14.2
14.3
14.4
14.5
14.6
ternary-04/naive time densities
mean
13.9 ms
14.0
14.1
14.2
14.3
14.4
14.5
14.6
ternary-04/naive times
lower bound estimate upper bound
Mean execution time 14.02 ms 14.04 ms 14.06 ms
Standard deviation 84.56 μs 112.7 μs 157.9 μs

Outlying measurements have no (1.0%) effect on estimated standard deviation.

ternary-04/lefteigen

152.5 ms
155.0
157.5
160.0
162.5
165.0
167.5
ternary-04/lefteigen time densities
mean
152.5 ms
155.0
157.5
160.0
162.5
165.0
167.5
ternary-04/lefteigen times
lower bound estimate upper bound
Mean execution time 153.0 ms 153.2 ms 153.8 ms
Standard deviation 829.8 μs 1.755 ms 3.599 ms

Outlying measurements have no (1.0%) effect on estimated standard deviation.

ternary-04/companion

50.0 ms
52.5
55.0
57.5
60.0
62.5
65.0
ternary-04/companion time densities
mean
50.0 ms
52.5
55.0
57.5
60.0
62.5
65.0
ternary-04/companion times
lower bound estimate upper bound
Mean execution time 51.20 ms 51.45 ms 52.04 ms
Standard deviation 777.9 μs 1.869 ms 3.309 ms

Outlying measurements have moderate (32.6%) effect on estimated standard deviation.

ternary-04/power

87.5 ms
90.0
92.5
95.0
97.5
100.0
102.5
ternary-04/power time densities
mean
87.5 ms
90.0
92.5
95.0
97.5
100.0
102.5
ternary-04/power times
lower bound estimate upper bound
Mean execution time 89.13 ms 89.39 ms 89.98 ms
Standard deviation 916.6 μs 1.916 ms 3.423 ms

Outlying measurements have moderate (14.2%) effect on estimated standard deviation.

4-ary-05/naive

650 μs
700
750
800
850
900
950
4-ary-05/naive time densities
mean
650 μs
700
750
800
850
900
950
4-ary-05/naive times
lower bound estimate upper bound
Mean execution time 685.5 μs 691.7 μs 703.5 μs
Standard deviation 26.69 μs 42.60 μs 67.45 μs

Outlying measurements have severe (58.5%) effect on estimated standard deviation.

4-ary-05/lefteigen

11.0 ms
11.5
12.0
12.5
4-ary-05/lefteigen time densities
mean
11.0 ms
11.5
12.0
12.5
4-ary-05/lefteigen times
lower bound estimate upper bound
Mean execution time 10.99 ms 11.03 ms 11.10 ms
Standard deviation 154.4 μs 253.9 μs 395.5 μs

Outlying measurements have moderate (16.1%) effect on estimated standard deviation.

4-ary-05/companion

2.5 ms
5.0
7.5
10.0
12.5
15.0
4-ary-05/companion time densities
mean
2.5 ms
5.0
7.5
10.0
12.5
15.0
4-ary-05/companion times
lower bound estimate upper bound
Mean execution time 2.406 ms 2.534 ms 3.114 ms
Standard deviation 134.8 μs 1.199 ms 2.844 ms

Outlying measurements have severe (98.9%) effect on estimated standard deviation.

4-ary-05/power

6.30 ms
6.40
6.50
6.60
6.70
4-ary-05/power time densities
mean
6.30 ms
6.40
6.50
6.60
6.70
4-ary-05/power times
lower bound estimate upper bound
Mean execution time 6.335 ms 6.347 ms 6.363 ms
Standard deviation 55.71 μs 70.43 μs 96.59 μs

Outlying measurements have no (1.0%) effect on estimated standard deviation.

4-ary-06/naive

4.20 ms
4.30
4.40
4.50
4.60
4.70
4-ary-06/naive time densities
mean
4.20 ms
4.30
4.40
4.50
4.60
4.70
4-ary-06/naive times
lower bound estimate upper bound
Mean execution time 4.318 ms 4.336 ms 4.359 ms
Standard deviation 85.07 μs 103.3 μs 124.1 μs

Outlying measurements have moderate (17.1%) effect on estimated standard deviation.

4-ary-06/lefteigen

44 ms
46
48
50
52
4-ary-06/lefteigen time densities
mean
44 ms
46
48
50
52
4-ary-06/lefteigen times
lower bound estimate upper bound
Mean execution time 43.61 ms 43.71 ms 44.08 ms
Standard deviation 246.7 μs 901.1 μs 2.063 ms

Outlying measurements have moderate (13.3%) effect on estimated standard deviation.

4-ary-06/companion

15.0 ms
16.0
17.0
18.0
19.0
20.0
4-ary-06/companion time densities
mean
15.0 ms
16.0
17.0
18.0
19.0
20.0
4-ary-06/companion times
lower bound estimate upper bound
Mean execution time 15.27 ms 15.35 ms 15.55 ms
Standard deviation 84.51 μs 574.2 μs 1.051 ms

Outlying measurements have moderate (33.6%) effect on estimated standard deviation.

4-ary-06/power

24 ms
25
26
27
28
29
30
31
4-ary-06/power time densities
mean
24 ms
25
26
27
28
29
30
31
4-ary-06/power times
lower bound estimate upper bound
Mean execution time 24.51 ms 24.58 ms 24.84 ms
Standard deviation 165.5 μs 623.9 μs 1.420 ms

Outlying measurements have moderate (19.0%) effect on estimated standard deviation.

4-ary-07/naive

2.12 ms
2.14
2.16
2.18
2.20
4-ary-07/naive time densities
mean
2.12 ms
2.14
2.16
2.18
2.20
4-ary-07/naive times
lower bound estimate upper bound
Mean execution time 2.133 ms 2.136 ms 2.140 ms
Standard deviation 16.35 μs 19.39 μs 23.08 μs

Outlying measurements have no (1.0%) effect on estimated standard deviation.

4-ary-07/lefteigen

22.5 ms
25.0
27.5
30.0
32.5
35.0
4-ary-07/lefteigen time densities
mean
22.5 ms
25.0
27.5
30.0
32.5
35.0
4-ary-07/lefteigen times
lower bound estimate upper bound
Mean execution time 22.10 ms 22.28 ms 22.83 ms
Standard deviation 577.1 μs 1.469 ms 3.202 ms

Outlying measurements have severe (61.6%) effect on estimated standard deviation.

4-ary-07/companion

6.5 ms
7.0
7.5
8.0
8.5
9.0
9.5
10.0
4-ary-07/companion time densities
mean
6.5 ms
7.0
7.5
8.0
8.5
9.0
9.5
10.0
4-ary-07/companion times
lower bound estimate upper bound
Mean execution time 6.922 ms 6.978 ms 7.094 ms
Standard deviation 236.8 μs 399.6 μs 708.1 μs

Outlying measurements have severe (55.5%) effect on estimated standard deviation.

4-ary-07/power

15.0 ms
15.5
16.0
16.5
17.0
17.5
18.0
4-ary-07/power time densities
mean
15.0 ms
15.5
16.0
16.5
17.0
17.5
18.0
4-ary-07/power times
lower bound estimate upper bound
Mean execution time 15.37 ms 15.42 ms 15.53 ms
Standard deviation 193.4 μs 384.9 μs 643.9 μs

Outlying measurements have moderate (19.0%) effect on estimated standard deviation.

4-ary-08/naive

26.5 ms
27.0
27.5
28.0
28.5
4-ary-08/naive time densities
mean
26.5 ms
27.0
27.5
28.0
28.5
4-ary-08/naive times
lower bound estimate upper bound
Mean execution time 27.00 ms 27.06 ms 27.14 ms
Standard deviation 256.9 μs 340.7 μs 452.4 μs

Outlying measurements have slight (2.4%) effect on estimated standard deviation.

4-ary-08/lefteigen

2.04 s
2.05
2.06
2.07
2.08
2.09
2.10
4-ary-08/lefteigen time densities
mean
2.04 s
2.05
2.06
2.07
2.08
2.09
2.10
4-ary-08/lefteigen times
lower bound estimate upper bound
Mean execution time 2.055 s 2.057 s 2.060 s
Standard deviation 11.38 ms 13.37 ms 15.57 ms

Outlying measurements have no (1.0%) effect on estimated standard deviation.

4-ary-08/companion

980 ms
990
1000
1010
1020
1030
1040
4-ary-08/companion time densities
mean
980 ms
990
1000
1010
1020
1030
1040
4-ary-08/companion times
lower bound estimate upper bound
Mean execution time 990.4 ms 992.0 ms 994.3 ms
Standard deviation 6.840 ms 9.574 ms 14.03 ms

Outlying measurements have no (1.0%) effect on estimated standard deviation.

4-ary-08/power

1.00 s
1.01
1.02
1.03
1.04
1.05
1.06
4-ary-08/power time densities
mean
1.00 s
1.01
1.02
1.03
1.04
1.05
1.06
4-ary-08/power times
lower bound estimate upper bound
Mean execution time 1.013 s 1.015 s 1.017 s
Standard deviation 9.176 ms 10.94 ms 13.43 ms

Outlying measurements have no (1.0%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. In each section, we display two charts, each with an x axis that represents measured execution time. These charts are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table displaying the mean and standard deviation of the measurements. We use a statistical technique called the bootstrap to provide confidence intervals on our estimates of these values. The bootstrap-derived upper and lower bounds on the mean and standard deviation let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.