Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/19e2ebbf45bdd9f2a9abc3f7760bdbea8c808dc0 (+ 462 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-aarch64-master-Oz_LTO-build/9/
View Options
Report
Filter

llvm-aarch64-master-Oz_LTO test results

Run Order Start Time Duration
Current 0009 (llvmorg-17-init-05087-g19e2ebbf45bd) 2023-03-13T05:10:44 0:00:00
Previous 0008 (llvmorg-17-init-04625-g2869d31a7ac1) 2023-03-10T03:14:23 0:00:00
Baseline 0001 (llvmorg-16-init-17110-gb1256f2345f6) 2022-12-17T20:51:03 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 7 49
Performance Improvements 35 60
Added Tests 3 7
Unchanged Tests 759 688
Total Tests 804

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
450.soplex 2.66% 488.000 501.000 - -96.42% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
450.soplex (unexplained diff vs previous) 85.71% 7.000 13.000 - 0.00% -
450.soplex (unexplained diff vs previous) (percentage) 81.12% 1.430 2.590 - 0.00% -
401.bzip2 1.67% 60.000 61.000 - 369.23% -
401.bzip2 (avg) 1.67% 60.000 61.000 - 369.23% -
429.mcf 1.19% 84.000 85.000 - 23.19% -
429.mcf (avg) 1.19% 84.000 85.000 - 23.19% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -5.63% 71.000 67.000 - -99.34% -
471.omnetpp -1.81% 2985.000 2931.000 - -75.40% -
456.hmmer -1.79% 2787.000 2737.000 - -70.00% -
400.perlbench -1.20% 1828.000 1806.000 - -83.71% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
401.bzip2 (max) (percentage) -10.01% 8.490 7.640 - 15180.00% -
450.soplex (max) (percentage) -9.96% 9.240 8.320 - 656.36% -
462.libquantum (max) (percentage) -9.94% 3.620 3.260 - 8050.00% -
401.bzip2 (max) -9.30% 215.000 195.000 - 3150.00% -
450.soplex (max) -8.89% 45.000 41.000 - -73.20% -
471.omnetpp (max) -6.82% 44.000 41.000 - -65.55% -
471.omnetpp (max) (percentage) -4.73% 1.480 1.410 - 41.00% -
456.hmmer (max) -4.35% 46.000 44.000 - -35.29% -
462.libquantum (avg) (percentage) -4.32% 1.620 1.550 - 1309.09% -
464.h264ref (max) -4.17% 24.000 23.000 - -28.12% -
450.soplex (avg) (percentage) -3.74% 4.810 4.630 - 757.41% -
429.mcf (max) (percentage) -3.74% 18.980 18.270 - 4975.00% -
473.astar (max) (percentage) -3.73% 7.780 7.490 - 9262.50% -
447.dealII (max) (percentage) -3.66% 1.910 1.840 - 32.37% -
456.hmmer (max) (percentage) -3.59% 1.670 1.610 - 114.67% -
482.sphinx3 (max) (percentage) -3.57% 22.410 21.610 - 3031.88% -
470.lbm (max) (percentage) -3.40% 21.160 20.440 - 33966.67% -
482.sphinx3 (max) -3.39% 177.000 171.000 - 4.27% -
473.astar (max) -3.38% 385.000 372.000 - 3620.00% -
433.milc (max) (percentage) -3.31% 3.320 3.210 - 73.51% -
429.mcf (max) -2.89% 311.000 302.000 - 344.12% -
447.dealII (max) -2.86% 35.000 34.000 - -73.02% -
470.lbm (max) -2.73% 659.000 641.000 - 10583.33% -
453.povray (max) (percentage) -2.52% 1.590 1.550 - 46.23% -
433.milc (max) -2.50% 40.000 39.000 - -85.50% -
458.sjeng (max) (percentage) -2.43% 2.060 2.010 - 773.91% -
483.xalancbmk (max) (percentage) -2.38% 0.420 0.410 - -19.61% -
445.gobmk (max) (percentage) -2.25% 0.890 0.870 - 45.00% -
445.gobmk (max) -2.17% 46.000 45.000 - -35.71% -
458.sjeng (max) -2.00% 150.000 147.000 - 320.00% -
464.h264ref (max) (percentage) -1.85% 0.540 0.530 - 96.30% -

Added Tests - sample variability
403.gcc
403.gcc (avg)
403.gcc (max)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 3.34% 224407.000 231906.000 - 0.00% -
447.dealII 1.05% 325219.000 328619.000 - -0.10% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
470.lbm (max) (percentage) 33966.67% 0.060 20.440 - -3.40% -
401.bzip2 (max) (percentage) 15180.00% 0.050 7.640 - -10.01% -
470.lbm (max) 10583.33% 6.000 641.000 - -2.73% -
473.astar (max) (percentage) 9262.50% 0.080 7.490 - -3.73% -
462.libquantum (max) (percentage) 8050.00% 0.040 3.260 - -9.94% -
429.mcf (max) (percentage) 4975.00% 0.360 18.270 - -3.74% -
473.astar (max) 3620.00% 10.000 372.000 - -3.38% -
401.bzip2 (max) 3150.00% 6.000 195.000 - -9.30% -
482.sphinx3 (max) (percentage) 3031.88% 0.690 21.610 - -3.57% -
473.astar (avg) (percentage) 2342.86% 0.070 1.710 - 0.00% -
401.bzip2 (avg) (percentage) 2300.00% 0.100 2.400 - 0.00% -
462.libquantum (avg) (percentage) 1309.09% 0.110 1.550 - -4.32% -
429.mcf (avg) (percentage) 1294.59% 0.370 5.160 - 0.00% -
473.astar 833.33% 9.000 84.000 - 0.00% -
473.astar (avg) 833.33% 9.000 84.000 - 0.00% -
458.sjeng (max) (percentage) 773.91% 0.230 2.010 - -2.43% -
450.soplex (avg) (percentage) 757.41% 0.540 4.630 - -3.74% -
470.lbm (avg) (percentage) 750.00% 0.580 4.930 - 0.00% -
450.soplex (max) (percentage) 656.36% 1.100 8.320 - -9.96% -
482.sphinx3 (avg) (percentage) 381.32% 0.910 4.380 - 0.00% -
401.bzip2 369.23% 13.000 61.000 - 1.67% -
401.bzip2 (avg) 369.23% 13.000 61.000 - 1.67% -
429.mcf (max) 344.12% 68.000 302.000 - -2.89% -
458.sjeng (max) 320.00% 35.000 147.000 - -2.00% -
458.sjeng (avg) (percentage) 147.83% 0.230 0.570 - 0.00% -
470.lbm 140.62% 64.000 154.000 - 0.65% -
470.lbm (avg) 140.62% 64.000 154.000 - 0.65% -
456.hmmer (max) (percentage) 114.67% 0.750 1.610 - -3.59% -
464.h264ref (max) (percentage) 96.30% 0.270 0.530 - -1.85% -
433.milc (max) (percentage) 73.51% 1.850 3.210 - -3.31% -
453.povray (max) (percentage) 46.23% 1.060 1.550 - -2.52% -
445.gobmk (max) (percentage) 45.00% 0.600 0.870 - -2.25% -
471.omnetpp (max) (percentage) 41.00% 1.000 1.410 - -4.73% -
445.gobmk (avg) (percentage) 34.37% 0.320 0.430 - 0.00% -
462.libquantum (max) 33.33% 3.000 4.000 - 0.00% -
447.dealII (max) (percentage) 32.37% 1.390 1.840 - -3.66% -
453.povray (avg) (percentage) 31.51% 0.730 0.960 - 0.00% -
429.mcf 23.19% 69.000 85.000 - 1.19% -
429.mcf (avg) 23.19% 69.000 85.000 - 1.19% -
458.sjeng 17.14% 35.000 41.000 - 0.00% -
458.sjeng (avg) 17.14% 35.000 41.000 - 0.00% -
447.dealII (avg) (percentage) 14.29% 0.910 1.040 - 0.00% -
456.hmmer (avg) (percentage) 4.55% 0.880 0.920 - 0.00% -
482.sphinx3 (max) 4.27% 164.000 171.000 - -3.39% -
433.milc (avg) (percentage) 3.61% 0.830 0.860 - 0.00% -

Performance Regressions - Score Δ (B) Baseline Current σ (B) Δ σ
nb_compile_successful -5.26% 19.000 18.000 - 0.00% -
nb_execution_successful -5.26% 19.000 18.000 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
471.omnetpp -2.85% 490147.000 476173.000 - 0.01% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.34% 10150.000 67.000 - -5.63% -
462.libquantum -98.62% 9659.000 133.000 - 0.76% -
482.sphinx3 -96.67% 23779.000 792.000 - 0.25% -
450.soplex -96.42% 13988.000 501.000 - 2.66% -
453.povray -92.75% 7301.000 529.000 - 0.38% -
444.namd -92.74% 9377.000 681.000 - 0.15% -
433.milc -91.62% 14590.000 1223.000 - 0.82% -
429.mcf -91.25% 18905.000 1655.000 - 0.79% -
400.perlbench -83.71% 11085.000 1806.000 - -1.20% -
401.bzip2 -81.68% 13962.000 2558.000 - 0.75% -
447.dealII -79.60% 9102.000 1857.000 - -0.43% -
471.omnetpp -75.40% 11917.000 2931.000 - -1.81% -
470.lbm -71.57% 11046.000 3140.000 - 0.74% -
456.hmmer -70.00% 9122.000 2737.000 - -1.79% -
464.h264ref -63.30% 12142.000 4456.000 - -0.85% -
473.astar -62.66% 13301.000 4967.000 - 0.20% -
445.gobmk -55.14% 11668.000 5234.000 - -0.40% -
458.sjeng -52.09% 15282.000 7322.000 - 0.33% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -97.32% 485.000 13.000 - 0.00% -
444.namd -95.31% 213.000 10.000 - 0.00% -
444.namd (avg) -95.31% 213.000 10.000 - 0.00% -
433.milc -91.74% 121.000 10.000 - 0.00% -
433.milc (avg) -91.74% 121.000 10.000 - 0.00% -
453.povray -90.57% 53.000 5.000 - 0.00% -
453.povray (avg) -90.57% 53.000 5.000 - 0.00% -
453.povray (max) -89.61% 77.000 8.000 - 0.00% -
400.perlbench (max) -87.01% 77.000 10.000 - 0.00% -
433.milc (max) -85.50% 269.000 39.000 - -2.50% -
482.sphinx3 -84.26% 216.000 34.000 - 0.00% -
482.sphinx3 (avg) -84.26% 216.000 34.000 - 0.00% -
400.perlbench -83.93% 56.000 9.000 - 0.00% -
400.perlbench (avg) -83.93% 56.000 9.000 - 0.00% -
462.libquantum -80.00% 10.000 2.000 - 0.00% -
462.libquantum (avg) -80.00% 10.000 2.000 - 0.00% -
447.dealII -76.83% 82.000 19.000 - 0.00% -
447.dealII (avg) -76.83% 82.000 19.000 - 0.00% -
471.omnetpp -76.06% 71.000 17.000 - 0.00% -
471.omnetpp (avg) -76.06% 71.000 17.000 - 0.00% -
450.soplex (max) -73.20% 153.000 41.000 - -8.89% -
447.dealII (max) -73.02% 126.000 34.000 - -2.86% -
450.soplex -69.33% 75.000 23.000 - 0.00% -
450.soplex (avg) -69.33% 75.000 23.000 - 0.00% -
456.hmmer -68.75% 80.000 25.000 - 0.00% -
456.hmmer (avg) -68.75% 80.000 25.000 - 0.00% -
471.omnetpp (max) -65.55% 119.000 41.000 - -6.82% -
464.h264ref -63.64% 44.000 16.000 - 0.00% -
464.h264ref (avg) -63.64% 44.000 16.000 - 0.00% -
444.namd (max) (percentage) -62.16% 5.180 1.960 - 0.00% -
445.gobmk -40.54% 37.000 22.000 - 0.00% -
445.gobmk (avg) -40.54% 37.000 22.000 - 0.00% -
445.gobmk (max) -35.71% 70.000 45.000 - -2.17% -
456.hmmer (max) -35.29% 68.000 44.000 - -4.35% -
444.namd (avg) (percentage) -33.77% 2.280 1.510 - 0.00% -
464.h264ref (max) -28.12% 32.000 23.000 - -4.17% -
400.perlbench (max) (percentage) -20.00% 0.700 0.560 - 0.00% -
483.xalancbmk (max) (percentage) -19.61% 0.510 0.410 - -2.38% -
403.gcc (max) (percentage) -15.00% 0.400 0.340 - 0.00% -
471.omnetpp (avg) (percentage) -1.67% 0.600 0.590 - 0.00% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 52.63% 19.000 29.000 - 0.00% -

Added Tests - sample variability
403.gcc
403.gcc (avg)
403.gcc (max)
450.soplex (unexplained diff vs previous)
450.soplex (unexplained diff vs previous) (percentage)
462.libquantum (unexplained diff vs previous)
462.libquantum (unexplained diff vs previous) (percentage)

Report Time: 0.35s
Execution (seconds)
Name Current %
400.perlbench 1806.000 -1.20%
401.bzip2 2558.000 -
403.gcc 67.000 -5.63%
429.mcf 1655.000 -
433.milc 1223.000 -
444.namd 681.000 -
445.gobmk 5234.000 -
447.dealII 1857.000 -
450.soplex 501.000 2.66%
453.povray 529.000 -
456.hmmer 2737.000 -1.79%
458.sjeng 7322.000 -
462.libquantum 133.000 -
464.h264ref 4456.000 -
470.lbm 3140.000 -
471.omnetpp 2931.000 -1.81%
473.astar 4967.000 -
482.sphinx3 792.000 -
Geometric Mean 1407.932 -
Code Size (bytes)
Name Current %
400.perlbench 900050.000 -
401.bzip2 44192.000 -
403.gcc 2484804.000 -
429.mcf 9248.000 -
433.milc 62842.000 -
444.namd 143408.000 -
445.gobmk 1851698.000 -
447.dealII 328619.000 -
450.soplex 231906.000 -
453.povray 691441.000 -
456.hmmer 113660.000 -
458.sjeng 106791.000 -
462.libquantum 16880.000 -
464.h264ref 400193.000 -
470.lbm 10320.000 -
471.omnetpp 476173.000 -
473.astar 23784.000 -
482.sphinx3 120121.000 -
Geometric Mean 146147.333 -
sample variability
Name Current %
400.perlbench 9.000 -
400.perlbench (avg) 9.000 -
400.perlbench (avg) (percentage) 0.510 -
400.perlbench (max) 10.000 -
400.perlbench (max) (percentage) 0.560 -
401.bzip2 61.000 1.67%
401.bzip2 (avg) 61.000 1.67%
401.bzip2 (avg) (percentage) 2.400 -
401.bzip2 (max) 195.000 -9.30%
401.bzip2 (max) (percentage) 7.640 -10.01%
403.gcc 0.000 -
403.gcc (avg) 0.000 -
403.gcc (avg) (percentage) 0.310 -
403.gcc (max) 0.000 -
403.gcc (max) (percentage) 0.340 -
429.mcf 85.000 1.19%
429.mcf (avg) 85.000 1.19%
429.mcf (avg) (percentage) 5.160 -
429.mcf (max) 302.000 -2.89%
429.mcf (max) (percentage) 18.270 -3.74%
433.milc 10.000 -
433.milc (avg) 10.000 -
433.milc (avg) (percentage) 0.860 -
433.milc (max) 39.000 -2.50%
433.milc (max) (percentage) 3.210 -3.31%
444.namd 10.000 -
444.namd (avg) 10.000 -
444.namd (avg) (percentage) 1.510 -
444.namd (max) 13.000 -
444.namd (max) (percentage) 1.960 -
445.gobmk 22.000 -
445.gobmk (avg) 22.000 -
445.gobmk (avg) (percentage) 0.430 -
445.gobmk (max) 45.000 -2.17%
445.gobmk (max) (percentage) 0.870 -2.25%
447.dealII 19.000 -
447.dealII (avg) 19.000 -
447.dealII (avg) (percentage) 1.040 -
447.dealII (max) 34.000 -2.86%
447.dealII (max) (percentage) 1.840 -3.66%
450.soplex 23.000 -
450.soplex (avg) 23.000 -
450.soplex (avg) (percentage) 4.630 -3.74%
450.soplex (max) 41.000 -8.89%
450.soplex (max) (percentage) 8.320 -9.96%
450.soplex (unexplained diff vs previous) 13.000 85.71%
450.soplex (unexplained diff vs previous) (percentage) 2.590 81.12%
453.povray 5.000 -
453.povray (avg) 5.000 -
453.povray (avg) (percentage) 0.960 -
453.povray (max) 8.000 -
453.povray (max) (percentage) 1.550 -2.52%
456.hmmer 25.000 -
456.hmmer (avg) 25.000 -
456.hmmer (avg) (percentage) 0.920 -
456.hmmer (max) 44.000 -4.35%
456.hmmer (max) (percentage) 1.610 -3.59%
458.sjeng 41.000 -
458.sjeng (avg) 41.000 -
458.sjeng (avg) (percentage) 0.570 -
458.sjeng (max) 147.000 -2.00%
458.sjeng (max) (percentage) 2.010 -2.43%
462.libquantum 2.000 -
462.libquantum (avg) 2.000 -
462.libquantum (avg) (percentage) 1.550 -4.32%
462.libquantum (max) 4.000 -
462.libquantum (max) (percentage) 3.260 -9.94%
462.libquantum (unexplained diff vs previous) 1.000 -
462.libquantum (unexplained diff vs previous) (percentage) 0.750 -
464.h264ref 16.000 -
464.h264ref (avg) 16.000 -
464.h264ref (avg) (percentage) 0.370 -
464.h264ref (max) 23.000 -4.17%
464.h264ref (max) (percentage) 0.530 -1.85%
470.lbm 154.000 -
470.lbm (avg) 154.000 -
470.lbm (avg) (percentage) 4.930 -
470.lbm (max) 641.000 -2.73%
470.lbm (max) (percentage) 20.440 -3.40%
471.omnetpp 17.000 -
471.omnetpp (avg) 17.000 -
471.omnetpp (avg) (percentage) 0.590 -
471.omnetpp (max) 41.000 -6.82%
471.omnetpp (max) (percentage) 1.410 -4.73%
473.astar 84.000 -
473.astar (avg) 84.000 -
473.astar (avg) (percentage) 1.710 -
473.astar (max) 372.000 -3.38%
473.astar (max) (percentage) 7.490 -3.73%
482.sphinx3 34.000 -
482.sphinx3 (avg) 34.000 -
482.sphinx3 (avg) (percentage) 4.380 -
482.sphinx3 (max) 171.000 -3.39%
482.sphinx3 (max) (percentage) 21.610 -3.57%
483.xalancbmk (avg) (percentage) 0.430 -
483.xalancbmk (max) (percentage) 0.410 -2.38%
Geometric Mean 5.729 -14.75%