Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/a590d8634308aadc96e71c8dbbcbd4348f2efd1d (+ 866 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-aarch64-master-Oz_LTO-build/16/
View Options
Report
Filter

llvm-aarch64-master-Oz_LTO test results

Run Order Start Time Duration
Current 0016 (llvmorg-17-init-06545-ga590d8634308) 2023-03-25T00:42:29 0:00:00
Previous 0011 (llvmorg-17-init-05679-g558b46fde2db) 2023-03-19T06:48:42 0:00:00
Baseline 0001 (llvmorg-16-init-17110-gb1256f2345f6) 2022-12-17T20:51:03 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 13 48
Performance Improvements 52 63
Added Tests 6 16
Unchanged Tests 769 713
Total Tests 840

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
453.povray 1.73% 520.000 529.000 - -92.75% -
445.gobmk 1.61% 5216.000 5300.000 - -54.58% -
462.libquantum 1.53% 131.000 133.000 - -98.62% -
429.mcf 1.38% 1591.000 1613.000 - -91.47% -
456.hmmer 1.06% 2745.000 2774.000 - -69.59% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
444.namd (unexplained diff vs previous) 300.00% 1.000 4.000 - 0.00% -
444.namd (unexplained diff vs previous) (percentage) 293.33% 0.150 0.590 - 0.00% -
429.mcf (unexplained diff vs previous) 83.33% 12.000 22.000 - 0.00% -
429.mcf (unexplained diff vs previous) (percentage) 81.33% 0.750 1.360 - 0.00% -
447.dealII 5.88% 17.000 18.000 - -78.05% -
447.dealII (avg) 5.88% 17.000 18.000 - -78.05% -
456.hmmer (max) 4.76% 21.000 22.000 - -67.65% -
447.dealII (avg) (percentage) 2.11% 0.950 0.970 - 6.59% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
464.h264ref -2.92% 4457.000 4327.000 - -64.36% -
450.soplex -2.88% 486.000 472.000 - -96.63% -
470.lbm -2.15% 3212.000 3143.000 - -71.55% -
482.sphinx3 -1.12% 803.000 794.000 - -96.66% -
447.dealII -1.06% 1878.000 1858.000 - -79.59% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
458.sjeng (unexplained diff vs previous) (percentage) -78.95% 0.380 0.080 - 0.00% -
458.sjeng (unexplained diff vs previous) -78.57% 28.000 6.000 - 0.00% -
473.astar (unexplained diff vs previous) -39.29% 56.000 34.000 - 0.00% -
473.astar (unexplained diff vs previous) (percentage) -38.39% 1.120 0.690 - 0.00% -
470.lbm (unexplained diff vs previous) -26.60% 94.000 69.000 - 0.00% -
470.lbm (unexplained diff vs previous) (percentage) -24.91% 2.930 2.200 - 0.00% -
462.libquantum (max) (percentage) -13.16% 0.760 0.660 - 1550.00% -
450.soplex (max) -10.00% 20.000 18.000 - -88.24% -
450.soplex (max) (percentage) -9.95% 4.320 3.890 - 253.64% -
433.milc -7.69% 13.000 12.000 - -90.08% -
433.milc (avg) -7.69% 13.000 12.000 - -90.08% -
458.sjeng -7.69% 39.000 36.000 - 2.86% -
458.sjeng (avg) -7.69% 39.000 36.000 - 2.86% -
458.sjeng (avg) (percentage) -7.41% 0.540 0.500 - 117.39% -
470.lbm -6.54% 153.000 143.000 - 123.44% -
470.lbm (avg) -6.54% 153.000 143.000 - 123.44% -
464.h264ref -6.25% 16.000 15.000 - -65.91% -
464.h264ref (avg) -6.25% 16.000 15.000 - -65.91% -
429.mcf (avg) (percentage) -5.38% 4.650 4.400 - 1089.19% -
471.omnetpp (max) -5.13% 39.000 37.000 - -68.91% -
473.astar -4.88% 82.000 78.000 - 766.67% -
473.astar (avg) -4.88% 82.000 78.000 - 766.67% -
470.lbm (avg) (percentage) -4.61% 4.770 4.550 - 684.48% -
464.h264ref (max) -4.55% 22.000 21.000 - -34.38% -
433.milc (avg) (percentage) -4.50% 1.110 1.060 - 27.71% -
433.milc (max) -4.44% 45.000 43.000 - -84.01% -
473.astar (avg) (percentage) -4.24% 1.650 1.580 - 2157.14% -
429.mcf -4.11% 73.000 70.000 - 1.45% -
429.mcf (avg) -4.11% 73.000 70.000 - 1.45% -
444.namd (avg) (percentage) -3.88% 1.290 1.240 - -45.61% -
433.milc (max) (percentage) -3.65% 3.840 3.700 - 100.00% -
429.mcf (max) (percentage) -3.57% 12.890 12.430 - 3352.78% -
401.bzip2 (max) (percentage) -3.53% 6.790 6.550 - 13000.00% -
401.bzip2 (max) -3.51% 171.000 165.000 - 2650.00% -
482.sphinx3 -3.33% 30.000 29.000 - -86.57% -
482.sphinx3 (avg) -3.33% 30.000 29.000 - -86.57% -
447.dealII (max) -3.12% 32.000 31.000 - -75.40% -
471.omnetpp (max) (percentage) -3.05% 1.310 1.270 - 27.00% -
482.sphinx3 (max) -3.03% 165.000 160.000 - -2.44% -
445.gobmk (max) (percentage) -2.50% 0.800 0.780 - 30.00% -
429.mcf (max) -2.44% 205.000 200.000 - 194.12% -
470.lbm (max) -2.44% 246.000 240.000 - 3900.00% -
447.dealII (max) (percentage) -2.30% 1.740 1.700 - 22.30% -
453.povray (max) (percentage) -2.03% 1.480 1.450 - 36.79% -
464.h264ref (max) (percentage) -2.00% 0.500 0.490 - 81.48% -
482.sphinx3 (max) (percentage) -1.85% 20.550 20.170 - 2823.19% -
458.sjeng (max) -1.59% 63.000 62.000 - 77.14% -

Added Tests - sample variability
403.gcc
403.gcc (avg)
403.gcc (max)
447.dealII (unexplained diff vs previous)
447.dealII (unexplained diff vs previous) (percentage)
462.libquantum (max)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 3.33% 224407.000 231874.000 - -0.01% -
447.dealII 1.04% 325219.000 328611.000 - 0.00% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
401.bzip2 (max) (percentage) 13000.00% 0.050 6.550 - -3.53% -
470.lbm (max) (percentage) 12666.67% 0.060 7.660 - 0.00% -
473.astar (max) (percentage) 4300.00% 0.080 3.520 - 0.00% -
470.lbm (max) 3900.00% 6.000 240.000 - -2.44% -
429.mcf (max) (percentage) 3352.78% 0.360 12.430 - -3.57% -
482.sphinx3 (max) (percentage) 2823.19% 0.690 20.170 - -1.85% -
401.bzip2 (max) 2650.00% 6.000 165.000 - -3.51% -
433.milc (unexplained diff vs previous) (percentage) 2450.00% 0.020 0.510 - 0.00% -
473.astar (avg) (percentage) 2157.14% 0.070 1.580 - -4.24% -
401.bzip2 (avg) (percentage) 2060.00% 0.100 2.160 - 0.00% -
473.astar (max) 1640.00% 10.000 174.000 - -0.57% -
462.libquantum (max) (percentage) 1550.00% 0.040 0.660 - -13.16% -
462.libquantum (avg) (percentage) 1200.00% 0.110 1.430 - 0.00% -
429.mcf (avg) (percentage) 1089.19% 0.370 4.400 - -5.38% -
473.astar 766.67% 9.000 78.000 - -4.88% -
473.astar (avg) 766.67% 9.000 78.000 - -4.88% -
450.soplex (avg) (percentage) 733.33% 0.540 4.500 - 0.00% -
470.lbm (avg) (percentage) 684.48% 0.580 4.550 - -4.61% -
401.bzip2 315.38% 13.000 54.000 - 0.00% -
401.bzip2 (avg) 315.38% 13.000 54.000 - 0.00% -
482.sphinx3 (avg) (percentage) 312.09% 0.910 3.750 - 0.00% -
458.sjeng (max) (percentage) 273.91% 0.230 0.860 - 0.00% -
450.soplex (max) (percentage) 253.64% 1.100 3.890 - -9.95% -
429.mcf (max) 194.12% 68.000 200.000 - -2.44% -
470.lbm 123.44% 64.000 143.000 - -6.54% -
470.lbm (avg) 123.44% 64.000 143.000 - -6.54% -
458.sjeng (avg) (percentage) 117.39% 0.230 0.500 - -7.41% -
433.milc (max) (percentage) 100.00% 1.850 3.700 - -3.65% -
433.milc (unexplained diff vs previous) 100.00% 3.000 6.000 - 0.00% -
464.h264ref (max) (percentage) 81.48% 0.270 0.490 - -2.00% -
458.sjeng (max) 77.14% 35.000 62.000 - -1.59% -
453.povray (max) (percentage) 36.79% 1.060 1.450 - -2.03% -
445.gobmk (max) (percentage) 30.00% 0.600 0.780 - -2.50% -
433.milc (avg) (percentage) 27.71% 0.830 1.060 - -4.50% -
471.omnetpp (max) (percentage) 27.00% 1.000 1.270 - -3.05% -
447.dealII (max) (percentage) 22.30% 1.390 1.700 - -2.30% -
453.povray (avg) (percentage) 20.55% 0.730 0.880 - 0.00% -
445.gobmk (avg) (percentage) 15.62% 0.320 0.370 - 0.00% -
456.hmmer (max) (percentage) 6.67% 0.750 0.800 - 0.00% -
447.dealII (avg) (percentage) 6.59% 0.910 0.970 - 2.11% -
458.sjeng 2.86% 35.000 36.000 - -7.69% -
458.sjeng (avg) 2.86% 35.000 36.000 - -7.69% -
429.mcf 1.45% 69.000 70.000 - -4.11% -
429.mcf (avg) 1.45% 69.000 70.000 - -4.11% -

Performance Regressions - Score Δ (B) Baseline Current σ (B) Δ σ
nb_compile_successful -5.26% 19.000 18.000 - 0.00% -
nb_execution_successful -5.26% 19.000 18.000 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
471.omnetpp -2.87% 490147.000 476073.000 - -0.02% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.32% 10150.000 69.000 - 0.00% -
462.libquantum -98.62% 9659.000 133.000 - 1.53% -
482.sphinx3 -96.66% 23779.000 794.000 - -1.12% -
450.soplex -96.63% 13988.000 472.000 - -2.88% -
444.namd -92.77% 9377.000 678.000 - -0.59% -
453.povray -92.75% 7301.000 529.000 - 1.73% -
433.milc -91.87% 14590.000 1186.000 - 0.51% -
429.mcf -91.47% 18905.000 1613.000 - 1.38% -
400.perlbench -83.69% 11085.000 1808.000 - 0.89% -
401.bzip2 -81.87% 13962.000 2532.000 - 0.16% -
447.dealII -79.59% 9102.000 1858.000 - -1.06% -
471.omnetpp -75.11% 11917.000 2966.000 - -0.84% -
470.lbm -71.55% 11046.000 3143.000 - -2.15% -
456.hmmer -69.59% 9122.000 2774.000 - 1.06% -
464.h264ref -64.36% 12142.000 4327.000 - -2.92% -
473.astar -62.72% 13301.000 4959.000 - -0.68% -
445.gobmk -54.58% 11668.000 5300.000 - 1.61% -
458.sjeng -52.09% 15282.000 7321.000 - -0.08% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -97.32% 485.000 13.000 - 0.00% -
444.namd -96.24% 213.000 8.000 - 0.00% -
444.namd (avg) -96.24% 213.000 8.000 - 0.00% -
453.povray -92.45% 53.000 4.000 - 0.00% -
453.povray (avg) -92.45% 53.000 4.000 - 0.00% -
453.povray (max) -90.91% 77.000 7.000 - 0.00% -
433.milc -90.08% 121.000 12.000 - -7.69% -
433.milc (avg) -90.08% 121.000 12.000 - -7.69% -
462.libquantum -90.00% 10.000 1.000 - 0.00% -
462.libquantum (avg) -90.00% 10.000 1.000 - 0.00% -
400.perlbench (max) -88.31% 77.000 9.000 - 0.00% -
450.soplex (max) -88.24% 153.000 18.000 - -10.00% -
482.sphinx3 -86.57% 216.000 29.000 - -3.33% -
482.sphinx3 (avg) -86.57% 216.000 29.000 - -3.33% -
433.milc (max) -84.01% 269.000 43.000 - -4.44% -
400.perlbench -83.93% 56.000 9.000 - 0.00% -
400.perlbench (avg) -83.93% 56.000 9.000 - 0.00% -
447.dealII -78.05% 82.000 18.000 - 5.88% -
447.dealII (avg) -78.05% 82.000 18.000 - 5.88% -
471.omnetpp -76.06% 71.000 17.000 - 0.00% -
471.omnetpp (avg) -76.06% 71.000 17.000 - 0.00% -
447.dealII (max) -75.40% 126.000 31.000 - -3.12% -
456.hmmer -72.50% 80.000 22.000 - 0.00% -
456.hmmer (avg) -72.50% 80.000 22.000 - 0.00% -
450.soplex -72.00% 75.000 21.000 - 0.00% -
450.soplex (avg) -72.00% 75.000 21.000 - 0.00% -
471.omnetpp (max) -68.91% 119.000 37.000 - -5.13% -
456.hmmer (max) -67.65% 68.000 22.000 - 4.76% -
464.h264ref -65.91% 44.000 15.000 - -6.25% -
464.h264ref (avg) -65.91% 44.000 15.000 - -6.25% -
444.namd (max) (percentage) -62.16% 5.180 1.960 - 0.00% -
445.gobmk -48.65% 37.000 19.000 - 0.00% -
445.gobmk (avg) -48.65% 37.000 19.000 - 0.00% -
444.namd (avg) (percentage) -45.61% 2.280 1.240 - -3.88% -
445.gobmk (max) -41.43% 70.000 41.000 - 0.00% -
464.h264ref (max) -34.38% 32.000 21.000 - -4.55% -
400.perlbench (max) (percentage) -22.86% 0.700 0.540 - 0.00% -
483.xalancbmk (max) (percentage) -19.61% 0.510 0.410 - 0.00% -
403.gcc (max) (percentage) -15.00% 0.400 0.340 - 0.00% -
456.hmmer (avg) (percentage) -7.95% 0.880 0.810 - 0.00% -
464.h264ref (avg) (percentage) -2.70% 0.370 0.360 - 0.00% -
482.sphinx3 (max) -2.44% 164.000 160.000 - -3.03% -
471.omnetpp (avg) (percentage) -1.67% 0.600 0.590 - 0.00% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 52.63% 19.000 29.000 - 0.00% -

Added Tests - sample variability
403.gcc
403.gcc (avg)
403.gcc (max)
429.mcf (unexplained diff vs previous)
429.mcf (unexplained diff vs previous) (percentage)
444.namd (unexplained diff vs previous)
444.namd (unexplained diff vs previous) (percentage)
447.dealII (unexplained diff vs previous)
447.dealII (unexplained diff vs previous) (percentage)
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)
462.libquantum (max)
470.lbm (unexplained diff vs previous)
470.lbm (unexplained diff vs previous) (percentage)
473.astar (unexplained diff vs previous)
473.astar (unexplained diff vs previous) (percentage)

Report Time: 0.35s
Execution (seconds)
Name Current %
400.perlbench 1808.000 -
401.bzip2 2532.000 -
403.gcc 69.000 -
429.mcf 1613.000 1.38%
433.milc 1186.000 -
444.namd 678.000 -
445.gobmk 5300.000 1.61%
447.dealII 1858.000 -1.06%
450.soplex 472.000 -2.88%
453.povray 529.000 1.73%
456.hmmer 2774.000 1.06%
458.sjeng 7321.000 -
462.libquantum 133.000 1.53%
464.h264ref 4327.000 -2.92%
470.lbm 3143.000 -2.15%
471.omnetpp 2966.000 -
473.astar 4959.000 -
482.sphinx3 794.000 -1.12%
Geometric Mean 1400.953 -
Code Size (bytes)
Name Current %
400.perlbench 899062.000 -
401.bzip2 43928.000 -
403.gcc 2484708.000 -
429.mcf 9248.000 -
433.milc 62842.000 -
444.namd 143408.000 -
445.gobmk 1852074.000 -
447.dealII 328611.000 -
450.soplex 231874.000 -
453.povray 691593.000 -
456.hmmer 113556.000 -
458.sjeng 106791.000 -
462.libquantum 16876.000 -
464.h264ref 400009.000 -
470.lbm 10320.000 -
471.omnetpp 476073.000 -
473.astar 23784.000 -
482.sphinx3 120117.000 -
Geometric Mean 146076.518 -
sample variability
Name Current %
400.perlbench 9.000 -
400.perlbench (avg) 9.000 -
400.perlbench (avg) (percentage) 0.510 -
400.perlbench (max) 9.000 -
400.perlbench (max) (percentage) 0.540 -
401.bzip2 54.000 -
401.bzip2 (avg) 54.000 -
401.bzip2 (avg) (percentage) 2.160 -
401.bzip2 (max) 165.000 -3.51%
401.bzip2 (max) (percentage) 6.550 -3.53%
403.gcc 0.000 -
403.gcc (avg) 0.000 -
403.gcc (avg) (percentage) 0.310 -
403.gcc (max) 0.000 -
403.gcc (max) (percentage) 0.340 -
429.mcf 70.000 -4.11%
429.mcf (avg) 70.000 -4.11%
429.mcf (avg) (percentage) 4.400 -5.38%
429.mcf (max) 200.000 -2.44%
429.mcf (max) (percentage) 12.430 -3.57%
429.mcf (unexplained diff vs previous) 22.000 83.33%
429.mcf (unexplained diff vs previous) (percentage) 1.360 81.33%
433.milc 12.000 -7.69%
433.milc (avg) 12.000 -7.69%
433.milc (avg) (percentage) 1.060 -4.50%
433.milc (max) 43.000 -4.44%
433.milc (max) (percentage) 3.700 -3.65%
433.milc (unexplained diff vs previous) 6.000 -
433.milc (unexplained diff vs previous) (percentage) 0.510 -
444.namd 8.000 -
444.namd (avg) 8.000 -
444.namd (avg) (percentage) 1.240 -3.88%
444.namd (max) 13.000 -
444.namd (max) (percentage) 1.960 -
444.namd (unexplained diff vs previous) 4.000 300.00%
444.namd (unexplained diff vs previous) (percentage) 0.590 293.33%
445.gobmk 19.000 -
445.gobmk (avg) 19.000 -
445.gobmk (avg) (percentage) 0.370 -
445.gobmk (max) 41.000 -
445.gobmk (max) (percentage) 0.780 -2.50%
447.dealII 18.000 5.88%
447.dealII (avg) 18.000 5.88%
447.dealII (avg) (percentage) 0.970 2.11%
447.dealII (max) 31.000 -3.12%
447.dealII (max) (percentage) 1.700 -2.30%
447.dealII (unexplained diff vs previous) 20.000 -
447.dealII (unexplained diff vs previous) (percentage) 1.080 -
450.soplex 21.000 -
450.soplex (avg) 21.000 -
450.soplex (avg) (percentage) 4.500 -
450.soplex (max) 18.000 -10.00%
450.soplex (max) (percentage) 3.890 -9.95%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.880 -
453.povray (max) 7.000 -
453.povray (max) (percentage) 1.450 -2.03%
456.hmmer 22.000 -
456.hmmer (avg) 22.000 -
456.hmmer (avg) (percentage) 0.810 -
456.hmmer (max) 22.000 4.76%
456.hmmer (max) (percentage) 0.800 -
458.sjeng 36.000 -7.69%
458.sjeng (avg) 36.000 -7.69%
458.sjeng (avg) (percentage) 0.500 -7.41%
458.sjeng (max) 62.000 -1.59%
458.sjeng (max) (percentage) 0.860 -
458.sjeng (unexplained diff vs previous) 6.000 -78.57%
458.sjeng (unexplained diff vs previous) (percentage) 0.080 -78.95%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.430 -
462.libquantum (max) 0.000 -
462.libquantum (max) (percentage) 0.660 -13.16%
464.h264ref 15.000 -6.25%
464.h264ref (avg) 15.000 -6.25%
464.h264ref (avg) (percentage) 0.360 -
464.h264ref (max) 21.000 -4.55%
464.h264ref (max) (percentage) 0.490 -2.00%
470.lbm 143.000 -6.54%
470.lbm (avg) 143.000 -6.54%
470.lbm (avg) (percentage) 4.550 -4.61%
470.lbm (max) 240.000 -2.44%
470.lbm (max) (percentage) 7.660 -
470.lbm (unexplained diff vs previous) 69.000 -26.60%
470.lbm (unexplained diff vs previous) (percentage) 2.200 -24.91%
471.omnetpp 17.000 -
471.omnetpp (avg) 17.000 -
471.omnetpp (avg) (percentage) 0.590 -
471.omnetpp (max) 37.000 -5.13%
471.omnetpp (max) (percentage) 1.270 -3.05%
473.astar 78.000 -4.88%
473.astar (avg) 78.000 -4.88%
473.astar (avg) (percentage) 1.580 -4.24%
473.astar (max) 174.000 -
473.astar (max) (percentage) 3.520 -
473.astar (unexplained diff vs previous) 34.000 -39.29%
473.astar (unexplained diff vs previous) (percentage) 0.690 -38.39%
482.sphinx3 29.000 -3.33%
482.sphinx3 (avg) 29.000 -3.33%
482.sphinx3 (avg) (percentage) 3.750 -
482.sphinx3 (max) 160.000 -3.03%
482.sphinx3 (max) (percentage) 20.170 -1.85%
483.xalancbmk (avg) (percentage) 0.430 -
483.xalancbmk (max) (percentage) 0.410 -
Geometric Mean 4.200 -36.22%