Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/37ce3c2208167b01bad29761a4e0b0a368cb1f36 (+ 923 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz_LTO-build/186/
View Options
Report
Filter

llvm-arm-master-Oz_LTO test results

Run Order Start Time Duration
Current 0186 (llvmorg-20-init-08115-g37ce3c220816) 2024-09-29T09:59:08 0:00:00
Previous 0184 (llvmorg-20-init-07192-gd1cd2c3a2610) 2024-09-22T13:59:51 0:00:00
Baseline 0001 (llvmorg-16-init-17036-g4f40923103d2) 2023-01-06T17:55:45 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 10 19
Performance Improvements 29 112
Added Tests 8 2
Unchanged Tests 709 623
Total Tests 756

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
462.libquantum 5.17% 174.000 183.000 - -98.73% -
447.dealII 1.83% 1908.000 1943.000 - -79.29% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
462.libquantum (max) 125.00% 4.000 9.000 - -99.02% -
462.libquantum (max) (percentage) 85.30% 2.790 5.170 - -18.97% -
462.libquantum (unexplained diff vs previous) 80.00% 5.000 9.000 0.000 800.00% 0.000
462.libquantum (unexplained diff vs previous) (percentage) 71.43% 2.870 4.920 0.000 49100.00% 0.000
462.libquantum (avg) (percentage) 4.00% 1.500 1.560 - 102.60% -

Performance Regressions - Score Δ Previous Current σ Δ (B) σ (B)
nb_compile_successful -50.00% 36.000 18.000 - -5.26% -
nb_execution_successful -50.00% 34.000 17.000 - -5.56% -
nb_total_tests -50.00% 40.000 20.000 - 5.26% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -1.67% 60.000 59.000 - -99.38% -
471.omnetpp -1.17% 2914.000 2880.000 - -72.51% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) -29.17% 48.000 34.000 0.000 -66.99% 0.000
471.omnetpp (unexplained diff vs previous) (percentage) -28.48% 1.650 1.180 0.000 20.41% 0.000
450.soplex (max) -25.00% 8.000 6.000 - -97.59% -
450.soplex (max) (percentage) -22.36% 1.610 1.250 - -24.24% -
471.omnetpp (max) (percentage) -16.77% 1.670 1.390 - -85.02% -
471.omnetpp (max) -16.67% 48.000 40.000 - -95.88% -
456.hmmer (max) -13.79% 29.000 25.000 - -97.99% -
456.hmmer (unexplained diff vs previous) -13.79% 29.000 25.000 0.000 -45.65% 0.000
456.hmmer (unexplained diff vs previous) (percentage) -13.16% 0.760 0.660 0.000 69.23% 0.000
456.hmmer (max) (percentage) -10.53% 0.760 0.680 - -93.51% -
429.mcf (max) -6.67% 15.000 14.000 - -99.39% -
433.milc (max) (percentage) -6.45% 0.930 0.870 - 8.75% -
444.namd (max) (percentage) -5.88% 0.170 0.160 - -98.02% -
470.lbm (max) (percentage) -5.71% 0.700 0.660 - -88.46% -
473.astar (max) (percentage) -5.26% 0.190 0.180 - -97.31% -
445.gobmk (max) (percentage) -5.00% 0.200 0.190 - -94.91% -
447.dealII (max) -4.55% 22.000 21.000 - -90.58% -
447.dealII (max) (percentage) -4.31% 1.160 1.110 - -53.36% -
470.lbm (max) -3.70% 27.000 26.000 - -96.57% -
464.h264ref (max) (percentage) -3.57% 0.280 0.270 - -35.71% -
453.povray (max) (percentage) -2.83% 1.060 1.030 - 186.11% -
444.namd (avg) (percentage) -2.60% 0.770 0.750 - -70.82% -
471.omnetpp -2.56% 39.000 38.000 - -77.25% -
471.omnetpp (avg) -2.56% 39.000 38.000 - -77.25% -
429.mcf (max) (percentage) -1.49% 1.340 1.320 - -91.91% -
482.sphinx3 (max) (percentage) -1.36% 1.470 1.450 - -56.19% -
456.hmmer (avg) (percentage) -1.28% 0.780 0.770 - -44.60% -

Added Tests - sample variability
401.bzip2 (unexplained diff vs previous)
401.bzip2 (unexplained diff vs previous) (percentage)
429.mcf (unexplained diff vs previous)
429.mcf (unexplained diff vs previous) (percentage)
450.soplex (unexplained diff vs previous)
450.soplex (unexplained diff vs previous) (percentage)
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.92% 146765.000 149580.000 - 0.00% -
471.omnetpp 1.67% 316676.000 321958.000 - -0.00% -
473.astar 1.04% 15470.000 15631.000 - -0.13% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
462.libquantum (unexplained diff vs previous) (percentage) 49100.00% 0.010 4.920 0.000 71.43% 0.000
462.libquantum (unexplained diff vs previous) 800.00% 1.000 9.000 0.000 80.00% 0.000
403.gcc (avg) (percentage) 494.55% 0.550 3.270 - 0.00% -
429.mcf (unexplained diff vs previous) (percentage) 343.75% 0.160 0.710 0.000 0.00% 0.000
403.gcc (max) (percentage) 343.35% 2.330 10.330 - -0.86% -
453.povray (max) (percentage) 186.11% 0.360 1.030 - -2.83% -
450.soplex (avg) (percentage) 160.00% 0.500 1.300 - -0.76% -
462.libquantum (avg) (percentage) 102.60% 0.770 1.560 - 4.00% -
444.namd (unexplained diff vs previous) (percentage) 100.00% 0.060 0.120 0.000 0.00% 0.000
456.hmmer (unexplained diff vs previous) (percentage) 69.23% 0.390 0.660 0.000 -13.16% 0.000
471.omnetpp (unexplained diff vs previous) (percentage) 20.41% 0.980 1.180 0.000 -28.48% 0.000
450.soplex (unexplained diff vs previous) (percentage) 19.57% 0.460 0.550 0.000 0.00% 0.000
453.povray (avg) (percentage) 13.75% 0.800 0.910 - 0.00% -
433.milc (max) (percentage) 8.75% 0.800 0.870 - -6.45% -

Performance Regressions - Score Δ (B) Baseline Current σ (B) Δ σ
nb_execution_successful -5.56% 18.000 17.000 - -50.00% -
nb_compile_successful -5.26% 19.000 18.000 - -50.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
429.mcf -4.20% 7239.000 6935.000 - 0.00% -
462.libquantum -3.59% 12504.000 12055.000 - 0.00% -
453.povray -2.32% 524493.000 512312.000 - -0.01% -
433.milc -1.94% 47001.000 46089.000 - -0.02% -
444.namd -1.87% 124963.000 122622.000 - 0.00% -
464.h264ref -1.65% 303767.000 298765.000 - 0.02% -
458.sjeng -1.63% 80297.000 78986.000 - 0.00% -
401.bzip2 -1.52% 33698.000 33185.000 - 0.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.38% 9509.000 59.000 - -1.67% -
462.libquantum -98.73% 14409.000 183.000 - 5.17% -
450.soplex -96.41% 15112.000 542.000 - 0.56% -
482.sphinx3 -96.31% 26682.000 985.000 - 0.10% -
444.namd -93.42% 12400.000 816.000 - 0.12% -
453.povray -92.90% 6987.000 496.000 - -0.40% -
429.mcf -91.93% 13988.000 1129.000 - -0.70% -
433.milc -91.38% 13601.000 1172.000 - 0.26% -
401.bzip2 -82.43% 16137.000 2836.000 - 0.21% -
447.dealII -79.29% 9380.000 1943.000 - 1.83% -
471.omnetpp -72.51% 10477.000 2880.000 - -1.17% -
470.lbm -69.85% 13259.000 3997.000 - 0.43% -
456.hmmer -67.83% 11854.000 3814.000 - -0.65% -
473.astar -62.39% 12272.000 4615.000 - -0.41% -
464.h264ref -59.01% 14498.000 5943.000 - 0.10% -
445.gobmk -55.91% 11093.000 4891.000 - 0.20% -
458.sjeng -51.98% 17055.000 8189.000 - -0.13% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -99.90% 1003.000 1.000 - 0.00% -
483.xalancbmk (max) (percentage) -99.61% 12.900 0.050 - 0.00% -
429.mcf (max) -99.39% 2281.000 14.000 - -6.67% -
473.astar (max) -99.03% 822.000 8.000 - 0.00% -
462.libquantum (max) -99.02% 919.000 9.000 - 125.00% -
482.sphinx3 -98.49% 464.000 7.000 - 0.00% -
482.sphinx3 (avg) -98.49% 464.000 7.000 - 0.00% -
482.sphinx3 (max) -98.41% 883.000 14.000 - 0.00% -
401.bzip2 (max) -98.36% 609.000 10.000 - 0.00% -
462.libquantum -98.18% 110.000 2.000 - 0.00% -
462.libquantum (avg) -98.18% 110.000 2.000 - 0.00% -
444.namd -98.11% 318.000 6.000 - 0.00% -
444.namd (avg) -98.11% 318.000 6.000 - 0.00% -
403.gcc -98.08% 52.000 1.000 - 0.00% -
403.gcc (avg) -98.08% 52.000 1.000 - 0.00% -
444.namd (max) (percentage) -98.02% 8.090 0.160 - -5.88% -
456.hmmer (max) -97.99% 1242.000 25.000 - -13.79% -
458.sjeng (max) -97.95% 876.000 18.000 - 0.00% -
445.gobmk (max) -97.82% 413.000 9.000 - 0.00% -
450.soplex (max) -97.59% 249.000 6.000 - -25.00% -
473.astar (max) (percentage) -97.31% 6.700 0.180 - -5.26% -
403.gcc (max) -97.29% 221.000 6.000 - 0.00% -
470.lbm (max) -96.57% 758.000 26.000 - -3.70% -
471.omnetpp (max) -95.88% 972.000 40.000 - -16.67% -
450.soplex (unexplained diff vs previous) -95.65% 69.000 3.000 0.000 0.00% 0.000
458.sjeng (max) (percentage) -95.53% 5.140 0.230 - 0.00% -
429.mcf -95.27% 296.000 14.000 - 0.00% -
429.mcf (avg) -95.27% 296.000 14.000 - 0.00% -
470.lbm -95.24% 441.000 21.000 - 0.00% -
470.lbm (avg) -95.24% 441.000 21.000 - 0.00% -
445.gobmk (max) (percentage) -94.91% 3.730 0.190 - -5.00% -
456.hmmer (max) (percentage) -93.51% 10.480 0.680 - -10.53% -
433.milc -93.10% 116.000 8.000 - 0.00% -
433.milc (avg) -93.10% 116.000 8.000 - 0.00% -
453.povray -92.73% 55.000 4.000 - 0.00% -
453.povray (avg) -92.73% 55.000 4.000 - 0.00% -
429.mcf (max) (percentage) -91.91% 16.310 1.320 - -1.49% -
433.milc (max) -90.74% 108.000 10.000 - 0.00% -
450.soplex -90.67% 75.000 7.000 - 0.00% -
450.soplex (avg) -90.67% 75.000 7.000 - 0.00% -
447.dealII (max) -90.58% 223.000 21.000 - -4.55% -
401.bzip2 (max) (percentage) -89.95% 3.780 0.380 - 0.00% -
401.bzip2 -89.42% 104.000 11.000 - 0.00% -
401.bzip2 (avg) -89.42% 104.000 11.000 - 0.00% -
470.lbm (max) (percentage) -88.46% 5.720 0.660 - -5.71% -
444.namd (unexplained diff vs previous) -87.50% 8.000 1.000 0.000 0.00% 0.000
447.dealII -85.71% 98.000 14.000 - 0.00% -
447.dealII (avg) -85.71% 98.000 14.000 - 0.00% -
471.omnetpp (max) (percentage) -85.02% 9.280 1.390 - -16.77% -
470.lbm (avg) (percentage) -83.48% 3.330 0.550 - 0.00% -
445.gobmk -82.43% 74.000 13.000 - 0.00% -
445.gobmk (avg) -82.43% 74.000 13.000 - 0.00% -
456.hmmer -82.32% 164.000 29.000 - 0.00% -
456.hmmer (avg) -82.32% 164.000 29.000 - 0.00% -
401.bzip2 (unexplained diff vs previous) -81.82% 33.000 6.000 0.000 0.00% 0.000
458.sjeng -80.15% 136.000 27.000 - 0.00% -
458.sjeng (avg) -80.15% 136.000 27.000 - 0.00% -
453.povray (max) -80.00% 25.000 5.000 - 0.00% -
473.astar -79.46% 112.000 23.000 - 0.00% -
473.astar (avg) -79.46% 112.000 23.000 - 0.00% -
471.omnetpp -77.25% 167.000 38.000 - -2.56% -
471.omnetpp (avg) -77.25% 167.000 38.000 - -2.56% -
464.h264ref (max) -73.33% 60.000 16.000 - 0.00% -
444.namd (avg) (percentage) -70.82% 2.570 0.750 - -2.60% -
464.h264ref -69.05% 42.000 13.000 - 0.00% -
464.h264ref (avg) -69.05% 42.000 13.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) -66.99% 103.000 34.000 0.000 -29.17% 0.000
429.mcf (unexplained diff vs previous) -63.64% 22.000 8.000 0.000 0.00% 0.000
458.sjeng (avg) (percentage) -58.75% 0.800 0.330 - 0.00% -
445.gobmk (avg) (percentage) -58.21% 0.670 0.280 - 0.00% -
482.sphinx3 (max) (percentage) -56.19% 3.310 1.450 - -1.36% -
482.sphinx3 (avg) (percentage) -54.02% 1.740 0.800 - 0.00% -
447.dealII (max) (percentage) -53.36% 2.380 1.110 - -4.31% -
456.hmmer (unexplained diff vs previous) -45.65% 46.000 25.000 0.000 -13.79% 0.000
456.hmmer (avg) (percentage) -44.60% 1.390 0.770 - -1.28% -
473.astar (avg) (percentage) -44.57% 0.920 0.510 - 0.00% -
429.mcf (avg) (percentage) -40.09% 2.120 1.270 - -0.78% -
464.h264ref (max) (percentage) -35.71% 0.420 0.270 - -3.57% -
401.bzip2 (avg) (percentage) -35.38% 0.650 0.420 - 0.00% -
483.xalancbmk (avg) (percentage) -32.08% 2.120 1.440 - 0.00% -
447.dealII (avg) (percentage) -29.52% 1.050 0.740 - 0.00% -
450.soplex (max) (percentage) -24.24% 1.650 1.250 - -22.36% -
464.h264ref (avg) (percentage) -24.14% 0.290 0.220 - 0.00% -
433.milc (avg) (percentage) -19.77% 0.860 0.690 - 0.00% -
462.libquantum (max) (percentage) -18.97% 6.380 5.170 - 85.30% -
471.omnetpp (avg) (percentage) -15.62% 1.600 1.350 - -0.74% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 5.26% 19.000 20.000 - -50.00% -

Added Tests - sample variability
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)

Report Time: 0.36s
Execution (seconds)
Name Current %
401.bzip2 2836.000 -
403.gcc 59.000 -1.67%
429.mcf 1129.000 -
433.milc 1172.000 -
444.namd 816.000 -
445.gobmk 4891.000 -
447.dealII 1943.000 1.83%
450.soplex 542.000 -
453.povray 496.000 -
456.hmmer 3814.000 -
458.sjeng 8189.000 -
462.libquantum 183.000 5.17%
464.h264ref 5943.000 -
470.lbm 3997.000 -
471.omnetpp 2880.000 -1.17%
473.astar 4615.000 -
482.sphinx3 985.000 -
Geometric Mean 1483.929 -
Code Size (bytes)
Name Current %
401.bzip2 33185.000 -
403.gcc 1655187.000 -
429.mcf 6935.000 -
433.milc 46089.000 -
444.namd 122622.000 -
445.gobmk 1255425.000 -
447.dealII 194907.000 -
450.soplex 149580.000 -
453.povray 512312.000 -
456.hmmer 80241.000 -
458.sjeng 78986.000 -
462.libquantum 12055.000 -
464.h264ref 298765.000 -
470.lbm 9256.000 -
471.omnetpp 321958.000 -
473.astar 15631.000 -
482.sphinx3 87628.000 -
Geometric Mean 94410.217 -
Score
Name Current %
nb_compile_successful 18.000 -50.00%
nb_execution_successful 17.000 -50.00%
nb_total_tests 20.000 -50.00%
Geometric Mean 18.292 -50.00%
sample variability
Name Current %
401.bzip2 11.000 -
401.bzip2 (avg) 11.000 -
401.bzip2 (avg) (percentage) 0.420 -
401.bzip2 (max) 10.000 -
401.bzip2 (max) (percentage) 0.380 -
401.bzip2 (unexplained diff vs previous) 6.000 -
401.bzip2 (unexplained diff vs previous) (percentage) 0.210 -
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 3.270 -
403.gcc (max) 6.000 -
403.gcc (max) (percentage) 10.330 -
429.mcf 14.000 -
429.mcf (avg) 14.000 -
429.mcf (avg) (percentage) 1.270 -
429.mcf (max) 14.000 -6.67%
429.mcf (max) (percentage) 1.320 -1.49%
429.mcf (unexplained diff vs previous) 8.000 -
429.mcf (unexplained diff vs previous) (percentage) 0.710 -
433.milc 8.000 -
433.milc (avg) 8.000 -
433.milc (avg) (percentage) 0.690 -
433.milc (max) 10.000 -
433.milc (max) (percentage) 0.870 -6.45%
444.namd 6.000 -
444.namd (avg) 6.000 -
444.namd (avg) (percentage) 0.750 -2.60%
444.namd (max) 1.000 -
444.namd (max) (percentage) 0.160 -5.88%
444.namd (unexplained diff vs previous) 1.000 -
444.namd (unexplained diff vs previous) (percentage) 0.120 -
445.gobmk 13.000 -
445.gobmk (avg) 13.000 -
445.gobmk (avg) (percentage) 0.280 -
445.gobmk (max) 9.000 -
445.gobmk (max) (percentage) 0.190 -5.00%
447.dealII 14.000 -
447.dealII (avg) 14.000 -
447.dealII (avg) (percentage) 0.740 -
447.dealII (max) 21.000 -4.55%
447.dealII (max) (percentage) 1.110 -4.31%
450.soplex 7.000 -
450.soplex (avg) 7.000 -
450.soplex (avg) (percentage) 1.300 -
450.soplex (max) 6.000 -25.00%
450.soplex (max) (percentage) 1.250 -22.36%
450.soplex (unexplained diff vs previous) 3.000 -
450.soplex (unexplained diff vs previous) (percentage) 0.550 -
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.910 -
453.povray (max) 5.000 -
453.povray (max) (percentage) 1.030 -2.83%
456.hmmer 29.000 -
456.hmmer (avg) 29.000 -
456.hmmer (avg) (percentage) 0.770 -1.28%
456.hmmer (max) 25.000 -13.79%
456.hmmer (max) (percentage) 0.680 -10.53%
456.hmmer (unexplained diff vs previous) 25.000 -13.79%
456.hmmer (unexplained diff vs previous) (percentage) 0.660 -13.16%
458.sjeng 27.000 -
458.sjeng (avg) 27.000 -
458.sjeng (avg) (percentage) 0.330 -
458.sjeng (max) 18.000 -
458.sjeng (max) (percentage) 0.230 -
458.sjeng (unexplained diff vs previous) 11.000 -
458.sjeng (unexplained diff vs previous) (percentage) 0.130 -
462.libquantum 2.000 -
462.libquantum (avg) 2.000 -
462.libquantum (avg) (percentage) 1.560 4.00%
462.libquantum (max) 9.000 125.00%
462.libquantum (max) (percentage) 5.170 85.30%
462.libquantum (unexplained diff vs previous) 9.000 80.00%
462.libquantum (unexplained diff vs previous) (percentage) 4.920 71.43%
464.h264ref 13.000 -
464.h264ref (avg) 13.000 -
464.h264ref (avg) (percentage) 0.220 -
464.h264ref (max) 16.000 -
464.h264ref (max) (percentage) 0.270 -3.57%
470.lbm 21.000 -
470.lbm (avg) 21.000 -
470.lbm (avg) (percentage) 0.550 -
470.lbm (max) 26.000 -3.70%
470.lbm (max) (percentage) 0.660 -5.71%
471.omnetpp 38.000 -2.56%
471.omnetpp (avg) 38.000 -2.56%
471.omnetpp (avg) (percentage) 1.350 -
471.omnetpp (max) 40.000 -16.67%
471.omnetpp (max) (percentage) 1.390 -16.77%
471.omnetpp (unexplained diff vs previous) 34.000 -29.17%
471.omnetpp (unexplained diff vs previous) (percentage) 1.180 -28.48%
473.astar 23.000 -
473.astar (avg) 23.000 -
473.astar (avg) (percentage) 0.510 -
473.astar (max) 8.000 -
473.astar (max) (percentage) 0.180 -5.26%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.800 -
482.sphinx3 (max) 14.000 -
482.sphinx3 (max) (percentage) 1.450 -1.36%
483.xalancbmk (avg) (percentage) 1.440 -
483.xalancbmk (max) (percentage) 0.050 -
Geometric Mean 3.124 -6.65%