Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/173907b5d77115623f160978a95159e36e05ee6c (+ 1501 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz_LTO-build/192/
View Options
Report
Filter

llvm-arm-master-Oz_LTO test results

Run Order Start Time Duration
Current 0192 (llvmorg-20-init-09616-g173907b5d771) 2024-10-11T22:12:30 0:00:00
Previous 0191 (no-change) 2024-10-02T19:58:46 0:00:00
Baseline 0001 (llvmorg-16-init-17036-g4f40923103d2) 2023-01-06T17:55:45 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 21 18
Performance Improvements 34 113
Added Tests 2 6
Unchanged Tests 711 631
Total Tests 768

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
450.soplex 1.68% 536.000 545.000 - -96.39% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
458.sjeng (unexplained diff vs previous) (percentage) 3700.00% 0.010 0.380 - 0.00% -
458.sjeng (unexplained diff vs previous) 3000.00% 1.000 31.000 - 0.00% -
464.h264ref (unexplained diff vs previous) 250.00% 6.000 21.000 - 0.00% -
464.h264ref (unexplained diff vs previous) (percentage) 250.00% 0.100 0.350 - 0.00% -
464.h264ref (max) (percentage) 105.88% 0.170 0.350 - -16.67% -
464.h264ref (max) 100.00% 10.000 20.000 - -66.67% -
458.sjeng (max) 72.22% 18.000 31.000 - -96.46% -
458.sjeng (max) (percentage) 65.22% 0.230 0.380 - -92.61% -
450.soplex (max) 50.00% 6.000 9.000 - -96.39% -
450.soplex (unexplained diff vs previous) 50.00% 6.000 9.000 - -86.96% -
450.soplex (unexplained diff vs previous) (percentage) 47.32% 1.120 1.650 - 258.70% -
450.soplex (max) (percentage) 43.59% 1.170 1.680 - 1.82% -
473.astar (max) 23.08% 13.000 16.000 - -98.05% -
450.soplex 16.67% 6.000 7.000 - -90.67% -
450.soplex (avg) 16.67% 6.000 7.000 - -90.67% -
473.astar (max) (percentage) 16.67% 0.300 0.350 - -94.78% -
473.astar (unexplained diff vs previous) (percentage) 16.67% 0.300 0.350 - 75.00% -
473.astar (unexplained diff vs previous) 14.29% 14.000 16.000 - -36.00% -
471.omnetpp (unexplained diff vs previous) (percentage) 4.42% 1.130 1.180 - 20.41% -
471.omnetpp (unexplained diff vs previous) 3.03% 33.000 34.000 - -66.99% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -5.00% 60.000 57.000 - -99.40% -
471.omnetpp -1.17% 2913.000 2879.000 - -72.52% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
433.milc (unexplained diff vs previous) -80.00% 15.000 3.000 - -76.92% -
433.milc (unexplained diff vs previous) (percentage) -80.00% 1.300 0.260 - 160.00% -
444.namd (max) -33.33% 3.000 2.000 - -99.80% -
401.bzip2 (unexplained diff vs previous) (percentage) -22.22% 0.180 0.140 - -30.00% -
401.bzip2 (max) -20.00% 5.000 4.000 - -99.34% -
401.bzip2 (unexplained diff vs previous) -20.00% 5.000 4.000 - -87.88% -
462.libquantum (max) -12.50% 8.000 7.000 - -99.24% -
401.bzip2 (max) (percentage) -10.53% 0.190 0.170 - -95.50% -
433.milc (max) (percentage) -10.16% 1.280 1.150 - 43.75% -
462.libquantum (max) (percentage) -8.33% 4.920 4.510 - -29.31% -
456.hmmer (max) -8.33% 24.000 22.000 - -98.23% -
471.omnetpp (max) -8.33% 36.000 33.000 - -96.60% -
429.mcf (max) (percentage) -8.20% 0.610 0.560 - -96.57% -
444.namd (max) (percentage) -8.11% 0.370 0.340 - -95.80% -
433.milc (max) -7.14% 14.000 13.000 - -87.96% -
445.gobmk (max) -7.14% 14.000 13.000 - -96.85% -
445.gobmk (max) (percentage) -6.90% 0.290 0.270 - -92.76% -
471.omnetpp (max) (percentage) -6.40% 1.250 1.170 - -87.39% -
447.dealII (max) -5.88% 17.000 16.000 - -92.83% -
453.povray (max) (percentage) -5.00% 0.600 0.570 - 58.33% -
470.lbm (max) (percentage) -5.00% 0.600 0.570 - -90.03% -
456.hmmer (max) (percentage) -4.76% 0.630 0.600 - -94.27% -
470.lbm (max) -4.35% 23.000 22.000 - -97.10% -
429.mcf (unexplained diff vs previous) (percentage) -3.70% 0.270 0.260 - 62.50% -
471.omnetpp -2.56% 39.000 38.000 - -77.25% -
471.omnetpp (avg) -2.56% 39.000 38.000 - -77.25% -
447.dealII (max) (percentage) -2.25% 0.890 0.870 - -63.45% -
473.astar (avg) (percentage) -1.96% 0.510 0.500 - -45.65% -
462.libquantum (avg) (percentage) -1.85% 1.620 1.590 - 106.49% -
433.milc (avg) (percentage) -1.41% 0.710 0.700 - -18.60% -
482.sphinx3 (max) (percentage) -1.35% 0.740 0.730 - -77.95% -
403.gcc (max) (percentage) -1.31% 9.900 9.770 - 319.31% -

Added Tests - sample variability
462.libquantum (unexplained diff vs previous)
462.libquantum (unexplained diff vs previous) (percentage)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.92% 146765.000 149580.000 - 0.00% -
471.omnetpp 1.67% 316676.000 321950.000 - -0.00% -
473.astar 1.04% 15470.000 15631.000 - 0.00% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
403.gcc (avg) (percentage) 474.55% 0.550 3.160 - 0.00% -
403.gcc (max) (percentage) 319.31% 2.330 9.770 - -1.31% -
450.soplex (unexplained diff vs previous) (percentage) 258.70% 0.460 1.650 - 47.32% -
450.soplex (avg) (percentage) 160.00% 0.500 1.300 - 0.78% -
433.milc (unexplained diff vs previous) (percentage) 160.00% 0.100 0.260 - -80.00% -
462.libquantum (avg) (percentage) 106.49% 0.770 1.590 - -1.85% -
473.astar (unexplained diff vs previous) (percentage) 75.00% 0.200 0.350 - 16.67% -
429.mcf (unexplained diff vs previous) (percentage) 62.50% 0.160 0.260 - -3.70% -
453.povray (max) (percentage) 58.33% 0.360 0.570 - -5.00% -
433.milc (max) (percentage) 43.75% 0.800 1.150 - -10.16% -
471.omnetpp (unexplained diff vs previous) (percentage) 20.41% 0.980 1.180 - 4.42% -
453.povray (avg) (percentage) 12.50% 0.800 0.900 - 0.00% -
450.soplex (max) (percentage) 1.82% 1.650 1.680 - 43.59% -

Performance Regressions - Score Δ (B) Baseline Current σ (B) Δ σ
nb_execution_successful -5.56% 18.000 17.000 - 0.00% -
nb_compile_successful -5.26% 19.000 18.000 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
429.mcf -4.20% 7239.000 6935.000 - 0.00% -
462.libquantum -3.59% 12504.000 12055.000 - 0.00% -
453.povray -2.32% 524493.000 512304.000 - -0.00% -
433.milc -1.94% 47001.000 46089.000 - 0.00% -
444.namd -1.91% 124963.000 122574.000 - -0.04% -
464.h264ref -1.64% 303767.000 298797.000 - 0.01% -
458.sjeng -1.63% 80297.000 78986.000 - 0.00% -
401.bzip2 -1.52% 33698.000 33185.000 - 0.00% -
470.lbm -1.14% 9338.000 9232.000 - -0.26% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.40% 9509.000 57.000 - -5.00% -
462.libquantum -98.79% 14409.000 174.000 - 0.00% -
450.soplex -96.39% 15112.000 545.000 - 1.68% -
482.sphinx3 -96.30% 26682.000 987.000 - -0.10% -
444.namd -93.42% 12400.000 816.000 - 0.37% -
453.povray -92.94% 6987.000 493.000 - 0.00% -
429.mcf -91.89% 13988.000 1135.000 - 0.27% -
433.milc -91.47% 13601.000 1160.000 - 0.26% -
401.bzip2 -82.43% 16137.000 2835.000 - 0.14% -
447.dealII -79.36% 9380.000 1936.000 - 0.47% -
471.omnetpp -72.52% 10477.000 2879.000 - -1.17% -
470.lbm -70.01% 13259.000 3976.000 - -0.05% -
456.hmmer -68.01% 11854.000 3792.000 - -0.58% -
473.astar -62.38% 12272.000 4617.000 - 0.35% -
464.h264ref -58.90% 14498.000 5958.000 - 0.35% -
445.gobmk -55.92% 11093.000 4890.000 - 0.27% -
458.sjeng -51.81% 17055.000 8219.000 - 0.38% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -99.80% 1003.000 2.000 - -33.33% -
429.mcf (max) -99.74% 2281.000 6.000 - 0.00% -
483.xalancbmk (max) (percentage) -99.61% 12.900 0.050 - 0.00% -
401.bzip2 (max) -99.34% 609.000 4.000 - -20.00% -
462.libquantum (max) -99.24% 919.000 7.000 - -12.50% -
482.sphinx3 (max) -99.21% 883.000 7.000 - 0.00% -
482.sphinx3 -98.49% 464.000 7.000 - 0.00% -
482.sphinx3 (avg) -98.49% 464.000 7.000 - 0.00% -
456.hmmer (max) -98.23% 1242.000 22.000 - -8.33% -
462.libquantum -98.18% 110.000 2.000 - 0.00% -
462.libquantum (avg) -98.18% 110.000 2.000 - 0.00% -
444.namd -98.11% 318.000 6.000 - 0.00% -
444.namd (avg) -98.11% 318.000 6.000 - 0.00% -
403.gcc -98.08% 52.000 1.000 - 0.00% -
403.gcc (avg) -98.08% 52.000 1.000 - 0.00% -
473.astar (max) -98.05% 822.000 16.000 - 23.08% -
403.gcc (max) -97.74% 221.000 5.000 - 0.00% -
470.lbm (max) -97.10% 758.000 22.000 - -4.35% -
445.gobmk (max) -96.85% 413.000 13.000 - -7.14% -
471.omnetpp (max) -96.60% 972.000 33.000 - -8.33% -
429.mcf (max) (percentage) -96.57% 16.310 0.560 - -8.20% -
458.sjeng (max) -96.46% 876.000 31.000 - 72.22% -
450.soplex (max) -96.39% 249.000 9.000 - 50.00% -
444.namd (max) (percentage) -95.80% 8.090 0.340 - -8.11% -
401.bzip2 (max) (percentage) -95.50% 3.780 0.170 - -10.53% -
429.mcf -95.27% 296.000 14.000 - 0.00% -
429.mcf (avg) -95.27% 296.000 14.000 - 0.00% -
470.lbm -95.24% 441.000 21.000 - 0.00% -
470.lbm (avg) -95.24% 441.000 21.000 - 0.00% -
473.astar (max) (percentage) -94.78% 6.700 0.350 - 16.67% -
456.hmmer (max) (percentage) -94.27% 10.480 0.600 - -4.76% -
433.milc -93.10% 116.000 8.000 - 0.00% -
433.milc (avg) -93.10% 116.000 8.000 - 0.00% -
447.dealII (max) -92.83% 223.000 16.000 - -5.88% -
445.gobmk (max) (percentage) -92.76% 3.730 0.270 - -6.90% -
453.povray -92.73% 55.000 4.000 - 0.00% -
453.povray (avg) -92.73% 55.000 4.000 - 0.00% -
458.sjeng (max) (percentage) -92.61% 5.140 0.380 - 65.22% -
453.povray (max) -92.00% 25.000 2.000 - 0.00% -
450.soplex -90.67% 75.000 7.000 - 16.67% -
450.soplex (avg) -90.67% 75.000 7.000 - 16.67% -
470.lbm (max) (percentage) -90.03% 5.720 0.570 - -5.00% -
401.bzip2 -89.42% 104.000 11.000 - 0.00% -
401.bzip2 (avg) -89.42% 104.000 11.000 - 0.00% -
433.milc (max) -87.96% 108.000 13.000 - -7.14% -
401.bzip2 (unexplained diff vs previous) -87.88% 33.000 4.000 - -20.00% -
471.omnetpp (max) (percentage) -87.39% 9.280 1.170 - -6.40% -
450.soplex (unexplained diff vs previous) -86.96% 69.000 9.000 - 50.00% -
429.mcf (unexplained diff vs previous) -86.36% 22.000 3.000 - 0.00% -
447.dealII -85.71% 98.000 14.000 - 0.00% -
447.dealII (avg) -85.71% 98.000 14.000 - 0.00% -
470.lbm (avg) (percentage) -83.48% 3.330 0.550 - 0.00% -
456.hmmer -82.93% 164.000 28.000 - 0.00% -
456.hmmer (avg) -82.93% 164.000 28.000 - 0.00% -
445.gobmk -82.43% 74.000 13.000 - 0.00% -
445.gobmk (avg) -82.43% 74.000 13.000 - 0.00% -
458.sjeng -80.88% 136.000 26.000 - 0.00% -
458.sjeng (avg) -80.88% 136.000 26.000 - 0.00% -
473.astar -79.46% 112.000 23.000 - 0.00% -
473.astar (avg) -79.46% 112.000 23.000 - 0.00% -
482.sphinx3 (max) (percentage) -77.95% 3.310 0.730 - -1.35% -
471.omnetpp -77.25% 167.000 38.000 - -2.56% -
471.omnetpp (avg) -77.25% 167.000 38.000 - -2.56% -
433.milc (unexplained diff vs previous) -76.92% 13.000 3.000 - -80.00% -
444.namd (avg) (percentage) -71.21% 2.570 0.740 - 0.00% -
464.h264ref -69.05% 42.000 13.000 - 0.00% -
464.h264ref (avg) -69.05% 42.000 13.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) -66.99% 103.000 34.000 - 3.03% -
464.h264ref (max) -66.67% 60.000 20.000 - 100.00% -
447.dealII (max) (percentage) -63.45% 2.380 0.870 - -2.25% -
458.sjeng (avg) (percentage) -60.00% 0.800 0.320 - 0.00% -
445.gobmk (avg) (percentage) -58.21% 0.670 0.280 - 0.00% -
482.sphinx3 (avg) (percentage) -54.60% 1.740 0.790 - 0.00% -
473.astar (avg) (percentage) -45.65% 0.920 0.500 - -1.96% -
456.hmmer (avg) (percentage) -45.32% 1.390 0.760 - 0.00% -
429.mcf (avg) (percentage) -41.51% 2.120 1.240 - -0.80% -
401.bzip2 (avg) (percentage) -36.92% 0.650 0.410 - 0.00% -
473.astar (unexplained diff vs previous) -36.00% 25.000 16.000 - 14.29% -
483.xalancbmk (avg) (percentage) -32.08% 2.120 1.440 - 0.00% -
401.bzip2 (unexplained diff vs previous) (percentage) -30.00% 0.200 0.140 - -22.22% -
462.libquantum (max) (percentage) -29.31% 6.380 4.510 - -8.33% -
447.dealII (avg) (percentage) -28.57% 1.050 0.750 - 0.00% -
464.h264ref (avg) (percentage) -24.14% 0.290 0.220 - 0.00% -
433.milc (avg) (percentage) -18.60% 0.860 0.700 - -1.41% -
464.h264ref (max) (percentage) -16.67% 0.420 0.350 - 105.88% -
471.omnetpp (avg) (percentage) -15.62% 1.600 1.350 - 0.00% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 5.26% 19.000 20.000 - 0.00% -

Added Tests - sample variability
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)
462.libquantum (unexplained diff vs previous)
462.libquantum (unexplained diff vs previous) (percentage)
464.h264ref (unexplained diff vs previous)
464.h264ref (unexplained diff vs previous) (percentage)

Report Time: 0.38s
Execution (seconds)
Name Current %
401.bzip2 2835.000 -
403.gcc 57.000 -5.00%
429.mcf 1135.000 -
433.milc 1160.000 -
444.namd 816.000 -
445.gobmk 4890.000 -
447.dealII 1936.000 -
450.soplex 545.000 1.68%
453.povray 493.000 -
456.hmmer 3792.000 -
458.sjeng 8219.000 -
462.libquantum 174.000 -
464.h264ref 5958.000 -
470.lbm 3976.000 -
471.omnetpp 2879.000 -1.17%
473.astar 4617.000 -
482.sphinx3 987.000 -
Geometric Mean 1475.453 -
Code Size (bytes)
Name Current %
401.bzip2 33185.000 -
403.gcc 1655307.000 -
429.mcf 6935.000 -
433.milc 46089.000 -
444.namd 122574.000 -
445.gobmk 1255433.000 -
447.dealII 194787.000 -
450.soplex 149580.000 -
453.povray 512304.000 -
456.hmmer 80057.000 -
458.sjeng 78986.000 -
462.libquantum 12055.000 -
464.h264ref 298797.000 -
470.lbm 9232.000 -
471.omnetpp 321950.000 -
473.astar 15631.000 -
482.sphinx3 87620.000 -
Geometric Mean 94377.761 -
sample variability
Name Current %
401.bzip2 11.000 -
401.bzip2 (avg) 11.000 -
401.bzip2 (avg) (percentage) 0.410 -
401.bzip2 (max) 4.000 -20.00%
401.bzip2 (max) (percentage) 0.170 -10.53%
401.bzip2 (unexplained diff vs previous) 4.000 -20.00%
401.bzip2 (unexplained diff vs previous) (percentage) 0.140 -22.22%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 3.160 -
403.gcc (max) 5.000 -
403.gcc (max) (percentage) 9.770 -1.31%
429.mcf 14.000 -
429.mcf (avg) 14.000 -
429.mcf (avg) (percentage) 1.240 -
429.mcf (max) 6.000 -
429.mcf (max) (percentage) 0.560 -8.20%
429.mcf (unexplained diff vs previous) 3.000 -
429.mcf (unexplained diff vs previous) (percentage) 0.260 -3.70%
433.milc 8.000 -
433.milc (avg) 8.000 -
433.milc (avg) (percentage) 0.700 -1.41%
433.milc (max) 13.000 -7.14%
433.milc (max) (percentage) 1.150 -10.16%
433.milc (unexplained diff vs previous) 3.000 -80.00%
433.milc (unexplained diff vs previous) (percentage) 0.260 -80.00%
444.namd 6.000 -
444.namd (avg) 6.000 -
444.namd (avg) (percentage) 0.740 -
444.namd (max) 2.000 -33.33%
444.namd (max) (percentage) 0.340 -8.11%
445.gobmk 13.000 -
445.gobmk (avg) 13.000 -
445.gobmk (avg) (percentage) 0.280 -
445.gobmk (max) 13.000 -7.14%
445.gobmk (max) (percentage) 0.270 -6.90%
447.dealII 14.000 -
447.dealII (avg) 14.000 -
447.dealII (avg) (percentage) 0.750 -
447.dealII (max) 16.000 -5.88%
447.dealII (max) (percentage) 0.870 -2.25%
450.soplex 7.000 16.67%
450.soplex (avg) 7.000 16.67%
450.soplex (avg) (percentage) 1.300 -
450.soplex (max) 9.000 50.00%
450.soplex (max) (percentage) 1.680 43.59%
450.soplex (unexplained diff vs previous) 9.000 50.00%
450.soplex (unexplained diff vs previous) (percentage) 1.650 47.32%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.900 -
453.povray (max) 2.000 -
453.povray (max) (percentage) 0.570 -5.00%
456.hmmer 28.000 -
456.hmmer (avg) 28.000 -
456.hmmer (avg) (percentage) 0.760 -
456.hmmer (max) 22.000 -8.33%
456.hmmer (max) (percentage) 0.600 -4.76%
458.sjeng 26.000 -
458.sjeng (avg) 26.000 -
458.sjeng (avg) (percentage) 0.320 -
458.sjeng (max) 31.000 72.22%
458.sjeng (max) (percentage) 0.380 65.22%
458.sjeng (unexplained diff vs previous) 31.000 3000.00%
458.sjeng (unexplained diff vs previous) (percentage) 0.380 3700.00%
462.libquantum 2.000 -
462.libquantum (avg) 2.000 -
462.libquantum (avg) (percentage) 1.590 -1.85%
462.libquantum (max) 7.000 -12.50%
462.libquantum (max) (percentage) 4.510 -8.33%
462.libquantum (unexplained diff vs previous) 0.000 -
462.libquantum (unexplained diff vs previous) (percentage) 0.000 -
464.h264ref 13.000 -
464.h264ref (avg) 13.000 -
464.h264ref (avg) (percentage) 0.220 -
464.h264ref (max) 20.000 100.00%
464.h264ref (max) (percentage) 0.350 105.88%
464.h264ref (unexplained diff vs previous) 21.000 250.00%
464.h264ref (unexplained diff vs previous) (percentage) 0.350 250.00%
470.lbm 21.000 -
470.lbm (avg) 21.000 -
470.lbm (avg) (percentage) 0.550 -
470.lbm (max) 22.000 -4.35%
470.lbm (max) (percentage) 0.570 -5.00%
471.omnetpp 38.000 -2.56%
471.omnetpp (avg) 38.000 -2.56%
471.omnetpp (avg) (percentage) 1.350 -
471.omnetpp (max) 33.000 -8.33%
471.omnetpp (max) (percentage) 1.170 -6.40%
471.omnetpp (unexplained diff vs previous) 34.000 3.03%
471.omnetpp (unexplained diff vs previous) (percentage) 1.180 4.42%
473.astar 23.000 -
473.astar (avg) 23.000 -
473.astar (avg) (percentage) 0.500 -1.96%
473.astar (max) 16.000 23.08%
473.astar (max) (percentage) 0.350 16.67%
473.astar (unexplained diff vs previous) 16.000 14.29%
473.astar (unexplained diff vs previous) (percentage) 0.350 16.67%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.790 -
482.sphinx3 (max) 7.000 -
482.sphinx3 (max) (percentage) 0.730 -1.35%
483.xalancbmk (avg) (percentage) 1.440 -
483.xalancbmk (max) (percentage) 0.050 -
Geometric Mean 2.520 -10.56%