Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/0defd832eb7c0618a67556e6fcbd32dd19e88b97 (+ 4066 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz_LTO-build/248/
View Options
Report
Filter

llvm-arm-master-Oz_LTO test results

Run Order Start Time Duration
Current 0248 (llvmorg-21-init-07628-g0defd832eb7c) 2025-03-18T14:22:42 0:00:00
Previous 0238 (llvmorg-21-init-03562-gcb7030dbe7f3) 2025-02-27T15:48:52 0:00:00
Baseline 0001 (llvmorg-16-init-17036-g4f40923103d2) 2023-01-06T17:55:45 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 17 13
Performance Improvements 39 117
Added Tests 2 2
Unchanged Tests 698 624
Total Tests 756

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc 3.51% 57.000 59.000 - -99.38% -
462.libquantum 1.64% 183.000 186.000 - -98.71% -
433.milc 1.21% 1161.000 1175.000 - -91.36% -
453.povray 1.00% 500.000 505.000 - -92.77% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) 4100.00% 1.000 42.000 - -59.22% -
471.omnetpp (unexplained diff vs previous) (percentage) 3650.00% 0.040 1.500 - 53.06% -
473.astar (unexplained diff vs previous) (percentage) 550.00% 0.020 0.130 - -35.00% -
473.astar (unexplained diff vs previous) 500.00% 1.000 6.000 - -76.00% -
471.omnetpp (max) (percentage) 196.00% 0.500 1.480 - -84.05% -
471.omnetpp (max) 192.86% 14.000 41.000 - -95.78% -
403.gcc 100.00% 1.000 2.000 - -96.15% -
403.gcc (avg) 100.00% 1.000 2.000 - -96.15% -
433.milc (max) 16.67% 12.000 14.000 - -87.04% -
433.milc (max) (percentage) 9.01% 1.110 1.210 - 51.25% -
433.milc (unexplained diff vs previous) 7.69% 13.000 14.000 - 7.69% -
433.milc (unexplained diff vs previous) (percentage) 6.25% 1.120 1.190 - 1090.00% -
433.milc (avg) (percentage) 1.33% 0.750 0.760 - -11.63% -

Performance Improvements - Code Size Δ Previous Current σ Δ (B) σ (B)
450.soplex -2.43% 150080.000 146428.000 - -0.23% -
471.omnetpp -1.80% 323274.000 317458.000 - 0.25% -
447.dealII -1.00% 193395.000 191458.000 - -2.21% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
450.soplex -2.71% 554.000 539.000 - -96.43% -
471.omnetpp -1.48% 2843.000 2801.000 - -73.27% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
456.hmmer (unexplained diff vs previous) -85.00% 20.000 3.000 - -93.48% -
456.hmmer (unexplained diff vs previous) (percentage) -84.62% 0.520 0.080 - -79.49% -
464.h264ref (unexplained diff vs previous) (percentage) -45.45% 0.330 0.180 - 0.00% -
464.h264ref (unexplained diff vs previous) -45.00% 20.000 11.000 - 0.00% -
444.namd (max) -33.33% 3.000 2.000 - -99.80% -
464.h264ref (max) (percentage) -24.24% 0.330 0.250 - -40.48% -
401.bzip2 (max) -22.22% 9.000 7.000 - -98.85% -
464.h264ref (max) -21.05% 19.000 15.000 - -75.00% -
456.hmmer (max) -20.00% 20.000 16.000 - -98.71% -
456.hmmer (max) (percentage) -16.98% 0.530 0.440 - -95.80% -
401.bzip2 (max) (percentage) -12.50% 0.320 0.280 - -92.59% -
473.astar (max) -10.00% 20.000 18.000 - -97.81% -
462.libquantum (max) (percentage) -10.00% 1.400 1.260 - -80.25% -
482.sphinx3 (max) (percentage) -10.00% 0.300 0.270 - -91.84% -
429.mcf (max) -8.70% 23.000 21.000 - -99.08% -
470.lbm (max) -8.70% 23.000 21.000 - -97.23% -
470.lbm (unexplained diff vs previous) -8.33% 24.000 22.000 - -97.26% -
470.lbm (max) (percentage) -8.33% 0.600 0.550 - -90.38% -
470.lbm (unexplained diff vs previous) (percentage) -8.33% 0.600 0.550 - -90.92% -
444.namd (max) (percentage) -7.89% 0.380 0.350 - -95.67% -
445.gobmk (max) -7.14% 28.000 26.000 - -93.70% -
445.gobmk (max) (percentage) -6.90% 0.580 0.540 - -85.52% -
473.astar (max) (percentage) -6.82% 0.440 0.410 - -93.88% -
429.mcf (max) (percentage) -6.31% 2.060 1.930 - -88.17% -
473.astar -4.55% 22.000 21.000 - -81.25% -
473.astar (avg) -4.55% 22.000 21.000 - -81.25% -
450.soplex (max) (percentage) -4.26% 0.470 0.450 - -72.73% -
453.povray (max) (percentage) -3.73% 1.340 1.290 - 258.33% -
403.gcc (max) (percentage) -3.16% 5.070 4.910 - 110.73% -
471.omnetpp -2.86% 35.000 34.000 - -79.64% -
471.omnetpp (avg) -2.86% 35.000 34.000 - -79.64% -
447.dealII (max) (percentage) -2.33% 0.860 0.840 - -64.71% -
473.astar (avg) (percentage) -2.08% 0.480 0.470 - -48.91% -
456.hmmer (avg) (percentage) -1.67% 0.600 0.590 - -57.55% -

Added Tests - sample variability
403.gcc (unexplained diff vs previous)
403.gcc (unexplained diff vs previous) (percentage)

Run-Over-Baseline Changes Detail

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
403.gcc (unexplained diff vs previous) (percentage) 2725.00% 0.120 3.390 - 0.00% -
433.milc (unexplained diff vs previous) (percentage) 1090.00% 0.100 1.190 - 6.25% -
403.gcc (avg) (percentage) 536.36% 0.550 3.500 - 0.00% -
453.povray (max) (percentage) 258.33% 0.360 1.290 - -3.73% -
450.soplex (avg) (percentage) 164.00% 0.500 1.320 - 0.00% -
403.gcc (max) (percentage) 110.73% 2.330 4.910 - -3.16% -
462.libquantum (avg) (percentage) 100.00% 0.770 1.540 - 0.00% -
471.omnetpp (unexplained diff vs previous) (percentage) 53.06% 0.980 1.500 - 3650.00% -
433.milc (max) (percentage) 51.25% 0.800 1.210 - 9.01% -
453.povray (avg) (percentage) 23.75% 0.800 0.990 - 0.00% -
433.milc (unexplained diff vs previous) 7.69% 13.000 14.000 - 7.69% -

Performance Regressions - Score Δ (B) Baseline Current σ (B) Δ σ
nb_execution_successful -5.56% 18.000 17.000 - 0.00% -
nb_compile_successful -5.26% 19.000 18.000 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
429.mcf -4.48% 7239.000 6915.000 - -0.17% -
462.libquantum -2.95% 12504.000 12135.000 - 0.66% -
433.milc -2.43% 47001.000 45861.000 - -0.17% -
453.povray -2.34% 524493.000 512222.000 - 0.03% -
447.dealII -2.21% 195792.000 191458.000 - -1.00% -
444.namd -2.02% 124963.000 122443.000 - -0.01% -
464.h264ref -1.93% 303767.000 297909.000 - 0.02% -
401.bzip2 -1.61% 33698.000 33157.000 - 0.07% -
458.sjeng -1.50% 80297.000 79091.000 - 0.13% -
456.hmmer -1.34% 80861.000 79777.000 - -0.12% -
470.lbm -1.22% 9338.000 9224.000 - 0.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.38% 9509.000 59.000 - 3.51% -
462.libquantum -98.71% 14409.000 186.000 - 1.64% -
450.soplex -96.43% 15112.000 539.000 - -2.71% -
482.sphinx3 -96.30% 26682.000 986.000 - -0.40% -
444.namd -93.50% 12400.000 806.000 - -0.25% -
453.povray -92.77% 6987.000 505.000 - 1.00% -
429.mcf -91.91% 13988.000 1131.000 - -0.44% -
433.milc -91.36% 13601.000 1175.000 - 1.21% -
401.bzip2 -82.43% 16137.000 2836.000 - 0.75% -
447.dealII -79.14% 9380.000 1957.000 - -0.05% -
471.omnetpp -73.27% 10477.000 2801.000 - -1.48% -
470.lbm -69.91% 13259.000 3990.000 - 0.55% -
456.hmmer -67.78% 11854.000 3819.000 - 0.08% -
473.astar -62.59% 12272.000 4591.000 - 0.13% -
464.h264ref -58.42% 14498.000 6028.000 - -0.18% -
445.gobmk -56.26% 11093.000 4852.000 - -0.68% -
458.sjeng -52.15% 17055.000 8160.000 - 0.25% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -99.80% 1003.000 2.000 - -33.33% -
462.libquantum (max) -99.78% 919.000 2.000 - 0.00% -
482.sphinx3 (max) -99.77% 883.000 2.000 - 0.00% -
483.xalancbmk (max) (percentage) -99.61% 12.900 0.050 - 0.00% -
450.soplex (max) -99.20% 249.000 2.000 - 0.00% -
403.gcc (max) -99.10% 221.000 2.000 - 0.00% -
429.mcf (max) -99.08% 2281.000 21.000 - -8.70% -
401.bzip2 (max) -98.85% 609.000 7.000 - -22.22% -
456.hmmer (max) -98.71% 1242.000 16.000 - -20.00% -
482.sphinx3 -98.49% 464.000 7.000 - 0.00% -
482.sphinx3 (avg) -98.49% 464.000 7.000 - 0.00% -
444.namd -98.43% 318.000 5.000 - 0.00% -
444.namd (avg) -98.43% 318.000 5.000 - 0.00% -
462.libquantum -98.18% 110.000 2.000 - 0.00% -
462.libquantum (avg) -98.18% 110.000 2.000 - 0.00% -
458.sjeng (max) -98.17% 876.000 16.000 - 0.00% -
473.astar (max) -97.81% 822.000 18.000 - -10.00% -
470.lbm (unexplained diff vs previous) -97.26% 804.000 22.000 - -8.33% -
470.lbm (max) -97.23% 758.000 21.000 - -8.70% -
403.gcc -96.15% 52.000 2.000 - 100.00% -
403.gcc (avg) -96.15% 52.000 2.000 - 100.00% -
458.sjeng (max) (percentage) -96.11% 5.140 0.200 - 0.00% -
456.hmmer (max) (percentage) -95.80% 10.480 0.440 - -16.98% -
471.omnetpp (max) -95.78% 972.000 41.000 - 192.86% -
444.namd (max) (percentage) -95.67% 8.090 0.350 - -7.89% -
429.mcf -95.27% 296.000 14.000 - 0.00% -
429.mcf (avg) -95.27% 296.000 14.000 - 0.00% -
470.lbm -95.01% 441.000 22.000 - 0.00% -
470.lbm (avg) -95.01% 441.000 22.000 - 0.00% -
473.astar (max) (percentage) -93.88% 6.700 0.410 - -6.82% -
445.gobmk (max) -93.70% 413.000 26.000 - -7.14% -
456.hmmer (unexplained diff vs previous) -93.48% 46.000 3.000 - -85.00% -
433.milc -93.10% 116.000 8.000 - 0.00% -
433.milc (avg) -93.10% 116.000 8.000 - 0.00% -
447.dealII (max) -92.83% 223.000 16.000 - 0.00% -
453.povray -92.73% 55.000 4.000 - 0.00% -
453.povray (avg) -92.73% 55.000 4.000 - 0.00% -
401.bzip2 (max) (percentage) -92.59% 3.780 0.280 - -12.50% -
482.sphinx3 (max) (percentage) -91.84% 3.310 0.270 - -10.00% -
470.lbm (unexplained diff vs previous) (percentage) -90.92% 6.060 0.550 - -8.33% -
450.soplex -90.67% 75.000 7.000 - 0.00% -
450.soplex (avg) -90.67% 75.000 7.000 - 0.00% -
470.lbm (max) (percentage) -90.38% 5.720 0.550 - -8.33% -
401.bzip2 -88.46% 104.000 12.000 - 0.00% -
401.bzip2 (avg) -88.46% 104.000 12.000 - 0.00% -
429.mcf (max) (percentage) -88.17% 16.310 1.930 - -6.31% -
433.milc (max) -87.04% 108.000 14.000 - 16.67% -
456.hmmer -86.59% 164.000 22.000 - 0.00% -
456.hmmer (avg) -86.59% 164.000 22.000 - 0.00% -
447.dealII -85.71% 98.000 14.000 - 0.00% -
447.dealII (avg) -85.71% 98.000 14.000 - 0.00% -
445.gobmk (max) (percentage) -85.52% 3.730 0.540 - -6.90% -
471.omnetpp (max) (percentage) -84.05% 9.280 1.480 - 196.00% -
445.gobmk -83.78% 74.000 12.000 - 0.00% -
445.gobmk (avg) -83.78% 74.000 12.000 - 0.00% -
470.lbm (avg) (percentage) -82.88% 3.330 0.570 - 0.00% -
458.sjeng -82.35% 136.000 24.000 - 0.00% -
458.sjeng (avg) -82.35% 136.000 24.000 - 0.00% -
403.gcc (unexplained diff vs previous) -81.82% 11.000 2.000 - 0.00% -
473.astar -81.25% 112.000 21.000 - -4.55% -
473.astar (avg) -81.25% 112.000 21.000 - -4.55% -
462.libquantum (max) (percentage) -80.25% 6.380 1.260 - -10.00% -
471.omnetpp -79.64% 167.000 34.000 - -2.86% -
471.omnetpp (avg) -79.64% 167.000 34.000 - -2.86% -
456.hmmer (unexplained diff vs previous) (percentage) -79.49% 0.390 0.080 - -84.62% -
453.povray (max) -76.00% 25.000 6.000 - 0.00% -
473.astar (unexplained diff vs previous) -76.00% 25.000 6.000 - 500.00% -
464.h264ref (max) -75.00% 60.000 15.000 - -21.05% -
444.namd (avg) (percentage) -73.93% 2.570 0.670 - 0.00% -
450.soplex (max) (percentage) -72.73% 1.650 0.450 - -4.26% -
464.h264ref -71.43% 42.000 12.000 - 0.00% -
464.h264ref (avg) -71.43% 42.000 12.000 - 0.00% -
447.dealII (max) (percentage) -64.71% 2.380 0.840 - -2.33% -
458.sjeng (avg) (percentage) -62.50% 0.800 0.300 - 0.00% -
445.gobmk (avg) (percentage) -61.19% 0.670 0.260 - 0.00% -
471.omnetpp (unexplained diff vs previous) -59.22% 103.000 42.000 - 4100.00% -
456.hmmer (avg) (percentage) -57.55% 1.390 0.590 - -1.67% -
482.sphinx3 (avg) (percentage) -55.17% 1.740 0.780 - 0.00% -
473.astar (avg) (percentage) -48.91% 0.920 0.470 - -2.08% -
464.h264ref (max) (percentage) -40.48% 0.420 0.250 - -24.24% -
429.mcf (avg) (percentage) -39.62% 2.120 1.280 - 0.00% -
473.astar (unexplained diff vs previous) (percentage) -35.00% 0.200 0.130 - 550.00% -
483.xalancbmk (avg) (percentage) -32.08% 2.120 1.440 - 0.00% -
401.bzip2 (avg) (percentage) -30.77% 0.650 0.450 - 0.00% -
447.dealII (avg) (percentage) -28.57% 1.050 0.750 - 0.00% -
464.h264ref (avg) (percentage) -27.59% 0.290 0.210 - 0.00% -
471.omnetpp (avg) (percentage) -22.50% 1.600 1.240 - 0.00% -
433.milc (avg) (percentage) -11.63% 0.860 0.760 - 1.33% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 5.26% 19.000 20.000 - 0.00% -

Added Tests - sample variability
464.h264ref (unexplained diff vs previous)
464.h264ref (unexplained diff vs previous) (percentage)

Report Time: 0.35s
Execution (seconds)
Name Current %
401.bzip2 2836.000 -
403.gcc 59.000 3.51%
429.mcf 1131.000 -
433.milc 1175.000 1.21%
444.namd 806.000 -
445.gobmk 4852.000 -
447.dealII 1957.000 -
450.soplex 539.000 -2.71%
453.povray 505.000 1.00%
456.hmmer 3819.000 -
458.sjeng 8160.000 -
462.libquantum 186.000 1.64%
464.h264ref 6028.000 -
470.lbm 3990.000 -
471.omnetpp 2801.000 -1.48%
473.astar 4591.000 -
482.sphinx3 986.000 -
Geometric Mean 1483.760 -
Code Size (bytes)
Name Current %
401.bzip2 33157.000 -
403.gcc 1653468.000 -
429.mcf 6915.000 -
433.milc 45861.000 -
444.namd 122443.000 -
445.gobmk 1255977.000 -
447.dealII 191458.000 -1.00%
450.soplex 146428.000 -2.43%
453.povray 512222.000 -
456.hmmer 79777.000 -
458.sjeng 79091.000 -
462.libquantum 12135.000 -
464.h264ref 297909.000 -
470.lbm 9224.000 -
471.omnetpp 317458.000 -1.80%
473.astar 15583.000 -
482.sphinx3 87408.000 -
Geometric Mean 94000.519 -
sample variability
Name Current %
401.bzip2 12.000 -
401.bzip2 (avg) 12.000 -
401.bzip2 (avg) (percentage) 0.450 -
401.bzip2 (max) 7.000 -22.22%
401.bzip2 (max) (percentage) 0.280 -12.50%
403.gcc 2.000 100.00%
403.gcc (avg) 2.000 100.00%
403.gcc (avg) (percentage) 3.500 -
403.gcc (max) 2.000 -
403.gcc (max) (percentage) 4.910 -3.16%
403.gcc (unexplained diff vs previous) 2.000 -
403.gcc (unexplained diff vs previous) (percentage) 3.390 -
429.mcf 14.000 -
429.mcf (avg) 14.000 -
429.mcf (avg) (percentage) 1.280 -
429.mcf (max) 21.000 -8.70%
429.mcf (max) (percentage) 1.930 -6.31%
433.milc 8.000 -
433.milc (avg) 8.000 -
433.milc (avg) (percentage) 0.760 1.33%
433.milc (max) 14.000 16.67%
433.milc (max) (percentage) 1.210 9.01%
433.milc (unexplained diff vs previous) 14.000 7.69%
433.milc (unexplained diff vs previous) (percentage) 1.190 6.25%
444.namd 5.000 -
444.namd (avg) 5.000 -
444.namd (avg) (percentage) 0.670 -
444.namd (max) 2.000 -33.33%
444.namd (max) (percentage) 0.350 -7.89%
445.gobmk 12.000 -
445.gobmk (avg) 12.000 -
445.gobmk (avg) (percentage) 0.260 -
445.gobmk (max) 26.000 -7.14%
445.gobmk (max) (percentage) 0.540 -6.90%
447.dealII 14.000 -
447.dealII (avg) 14.000 -
447.dealII (avg) (percentage) 0.750 -
447.dealII (max) 16.000 -
447.dealII (max) (percentage) 0.840 -2.33%
450.soplex 7.000 -
450.soplex (avg) 7.000 -
450.soplex (avg) (percentage) 1.320 -
450.soplex (max) 2.000 -
450.soplex (max) (percentage) 0.450 -4.26%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.990 -
453.povray (max) 6.000 -
453.povray (max) (percentage) 1.290 -3.73%
456.hmmer 22.000 -
456.hmmer (avg) 22.000 -
456.hmmer (avg) (percentage) 0.590 -1.67%
456.hmmer (max) 16.000 -20.00%
456.hmmer (max) (percentage) 0.440 -16.98%
456.hmmer (unexplained diff vs previous) 3.000 -85.00%
456.hmmer (unexplained diff vs previous) (percentage) 0.080 -84.62%
458.sjeng 24.000 -
458.sjeng (avg) 24.000 -
458.sjeng (avg) (percentage) 0.300 -
458.sjeng (max) 16.000 -
458.sjeng (max) (percentage) 0.200 -
462.libquantum 2.000 -
462.libquantum (avg) 2.000 -
462.libquantum (avg) (percentage) 1.540 -
462.libquantum (max) 2.000 -
462.libquantum (max) (percentage) 1.260 -10.00%
464.h264ref 12.000 -
464.h264ref (avg) 12.000 -
464.h264ref (avg) (percentage) 0.210 -
464.h264ref (max) 15.000 -21.05%
464.h264ref (max) (percentage) 0.250 -24.24%
464.h264ref (unexplained diff vs previous) 11.000 -45.00%
464.h264ref (unexplained diff vs previous) (percentage) 0.180 -45.45%
470.lbm 22.000 -
470.lbm (avg) 22.000 -
470.lbm (avg) (percentage) 0.570 -
470.lbm (max) 21.000 -8.70%
470.lbm (max) (percentage) 0.550 -8.33%
470.lbm (unexplained diff vs previous) 22.000 -8.33%
470.lbm (unexplained diff vs previous) (percentage) 0.550 -8.33%
471.omnetpp 34.000 -2.86%
471.omnetpp (avg) 34.000 -2.86%
471.omnetpp (avg) (percentage) 1.240 -
471.omnetpp (max) 41.000 192.86%
471.omnetpp (max) (percentage) 1.480 196.00%
471.omnetpp (unexplained diff vs previous) 42.000 4100.00%
471.omnetpp (unexplained diff vs previous) (percentage) 1.500 3650.00%
473.astar 21.000 -4.55%
473.astar (avg) 21.000 -4.55%
473.astar (avg) (percentage) 0.470 -2.08%
473.astar (max) 18.000 -10.00%
473.astar (max) (percentage) 0.410 -6.82%
473.astar (unexplained diff vs previous) 6.000 500.00%
473.astar (unexplained diff vs previous) (percentage) 0.130 550.00%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.780 -
482.sphinx3 (max) 2.000 -
482.sphinx3 (max) (percentage) 0.270 -10.00%
483.xalancbmk (avg) (percentage) 1.440 -
483.xalancbmk (max) (percentage) 0.050 -
Geometric Mean 2.967 6.85%