Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/e439fdf4ea0dbc6f001428f4d4956700bf26bb97 (+ 390 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz_LTO-build/174/
View Options
Report
Filter

llvm-arm-master-Oz_LTO test results

Run Order Start Time Duration
Current 0174 (llvmorg-20-init-03486-ge439fdf4ea0d) 2024-08-19T20:26:11 0:00:00
Previous 0173 (llvmorg-20-init-03096-g5ec73b7dcf7f) 2024-08-15T15:47:52 0:00:00
Baseline 0001 (llvmorg-16-init-17036-g4f40923103d2) 2023-01-06T17:55:45 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 19 12
Performance Improvements 29 114
Added Tests 4 6
Unchanged Tests 716 636
Total Tests 768

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
462.libquantum 2.79% 179.000 184.000 - -98.72% -
444.namd 2.13% 799.000 816.000 - -93.42% -
401.bzip2 1.03% 2811.000 2840.000 - -82.40% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) (percentage) 1266.67% 0.030 0.410 0.000 -58.16% 0.000
471.omnetpp (unexplained diff vs previous) 1100.00% 1.000 12.000 0.000 -88.35% 0.000
470.lbm (unexplained diff vs previous) 600.00% 3.000 21.000 0.000 -97.39% 0.000
458.sjeng (max) 562.50% 8.000 53.000 - -93.95% -
458.sjeng (max) (percentage) 560.00% 0.100 0.660 - -87.16% -
470.lbm (unexplained diff vs previous) (percentage) 550.00% 0.080 0.520 0.000 -91.42% 0.000
401.bzip2 (unexplained diff vs previous) 480.00% 5.000 29.000 0.000 -12.12% 0.000
401.bzip2 (unexplained diff vs previous) (percentage) 466.67% 0.180 1.020 0.000 410.00% 0.000
401.bzip2 (max) 123.08% 13.000 29.000 - -95.24% -
401.bzip2 (max) (percentage) 119.15% 0.470 1.030 - -72.75% -
470.lbm (max) 16.67% 18.000 21.000 - -97.23% -
470.lbm (max) (percentage) 15.22% 0.460 0.530 - -90.73% -
458.sjeng 3.85% 26.000 27.000 - -80.15% -
458.sjeng (avg) 3.85% 26.000 27.000 - -80.15% -
458.sjeng (avg) (percentage) 3.03% 0.330 0.340 - -57.50% -
401.bzip2 (avg) (percentage) 2.33% 0.430 0.440 - -32.31% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -1.67% 60.000 59.000 - -99.38% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
447.dealII (unexplained diff vs previous) (percentage) -48.46% 1.300 0.670 0.000 15.52% 0.000
447.dealII (unexplained diff vs previous) -48.00% 25.000 13.000 0.000 -75.93% 0.000
433.milc (max) -12.50% 16.000 14.000 - -87.04% -
456.hmmer (max) -12.50% 8.000 7.000 - -99.44% -
444.namd (max) (percentage) -10.81% 0.370 0.330 - -95.92% -
445.gobmk (max) -10.00% 10.000 9.000 - -97.82% -
429.mcf (max) (percentage) -9.30% 0.430 0.390 - -97.61% -
447.dealII (max) (percentage) -8.89% 1.350 1.230 - -48.32% -
471.omnetpp (max) -8.33% 12.000 11.000 - -98.87% -
473.astar (max) -8.33% 24.000 22.000 - -97.32% -
447.dealII (max) -8.00% 25.000 23.000 - -89.69% -
482.sphinx3 (max) -7.69% 13.000 12.000 - -98.64% -
429.mcf -7.14% 14.000 13.000 - -95.61% -
429.mcf (avg) -7.14% 14.000 13.000 - -95.61% -
471.omnetpp (max) (percentage) -6.82% 0.440 0.410 - -95.58% -
450.soplex (max) -6.67% 15.000 14.000 - -94.38% -
433.milc (max) (percentage) -6.57% 1.370 1.280 - 60.00% -
473.astar (max) (percentage) -5.77% 0.520 0.490 - -92.69% -
450.soplex (max) (percentage) -5.69% 2.810 2.650 - 60.61% -
462.libquantum (max) (percentage) -4.69% 1.920 1.830 - -71.32% -
473.astar -4.00% 25.000 24.000 - -78.57% -
473.astar (avg) -4.00% 25.000 24.000 - -78.57% -
482.sphinx3 (max) (percentage) -3.68% 1.360 1.310 - -60.42% -
453.povray (max) (percentage) -2.59% 1.160 1.130 - 213.89% -
429.mcf (avg) (percentage) -2.38% 1.260 1.230 - -41.98% -
471.omnetpp -2.38% 42.000 41.000 - -75.45% -
471.omnetpp (avg) -2.38% 42.000 41.000 - -75.45% -
471.omnetpp (avg) (percentage) -1.39% 1.440 1.420 - -11.25% -

Added Tests - sample variability
429.mcf (unexplained diff vs previous)
429.mcf (unexplained diff vs previous) (percentage)
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.89% 146765.000 149544.000 - 0.00% -
471.omnetpp 1.61% 316676.000 321790.000 - -0.00% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
401.bzip2 (unexplained diff vs previous) (percentage) 410.00% 0.200 1.020 0.000 466.67% 0.000
403.gcc (avg) (percentage) 372.73% 0.550 2.600 - 0.00% -
453.povray (max) (percentage) 213.89% 0.360 1.130 - -2.59% -
450.soplex (avg) (percentage) 154.00% 0.500 1.270 - 0.00% -
403.gcc (max) (percentage) 130.04% 2.330 5.360 - -0.56% -
462.libquantum (avg) (percentage) 100.00% 0.770 1.540 - 0.00% -
450.soplex (max) (percentage) 60.61% 1.650 2.650 - -5.69% -
433.milc (max) (percentage) 60.00% 0.800 1.280 - -6.57% -
453.povray (avg) (percentage) 18.75% 0.800 0.950 - 0.00% -
447.dealII (unexplained diff vs previous) (percentage) 15.52% 0.580 0.670 0.000 -48.46% 0.000

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
429.mcf -3.87% 7239.000 6959.000 - 0.00% -
462.libquantum -3.59% 12504.000 12055.000 - -0.33% -
453.povray -2.19% 524493.000 513024.000 - -0.03% -
444.namd -1.86% 124963.000 122638.000 - -0.29% -
433.milc -1.84% 47001.000 46137.000 - -0.02% -
401.bzip2 -1.76% 33698.000 33105.000 - 0.00% -
458.sjeng -1.52% 80297.000 79078.000 - 0.00% -
464.h264ref -1.46% 303767.000 299337.000 - 0.01% -
470.lbm -1.14% 9338.000 9232.000 - 0.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.38% 9509.000 59.000 - -1.67% -
462.libquantum -98.72% 14409.000 184.000 - 2.79% -
450.soplex -96.44% 15112.000 538.000 - -0.19% -
482.sphinx3 -96.30% 26682.000 988.000 - -0.10% -
444.namd -93.42% 12400.000 816.000 - 2.13% -
453.povray -92.87% 6987.000 498.000 - 0.20% -
429.mcf -91.86% 13988.000 1138.000 - 0.00% -
433.milc -91.39% 13601.000 1171.000 - -0.76% -
401.bzip2 -82.40% 16137.000 2840.000 - 1.03% -
447.dealII -79.43% 9380.000 1929.000 - 0.68% -
471.omnetpp -72.15% 10477.000 2918.000 - -0.41% -
470.lbm -69.79% 13259.000 4005.000 - 0.53% -
456.hmmer -67.87% 11854.000 3809.000 - -0.10% -
473.astar -62.31% 12272.000 4625.000 - -0.45% -
464.h264ref -58.97% 14498.000 5949.000 - 0.12% -
445.gobmk -56.25% 11093.000 4853.000 - -0.06% -
458.sjeng -52.55% 17055.000 8092.000 - -0.66% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
429.mcf (max) -99.82% 2281.000 4.000 - 0.00% -
444.namd (max) -99.80% 1003.000 2.000 - 0.00% -
462.libquantum (max) -99.67% 919.000 3.000 - 0.00% -
483.xalancbmk (max) (percentage) -99.61% 12.900 0.050 - 0.00% -
456.hmmer (max) -99.44% 1242.000 7.000 - -12.50% -
471.omnetpp (max) -98.87% 972.000 11.000 - -8.33% -
403.gcc (max) -98.64% 221.000 3.000 - 0.00% -
482.sphinx3 (max) -98.64% 883.000 12.000 - -7.69% -
482.sphinx3 -98.49% 464.000 7.000 - 0.00% -
482.sphinx3 (avg) -98.49% 464.000 7.000 - 0.00% -
462.libquantum -98.18% 110.000 2.000 - 0.00% -
462.libquantum (avg) -98.18% 110.000 2.000 - 0.00% -
444.namd -98.11% 318.000 6.000 - 0.00% -
444.namd (avg) -98.11% 318.000 6.000 - 0.00% -
456.hmmer (max) (percentage) -98.09% 10.480 0.200 - -4.76% -
403.gcc -98.08% 52.000 1.000 - 0.00% -
403.gcc (avg) -98.08% 52.000 1.000 - 0.00% -
445.gobmk (max) -97.82% 413.000 9.000 - -10.00% -
429.mcf (max) (percentage) -97.61% 16.310 0.390 - -9.30% -
470.lbm (unexplained diff vs previous) -97.39% 804.000 21.000 0.000 600.00% 0.000
473.astar (max) -97.32% 822.000 22.000 - -8.33% -
470.lbm (max) -97.23% 758.000 21.000 - 16.67% -
444.namd (max) (percentage) -95.92% 8.090 0.330 - -10.81% -
429.mcf -95.61% 296.000 13.000 - -7.14% -
429.mcf (avg) -95.61% 296.000 13.000 - -7.14% -
471.omnetpp (max) (percentage) -95.58% 9.280 0.410 - -6.82% -
401.bzip2 (max) -95.24% 609.000 29.000 - 123.08% -
470.lbm -95.01% 441.000 22.000 - 0.00% -
470.lbm (avg) -95.01% 441.000 22.000 - 0.00% -
445.gobmk (max) (percentage) -94.64% 3.730 0.200 - -4.76% -
450.soplex (max) -94.38% 249.000 14.000 - -6.67% -
433.milc -93.97% 116.000 7.000 - 0.00% -
433.milc (avg) -93.97% 116.000 7.000 - 0.00% -
458.sjeng (max) -93.95% 876.000 53.000 - 562.50% -
453.povray -92.73% 55.000 4.000 - 0.00% -
453.povray (avg) -92.73% 55.000 4.000 - 0.00% -
473.astar (max) (percentage) -92.69% 6.700 0.490 - -5.77% -
450.soplex -92.00% 75.000 6.000 - 0.00% -
450.soplex (avg) -92.00% 75.000 6.000 - 0.00% -
470.lbm (unexplained diff vs previous) (percentage) -91.42% 6.060 0.520 0.000 550.00% 0.000
470.lbm (max) (percentage) -90.73% 5.720 0.530 - 15.22% -
447.dealII (max) -89.69% 223.000 23.000 - -8.00% -
401.bzip2 -88.46% 104.000 12.000 - 0.00% -
401.bzip2 (avg) -88.46% 104.000 12.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) -88.35% 103.000 12.000 0.000 1100.00% 0.000
458.sjeng (max) (percentage) -87.16% 5.140 0.660 - 560.00% -
433.milc (max) -87.04% 108.000 14.000 - -12.50% -
464.h264ref (max) -86.67% 60.000 8.000 - 0.00% -
447.dealII -85.71% 98.000 14.000 - 0.00% -
447.dealII (avg) -85.71% 98.000 14.000 - 0.00% -
470.lbm (avg) (percentage) -83.18% 3.330 0.560 - 0.00% -
445.gobmk -81.08% 74.000 14.000 - 0.00% -
445.gobmk (avg) -81.08% 74.000 14.000 - 0.00% -
456.hmmer -80.49% 164.000 32.000 - 0.00% -
456.hmmer (avg) -80.49% 164.000 32.000 - 0.00% -
458.sjeng -80.15% 136.000 27.000 - 3.85% -
458.sjeng (avg) -80.15% 136.000 27.000 - 3.85% -
453.povray (max) -80.00% 25.000 5.000 - 0.00% -
473.astar -78.57% 112.000 24.000 - -4.00% -
473.astar (avg) -78.57% 112.000 24.000 - -4.00% -
447.dealII (unexplained diff vs previous) -75.93% 54.000 13.000 0.000 -48.00% 0.000
471.omnetpp -75.45% 167.000 41.000 - -2.38% -
471.omnetpp (avg) -75.45% 167.000 41.000 - -2.38% -
401.bzip2 (max) (percentage) -72.75% 3.780 1.030 - 119.15% -
462.libquantum (max) (percentage) -71.32% 6.380 1.830 - -4.69% -
464.h264ref -69.05% 42.000 13.000 - 0.00% -
464.h264ref (avg) -69.05% 42.000 13.000 - 0.00% -
444.namd (avg) (percentage) -66.93% 2.570 0.850 - 0.00% -
464.h264ref (max) (percentage) -66.67% 0.420 0.140 - 0.00% -
482.sphinx3 (max) (percentage) -60.42% 3.310 1.310 - -3.68% -
471.omnetpp (unexplained diff vs previous) (percentage) -58.16% 0.980 0.410 0.000 1266.67% 0.000
458.sjeng (avg) (percentage) -57.50% 0.800 0.340 - 3.03% -
445.gobmk (avg) (percentage) -56.72% 0.670 0.290 - 0.00% -
482.sphinx3 (avg) (percentage) -55.17% 1.740 0.780 - 0.00% -
447.dealII (max) (percentage) -48.32% 2.380 1.230 - -8.89% -
429.mcf (avg) (percentage) -41.98% 2.120 1.230 - -2.38% -
473.astar (avg) (percentage) -41.30% 0.920 0.540 - 0.00% -
456.hmmer (avg) (percentage) -38.13% 1.390 0.860 - 0.00% -
401.bzip2 (avg) (percentage) -32.31% 0.650 0.440 - 2.33% -
483.xalancbmk (avg) (percentage) -32.08% 2.120 1.440 - 0.00% -
433.milc (avg) (percentage) -27.91% 0.860 0.620 - 0.00% -
447.dealII (avg) (percentage) -27.62% 1.050 0.760 - 0.00% -
464.h264ref (avg) (percentage) -24.14% 0.290 0.220 - 0.00% -
401.bzip2 (unexplained diff vs previous) -12.12% 33.000 29.000 0.000 480.00% 0.000
471.omnetpp (avg) (percentage) -11.25% 1.600 1.420 - -1.39% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 110.53% 19.000 40.000 - 0.00% -
nb_compile_successful 89.47% 19.000 36.000 - 0.00% -
nb_execution_successful 88.89% 18.000 34.000 - 0.00% -

Added Tests - sample variability
429.mcf (unexplained diff vs previous)
429.mcf (unexplained diff vs previous) (percentage)
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)
464.h264ref (unexplained diff vs previous)
464.h264ref (unexplained diff vs previous) (percentage)

Report Time: 0.35s
Execution (seconds)
Name Current %
401.bzip2 2840.000 1.03%
403.gcc 59.000 -1.67%
429.mcf 1138.000 -
433.milc 1171.000 -
444.namd 816.000 2.13%
445.gobmk 4853.000 -
447.dealII 1929.000 -
450.soplex 538.000 -
453.povray 498.000 -
456.hmmer 3809.000 -
458.sjeng 8092.000 -
462.libquantum 184.000 2.79%
464.h264ref 5949.000 -
470.lbm 4005.000 -
471.omnetpp 2918.000 -
473.astar 4625.000 -
482.sphinx3 988.000 -
Geometric Mean 1484.246 -
Code Size (bytes)
Name Current %
401.bzip2 33105.000 -
403.gcc 1655939.000 -
429.mcf 6959.000 -
433.milc 46137.000 -
444.namd 122638.000 -
445.gobmk 1255901.000 -
447.dealII 195814.000 -
450.soplex 149544.000 -
453.povray 513024.000 -
456.hmmer 80329.000 -
458.sjeng 79078.000 -
462.libquantum 12055.000 -
464.h264ref 299337.000 -
470.lbm 9232.000 -
471.omnetpp 321790.000 -
473.astar 15603.000 -
482.sphinx3 87484.000 -
Geometric Mean 94446.065 -
sample variability
Name Current %
401.bzip2 12.000 -
401.bzip2 (avg) 12.000 -
401.bzip2 (avg) (percentage) 0.440 2.33%
401.bzip2 (max) 29.000 123.08%
401.bzip2 (max) (percentage) 1.030 119.15%
401.bzip2 (unexplained diff vs previous) 29.000 480.00%
401.bzip2 (unexplained diff vs previous) (percentage) 1.020 466.67%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 2.600 -
403.gcc (max) 3.000 -
403.gcc (max) (percentage) 5.360 -
429.mcf 13.000 -7.14%
429.mcf (avg) 13.000 -7.14%
429.mcf (avg) (percentage) 1.230 -2.38%
429.mcf (max) 4.000 -
429.mcf (max) (percentage) 0.390 -9.30%
429.mcf (unexplained diff vs previous) 0.000 -
429.mcf (unexplained diff vs previous) (percentage) 0.000 -
433.milc 7.000 -
433.milc (avg) 7.000 -
433.milc (avg) (percentage) 0.620 -
433.milc (max) 14.000 -12.50%
433.milc (max) (percentage) 1.280 -6.57%
444.namd 6.000 -
444.namd (avg) 6.000 -
444.namd (avg) (percentage) 0.850 -
444.namd (max) 2.000 -
444.namd (max) (percentage) 0.330 -10.81%
445.gobmk 14.000 -
445.gobmk (avg) 14.000 -
445.gobmk (avg) (percentage) 0.290 -
445.gobmk (max) 9.000 -10.00%
445.gobmk (max) (percentage) 0.200 -
447.dealII 14.000 -
447.dealII (avg) 14.000 -
447.dealII (avg) (percentage) 0.760 -
447.dealII (max) 23.000 -8.00%
447.dealII (max) (percentage) 1.230 -8.89%
447.dealII (unexplained diff vs previous) 13.000 -48.00%
447.dealII (unexplained diff vs previous) (percentage) 0.670 -48.46%
450.soplex 6.000 -
450.soplex (avg) 6.000 -
450.soplex (avg) (percentage) 1.270 -
450.soplex (max) 14.000 -6.67%
450.soplex (max) (percentage) 2.650 -5.69%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.950 -
453.povray (max) 5.000 -
453.povray (max) (percentage) 1.130 -2.59%
456.hmmer 32.000 -
456.hmmer (avg) 32.000 -
456.hmmer (avg) (percentage) 0.860 -
456.hmmer (max) 7.000 -12.50%
456.hmmer (max) (percentage) 0.200 -
458.sjeng 27.000 3.85%
458.sjeng (avg) 27.000 3.85%
458.sjeng (avg) (percentage) 0.340 3.03%
458.sjeng (max) 53.000 562.50%
458.sjeng (max) (percentage) 0.660 560.00%
458.sjeng (unexplained diff vs previous) 54.000 -
458.sjeng (unexplained diff vs previous) (percentage) 0.670 -
462.libquantum 2.000 -
462.libquantum (avg) 2.000 -
462.libquantum (avg) (percentage) 1.540 -
462.libquantum (max) 3.000 -
462.libquantum (max) (percentage) 1.830 -4.69%
464.h264ref 13.000 -
464.h264ref (avg) 13.000 -
464.h264ref (avg) (percentage) 0.220 -
464.h264ref (max) 8.000 -
464.h264ref (max) (percentage) 0.140 -
464.h264ref (unexplained diff vs previous) 7.000 -
464.h264ref (unexplained diff vs previous) (percentage) 0.120 -
470.lbm 22.000 -
470.lbm (avg) 22.000 -
470.lbm (avg) (percentage) 0.560 -
470.lbm (max) 21.000 16.67%
470.lbm (max) (percentage) 0.530 15.22%
470.lbm (unexplained diff vs previous) 21.000 600.00%
470.lbm (unexplained diff vs previous) (percentage) 0.520 550.00%
471.omnetpp 41.000 -2.38%
471.omnetpp (avg) 41.000 -2.38%
471.omnetpp (avg) (percentage) 1.420 -1.39%
471.omnetpp (max) 11.000 -8.33%
471.omnetpp (max) (percentage) 0.410 -6.82%
471.omnetpp (unexplained diff vs previous) 12.000 1100.00%
471.omnetpp (unexplained diff vs previous) (percentage) 0.410 1266.67%
473.astar 24.000 -4.00%
473.astar (avg) 24.000 -4.00%
473.astar (avg) (percentage) 0.540 -
473.astar (max) 22.000 -8.33%
473.astar (max) (percentage) 0.490 -5.77%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.780 -
482.sphinx3 (max) 12.000 -7.69%
482.sphinx3 (max) (percentage) 1.310 -3.68%
483.xalancbmk (avg) (percentage) 1.440 -
483.xalancbmk (max) (percentage) 0.050 -
Geometric Mean 2.649 16.15%