Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/d31fb264821ceac90b55c87c3f777053ab30a9af (+ 1206 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz-build/208/
View Options
Report
Filter

llvm-arm-master-Oz test results

Run Order Start Time Duration
Current 0208 (llvmorg-20-init-14655-gd31fb264821c) 2024-11-30T18:13:09 0:00:00
Previous 0207 (no-change) 2024-11-23T05:28:53 0:00:00
Baseline 0014 (no-change) 2023-02-19T08:08:29 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 8 11
Performance Improvements 27 94
Added Tests 0 6
Unchanged Tests 733 657
Total Tests 768

Run-Over-Run Changes Detail

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) 23.53% 17.000 21.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) (percentage) 22.64% 0.530 0.650 - 0.00% -
471.omnetpp (max) (percentage) 20.37% 0.540 0.650 - -96.01% -
471.omnetpp (max) 17.65% 17.000 20.000 - -96.06% -
433.milc (unexplained diff vs previous) (percentage) 10.59% 0.850 0.940 - 0.00% -
433.milc (unexplained diff vs previous) 10.00% 10.000 11.000 - 0.00% -
433.milc (max) (percentage) 9.41% 0.850 0.930 - -92.98% -
433.milc (avg) (percentage) 1.39% 0.720 0.730 - -64.22% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -9.38% 64.000 58.000 - -3.33% -
450.soplex -1.98% 556.000 545.000 - -1.80% -
429.mcf -1.91% 1150.000 1128.000 - -2.34% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
403.gcc (max) -20.00% 5.000 4.000 - -33.33% -
447.dealII (max) -20.00% 5.000 4.000 - -96.52% -
473.astar (max) -16.67% 12.000 10.000 - -99.68% -
450.soplex -12.50% 8.000 7.000 - 133.33% -
450.soplex (avg) -12.50% 8.000 7.000 - 133.33% -
456.hmmer (max) -12.50% 16.000 14.000 - -95.65% -
464.h264ref (max) (percentage) -12.12% 0.330 0.290 - -94.01% -
473.astar (max) (percentage) -12.00% 0.250 0.220 - -99.65% -
429.mcf (max) -11.76% 17.000 15.000 - -93.06% -
444.namd (max) -11.11% 9.000 8.000 - -92.31% -
456.hmmer (max) (percentage) -10.87% 0.460 0.410 - -95.40% -
464.h264ref (max) -10.53% 19.000 17.000 - -94.18% -
429.mcf (max) (percentage) -10.00% 1.500 1.350 - -92.79% -
450.soplex (max) -8.33% 12.000 11.000 - 57.14% -
462.libquantum (max) (percentage) -8.22% 1.460 1.340 - -77.13% -
444.namd (max) (percentage) -6.14% 1.140 1.070 - -91.86% -
482.sphinx3 (max) (percentage) -5.94% 1.010 0.950 - -93.38% -
470.lbm (max) -5.88% 17.000 16.000 - -96.39% -
450.soplex (max) (percentage) -5.70% 2.280 2.150 - 61.65% -
470.lbm (max) (percentage) -4.55% 0.440 0.420 - -96.28% -
458.sjeng (max) -4.35% 46.000 44.000 - -95.05% -
458.sjeng (max) (percentage) -4.08% 0.490 0.470 - -94.91% -
401.bzip2 (max) (percentage) -1.79% 0.560 0.550 - -80.63% -
403.gcc (max) (percentage) -1.34% 8.220 8.110 - -30.21% -

Run-Over-Baseline Changes Detail

Performance Regressions - Execution Δ (B) Baseline Current σ (B) Δ σ
471.omnetpp 3.66% 3116.000 3230.000 - 0.65% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
450.soplex 133.33% 3.000 7.000 - -12.50% -
450.soplex (avg) 133.33% 3.000 7.000 - -12.50% -
450.soplex (avg) (percentage) 119.70% 0.660 1.450 - 0.00% -
403.gcc (avg) (percentage) 109.20% 1.740 3.640 - 0.00% -
403.gcc 100.00% 1.000 2.000 - 0.00% -
403.gcc (avg) 100.00% 1.000 2.000 - 0.00% -
450.soplex (max) (percentage) 61.65% 1.330 2.150 - -5.70% -
450.soplex (max) 57.14% 7.000 11.000 - -8.33% -

Performance Regressions - Score Δ (B) Baseline Current σ (B) Δ σ
nb_execution_successful -5.56% 18.000 17.000 - 0.00% -
nb_compile_successful -5.26% 19.000 18.000 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -5.82% 22066.000 20782.000 - 0.00% -
401.bzip2 -2.10% 35781.000 35029.000 - 0.11% -
453.povray -1.45% 549774.000 541811.000 - -0.05% -
444.namd -1.29% 154855.000 152863.000 - -0.01% -
464.h264ref -1.28% 334968.000 330678.000 - -0.07% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -3.33% 60.000 58.000 - -9.38% -
462.libquantum -3.26% 184.000 178.000 - -0.56% -
453.povray -2.46% 529.000 516.000 - 0.78% -
429.mcf -2.34% 1155.000 1128.000 - -1.91% -
401.bzip2 -1.82% 3014.000 2959.000 - -0.60% -
450.soplex -1.80% 555.000 545.000 - -1.98% -
445.gobmk -1.62% 5439.000 5351.000 - 0.07% -
458.sjeng -1.47% 9638.000 9496.000 - -0.02% -
464.h264ref -1.16% 6043.000 5973.000 - 0.47% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
473.astar (max) -99.68% 3144.000 10.000 - -16.67% -
473.astar (max) (percentage) -99.65% 63.460 0.220 - -12.00% -
483.xalancbmk (max) (percentage) -99.34% 47.200 0.310 - 0.00% -
447.dealII (max) -96.52% 115.000 4.000 - -20.00% -
470.lbm (max) -96.39% 443.000 16.000 - -5.88% -
470.lbm (max) (percentage) -96.28% 11.290 0.420 - -4.55% -
471.omnetpp (max) -96.06% 507.000 20.000 - 17.65% -
471.omnetpp (max) (percentage) -96.01% 16.280 0.650 - 20.37% -
447.dealII (max) (percentage) -95.80% 4.760 0.200 - -4.76% -
456.hmmer (max) -95.65% 322.000 14.000 - -12.50% -
456.hmmer (max) (percentage) -95.40% 8.910 0.410 - -10.87% -
458.sjeng (max) -95.05% 889.000 44.000 - -4.35% -
458.sjeng (max) (percentage) -94.91% 9.230 0.470 - -4.08% -
464.h264ref (max) -94.18% 292.000 17.000 - -10.53% -
464.h264ref (max) (percentage) -94.01% 4.840 0.290 - -12.12% -
482.sphinx3 (max) -93.66% 142.000 9.000 - 0.00% -
433.milc (max) -93.46% 153.000 10.000 - 0.00% -
445.gobmk (max) -93.41% 182.000 12.000 - 0.00% -
482.sphinx3 (max) (percentage) -93.38% 14.360 0.950 - -5.94% -
445.gobmk (max) (percentage) -93.13% 3.350 0.230 - -4.17% -
429.mcf (max) -93.06% 216.000 15.000 - -11.76% -
433.milc (max) (percentage) -92.98% 13.240 0.930 - 9.41% -
429.mcf (max) (percentage) -92.79% 18.730 1.350 - -10.00% -
444.namd (max) -92.31% 104.000 8.000 - -11.11% -
453.povray (max) -92.00% 25.000 2.000 - 0.00% -
444.namd (max) (percentage) -91.86% 13.140 1.070 - -6.14% -
453.povray (max) (percentage) -91.51% 4.830 0.410 - 0.00% -
473.astar -88.31% 248.000 29.000 - 0.00% -
473.astar (avg) -88.31% 248.000 29.000 - 0.00% -
473.astar (avg) (percentage) -88.25% 5.020 0.590 - 0.00% -
456.hmmer -87.02% 131.000 17.000 - 0.00% -
456.hmmer (avg) -87.02% 131.000 17.000 - 0.00% -
456.hmmer (avg) (percentage) -86.54% 3.640 0.490 - 0.00% -
470.lbm -86.54% 156.000 21.000 - 0.00% -
470.lbm (avg) -86.54% 156.000 21.000 - 0.00% -
470.lbm (avg) (percentage) -86.43% 3.980 0.540 - 0.00% -
464.h264ref -84.17% 120.000 19.000 - 0.00% -
464.h264ref (avg) -84.17% 120.000 19.000 - 0.00% -
464.h264ref (avg) (percentage) -84.00% 2.000 0.320 - 0.00% -
444.namd -82.35% 34.000 6.000 - 0.00% -
444.namd (avg) -82.35% 34.000 6.000 - 0.00% -
444.namd (avg) (percentage) -81.22% 4.260 0.800 - 0.00% -
401.bzip2 (max) -81.18% 85.000 16.000 - 0.00% -
401.bzip2 (max) (percentage) -80.63% 2.840 0.550 - -1.79% -
429.mcf -80.00% 55.000 11.000 - 0.00% -
429.mcf (avg) -80.00% 55.000 11.000 - 0.00% -
462.libquantum (max) -80.00% 10.000 2.000 - 0.00% -
482.sphinx3 -79.41% 34.000 7.000 - 0.00% -
482.sphinx3 (avg) -79.41% 34.000 7.000 - 0.00% -
482.sphinx3 (avg) (percentage) -78.61% 3.460 0.740 - 0.00% -
429.mcf (avg) (percentage) -78.20% 4.770 1.040 - 0.00% -
445.gobmk -77.46% 71.000 16.000 - 0.00% -
445.gobmk (avg) -77.46% 71.000 16.000 - 0.00% -
458.sjeng -77.36% 159.000 36.000 - 0.00% -
458.sjeng (avg) -77.36% 159.000 36.000 - 0.00% -
462.libquantum (max) (percentage) -77.13% 5.860 1.340 - -8.22% -
445.gobmk (avg) (percentage) -77.10% 1.310 0.300 - 0.00% -
458.sjeng (avg) (percentage) -76.97% 1.650 0.380 - 0.00% -
471.omnetpp (avg) (percentage) -69.49% 3.540 1.080 - 0.00% -
471.omnetpp -69.09% 110.000 34.000 - 0.00% -
471.omnetpp (avg) -69.09% 110.000 34.000 - 0.00% -
401.bzip2 -67.65% 34.000 11.000 - 0.00% -
401.bzip2 (avg) -67.65% 34.000 11.000 - 0.00% -
401.bzip2 (avg) (percentage) -67.24% 1.160 0.380 - 0.00% -
433.milc -65.22% 23.000 8.000 - 0.00% -
433.milc (avg) -65.22% 23.000 8.000 - 0.00% -
433.milc (avg) (percentage) -64.22% 2.040 0.730 - 1.39% -
483.xalancbmk (avg) (percentage) -62.26% 6.810 2.570 - 0.00% -
453.povray -57.14% 7.000 3.000 - 0.00% -
453.povray (avg) -57.14% 7.000 3.000 - 0.00% -
447.dealII -55.56% 27.000 12.000 - 0.00% -
447.dealII (avg) -55.56% 27.000 12.000 - 0.00% -
447.dealII (avg) (percentage) -53.51% 1.140 0.530 - 0.00% -
462.libquantum -50.00% 2.000 1.000 - 0.00% -
462.libquantum (avg) -50.00% 2.000 1.000 - 0.00% -
453.povray (avg) (percentage) -48.67% 1.500 0.770 - 0.00% -
462.libquantum (avg) (percentage) -43.12% 1.600 0.910 - 0.00% -
403.gcc (max) -33.33% 6.000 4.000 - -20.00% -
403.gcc (max) (percentage) -30.21% 11.620 8.110 - -1.34% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 5.26% 19.000 20.000 - 0.00% -

Added Tests - sample variability
433.milc (unexplained diff vs previous)
433.milc (unexplained diff vs previous) (percentage)
470.lbm (unexplained diff vs previous)
470.lbm (unexplained diff vs previous) (percentage)
471.omnetpp (unexplained diff vs previous)
471.omnetpp (unexplained diff vs previous) (percentage)

Report Time: 0.24s
Execution (seconds)
Name Current %
401.bzip2 2959.000 -
403.gcc 58.000 -9.38%
429.mcf 1128.000 -1.91%
433.milc 1172.000 -
444.namd 801.000 -
445.gobmk 5351.000 -
447.dealII 2437.000 -
450.soplex 545.000 -1.98%
453.povray 516.000 -
456.hmmer 3620.000 -
458.sjeng 9496.000 -
462.libquantum 178.000 -
464.h264ref 5973.000 -
470.lbm 3921.000 -
471.omnetpp 3230.000 -
473.astar 4992.000 -
482.sphinx3 986.000 -
Geometric Mean 1538.653 -
Code Size (bytes)
Name Current %
401.bzip2 35029.000 -
403.gcc 1837421.000 -
429.mcf 7458.000 -
433.milc 69085.000 -
444.namd 152863.000 -
445.gobmk 1288986.000 -
447.dealII 1470587.000 -
450.soplex 179990.000 -
453.povray 541811.000 -
456.hmmer 155802.000 -
458.sjeng 89781.000 -
462.libquantum 20782.000 -
464.h264ref 330678.000 -
470.lbm 9698.000 -
471.omnetpp 363239.000 -
473.astar 21331.000 -
482.sphinx3 107583.000 -
Geometric Mean 128689.966 -
sample variability
Name Current %
401.bzip2 11.000 -
401.bzip2 (avg) 11.000 -
401.bzip2 (avg) (percentage) 0.380 -
401.bzip2 (max) 16.000 -
401.bzip2 (max) (percentage) 0.550 -1.79%
403.gcc 2.000 -
403.gcc (avg) 2.000 -
403.gcc (avg) (percentage) 3.640 -
403.gcc (max) 4.000 -20.00%
403.gcc (max) (percentage) 8.110 -1.34%
429.mcf 11.000 -
429.mcf (avg) 11.000 -
429.mcf (avg) (percentage) 1.040 -
429.mcf (max) 15.000 -11.76%
429.mcf (max) (percentage) 1.350 -10.00%
433.milc 8.000 -
433.milc (avg) 8.000 -
433.milc (avg) (percentage) 0.730 1.39%
433.milc (max) 10.000 -
433.milc (max) (percentage) 0.930 9.41%
433.milc (unexplained diff vs previous) 11.000 10.00%
433.milc (unexplained diff vs previous) (percentage) 0.940 10.59%
444.namd 6.000 -
444.namd (avg) 6.000 -
444.namd (avg) (percentage) 0.800 -
444.namd (max) 8.000 -11.11%
444.namd (max) (percentage) 1.070 -6.14%
445.gobmk 16.000 -
445.gobmk (avg) 16.000 -
445.gobmk (avg) (percentage) 0.300 -
445.gobmk (max) 12.000 -
445.gobmk (max) (percentage) 0.230 -
447.dealII 12.000 -
447.dealII (avg) 12.000 -
447.dealII (avg) (percentage) 0.530 -
447.dealII (max) 4.000 -20.00%
447.dealII (max) (percentage) 0.200 -
450.soplex 7.000 -12.50%
450.soplex (avg) 7.000 -12.50%
450.soplex (avg) (percentage) 1.450 -
450.soplex (max) 11.000 -8.33%
450.soplex (max) (percentage) 2.150 -5.70%
453.povray 3.000 -
453.povray (avg) 3.000 -
453.povray (avg) (percentage) 0.770 -
453.povray (max) 2.000 -
453.povray (max) (percentage) 0.410 -
456.hmmer 17.000 -
456.hmmer (avg) 17.000 -
456.hmmer (avg) (percentage) 0.490 -
456.hmmer (max) 14.000 -12.50%
456.hmmer (max) (percentage) 0.410 -10.87%
458.sjeng 36.000 -
458.sjeng (avg) 36.000 -
458.sjeng (avg) (percentage) 0.380 -
458.sjeng (max) 44.000 -4.35%
458.sjeng (max) (percentage) 0.470 -4.08%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 0.910 -
462.libquantum (max) 2.000 -
462.libquantum (max) (percentage) 1.340 -8.22%
464.h264ref 19.000 -
464.h264ref (avg) 19.000 -
464.h264ref (avg) (percentage) 0.320 -
464.h264ref (max) 17.000 -10.53%
464.h264ref (max) (percentage) 0.290 -12.12%
470.lbm 21.000 -
470.lbm (avg) 21.000 -
470.lbm (avg) (percentage) 0.540 -
470.lbm (max) 16.000 -5.88%
470.lbm (max) (percentage) 0.420 -4.55%
470.lbm (unexplained diff vs previous) 2.000 -
470.lbm (unexplained diff vs previous) (percentage) 0.050 -
471.omnetpp 34.000 -
471.omnetpp (avg) 34.000 -
471.omnetpp (avg) (percentage) 1.080 -
471.omnetpp (max) 20.000 17.65%
471.omnetpp (max) (percentage) 0.650 20.37%
471.omnetpp (unexplained diff vs previous) 21.000 23.53%
471.omnetpp (unexplained diff vs previous) (percentage) 0.650 22.64%
473.astar 29.000 -
473.astar (avg) 29.000 -
473.astar (avg) (percentage) 0.590 -
473.astar (max) 10.000 -16.67%
473.astar (max) (percentage) 0.220 -12.00%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.740 -
482.sphinx3 (max) 9.000 -
482.sphinx3 (max) (percentage) 0.950 -5.94%
483.xalancbmk (avg) (percentage) 2.570 -
483.xalancbmk (max) (percentage) 0.310 -
Geometric Mean 3.113 17.06%