Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/e13bed4c5f3544c076ce57e36d9a11eefa5a7815 (+ 1 commits)
regression LLVM-1001
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz-build/74/
View Options
Report
Filter

llvm-arm-master-Oz test results

Run Order Start Time Duration
Current 0074 (llvmorg-18-init-07933-ge13bed4c5f35) 2023-10-04T08:18:28 0:00:00
Previous 0073 (llvmorg-18-init-07932-ga16f6462d756) 2023-10-02T05:07:50 0:00:00
Baseline 0014 (no-change) 2023-02-19T08:08:29 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 11 7
Performance Improvements 36 62
Added Tests 0 4
Unchanged Tests 721 695
Total Tests 768

Run-Over-Run Changes Detail

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
458.sjeng (unexplained diff vs previous) 261.54% 13.000 47.000 - 0.00% -
444.namd (unexplained diff vs previous) (percentage) 252.00% 0.250 0.880 - 0.00% -
444.namd (unexplained diff vs previous) 250.00% 2.000 7.000 - 0.00% -
458.sjeng (unexplained diff vs previous) (percentage) 250.00% 0.140 0.490 - 0.00% -
458.sjeng (max) (percentage) 42.86% 0.350 0.500 - -94.58% -
458.sjeng (max) 42.42% 33.000 47.000 - -94.71% -
444.namd (max) (percentage) 1.16% 0.860 0.870 - -93.38% -
473.astar 1.01% 99.000 100.000 - -59.68% -
473.astar (avg) 1.01% 99.000 100.000 - -59.68% -

Performance Regressions - Score Δ Previous Current σ Δ (B) σ (B)
nb_compile_successful -52.63% 19.000 9.000 - -52.63% -
nb_execution_successful -50.00% 18.000 9.000 - -50.00% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
429.mcf -7.19% 1154.000 1071.000 - -7.27% -
450.soplex -7.13% 561.000 521.000 - -6.13% -
401.bzip2 -2.86% 3006.000 2920.000 - -3.12% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
450.soplex (max) -16.67% 12.000 10.000 - 42.86% -
450.soplex -12.50% 8.000 7.000 - 133.33% -
450.soplex (avg) -12.50% 8.000 7.000 - 133.33% -
482.sphinx3 (max) -12.50% 8.000 7.000 - -95.07% -
470.lbm (max) -10.53% 19.000 17.000 - -96.16% -
473.astar (max) -10.00% 10.000 9.000 - -99.71% -
429.mcf (max) -9.09% 11.000 10.000 - -95.37% -
401.bzip2 (max) -7.69% 26.000 24.000 - -71.76% -
429.mcf -7.69% 26.000 24.000 - -56.36% -
429.mcf (avg) -7.69% 26.000 24.000 - -56.36% -
450.soplex (max) (percentage) -7.31% 2.190 2.030 - 52.63% -
471.omnetpp (max) (percentage) -6.96% 1.580 1.470 - -90.97% -
464.h264ref (max) (percentage) -6.25% 0.480 0.450 - -90.70% -
470.lbm (max) (percentage) -6.12% 0.490 0.460 - -95.93% -
429.mcf (max) (percentage) -5.94% 1.010 0.950 - -94.93% -
444.namd -5.88% 17.000 16.000 - -52.94% -
444.namd (avg) -5.88% 17.000 16.000 - -52.94% -
433.milc (max) (percentage) -5.83% 1.030 0.970 - -92.67% -
401.bzip2 -4.76% 21.000 20.000 - -41.18% -
401.bzip2 (avg) -4.76% 21.000 20.000 - -41.18% -
401.bzip2 (max) (percentage) -4.55% 0.880 0.840 - -70.42% -
456.hmmer (max) (percentage) -4.35% 0.460 0.440 - -95.06% -
445.gobmk (max) (percentage) -3.57% 0.280 0.270 - -91.94% -
464.h264ref (max) -3.57% 28.000 27.000 - -90.75% -
444.namd (avg) (percentage) -2.84% 2.110 2.050 - -51.88% -
462.libquantum (max) (percentage) -2.67% 0.750 0.730 - -87.54% -
482.sphinx3 (max) (percentage) -2.41% 0.830 0.810 - -94.36% -
483.xalancbmk (max) (percentage) -2.25% 0.890 0.870 - -98.16% -
453.povray (max) (percentage) -2.14% 1.870 1.830 - -62.11% -
458.sjeng -1.41% 71.000 70.000 - -55.97% -
458.sjeng (avg) -1.41% 71.000 70.000 - -55.97% -
458.sjeng (avg) (percentage) -1.33% 0.750 0.740 - -55.15% -
447.dealII (max) (percentage) -1.04% 0.960 0.950 - -80.04% -

Run-Over-Baseline Changes Detail

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
450.soplex 133.33% 3.000 7.000 - -12.50% -
450.soplex (avg) 133.33% 3.000 7.000 - -12.50% -
450.soplex (avg) (percentage) 116.67% 0.660 1.430 - 0.00% -
450.soplex (max) (percentage) 52.63% 1.330 2.030 - -7.31% -
450.soplex (max) 42.86% 7.000 10.000 - -16.67% -

Performance Regressions - Score Δ (B) Baseline Current σ (B) Δ σ
nb_compile_successful -52.63% 19.000 9.000 - -52.63% -
nb_execution_successful -50.00% 18.000 9.000 - -50.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
429.mcf -7.27% 1155.000 1071.000 - -7.19% -
450.soplex -6.13% 555.000 521.000 - -7.13% -
401.bzip2 -3.12% 3014.000 2920.000 - -2.86% -
482.sphinx3 -1.21% 992.000 980.000 - -0.71% -
458.sjeng -1.19% 9638.000 9523.000 - 0.50% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
473.astar (max) -99.71% 3144.000 9.000 - -10.00% -
473.astar (max) (percentage) -99.68% 63.460 0.200 - -4.76% -
483.xalancbmk (max) (percentage) -98.16% 47.200 0.870 - -2.25% -
470.lbm (max) -96.16% 443.000 17.000 - -10.53% -
470.lbm (max) (percentage) -95.93% 11.290 0.460 - -6.12% -
429.mcf (max) -95.37% 216.000 10.000 - -9.09% -
482.sphinx3 (max) -95.07% 142.000 7.000 - -12.50% -
456.hmmer (max) (percentage) -95.06% 8.910 0.440 - -4.35% -
429.mcf (max) (percentage) -94.93% 18.730 0.950 - -5.94% -
458.sjeng (max) -94.71% 889.000 47.000 - 42.42% -
458.sjeng (max) (percentage) -94.58% 9.230 0.500 - 42.86% -
482.sphinx3 (max) (percentage) -94.36% 14.360 0.810 - -2.41% -
444.namd (max) -94.23% 104.000 6.000 - 0.00% -
444.namd (max) (percentage) -93.38% 13.140 0.870 - 1.16% -
433.milc (max) (percentage) -92.67% 13.240 0.970 - -5.83% -
445.gobmk (max) (percentage) -91.94% 3.350 0.270 - -3.57% -
471.omnetpp (max) (percentage) -90.97% 16.280 1.470 - -6.96% -
464.h264ref (max) -90.75% 292.000 27.000 - -3.57% -
464.h264ref (max) (percentage) -90.70% 4.840 0.450 - -6.25% -
462.libquantum (max) (percentage) -87.54% 5.860 0.730 - -2.67% -
447.dealII (max) (percentage) -80.04% 4.760 0.950 - -1.04% -
401.bzip2 (max) -71.76% 85.000 24.000 - -7.69% -
401.bzip2 (max) (percentage) -70.42% 2.840 0.840 - -4.55% -
453.povray (max) (percentage) -62.11% 4.830 1.830 - -2.14% -
470.lbm -60.26% 156.000 62.000 - 0.00% -
470.lbm (avg) -60.26% 156.000 62.000 - 0.00% -
470.lbm (avg) (percentage) -60.05% 3.980 1.590 - 0.00% -
473.astar (avg) (percentage) -59.76% 5.020 2.020 - 0.00% -
473.astar -59.68% 248.000 100.000 - 1.01% -
473.astar (avg) -59.68% 248.000 100.000 - 1.01% -
464.h264ref -56.67% 120.000 52.000 - 0.00% -
464.h264ref (avg) -56.67% 120.000 52.000 - 0.00% -
464.h264ref (avg) (percentage) -56.50% 2.000 0.870 - 0.00% -
429.mcf -56.36% 55.000 24.000 - -7.69% -
429.mcf (avg) -56.36% 55.000 24.000 - -7.69% -
458.sjeng -55.97% 159.000 70.000 - -1.41% -
458.sjeng (avg) -55.97% 159.000 70.000 - -1.41% -
458.sjeng (avg) (percentage) -55.15% 1.650 0.740 - -1.33% -
445.gobmk (avg) (percentage) -54.20% 1.310 0.600 - 0.00% -
444.namd -52.94% 34.000 16.000 - -5.88% -
444.namd (avg) -52.94% 34.000 16.000 - -5.88% -
444.namd (avg) (percentage) -51.88% 4.260 2.050 - -2.84% -
471.omnetpp (avg) (percentage) -51.41% 3.540 1.720 - 0.00% -
456.hmmer (avg) (percentage) -51.37% 3.640 1.770 - 0.00% -
429.mcf (avg) (percentage) -51.15% 4.770 2.330 - 0.00% -
482.sphinx3 -50.00% 34.000 17.000 - 0.00% -
482.sphinx3 (avg) -50.00% 34.000 17.000 - 0.00% -
482.sphinx3 (avg) (percentage) -47.69% 3.460 1.810 - 0.00% -
403.gcc (max) (percentage) -42.34% 11.620 6.700 - -0.30% -
483.xalancbmk (avg) (percentage) -41.85% 6.810 3.960 - 0.00% -
401.bzip2 -41.18% 34.000 20.000 - -4.76% -
401.bzip2 (avg) -41.18% 34.000 20.000 - -4.76% -
462.libquantum (avg) (percentage) -40.00% 1.600 0.960 - 0.00% -
401.bzip2 (avg) (percentage) -38.79% 1.160 0.710 - 0.00% -
433.milc (avg) (percentage) -36.76% 2.040 1.290 - 0.00% -
447.dealII (avg) (percentage) -17.54% 1.140 0.940 - 0.00% -
453.povray (avg) (percentage) -17.33% 1.500 1.240 - 0.00% -

Added Tests - sample variability
444.namd (unexplained diff vs previous)
444.namd (unexplained diff vs previous) (percentage)
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)

Report Time: 0.23s
Execution (seconds)
Name Current %
401.bzip2 2920.000 -2.86%
429.mcf 1071.000 -7.19%
444.namd 799.000 -
450.soplex 521.000 -7.13%
458.sjeng 9523.000 -
464.h264ref 6009.000 -
470.lbm 3909.000 -
473.astar 4978.000 -
482.sphinx3 980.000 -
Geometric Mean 2240.142 -2.05%
Code Size (bytes)
Name Current %
401.bzip2 35825.000 -
429.mcf 7534.000 -
444.namd 154727.000 -
450.soplex 180234.000 -
458.sjeng 90393.000 -
464.h264ref 335688.000 -
470.lbm 9742.000 -
473.astar 21387.000 -
482.sphinx3 108107.000 -
Geometric Mean 55680.627 -
Score
Name Current %
nb_compile_successful 9.000 -52.63%
nb_execution_successful 9.000 -50.00%
nb_total_tests 19.000 -
Geometric Mean 11.546 -38.13%
sample variability
Name Current %
401.bzip2 20.000 -4.76%
401.bzip2 (avg) 20.000 -4.76%
401.bzip2 (avg) (percentage) 0.710 -
401.bzip2 (max) 24.000 -7.69%
401.bzip2 (max) (percentage) 0.840 -4.55%
403.gcc (avg) (percentage) 1.730 -
403.gcc (max) (percentage) 6.700 -
429.mcf 24.000 -7.69%
429.mcf (avg) 24.000 -7.69%
429.mcf (avg) (percentage) 2.330 -
429.mcf (max) 10.000 -9.09%
429.mcf (max) (percentage) 0.950 -5.94%
433.milc (avg) (percentage) 1.290 -
433.milc (max) (percentage) 0.970 -5.83%
444.namd 16.000 -5.88%
444.namd (avg) 16.000 -5.88%
444.namd (avg) (percentage) 2.050 -2.84%
444.namd (max) 6.000 -
444.namd (max) (percentage) 0.870 1.16%
444.namd (unexplained diff vs previous) 7.000 250.00%
444.namd (unexplained diff vs previous) (percentage) 0.880 252.00%
445.gobmk (avg) (percentage) 0.600 -
445.gobmk (max) (percentage) 0.270 -3.57%
447.dealII (avg) (percentage) 0.940 -
447.dealII (max) (percentage) 0.950 -1.04%
450.soplex 7.000 -12.50%
450.soplex (avg) 7.000 -12.50%
450.soplex (avg) (percentage) 1.430 -
450.soplex (max) 10.000 -16.67%
450.soplex (max) (percentage) 2.030 -7.31%
453.povray (avg) (percentage) 1.240 -
453.povray (max) (percentage) 1.830 -2.14%
456.hmmer (avg) (percentage) 1.770 -
456.hmmer (max) (percentage) 0.440 -4.35%
458.sjeng 70.000 -1.41%
458.sjeng (avg) 70.000 -1.41%
458.sjeng (avg) (percentage) 0.740 -1.33%
458.sjeng (max) 47.000 42.42%
458.sjeng (max) (percentage) 0.500 42.86%
458.sjeng (unexplained diff vs previous) 47.000 261.54%
458.sjeng (unexplained diff vs previous) (percentage) 0.490 250.00%
462.libquantum (avg) (percentage) 0.960 -
462.libquantum (max) (percentage) 0.730 -2.67%
464.h264ref 52.000 -
464.h264ref (avg) 52.000 -
464.h264ref (avg) (percentage) 0.870 -
464.h264ref (max) 27.000 -3.57%
464.h264ref (max) (percentage) 0.450 -6.25%
470.lbm 62.000 -
470.lbm (avg) 62.000 -
470.lbm (avg) (percentage) 1.590 -
470.lbm (max) 17.000 -10.53%
470.lbm (max) (percentage) 0.460 -6.12%
471.omnetpp (avg) (percentage) 1.720 -
471.omnetpp (max) (percentage) 1.470 -6.96%
473.astar 100.000 1.01%
473.astar (avg) 100.000 1.01%
473.astar (avg) (percentage) 2.020 -
473.astar (max) 9.000 -10.00%
473.astar (max) (percentage) 0.200 -
482.sphinx3 17.000 -
482.sphinx3 (avg) 17.000 -
482.sphinx3 (avg) (percentage) 1.810 -
482.sphinx3 (max) 7.000 -12.50%
482.sphinx3 (max) (percentage) 0.810 -2.41%
483.xalancbmk (avg) (percentage) 3.960 -
483.xalancbmk (max) (percentage) 0.870 -2.25%
Geometric Mean 3.962 5.59%