Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/b3b9f8dd4cad55cf4106570af518c5075c2cdcee (+ 3558 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz-build/148/
View Options
Report
Filter

llvm-arm-master-Oz test results

Run Order Start Time Duration
Current 0148 (llvmorg-19-init-13502-gb3b9f8dd4cad) 2024-05-17T18:56:34 0:00:00
Previous 0147 (llvmorg-19-init-09944-g8009bbec59d1) 2024-04-28T03:12:02 0:00:00
Baseline 0014 (no-change) 2023-02-19T08:08:29 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 4 11
Performance Improvements 48 96
Added Tests 0 14
Unchanged Tests 728 659
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc 6.90% 58.000 62.000 - 3.33% -
447.dealII 3.28% 2408.000 2487.000 - 2.26% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
401.bzip2 (unexplained diff vs previous) 36.36% 11.000 15.000 0.000 0.00% 0.000
401.bzip2 (unexplained diff vs previous) (percentage) 35.14% 0.370 0.500 0.000 0.00% 0.000

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
453.povray -2.29% 523.000 511.000 - -3.40% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
433.milc (unexplained diff vs previous) -80.00% 10.000 2.000 0.000 0.00% 0.000
433.milc (unexplained diff vs previous) (percentage) -79.76% 0.840 0.170 0.000 0.00% 0.000
473.astar (unexplained diff vs previous) -76.19% 21.000 5.000 0.000 -99.94% 0.000
473.astar (unexplained diff vs previous) (percentage) -76.19% 0.420 0.100 0.000 -99.94% 0.000
462.libquantum (unexplained diff vs previous) (percentage) -66.87% 1.660 0.550 0.000 0.00% 0.000
462.libquantum (unexplained diff vs previous) -66.67% 3.000 1.000 0.000 0.00% 0.000
470.lbm (unexplained diff vs previous) -66.67% 15.000 5.000 0.000 0.00% 0.000
470.lbm (unexplained diff vs previous) (percentage) -65.79% 0.380 0.130 0.000 0.00% 0.000
464.h264ref (max) -54.29% 35.000 16.000 - -94.52% -
464.h264ref (max) (percentage) -54.24% 0.590 0.270 - -94.42% -
471.omnetpp (unexplained diff vs previous) (percentage) -31.67% 1.200 0.820 0.000 0.00% 0.000
471.omnetpp (unexplained diff vs previous) -31.58% 38.000 26.000 0.000 0.00% 0.000
429.mcf (unexplained diff vs previous) -28.57% 7.000 5.000 0.000 0.00% 0.000
429.mcf (unexplained diff vs previous) (percentage) -27.87% 0.610 0.440 0.000 0.00% 0.000
464.h264ref (unexplained diff vs previous) (percentage) -26.47% 0.340 0.250 0.000 0.00% 0.000
462.libquantum (max) (percentage) -25.15% 1.630 1.220 - -79.18% -
464.h264ref (unexplained diff vs previous) -25.00% 20.000 15.000 0.000 0.00% 0.000
473.astar (max) -23.81% 21.000 16.000 - -99.49% -
473.astar (max) (percentage) -23.81% 0.420 0.320 - -99.50% -
433.milc (max) -20.00% 10.000 8.000 - -94.77% -
433.milc (max) (percentage) -16.47% 0.850 0.710 - -94.64% -
470.lbm (max) (percentage) -15.79% 0.380 0.320 - -97.17% -
429.mcf (max) -14.29% 7.000 6.000 - -97.22% -
470.lbm (max) -14.29% 14.000 12.000 - -97.29% -
429.mcf (max) (percentage) -13.11% 0.610 0.530 - -97.17% -
471.omnetpp (max) (percentage) -10.66% 1.220 1.090 - -93.30% -
471.omnetpp (max) -10.53% 38.000 34.000 - -93.29% -
456.hmmer (max) (percentage) -10.34% 0.290 0.260 - -97.08% -
456.hmmer (max) -10.00% 10.000 9.000 - -97.20% -
401.bzip2 (max) (percentage) -8.77% 0.570 0.520 - -81.69% -
445.gobmk (max) (percentage) -8.11% 0.370 0.340 - -89.85% -
444.namd (max) (percentage) -8.00% 0.500 0.460 - -96.50% -
482.sphinx3 (max) (percentage) -6.88% 1.600 1.490 - -89.62% -
482.sphinx3 (max) -6.67% 15.000 14.000 - -90.14% -
401.bzip2 (max) -6.25% 16.000 15.000 - -82.35% -
445.gobmk (max) -5.26% 19.000 18.000 - -90.11% -
458.sjeng (max) -4.88% 41.000 39.000 - -95.61% -
458.sjeng (max) (percentage) -4.55% 0.440 0.420 - -95.45% -
447.dealII (max) (percentage) -4.29% 0.700 0.670 - -85.92% -
470.lbm -3.57% 28.000 27.000 - -82.69% -
470.lbm (avg) -3.57% 28.000 27.000 - -82.69% -
453.povray (max) (percentage) -3.45% 0.870 0.840 - -82.61% -
483.xalancbmk (max) (percentage) -2.56% 0.390 0.380 - -99.19% -
429.mcf (avg) (percentage) -1.55% 1.290 1.270 - -73.38% -
470.lbm (avg) (percentage) -1.39% 0.720 0.710 - -82.16% -
433.milc (avg) (percentage) -1.19% 0.840 0.830 - -59.31% -
473.astar (avg) (percentage) -1.12% 0.890 0.880 - -82.47% -

Run-Over-Baseline Changes Detail

Performance Regressions - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc 3.33% 60.000 62.000 - 6.90% -
447.dealII 2.26% 2432.000 2487.000 - 3.28% -
471.omnetpp 2.21% 3116.000 3185.000 - 0.82% -
433.milc 1.98% 1162.000 1185.000 - -0.17% -
473.astar 1.27% 4955.000 5018.000 - -0.10% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
450.soplex 133.33% 3.000 7.000 - 0.00% -
450.soplex (avg) 133.33% 3.000 7.000 - 0.00% -
403.gcc (avg) (percentage) 114.37% 1.740 3.730 - 0.00% -
450.soplex (avg) (percentage) 104.55% 0.660 1.350 - 0.00% -
403.gcc 100.00% 1.000 2.000 - 0.00% -
403.gcc (avg) 100.00% 1.000 2.000 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -5.29% 22066.000 20898.000 - 0.00% -
401.bzip2 -1.68% 35781.000 35181.000 - 0.00% -
453.povray -1.19% 549774.000 543255.000 - 0.05% -
464.h264ref -1.10% 334968.000 331286.000 - -0.06% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
453.povray -3.40% 529.000 511.000 - -2.29% -
464.h264ref -1.69% 6043.000 5941.000 - -0.25% -
445.gobmk -1.64% 5439.000 5350.000 - -0.21% -
458.sjeng -1.58% 9638.000 9486.000 - 0.25% -
462.libquantum -1.09% 184.000 182.000 - 0.55% -
401.bzip2 -1.03% 3014.000 2983.000 - 0.51% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
473.astar (unexplained diff vs previous) (percentage) -99.94% 173.660 0.100 0.000 -76.19% 0.000
473.astar (unexplained diff vs previous) -99.94% 8605.000 5.000 0.000 -76.19% 0.000
473.astar (max) (percentage) -99.50% 63.460 0.320 - -23.81% -
473.astar (max) -99.49% 3144.000 16.000 - -23.81% -
483.xalancbmk (max) (percentage) -99.19% 47.200 0.380 - -2.56% -
470.lbm (max) -97.29% 443.000 12.000 - -14.29% -
429.mcf (max) -97.22% 216.000 6.000 - -14.29% -
456.hmmer (max) -97.20% 322.000 9.000 - -10.00% -
429.mcf (max) (percentage) -97.17% 18.730 0.530 - -13.11% -
470.lbm (max) (percentage) -97.17% 11.290 0.320 - -15.79% -
444.namd (max) -97.12% 104.000 3.000 - 0.00% -
456.hmmer (max) (percentage) -97.08% 8.910 0.260 - -10.34% -
444.namd (max) (percentage) -96.50% 13.140 0.460 - -8.00% -
458.sjeng (max) -95.61% 889.000 39.000 - -4.88% -
458.sjeng (max) (percentage) -95.45% 9.230 0.420 - -4.55% -
433.milc (max) -94.77% 153.000 8.000 - -20.00% -
433.milc (max) (percentage) -94.64% 13.240 0.710 - -16.47% -
464.h264ref (max) -94.52% 292.000 16.000 - -54.29% -
464.h264ref (max) (percentage) -94.42% 4.840 0.270 - -54.24% -
471.omnetpp (max) (percentage) -93.30% 16.280 1.090 - -10.66% -
471.omnetpp (max) -93.29% 507.000 34.000 - -10.53% -
482.sphinx3 (max) -90.14% 142.000 14.000 - -6.67% -
445.gobmk (max) -90.11% 182.000 18.000 - -5.26% -
445.gobmk (max) (percentage) -89.85% 3.350 0.340 - -8.11% -
482.sphinx3 (max) (percentage) -89.62% 14.360 1.490 - -6.88% -
447.dealII (max) -86.09% 115.000 16.000 - 0.00% -
447.dealII (max) (percentage) -85.92% 4.760 0.670 - -4.29% -
453.povray (max) -84.00% 25.000 4.000 - 0.00% -
470.lbm -82.69% 156.000 27.000 - -3.57% -
470.lbm (avg) -82.69% 156.000 27.000 - -3.57% -
453.povray (max) (percentage) -82.61% 4.830 0.840 - -3.45% -
473.astar (avg) (percentage) -82.47% 5.020 0.880 - -1.12% -
401.bzip2 (max) -82.35% 85.000 15.000 - -6.25% -
473.astar -82.26% 248.000 44.000 - 0.00% -
473.astar (avg) -82.26% 248.000 44.000 - 0.00% -
470.lbm (avg) (percentage) -82.16% 3.980 0.710 - -1.39% -
401.bzip2 (max) (percentage) -81.69% 2.840 0.520 - -8.77% -
456.hmmer -81.68% 131.000 24.000 - 0.00% -
456.hmmer (avg) -81.68% 131.000 24.000 - 0.00% -
456.hmmer (avg) (percentage) -81.59% 3.640 0.670 - 0.00% -
462.libquantum (max) -80.00% 10.000 2.000 - 0.00% -
444.namd -79.41% 34.000 7.000 - 0.00% -
444.namd (avg) -79.41% 34.000 7.000 - 0.00% -
462.libquantum (max) (percentage) -79.18% 5.860 1.220 - -25.15% -
464.h264ref -79.17% 120.000 25.000 - 0.00% -
464.h264ref (avg) -79.17% 120.000 25.000 - 0.00% -
464.h264ref (avg) (percentage) -78.50% 2.000 0.430 - 0.00% -
444.namd (avg) (percentage) -77.93% 4.260 0.940 - 0.00% -
482.sphinx3 -76.47% 34.000 8.000 - 0.00% -
482.sphinx3 (avg) -76.47% 34.000 8.000 - 0.00% -
445.gobmk -74.65% 71.000 18.000 - 0.00% -
445.gobmk (avg) -74.65% 71.000 18.000 - 0.00% -
482.sphinx3 (avg) (percentage) -74.57% 3.460 0.880 - 0.00% -
429.mcf -74.55% 55.000 14.000 - 0.00% -
429.mcf (avg) -74.55% 55.000 14.000 - 0.00% -
445.gobmk (avg) (percentage) -74.05% 1.310 0.340 - 0.00% -
458.sjeng -73.58% 159.000 42.000 - 0.00% -
458.sjeng (avg) -73.58% 159.000 42.000 - 0.00% -
429.mcf (avg) (percentage) -73.38% 4.770 1.270 - -1.55% -
458.sjeng (avg) (percentage) -72.73% 1.650 0.450 - 0.00% -
471.omnetpp (avg) (percentage) -64.69% 3.540 1.250 - -0.79% -
471.omnetpp -64.55% 110.000 39.000 - 0.00% -
471.omnetpp (avg) -64.55% 110.000 39.000 - 0.00% -
462.libquantum (avg) (percentage) -64.38% 1.600 0.570 - 0.00% -
483.xalancbmk (avg) (percentage) -62.26% 6.810 2.570 - 0.00% -
401.bzip2 -61.76% 34.000 13.000 - 0.00% -
401.bzip2 (avg) -61.76% 34.000 13.000 - 0.00% -
401.bzip2 (avg) (percentage) -61.21% 1.160 0.450 - 0.00% -
433.milc -60.87% 23.000 9.000 - 0.00% -
433.milc (avg) -60.87% 23.000 9.000 - 0.00% -
433.milc (avg) (percentage) -59.31% 2.040 0.830 - -1.19% -
403.gcc (max) -50.00% 6.000 3.000 - 0.00% -
462.libquantum -50.00% 2.000 1.000 - 0.00% -
462.libquantum (avg) -50.00% 2.000 1.000 - 0.00% -
403.gcc (max) (percentage) -48.54% 11.620 5.980 - -0.33% -
453.povray -42.86% 7.000 4.000 - 0.00% -
453.povray (avg) -42.86% 7.000 4.000 - 0.00% -
453.povray (avg) (percentage) -42.67% 1.500 0.860 - 0.00% -
447.dealII (avg) (percentage) -41.23% 1.140 0.670 - 0.00% -
447.dealII -40.74% 27.000 16.000 - 0.00% -
447.dealII (avg) -40.74% 27.000 16.000 - 0.00% -
450.soplex (max) (percentage) -31.58% 1.330 0.910 - 0.00% -
450.soplex (max) -28.57% 7.000 5.000 - 0.00% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 110.53% 19.000 40.000 - 0.00% -
nb_compile_successful 89.47% 19.000 36.000 - 0.00% -
nb_execution_successful 88.89% 18.000 34.000 - 0.00% -

Added Tests - sample variability
401.bzip2 (unexplained diff vs previous)
401.bzip2 (unexplained diff vs previous) (percentage)
429.mcf (unexplained diff vs previous)
429.mcf (unexplained diff vs previous) (percentage)
433.milc (unexplained diff vs previous)
433.milc (unexplained diff vs previous) (percentage)
462.libquantum (unexplained diff vs previous)
462.libquantum (unexplained diff vs previous) (percentage)
464.h264ref (unexplained diff vs previous)
464.h264ref (unexplained diff vs previous) (percentage)
470.lbm (unexplained diff vs previous)
470.lbm (unexplained diff vs previous) (percentage)
471.omnetpp (unexplained diff vs previous)
471.omnetpp (unexplained diff vs previous) (percentage)

Report Time: 0.35s
Execution (seconds)
Name Current %
401.bzip2 2983.000 -
403.gcc 62.000 6.90%
429.mcf 1145.000 -
433.milc 1185.000 -
444.namd 796.000 -
445.gobmk 5350.000 -
447.dealII 2487.000 3.28%
450.soplex 551.000 -
453.povray 511.000 -2.29%
456.hmmer 3608.000 -
458.sjeng 9486.000 -
462.libquantum 182.000 -
464.h264ref 5941.000 -
470.lbm 3923.000 -
471.omnetpp 3185.000 -
473.astar 5018.000 -
482.sphinx3 984.000 -
Geometric Mean 1549.366 -
Code Size (bytes)
Name Current %
401.bzip2 35181.000 -
403.gcc 1839525.000 -
429.mcf 7454.000 -
433.milc 69173.000 -
444.namd 153559.000 -
445.gobmk 1289046.000 -
447.dealII 1472723.000 -
450.soplex 180010.000 -
453.povray 543255.000 -
456.hmmer 156034.000 -
458.sjeng 89905.000 -
462.libquantum 20898.000 -
464.h264ref 331286.000 -
470.lbm 9694.000 -
471.omnetpp 363423.000 -
473.astar 21347.000 -
482.sphinx3 107731.000 -
Geometric Mean 128898.416 -
sample variability
Name Current %
401.bzip2 13.000 -
401.bzip2 (avg) 13.000 -
401.bzip2 (avg) (percentage) 0.450 -
401.bzip2 (max) 15.000 -6.25%
401.bzip2 (max) (percentage) 0.520 -8.77%
401.bzip2 (unexplained diff vs previous) 15.000 36.36%
401.bzip2 (unexplained diff vs previous) (percentage) 0.500 35.14%
403.gcc 2.000 -
403.gcc (avg) 2.000 -
403.gcc (avg) (percentage) 3.730 -
403.gcc (max) 3.000 -
403.gcc (max) (percentage) 5.980 -
429.mcf 14.000 -
429.mcf (avg) 14.000 -
429.mcf (avg) (percentage) 1.270 -1.55%
429.mcf (max) 6.000 -14.29%
429.mcf (max) (percentage) 0.530 -13.11%
429.mcf (unexplained diff vs previous) 5.000 -28.57%
429.mcf (unexplained diff vs previous) (percentage) 0.440 -27.87%
433.milc 9.000 -
433.milc (avg) 9.000 -
433.milc (avg) (percentage) 0.830 -1.19%
433.milc (max) 8.000 -20.00%
433.milc (max) (percentage) 0.710 -16.47%
433.milc (unexplained diff vs previous) 2.000 -80.00%
433.milc (unexplained diff vs previous) (percentage) 0.170 -79.76%
444.namd 7.000 -
444.namd (avg) 7.000 -
444.namd (avg) (percentage) 0.940 -
444.namd (max) 3.000 -
444.namd (max) (percentage) 0.460 -8.00%
445.gobmk 18.000 -
445.gobmk (avg) 18.000 -
445.gobmk (avg) (percentage) 0.340 -
445.gobmk (max) 18.000 -5.26%
445.gobmk (max) (percentage) 0.340 -8.11%
447.dealII 16.000 -
447.dealII (avg) 16.000 -
447.dealII (avg) (percentage) 0.670 -
447.dealII (max) 16.000 -
447.dealII (max) (percentage) 0.670 -4.29%
450.soplex 7.000 -
450.soplex (avg) 7.000 -
450.soplex (avg) (percentage) 1.350 -
450.soplex (max) 5.000 -
450.soplex (max) (percentage) 0.910 -
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.860 -
453.povray (max) 4.000 -
453.povray (max) (percentage) 0.840 -3.45%
456.hmmer 24.000 -
456.hmmer (avg) 24.000 -
456.hmmer (avg) (percentage) 0.670 -
456.hmmer (max) 9.000 -10.00%
456.hmmer (max) (percentage) 0.260 -10.34%
458.sjeng 42.000 -
458.sjeng (avg) 42.000 -
458.sjeng (avg) (percentage) 0.450 -
458.sjeng (max) 39.000 -4.88%
458.sjeng (max) (percentage) 0.420 -4.55%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 0.570 -
462.libquantum (max) 2.000 -
462.libquantum (max) (percentage) 1.220 -25.15%
462.libquantum (unexplained diff vs previous) 1.000 -66.67%
462.libquantum (unexplained diff vs previous) (percentage) 0.550 -66.87%
464.h264ref 25.000 -
464.h264ref (avg) 25.000 -
464.h264ref (avg) (percentage) 0.430 -
464.h264ref (max) 16.000 -54.29%
464.h264ref (max) (percentage) 0.270 -54.24%
464.h264ref (unexplained diff vs previous) 15.000 -25.00%
464.h264ref (unexplained diff vs previous) (percentage) 0.250 -26.47%
470.lbm 27.000 -3.57%
470.lbm (avg) 27.000 -3.57%
470.lbm (avg) (percentage) 0.710 -1.39%
470.lbm (max) 12.000 -14.29%
470.lbm (max) (percentage) 0.320 -15.79%
470.lbm (unexplained diff vs previous) 5.000 -66.67%
470.lbm (unexplained diff vs previous) (percentage) 0.130 -65.79%
471.omnetpp 39.000 -
471.omnetpp (avg) 39.000 -
471.omnetpp (avg) (percentage) 1.250 -
471.omnetpp (max) 34.000 -10.53%
471.omnetpp (max) (percentage) 1.090 -10.66%
471.omnetpp (unexplained diff vs previous) 26.000 -31.58%
471.omnetpp (unexplained diff vs previous) (percentage) 0.820 -31.67%
473.astar 44.000 -
473.astar (avg) 44.000 -
473.astar (avg) (percentage) 0.880 -1.12%
473.astar (max) 16.000 -23.81%
473.astar (max) (percentage) 0.320 -23.81%
473.astar (unexplained diff vs previous) 5.000 -76.19%
473.astar (unexplained diff vs previous) (percentage) 0.100 -76.19%
482.sphinx3 8.000 -
482.sphinx3 (avg) 8.000 -
482.sphinx3 (avg) (percentage) 0.880 -
482.sphinx3 (max) 14.000 -6.67%
482.sphinx3 (max) (percentage) 1.490 -6.88%
483.xalancbmk (avg) (percentage) 2.570 -
483.xalancbmk (max) (percentage) 0.380 -2.56%
Geometric Mean 3.062 -14.86%