Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/aa9e4f0bc93fc735e83ea8b5269e699f31fb9d28 (+ 153 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz-build/159/
View Options
Report
Filter

llvm-arm-master-Oz test results

Run Order Start Time Duration
Current 0159 (llvmorg-19-init-16734-gaa9e4f0bc93f) 2024-07-04T20:54:27 0:00:00
Previous 0158 (llvmorg-19-init-16581-g74984dee5130) 2024-07-01T11:29:30 0:00:00
Baseline 0014 (no-change) 2023-02-19T08:08:29 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 3 10
Performance Improvements 45 95
Added Tests 0 6
Unchanged Tests 588 525
Total Tests 636

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
444.namd 1.27% 788.000 798.000 - -0.13% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
456.hmmer 4.76% 21.000 22.000 - -83.21% -
456.hmmer (avg) 4.76% 21.000 22.000 - -83.21% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
462.libquantum -1.10% 181.000 179.000 - -2.72% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
473.astar (unexplained diff vs previous) (percentage) -91.67% 0.240 0.020 0.000 -99.99% 0.000
473.astar (unexplained diff vs previous) -91.67% 12.000 1.000 0.000 -99.99% 0.000
470.lbm (unexplained diff vs previous) (percentage) -89.13% 0.460 0.050 0.000 0.00% 0.000
470.lbm (unexplained diff vs previous) -88.89% 18.000 2.000 0.000 0.00% 0.000
433.milc (unexplained diff vs previous) (percentage) -67.11% 0.760 0.250 0.000 0.00% 0.000
433.milc (unexplained diff vs previous) -66.67% 9.000 3.000 0.000 0.00% 0.000
433.milc (max) -33.33% 9.000 6.000 - -96.08% -
471.omnetpp (unexplained diff vs previous) (percentage) -31.71% 0.410 0.280 0.000 0.00% 0.000
471.omnetpp (unexplained diff vs previous) -30.77% 13.000 9.000 0.000 0.00% 0.000
433.milc (max) (percentage) -25.00% 0.760 0.570 - -95.69% -
429.mcf (max) -16.67% 6.000 5.000 - -97.69% -
470.lbm (max) -16.67% 18.000 15.000 - -96.61% -
473.astar (max) -16.67% 12.000 10.000 - -99.68% -
473.astar (max) (percentage) -16.67% 0.240 0.200 - -99.68% -
471.omnetpp (max) -13.64% 22.000 19.000 - -96.25% -
470.lbm (max) (percentage) -13.04% 0.460 0.400 - -96.46% -
450.soplex (max) -12.50% 8.000 7.000 - 0.00% -
471.omnetpp (max) (percentage) -11.43% 0.700 0.620 - -96.19% -
456.hmmer (max) -8.33% 12.000 11.000 - -96.58% -
429.mcf (max) (percentage) -7.14% 0.560 0.520 - -97.22% -
401.bzip2 (max) -6.67% 15.000 14.000 - -83.53% -
450.soplex (max) (percentage) -5.88% 1.530 1.440 - 8.27% -
456.hmmer (max) (percentage) -5.88% 0.340 0.320 - -96.41% -
401.bzip2 (max) (percentage) -5.77% 0.520 0.490 - -82.75% -
447.dealII (max) -4.55% 22.000 21.000 - -81.74% -
458.sjeng (max) -4.55% 44.000 42.000 - -95.28% -
462.libquantum (max) (percentage) -4.52% 1.550 1.480 - -74.74% -
458.sjeng (max) (percentage) -4.26% 0.470 0.450 - -95.12% -
470.lbm -4.00% 25.000 24.000 - -84.62% -
470.lbm (avg) -4.00% 25.000 24.000 - -84.62% -
482.sphinx3 (max) (percentage) -3.85% 0.260 0.250 - -98.26% -
444.namd (max) (percentage) -3.30% 0.910 0.880 - -93.30% -
471.omnetpp -2.63% 38.000 37.000 - -66.36% -
471.omnetpp (avg) -2.63% 38.000 37.000 - -66.36% -
445.gobmk (max) (percentage) -2.56% 0.390 0.380 - -88.66% -
473.astar -2.56% 39.000 38.000 - -84.68% -
473.astar (avg) -2.56% 39.000 38.000 - -84.68% -
473.astar (avg) (percentage) -2.53% 0.790 0.770 - -84.66% -
447.dealII (max) (percentage) -2.25% 0.890 0.870 - -81.72% -
471.omnetpp (avg) (percentage) -1.68% 1.190 1.170 - -66.95% -
453.povray (max) (percentage) -1.64% 1.220 1.200 - -75.16% -
470.lbm (avg) (percentage) -1.56% 0.640 0.630 - -84.17% -
433.milc (avg) (percentage) -1.28% 0.780 0.770 - -62.25% -
403.gcc (max) (percentage) -1.03% 2.910 2.880 - -75.22% -

Run-Over-Baseline Changes Detail

Performance Regressions - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc 3.33% 60.000 62.000 - 0.00% -
471.omnetpp 3.05% 3116.000 3211.000 - 0.28% -
447.dealII 2.34% 2432.000 2489.000 - 0.48% -
433.milc 1.89% 1162.000 1184.000 - -0.25% -
473.astar 1.88% 4955.000 5048.000 - -0.02% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
450.soplex 133.33% 3.000 7.000 - 0.00% -
450.soplex (avg) 133.33% 3.000 7.000 - 0.00% -
450.soplex (avg) (percentage) 96.97% 0.660 1.300 - 0.00% -
403.gcc (avg) (percentage) 83.91% 1.740 3.200 - 0.00% -
450.soplex (max) (percentage) 8.27% 1.330 1.440 - -5.88% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -5.17% 22066.000 20926.000 - -0.08% -
401.bzip2 -1.67% 35781.000 35185.000 - 0.02% -
464.h264ref -1.22% 334968.000 330882.000 - -0.01% -
453.povray -1.21% 549774.000 543131.000 - 0.01% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -2.72% 184.000 179.000 - -1.10% -
458.sjeng -1.75% 9638.000 9469.000 - -0.41% -
445.gobmk -1.69% 5439.000 5347.000 - -0.69% -
464.h264ref -1.61% 6043.000 5946.000 - 0.12% -
429.mcf -1.56% 1155.000 1137.000 - -0.35% -
401.bzip2 -1.26% 3014.000 2976.000 - 0.27% -
453.povray -1.13% 529.000 523.000 - 0.97% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
473.astar (unexplained diff vs previous) (percentage) -99.99% 173.660 0.020 0.000 -91.67% 0.000
473.astar (unexplained diff vs previous) -99.99% 8605.000 1.000 0.000 -91.67% 0.000
473.astar (max) (percentage) -99.68% 63.460 0.200 - -16.67% -
473.astar (max) -99.68% 3144.000 10.000 - -16.67% -
483.xalancbmk (max) (percentage) -99.28% 47.200 0.340 - -2.86% -
482.sphinx3 (max) -98.59% 142.000 2.000 - 0.00% -
482.sphinx3 (max) (percentage) -98.26% 14.360 0.250 - -3.85% -
429.mcf (max) -97.69% 216.000 5.000 - -16.67% -
464.h264ref (max) -97.60% 292.000 7.000 - 0.00% -
464.h264ref (max) (percentage) -97.52% 4.840 0.120 - 0.00% -
429.mcf (max) (percentage) -97.22% 18.730 0.520 - -7.14% -
470.lbm (max) -96.61% 443.000 15.000 - -16.67% -
456.hmmer (max) -96.58% 322.000 11.000 - -8.33% -
470.lbm (max) (percentage) -96.46% 11.290 0.400 - -13.04% -
456.hmmer (max) (percentage) -96.41% 8.910 0.320 - -5.88% -
471.omnetpp (max) -96.25% 507.000 19.000 - -13.64% -
471.omnetpp (max) (percentage) -96.19% 16.280 0.620 - -11.43% -
433.milc (max) -96.08% 153.000 6.000 - -33.33% -
433.milc (max) (percentage) -95.69% 13.240 0.570 - -25.00% -
458.sjeng (max) -95.28% 889.000 42.000 - -4.55% -
458.sjeng (max) (percentage) -95.12% 9.230 0.450 - -4.26% -
444.namd (max) (percentage) -93.30% 13.140 0.880 - -3.30% -
444.namd (max) -93.27% 104.000 7.000 - 0.00% -
445.gobmk (max) -89.01% 182.000 20.000 - 0.00% -
445.gobmk (max) (percentage) -88.66% 3.350 0.380 - -2.56% -
473.astar -84.68% 248.000 38.000 - -2.56% -
473.astar (avg) -84.68% 248.000 38.000 - -2.56% -
473.astar (avg) (percentage) -84.66% 5.020 0.770 - -2.53% -
470.lbm -84.62% 156.000 24.000 - -4.00% -
470.lbm (avg) -84.62% 156.000 24.000 - -4.00% -
470.lbm (avg) (percentage) -84.17% 3.980 0.630 - -1.56% -
401.bzip2 (max) -83.53% 85.000 14.000 - -6.67% -
403.gcc (max) -83.33% 6.000 1.000 - 0.00% -
456.hmmer (avg) (percentage) -83.24% 3.640 0.610 - 0.00% -
456.hmmer -83.21% 131.000 22.000 - 4.76% -
456.hmmer (avg) -83.21% 131.000 22.000 - 4.76% -
401.bzip2 (max) (percentage) -82.75% 2.840 0.490 - -5.77% -
447.dealII (max) -81.74% 115.000 21.000 - -4.55% -
447.dealII (max) (percentage) -81.72% 4.760 0.870 - -2.25% -
464.h264ref -81.67% 120.000 22.000 - 0.00% -
464.h264ref (avg) -81.67% 120.000 22.000 - 0.00% -
464.h264ref (avg) (percentage) -81.00% 2.000 0.380 - 0.00% -
462.libquantum (max) -80.00% 10.000 2.000 - 0.00% -
444.namd -79.41% 34.000 7.000 - 0.00% -
444.namd (avg) -79.41% 34.000 7.000 - 0.00% -
482.sphinx3 -79.41% 34.000 7.000 - 0.00% -
482.sphinx3 (avg) -79.41% 34.000 7.000 - 0.00% -
444.namd (avg) (percentage) -78.87% 4.260 0.900 - 0.00% -
482.sphinx3 (avg) (percentage) -77.17% 3.460 0.790 - 0.00% -
429.mcf -76.36% 55.000 13.000 - 0.00% -
429.mcf (avg) -76.36% 55.000 13.000 - 0.00% -
445.gobmk -76.06% 71.000 17.000 - 0.00% -
445.gobmk (avg) -76.06% 71.000 17.000 - 0.00% -
453.povray (max) -76.00% 25.000 6.000 - 0.00% -
445.gobmk (avg) (percentage) -75.57% 1.310 0.320 - 0.00% -
403.gcc (max) (percentage) -75.22% 11.620 2.880 - -1.03% -
453.povray (max) (percentage) -75.16% 4.830 1.200 - -1.64% -
429.mcf (avg) (percentage) -74.84% 4.770 1.200 - 0.00% -
458.sjeng -74.84% 159.000 40.000 - 0.00% -
458.sjeng (avg) -74.84% 159.000 40.000 - 0.00% -
462.libquantum (max) (percentage) -74.74% 5.860 1.480 - -4.52% -
458.sjeng (avg) (percentage) -73.94% 1.650 0.430 - 0.00% -
471.omnetpp (avg) (percentage) -66.95% 3.540 1.170 - -1.68% -
471.omnetpp -66.36% 110.000 37.000 - -2.63% -
471.omnetpp (avg) -66.36% 110.000 37.000 - -2.63% -
483.xalancbmk (avg) (percentage) -62.26% 6.810 2.570 - 0.00% -
433.milc (avg) (percentage) -62.25% 2.040 0.770 - -1.28% -
401.bzip2 (avg) (percentage) -62.07% 1.160 0.440 - 0.00% -
401.bzip2 -61.76% 34.000 13.000 - 0.00% -
401.bzip2 (avg) -61.76% 34.000 13.000 - 0.00% -
433.milc -60.87% 23.000 9.000 - 0.00% -
433.milc (avg) -60.87% 23.000 9.000 - 0.00% -
462.libquantum (avg) (percentage) -58.13% 1.600 0.670 - 0.00% -
462.libquantum -50.00% 2.000 1.000 - 0.00% -
462.libquantum (avg) -50.00% 2.000 1.000 - 0.00% -
453.povray -42.86% 7.000 4.000 - 0.00% -
453.povray (avg) -42.86% 7.000 4.000 - 0.00% -
447.dealII (avg) (percentage) -42.11% 1.140 0.660 - 0.00% -
447.dealII -40.74% 27.000 16.000 - 0.00% -
447.dealII (avg) -40.74% 27.000 16.000 - 0.00% -
453.povray (avg) (percentage) -39.33% 1.500 0.910 - 0.00% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 110.53% 19.000 40.000 - 0.00% -
nb_compile_successful 89.47% 19.000 36.000 - 0.00% -
nb_execution_successful 88.89% 18.000 34.000 - 0.00% -

Added Tests - sample variability
433.milc (unexplained diff vs previous)
433.milc (unexplained diff vs previous) (percentage)
470.lbm (unexplained diff vs previous)
470.lbm (unexplained diff vs previous) (percentage)
471.omnetpp (unexplained diff vs previous)
471.omnetpp (unexplained diff vs previous) (percentage)

Report Time: 0.24s
Execution (seconds)
Name Current %
401.bzip2 2976.000 -
403.gcc 62.000 -
429.mcf 1137.000 -
433.milc 1184.000 -
444.namd 798.000 1.27%
445.gobmk 5347.000 -
447.dealII 2489.000 -
450.soplex 552.000 -
453.povray 523.000 -
456.hmmer 3613.000 -
458.sjeng 9469.000 -
462.libquantum 179.000 -1.10%
464.h264ref 5946.000 -
470.lbm 3922.000 -
471.omnetpp 3211.000 -
473.astar 5048.000 -
482.sphinx3 985.000 -
Geometric Mean 1550.847 -
Code Size (bytes)
Name Current %
401.bzip2 35185.000 -
403.gcc 1839417.000 -
429.mcf 7466.000 -
433.milc 69141.000 -
444.namd 153347.000 -
445.gobmk 1289114.000 -
447.dealII 1472639.000 -
450.soplex 180014.000 -
453.povray 543131.000 -
456.hmmer 156046.000 -
458.sjeng 89901.000 -
462.libquantum 20926.000 -
464.h264ref 330882.000 -
470.lbm 9706.000 -
471.omnetpp 363427.000 -
473.astar 21359.000 -
482.sphinx3 107751.000 -
Geometric Mean 128911.730 -
sample variability
Name Current %
401.bzip2 13.000 -
401.bzip2 (avg) 13.000 -
401.bzip2 (avg) (percentage) 0.440 -
401.bzip2 (max) 14.000 -6.67%
401.bzip2 (max) (percentage) 0.490 -5.77%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 3.200 -
403.gcc (max) 1.000 -
403.gcc (max) (percentage) 2.880 -1.03%
429.mcf 13.000 -
429.mcf (avg) 13.000 -
429.mcf (avg) (percentage) 1.200 -
429.mcf (max) 5.000 -16.67%
429.mcf (max) (percentage) 0.520 -7.14%
433.milc 9.000 -
433.milc (avg) 9.000 -
433.milc (avg) (percentage) 0.770 -1.28%
433.milc (max) 6.000 -33.33%
433.milc (max) (percentage) 0.570 -25.00%
433.milc (unexplained diff vs previous) 3.000 -66.67%
433.milc (unexplained diff vs previous) (percentage) 0.250 -67.11%
444.namd 7.000 -
444.namd (avg) 7.000 -
444.namd (avg) (percentage) 0.900 -
444.namd (max) 7.000 -
444.namd (max) (percentage) 0.880 -3.30%
445.gobmk 17.000 -
445.gobmk (avg) 17.000 -
445.gobmk (avg) (percentage) 0.320 -
445.gobmk (max) 20.000 -
445.gobmk (max) (percentage) 0.380 -2.56%
447.dealII 16.000 -
447.dealII (avg) 16.000 -
447.dealII (avg) (percentage) 0.660 -
447.dealII (max) 21.000 -4.55%
447.dealII (max) (percentage) 0.870 -2.25%
450.soplex 7.000 -
450.soplex (avg) 7.000 -
450.soplex (avg) (percentage) 1.300 -
450.soplex (max) 7.000 -12.50%
450.soplex (max) (percentage) 1.440 -5.88%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.910 -
453.povray (max) 6.000 -
453.povray (max) (percentage) 1.200 -1.64%
456.hmmer 22.000 4.76%
456.hmmer (avg) 22.000 4.76%
456.hmmer (avg) (percentage) 0.610 -
456.hmmer (max) 11.000 -8.33%
456.hmmer (max) (percentage) 0.320 -5.88%
458.sjeng 40.000 -
458.sjeng (avg) 40.000 -
458.sjeng (avg) (percentage) 0.430 -
458.sjeng (max) 42.000 -4.55%
458.sjeng (max) (percentage) 0.450 -4.26%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 0.670 -
462.libquantum (max) 2.000 -
462.libquantum (max) (percentage) 1.480 -4.52%
464.h264ref 22.000 -
464.h264ref (avg) 22.000 -
464.h264ref (avg) (percentage) 0.380 -
464.h264ref (max) 7.000 -
464.h264ref (max) (percentage) 0.120 -
470.lbm 24.000 -4.00%
470.lbm (avg) 24.000 -4.00%
470.lbm (avg) (percentage) 0.630 -1.56%
470.lbm (max) 15.000 -16.67%
470.lbm (max) (percentage) 0.400 -13.04%
470.lbm (unexplained diff vs previous) 2.000 -88.89%
470.lbm (unexplained diff vs previous) (percentage) 0.050 -89.13%
471.omnetpp 37.000 -2.63%
471.omnetpp (avg) 37.000 -2.63%
471.omnetpp (avg) (percentage) 1.170 -1.68%
471.omnetpp (max) 19.000 -13.64%
471.omnetpp (max) (percentage) 0.620 -11.43%
471.omnetpp (unexplained diff vs previous) 9.000 -30.77%
471.omnetpp (unexplained diff vs previous) (percentage) 0.280 -31.71%
473.astar 38.000 -2.56%
473.astar (avg) 38.000 -2.56%
473.astar (avg) (percentage) 0.770 -2.53%
473.astar (max) 10.000 -16.67%
473.astar (max) (percentage) 0.200 -16.67%
473.astar (unexplained diff vs previous) 1.000 -91.67%
473.astar (unexplained diff vs previous) (percentage) 0.020 -91.67%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.790 -
482.sphinx3 (max) 2.000 -
482.sphinx3 (max) (percentage) 0.250 -3.85%
483.xalancbmk (avg) (percentage) 2.570 -
483.xalancbmk (max) (percentage) 0.340 -
Geometric Mean 2.748 -14.82%