Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/54f040fff301a629f2ed032863408ed119789b0e (+ 656 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz-build/157/
View Options
Report
Filter

llvm-arm-master-Oz test results

Run Order Start Time Duration
Current 0157 (llvmorg-19-init-16223-g54f040fff301) 2024-06-27T16:23:19 0:00:00
Previous 0156 (llvmorg-19-init-15567-g28a3fbbe8c6c) 2024-06-23T01:46:24 0:00:00
Baseline 0014 (no-change) 2023-02-19T08:08:29 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 25 10
Performance Improvements 38 95
Added Tests 0 18
Unchanged Tests 705 645
Total Tests 768

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
447.dealII 1.52% 2436.000 2473.000 - 1.69% -
450.soplex 1.27% 552.000 559.000 - 0.72% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
470.lbm (unexplained diff vs previous) 1700.00% 1.000 18.000 0.000 0.00% 0.000
470.lbm (unexplained diff vs previous) (percentage) 1433.33% 0.030 0.460 0.000 0.00% 0.000
458.sjeng (unexplained diff vs previous) 411.11% 9.000 46.000 0.000 0.00% 0.000
464.h264ref (unexplained diff vs previous) 400.00% 1.000 5.000 0.000 0.00% 0.000
458.sjeng (unexplained diff vs previous) (percentage) 390.00% 0.100 0.490 0.000 0.00% 0.000
464.h264ref (unexplained diff vs previous) (percentage) 300.00% 0.020 0.080 0.000 0.00% 0.000
445.gobmk (unexplained diff vs previous) 175.00% 8.000 22.000 0.000 0.00% 0.000
445.gobmk (unexplained diff vs previous) (percentage) 173.33% 0.150 0.410 0.000 0.00% 0.000
470.lbm (max) (percentage) 100.00% 0.230 0.460 - -95.93% -
470.lbm (max) 88.89% 9.000 17.000 - -96.16% -
429.mcf (unexplained diff vs previous) 75.00% 4.000 7.000 0.000 0.00% 0.000
429.mcf (unexplained diff vs previous) (percentage) 74.29% 0.350 0.610 0.000 0.00% 0.000
462.libquantum (unexplained diff vs previous) (percentage) 53.21% 1.090 1.670 0.000 0.00% 0.000
462.libquantum (unexplained diff vs previous) 50.00% 2.000 3.000 0.000 0.00% 0.000
471.omnetpp (unexplained diff vs previous) (percentage) 44.62% 0.650 0.940 0.000 0.00% 0.000
471.omnetpp (unexplained diff vs previous) 42.86% 21.000 30.000 0.000 0.00% 0.000
458.sjeng (max) (percentage) 40.00% 0.350 0.490 - -94.69% -
458.sjeng (max) 39.39% 33.000 46.000 - -94.83% -
462.libquantum (max) (percentage) 16.31% 1.410 1.640 - -72.01% -
429.mcf (max) (percentage) 5.17% 0.580 0.610 - -96.74% -
445.gobmk (max) (percentage) 5.13% 0.390 0.410 - -87.76% -
445.gobmk (max) 5.00% 20.000 21.000 - -88.46% -
462.libquantum (avg) (percentage) 3.08% 0.650 0.670 - -58.13% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -3.28% 61.000 59.000 - -1.67% -
462.libquantum -1.64% 183.000 180.000 - -2.17% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
473.astar (unexplained diff vs previous) -66.67% 3.000 1.000 0.000 -99.99% 0.000
473.astar (unexplained diff vs previous) (percentage) -66.67% 0.060 0.020 0.000 -99.99% 0.000
433.milc (unexplained diff vs previous) (percentage) -55.84% 0.770 0.340 0.000 0.00% 0.000
433.milc (unexplained diff vs previous) -55.56% 9.000 4.000 0.000 0.00% 0.000
450.soplex (unexplained diff vs previous) (percentage) -47.03% 2.360 1.250 0.000 0.00% 0.000
450.soplex (unexplained diff vs previous) -46.15% 13.000 7.000 0.000 0.00% 0.000
482.sphinx3 (max) -33.33% 3.000 2.000 - -98.59% -
450.soplex (max) (percentage) -25.22% 2.300 1.720 - 29.32% -
450.soplex (max) -25.00% 12.000 9.000 - 28.57% -
433.milc (max) (percentage) -17.11% 0.760 0.630 - -95.24% -
464.h264ref (max) (percentage) -13.33% 0.150 0.130 - -97.31% -
482.sphinx3 (max) (percentage) -12.90% 0.310 0.270 - -98.12% -
433.milc (max) -12.50% 8.000 7.000 - -95.42% -
464.h264ref (max) -12.50% 8.000 7.000 - -97.60% -
473.astar (max) -12.50% 8.000 7.000 - -99.78% -
471.omnetpp (max) -11.43% 35.000 31.000 - -93.89% -
401.bzip2 (max) -11.11% 18.000 16.000 - -81.18% -
471.omnetpp (max) (percentage) -10.91% 1.100 0.980 - -93.98% -
401.bzip2 (max) (percentage) -9.68% 0.620 0.560 - -80.28% -
456.hmmer (max) -7.69% 13.000 12.000 - -96.27% -
453.povray (max) (percentage) -7.41% 1.350 1.250 - -74.12% -
473.astar (max) (percentage) -6.25% 0.160 0.150 - -99.76% -
456.hmmer (max) (percentage) -5.41% 0.370 0.350 - -96.07% -
464.h264ref -4.35% 23.000 22.000 - -81.67% -
464.h264ref (avg) -4.35% 23.000 22.000 - -81.67% -
470.lbm -4.00% 25.000 24.000 - -84.62% -
470.lbm (avg) -4.00% 25.000 24.000 - -84.62% -
444.namd (max) (percentage) -3.09% 0.970 0.940 - -92.85% -
464.h264ref (avg) (percentage) -2.56% 0.390 0.380 - -81.00% -
473.astar -2.50% 40.000 39.000 - -84.27% -
473.astar (avg) -2.50% 40.000 39.000 - -84.27% -
473.astar (avg) (percentage) -2.47% 0.810 0.790 - -84.26% -
447.dealII (max) (percentage) -2.15% 0.930 0.910 - -80.88% -
470.lbm (avg) (percentage) -1.54% 0.650 0.640 - -83.92% -
433.milc (avg) (percentage) -1.27% 0.790 0.780 - -61.76% -
403.gcc (max) (percentage) -1.01% 2.980 2.950 - -74.61% -

Run-Over-Baseline Changes Detail

Performance Regressions - Execution Δ (B) Baseline Current σ (B) Δ σ
471.omnetpp 2.34% 3116.000 3189.000 - -0.93% -
447.dealII 1.69% 2432.000 2473.000 - 1.52% -
473.astar 1.65% 4955.000 5037.000 - -0.02% -
433.milc 1.38% 1162.000 1178.000 - 0.34% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
450.soplex 133.33% 3.000 7.000 - 0.00% -
450.soplex (avg) 133.33% 3.000 7.000 - 0.00% -
450.soplex (avg) (percentage) 95.45% 0.660 1.290 - 0.00% -
403.gcc (avg) (percentage) 83.91% 1.740 3.200 - 0.00% -
450.soplex (max) (percentage) 29.32% 1.330 1.720 - -25.22% -
450.soplex (max) 28.57% 7.000 9.000 - -25.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -5.17% 22066.000 20926.000 - 0.00% -
401.bzip2 -1.67% 35781.000 35185.000 - -0.02% -
464.h264ref -1.22% 334968.000 330882.000 - 0.00% -
453.povray -1.21% 549774.000 543147.000 - -0.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
453.povray -2.65% 529.000 515.000 - 0.39% -
462.libquantum -2.17% 184.000 180.000 - -1.64% -
445.gobmk -1.80% 5439.000 5341.000 - -0.41% -
464.h264ref -1.69% 6043.000 5941.000 - -0.08% -
403.gcc -1.67% 60.000 59.000 - -3.28% -
458.sjeng -1.65% 9638.000 9479.000 - 0.49% -
401.bzip2 -1.13% 3014.000 2980.000 - -0.17% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
473.astar (unexplained diff vs previous) (percentage) -99.99% 173.660 0.020 0.000 -66.67% 0.000
473.astar (unexplained diff vs previous) -99.99% 8605.000 1.000 0.000 -66.67% 0.000
473.astar (max) -99.78% 3144.000 7.000 - -12.50% -
473.astar (max) (percentage) -99.76% 63.460 0.150 - -6.25% -
483.xalancbmk (max) (percentage) -99.26% 47.200 0.350 - 0.00% -
482.sphinx3 (max) -98.59% 142.000 2.000 - -33.33% -
482.sphinx3 (max) (percentage) -98.12% 14.360 0.270 - -12.90% -
464.h264ref (max) -97.60% 292.000 7.000 - -12.50% -
464.h264ref (max) (percentage) -97.31% 4.840 0.130 - -13.33% -
429.mcf (max) -97.22% 216.000 6.000 - 0.00% -
429.mcf (max) (percentage) -96.74% 18.730 0.610 - 5.17% -
456.hmmer (max) -96.27% 322.000 12.000 - -7.69% -
470.lbm (max) -96.16% 443.000 17.000 - 88.89% -
456.hmmer (max) (percentage) -96.07% 8.910 0.350 - -5.41% -
470.lbm (max) (percentage) -95.93% 11.290 0.460 - 100.00% -
433.milc (max) -95.42% 153.000 7.000 - -12.50% -
433.milc (max) (percentage) -95.24% 13.240 0.630 - -17.11% -
458.sjeng (max) -94.83% 889.000 46.000 - 39.39% -
458.sjeng (max) (percentage) -94.69% 9.230 0.490 - 40.00% -
471.omnetpp (max) (percentage) -93.98% 16.280 0.980 - -10.91% -
471.omnetpp (max) -93.89% 507.000 31.000 - -11.43% -
444.namd (max) -93.27% 104.000 7.000 - 0.00% -
444.namd (max) (percentage) -92.85% 13.140 0.940 - -3.09% -
445.gobmk (max) -88.46% 182.000 21.000 - 5.00% -
445.gobmk (max) (percentage) -87.76% 3.350 0.410 - 5.13% -
470.lbm -84.62% 156.000 24.000 - -4.00% -
470.lbm (avg) -84.62% 156.000 24.000 - -4.00% -
473.astar -84.27% 248.000 39.000 - -2.50% -
473.astar (avg) -84.27% 248.000 39.000 - -2.50% -
473.astar (avg) (percentage) -84.26% 5.020 0.790 - -2.47% -
470.lbm (avg) (percentage) -83.92% 3.980 0.640 - -1.54% -
403.gcc (max) -83.33% 6.000 1.000 - 0.00% -
456.hmmer (avg) (percentage) -83.24% 3.640 0.610 - 0.00% -
456.hmmer -83.21% 131.000 22.000 - 0.00% -
456.hmmer (avg) -83.21% 131.000 22.000 - 0.00% -
464.h264ref -81.67% 120.000 22.000 - -4.35% -
464.h264ref (avg) -81.67% 120.000 22.000 - -4.35% -
401.bzip2 (max) -81.18% 85.000 16.000 - -11.11% -
464.h264ref (avg) (percentage) -81.00% 2.000 0.380 - -2.56% -
447.dealII (max) (percentage) -80.88% 4.760 0.910 - -2.15% -
447.dealII (max) -80.87% 115.000 22.000 - 0.00% -
401.bzip2 (max) (percentage) -80.28% 2.840 0.560 - -9.68% -
462.libquantum (max) -80.00% 10.000 2.000 - 0.00% -
444.namd -79.41% 34.000 7.000 - 0.00% -
444.namd (avg) -79.41% 34.000 7.000 - 0.00% -
482.sphinx3 -79.41% 34.000 7.000 - 0.00% -
482.sphinx3 (avg) -79.41% 34.000 7.000 - 0.00% -
444.namd (avg) (percentage) -78.87% 4.260 0.900 - 0.00% -
482.sphinx3 (avg) (percentage) -77.17% 3.460 0.790 - 0.00% -
429.mcf -76.36% 55.000 13.000 - 0.00% -
429.mcf (avg) -76.36% 55.000 13.000 - 0.00% -
445.gobmk -76.06% 71.000 17.000 - 0.00% -
445.gobmk (avg) -76.06% 71.000 17.000 - 0.00% -
453.povray (max) -76.00% 25.000 6.000 - 0.00% -
445.gobmk (avg) (percentage) -75.57% 1.310 0.320 - 0.00% -
429.mcf (avg) (percentage) -74.84% 4.770 1.200 - -0.83% -
458.sjeng -74.84% 159.000 40.000 - 0.00% -
458.sjeng (avg) -74.84% 159.000 40.000 - 0.00% -
403.gcc (max) (percentage) -74.61% 11.620 2.950 - -1.01% -
453.povray (max) (percentage) -74.12% 4.830 1.250 - -7.41% -
458.sjeng (avg) (percentage) -73.94% 1.650 0.430 - 0.00% -
462.libquantum (max) (percentage) -72.01% 5.860 1.640 - 16.31% -
471.omnetpp (avg) (percentage) -66.10% 3.540 1.200 - -0.83% -
471.omnetpp -65.45% 110.000 38.000 - 0.00% -
471.omnetpp (avg) -65.45% 110.000 38.000 - 0.00% -
483.xalancbmk (avg) (percentage) -62.26% 6.810 2.570 - 0.00% -
401.bzip2 (avg) (percentage) -62.07% 1.160 0.440 - 0.00% -
401.bzip2 -61.76% 34.000 13.000 - 0.00% -
401.bzip2 (avg) -61.76% 34.000 13.000 - 0.00% -
433.milc (avg) (percentage) -61.76% 2.040 0.780 - -1.27% -
433.milc -60.87% 23.000 9.000 - 0.00% -
433.milc (avg) -60.87% 23.000 9.000 - 0.00% -
462.libquantum (avg) (percentage) -58.13% 1.600 0.670 - 3.08% -
462.libquantum -50.00% 2.000 1.000 - 0.00% -
462.libquantum (avg) -50.00% 2.000 1.000 - 0.00% -
453.povray -42.86% 7.000 4.000 - 0.00% -
453.povray (avg) -42.86% 7.000 4.000 - 0.00% -
447.dealII (avg) (percentage) -42.11% 1.140 0.660 - 0.00% -
447.dealII -40.74% 27.000 16.000 - 0.00% -
447.dealII (avg) -40.74% 27.000 16.000 - 0.00% -
453.povray (avg) (percentage) -39.33% 1.500 0.910 - 0.00% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 110.53% 19.000 40.000 - 0.00% -
nb_compile_successful 89.47% 19.000 36.000 - 0.00% -
nb_execution_successful 88.89% 18.000 34.000 - 0.00% -

Added Tests - sample variability
429.mcf (unexplained diff vs previous)
429.mcf (unexplained diff vs previous) (percentage)
433.milc (unexplained diff vs previous)
433.milc (unexplained diff vs previous) (percentage)
445.gobmk (unexplained diff vs previous)
445.gobmk (unexplained diff vs previous) (percentage)
450.soplex (unexplained diff vs previous)
450.soplex (unexplained diff vs previous) (percentage)
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)
462.libquantum (unexplained diff vs previous)
462.libquantum (unexplained diff vs previous) (percentage)
464.h264ref (unexplained diff vs previous)
464.h264ref (unexplained diff vs previous) (percentage)
470.lbm (unexplained diff vs previous)
470.lbm (unexplained diff vs previous) (percentage)
471.omnetpp (unexplained diff vs previous)
471.omnetpp (unexplained diff vs previous) (percentage)

Report Time: 0.34s
Execution (seconds)
Name Current %
401.bzip2 2980.000 -
403.gcc 59.000 -3.28%
429.mcf 1144.000 -
433.milc 1178.000 -
444.namd 794.000 -
445.gobmk 5341.000 -
447.dealII 2473.000 1.52%
450.soplex 559.000 1.27%
453.povray 515.000 -
456.hmmer 3612.000 -
458.sjeng 9479.000 -
462.libquantum 180.000 -1.64%
464.h264ref 5941.000 -
470.lbm 3906.000 -
471.omnetpp 3189.000 -
473.astar 5037.000 -
482.sphinx3 985.000 -
Geometric Mean 1544.452 -
Code Size (bytes)
Name Current %
401.bzip2 35185.000 -
403.gcc 1839401.000 -
429.mcf 7466.000 -
433.milc 69141.000 -
444.namd 153347.000 -
445.gobmk 1289114.000 -
447.dealII 1472639.000 -
450.soplex 180014.000 -
453.povray 543147.000 -
456.hmmer 156046.000 -
458.sjeng 89901.000 -
462.libquantum 20926.000 -
464.h264ref 330882.000 -
470.lbm 9706.000 -
471.omnetpp 363435.000 -
473.astar 21359.000 -
482.sphinx3 107751.000 -
Geometric Mean 128912.054 -
sample variability
Name Current %
401.bzip2 13.000 -
401.bzip2 (avg) 13.000 -
401.bzip2 (avg) (percentage) 0.440 -
401.bzip2 (max) 16.000 -11.11%
401.bzip2 (max) (percentage) 0.560 -9.68%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 3.200 -
403.gcc (max) 1.000 -
403.gcc (max) (percentage) 2.950 -1.01%
429.mcf 13.000 -
429.mcf (avg) 13.000 -
429.mcf (avg) (percentage) 1.200 -
429.mcf (max) 6.000 -
429.mcf (max) (percentage) 0.610 5.17%
429.mcf (unexplained diff vs previous) 7.000 75.00%
429.mcf (unexplained diff vs previous) (percentage) 0.610 74.29%
433.milc 9.000 -
433.milc (avg) 9.000 -
433.milc (avg) (percentage) 0.780 -1.27%
433.milc (max) 7.000 -12.50%
433.milc (max) (percentage) 0.630 -17.11%
433.milc (unexplained diff vs previous) 4.000 -55.56%
433.milc (unexplained diff vs previous) (percentage) 0.340 -55.84%
444.namd 7.000 -
444.namd (avg) 7.000 -
444.namd (avg) (percentage) 0.900 -
444.namd (max) 7.000 -
444.namd (max) (percentage) 0.940 -3.09%
445.gobmk 17.000 -
445.gobmk (avg) 17.000 -
445.gobmk (avg) (percentage) 0.320 -
445.gobmk (max) 21.000 5.00%
445.gobmk (max) (percentage) 0.410 5.13%
445.gobmk (unexplained diff vs previous) 22.000 175.00%
445.gobmk (unexplained diff vs previous) (percentage) 0.410 173.33%
447.dealII 16.000 -
447.dealII (avg) 16.000 -
447.dealII (avg) (percentage) 0.660 -
447.dealII (max) 22.000 -
447.dealII (max) (percentage) 0.910 -2.15%
450.soplex 7.000 -
450.soplex (avg) 7.000 -
450.soplex (avg) (percentage) 1.290 -
450.soplex (max) 9.000 -25.00%
450.soplex (max) (percentage) 1.720 -25.22%
450.soplex (unexplained diff vs previous) 7.000 -46.15%
450.soplex (unexplained diff vs previous) (percentage) 1.250 -47.03%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.910 -
453.povray (max) 6.000 -
453.povray (max) (percentage) 1.250 -7.41%
456.hmmer 22.000 -
456.hmmer (avg) 22.000 -
456.hmmer (avg) (percentage) 0.610 -
456.hmmer (max) 12.000 -7.69%
456.hmmer (max) (percentage) 0.350 -5.41%
458.sjeng 40.000 -
458.sjeng (avg) 40.000 -
458.sjeng (avg) (percentage) 0.430 -
458.sjeng (max) 46.000 39.39%
458.sjeng (max) (percentage) 0.490 40.00%
458.sjeng (unexplained diff vs previous) 46.000 411.11%
458.sjeng (unexplained diff vs previous) (percentage) 0.490 390.00%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 0.670 3.08%
462.libquantum (max) 2.000 -
462.libquantum (max) (percentage) 1.640 16.31%
462.libquantum (unexplained diff vs previous) 3.000 50.00%
462.libquantum (unexplained diff vs previous) (percentage) 1.670 53.21%
464.h264ref 22.000 -4.35%
464.h264ref (avg) 22.000 -4.35%
464.h264ref (avg) (percentage) 0.380 -2.56%
464.h264ref (max) 7.000 -12.50%
464.h264ref (max) (percentage) 0.130 -13.33%
464.h264ref (unexplained diff vs previous) 5.000 400.00%
464.h264ref (unexplained diff vs previous) (percentage) 0.080 300.00%
470.lbm 24.000 -4.00%
470.lbm (avg) 24.000 -4.00%
470.lbm (avg) (percentage) 0.640 -1.54%
470.lbm (max) 17.000 88.89%
470.lbm (max) (percentage) 0.460 100.00%
470.lbm (unexplained diff vs previous) 18.000 1700.00%
470.lbm (unexplained diff vs previous) (percentage) 0.460 1433.33%
471.omnetpp 38.000 -
471.omnetpp (avg) 38.000 -
471.omnetpp (avg) (percentage) 1.200 -
471.omnetpp (max) 31.000 -11.43%
471.omnetpp (max) (percentage) 0.980 -10.91%
471.omnetpp (unexplained diff vs previous) 30.000 42.86%
471.omnetpp (unexplained diff vs previous) (percentage) 0.940 44.62%
473.astar 39.000 -2.50%
473.astar (avg) 39.000 -2.50%
473.astar (avg) (percentage) 0.790 -2.47%
473.astar (max) 7.000 -12.50%
473.astar (max) (percentage) 0.150 -6.25%
473.astar (unexplained diff vs previous) 1.000 -66.67%
473.astar (unexplained diff vs previous) (percentage) 0.020 -66.67%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.790 -
482.sphinx3 (max) 2.000 -33.33%
482.sphinx3 (max) (percentage) 0.270 -12.90%
483.xalancbmk (avg) (percentage) 2.570 -
483.xalancbmk (max) (percentage) 0.350 -
Geometric Mean 2.960 10.62%