Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/74984dee51307779a3eab10a8cd6102be37e1081 (+ 358 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz-build/158/
View Options
Report
Filter

llvm-arm-master-Oz test results

Run Order Start Time Duration
Current 0158 (llvmorg-19-init-16581-g74984dee5130) 2024-07-01T11:29:30 0:00:00
Previous 0157 (llvmorg-19-init-16223-g54f040fff301) 2024-06-27T16:23:19 0:00:00
Baseline 0014 (no-change) 2023-02-19T08:08:29 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 14 11
Performance Improvements 23 98
Added Tests 0 8
Unchanged Tests 659 579
Total Tests 696

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc 5.08% 59.000 62.000 - 3.33% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
473.astar (unexplained diff vs previous) 1100.00% 1.000 12.000 0.000 -99.86% 0.000
473.astar (unexplained diff vs previous) (percentage) 1100.00% 0.020 0.240 0.000 -99.86% 0.000
433.milc (unexplained diff vs previous) 125.00% 4.000 9.000 0.000 0.00% 0.000
433.milc (unexplained diff vs previous) (percentage) 123.53% 0.340 0.760 0.000 0.00% 0.000
473.astar (max) 71.43% 7.000 12.000 - -99.62% -
473.astar (max) (percentage) 60.00% 0.150 0.240 - -99.62% -
433.milc (max) 28.57% 7.000 9.000 - -94.12% -
433.milc (max) (percentage) 20.63% 0.630 0.760 - -94.26% -
450.soplex (unexplained diff vs previous) (percentage) 16.00% 1.250 1.450 0.000 0.00% 0.000
450.soplex (unexplained diff vs previous) 14.29% 7.000 8.000 0.000 0.00% 0.000
470.lbm (max) 5.88% 17.000 18.000 - -95.94% -
470.lbm 4.17% 24.000 25.000 - -83.97% -
470.lbm (avg) 4.17% 24.000 25.000 - -83.97% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
450.soplex -1.43% 559.000 551.000 - -0.72% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) -56.67% 30.000 13.000 0.000 0.00% 0.000
471.omnetpp (unexplained diff vs previous) (percentage) -56.38% 0.940 0.410 0.000 0.00% 0.000
471.omnetpp (max) -29.03% 31.000 22.000 - -95.66% -
471.omnetpp (max) (percentage) -28.57% 0.980 0.700 - -95.70% -
450.soplex (max) -11.11% 9.000 8.000 - 14.29% -
450.soplex (max) (percentage) -11.05% 1.720 1.530 - 15.04% -
429.mcf (max) (percentage) -8.20% 0.610 0.560 - -97.01% -
464.h264ref (max) (percentage) -7.69% 0.130 0.120 - -97.52% -
401.bzip2 (max) (percentage) -7.14% 0.560 0.520 - -81.69% -
401.bzip2 (max) -6.25% 16.000 15.000 - -82.35% -
462.libquantum (max) (percentage) -5.49% 1.640 1.550 - -73.55% -
445.gobmk (max) (percentage) -4.88% 0.410 0.390 - -88.36% -
445.gobmk (max) -4.76% 21.000 20.000 - -89.01% -
456.hmmer -4.55% 22.000 21.000 - -83.97% -
456.hmmer (avg) -4.55% 22.000 21.000 - -83.97% -
458.sjeng (max) -4.35% 46.000 44.000 - -95.05% -
458.sjeng (max) (percentage) -4.08% 0.490 0.470 - -94.91% -
482.sphinx3 (max) (percentage) -3.70% 0.270 0.260 - -98.19% -
444.namd (max) (percentage) -3.19% 0.940 0.910 - -93.07% -
453.povray (max) (percentage) -2.40% 1.250 1.220 - -74.74% -
447.dealII (max) (percentage) -2.20% 0.910 0.890 - -81.30% -
403.gcc (max) (percentage) -1.36% 2.950 2.910 - -74.96% -

Run-Over-Baseline Changes Detail

Performance Regressions - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc 3.33% 60.000 62.000 - 5.08% -
471.omnetpp 2.76% 3116.000 3202.000 - 0.41% -
433.milc 2.15% 1162.000 1187.000 - 0.76% -
473.astar 1.90% 4955.000 5049.000 - 0.24% -
447.dealII 1.85% 2432.000 2477.000 - 0.16% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
450.soplex 133.33% 3.000 7.000 - 0.00% -
450.soplex (avg) 133.33% 3.000 7.000 - 0.00% -
450.soplex (avg) (percentage) 96.97% 0.660 1.300 - 0.78% -
403.gcc (avg) (percentage) 83.91% 1.740 3.200 - 0.00% -
450.soplex (max) (percentage) 15.04% 1.330 1.530 - -11.05% -
450.soplex (max) 14.29% 7.000 8.000 - -11.11% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -5.09% 22066.000 20942.000 - 0.08% -
401.bzip2 -1.69% 35781.000 35177.000 - -0.02% -
453.povray -1.21% 549774.000 543099.000 - -0.01% -
464.h264ref -1.21% 334968.000 330922.000 - 0.01% -
444.namd -1.06% 154855.000 153211.000 - -0.09% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
453.povray -2.08% 529.000 518.000 - 0.58% -
464.h264ref -1.72% 6043.000 5939.000 - -0.03% -
462.libquantum -1.63% 184.000 181.000 - 0.56% -
401.bzip2 -1.53% 3014.000 2968.000 - -0.40% -
444.namd -1.38% 799.000 788.000 - -0.76% -
458.sjeng -1.35% 9638.000 9508.000 - 0.31% -
429.mcf -1.21% 1155.000 1141.000 - -0.26% -
482.sphinx3 -1.21% 992.000 980.000 - -0.51% -
445.gobmk -1.01% 5439.000 5384.000 - 0.81% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
473.astar (unexplained diff vs previous) (percentage) -99.86% 173.660 0.240 0.000 1100.00% 0.000
473.astar (unexplained diff vs previous) -99.86% 8605.000 12.000 0.000 1100.00% 0.000
473.astar (max) (percentage) -99.62% 63.460 0.240 - 60.00% -
473.astar (max) -99.62% 3144.000 12.000 - 71.43% -
483.xalancbmk (max) (percentage) -99.26% 47.200 0.350 - 0.00% -
482.sphinx3 (max) -98.59% 142.000 2.000 - 0.00% -
482.sphinx3 (max) (percentage) -98.19% 14.360 0.260 - -3.70% -
464.h264ref (max) -97.60% 292.000 7.000 - 0.00% -
464.h264ref (max) (percentage) -97.52% 4.840 0.120 - -7.69% -
429.mcf (max) -97.22% 216.000 6.000 - 0.00% -
429.mcf (max) (percentage) -97.01% 18.730 0.560 - -8.20% -
456.hmmer (max) -96.27% 322.000 12.000 - 0.00% -
456.hmmer (max) (percentage) -96.18% 8.910 0.340 - -2.86% -
470.lbm (max) -95.94% 443.000 18.000 - 5.88% -
470.lbm (max) (percentage) -95.93% 11.290 0.460 - 0.00% -
471.omnetpp (max) (percentage) -95.70% 16.280 0.700 - -28.57% -
471.omnetpp (max) -95.66% 507.000 22.000 - -29.03% -
458.sjeng (max) -95.05% 889.000 44.000 - -4.35% -
458.sjeng (max) (percentage) -94.91% 9.230 0.470 - -4.08% -
433.milc (max) (percentage) -94.26% 13.240 0.760 - 20.63% -
433.milc (max) -94.12% 153.000 9.000 - 28.57% -
444.namd (max) -93.27% 104.000 7.000 - 0.00% -
444.namd (max) (percentage) -93.07% 13.140 0.910 - -3.19% -
445.gobmk (max) -89.01% 182.000 20.000 - -4.76% -
445.gobmk (max) (percentage) -88.36% 3.350 0.390 - -4.88% -
473.astar -84.27% 248.000 39.000 - 0.00% -
473.astar (avg) -84.27% 248.000 39.000 - 0.00% -
473.astar (avg) (percentage) -84.26% 5.020 0.790 - 0.00% -
470.lbm -83.97% 156.000 25.000 - 4.17% -
470.lbm (avg) -83.97% 156.000 25.000 - 4.17% -
456.hmmer -83.97% 131.000 21.000 - -4.55% -
456.hmmer (avg) -83.97% 131.000 21.000 - -4.55% -
470.lbm (avg) (percentage) -83.92% 3.980 0.640 - 0.00% -
403.gcc (max) -83.33% 6.000 1.000 - 0.00% -
456.hmmer (avg) (percentage) -83.24% 3.640 0.610 - 0.00% -
401.bzip2 (max) -82.35% 85.000 15.000 - -6.25% -
401.bzip2 (max) (percentage) -81.69% 2.840 0.520 - -7.14% -
464.h264ref -81.67% 120.000 22.000 - 0.00% -
464.h264ref (avg) -81.67% 120.000 22.000 - 0.00% -
447.dealII (max) (percentage) -81.30% 4.760 0.890 - -2.20% -
464.h264ref (avg) (percentage) -81.00% 2.000 0.380 - 0.00% -
447.dealII (max) -80.87% 115.000 22.000 - 0.00% -
462.libquantum (max) -80.00% 10.000 2.000 - 0.00% -
444.namd -79.41% 34.000 7.000 - 0.00% -
444.namd (avg) -79.41% 34.000 7.000 - 0.00% -
482.sphinx3 -79.41% 34.000 7.000 - 0.00% -
482.sphinx3 (avg) -79.41% 34.000 7.000 - 0.00% -
444.namd (avg) (percentage) -78.87% 4.260 0.900 - 0.00% -
482.sphinx3 (avg) (percentage) -77.17% 3.460 0.790 - 0.00% -
429.mcf -76.36% 55.000 13.000 - 0.00% -
429.mcf (avg) -76.36% 55.000 13.000 - 0.00% -
445.gobmk -76.06% 71.000 17.000 - 0.00% -
445.gobmk (avg) -76.06% 71.000 17.000 - 0.00% -
453.povray (max) -76.00% 25.000 6.000 - 0.00% -
445.gobmk (avg) (percentage) -75.57% 1.310 0.320 - 0.00% -
403.gcc (max) (percentage) -74.96% 11.620 2.910 - -1.36% -
429.mcf (avg) (percentage) -74.84% 4.770 1.200 - 0.00% -
458.sjeng -74.84% 159.000 40.000 - 0.00% -
458.sjeng (avg) -74.84% 159.000 40.000 - 0.00% -
453.povray (max) (percentage) -74.74% 4.830 1.220 - -2.40% -
458.sjeng (avg) (percentage) -73.94% 1.650 0.430 - 0.00% -
462.libquantum (max) (percentage) -73.55% 5.860 1.550 - -5.49% -
471.omnetpp (avg) (percentage) -66.38% 3.540 1.190 - -0.83% -
471.omnetpp -65.45% 110.000 38.000 - 0.00% -
471.omnetpp (avg) -65.45% 110.000 38.000 - 0.00% -
483.xalancbmk (avg) (percentage) -62.26% 6.810 2.570 - 0.00% -
401.bzip2 (avg) (percentage) -62.07% 1.160 0.440 - 0.00% -
401.bzip2 -61.76% 34.000 13.000 - 0.00% -
401.bzip2 (avg) -61.76% 34.000 13.000 - 0.00% -
433.milc (avg) (percentage) -61.76% 2.040 0.780 - 0.00% -
433.milc -60.87% 23.000 9.000 - 0.00% -
433.milc (avg) -60.87% 23.000 9.000 - 0.00% -
462.libquantum (avg) (percentage) -58.13% 1.600 0.670 - 0.00% -
462.libquantum -50.00% 2.000 1.000 - 0.00% -
462.libquantum (avg) -50.00% 2.000 1.000 - 0.00% -
453.povray -42.86% 7.000 4.000 - 0.00% -
453.povray (avg) -42.86% 7.000 4.000 - 0.00% -
447.dealII (avg) (percentage) -42.11% 1.140 0.660 - 0.00% -
447.dealII -40.74% 27.000 16.000 - 0.00% -
447.dealII (avg) -40.74% 27.000 16.000 - 0.00% -
453.povray (avg) (percentage) -39.33% 1.500 0.910 - 0.00% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 110.53% 19.000 40.000 - 0.00% -
nb_compile_successful 89.47% 19.000 36.000 - 0.00% -
nb_execution_successful 88.89% 18.000 34.000 - 0.00% -

Added Tests - sample variability
433.milc (unexplained diff vs previous)
433.milc (unexplained diff vs previous) (percentage)
450.soplex (unexplained diff vs previous)
450.soplex (unexplained diff vs previous) (percentage)
470.lbm (unexplained diff vs previous)
470.lbm (unexplained diff vs previous) (percentage)
471.omnetpp (unexplained diff vs previous)
471.omnetpp (unexplained diff vs previous) (percentage)

Report Time: 0.23s
Execution (seconds)
Name Current %
401.bzip2 2968.000 -
403.gcc 62.000 5.08%
429.mcf 1141.000 -
433.milc 1187.000 -
444.namd 788.000 -
445.gobmk 5384.000 -
447.dealII 2477.000 -
450.soplex 551.000 -1.43%
453.povray 518.000 -
456.hmmer 3602.000 -
458.sjeng 9508.000 -
462.libquantum 181.000 -
464.h264ref 5939.000 -
470.lbm 3924.000 -
471.omnetpp 3202.000 -
473.astar 5049.000 -
482.sphinx3 980.000 -
Geometric Mean 1549.496 -
Code Size (bytes)
Name Current %
401.bzip2 35177.000 -
403.gcc 1839225.000 -
429.mcf 7466.000 -
433.milc 69149.000 -
444.namd 153211.000 -
445.gobmk 1289306.000 -
447.dealII 1472651.000 -
450.soplex 180054.000 -
453.povray 543099.000 -
456.hmmer 156030.000 -
458.sjeng 90005.000 -
462.libquantum 20942.000 -
464.h264ref 330922.000 -
470.lbm 9706.000 -
471.omnetpp 363427.000 -
473.astar 21359.000 -
482.sphinx3 107767.000 -
Geometric Mean 128921.621 -
sample variability
Name Current %
401.bzip2 13.000 -
401.bzip2 (avg) 13.000 -
401.bzip2 (avg) (percentage) 0.440 -
401.bzip2 (max) 15.000 -6.25%
401.bzip2 (max) (percentage) 0.520 -7.14%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 3.200 -
403.gcc (max) 1.000 -
403.gcc (max) (percentage) 2.910 -1.36%
429.mcf 13.000 -
429.mcf (avg) 13.000 -
429.mcf (avg) (percentage) 1.200 -
429.mcf (max) 6.000 -
429.mcf (max) (percentage) 0.560 -8.20%
433.milc 9.000 -
433.milc (avg) 9.000 -
433.milc (avg) (percentage) 0.780 -
433.milc (max) 9.000 28.57%
433.milc (max) (percentage) 0.760 20.63%
433.milc (unexplained diff vs previous) 9.000 125.00%
433.milc (unexplained diff vs previous) (percentage) 0.760 123.53%
444.namd 7.000 -
444.namd (avg) 7.000 -
444.namd (avg) (percentage) 0.900 -
444.namd (max) 7.000 -
444.namd (max) (percentage) 0.910 -3.19%
445.gobmk 17.000 -
445.gobmk (avg) 17.000 -
445.gobmk (avg) (percentage) 0.320 -
445.gobmk (max) 20.000 -4.76%
445.gobmk (max) (percentage) 0.390 -4.88%
447.dealII 16.000 -
447.dealII (avg) 16.000 -
447.dealII (avg) (percentage) 0.660 -
447.dealII (max) 22.000 -
447.dealII (max) (percentage) 0.890 -2.20%
450.soplex 7.000 -
450.soplex (avg) 7.000 -
450.soplex (avg) (percentage) 1.300 -
450.soplex (max) 8.000 -11.11%
450.soplex (max) (percentage) 1.530 -11.05%
450.soplex (unexplained diff vs previous) 8.000 14.29%
450.soplex (unexplained diff vs previous) (percentage) 1.450 16.00%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.910 -
453.povray (max) 6.000 -
453.povray (max) (percentage) 1.220 -2.40%
456.hmmer 21.000 -4.55%
456.hmmer (avg) 21.000 -4.55%
456.hmmer (avg) (percentage) 0.610 -
456.hmmer (max) 12.000 -
456.hmmer (max) (percentage) 0.340 -
458.sjeng 40.000 -
458.sjeng (avg) 40.000 -
458.sjeng (avg) (percentage) 0.430 -
458.sjeng (max) 44.000 -4.35%
458.sjeng (max) (percentage) 0.470 -4.08%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 0.670 -
462.libquantum (max) 2.000 -
462.libquantum (max) (percentage) 1.550 -5.49%
464.h264ref 22.000 -
464.h264ref (avg) 22.000 -
464.h264ref (avg) (percentage) 0.380 -
464.h264ref (max) 7.000 -
464.h264ref (max) (percentage) 0.120 -7.69%
470.lbm 25.000 4.17%
470.lbm (avg) 25.000 4.17%
470.lbm (avg) (percentage) 0.640 -
470.lbm (max) 18.000 5.88%
470.lbm (max) (percentage) 0.460 -
470.lbm (unexplained diff vs previous) 18.000 -
470.lbm (unexplained diff vs previous) (percentage) 0.460 -
471.omnetpp 38.000 -
471.omnetpp (avg) 38.000 -
471.omnetpp (avg) (percentage) 1.190 -
471.omnetpp (max) 22.000 -29.03%
471.omnetpp (max) (percentage) 0.700 -28.57%
471.omnetpp (unexplained diff vs previous) 13.000 -56.67%
471.omnetpp (unexplained diff vs previous) (percentage) 0.410 -56.38%
473.astar 39.000 -
473.astar (avg) 39.000 -
473.astar (avg) (percentage) 0.790 -
473.astar (max) 12.000 71.43%
473.astar (max) (percentage) 0.240 60.00%
473.astar (unexplained diff vs previous) 12.000 1100.00%
473.astar (unexplained diff vs previous) (percentage) 0.240 1100.00%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.790 -
482.sphinx3 (max) 2.000 -
482.sphinx3 (max) (percentage) 0.260 -3.70%
483.xalancbmk (avg) (percentage) 2.570 -
483.xalancbmk (max) (percentage) 0.350 -
Geometric Mean 3.230 5.37%