Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/cbdccb30c23f71f20d05b19256232419e7c5e517 (+ 1 commits)
regression LLVM-928
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Os-build/71/
View Options
Report
Filter

llvm-arm-master-Os test results

Run Order Start Time Duration
Current 0071 (llvmorg-18-init-05853-gcbdccb30c23f) 2023-09-13T20:14:15 0:00:00
Previous 0070 (llvmorg-18-init-05852-g01e80a0f4102) 2023-09-11T20:35:40 0:00:00
Baseline 0001 (llvmorg-16-init-16061-ge7790fbed32b) 2022-12-29T19:41:07 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 17 37
Performance Improvements 42 91
Unchanged Tests 721 652
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Code Size Δ Previous Current σ Δ (B) σ (B)
401.bzip2 8.96% 38377.000 41817.000 - 5.98% -

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
401.bzip2 2.78% 2701.000 2776.000 - -82.30% -
429.mcf 1.37% 1164.000 1180.000 - -90.17% -
462.libquantum 1.01% 99.000 100.000 - -99.24% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
464.h264ref (unexplained diff vs previous) (percentage) 220.00% 0.050 0.160 - -58.97% -
464.h264ref (unexplained diff vs previous) 200.00% 3.000 9.000 - -85.00% -
433.milc (unexplained diff vs previous) 100.00% 3.000 6.000 - -96.70% -
433.milc (unexplained diff vs previous) (percentage) 100.00% 0.260 0.520 - -61.48% -
470.lbm (max) (percentage) 29.63% 0.540 0.700 - -87.36% -
470.lbm (max) 29.41% 17.000 22.000 - -97.14% -
470.lbm (unexplained diff vs previous) 29.41% 17.000 22.000 - -95.57% -
470.lbm (unexplained diff vs previous) (percentage) 27.78% 0.540 0.690 - -80.73% -
464.h264ref (max) 14.29% 7.000 8.000 - -86.67% -
464.h264ref (max) (percentage) 14.29% 0.140 0.160 - -58.97% -
401.bzip2 5.56% 18.000 19.000 - -44.12% -
401.bzip2 (avg) 5.56% 18.000 19.000 - -44.12% -
458.sjeng (max) 2.94% 34.000 35.000 - -46.15% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
483.xalancbmk (unexplained diff vs previous) (percentage) -64.22% 1.090 0.390 - -40.00% -
483.xalancbmk (unexplained diff vs previous) -64.10% 39.000 14.000 - -77.05% -
471.omnetpp (unexplained diff vs previous) (percentage) -50.31% 1.590 0.790 - -8.14% -
450.soplex (unexplained diff vs previous) -50.00% 2.000 1.000 - -98.78% -
450.soplex (unexplained diff vs previous) (percentage) -50.00% 0.380 0.190 - -66.67% -
471.omnetpp (unexplained diff vs previous) -50.00% 44.000 22.000 - -76.09% -
471.omnetpp (max) -25.58% 43.000 32.000 - -83.07% -
471.omnetpp (max) (percentage) -24.84% 1.570 1.180 - -32.95% -
483.xalancbmk (max) -13.16% 38.000 33.000 - -45.90% -
483.xalancbmk (max) (percentage) -12.96% 1.080 0.940 - 44.62% -
429.mcf (max) (percentage) -12.73% 1.100 0.960 - 71.43% -
450.soplex (max) -12.50% 8.000 7.000 - -91.46% -
456.hmmer (max) -12.50% 16.000 14.000 - -65.00% -
456.hmmer (max) (percentage) -11.11% 0.450 0.400 - 33.33% -
401.bzip2 (max) (percentage) -10.17% 0.590 0.530 - 70.97% -
482.sphinx3 (max) -10.00% 10.000 9.000 - -98.97% -
482.sphinx3 (max) (percentage) -9.09% 1.210 1.100 - -66.87% -
429.mcf (max) -8.33% 12.000 11.000 - -83.58% -
444.namd (max) (percentage) -8.16% 0.490 0.450 - -93.12% -
401.bzip2 (max) -6.67% 15.000 14.000 - -70.83% -
450.soplex (max) (percentage) -6.17% 1.620 1.520 - 166.67% -
433.milc (max) (percentage) -5.45% 0.550 0.520 - -62.04% -
445.gobmk (max) (percentage) -5.26% 0.380 0.360 - 50.00% -
473.astar (max) -4.76% 21.000 20.000 - -82.46% -
473.astar (max) (percentage) -4.35% 0.460 0.440 - -45.00% -
462.libquantum (max) (percentage) -3.85% 1.560 1.500 - 650.00% -
483.xalancbmk -3.62% 138.000 133.000 - 182.98% -
483.xalancbmk (avg) -3.62% 138.000 133.000 - 182.98% -
447.dealII (max) (percentage) -3.57% 0.280 0.270 - -82.91% -
464.h264ref -3.57% 56.000 54.000 - 5.88% -
464.h264ref (avg) -3.57% 56.000 54.000 - 5.88% -
453.povray (max) (percentage) -3.54% 2.540 2.450 - 26.94% -
483.xalancbmk (avg) (percentage) -3.37% 3.860 3.730 - 646.00% -
450.soplex (avg) (percentage) -3.26% 0.920 0.890 - 147.22% -
464.h264ref (avg) (percentage) -2.94% 1.020 0.990 - 200.00% -
471.omnetpp (avg) (percentage) -2.35% 3.400 3.320 - 325.64% -
470.lbm (avg) (percentage) -2.02% 1.980 1.940 - -16.38% -
403.gcc (max) (percentage) -1.74% 5.170 5.080 - 1236.84% -
470.lbm -1.61% 62.000 61.000 - -81.06% -
470.lbm (avg) -1.61% 62.000 61.000 - -81.06% -
471.omnetpp -1.08% 93.000 92.000 - 10.84% -
471.omnetpp (avg) -1.08% 93.000 92.000 - 10.84% -

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
401.bzip2 5.98% 39458.000 41817.000 - 8.96% -
450.soplex 1.73% 198708.000 202143.000 - 0.09% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
403.gcc (max) (percentage) 1236.84% 0.380 5.080 - -1.74% -
462.libquantum (avg) (percentage) 926.67% 0.150 1.540 - 0.00% -
429.mcf (avg) (percentage) 746.67% 0.450 3.810 - 0.00% -
462.libquantum (max) (percentage) 650.00% 0.200 1.500 - -3.85% -
483.xalancbmk (avg) (percentage) 646.00% 0.500 3.730 - -3.37% -
473.astar (avg) (percentage) 504.00% 0.500 3.020 - 0.00% -
445.gobmk (avg) (percentage) 355.56% 0.180 0.820 - 0.00% -
403.gcc (avg) (percentage) 345.83% 0.480 2.140 - 0.00% -
471.omnetpp (avg) (percentage) 325.64% 0.780 3.320 - -2.35% -
456.hmmer (avg) (percentage) 319.44% 0.360 1.510 - 0.00% -
401.bzip2 (avg) (percentage) 213.64% 0.220 0.690 - 0.00% -
464.h264ref (avg) (percentage) 200.00% 0.330 0.990 - -2.94% -
483.xalancbmk 182.98% 47.000 133.000 - -3.62% -
483.xalancbmk (avg) 182.98% 47.000 133.000 - -3.62% -
450.soplex (max) (percentage) 166.67% 0.570 1.520 - -6.17% -
450.soplex (avg) (percentage) 147.22% 0.360 0.890 - -3.26% -
473.astar 101.41% 71.000 143.000 - 0.00% -
473.astar (avg) 101.41% 71.000 143.000 - 0.00% -
445.gobmk 95.45% 22.000 43.000 - 0.00% -
445.gobmk (avg) 95.45% 22.000 43.000 - 0.00% -
429.mcf (max) (percentage) 71.43% 0.560 0.960 - -12.73% -
401.bzip2 (max) (percentage) 70.97% 0.310 0.530 - -10.17% -
458.sjeng (avg) (percentage) 62.07% 0.580 0.940 - 0.00% -
445.gobmk (max) (percentage) 50.00% 0.240 0.360 - -5.26% -
483.xalancbmk (max) (percentage) 44.62% 0.650 0.940 - -12.96% -
456.hmmer (max) (percentage) 33.33% 0.300 0.400 - -11.11% -
453.povray (max) (percentage) 26.94% 1.930 2.450 - -3.54% -
471.omnetpp 10.84% 83.000 92.000 - -1.08% -
471.omnetpp (avg) 10.84% 83.000 92.000 - -1.08% -
456.hmmer 10.42% 48.000 53.000 - 0.00% -
456.hmmer (avg) 10.42% 48.000 53.000 - 0.00% -
458.sjeng (max) (percentage) 7.50% 0.400 0.430 - 0.00% -
453.povray (avg) (percentage) 6.57% 1.370 1.460 - 0.00% -
464.h264ref 5.88% 51.000 54.000 - -3.57% -
464.h264ref (avg) 5.88% 51.000 54.000 - -3.57% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -6.38% 26756.000 25049.000 - 0.45% -
453.povray -3.25% 593097.000 573843.000 - 0.05% -
433.milc -2.79% 74971.000 72883.000 - -0.01% -
447.dealII -2.02% 1777992.000 1742026.000 - 0.05% -
470.lbm -1.65% 9723.000 9563.000 - 0.00% -
482.sphinx3 -1.19% 115736.000 114363.000 - 0.07% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.33% 9099.000 61.000 - 0.00% -
462.libquantum -99.24% 13126.000 100.000 - 1.01% -
482.sphinx3 -96.67% 26310.000 876.000 - -0.68% -
450.soplex -96.38% 14429.000 523.000 - 0.19% -
444.namd -94.71% 13495.000 714.000 - 0.42% -
453.povray -93.31% 6857.000 459.000 - -0.43% -
433.milc -91.43% 13482.000 1155.000 - -0.52% -
429.mcf -90.17% 11999.000 1180.000 - 1.37% -
401.bzip2 -82.30% 15683.000 2776.000 - 2.78% -
447.dealII -78.57% 8196.000 1756.000 - -0.73% -
470.lbm -77.15% 13882.000 3172.000 - 0.70% -
471.omnetpp -74.09% 10746.000 2784.000 - 0.80% -
456.hmmer -73.30% 13365.000 3569.000 - 0.03% -
473.astar -66.65% 14262.000 4756.000 - -0.08% -
464.h264ref -64.42% 15493.000 5513.000 - -0.16% -
483.xalancbmk -61.92% 9427.000 3590.000 - 0.39% -
445.gobmk -57.40% 12400.000 5282.000 - 0.46% -
458.sjeng -50.34% 16406.000 8148.000 - 0.49% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -99.66% 882.000 3.000 - 0.00% -
482.sphinx3 (max) -98.97% 873.000 9.000 - -10.00% -
450.soplex (unexplained diff vs previous) -98.78% 82.000 1.000 - -50.00% -
403.gcc -97.67% 43.000 1.000 - 0.00% -
403.gcc (avg) -97.67% 43.000 1.000 - 0.00% -
470.lbm (max) -97.14% 769.000 22.000 - 29.41% -
482.sphinx3 -97.14% 489.000 14.000 - 0.00% -
482.sphinx3 (avg) -97.14% 489.000 14.000 - 0.00% -
444.namd -97.08% 514.000 15.000 - 0.00% -
444.namd (avg) -97.08% 514.000 15.000 - 0.00% -
447.dealII (max) -96.90% 129.000 4.000 - 0.00% -
433.milc (max) -96.74% 184.000 6.000 - 0.00% -
433.milc (unexplained diff vs previous) -96.70% 182.000 6.000 - 100.00% -
462.libquantum (max) -96.15% 26.000 1.000 - 0.00% -
470.lbm (unexplained diff vs previous) -95.57% 497.000 22.000 - 29.41% -
462.libquantum -94.74% 19.000 1.000 - 0.00% -
462.libquantum (avg) -94.74% 19.000 1.000 - 0.00% -
433.milc -94.12% 136.000 8.000 - 0.00% -
433.milc (avg) -94.12% 136.000 8.000 - 0.00% -
453.povray -93.55% 93.000 6.000 - 0.00% -
453.povray (avg) -93.55% 93.000 6.000 - 0.00% -
444.namd (max) (percentage) -93.12% 6.540 0.450 - -8.16% -
450.soplex -92.16% 51.000 4.000 - 0.00% -
450.soplex (avg) -92.16% 51.000 4.000 - 0.00% -
453.povray (max) -91.67% 132.000 11.000 - 0.00% -
450.soplex (max) -91.46% 82.000 7.000 - -12.50% -
403.gcc (max) -91.18% 34.000 3.000 - 0.00% -
470.lbm (max) (percentage) -87.36% 5.540 0.700 - 29.63% -
447.dealII -86.67% 120.000 16.000 - 0.00% -
447.dealII (avg) -86.67% 120.000 16.000 - 0.00% -
464.h264ref (max) -86.67% 60.000 8.000 - 14.29% -
464.h264ref (unexplained diff vs previous) -85.00% 60.000 9.000 - 200.00% -
429.mcf (max) -83.58% 67.000 11.000 - -8.33% -
471.omnetpp (max) -83.07% 189.000 32.000 - -25.58% -
447.dealII (max) (percentage) -82.91% 1.580 0.270 - -3.57% -
473.astar (max) -82.46% 114.000 20.000 - -4.76% -
470.lbm -81.06% 322.000 61.000 - -1.61% -
470.lbm (avg) -81.06% 322.000 61.000 - -1.61% -
470.lbm (unexplained diff vs previous) (percentage) -80.73% 3.580 0.690 - 27.78% -
483.xalancbmk (unexplained diff vs previous) -77.05% 61.000 14.000 - -64.10% -
471.omnetpp (unexplained diff vs previous) -76.09% 92.000 22.000 - -50.00% -
401.bzip2 (max) -70.83% 48.000 14.000 - -6.67% -
482.sphinx3 (max) (percentage) -66.87% 3.320 1.100 - -9.09% -
450.soplex (unexplained diff vs previous) (percentage) -66.67% 0.570 0.190 - -50.00% -
456.hmmer (max) -65.00% 40.000 14.000 - -12.50% -
433.milc (max) (percentage) -62.04% 1.370 0.520 - -5.45% -
433.milc (unexplained diff vs previous) (percentage) -61.48% 1.350 0.520 - 100.00% -
464.h264ref (max) (percentage) -58.97% 0.390 0.160 - 14.29% -
464.h264ref (unexplained diff vs previous) (percentage) -58.97% 0.390 0.160 - 220.00% -
458.sjeng (max) -46.15% 65.000 35.000 - 2.94% -
483.xalancbmk (max) -45.90% 61.000 33.000 - -13.16% -
473.astar (max) (percentage) -45.00% 0.800 0.440 - -4.35% -
401.bzip2 -44.12% 34.000 19.000 - 5.56% -
401.bzip2 (avg) -44.12% 34.000 19.000 - 5.56% -
444.namd (avg) (percentage) -41.21% 3.810 2.240 - 0.00% -
483.xalancbmk (unexplained diff vs previous) (percentage) -40.00% 0.650 0.390 - -64.22% -
447.dealII (avg) (percentage) -34.69% 1.470 0.960 - 0.00% -
445.gobmk (max) -34.48% 29.000 19.000 - 0.00% -
471.omnetpp (max) (percentage) -32.95% 1.760 1.180 - -24.84% -
433.milc (avg) (percentage) -27.72% 1.010 0.730 - 0.00% -
458.sjeng -20.00% 95.000 76.000 - 0.00% -
458.sjeng (avg) -20.00% 95.000 76.000 - 0.00% -
429.mcf -16.98% 53.000 44.000 - 0.00% -
429.mcf (avg) -16.98% 53.000 44.000 - 0.00% -
470.lbm (avg) (percentage) -16.38% 2.320 1.940 - -2.02% -
482.sphinx3 (avg) (percentage) -10.75% 1.860 1.660 - 0.00% -
471.omnetpp (unexplained diff vs previous) (percentage) -8.14% 0.860 0.790 - -50.31% -

Report Time: 0.35s
Execution (seconds)
Name Current %
401.bzip2 2776.000 2.78%
403.gcc 61.000 -
429.mcf 1180.000 1.37%
433.milc 1155.000 -
444.namd 714.000 -
445.gobmk 5282.000 -
447.dealII 1756.000 -
450.soplex 523.000 -
453.povray 459.000 -
456.hmmer 3569.000 -
458.sjeng 8148.000 -
462.libquantum 100.000 1.01%
464.h264ref 5513.000 -
470.lbm 3172.000 -
471.omnetpp 2784.000 -
473.astar 4756.000 -
482.sphinx3 876.000 -
483.xalancbmk 3590.000 -
Geometric Mean 1447.360 -
Code Size (bytes)
Name Current %
401.bzip2 41817.000 8.96%
403.gcc 1961341.000 -
429.mcf 7922.000 -
433.milc 72883.000 -
444.namd 174634.000 -
445.gobmk 1309912.000 -
447.dealII 1742026.000 -
450.soplex 202143.000 -
453.povray 573843.000 -
456.hmmer 166635.000 -
458.sjeng 97097.000 -
462.libquantum 25049.000 -
464.h264ref 352103.000 -
470.lbm 9563.000 -
471.omnetpp 376599.000 -
473.astar 23051.000 -
482.sphinx3 114363.000 -
483.xalancbmk 2287201.000 -
Geometric Mean 163251.506 -
sample variability
Name Current %
401.bzip2 19.000 5.56%
401.bzip2 (avg) 19.000 5.56%
401.bzip2 (avg) (percentage) 0.690 -
401.bzip2 (max) 14.000 -6.67%
401.bzip2 (max) (percentage) 0.530 -10.17%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 2.140 -
403.gcc (max) 3.000 -
403.gcc (max) (percentage) 5.080 -1.74%
429.mcf 44.000 -
429.mcf (avg) 44.000 -
429.mcf (avg) (percentage) 3.810 -
429.mcf (max) 11.000 -8.33%
429.mcf (max) (percentage) 0.960 -12.73%
433.milc 8.000 -
433.milc (avg) 8.000 -
433.milc (avg) (percentage) 0.730 -
433.milc (max) 6.000 -
433.milc (max) (percentage) 0.520 -5.45%
433.milc (unexplained diff vs previous) 6.000 100.00%
433.milc (unexplained diff vs previous) (percentage) 0.520 100.00%
444.namd 15.000 -
444.namd (avg) 15.000 -
444.namd (avg) (percentage) 2.240 -
444.namd (max) 3.000 -
444.namd (max) (percentage) 0.450 -8.16%
445.gobmk 43.000 -
445.gobmk (avg) 43.000 -
445.gobmk (avg) (percentage) 0.820 -
445.gobmk (max) 19.000 -
445.gobmk (max) (percentage) 0.360 -5.26%
447.dealII 16.000 -
447.dealII (avg) 16.000 -
447.dealII (avg) (percentage) 0.960 -
447.dealII (max) 4.000 -
447.dealII (max) (percentage) 0.270 -3.57%
450.soplex 4.000 -
450.soplex (avg) 4.000 -
450.soplex (avg) (percentage) 0.890 -3.26%
450.soplex (max) 7.000 -12.50%
450.soplex (max) (percentage) 1.520 -6.17%
450.soplex (unexplained diff vs previous) 1.000 -50.00%
450.soplex (unexplained diff vs previous) (percentage) 0.190 -50.00%
453.povray 6.000 -
453.povray (avg) 6.000 -
453.povray (avg) (percentage) 1.460 -
453.povray (max) 11.000 -
453.povray (max) (percentage) 2.450 -3.54%
456.hmmer 53.000 -
456.hmmer (avg) 53.000 -
456.hmmer (avg) (percentage) 1.510 -
456.hmmer (max) 14.000 -12.50%
456.hmmer (max) (percentage) 0.400 -11.11%
458.sjeng 76.000 -
458.sjeng (avg) 76.000 -
458.sjeng (avg) (percentage) 0.940 -
458.sjeng (max) 35.000 2.94%
458.sjeng (max) (percentage) 0.430 -
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.540 -
462.libquantum (max) 1.000 -
462.libquantum (max) (percentage) 1.500 -3.85%
464.h264ref 54.000 -3.57%
464.h264ref (avg) 54.000 -3.57%
464.h264ref (avg) (percentage) 0.990 -2.94%
464.h264ref (max) 8.000 14.29%
464.h264ref (max) (percentage) 0.160 14.29%
464.h264ref (unexplained diff vs previous) 9.000 200.00%
464.h264ref (unexplained diff vs previous) (percentage) 0.160 220.00%
470.lbm 61.000 -1.61%
470.lbm (avg) 61.000 -1.61%
470.lbm (avg) (percentage) 1.940 -2.02%
470.lbm (max) 22.000 29.41%
470.lbm (max) (percentage) 0.700 29.63%
470.lbm (unexplained diff vs previous) 22.000 29.41%
470.lbm (unexplained diff vs previous) (percentage) 0.690 27.78%
471.omnetpp 92.000 -1.08%
471.omnetpp (avg) 92.000 -1.08%
471.omnetpp (avg) (percentage) 3.320 -2.35%
471.omnetpp (max) 32.000 -25.58%
471.omnetpp (max) (percentage) 1.180 -24.84%
471.omnetpp (unexplained diff vs previous) 22.000 -50.00%
471.omnetpp (unexplained diff vs previous) (percentage) 0.790 -50.31%
473.astar 143.000 -
473.astar (avg) 143.000 -
473.astar (avg) (percentage) 3.020 -
473.astar (max) 20.000 -4.76%
473.astar (max) (percentage) 0.440 -4.35%
482.sphinx3 14.000 -
482.sphinx3 (avg) 14.000 -
482.sphinx3 (avg) (percentage) 1.660 -
482.sphinx3 (max) 9.000 -10.00%
482.sphinx3 (max) (percentage) 1.100 -9.09%
483.xalancbmk 133.000 -3.62%
483.xalancbmk (avg) 133.000 -3.62%
483.xalancbmk (avg) (percentage) 3.730 -3.37%
483.xalancbmk (max) 33.000 -13.16%
483.xalancbmk (max) (percentage) 0.940 -12.96%
483.xalancbmk (unexplained diff vs previous) 14.000 -64.10%
483.xalancbmk (unexplained diff vs previous) (percentage) 0.390 -64.22%
Geometric Mean 4.788 -2.35%