Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/675231eb09ca37a8b76f748c0b73a1e26604ff20 (+ 878 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Os-build/91/
View Options
Report
Filter

llvm-arm-master-Os test results

Run Order Start Time Duration
Current 0091 (llvmorg-18-init-08811-g675231eb09ca) 2023-10-11T22:07:58 0:00:00
Previous 0090 (llvmorg-18-init-07933-ge13bed4c5f35) 2023-10-05T10:35:50 0:00:00
Baseline 0001 (llvmorg-16-init-16061-ge7790fbed32b) 2022-12-29T19:41:07 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 18 33
Performance Improvements 37 94
Added Tests 4 0
Unchanged Tests 721 653
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
464.h264ref 4.54% 5393.000 5638.000 - -63.61% -
403.gcc 1.69% 59.000 60.000 - -99.34% -
458.sjeng 1.23% 8103.000 8203.000 - -50.00% -
471.omnetpp 1.16% 2758.000 2790.000 - -74.04% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
464.h264ref (max) 1316.67% 18.000 255.000 - 325.00% -
464.h264ref (max) (percentage) 1197.14% 0.350 4.540 - 1064.10% -
433.milc (unexplained diff vs previous) (percentage) 542.31% 0.260 1.670 - 23.70% -
433.milc (unexplained diff vs previous) 533.33% 3.000 19.000 - -89.56% -
450.soplex (unexplained diff vs previous) (percentage) 515.79% 0.190 1.170 - 105.26% -
450.soplex (unexplained diff vs previous) 500.00% 1.000 6.000 - -92.68% -
433.milc (max) 350.00% 4.000 18.000 - -90.22% -
433.milc (max) (percentage) 290.48% 0.420 1.640 - 19.71% -
464.h264ref 19.05% 42.000 50.000 - -1.96% -
464.h264ref (avg) 19.05% 42.000 50.000 - -1.96% -
464.h264ref (avg) (percentage) 13.92% 0.790 0.900 - 172.73% -
433.milc (avg) (percentage) 4.76% 0.630 0.660 - -34.65% -
458.sjeng 1.61% 62.000 63.000 - -33.68% -
458.sjeng (avg) 1.61% 62.000 63.000 - -33.68% -

Performance Improvements - Code Size Δ Previous Current σ Δ (B) σ (B)
401.bzip2 -8.23% 41777.000 38337.000 - -2.84% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
401.bzip2 -2.62% 2752.000 2680.000 - -82.91% -
433.milc -1.64% 1157.000 1138.000 - -91.56% -
429.mcf -1.20% 1170.000 1156.000 - -90.37% -
450.soplex -1.16% 519.000 513.000 - -96.44% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
450.soplex (max) -20.00% 10.000 8.000 - -90.24% -
483.xalancbmk (unexplained diff vs previous) -20.00% 20.000 16.000 - -73.77% -
483.xalancbmk (unexplained diff vs previous) (percentage) -19.64% 0.560 0.450 - -30.77% -
401.bzip2 (max) -14.29% 14.000 12.000 - -75.00% -
453.povray (max) -14.29% 7.000 6.000 - -95.45% -
444.namd (max) (percentage) -12.23% 1.390 1.220 - -81.35% -
450.soplex (max) (percentage) -11.17% 1.970 1.750 - 207.02% -
444.namd (max) -11.11% 9.000 8.000 - -99.09% -
429.mcf (max) (percentage) -10.00% 0.200 0.180 - -67.86% -
458.sjeng (max) (percentage) -8.05% 0.870 0.800 - 100.00% -
401.bzip2 (max) (percentage) -7.69% 0.520 0.480 - 54.84% -
483.xalancbmk (max) -7.41% 27.000 25.000 - -59.02% -
458.sjeng (max) -7.14% 70.000 65.000 - 0.00% -
473.astar (max) -7.14% 14.000 13.000 - -88.60% -
473.astar (max) (percentage) -6.67% 0.300 0.280 - -65.00% -
483.xalancbmk (max) (percentage) -6.58% 0.760 0.710 - 9.23% -
470.lbm (max) (percentage) -6.41% 0.780 0.730 - -86.82% -
401.bzip2 -5.88% 17.000 16.000 - -52.94% -
401.bzip2 (avg) -5.88% 17.000 16.000 - -52.94% -
453.povray (max) (percentage) -5.37% 1.490 1.410 - -26.94% -
447.dealII (max) (percentage) -4.82% 0.830 0.790 - -50.00% -
470.lbm (max) -4.17% 24.000 23.000 - -97.01% -
482.sphinx3 (max) (percentage) -4.08% 0.980 0.940 - -71.69% -
403.gcc (max) (percentage) -4.07% 4.910 4.710 - 1139.47% -
462.libquantum (max) (percentage) -4.00% 1.250 1.200 - 500.00% -
483.xalancbmk (avg) (percentage) -2.43% 2.880 2.810 - 462.00% -
470.lbm -2.00% 50.000 49.000 - -84.78% -
470.lbm (avg) -2.00% 50.000 49.000 - -84.78% -
483.xalancbmk -1.96% 102.000 100.000 - 112.77% -
483.xalancbmk (avg) -1.96% 102.000 100.000 - 112.77% -
471.omnetpp (avg) (percentage) -1.44% 2.770 2.730 - 250.00% -
470.lbm (avg) (percentage) -1.26% 1.590 1.570 - -32.33% -

Added Tests - sample variability
464.h264ref (unexplained diff vs previous)
464.h264ref (unexplained diff vs previous) (percentage)
470.lbm (unexplained diff vs previous)
470.lbm (unexplained diff vs previous) (percentage)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.67% 198708.000 202018.000 - -0.11% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
403.gcc (max) (percentage) 1139.47% 0.380 4.710 - -4.07% -
464.h264ref (max) (percentage) 1064.10% 0.390 4.540 - 1197.14% -
464.h264ref (unexplained diff vs previous) (percentage) 1015.38% 0.390 4.350 - 0.00% -
462.libquantum (avg) (percentage) 846.67% 0.150 1.420 - 0.00% -
429.mcf (avg) (percentage) 562.22% 0.450 2.980 - 0.00% -
462.libquantum (max) (percentage) 500.00% 0.200 1.200 - -4.00% -
403.gcc (avg) (percentage) 470.83% 0.480 2.740 - 0.00% -
483.xalancbmk (avg) (percentage) 462.00% 0.500 2.810 - -2.43% -
473.astar (avg) (percentage) 364.00% 0.500 2.320 - 0.00% -
464.h264ref (max) 325.00% 60.000 255.000 - 1316.67% -
464.h264ref (unexplained diff vs previous) 308.33% 60.000 245.000 - 0.00% -
471.omnetpp (avg) (percentage) 250.00% 0.780 2.730 - -1.44% -
445.gobmk (avg) (percentage) 244.44% 0.180 0.620 - 0.00% -
456.hmmer (avg) (percentage) 238.89% 0.360 1.220 - 0.00% -
450.soplex (max) (percentage) 207.02% 0.570 1.750 - -11.17% -
450.soplex (avg) (percentage) 191.67% 0.360 1.050 - 0.00% -
401.bzip2 (avg) (percentage) 181.82% 0.220 0.620 - 0.00% -
464.h264ref (avg) (percentage) 172.73% 0.330 0.900 - 13.92% -
483.xalancbmk 112.77% 47.000 100.000 - -1.96% -
483.xalancbmk (avg) 112.77% 47.000 100.000 - -1.96% -
450.soplex (unexplained diff vs previous) (percentage) 105.26% 0.570 1.170 - 515.79% -
458.sjeng (max) (percentage) 100.00% 0.400 0.800 - -8.05% -
473.astar 59.15% 71.000 113.000 - 0.00% -
473.astar (avg) 59.15% 71.000 113.000 - 0.00% -
401.bzip2 (max) (percentage) 54.84% 0.310 0.480 - -7.69% -
445.gobmk 45.45% 22.000 32.000 - 0.00% -
445.gobmk (avg) 45.45% 22.000 32.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) (percentage) 33.72% 0.860 1.150 - -0.86% -
458.sjeng (avg) (percentage) 32.76% 0.580 0.770 - 0.00% -
433.milc (unexplained diff vs previous) (percentage) 23.70% 1.350 1.670 - 542.31% -
433.milc (max) (percentage) 19.71% 1.370 1.640 - 290.48% -
483.xalancbmk (max) (percentage) 9.23% 0.650 0.710 - -6.58% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -9.36% 26756.000 24251.000 - -0.43% -
401.bzip2 -2.84% 39458.000 38337.000 - -8.23% -
453.povray -2.74% 593097.000 576846.000 - -0.08% -
433.milc -2.49% 74971.000 73102.000 - -0.01% -
447.dealII -1.80% 1777992.000 1745927.000 - -0.05% -
482.sphinx3 -1.20% 115736.000 114347.000 - -0.08% -
456.hmmer -1.06% 168072.000 166298.000 - -0.07% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.34% 9099.000 60.000 - 1.69% -
462.libquantum -99.25% 13126.000 99.000 - 0.00% -
482.sphinx3 -96.64% 26310.000 884.000 - 0.68% -
450.soplex -96.44% 14429.000 513.000 - -1.16% -
444.namd -94.72% 13495.000 712.000 - 0.14% -
453.povray -93.13% 6857.000 471.000 - 0.21% -
433.milc -91.56% 13482.000 1138.000 - -1.64% -
429.mcf -90.37% 11999.000 1156.000 - -1.20% -
401.bzip2 -82.91% 15683.000 2680.000 - -2.62% -
447.dealII -78.10% 8196.000 1795.000 - 0.11% -
470.lbm -77.14% 13882.000 3173.000 - 0.73% -
471.omnetpp -74.04% 10746.000 2790.000 - 1.16% -
456.hmmer -73.29% 13365.000 3570.000 - 0.28% -
473.astar -65.79% 14262.000 4879.000 - -0.06% -
464.h264ref -63.61% 15493.000 5638.000 - 4.54% -
483.xalancbmk -61.92% 9427.000 3590.000 - 0.45% -
445.gobmk -57.53% 12400.000 5266.000 - 0.34% -
458.sjeng -50.00% 16406.000 8203.000 - 1.23% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -99.09% 882.000 8.000 - -11.11% -
482.sphinx3 (max) -99.08% 873.000 8.000 - 0.00% -
403.gcc -97.67% 43.000 1.000 - 0.00% -
403.gcc (avg) -97.67% 43.000 1.000 - 0.00% -
482.sphinx3 -97.55% 489.000 12.000 - 0.00% -
482.sphinx3 (avg) -97.55% 489.000 12.000 - 0.00% -
444.namd -97.28% 514.000 14.000 - 0.00% -
444.namd (avg) -97.28% 514.000 14.000 - 0.00% -
429.mcf (max) -97.01% 67.000 2.000 - 0.00% -
470.lbm (max) -97.01% 769.000 23.000 - -4.17% -
462.libquantum (max) -96.15% 26.000 1.000 - 0.00% -
453.povray (max) -95.45% 132.000 6.000 - -14.29% -
470.lbm (unexplained diff vs previous) -95.37% 497.000 23.000 - 0.00% -
433.milc -94.85% 136.000 7.000 - 0.00% -
433.milc (avg) -94.85% 136.000 7.000 - 0.00% -
462.libquantum -94.74% 19.000 1.000 - 0.00% -
462.libquantum (avg) -94.74% 19.000 1.000 - 0.00% -
453.povray -94.62% 93.000 5.000 - 0.00% -
453.povray (avg) -94.62% 93.000 5.000 - 0.00% -
403.gcc (max) -94.12% 34.000 2.000 - 0.00% -
450.soplex (unexplained diff vs previous) -92.68% 82.000 6.000 - 500.00% -
456.hmmer (max) -92.50% 40.000 3.000 - 0.00% -
450.soplex (max) -90.24% 82.000 8.000 - -20.00% -
433.milc (max) -90.22% 184.000 18.000 - 350.00% -
450.soplex -90.20% 51.000 5.000 - 0.00% -
450.soplex (avg) -90.20% 51.000 5.000 - 0.00% -
433.milc (unexplained diff vs previous) -89.56% 182.000 19.000 - 533.33% -
447.dealII (max) -89.15% 129.000 14.000 - 0.00% -
473.astar (max) -88.60% 114.000 13.000 - -7.14% -
447.dealII -87.50% 120.000 15.000 - 0.00% -
447.dealII (avg) -87.50% 120.000 15.000 - 0.00% -
470.lbm (max) (percentage) -86.82% 5.540 0.730 - -6.41% -
470.lbm -84.78% 322.000 49.000 - -2.00% -
470.lbm (avg) -84.78% 322.000 49.000 - -2.00% -
471.omnetpp (max) -83.07% 189.000 32.000 - 0.00% -
444.namd (max) (percentage) -81.35% 6.540 1.220 - -12.23% -
470.lbm (unexplained diff vs previous) (percentage) -79.89% 3.580 0.720 - 0.00% -
401.bzip2 (max) -75.00% 48.000 12.000 - -14.29% -
483.xalancbmk (unexplained diff vs previous) -73.77% 61.000 16.000 - -20.00% -
445.gobmk (max) -72.41% 29.000 8.000 - 0.00% -
482.sphinx3 (max) (percentage) -71.69% 3.320 0.940 - -4.08% -
429.mcf (max) (percentage) -67.86% 0.560 0.180 - -10.00% -
471.omnetpp (unexplained diff vs previous) -65.22% 92.000 32.000 - 0.00% -
473.astar (max) (percentage) -65.00% 0.800 0.280 - -6.67% -
456.hmmer (max) (percentage) -63.33% 0.300 0.110 - 0.00% -
483.xalancbmk (max) -59.02% 61.000 25.000 - -7.41% -
401.bzip2 -52.94% 34.000 16.000 - -5.88% -
401.bzip2 (avg) -52.94% 34.000 16.000 - -5.88% -
447.dealII (max) (percentage) -50.00% 1.580 0.790 - -4.82% -
444.namd (avg) (percentage) -47.77% 3.810 1.990 - 0.00% -
447.dealII (avg) (percentage) -42.86% 1.470 0.840 - 0.00% -
429.mcf -35.85% 53.000 34.000 - 0.00% -
429.mcf (avg) -35.85% 53.000 34.000 - 0.00% -
433.milc (avg) (percentage) -34.65% 1.010 0.660 - 4.76% -
471.omnetpp (max) (percentage) -34.09% 1.760 1.160 - -0.85% -
458.sjeng -33.68% 95.000 63.000 - 1.61% -
458.sjeng (avg) -33.68% 95.000 63.000 - 1.61% -
470.lbm (avg) (percentage) -32.33% 2.320 1.570 - -1.26% -
483.xalancbmk (unexplained diff vs previous) (percentage) -30.77% 0.650 0.450 - -19.64% -
445.gobmk (max) (percentage) -29.17% 0.240 0.170 - 0.00% -
453.povray (max) (percentage) -26.94% 1.930 1.410 - -5.37% -
482.sphinx3 (avg) (percentage) -22.58% 1.860 1.440 - 0.00% -
456.hmmer -10.42% 48.000 43.000 - 0.00% -
456.hmmer (avg) -10.42% 48.000 43.000 - 0.00% -
471.omnetpp -8.43% 83.000 76.000 - 0.00% -
471.omnetpp (avg) -8.43% 83.000 76.000 - 0.00% -
453.povray (avg) (percentage) -8.03% 1.370 1.260 - 0.00% -
464.h264ref -1.96% 51.000 50.000 - 19.05% -
464.h264ref (avg) -1.96% 51.000 50.000 - 19.05% -

Report Time: 0.33s
Execution (seconds)
Name Current %
401.bzip2 2680.000 -2.62%
403.gcc 60.000 1.69%
429.mcf 1156.000 -1.20%
433.milc 1138.000 -1.64%
444.namd 712.000 -
445.gobmk 5266.000 -
447.dealII 1795.000 -
450.soplex 513.000 -1.16%
453.povray 471.000 -
456.hmmer 3570.000 -
458.sjeng 8203.000 1.23%
462.libquantum 99.000 -
464.h264ref 5638.000 4.54%
470.lbm 3173.000 -
471.omnetpp 2790.000 1.16%
473.astar 4879.000 -
482.sphinx3 884.000 -
483.xalancbmk 3590.000 -
Geometric Mean 1446.717 -
Code Size (bytes)
Name Current %
401.bzip2 38337.000 -8.23%
403.gcc 1957377.000 -
429.mcf 7846.000 -
433.milc 73102.000 -
444.namd 174865.000 -
445.gobmk 1310280.000 -
447.dealII 1745927.000 -
450.soplex 202018.000 -
453.povray 576846.000 -
456.hmmer 166298.000 -
458.sjeng 97129.000 -
462.libquantum 24251.000 -
464.h264ref 354558.000 -
470.lbm 9718.000 -
471.omnetpp 376723.000 -
473.astar 23367.000 -
482.sphinx3 114347.000 -
483.xalancbmk 2285568.000 -
Geometric Mean 162481.842 -
sample variability
Name Current %
401.bzip2 16.000 -5.88%
401.bzip2 (avg) 16.000 -5.88%
401.bzip2 (avg) (percentage) 0.620 -
401.bzip2 (max) 12.000 -14.29%
401.bzip2 (max) (percentage) 0.480 -7.69%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 2.740 -
403.gcc (max) 2.000 -
403.gcc (max) (percentage) 4.710 -4.07%
429.mcf 34.000 -
429.mcf (avg) 34.000 -
429.mcf (avg) (percentage) 2.980 -
429.mcf (max) 2.000 -
429.mcf (max) (percentage) 0.180 -10.00%
433.milc 7.000 -
433.milc (avg) 7.000 -
433.milc (avg) (percentage) 0.660 4.76%
433.milc (max) 18.000 350.00%
433.milc (max) (percentage) 1.640 290.48%
433.milc (unexplained diff vs previous) 19.000 533.33%
433.milc (unexplained diff vs previous) (percentage) 1.670 542.31%
444.namd 14.000 -
444.namd (avg) 14.000 -
444.namd (avg) (percentage) 1.990 -
444.namd (max) 8.000 -11.11%
444.namd (max) (percentage) 1.220 -12.23%
445.gobmk 32.000 -
445.gobmk (avg) 32.000 -
445.gobmk (avg) (percentage) 0.620 -
445.gobmk (max) 8.000 -
445.gobmk (max) (percentage) 0.170 -
447.dealII 15.000 -
447.dealII (avg) 15.000 -
447.dealII (avg) (percentage) 0.840 -
447.dealII (max) 14.000 -
447.dealII (max) (percentage) 0.790 -4.82%
450.soplex 5.000 -
450.soplex (avg) 5.000 -
450.soplex (avg) (percentage) 1.050 -
450.soplex (max) 8.000 -20.00%
450.soplex (max) (percentage) 1.750 -11.17%
450.soplex (unexplained diff vs previous) 6.000 500.00%
450.soplex (unexplained diff vs previous) (percentage) 1.170 515.79%
453.povray 5.000 -
453.povray (avg) 5.000 -
453.povray (avg) (percentage) 1.260 -
453.povray (max) 6.000 -14.29%
453.povray (max) (percentage) 1.410 -5.37%
456.hmmer 43.000 -
456.hmmer (avg) 43.000 -
456.hmmer (avg) (percentage) 1.220 -
456.hmmer (max) 3.000 -
456.hmmer (max) (percentage) 0.110 -
458.sjeng 63.000 1.61%
458.sjeng (avg) 63.000 1.61%
458.sjeng (avg) (percentage) 0.770 -
458.sjeng (max) 65.000 -7.14%
458.sjeng (max) (percentage) 0.800 -8.05%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.420 -
462.libquantum (max) 1.000 -
462.libquantum (max) (percentage) 1.200 -4.00%
464.h264ref 50.000 19.05%
464.h264ref (avg) 50.000 19.05%
464.h264ref (avg) (percentage) 0.900 13.92%
464.h264ref (max) 255.000 1316.67%
464.h264ref (max) (percentage) 4.540 1197.14%
464.h264ref (unexplained diff vs previous) 245.000 -
464.h264ref (unexplained diff vs previous) (percentage) 4.350 -
470.lbm 49.000 -2.00%
470.lbm (avg) 49.000 -2.00%
470.lbm (avg) (percentage) 1.570 -1.26%
470.lbm (max) 23.000 -4.17%
470.lbm (max) (percentage) 0.730 -6.41%
470.lbm (unexplained diff vs previous) 23.000 -
470.lbm (unexplained diff vs previous) (percentage) 0.720 -
471.omnetpp 76.000 -
471.omnetpp (avg) 76.000 -
471.omnetpp (avg) (percentage) 2.730 -1.44%
471.omnetpp (max) 32.000 -
471.omnetpp (max) (percentage) 1.160 -
471.omnetpp (unexplained diff vs previous) 32.000 -
471.omnetpp (unexplained diff vs previous) (percentage) 1.150 -
473.astar 113.000 -
473.astar (avg) 113.000 -
473.astar (avg) (percentage) 2.320 -
473.astar (max) 13.000 -7.14%
473.astar (max) (percentage) 0.280 -6.67%
482.sphinx3 12.000 -
482.sphinx3 (avg) 12.000 -
482.sphinx3 (avg) (percentage) 1.440 -
482.sphinx3 (max) 8.000 -
482.sphinx3 (max) (percentage) 0.940 -4.08%
483.xalancbmk 100.000 -1.96%
483.xalancbmk (avg) 100.000 -1.96%
483.xalancbmk (avg) (percentage) 2.810 -2.43%
483.xalancbmk (max) 25.000 -7.41%
483.xalancbmk (max) (percentage) 0.710 -6.58%
483.xalancbmk (unexplained diff vs previous) 16.000 -20.00%
483.xalancbmk (unexplained diff vs previous) (percentage) 0.450 -19.64%
Geometric Mean 5.256 18.35%