Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/e13bed4c5f3544c076ce57e36d9a11eefa5a7815 (+ 1 commits)
regression LLVM-1001
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Os-build/90/
View Options
Report
Filter

llvm-arm-master-Os test results

Run Order Start Time Duration
Current 0090 (llvmorg-18-init-07933-ge13bed4c5f35) 2023-10-05T10:35:50 0:00:00
Previous 0084 (llvmorg-18-init-07932-ga16f6462d756) 2023-10-04T09:42:34 0:00:00
Baseline 0001 (llvmorg-16-init-16061-ge7790fbed32b) 2022-12-29T19:41:07 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 18 30
Performance Improvements 49 97
Added Tests 2 2
Unchanged Tests 711 651
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Code Size Δ Previous Current σ Δ (B) σ (B)
473.astar 1.02% 23183.000 23419.000 - 0.87% -

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
453.povray 4.91% 448.000 470.000 - -93.15% -
447.dealII 2.69% 1746.000 1793.000 - -78.12% -
473.astar 2.37% 4769.000 4882.000 - -65.77% -
471.omnetpp 1.17% 2726.000 2758.000 - -74.33% -
462.libquantum 1.02% 98.000 99.000 - -99.25% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) 190.91% 11.000 32.000 - -65.22% -
471.omnetpp (unexplained diff vs previous) (percentage) 190.00% 0.400 1.160 - 34.88% -
458.sjeng (unexplained diff vs previous) (percentage) 131.58% 0.380 0.880 - 252.00% -
458.sjeng (unexplained diff vs previous) 129.03% 31.000 71.000 - 73.17% -
458.sjeng (max) (percentage) 128.95% 0.380 0.870 - 117.50% -
458.sjeng (max) 125.81% 31.000 70.000 - 7.69% -
471.omnetpp (max) 77.78% 18.000 32.000 - -83.07% -
471.omnetpp (max) (percentage) 72.06% 0.680 1.170 - -33.52% -
447.dealII 7.14% 14.000 15.000 - -87.50% -
447.dealII (avg) 7.14% 14.000 15.000 - -87.50% -
473.astar 2.73% 110.000 113.000 - 59.15% -
473.astar (avg) 2.73% 110.000 113.000 - 59.15% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -3.28% 61.000 59.000 - -99.35% -
464.h264ref -2.39% 5525.000 5393.000 - -65.19% -
444.namd -1.39% 721.000 711.000 - -94.73% -
401.bzip2 -1.22% 2786.000 2752.000 - -82.45% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
450.soplex (unexplained diff vs previous) (percentage) -92.96% 2.700 0.190 - -66.67% -
450.soplex (unexplained diff vs previous) -92.86% 14.000 1.000 - -98.78% -
445.gobmk (max) -50.00% 16.000 8.000 - -72.41% -
445.gobmk (max) (percentage) -45.16% 0.310 0.170 - -29.17% -
403.gcc (max) -33.33% 3.000 2.000 - -94.12% -
483.xalancbmk (unexplained diff vs previous) -33.33% 30.000 20.000 - -67.21% -
483.xalancbmk (unexplained diff vs previous) (percentage) -32.53% 0.830 0.560 - -13.85% -
450.soplex (max) (percentage) -25.10% 2.630 1.970 - 245.61% -
456.hmmer (max) -25.00% 4.000 3.000 - -92.50% -
450.soplex (max) -23.08% 13.000 10.000 - -87.80% -
433.milc (max) -20.00% 5.000 4.000 - -97.83% -
444.namd (max) -18.18% 11.000 9.000 - -98.98% -
464.h264ref (max) -14.29% 21.000 18.000 - -70.00% -
401.bzip2 (max) -12.50% 16.000 14.000 - -70.83% -
482.sphinx3 (max) -11.11% 9.000 8.000 - -99.08% -
483.xalancbmk (max) (percentage) -10.59% 0.850 0.760 - 16.92% -
401.bzip2 (max) (percentage) -10.34% 0.580 0.520 - 67.74% -
444.namd (max) (percentage) -10.32% 1.550 1.390 - -78.75% -
464.h264ref (max) (percentage) -10.26% 0.390 0.350 - -10.26% -
483.xalancbmk (max) -10.00% 30.000 27.000 - -55.74% -
444.namd (unexplained diff vs previous) -9.09% 11.000 10.000 - -98.79% -
429.mcf (max) (percentage) -9.09% 0.220 0.200 - -64.29% -
433.milc (max) (percentage) -8.70% 0.460 0.420 - -69.34% -
470.lbm (max) (percentage) -8.24% 0.850 0.780 - -85.92% -
444.namd (unexplained diff vs previous) (percentage) -7.84% 1.530 1.410 - -77.04% -
470.lbm (max) -7.69% 26.000 24.000 - -96.88% -
453.povray (max) (percentage) -6.88% 1.600 1.490 - -22.80% -
447.dealII (max) (percentage) -6.74% 0.890 0.830 - -47.47% -
447.dealII (max) -6.67% 15.000 14.000 - -89.15% -
473.astar (max) -6.67% 15.000 14.000 - -87.72% -
473.astar (max) (percentage) -6.25% 0.320 0.300 - -62.50% -
462.libquantum (max) (percentage) -6.02% 1.330 1.250 - 525.00% -
445.gobmk -5.88% 34.000 32.000 - 45.45% -
445.gobmk (avg) -5.88% 34.000 32.000 - 45.45% -
403.gcc (max) (percentage) -5.03% 5.170 4.910 - 1192.11% -
482.sphinx3 (max) (percentage) -4.85% 1.030 0.980 - -70.48% -
445.gobmk (avg) (percentage) -4.62% 0.650 0.620 - 244.44% -
483.xalancbmk -3.77% 106.000 102.000 - 117.02% -
483.xalancbmk (avg) -3.77% 106.000 102.000 - 117.02% -
483.xalancbmk (avg) (percentage) -2.70% 2.960 2.880 - 476.00% -
464.h264ref -2.33% 43.000 42.000 - -17.65% -
464.h264ref (avg) -2.33% 43.000 42.000 - -17.65% -
450.soplex (avg) (percentage) -1.87% 1.070 1.050 - 191.67% -
471.omnetpp (avg) (percentage) -1.77% 2.820 2.770 - 255.13% -
433.milc (avg) (percentage) -1.56% 0.640 0.630 - -37.62% -

Added Tests - sample variability
445.gobmk (unexplained diff vs previous)
445.gobmk (unexplained diff vs previous) (percentage)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
401.bzip2 5.88% 39458.000 41777.000 - -0.10% -
450.soplex 1.78% 198708.000 202238.000 - -0.01% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
403.gcc (max) (percentage) 1192.11% 0.380 4.910 - -5.03% -
462.libquantum (avg) (percentage) 846.67% 0.150 1.420 - 0.00% -
429.mcf (avg) (percentage) 562.22% 0.450 2.980 - 0.00% -
462.libquantum (max) (percentage) 525.00% 0.200 1.250 - -6.02% -
483.xalancbmk (avg) (percentage) 476.00% 0.500 2.880 - -2.70% -
403.gcc (avg) (percentage) 470.83% 0.480 2.740 - 0.00% -
473.astar (avg) (percentage) 364.00% 0.500 2.320 - 0.00% -
471.omnetpp (avg) (percentage) 255.13% 0.780 2.770 - -1.77% -
458.sjeng (unexplained diff vs previous) (percentage) 252.00% 0.250 0.880 - 131.58% -
450.soplex (max) (percentage) 245.61% 0.570 1.970 - -25.10% -
445.gobmk (avg) (percentage) 244.44% 0.180 0.620 - -4.62% -
456.hmmer (avg) (percentage) 238.89% 0.360 1.220 - 0.00% -
450.soplex (avg) (percentage) 191.67% 0.360 1.050 - -1.87% -
401.bzip2 (avg) (percentage) 181.82% 0.220 0.620 - 0.00% -
464.h264ref (avg) (percentage) 139.39% 0.330 0.790 - 0.00% -
458.sjeng (max) (percentage) 117.50% 0.400 0.870 - 128.95% -
483.xalancbmk 117.02% 47.000 102.000 - -3.77% -
483.xalancbmk (avg) 117.02% 47.000 102.000 - -3.77% -
458.sjeng (unexplained diff vs previous) 73.17% 41.000 71.000 - 129.03% -
401.bzip2 (max) (percentage) 67.74% 0.310 0.520 - -10.34% -
473.astar 59.15% 71.000 113.000 - 2.73% -
473.astar (avg) 59.15% 71.000 113.000 - 2.73% -
445.gobmk 45.45% 22.000 32.000 - -5.88% -
445.gobmk (avg) 45.45% 22.000 32.000 - -5.88% -
471.omnetpp (unexplained diff vs previous) (percentage) 34.88% 0.860 1.160 - 190.00% -
458.sjeng (avg) (percentage) 32.76% 0.580 0.770 - 0.00% -
483.xalancbmk (max) (percentage) 16.92% 0.650 0.760 - -10.59% -
458.sjeng (max) 7.69% 65.000 70.000 - 125.81% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -8.97% 26756.000 24355.000 - 0.03% -
453.povray -2.67% 593097.000 577290.000 - -0.01% -
433.milc -2.48% 74971.000 73110.000 - 0.04% -
447.dealII -1.75% 1777992.000 1746855.000 - 0.04% -
482.sphinx3 -1.12% 115736.000 114439.000 - -0.03% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.35% 9099.000 59.000 - -3.28% -
462.libquantum -99.25% 13126.000 99.000 - 1.02% -
482.sphinx3 -96.66% 26310.000 878.000 - -0.45% -
450.soplex -96.40% 14429.000 519.000 - 0.19% -
444.namd -94.73% 13495.000 711.000 - -1.39% -
453.povray -93.15% 6857.000 470.000 - 4.91% -
433.milc -91.42% 13482.000 1157.000 - -0.26% -
429.mcf -90.25% 11999.000 1170.000 - -0.09% -
401.bzip2 -82.45% 15683.000 2752.000 - -1.22% -
447.dealII -78.12% 8196.000 1793.000 - 2.69% -
470.lbm -77.31% 13882.000 3150.000 - 0.10% -
471.omnetpp -74.33% 10746.000 2758.000 - 1.17% -
456.hmmer -73.36% 13365.000 3560.000 - -0.25% -
473.astar -65.77% 14262.000 4882.000 - 2.37% -
464.h264ref -65.19% 15493.000 5393.000 - -2.39% -
483.xalancbmk -62.09% 9427.000 3574.000 - -0.56% -
445.gobmk -57.68% 12400.000 5248.000 - 0.00% -
458.sjeng -50.61% 16406.000 8103.000 - -0.87% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
482.sphinx3 (max) -99.08% 873.000 8.000 - -11.11% -
444.namd (max) -98.98% 882.000 9.000 - -18.18% -
444.namd (unexplained diff vs previous) -98.79% 828.000 10.000 - -9.09% -
450.soplex (unexplained diff vs previous) -98.78% 82.000 1.000 - -92.86% -
433.milc (unexplained diff vs previous) -98.35% 182.000 3.000 - 0.00% -
433.milc (max) -97.83% 184.000 4.000 - -20.00% -
403.gcc -97.67% 43.000 1.000 - 0.00% -
403.gcc (avg) -97.67% 43.000 1.000 - 0.00% -
482.sphinx3 -97.55% 489.000 12.000 - 0.00% -
482.sphinx3 (avg) -97.55% 489.000 12.000 - 0.00% -
444.namd -97.28% 514.000 14.000 - 0.00% -
444.namd (avg) -97.28% 514.000 14.000 - 0.00% -
429.mcf (max) -97.01% 67.000 2.000 - 0.00% -
470.lbm (max) -96.88% 769.000 24.000 - -7.69% -
462.libquantum (max) -96.15% 26.000 1.000 - 0.00% -
433.milc -94.85% 136.000 7.000 - 0.00% -
433.milc (avg) -94.85% 136.000 7.000 - 0.00% -
462.libquantum -94.74% 19.000 1.000 - 0.00% -
462.libquantum (avg) -94.74% 19.000 1.000 - 0.00% -
453.povray (max) -94.70% 132.000 7.000 - 0.00% -
453.povray -94.62% 93.000 5.000 - 0.00% -
453.povray (avg) -94.62% 93.000 5.000 - 0.00% -
403.gcc (max) -94.12% 34.000 2.000 - -33.33% -
456.hmmer (max) -92.50% 40.000 3.000 - -25.00% -
450.soplex -90.20% 51.000 5.000 - 0.00% -
450.soplex (avg) -90.20% 51.000 5.000 - 0.00% -
447.dealII (max) -89.15% 129.000 14.000 - -6.67% -
450.soplex (max) -87.80% 82.000 10.000 - -23.08% -
473.astar (max) -87.72% 114.000 14.000 - -6.67% -
447.dealII -87.50% 120.000 15.000 - 7.14% -
447.dealII (avg) -87.50% 120.000 15.000 - 7.14% -
470.lbm (max) (percentage) -85.92% 5.540 0.780 - -8.24% -
470.lbm -84.47% 322.000 50.000 - 0.00% -
470.lbm (avg) -84.47% 322.000 50.000 - 0.00% -
471.omnetpp (max) -83.07% 189.000 32.000 - 77.78% -
433.milc (unexplained diff vs previous) (percentage) -80.74% 1.350 0.260 - 0.00% -
444.namd (max) (percentage) -78.75% 6.540 1.390 - -10.32% -
444.namd (unexplained diff vs previous) (percentage) -77.04% 6.140 1.410 - -7.84% -
445.gobmk (max) -72.41% 29.000 8.000 - -50.00% -
401.bzip2 (max) -70.83% 48.000 14.000 - -12.50% -
482.sphinx3 (max) (percentage) -70.48% 3.320 0.980 - -4.85% -
464.h264ref (max) -70.00% 60.000 18.000 - -14.29% -
433.milc (max) (percentage) -69.34% 1.370 0.420 - -8.70% -
483.xalancbmk (unexplained diff vs previous) -67.21% 61.000 20.000 - -33.33% -
450.soplex (unexplained diff vs previous) (percentage) -66.67% 0.570 0.190 - -92.96% -
471.omnetpp (unexplained diff vs previous) -65.22% 92.000 32.000 - 190.91% -
429.mcf (max) (percentage) -64.29% 0.560 0.200 - -9.09% -
456.hmmer (max) (percentage) -63.33% 0.300 0.110 - -8.33% -
473.astar (max) (percentage) -62.50% 0.800 0.300 - -6.25% -
483.xalancbmk (max) -55.74% 61.000 27.000 - -10.00% -
401.bzip2 -50.00% 34.000 17.000 - 0.00% -
401.bzip2 (avg) -50.00% 34.000 17.000 - 0.00% -
444.namd (avg) (percentage) -47.77% 3.810 1.990 - -1.00% -
447.dealII (max) (percentage) -47.47% 1.580 0.830 - -6.74% -
447.dealII (avg) (percentage) -42.86% 1.470 0.840 - 0.00% -
433.milc (avg) (percentage) -37.62% 1.010 0.630 - -1.56% -
429.mcf -35.85% 53.000 34.000 - 0.00% -
429.mcf (avg) -35.85% 53.000 34.000 - 0.00% -
458.sjeng -34.74% 95.000 62.000 - 0.00% -
458.sjeng (avg) -34.74% 95.000 62.000 - 0.00% -
471.omnetpp (max) (percentage) -33.52% 1.760 1.170 - 72.06% -
470.lbm (avg) (percentage) -31.47% 2.320 1.590 - 0.00% -
445.gobmk (max) (percentage) -29.17% 0.240 0.170 - -45.16% -
453.povray (max) (percentage) -22.80% 1.930 1.490 - -6.88% -
482.sphinx3 (avg) (percentage) -22.58% 1.860 1.440 - 0.00% -
464.h264ref -17.65% 51.000 42.000 - -2.33% -
464.h264ref (avg) -17.65% 51.000 42.000 - -2.33% -
483.xalancbmk (unexplained diff vs previous) (percentage) -13.85% 0.650 0.560 - -32.53% -
456.hmmer -10.42% 48.000 43.000 - 0.00% -
456.hmmer (avg) -10.42% 48.000 43.000 - 0.00% -
464.h264ref (max) (percentage) -10.26% 0.390 0.350 - -10.26% -
471.omnetpp -8.43% 83.000 76.000 - 0.00% -
471.omnetpp (avg) -8.43% 83.000 76.000 - 0.00% -
453.povray (avg) (percentage) -8.03% 1.370 1.260 - 0.00% -

Added Tests - sample variability
445.gobmk (unexplained diff vs previous)
445.gobmk (unexplained diff vs previous) (percentage)

Report Time: 0.34s
Execution (seconds)
Name Current %
401.bzip2 2752.000 -1.22%
403.gcc 59.000 -3.28%
429.mcf 1170.000 -
433.milc 1157.000 -
444.namd 711.000 -1.39%
445.gobmk 5248.000 -
447.dealII 1793.000 2.69%
450.soplex 519.000 -
453.povray 470.000 4.91%
456.hmmer 3560.000 -
458.sjeng 8103.000 -
462.libquantum 99.000 1.02%
464.h264ref 5393.000 -2.39%
470.lbm 3150.000 -
471.omnetpp 2758.000 1.17%
473.astar 4882.000 2.37%
482.sphinx3 878.000 -
483.xalancbmk 3574.000 -
Geometric Mean 1442.935 -
Code Size (bytes)
Name Current %
401.bzip2 41777.000 -
403.gcc 1960805.000 -
429.mcf 7874.000 -
433.milc 73110.000 -
444.namd 174849.000 -
445.gobmk 1309848.000 -
447.dealII 1746855.000 -
450.soplex 202238.000 -
453.povray 577290.000 -
456.hmmer 166414.000 -
458.sjeng 97185.000 -
462.libquantum 24355.000 -
464.h264ref 353206.000 -
470.lbm 9734.000 -
471.omnetpp 376679.000 -
473.astar 23419.000 1.02%
482.sphinx3 114439.000 -
483.xalancbmk 2286412.000 -
Geometric Mean 163386.823 -
sample variability
Name Current %
401.bzip2 17.000 -
401.bzip2 (avg) 17.000 -
401.bzip2 (avg) (percentage) 0.620 -
401.bzip2 (max) 14.000 -12.50%
401.bzip2 (max) (percentage) 0.520 -10.34%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 2.740 -
403.gcc (max) 2.000 -33.33%
403.gcc (max) (percentage) 4.910 -5.03%
429.mcf 34.000 -
429.mcf (avg) 34.000 -
429.mcf (avg) (percentage) 2.980 -
429.mcf (max) 2.000 -
429.mcf (max) (percentage) 0.200 -9.09%
433.milc 7.000 -
433.milc (avg) 7.000 -
433.milc (avg) (percentage) 0.630 -1.56%
433.milc (max) 4.000 -20.00%
433.milc (max) (percentage) 0.420 -8.70%
433.milc (unexplained diff vs previous) 3.000 -
433.milc (unexplained diff vs previous) (percentage) 0.260 -
444.namd 14.000 -
444.namd (avg) 14.000 -
444.namd (avg) (percentage) 1.990 -
444.namd (max) 9.000 -18.18%
444.namd (max) (percentage) 1.390 -10.32%
444.namd (unexplained diff vs previous) 10.000 -9.09%
444.namd (unexplained diff vs previous) (percentage) 1.410 -7.84%
445.gobmk 32.000 -5.88%
445.gobmk (avg) 32.000 -5.88%
445.gobmk (avg) (percentage) 0.620 -4.62%
445.gobmk (max) 8.000 -50.00%
445.gobmk (max) (percentage) 0.170 -45.16%
445.gobmk (unexplained diff vs previous) 0.000 -
445.gobmk (unexplained diff vs previous) (percentage) 0.000 -
447.dealII 15.000 7.14%
447.dealII (avg) 15.000 7.14%
447.dealII (avg) (percentage) 0.840 -
447.dealII (max) 14.000 -6.67%
447.dealII (max) (percentage) 0.830 -6.74%
450.soplex 5.000 -
450.soplex (avg) 5.000 -
450.soplex (avg) (percentage) 1.050 -1.87%
450.soplex (max) 10.000 -23.08%
450.soplex (max) (percentage) 1.970 -25.10%
450.soplex (unexplained diff vs previous) 1.000 -92.86%
450.soplex (unexplained diff vs previous) (percentage) 0.190 -92.96%
453.povray 5.000 -
453.povray (avg) 5.000 -
453.povray (avg) (percentage) 1.260 -
453.povray (max) 7.000 -
453.povray (max) (percentage) 1.490 -6.88%
456.hmmer 43.000 -
456.hmmer (avg) 43.000 -
456.hmmer (avg) (percentage) 1.220 -
456.hmmer (max) 3.000 -25.00%
456.hmmer (max) (percentage) 0.110 -
458.sjeng 62.000 -
458.sjeng (avg) 62.000 -
458.sjeng (avg) (percentage) 0.770 -
458.sjeng (max) 70.000 125.81%
458.sjeng (max) (percentage) 0.870 128.95%
458.sjeng (unexplained diff vs previous) 71.000 129.03%
458.sjeng (unexplained diff vs previous) (percentage) 0.880 131.58%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.420 -
462.libquantum (max) 1.000 -
462.libquantum (max) (percentage) 1.250 -6.02%
464.h264ref 42.000 -2.33%
464.h264ref (avg) 42.000 -2.33%
464.h264ref (avg) (percentage) 0.790 -
464.h264ref (max) 18.000 -14.29%
464.h264ref (max) (percentage) 0.350 -10.26%
470.lbm 50.000 -
470.lbm (avg) 50.000 -
470.lbm (avg) (percentage) 1.590 -
470.lbm (max) 24.000 -7.69%
470.lbm (max) (percentage) 0.780 -8.24%
471.omnetpp 76.000 -
471.omnetpp (avg) 76.000 -
471.omnetpp (avg) (percentage) 2.770 -1.77%
471.omnetpp (max) 32.000 77.78%
471.omnetpp (max) (percentage) 1.170 72.06%
471.omnetpp (unexplained diff vs previous) 32.000 190.91%
471.omnetpp (unexplained diff vs previous) (percentage) 1.160 190.00%
473.astar 113.000 2.73%
473.astar (avg) 113.000 2.73%
473.astar (avg) (percentage) 2.320 -
473.astar (max) 14.000 -6.67%
473.astar (max) (percentage) 0.300 -6.25%
482.sphinx3 12.000 -
482.sphinx3 (avg) 12.000 -
482.sphinx3 (avg) (percentage) 1.440 -
482.sphinx3 (max) 8.000 -11.11%
482.sphinx3 (max) (percentage) 0.980 -4.85%
483.xalancbmk 102.000 -3.77%
483.xalancbmk (avg) 102.000 -3.77%
483.xalancbmk (avg) (percentage) 2.880 -2.70%
483.xalancbmk (max) 27.000 -10.00%
483.xalancbmk (max) (percentage) 0.760 -10.59%
483.xalancbmk (unexplained diff vs previous) 20.000 -33.33%
483.xalancbmk (unexplained diff vs previous) (percentage) 0.560 -32.53%
Geometric Mean 3.643 -22.32%