Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/b88ebb3d94cbd2bdf6ffd17fa7c5559579a249a2 (+ 617 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6-llvm-arm-master-Os-build/7/
View Options
Report
Filter

llvm-arm-master-Os test results

Run Order Start Time Duration
Current 0007 (llvmorg-17-init-00954-gb88ebb3d94cb) 2023-01-29T09:07:58 0:00:00
Previous 0006 (llvmorg-17-init-00337-gfeedb4ad7d30) 2023-01-24T18:27:06 0:00:00
Baseline 0001 (llvmorg-16-init-16061-ge7790fbed32b) 2022-12-29T19:41:07 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 14 65
Performance Improvements 67 53
Unchanged Tests 699 662
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
471.omnetpp 3.63% 9289.000 9626.000 - -10.42% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
462.libquantum (unexplained diff vs previous) (percentage) 480.00% 0.050 0.290 - 141.67% -
462.libquantum (unexplained diff vs previous) 466.67% 6.000 34.000 - 112.50% -
470.lbm (unexplained diff vs previous) 76.92% 13.000 23.000 - -95.37% -
470.lbm (unexplained diff vs previous) (percentage) 72.73% 0.110 0.190 - -94.69% -
471.omnetpp (max) 48.51% 235.000 349.000 - 84.66% -
471.omnetpp (max) (percentage) 42.91% 2.540 3.630 - 106.25% -
471.omnetpp (unexplained diff vs previous) 39.26% 242.000 337.000 - 266.30% -
471.omnetpp (unexplained diff vs previous) (percentage) 34.10% 2.610 3.500 - 306.98% -
471.omnetpp 12.88% 132.000 149.000 - 79.52% -
471.omnetpp (avg) 12.88% 132.000 149.000 - 79.52% -
471.omnetpp (avg) (percentage) 8.39% 1.430 1.550 - 98.72% -
464.h264ref 1.06% 94.000 95.000 - 86.27% -
464.h264ref (avg) 1.06% 94.000 95.000 - 86.27% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
453.povray -2.73% 6877.000 6689.000 - -2.45% -
447.dealII -1.74% 8042.000 7902.000 - -3.59% -
445.gobmk -1.62% 11877.000 11685.000 - -5.77% -
456.hmmer -1.25% 11954.000 11805.000 - -11.67% -
403.gcc -1.18% 8996.000 8890.000 - -2.30% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
450.soplex (unexplained diff vs previous) (percentage) -73.33% 0.150 0.040 - -92.98% -
433.milc (unexplained diff vs previous) -71.88% 96.000 27.000 - -85.16% -
433.milc (unexplained diff vs previous) (percentage) -71.62% 0.740 0.210 - -84.44% -
450.soplex (unexplained diff vs previous) -71.43% 21.000 6.000 - -92.68% -
433.milc (max) -67.53% 271.000 88.000 - -52.17% -
433.milc (max) (percentage) -67.14% 2.100 0.690 - -49.64% -
429.mcf (unexplained diff vs previous) -60.00% 15.000 6.000 - -91.04% -
429.mcf (unexplained diff vs previous) (percentage) -60.00% 0.100 0.040 - -92.86% -
450.soplex (max) (percentage) -12.50% 0.320 0.280 - -50.88% -
450.soplex (max) -11.11% 45.000 40.000 - -51.22% -
464.h264ref (max) -9.38% 32.000 29.000 - -51.67% -
464.h264ref (max) (percentage) -8.70% 0.230 0.210 - -46.15% -
445.gobmk (max) -7.53% 93.000 86.000 - 196.55% -
444.namd (max) -6.65% 707.000 660.000 - -25.17% -
444.namd (max) (percentage) -6.61% 6.200 5.790 - -11.47% -
453.povray (max) -6.43% 140.000 131.000 - -0.76% -
429.mcf -6.35% 189.000 177.000 - 233.96% -
429.mcf (avg) -6.35% 189.000 177.000 - 233.96% -
445.gobmk (max) (percentage) -6.33% 0.790 0.740 - 208.33% -
429.mcf (avg) (percentage) -6.11% 1.310 1.230 - 173.33% -
470.lbm (max) -5.93% 472.000 444.000 - -42.26% -
429.mcf (max) (percentage) -5.88% 0.170 0.160 - -71.43% -
470.lbm (max) (percentage) -5.70% 3.860 3.640 - -34.30% -
470.lbm -5.56% 396.000 374.000 - 16.15% -
470.lbm (avg) -5.56% 396.000 374.000 - 16.15% -
470.lbm (avg) (percentage) -5.25% 3.240 3.070 - 32.33% -
456.hmmer (max) -5.08% 59.000 56.000 - 40.00% -
473.astar (max) -4.88% 82.000 78.000 - -31.58% -
473.astar (max) (percentage) -4.69% 0.640 0.610 - -23.75% -
482.sphinx3 (max) -4.61% 586.000 559.000 - -35.97% -
482.sphinx3 (max) (percentage) -4.58% 2.400 2.290 - -31.02% -
462.libquantum (max) (percentage) -4.55% 0.440 0.420 - 110.00% -
429.mcf (max) -4.17% 24.000 23.000 - -65.67% -
456.hmmer (max) (percentage) -4.00% 0.500 0.480 - 60.00% -
462.libquantum (avg) (percentage) -4.00% 0.750 0.720 - 380.00% -
450.soplex -4.00% 50.000 48.000 - -5.88% -
450.soplex (avg) -4.00% 50.000 48.000 - -5.88% -
462.libquantum (max) -3.92% 51.000 49.000 - 88.46% -
433.milc -3.54% 113.000 109.000 - -19.85% -
433.milc (avg) -3.54% 113.000 109.000 - -19.85% -
462.libquantum -3.45% 87.000 84.000 - 342.11% -
462.libquantum (avg) -3.45% 87.000 84.000 - 342.11% -
453.povray (max) (percentage) -3.43% 2.040 1.970 - 2.07% -
447.dealII (max) -3.43% 175.000 169.000 - 31.01% -
433.milc (avg) (percentage) -3.41% 0.880 0.850 - -15.84% -
453.povray -3.33% 90.000 87.000 - -6.45% -
453.povray (avg) -3.33% 90.000 87.000 - -6.45% -
403.gcc (max) -2.94% 136.000 132.000 - 288.24% -
483.xalancbmk (max) -2.46% 977.000 953.000 - 1462.30% -
458.sjeng (max) -2.37% 674.000 658.000 - 912.31% -
401.bzip2 (max) (percentage) -2.37% 2.950 2.880 - 829.03% -
401.bzip2 (max) -2.26% 443.000 433.000 - 802.08% -
483.xalancbmk (max) (percentage) -2.22% 12.600 12.320 - 1795.38% -
403.gcc (max) (percentage) -1.97% 1.520 1.490 - 292.11% -
447.dealII (max) (percentage) -1.83% 2.180 2.140 - 35.44% -
458.sjeng (max) (percentage) -1.82% 3.850 3.780 - 845.00% -
403.gcc -1.75% 57.000 56.000 - 30.23% -
403.gcc (avg) -1.75% 57.000 56.000 - 30.23% -
445.gobmk -1.69% 59.000 58.000 - 163.64% -
445.gobmk (avg) -1.69% 59.000 58.000 - 163.64% -
456.hmmer -1.41% 142.000 140.000 - 191.67% -
456.hmmer (avg) -1.41% 142.000 140.000 - 191.67% -

Run-Over-Baseline Changes Detail

Performance Regressions - Execution Δ (B) Baseline Current σ (B) Δ σ
429.mcf 20.39% 11999.000 14446.000 - -0.04% -
458.sjeng 6.19% 16406.000 17422.000 - -0.62% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
483.xalancbmk (max) (percentage) 1795.38% 0.650 12.320 - -2.22% -
483.xalancbmk (max) 1462.30% 61.000 953.000 - -2.46% -
458.sjeng (max) 912.31% 65.000 658.000 - -2.37% -
458.sjeng (max) (percentage) 845.00% 0.400 3.780 - -1.82% -
401.bzip2 (max) (percentage) 829.03% 0.310 2.880 - -2.37% -
401.bzip2 (max) 802.08% 48.000 433.000 - -2.26% -
462.libquantum (avg) (percentage) 380.00% 0.150 0.720 - -4.00% -
462.libquantum 342.11% 19.000 84.000 - -3.45% -
462.libquantum (avg) 342.11% 19.000 84.000 - -3.45% -
471.omnetpp (unexplained diff vs previous) (percentage) 306.98% 0.860 3.500 - 34.10% -
403.gcc (max) (percentage) 292.11% 0.380 1.490 - -1.97% -
403.gcc (max) 288.24% 34.000 132.000 - -2.94% -
471.omnetpp (unexplained diff vs previous) 266.30% 92.000 337.000 - 39.26% -
483.xalancbmk (avg) (percentage) 264.00% 0.500 1.820 - 0.00% -
429.mcf 233.96% 53.000 177.000 - -6.35% -
429.mcf (avg) 233.96% 53.000 177.000 - -6.35% -
456.hmmer (avg) (percentage) 230.56% 0.360 1.190 - 0.00% -
445.gobmk (max) (percentage) 208.33% 0.240 0.740 - -6.33% -
483.xalancbmk 197.87% 47.000 140.000 - -0.71% -
483.xalancbmk (avg) 197.87% 47.000 140.000 - -0.71% -
445.gobmk (max) 196.55% 29.000 86.000 - -7.53% -
456.hmmer 191.67% 48.000 140.000 - -1.41% -
456.hmmer (avg) 191.67% 48.000 140.000 - -1.41% -
445.gobmk (avg) (percentage) 177.78% 0.180 0.500 - 0.00% -
429.mcf (avg) (percentage) 173.33% 0.450 1.230 - -6.11% -
445.gobmk 163.64% 22.000 58.000 - -1.69% -
445.gobmk (avg) 163.64% 22.000 58.000 - -1.69% -
462.libquantum (unexplained diff vs previous) (percentage) 141.67% 0.120 0.290 - 480.00% -
462.libquantum (unexplained diff vs previous) 112.50% 16.000 34.000 - 466.67% -
473.astar (avg) (percentage) 112.00% 0.500 1.060 - 0.00% -
462.libquantum (max) (percentage) 110.00% 0.200 0.420 - -4.55% -
471.omnetpp (max) (percentage) 106.25% 1.760 3.630 - 42.91% -
464.h264ref (avg) (percentage) 103.03% 0.330 0.670 - 0.00% -
471.omnetpp (avg) (percentage) 98.72% 0.780 1.550 - 8.39% -
473.astar 91.55% 71.000 136.000 - 0.00% -
473.astar (avg) 91.55% 71.000 136.000 - 0.00% -
401.bzip2 (avg) (percentage) 90.91% 0.220 0.420 - 0.00% -
462.libquantum (max) 88.46% 26.000 49.000 - -3.92% -
464.h264ref 86.27% 51.000 95.000 - 1.06% -
464.h264ref (avg) 86.27% 51.000 95.000 - 1.06% -
401.bzip2 85.29% 34.000 63.000 - 0.00% -
401.bzip2 (avg) 85.29% 34.000 63.000 - 0.00% -
471.omnetpp (max) 84.66% 189.000 349.000 - 48.51% -
471.omnetpp 79.52% 83.000 149.000 - 12.88% -
471.omnetpp (avg) 79.52% 83.000 149.000 - 12.88% -
456.hmmer (max) (percentage) 60.00% 0.300 0.480 - -4.00% -
458.sjeng 46.32% 95.000 139.000 - -0.71% -
458.sjeng (avg) 46.32% 95.000 139.000 - -0.71% -
456.hmmer (max) 40.00% 40.000 56.000 - -5.08% -
458.sjeng (avg) (percentage) 37.93% 0.580 0.800 - 0.00% -
447.dealII (max) (percentage) 35.44% 1.580 2.140 - -1.83% -
403.gcc (avg) (percentage) 33.33% 0.480 0.640 - 0.00% -
470.lbm (avg) (percentage) 32.33% 2.320 3.070 - -5.25% -
447.dealII (max) 31.01% 129.000 169.000 - -3.43% -
403.gcc 30.23% 43.000 56.000 - -1.75% -
403.gcc (avg) 30.23% 43.000 56.000 - -1.75% -
470.lbm 16.15% 322.000 374.000 - -5.56% -
470.lbm (avg) 16.15% 322.000 374.000 - -5.56% -
482.sphinx3 (avg) (percentage) 14.52% 1.860 2.130 - 0.00% -
482.sphinx3 6.34% 489.000 520.000 - 0.00% -
482.sphinx3 (avg) 6.34% 489.000 520.000 - 0.00% -
444.namd (avg) (percentage) 5.77% 3.810 4.030 - 0.00% -
453.povray (max) (percentage) 2.07% 1.930 1.970 - -3.43% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
483.xalancbmk -17.93% 9427.000 7737.000 - -0.27% -
444.namd -15.45% 13495.000 11410.000 - -0.03% -
470.lbm -12.06% 13882.000 12208.000 - -0.19% -
456.hmmer -11.67% 13365.000 11805.000 - -1.25% -
462.libquantum -10.53% 13126.000 11744.000 - 0.29% -
471.omnetpp -10.42% 10746.000 9626.000 - 3.63% -
473.astar -9.84% 14262.000 12859.000 - -0.38% -
464.h264ref -8.46% 15493.000 14183.000 - 0.72% -
482.sphinx3 -7.12% 26310.000 24436.000 - -0.01% -
445.gobmk -5.77% 12400.000 11685.000 - -1.62% -
433.milc -4.48% 13482.000 12878.000 - -0.21% -
401.bzip2 -4.13% 15683.000 15035.000 - 0.04% -
447.dealII -3.59% 8196.000 7902.000 - -1.74% -
453.povray -2.45% 6857.000 6689.000 - -2.73% -
403.gcc -2.30% 9099.000 8890.000 - -1.18% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
470.lbm (unexplained diff vs previous) -95.37% 497.000 23.000 - 76.92% -
470.lbm (unexplained diff vs previous) (percentage) -94.69% 3.580 0.190 - 72.73% -
450.soplex (unexplained diff vs previous) (percentage) -92.98% 0.570 0.040 - -73.33% -
429.mcf (unexplained diff vs previous) (percentage) -92.86% 0.560 0.040 - -60.00% -
450.soplex (unexplained diff vs previous) -92.68% 82.000 6.000 - -71.43% -
429.mcf (unexplained diff vs previous) -91.04% 67.000 6.000 - -60.00% -
433.milc (unexplained diff vs previous) -85.16% 182.000 27.000 - -71.88% -
433.milc (unexplained diff vs previous) (percentage) -84.44% 1.350 0.210 - -71.62% -
429.mcf (max) (percentage) -71.43% 0.560 0.160 - -5.88% -
429.mcf (max) -65.67% 67.000 23.000 - -4.17% -
433.milc (max) -52.17% 184.000 88.000 - -67.53% -
464.h264ref (max) -51.67% 60.000 29.000 - -9.38% -
450.soplex (max) -51.22% 82.000 40.000 - -11.11% -
450.soplex (max) (percentage) -50.88% 0.570 0.280 - -12.50% -
433.milc (max) (percentage) -49.64% 1.370 0.690 - -67.14% -
464.h264ref (max) (percentage) -46.15% 0.390 0.210 - -8.70% -
470.lbm (max) -42.26% 769.000 444.000 - -5.93% -
482.sphinx3 (max) -35.97% 873.000 559.000 - -4.61% -
470.lbm (max) (percentage) -34.30% 5.540 3.640 - -5.70% -
473.astar (max) -31.58% 114.000 78.000 - -4.88% -
482.sphinx3 (max) (percentage) -31.02% 3.320 2.290 - -4.58% -
444.namd (max) -25.17% 882.000 660.000 - -6.65% -
473.astar (max) (percentage) -23.75% 0.800 0.610 - -4.69% -
433.milc -19.85% 136.000 109.000 - -3.54% -
433.milc (avg) -19.85% 136.000 109.000 - -3.54% -
433.milc (avg) (percentage) -15.84% 1.010 0.850 - -3.41% -
447.dealII -15.83% 120.000 101.000 - -0.98% -
447.dealII (avg) -15.83% 120.000 101.000 - -0.98% -
447.dealII (avg) (percentage) -12.93% 1.470 1.280 - 0.00% -
444.namd (max) (percentage) -11.47% 6.540 5.790 - -6.61% -
444.namd -10.70% 514.000 459.000 - 0.00% -
444.namd (avg) -10.70% 514.000 459.000 - 0.00% -
453.povray -6.45% 93.000 87.000 - -3.33% -
453.povray (avg) -6.45% 93.000 87.000 - -3.33% -
450.soplex -5.88% 51.000 48.000 - -4.00% -
450.soplex (avg) -5.88% 51.000 48.000 - -4.00% -
450.soplex (avg) (percentage) -5.56% 0.360 0.340 - -2.86% -
453.povray (avg) (percentage) -4.38% 1.370 1.310 - 0.00% -

Report Time: 0.33s
Execution (seconds)
Name Current %
401.bzip2 15035.000 -
403.gcc 8890.000 -1.18%
429.mcf 14446.000 -
433.milc 12878.000 -
444.namd 11410.000 -
445.gobmk 11685.000 -1.62%
447.dealII 7902.000 -1.74%
450.soplex 14301.000 -
453.povray 6689.000 -2.73%
456.hmmer 11805.000 -1.25%
458.sjeng 17422.000 -
462.libquantum 11744.000 -
464.h264ref 14183.000 -
470.lbm 12208.000 -
471.omnetpp 9626.000 3.63%
473.astar 12859.000 -
482.sphinx3 24436.000 -
483.xalancbmk 7737.000 -
Geometric Mean 11945.425 -
Code Size (bytes)
Name Current %
401.bzip2 39458.000 -
403.gcc 1959962.000 -
429.mcf 7915.000 -
433.milc 74971.000 -
444.namd 175160.000 -
445.gobmk 1312180.000 -
447.dealII 1778640.000 -
450.soplex 198708.000 -
453.povray 593201.000 -
456.hmmer 168088.000 -
458.sjeng 97694.000 -
462.libquantum 26756.000 -
464.h264ref 354530.000 -
470.lbm 9723.000 -
471.omnetpp 373571.000 -
473.astar 23216.000 -
482.sphinx3 115784.000 -
483.xalancbmk 2287101.000 -
Geometric Mean 164395.825 -
sample variability
Name Current %
401.bzip2 63.000 -
401.bzip2 (avg) 63.000 -
401.bzip2 (avg) (percentage) 0.420 -
401.bzip2 (max) 433.000 -2.26%
401.bzip2 (max) (percentage) 2.880 -2.37%
403.gcc 56.000 -1.75%
403.gcc (avg) 56.000 -1.75%
403.gcc (avg) (percentage) 0.640 -
403.gcc (max) 132.000 -2.94%
403.gcc (max) (percentage) 1.490 -1.97%
429.mcf 177.000 -6.35%
429.mcf (avg) 177.000 -6.35%
429.mcf (avg) (percentage) 1.230 -6.11%
429.mcf (max) 23.000 -4.17%
429.mcf (max) (percentage) 0.160 -5.88%
429.mcf (unexplained diff vs previous) 6.000 -60.00%
429.mcf (unexplained diff vs previous) (percentage) 0.040 -60.00%
433.milc 109.000 -3.54%
433.milc (avg) 109.000 -3.54%
433.milc (avg) (percentage) 0.850 -3.41%
433.milc (max) 88.000 -67.53%
433.milc (max) (percentage) 0.690 -67.14%
433.milc (unexplained diff vs previous) 27.000 -71.88%
433.milc (unexplained diff vs previous) (percentage) 0.210 -71.62%
444.namd 459.000 -
444.namd (avg) 459.000 -
444.namd (avg) (percentage) 4.030 -
444.namd (max) 660.000 -6.65%
444.namd (max) (percentage) 5.790 -6.61%
445.gobmk 58.000 -1.69%
445.gobmk (avg) 58.000 -1.69%
445.gobmk (avg) (percentage) 0.500 -
445.gobmk (max) 86.000 -7.53%
445.gobmk (max) (percentage) 0.740 -6.33%
447.dealII 101.000 -
447.dealII (avg) 101.000 -
447.dealII (avg) (percentage) 1.280 -
447.dealII (max) 169.000 -3.43%
447.dealII (max) (percentage) 2.140 -1.83%
450.soplex 48.000 -4.00%
450.soplex (avg) 48.000 -4.00%
450.soplex (avg) (percentage) 0.340 -
450.soplex (max) 40.000 -11.11%
450.soplex (max) (percentage) 0.280 -12.50%
450.soplex (unexplained diff vs previous) 6.000 -71.43%
450.soplex (unexplained diff vs previous) (percentage) 0.040 -73.33%
453.povray 87.000 -3.33%
453.povray (avg) 87.000 -3.33%
453.povray (avg) (percentage) 1.310 -
453.povray (max) 131.000 -6.43%
453.povray (max) (percentage) 1.970 -3.43%
456.hmmer 140.000 -1.41%
456.hmmer (avg) 140.000 -1.41%
456.hmmer (avg) (percentage) 1.190 -
456.hmmer (max) 56.000 -5.08%
456.hmmer (max) (percentage) 0.480 -4.00%
458.sjeng 139.000 -
458.sjeng (avg) 139.000 -
458.sjeng (avg) (percentage) 0.800 -
458.sjeng (max) 658.000 -2.37%
458.sjeng (max) (percentage) 3.780 -1.82%
462.libquantum 84.000 -3.45%
462.libquantum (avg) 84.000 -3.45%
462.libquantum (avg) (percentage) 0.720 -4.00%
462.libquantum (max) 49.000 -3.92%
462.libquantum (max) (percentage) 0.420 -4.55%
462.libquantum (unexplained diff vs previous) 34.000 466.67%
462.libquantum (unexplained diff vs previous) (percentage) 0.290 480.00%
464.h264ref 95.000 1.06%
464.h264ref (avg) 95.000 1.06%
464.h264ref (avg) (percentage) 0.670 -
464.h264ref (max) 29.000 -9.38%
464.h264ref (max) (percentage) 0.210 -8.70%
470.lbm 374.000 -5.56%
470.lbm (avg) 374.000 -5.56%
470.lbm (avg) (percentage) 3.070 -5.25%
470.lbm (max) 444.000 -5.93%
470.lbm (max) (percentage) 3.640 -5.70%
470.lbm (unexplained diff vs previous) 23.000 76.92%
470.lbm (unexplained diff vs previous) (percentage) 0.190 72.73%
471.omnetpp 149.000 12.88%
471.omnetpp (avg) 149.000 12.88%
471.omnetpp (avg) (percentage) 1.550 8.39%
471.omnetpp (max) 349.000 48.51%
471.omnetpp (max) (percentage) 3.630 42.91%
471.omnetpp (unexplained diff vs previous) 337.000 39.26%
471.omnetpp (unexplained diff vs previous) (percentage) 3.500 34.10%
473.astar 136.000 -
473.astar (avg) 136.000 -
473.astar (avg) (percentage) 1.060 -
473.astar (max) 78.000 -4.88%
473.astar (max) (percentage) 0.610 -4.69%
482.sphinx3 520.000 -
482.sphinx3 (avg) 520.000 -
482.sphinx3 (avg) (percentage) 2.130 -
482.sphinx3 (max) 559.000 -4.61%
482.sphinx3 (max) (percentage) 2.290 -4.58%
483.xalancbmk 140.000 -
483.xalancbmk (avg) 140.000 -
483.xalancbmk (avg) (percentage) 1.820 -
483.xalancbmk (max) 953.000 -2.46%
483.xalancbmk (max) (percentage) 12.320 -2.22%
Geometric Mean 15.548 -5.16%