Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/a6f621b8cacca926d809010c135be038fd05652c (+ 1144 commits)
test_url https://ci.linaro.org/job/tcwg_bmk_ci_llvm-build-tcwg_bmk_apm-llvm-master-arm-spec2k6-Os/189/
View Options
Report
Filter

llvm-arm-master-Os test results

Run Order Start Time Duration
Current 0189 (llvmorg-16-init-08494-ga6f621b8cacc) 2022-10-17T12:49:47 0:00:00
Previous 0188 (no-change) 2022-10-11T04:02:49 0:00:00
Baseline 0001 (llvmorg-16-init-16061-ge7790fbed32b) 2022-12-29T19:41:07 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 15 67
Performance Improvements 62 48
Unchanged Tests 703 665
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) 1471.43% 7.000 110.000 - 19.57% -
471.omnetpp (unexplained diff vs previous) (percentage) 1371.43% 0.070 1.030 - 19.77% -
462.libquantum (unexplained diff vs previous) (percentage) 220.00% 0.050 0.160 - 33.33% -
462.libquantum (unexplained diff vs previous) 200.00% 7.000 21.000 - 31.25% -
464.h264ref (unexplained diff vs previous) (percentage) 52.38% 0.210 0.320 - -17.95% -
464.h264ref (unexplained diff vs previous) 51.52% 33.000 50.000 - -16.67% -
462.libquantum (max) 50.00% 14.000 21.000 - -19.23% -
471.omnetpp (max) (percentage) 50.00% 0.680 1.020 - -42.05% -
471.omnetpp (max) 47.95% 73.000 108.000 - -42.86% -
462.libquantum (max) (percentage) 45.45% 0.110 0.160 - -20.00% -
471.omnetpp (avg) (percentage) 5.19% 0.770 0.810 - 3.85% -
471.omnetpp 4.88% 82.000 86.000 - 3.61% -
471.omnetpp (avg) 4.88% 82.000 86.000 - 3.61% -
464.h264ref (max) (percentage) 3.23% 0.310 0.320 - -17.95% -
464.h264ref (max) 2.08% 48.000 49.000 - -18.33% -

Performance Improvements - Code Size Δ Previous Current σ Δ (B) σ (B)
450.soplex -1.58% 203060.000 199849.000 - 0.57% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
471.omnetpp -1.02% 10756.000 10646.000 - -0.93% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
482.sphinx3 (unexplained diff vs previous) -95.56% 856.000 38.000 - -95.80% -
482.sphinx3 (unexplained diff vs previous) (percentage) -95.56% 3.150 0.140 - -95.93% -
470.lbm (unexplained diff vs previous) -91.61% 560.000 47.000 - -90.54% -
470.lbm (unexplained diff vs previous) (percentage) -91.58% 3.920 0.330 - -90.78% -
429.mcf (unexplained diff vs previous) (percentage) -83.61% 0.610 0.100 - -82.14% -
429.mcf (unexplained diff vs previous) -83.33% 72.000 12.000 - -82.09% -
433.milc (unexplained diff vs previous) -76.10% 251.000 60.000 - -67.03% -
433.milc (unexplained diff vs previous) (percentage) -76.06% 1.880 0.450 - -66.67% -
450.soplex (unexplained diff vs previous) -69.61% 102.000 31.000 - -62.20% -
450.soplex (unexplained diff vs previous) (percentage) -69.01% 0.710 0.220 - -61.40% -
450.soplex (max) -25.74% 101.000 75.000 - -8.54% -
450.soplex (max) (percentage) -25.35% 0.710 0.530 - -7.02% -
482.sphinx3 (max) -25.23% 884.000 661.000 - -24.28% -
482.sphinx3 (max) (percentage) -25.15% 3.260 2.440 - -26.51% -
470.lbm (max) (percentage) -25.00% 4.080 3.060 - -44.77% -
470.lbm (max) -24.74% 582.000 438.000 - -43.04% -
429.mcf (max) (percentage) -24.59% 0.610 0.460 - -17.86% -
429.mcf (max) -23.94% 71.000 54.000 - -19.40% -
433.milc (max) -17.07% 246.000 204.000 - 10.87% -
433.milc (max) (percentage) -16.76% 1.850 1.540 - 12.41% -
482.sphinx3 -15.53% 586.000 495.000 - 1.23% -
482.sphinx3 (avg) -15.53% 586.000 495.000 - 1.23% -
482.sphinx3 (avg) (percentage) -15.28% 2.160 1.830 - -1.61% -
429.mcf (avg) (percentage) -13.70% 0.730 0.630 - 40.00% -
470.lbm (avg) (percentage) -13.38% 2.690 2.330 - 0.43% -
470.lbm -13.05% 383.000 333.000 - 3.42% -
470.lbm (avg) -13.05% 383.000 333.000 - 3.42% -
429.mcf -12.94% 85.000 74.000 - 39.62% -
429.mcf (avg) -12.94% 85.000 74.000 - 39.62% -
450.soplex -10.31% 97.000 87.000 - 70.59% -
450.soplex (avg) -10.31% 97.000 87.000 - 70.59% -
450.soplex (avg) (percentage) -10.29% 0.680 0.610 - 69.44% -
433.milc -10.29% 175.000 157.000 - 15.44% -
433.milc (avg) -10.29% 175.000 157.000 - 15.44% -
433.milc (avg) (percentage) -9.85% 1.320 1.190 - 17.82% -
458.sjeng (max) (percentage) -9.68% 0.310 0.280 - -30.00% -
445.gobmk (max) -9.09% 44.000 40.000 - 37.93% -
445.gobmk (max) (percentage) -8.33% 0.360 0.330 - 37.50% -
458.sjeng (max) -8.00% 50.000 46.000 - -29.23% -
403.gcc (max) -7.50% 40.000 37.000 - 8.82% -
447.dealII (max) (percentage) -7.28% 2.060 1.910 - 20.89% -
403.gcc (max) (percentage) -6.82% 0.440 0.410 - 7.89% -
447.dealII (max) -6.67% 165.000 154.000 - 19.38% -
453.povray (max) -4.69% 128.000 122.000 - -7.58% -
462.libquantum (avg) (percentage) -4.55% 0.220 0.210 - 40.00% -
453.povray (max) (percentage) -4.19% 1.910 1.830 - -5.18% -
444.namd (max) (percentage) -4.13% 6.050 5.800 - -11.31% -
444.namd (max) -3.92% 766.000 736.000 - -16.55% -
456.hmmer (max) (percentage) -3.88% 1.030 0.990 - 230.00% -
456.hmmer (max) -3.57% 140.000 135.000 - 237.50% -
462.libquantum -3.57% 28.000 27.000 - 42.11% -
462.libquantum (avg) -3.57% 28.000 27.000 - 42.11% -
401.bzip2 (max) (percentage) -2.78% 0.360 0.350 - 12.90% -
473.astar (max) -2.50% 120.000 117.000 - 2.63% -
445.gobmk -2.44% 41.000 40.000 - 81.82% -
445.gobmk (avg) -2.44% 41.000 40.000 - 81.82% -
473.astar (max) (percentage) -2.38% 0.840 0.820 - 2.50% -
483.xalancbmk (max) (percentage) -2.22% 0.900 0.880 - 35.38% -
401.bzip2 (max) -1.82% 55.000 54.000 - 12.50% -
483.xalancbmk (max) -1.19% 84.000 83.000 - 36.07% -

Run-Over-Baseline Changes Detail

Performance Regressions - Execution Δ (B) Baseline Current σ (B) Δ σ
470.lbm 3.16% 13882.000 14321.000 - 0.33% -
482.sphinx3 2.99% 26310.000 27096.000 - -0.14% -
456.hmmer 2.54% 13365.000 13704.000 - 0.23% -
403.gcc 1.09% 9099.000 9198.000 - 0.46% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
456.hmmer (max) 237.50% 40.000 135.000 - -3.57% -
456.hmmer (max) (percentage) 230.00% 0.300 0.990 - -3.88% -
458.sjeng 221.05% 95.000 305.000 - 0.00% -
458.sjeng (avg) 221.05% 95.000 305.000 - 0.00% -
458.sjeng (avg) (percentage) 220.69% 0.580 1.860 - 0.00% -
403.gcc 116.28% 43.000 93.000 - 0.00% -
403.gcc (avg) 116.28% 43.000 93.000 - 0.00% -
403.gcc (avg) (percentage) 112.50% 0.480 1.020 - 0.00% -
445.gobmk (avg) (percentage) 83.33% 0.180 0.330 - 0.00% -
445.gobmk 81.82% 22.000 40.000 - -2.44% -
445.gobmk (avg) 81.82% 22.000 40.000 - -2.44% -
450.soplex 70.59% 51.000 87.000 - -10.31% -
450.soplex (avg) 70.59% 51.000 87.000 - -10.31% -
450.soplex (avg) (percentage) 69.44% 0.360 0.610 - -10.29% -
456.hmmer 64.58% 48.000 79.000 - 0.00% -
456.hmmer (avg) 64.58% 48.000 79.000 - 0.00% -
456.hmmer (avg) (percentage) 61.11% 0.360 0.580 - 0.00% -
462.libquantum 42.11% 19.000 27.000 - -3.57% -
462.libquantum (avg) 42.11% 19.000 27.000 - -3.57% -
462.libquantum (avg) (percentage) 40.00% 0.150 0.210 - -4.55% -
429.mcf (avg) (percentage) 40.00% 0.450 0.630 - -13.70% -
429.mcf 39.62% 53.000 74.000 - -12.94% -
429.mcf (avg) 39.62% 53.000 74.000 - -12.94% -
445.gobmk (max) 37.93% 29.000 40.000 - -9.09% -
445.gobmk (max) (percentage) 37.50% 0.240 0.330 - -8.33% -
483.xalancbmk (max) 36.07% 61.000 83.000 - -1.19% -
483.xalancbmk (max) (percentage) 35.38% 0.650 0.880 - -2.22% -
462.libquantum (unexplained diff vs previous) (percentage) 33.33% 0.120 0.160 - 220.00% -
462.libquantum (unexplained diff vs previous) 31.25% 16.000 21.000 - 200.00% -
483.xalancbmk (avg) (percentage) 28.00% 0.500 0.640 - 0.00% -
483.xalancbmk 27.66% 47.000 60.000 - 0.00% -
483.xalancbmk (avg) 27.66% 47.000 60.000 - 0.00% -
473.astar 22.54% 71.000 87.000 - 0.00% -
473.astar (avg) 22.54% 71.000 87.000 - 0.00% -
473.astar (avg) (percentage) 22.00% 0.500 0.610 - 0.00% -
447.dealII (max) (percentage) 20.89% 1.580 1.910 - -7.28% -
471.omnetpp (unexplained diff vs previous) (percentage) 19.77% 0.860 1.030 - 1371.43% -
471.omnetpp (unexplained diff vs previous) 19.57% 92.000 110.000 - 1471.43% -
447.dealII (max) 19.38% 129.000 154.000 - -6.67% -
444.namd (avg) (percentage) 19.16% 3.810 4.540 - 0.00% -
433.milc (avg) (percentage) 17.82% 1.010 1.190 - -9.85% -
433.milc 15.44% 136.000 157.000 - -10.29% -
433.milc (avg) 15.44% 136.000 157.000 - -10.29% -
401.bzip2 (max) (percentage) 12.90% 0.310 0.350 - -2.78% -
401.bzip2 (max) 12.50% 48.000 54.000 - -1.82% -
433.milc (max) (percentage) 12.41% 1.370 1.540 - -16.76% -
444.namd 12.06% 514.000 576.000 - 0.17% -
444.namd (avg) 12.06% 514.000 576.000 - 0.17% -
433.milc (max) 10.87% 184.000 204.000 - -17.07% -
403.gcc (max) 8.82% 34.000 37.000 - -7.50% -
403.gcc (max) (percentage) 7.89% 0.380 0.410 - -6.82% -
447.dealII (avg) (percentage) 4.76% 1.470 1.540 - 0.00% -
471.omnetpp (avg) (percentage) 3.85% 0.780 0.810 - 5.19% -
471.omnetpp 3.61% 83.000 86.000 - 4.88% -
471.omnetpp (avg) 3.61% 83.000 86.000 - 4.88% -
470.lbm 3.42% 322.000 333.000 - -13.05% -
470.lbm (avg) 3.42% 322.000 333.000 - -13.05% -
447.dealII 3.33% 120.000 124.000 - 0.81% -
447.dealII (avg) 3.33% 120.000 124.000 - 0.81% -
473.astar (max) 2.63% 114.000 117.000 - -2.50% -
473.astar (max) (percentage) 2.50% 0.800 0.820 - -2.38% -
482.sphinx3 1.23% 489.000 495.000 - -15.53% -
482.sphinx3 (avg) 1.23% 489.000 495.000 - -15.53% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
444.namd -5.90% 13495.000 12699.000 - 0.24% -
429.mcf -2.09% 11999.000 11748.000 - -0.10% -
453.povray -2.06% 6857.000 6716.000 - -0.34% -
433.milc -1.55% 13482.000 13273.000 - -0.45% -
447.dealII -1.42% 8196.000 8080.000 - 0.49% -
401.bzip2 -1.05% 15683.000 15519.000 - 0.23% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
482.sphinx3 (unexplained diff vs previous) (percentage) -95.93% 3.440 0.140 - -95.56% -
482.sphinx3 (unexplained diff vs previous) -95.80% 904.000 38.000 - -95.56% -
470.lbm (unexplained diff vs previous) (percentage) -90.78% 3.580 0.330 - -91.58% -
470.lbm (unexplained diff vs previous) -90.54% 497.000 47.000 - -91.61% -
429.mcf (unexplained diff vs previous) (percentage) -82.14% 0.560 0.100 - -83.61% -
429.mcf (unexplained diff vs previous) -82.09% 67.000 12.000 - -83.33% -
433.milc (unexplained diff vs previous) -67.03% 182.000 60.000 - -76.10% -
433.milc (unexplained diff vs previous) (percentage) -66.67% 1.350 0.450 - -76.06% -
450.soplex (unexplained diff vs previous) -62.20% 82.000 31.000 - -69.61% -
450.soplex (unexplained diff vs previous) (percentage) -61.40% 0.570 0.220 - -69.01% -
470.lbm (max) (percentage) -44.77% 5.540 3.060 - -25.00% -
470.lbm (max) -43.04% 769.000 438.000 - -24.74% -
471.omnetpp (max) -42.86% 189.000 108.000 - 47.95% -
471.omnetpp (max) (percentage) -42.05% 1.760 1.020 - 50.00% -
458.sjeng (max) (percentage) -30.00% 0.400 0.280 - -9.68% -
458.sjeng (max) -29.23% 65.000 46.000 - -8.00% -
482.sphinx3 (max) (percentage) -26.51% 3.320 2.440 - -25.15% -
482.sphinx3 (max) -24.28% 873.000 661.000 - -25.23% -
462.libquantum (max) (percentage) -20.00% 0.200 0.160 - 45.45% -
429.mcf (max) -19.40% 67.000 54.000 - -23.94% -
462.libquantum (max) -19.23% 26.000 21.000 - 50.00% -
464.h264ref (max) -18.33% 60.000 49.000 - 2.08% -
464.h264ref (max) (percentage) -17.95% 0.390 0.320 - 3.23% -
464.h264ref (unexplained diff vs previous) (percentage) -17.95% 0.390 0.320 - 52.38% -
429.mcf (max) (percentage) -17.86% 0.560 0.460 - -24.59% -
464.h264ref (unexplained diff vs previous) -16.67% 60.000 50.000 - 51.52% -
444.namd (max) -16.55% 882.000 736.000 - -3.92% -
444.namd (max) (percentage) -11.31% 6.540 5.800 - -4.13% -
464.h264ref -9.80% 51.000 46.000 - 0.00% -
464.h264ref (avg) -9.80% 51.000 46.000 - 0.00% -
464.h264ref (avg) (percentage) -9.09% 0.330 0.300 - 0.00% -
450.soplex (max) -8.54% 82.000 75.000 - -25.74% -
453.povray (max) -7.58% 132.000 122.000 - -4.69% -
453.povray -7.53% 93.000 86.000 - 0.00% -
453.povray (avg) -7.53% 93.000 86.000 - 0.00% -
450.soplex (max) (percentage) -7.02% 0.570 0.530 - -25.35% -
401.bzip2 -5.88% 34.000 32.000 - 0.00% -
401.bzip2 (avg) -5.88% 34.000 32.000 - 0.00% -
453.povray (avg) (percentage) -5.84% 1.370 1.290 - 0.00% -
453.povray (max) (percentage) -5.18% 1.930 1.830 - -4.19% -
401.bzip2 (avg) (percentage) -4.55% 0.220 0.210 - 0.00% -
482.sphinx3 (avg) (percentage) -1.61% 1.860 1.830 - -15.28% -

Report Time: 0.38s
Execution (seconds)
Name Current %
401.bzip2 15519.000 -
403.gcc 9198.000 -
429.mcf 11748.000 -
433.milc 13273.000 -
444.namd 12699.000 -
445.gobmk 12403.000 -
447.dealII 8080.000 -
450.soplex 14332.000 -
453.povray 6716.000 -
456.hmmer 13704.000 -
458.sjeng 16432.000 -
462.libquantum 13135.000 -
464.h264ref 15582.000 -
470.lbm 14321.000 -
471.omnetpp 10646.000 -1.02%
473.astar 14287.000 -
482.sphinx3 27096.000 -
483.xalancbmk 9456.000 -
Geometric Mean 12663.423 -
Code Size (bytes)
Name Current %
401.bzip2 39602.000 -
403.gcc 1962298.000 -
429.mcf 7935.000 -
433.milc 74979.000 -
444.namd 175704.000 -
445.gobmk 1312252.000 -
447.dealII 1775345.000 -
450.soplex 199849.000 -1.58%
453.povray 593317.000 -
456.hmmer 168296.000 -
458.sjeng 97982.000 -
462.libquantum 26720.000 -
464.h264ref 355338.000 -
470.lbm 9739.000 -
471.omnetpp 377123.000 -
473.astar 23248.000 -
482.sphinx3 115840.000 -
483.xalancbmk 2287837.000 -
Geometric Mean 164698.296 -
sample variability
Name Current %
401.bzip2 32.000 -
401.bzip2 (avg) 32.000 -
401.bzip2 (avg) (percentage) 0.210 -
401.bzip2 (max) 54.000 -1.82%
401.bzip2 (max) (percentage) 0.350 -2.78%
403.gcc 93.000 -
403.gcc (avg) 93.000 -
403.gcc (avg) (percentage) 1.020 -
403.gcc (max) 37.000 -7.50%
403.gcc (max) (percentage) 0.410 -6.82%
429.mcf 74.000 -12.94%
429.mcf (avg) 74.000 -12.94%
429.mcf (avg) (percentage) 0.630 -13.70%
429.mcf (max) 54.000 -23.94%
429.mcf (max) (percentage) 0.460 -24.59%
429.mcf (unexplained diff vs previous) 12.000 -83.33%
429.mcf (unexplained diff vs previous) (percentage) 0.100 -83.61%
433.milc 157.000 -10.29%
433.milc (avg) 157.000 -10.29%
433.milc (avg) (percentage) 1.190 -9.85%
433.milc (max) 204.000 -17.07%
433.milc (max) (percentage) 1.540 -16.76%
433.milc (unexplained diff vs previous) 60.000 -76.10%
433.milc (unexplained diff vs previous) (percentage) 0.450 -76.06%
444.namd 576.000 -
444.namd (avg) 576.000 -
444.namd (avg) (percentage) 4.540 -
444.namd (max) 736.000 -3.92%
444.namd (max) (percentage) 5.800 -4.13%
445.gobmk 40.000 -2.44%
445.gobmk (avg) 40.000 -2.44%
445.gobmk (avg) (percentage) 0.330 -
445.gobmk (max) 40.000 -9.09%
445.gobmk (max) (percentage) 0.330 -8.33%
447.dealII 124.000 -
447.dealII (avg) 124.000 -
447.dealII (avg) (percentage) 1.540 -
447.dealII (max) 154.000 -6.67%
447.dealII (max) (percentage) 1.910 -7.28%
450.soplex 87.000 -10.31%
450.soplex (avg) 87.000 -10.31%
450.soplex (avg) (percentage) 0.610 -10.29%
450.soplex (max) 75.000 -25.74%
450.soplex (max) (percentage) 0.530 -25.35%
450.soplex (unexplained diff vs previous) 31.000 -69.61%
450.soplex (unexplained diff vs previous) (percentage) 0.220 -69.01%
453.povray 86.000 -
453.povray (avg) 86.000 -
453.povray (avg) (percentage) 1.290 -
453.povray (max) 122.000 -4.69%
453.povray (max) (percentage) 1.830 -4.19%
456.hmmer 79.000 -
456.hmmer (avg) 79.000 -
456.hmmer (avg) (percentage) 0.580 -
456.hmmer (max) 135.000 -3.57%
456.hmmer (max) (percentage) 0.990 -3.88%
458.sjeng 305.000 -
458.sjeng (avg) 305.000 -
458.sjeng (avg) (percentage) 1.860 -
458.sjeng (max) 46.000 -8.00%
458.sjeng (max) (percentage) 0.280 -9.68%
462.libquantum 27.000 -3.57%
462.libquantum (avg) 27.000 -3.57%
462.libquantum (avg) (percentage) 0.210 -4.55%
462.libquantum (max) 21.000 50.00%
462.libquantum (max) (percentage) 0.160 45.45%
462.libquantum (unexplained diff vs previous) 21.000 200.00%
462.libquantum (unexplained diff vs previous) (percentage) 0.160 220.00%
464.h264ref 46.000 -
464.h264ref (avg) 46.000 -
464.h264ref (avg) (percentage) 0.300 -
464.h264ref (max) 49.000 2.08%
464.h264ref (max) (percentage) 0.320 3.23%
464.h264ref (unexplained diff vs previous) 50.000 51.52%
464.h264ref (unexplained diff vs previous) (percentage) 0.320 52.38%
470.lbm 333.000 -13.05%
470.lbm (avg) 333.000 -13.05%
470.lbm (avg) (percentage) 2.330 -13.38%
470.lbm (max) 438.000 -24.74%
470.lbm (max) (percentage) 3.060 -25.00%
470.lbm (unexplained diff vs previous) 47.000 -91.61%
470.lbm (unexplained diff vs previous) (percentage) 0.330 -91.58%
471.omnetpp 86.000 4.88%
471.omnetpp (avg) 86.000 4.88%
471.omnetpp (avg) (percentage) 0.810 5.19%
471.omnetpp (max) 108.000 47.95%
471.omnetpp (max) (percentage) 1.020 50.00%
471.omnetpp (unexplained diff vs previous) 110.000 1471.43%
471.omnetpp (unexplained diff vs previous) (percentage) 1.030 1371.43%
473.astar 87.000 -
473.astar (avg) 87.000 -
473.astar (avg) (percentage) 0.610 -
473.astar (max) 117.000 -2.50%
473.astar (max) (percentage) 0.820 -2.38%
482.sphinx3 495.000 -15.53%
482.sphinx3 (avg) 495.000 -15.53%
482.sphinx3 (avg) (percentage) 1.830 -15.28%
482.sphinx3 (max) 661.000 -25.23%
482.sphinx3 (max) (percentage) 2.440 -25.15%
482.sphinx3 (unexplained diff vs previous) 38.000 -95.56%
482.sphinx3 (unexplained diff vs previous) (percentage) 0.140 -95.56%
483.xalancbmk 60.000 -
483.xalancbmk (avg) 60.000 -
483.xalancbmk (avg) (percentage) 0.640 -
483.xalancbmk (max) 83.000 -1.19%
483.xalancbmk (max) (percentage) 0.880 -2.22%
Geometric Mean 11.795 -13.64%