Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/374a600df7207fbe2002e754a799c7595a0e4833 (+ 1173 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz-build/110/
View Options
Report
Filter

llvm-arm-master-Oz test results

Run Order Start Time Duration
Current 0110 (llvmorg-19-init-01061-g374a600df720) 2024-01-26T14:00:04 0:00:00
Previous 0108 (llvmorg-18-init-18248-ge96242b4e430) 2024-01-19T21:55:17 0:00:00
Baseline 0014 (no-change) 2023-02-19T08:08:29 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 2 11
Performance Improvements 49 91
Added Tests 1 7
Unchanged Tests 716 659
Total Tests 768

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc 4.92% 61.000 64.000 - 6.67% -
453.povray 1.15% 520.000 526.000 - -0.57% -

Performance Improvements - Code Size Δ Previous Current σ Δ (B) σ (B)
401.bzip2 -1.48% 35769.000 35241.000 - -1.51% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
444.namd -1.73% 810.000 796.000 - -0.38% -
462.libquantum -1.64% 183.000 180.000 - -2.17% -
483.xalancbmk -1.52% 4014.000 3953.000 - -0.95% -
458.sjeng -1.07% 9582.000 9479.000 - -1.65% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) (percentage) -71.79% 1.560 0.440 - 0.00% -
471.omnetpp (unexplained diff vs previous) -71.43% 49.000 14.000 - 0.00% -
450.soplex (max) -50.00% 8.000 4.000 - -42.86% -
450.soplex (max) (percentage) -40.13% 1.520 0.910 - -31.58% -
471.omnetpp (max) (percentage) -35.53% 1.970 1.270 - -92.20% -
471.omnetpp (max) -34.43% 61.000 40.000 - -92.11% -
429.mcf (max) -16.67% 6.000 5.000 - -97.69% -
450.soplex (unexplained diff vs previous) -16.67% 6.000 5.000 - 0.00% -
429.mcf (max) (percentage) -16.67% 0.600 0.500 - -97.33% -
450.soplex (unexplained diff vs previous) (percentage) -16.51% 1.090 0.910 - 0.00% -
482.sphinx3 (max) -14.29% 7.000 6.000 - -95.77% -
470.lbm (max) -13.33% 15.000 13.000 - -97.07% -
470.lbm (max) (percentage) -12.50% 0.400 0.350 - -96.90% -
450.soplex -12.50% 8.000 7.000 - 133.33% -
450.soplex (avg) -12.50% 8.000 7.000 - 133.33% -
445.gobmk (max) -11.11% 18.000 16.000 - -91.21% -
456.hmmer (max) -11.11% 9.000 8.000 - -97.52% -
433.milc (max) -9.09% 11.000 10.000 - -93.46% -
464.h264ref (max) -9.09% 22.000 20.000 - -93.15% -
445.gobmk (max) (percentage) -8.82% 0.340 0.310 - -90.75% -
482.sphinx3 (max) (percentage) -8.45% 0.710 0.650 - -95.47% -
473.astar (max) -8.33% 12.000 11.000 - -99.65% -
456.hmmer (max) (percentage) -7.69% 0.260 0.240 - -97.31% -
401.bzip2 (max) -7.69% 13.000 12.000 - -85.88% -
458.sjeng (max) -6.45% 62.000 58.000 - -93.48% -
453.povray (max) (percentage) -6.35% 1.890 1.770 - -63.35% -
447.dealII (max) -6.25% 32.000 30.000 - -73.91% -
433.milc (max) (percentage) -6.25% 0.960 0.900 - -93.20% -
447.dealII (max) (percentage) -6.11% 1.310 1.230 - -74.16% -
464.h264ref (max) (percentage) -5.41% 0.370 0.350 - -92.77% -
462.libquantum (max) (percentage) -5.00% 0.400 0.380 - -93.52% -
458.sjeng (max) (percentage) -4.62% 0.650 0.620 - -93.28% -
401.bzip2 (max) (percentage) -4.55% 0.440 0.420 - -85.21% -
444.namd (max) (percentage) -2.74% 0.730 0.710 - -94.60% -
470.lbm -2.56% 39.000 38.000 - -75.64% -
470.lbm (avg) -2.56% 39.000 38.000 - -75.64% -
471.omnetpp -2.33% 43.000 42.000 - -61.82% -
471.omnetpp (avg) -2.33% 43.000 42.000 - -61.82% -
403.gcc (max) (percentage) -2.18% 7.340 7.180 - -38.21% -
470.lbm (avg) (percentage) -1.98% 1.010 0.990 - -75.13% -
483.xalancbmk -1.52% 132.000 130.000 - -52.03% -
483.xalancbmk (avg) -1.52% 132.000 130.000 - -52.03% -
471.omnetpp (avg) (percentage) -1.45% 1.380 1.360 - -61.58% -
450.soplex (avg) (percentage) -1.37% 1.460 1.440 - 118.18% -

Added Tests - sample variability
462.libquantum (max)

Run-Over-Baseline Changes Detail

Performance Regressions - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc 6.67% 60.000 64.000 - 4.92% -
433.milc 1.64% 1162.000 1181.000 - 0.17% -
473.astar 1.41% 4955.000 5025.000 - -0.67% -
471.omnetpp 1.38% 3116.000 3159.000 - 0.45% -
447.dealII 1.32% 2432.000 2464.000 - 0.37% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
450.soplex 133.33% 3.000 7.000 - -12.50% -
450.soplex (avg) 133.33% 3.000 7.000 - -12.50% -
450.soplex (avg) (percentage) 118.18% 0.660 1.440 - -1.37% -
403.gcc (avg) (percentage) 116.67% 1.740 3.770 - 0.00% -
403.gcc 100.00% 1.000 2.000 - 0.00% -
403.gcc (avg) 100.00% 1.000 2.000 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -4.28% 22066.000 21122.000 - 0.00% -
401.bzip2 -1.51% 35781.000 35241.000 - -1.48% -
453.povray -1.24% 549774.000 542938.000 - -0.06% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -2.17% 184.000 180.000 - -1.64% -
458.sjeng -1.65% 9638.000 9479.000 - -1.07% -
450.soplex -1.44% 555.000 547.000 - -0.91% -
464.h264ref -1.31% 6043.000 5964.000 - -0.32% -
401.bzip2 -1.09% 3014.000 2981.000 - 0.37% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
473.astar (max) -99.65% 3144.000 11.000 - -8.33% -
473.astar (max) (percentage) -99.64% 63.460 0.230 - -4.17% -
483.xalancbmk (max) -99.63% 1883.000 7.000 - 0.00% -
483.xalancbmk (max) (percentage) -99.62% 47.200 0.180 - 0.00% -
429.mcf (max) -97.69% 216.000 5.000 - -16.67% -
456.hmmer (max) -97.52% 322.000 8.000 - -11.11% -
429.mcf (max) (percentage) -97.33% 18.730 0.500 - -16.67% -
456.hmmer (max) (percentage) -97.31% 8.910 0.240 - -7.69% -
470.lbm (max) -97.07% 443.000 13.000 - -13.33% -
470.lbm (max) (percentage) -96.90% 11.290 0.350 - -12.50% -
482.sphinx3 (max) -95.77% 142.000 6.000 - -14.29% -
482.sphinx3 (max) (percentage) -95.47% 14.360 0.650 - -8.45% -
444.namd (max) -95.19% 104.000 5.000 - 0.00% -
444.namd (max) (percentage) -94.60% 13.140 0.710 - -2.74% -
462.libquantum (max) (percentage) -93.52% 5.860 0.380 - -5.00% -
458.sjeng (max) -93.48% 889.000 58.000 - -6.45% -
433.milc (max) -93.46% 153.000 10.000 - -9.09% -
458.sjeng (max) (percentage) -93.28% 9.230 0.620 - -4.62% -
433.milc (max) (percentage) -93.20% 13.240 0.900 - -6.25% -
464.h264ref (max) -93.15% 292.000 20.000 - -9.09% -
464.h264ref (max) (percentage) -92.77% 4.840 0.350 - -5.41% -
471.omnetpp (max) (percentage) -92.20% 16.280 1.270 - -35.53% -
471.omnetpp (max) -92.11% 507.000 40.000 - -34.43% -
445.gobmk (max) -91.21% 182.000 16.000 - -11.11% -
445.gobmk (max) (percentage) -90.75% 3.350 0.310 - -8.82% -
401.bzip2 (max) -85.88% 85.000 12.000 - -7.69% -
401.bzip2 (max) (percentage) -85.21% 2.840 0.420 - -4.55% -
470.lbm -75.64% 156.000 38.000 - -2.56% -
470.lbm (avg) -75.64% 156.000 38.000 - -2.56% -
470.lbm (avg) (percentage) -75.13% 3.980 0.990 - -1.98% -
473.astar (avg) (percentage) -74.90% 5.020 1.260 - 0.00% -
473.astar -74.60% 248.000 63.000 - 0.00% -
473.astar (avg) -74.60% 248.000 63.000 - 0.00% -
447.dealII (max) (percentage) -74.16% 4.760 1.230 - -6.11% -
447.dealII (max) -73.91% 115.000 30.000 - -6.25% -
456.hmmer (avg) (percentage) -73.35% 3.640 0.970 - 0.00% -
464.h264ref -73.33% 120.000 32.000 - 0.00% -
464.h264ref (avg) -73.33% 120.000 32.000 - 0.00% -
456.hmmer -73.28% 131.000 35.000 - 0.00% -
456.hmmer (avg) -73.28% 131.000 35.000 - 0.00% -
464.h264ref (avg) (percentage) -73.00% 2.000 0.540 - 0.00% -
444.namd -70.59% 34.000 10.000 - 0.00% -
444.namd (avg) -70.59% 34.000 10.000 - 0.00% -
482.sphinx3 -70.59% 34.000 10.000 - 0.00% -
482.sphinx3 (avg) -70.59% 34.000 10.000 - 0.00% -
445.gobmk -70.42% 71.000 21.000 - 0.00% -
445.gobmk (avg) -70.42% 71.000 21.000 - 0.00% -
445.gobmk (avg) (percentage) -69.47% 1.310 0.400 - 0.00% -
444.namd (avg) (percentage) -68.54% 4.260 1.340 - 0.00% -
482.sphinx3 (avg) (percentage) -68.50% 3.460 1.090 - 0.00% -
458.sjeng -67.30% 159.000 52.000 - 0.00% -
458.sjeng (avg) -67.30% 159.000 52.000 - 0.00% -
458.sjeng (avg) (percentage) -66.67% 1.650 0.550 - 0.00% -
453.povray (max) -64.00% 25.000 9.000 - 0.00% -
429.mcf -63.64% 55.000 20.000 - 0.00% -
429.mcf (avg) -63.64% 55.000 20.000 - 0.00% -
453.povray (max) (percentage) -63.35% 4.830 1.770 - -6.35% -
429.mcf (avg) (percentage) -62.89% 4.770 1.770 - 0.00% -
471.omnetpp -61.82% 110.000 42.000 - -2.33% -
471.omnetpp (avg) -61.82% 110.000 42.000 - -2.33% -
471.omnetpp (avg) (percentage) -61.58% 3.540 1.360 - -1.45% -
462.libquantum (avg) (percentage) -56.25% 1.600 0.700 - 0.00% -
401.bzip2 -52.94% 34.000 16.000 - 0.00% -
401.bzip2 (avg) -52.94% 34.000 16.000 - 0.00% -
401.bzip2 (avg) (percentage) -52.59% 1.160 0.550 - 0.00% -
483.xalancbmk -52.03% 271.000 130.000 - -1.52% -
483.xalancbmk (avg) -52.03% 271.000 130.000 - -1.52% -
483.xalancbmk (avg) (percentage) -51.69% 6.810 3.290 - 0.00% -
462.libquantum -50.00% 2.000 1.000 - 0.00% -
462.libquantum (avg) -50.00% 2.000 1.000 - 0.00% -
433.milc (avg) (percentage) -49.02% 2.040 1.040 - 0.00% -
433.milc -47.83% 23.000 12.000 - 0.00% -
433.milc (avg) -47.83% 23.000 12.000 - 0.00% -
450.soplex (max) -42.86% 7.000 4.000 - -50.00% -
453.povray -42.86% 7.000 4.000 - 0.00% -
453.povray (avg) -42.86% 7.000 4.000 - 0.00% -
453.povray (avg) (percentage) -39.33% 1.500 0.910 - 0.00% -
403.gcc (max) (percentage) -38.21% 11.620 7.180 - -2.18% -
447.dealII -37.04% 27.000 17.000 - 0.00% -
447.dealII (avg) -37.04% 27.000 17.000 - 0.00% -
447.dealII (avg) (percentage) -36.84% 1.140 0.720 - 0.00% -
403.gcc (max) -33.33% 6.000 4.000 - 0.00% -
450.soplex (max) (percentage) -31.58% 1.330 0.910 - -40.13% -

Added Tests - sample variability
450.soplex (unexplained diff vs previous)
450.soplex (unexplained diff vs previous) (percentage)
462.libquantum (max)
470.lbm (unexplained diff vs previous)
470.lbm (unexplained diff vs previous) (percentage)
471.omnetpp (unexplained diff vs previous)
471.omnetpp (unexplained diff vs previous) (percentage)

Report Time: 0.35s
Execution (seconds)
Name Current %
401.bzip2 2981.000 -
403.gcc 64.000 4.92%
429.mcf 1157.000 -
433.milc 1181.000 -
444.namd 796.000 -1.73%
445.gobmk 5411.000 -
447.dealII 2464.000 -
450.soplex 547.000 -
453.povray 526.000 1.15%
456.hmmer 3610.000 -
458.sjeng 9479.000 -1.07%
462.libquantum 180.000 -1.64%
464.h264ref 5964.000 -
470.lbm 3901.000 -
471.omnetpp 3159.000 -
473.astar 5025.000 -
482.sphinx3 988.000 -
483.xalancbmk 3953.000 -1.52%
Geometric Mean 1636.305 -
Code Size (bytes)
Name Current %
401.bzip2 35241.000 -1.48%
403.gcc 1842737.000 -
429.mcf 7466.000 -
433.milc 69213.000 -
444.namd 153471.000 -
445.gobmk 1289606.000 -
447.dealII 1472199.000 -
450.soplex 180010.000 -
453.povray 542938.000 -
456.hmmer 156162.000 -
458.sjeng 90273.000 -
462.libquantum 21122.000 -
464.h264ref 331704.000 -
470.lbm 9718.000 -
471.omnetpp 361618.000 -
473.astar 21347.000 -
482.sphinx3 107819.000 -
483.xalancbmk 2166390.000 -
Geometric Mean 150940.396 -
sample variability
Name Current %
401.bzip2 16.000 -
401.bzip2 (avg) 16.000 -
401.bzip2 (avg) (percentage) 0.550 -
401.bzip2 (max) 12.000 -7.69%
401.bzip2 (max) (percentage) 0.420 -4.55%
403.gcc 2.000 -
403.gcc (avg) 2.000 -
403.gcc (avg) (percentage) 3.770 -
403.gcc (max) 4.000 -
403.gcc (max) (percentage) 7.180 -2.18%
429.mcf 20.000 -
429.mcf (avg) 20.000 -
429.mcf (avg) (percentage) 1.770 -
429.mcf (max) 5.000 -16.67%
429.mcf (max) (percentage) 0.500 -16.67%
433.milc 12.000 -
433.milc (avg) 12.000 -
433.milc (avg) (percentage) 1.040 -
433.milc (max) 10.000 -9.09%
433.milc (max) (percentage) 0.900 -6.25%
444.namd 10.000 -
444.namd (avg) 10.000 -
444.namd (avg) (percentage) 1.340 -
444.namd (max) 5.000 -
444.namd (max) (percentage) 0.710 -2.74%
445.gobmk 21.000 -
445.gobmk (avg) 21.000 -
445.gobmk (avg) (percentage) 0.400 -
445.gobmk (max) 16.000 -11.11%
445.gobmk (max) (percentage) 0.310 -8.82%
447.dealII 17.000 -
447.dealII (avg) 17.000 -
447.dealII (avg) (percentage) 0.720 -
447.dealII (max) 30.000 -6.25%
447.dealII (max) (percentage) 1.230 -6.11%
450.soplex 7.000 -12.50%
450.soplex (avg) 7.000 -12.50%
450.soplex (avg) (percentage) 1.440 -1.37%
450.soplex (max) 4.000 -50.00%
450.soplex (max) (percentage) 0.910 -40.13%
450.soplex (unexplained diff vs previous) 5.000 -16.67%
450.soplex (unexplained diff vs previous) (percentage) 0.910 -16.51%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.910 -
453.povray (max) 9.000 -
453.povray (max) (percentage) 1.770 -6.35%
456.hmmer 35.000 -
456.hmmer (avg) 35.000 -
456.hmmer (avg) (percentage) 0.970 -
456.hmmer (max) 8.000 -11.11%
456.hmmer (max) (percentage) 0.240 -7.69%
458.sjeng 52.000 -
458.sjeng (avg) 52.000 -
458.sjeng (avg) (percentage) 0.550 -
458.sjeng (max) 58.000 -6.45%
458.sjeng (max) (percentage) 0.620 -4.62%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 0.700 -
462.libquantum (max) 0.000 -
462.libquantum (max) (percentage) 0.380 -5.00%
464.h264ref 32.000 -
464.h264ref (avg) 32.000 -
464.h264ref (avg) (percentage) 0.540 -
464.h264ref (max) 20.000 -9.09%
464.h264ref (max) (percentage) 0.350 -5.41%
470.lbm 38.000 -2.56%
470.lbm (avg) 38.000 -2.56%
470.lbm (avg) (percentage) 0.990 -1.98%
470.lbm (max) 13.000 -13.33%
470.lbm (max) (percentage) 0.350 -12.50%
470.lbm (unexplained diff vs previous) 6.000 -
470.lbm (unexplained diff vs previous) (percentage) 0.150 -
471.omnetpp 42.000 -2.33%
471.omnetpp (avg) 42.000 -2.33%
471.omnetpp (avg) (percentage) 1.360 -1.45%
471.omnetpp (max) 40.000 -34.43%
471.omnetpp (max) (percentage) 1.270 -35.53%
471.omnetpp (unexplained diff vs previous) 14.000 -71.43%
471.omnetpp (unexplained diff vs previous) (percentage) 0.440 -71.79%
473.astar 63.000 -
473.astar (avg) 63.000 -
473.astar (avg) (percentage) 1.260 -
473.astar (max) 11.000 -8.33%
473.astar (max) (percentage) 0.230 -
482.sphinx3 10.000 -
482.sphinx3 (avg) 10.000 -
482.sphinx3 (avg) (percentage) 1.090 -
482.sphinx3 (max) 6.000 -14.29%
482.sphinx3 (max) (percentage) 0.650 -8.45%
483.xalancbmk 130.000 -1.52%
483.xalancbmk (avg) 130.000 -1.52%
483.xalancbmk (avg) (percentage) 3.290 -
483.xalancbmk (max) 7.000 -
483.xalancbmk (max) (percentage) 0.180 -
Geometric Mean 3.751 -