Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/3d5932b114fbd193009d23a18ddd9a5c3135202e (+ 1077 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz_LTO-build/24/
View Options
Report
Filter

llvm-arm-master-Oz_LTO test results

Run Order Start Time Duration
Current 0024 (llvmorg-17-init-10021-g3d5932b114fb) 2023-04-24T11:39:05 0:00:00
Previous 0024 (llvmorg-17-init-10021-g3d5932b114fb) 2023-04-24T11:39:05 0:00:00
Baseline 0001 (llvmorg-16-init-17036-g4f40923103d2) 2023-01-06T17:55:45 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 0 39
Performance Improvements 0 91
Added Tests 2 6
Unchanged Tests 766 632
Total Tests 768

Run-Over-Run Changes Detail

Added Tests - sample variability
403.gcc
403.gcc (avg)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 2.13% 146765.000 149884.000 - 0.00% -
471.omnetpp 1.50% 316676.000 321429.000 - 0.00% -
401.bzip2 1.07% 33698.000 34057.000 - 0.00% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
462.libquantum (unexplained diff vs previous) (percentage) 41600.00% 0.010 4.170 - 0.00% -
433.milc (unexplained diff vs previous) (percentage) 850.00% 0.100 0.950 - 0.00% -
462.libquantum (unexplained diff vs previous) 700.00% 1.000 8.000 - 0.00% -
453.povray (max) (percentage) 630.56% 0.360 2.630 - 0.00% -
401.bzip2 (unexplained diff vs previous) (percentage) 540.00% 0.200 1.280 - 0.00% -
464.h264ref (avg) (percentage) 289.66% 0.290 1.130 - 0.00% -
429.mcf (unexplained diff vs previous) (percentage) 275.00% 0.160 0.600 - 0.00% -
456.hmmer (avg) (percentage) 269.06% 1.390 5.130 - 0.00% -
483.xalancbmk (max) (percentage) 262.25% 12.900 46.730 - 0.00% -
483.xalancbmk (avg) (percentage) 252.83% 2.120 7.480 - 0.00% -
471.omnetpp (avg) (percentage) 244.37% 1.600 5.510 - 0.00% -
462.libquantum (avg) (percentage) 180.52% 0.770 2.160 - 0.00% -
473.astar (avg) (percentage) 161.96% 0.920 2.410 - 0.00% -
429.mcf (avg) (percentage) 97.64% 2.120 4.190 - 0.00% -
433.milc (max) (percentage) 65.00% 0.800 1.320 - 0.00% -
458.sjeng (avg) (percentage) 62.50% 0.800 1.300 - 0.00% -
464.h264ref 59.52% 42.000 67.000 - 0.00% -
464.h264ref (avg) 59.52% 42.000 67.000 - 0.00% -
482.sphinx3 (max) (percentage) 58.61% 3.310 5.250 - 0.00% -
450.soplex (avg) (percentage) 40.00% 0.500 0.700 - 0.00% -
473.astar (max) (percentage) 38.81% 6.700 9.300 - 0.00% -
403.gcc (avg) (percentage) 36.36% 0.550 0.750 - 0.00% -
445.gobmk (avg) (percentage) 35.82% 0.670 0.910 - 0.00% -
483.xalancbmk (max) 34.33% 1209.000 1624.000 - 0.00% -
401.bzip2 (avg) (percentage) 33.85% 0.650 0.870 - 0.00% -
483.xalancbmk 31.31% 198.000 260.000 - 0.00% -
483.xalancbmk (avg) 31.31% 198.000 260.000 - 0.00% -
447.dealII (max) (percentage) 28.15% 2.380 3.050 - 0.00% -
482.sphinx3 (avg) (percentage) 27.59% 1.740 2.220 - 0.00% -
447.dealII (avg) (percentage) 19.05% 1.050 1.250 - 0.00% -
456.hmmer 18.90% 164.000 195.000 - 0.00% -
456.hmmer (avg) 18.90% 164.000 195.000 - 0.00% -
401.bzip2 (unexplained diff vs previous) 18.18% 33.000 39.000 - 0.00% -
453.povray (avg) (percentage) 6.25% 0.800 0.850 - 0.00% -
450.soplex (max) (percentage) 4.85% 1.650 1.730 - 0.00% -
444.namd (avg) (percentage) 3.89% 2.570 2.670 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
429.mcf -2.60% 7239.000 7051.000 - 0.00% -
453.povray -1.97% 524493.000 514139.000 - 0.00% -
462.libquantum -1.71% 12504.000 12290.000 - 0.00% -
433.milc -1.17% 47001.000 46453.000 - 0.00% -
470.lbm -1.05% 9338.000 9240.000 - 0.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.40% 9509.000 57.000 - 0.00% -
462.libquantum -98.67% 14409.000 192.000 - 0.00% -
450.soplex -96.47% 15112.000 534.000 - 0.00% -
482.sphinx3 -96.23% 26682.000 1006.000 - 0.00% -
444.namd -93.44% 12400.000 813.000 - 0.00% -
453.povray -92.80% 6987.000 503.000 - 0.00% -
429.mcf -91.63% 13988.000 1171.000 - 0.00% -
433.milc -91.46% 13601.000 1162.000 - 0.00% -
401.bzip2 -81.17% 16137.000 3039.000 - 0.00% -
447.dealII -78.46% 9380.000 2020.000 - 0.00% -
471.omnetpp -72.71% 10477.000 2859.000 - 0.00% -
470.lbm -70.00% 13259.000 3978.000 - 0.00% -
456.hmmer -67.83% 11854.000 3814.000 - 0.00% -
483.xalancbmk -62.93% 9376.000 3476.000 - 0.00% -
473.astar -62.56% 12272.000 4595.000 - 0.00% -
464.h264ref -58.57% 14498.000 6006.000 - 0.00% -
445.gobmk -55.85% 11093.000 4898.000 - 0.00% -
458.sjeng -51.36% 17055.000 8295.000 - 0.00% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
429.mcf (max) -99.69% 2281.000 7.000 - 0.00% -
403.gcc (max) -99.55% 221.000 1.000 - 0.00% -
470.lbm (unexplained diff vs previous) -99.38% 804.000 5.000 - 0.00% -
462.libquantum (max) -99.02% 919.000 9.000 - 0.00% -
456.hmmer (max) -98.95% 1242.000 13.000 - 0.00% -
471.omnetpp (max) -97.94% 972.000 20.000 - 0.00% -
470.lbm (unexplained diff vs previous) (percentage) -97.85% 6.060 0.130 - 0.00% -
456.hmmer (max) (percentage) -96.66% 10.480 0.350 - 0.00% -
450.soplex (max) -96.39% 249.000 9.000 - 0.00% -
462.libquantum -96.36% 110.000 4.000 - 0.00% -
462.libquantum (avg) -96.36% 110.000 4.000 - 0.00% -
429.mcf (max) (percentage) -96.32% 16.310 0.600 - 0.00% -
444.namd (max) -96.21% 1003.000 38.000 - 0.00% -
450.soplex -96.00% 75.000 3.000 - 0.00% -
450.soplex (avg) -96.00% 75.000 3.000 - 0.00% -
482.sphinx3 -95.26% 464.000 22.000 - 0.00% -
482.sphinx3 (avg) -95.26% 464.000 22.000 - 0.00% -
482.sphinx3 (max) -94.11% 883.000 52.000 - 0.00% -
444.namd -93.40% 318.000 21.000 - 0.00% -
444.namd (avg) -93.40% 318.000 21.000 - 0.00% -
453.povray -92.73% 55.000 4.000 - 0.00% -
453.povray (avg) -92.73% 55.000 4.000 - 0.00% -
471.omnetpp (max) (percentage) -92.35% 9.280 0.710 - 0.00% -
433.milc -92.24% 116.000 9.000 - 0.00% -
433.milc (avg) -92.24% 116.000 9.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) -92.23% 103.000 8.000 - 0.00% -
401.bzip2 (max) -90.31% 609.000 59.000 - 0.00% -
456.hmmer (unexplained diff vs previous) -89.13% 46.000 5.000 - 0.00% -
458.sjeng (max) -88.13% 876.000 104.000 - 0.00% -
433.milc (max) -86.11% 108.000 15.000 - 0.00% -
429.mcf -83.45% 296.000 49.000 - 0.00% -
429.mcf (avg) -83.45% 296.000 49.000 - 0.00% -
445.gobmk (unexplained diff vs previous) -80.00% 40.000 8.000 - 0.00% -
458.sjeng (max) (percentage) -75.49% 5.140 1.260 - 0.00% -
470.lbm (max) -75.33% 758.000 187.000 - 0.00% -
401.bzip2 -75.00% 104.000 26.000 - 0.00% -
401.bzip2 (avg) -75.00% 104.000 26.000 - 0.00% -
447.dealII -74.49% 98.000 25.000 - 0.00% -
447.dealII (avg) -74.49% 98.000 25.000 - 0.00% -
464.h264ref (max) -73.33% 60.000 16.000 - 0.00% -
447.dealII (max) -72.65% 223.000 61.000 - 0.00% -
470.lbm -71.88% 441.000 124.000 - 0.00% -
470.lbm (avg) -71.88% 441.000 124.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) (percentage) -71.43% 0.980 0.280 - 0.00% -
429.mcf (unexplained diff vs previous) -68.18% 22.000 7.000 - 0.00% -
445.gobmk (max) -67.80% 413.000 133.000 - 0.00% -
456.hmmer (unexplained diff vs previous) (percentage) -66.67% 0.390 0.130 - 0.00% -
445.gobmk (unexplained diff vs previous) (percentage) -55.56% 0.360 0.160 - 0.00% -
401.bzip2 (max) (percentage) -48.41% 3.780 1.950 - 0.00% -
473.astar (max) -48.05% 822.000 427.000 - 0.00% -
453.povray (max) -48.00% 25.000 13.000 - 0.00% -
444.namd (max) (percentage) -41.53% 8.090 4.730 - 0.00% -
445.gobmk -40.54% 74.000 44.000 - 0.00% -
445.gobmk (avg) -40.54% 74.000 44.000 - 0.00% -
464.h264ref (max) (percentage) -35.71% 0.420 0.270 - 0.00% -
445.gobmk (max) (percentage) -26.81% 3.730 2.730 - 0.00% -
462.libquantum (max) (percentage) -25.08% 6.380 4.780 - 0.00% -
458.sjeng -21.32% 136.000 107.000 - 0.00% -
458.sjeng (avg) -21.32% 136.000 107.000 - 0.00% -
470.lbm (max) (percentage) -17.66% 5.720 4.710 - 0.00% -
433.milc (unexplained diff vs previous) -15.38% 13.000 11.000 - 0.00% -
403.gcc (max) (percentage) -11.59% 2.330 2.060 - 0.00% -
433.milc (avg) (percentage) -6.98% 0.860 0.800 - 0.00% -
470.lbm (avg) (percentage) -6.01% 3.330 3.130 - 0.00% -
471.omnetpp -5.99% 167.000 157.000 - 0.00% -
471.omnetpp (avg) -5.99% 167.000 157.000 - 0.00% -
473.astar -1.79% 112.000 110.000 - 0.00% -
473.astar (avg) -1.79% 112.000 110.000 - 0.00% -

Added Tests - sample variability
403.gcc
403.gcc (avg)
464.h264ref (unexplained diff vs previous)
464.h264ref (unexplained diff vs previous) (percentage)
483.xalancbmk (unexplained diff vs previous)
483.xalancbmk (unexplained diff vs previous) (percentage)

Report Time: 0.34s
Execution (seconds)
Name Current %
401.bzip2 3039.000 -
403.gcc 57.000 -
429.mcf 1171.000 -
433.milc 1162.000 -
444.namd 813.000 -
445.gobmk 4898.000 -
447.dealII 2020.000 -
450.soplex 534.000 -
453.povray 503.000 -
456.hmmer 3814.000 -
458.sjeng 8295.000 -
462.libquantum 192.000 -
464.h264ref 6006.000 -
470.lbm 3978.000 -
471.omnetpp 2859.000 -
473.astar 4595.000 -
482.sphinx3 1006.000 -
483.xalancbmk 3476.000 -
Geometric Mean 1570.938 -
Code Size (bytes)
Name Current %
401.bzip2 34057.000 -
403.gcc 1667723.000 -
429.mcf 7051.000 -
433.milc 46453.000 -
444.namd 124598.000 -
445.gobmk 1259121.000 -
447.dealII 197298.000 -
450.soplex 149884.000 -
453.povray 514139.000 -
456.hmmer 80349.000 -
458.sjeng 79714.000 -
462.libquantum 12290.000 -
464.h264ref 301967.000 -
470.lbm 9240.000 -
471.omnetpp 321429.000 -
473.astar 15375.000 -
482.sphinx3 87516.000 -
483.xalancbmk 1612616.000 -
Geometric Mean 111235.037 -
sample variability
Name Current %
401.bzip2 26.000 -
401.bzip2 (avg) 26.000 -
401.bzip2 (avg) (percentage) 0.870 -
401.bzip2 (max) 59.000 -
401.bzip2 (max) (percentage) 1.950 -
401.bzip2 (unexplained diff vs previous) 39.000 -
401.bzip2 (unexplained diff vs previous) (percentage) 1.280 -
403.gcc 0.000 -
403.gcc (avg) 0.000 -
403.gcc (avg) (percentage) 0.750 -
403.gcc (max) 1.000 -
403.gcc (max) (percentage) 2.060 -
429.mcf 49.000 -
429.mcf (avg) 49.000 -
429.mcf (avg) (percentage) 4.190 -
429.mcf (max) 7.000 -
429.mcf (max) (percentage) 0.600 -
429.mcf (unexplained diff vs previous) 7.000 -
429.mcf (unexplained diff vs previous) (percentage) 0.600 -
433.milc 9.000 -
433.milc (avg) 9.000 -
433.milc (avg) (percentage) 0.800 -
433.milc (max) 15.000 -
433.milc (max) (percentage) 1.320 -
433.milc (unexplained diff vs previous) 11.000 -
433.milc (unexplained diff vs previous) (percentage) 0.950 -
444.namd 21.000 -
444.namd (avg) 21.000 -
444.namd (avg) (percentage) 2.670 -
444.namd (max) 38.000 -
444.namd (max) (percentage) 4.730 -
445.gobmk 44.000 -
445.gobmk (avg) 44.000 -
445.gobmk (avg) (percentage) 0.910 -
445.gobmk (max) 133.000 -
445.gobmk (max) (percentage) 2.730 -
445.gobmk (unexplained diff vs previous) 8.000 -
445.gobmk (unexplained diff vs previous) (percentage) 0.160 -
447.dealII 25.000 -
447.dealII (avg) 25.000 -
447.dealII (avg) (percentage) 1.250 -
447.dealII (max) 61.000 -
447.dealII (max) (percentage) 3.050 -
450.soplex 3.000 -
450.soplex (avg) 3.000 -
450.soplex (avg) (percentage) 0.700 -
450.soplex (max) 9.000 -
450.soplex (max) (percentage) 1.730 -
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.850 -
453.povray (max) 13.000 -
453.povray (max) (percentage) 2.630 -
456.hmmer 195.000 -
456.hmmer (avg) 195.000 -
456.hmmer (avg) (percentage) 5.130 -
456.hmmer (max) 13.000 -
456.hmmer (max) (percentage) 0.350 -
456.hmmer (unexplained diff vs previous) 5.000 -
456.hmmer (unexplained diff vs previous) (percentage) 0.130 -
458.sjeng 107.000 -
458.sjeng (avg) 107.000 -
458.sjeng (avg) (percentage) 1.300 -
458.sjeng (max) 104.000 -
458.sjeng (max) (percentage) 1.260 -
462.libquantum 4.000 -
462.libquantum (avg) 4.000 -
462.libquantum (avg) (percentage) 2.160 -
462.libquantum (max) 9.000 -
462.libquantum (max) (percentage) 4.780 -
462.libquantum (unexplained diff vs previous) 8.000 -
462.libquantum (unexplained diff vs previous) (percentage) 4.170 -
464.h264ref 67.000 -
464.h264ref (avg) 67.000 -
464.h264ref (avg) (percentage) 1.130 -
464.h264ref (max) 16.000 -
464.h264ref (max) (percentage) 0.270 -
464.h264ref (unexplained diff vs previous) 10.000 -
464.h264ref (unexplained diff vs previous) (percentage) 0.170 -
470.lbm 124.000 -
470.lbm (avg) 124.000 -
470.lbm (avg) (percentage) 3.130 -
470.lbm (max) 187.000 -
470.lbm (max) (percentage) 4.710 -
470.lbm (unexplained diff vs previous) 5.000 -
470.lbm (unexplained diff vs previous) (percentage) 0.130 -
471.omnetpp 157.000 -
471.omnetpp (avg) 157.000 -
471.omnetpp (avg) (percentage) 5.510 -
471.omnetpp (max) 20.000 -
471.omnetpp (max) (percentage) 0.710 -
471.omnetpp (unexplained diff vs previous) 8.000 -
471.omnetpp (unexplained diff vs previous) (percentage) 0.280 -
473.astar 110.000 -
473.astar (avg) 110.000 -
473.astar (avg) (percentage) 2.410 -
473.astar (max) 427.000 -
473.astar (max) (percentage) 9.300 -
482.sphinx3 22.000 -
482.sphinx3 (avg) 22.000 -
482.sphinx3 (avg) (percentage) 2.220 -
482.sphinx3 (max) 52.000 -
482.sphinx3 (max) (percentage) 5.250 -
483.xalancbmk 260.000 -
483.xalancbmk (avg) 260.000 -
483.xalancbmk (avg) (percentage) 7.480 -
483.xalancbmk (max) 1624.000 -
483.xalancbmk (max) (percentage) 46.730 -
483.xalancbmk (unexplained diff vs previous) 4.000 -
483.xalancbmk (unexplained diff vs previous) (percentage) 0.120 -
Geometric Mean 6.356 -18.52%