Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/a292e7edf8b2fc51d3e86a96ff5dff45d16bd264 (+ 1 commits)
regression LLVM-929
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-aarch64-master-Oz_LTO-build/77/
View Options
Report
Filter

llvm-aarch64-master-Oz_LTO test results

Run Order Start Time Duration
Current 0077 (llvmorg-18-init-06129-ga292e7edf8b2) 2023-08-31T05:50:24 0:00:00
Previous 0076 (llvmorg-18-init-06128-g6d1c6ecbdc4e) 2023-08-11T22:27:47 0:00:00
Baseline 0001 (llvmorg-16-init-17110-gb1256f2345f6) 2022-12-17T20:51:03 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 9 32
Performance Improvements 38 87
Added Tests 8 8
Unchanged Tests 677 605
Total Tests 732

Run-Over-Run Changes Detail

Performance Regressions - Code Size Δ Previous Current σ Δ (B) σ (B)
470.lbm 3.08% 9957.000 10264.000 - -0.48% -
462.libquantum 1.24% 16614.000 16820.000 - -0.99% -

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
450.soplex 2.09% 526.000 537.000 - -96.16% -
447.dealII 1.31% 1834.000 1858.000 - -79.59% -
444.namd 1.04% 673.000 680.000 - -92.75% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
403.gcc (max) (percentage) 2390.91% 0.330 8.220 - 1955.00% -
403.gcc (avg) (percentage) 829.03% 0.310 2.880 - 829.03% -
447.dealII 8.33% 12.000 13.000 - -84.15% -
447.dealII (avg) 8.33% 12.000 13.000 - -84.15% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -8.22% 73.000 67.000 - -99.34% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
401.bzip2 (max) (percentage) -48.39% 0.310 0.160 - 220.00% -
401.bzip2 (max) -42.86% 7.000 4.000 - -33.33% -
429.mcf (max) -38.46% 13.000 8.000 - -88.24% -
429.mcf (max) (percentage) -36.99% 0.730 0.460 - 27.78% -
400.perlbench -20.00% 5.000 4.000 - -92.86% -
400.perlbench (avg) -20.00% 5.000 4.000 - -92.86% -
401.bzip2 -8.33% 24.000 22.000 - 69.23% -
401.bzip2 (avg) -8.33% 24.000 22.000 - 69.23% -
482.sphinx3 (max) (percentage) -7.35% 0.680 0.630 - -8.70% -
433.milc (max) (percentage) -7.27% 0.550 0.510 - -72.43% -
471.omnetpp (max) (percentage) -6.55% 1.680 1.570 - 57.00% -
470.lbm (max) (percentage) -6.45% 0.310 0.290 - 383.33% -
450.soplex (max) (percentage) -6.25% 1.280 1.200 - 9.09% -
458.sjeng (avg) (percentage) -6.25% 0.320 0.300 - 30.43% -
471.omnetpp (max) -6.12% 49.000 46.000 - -61.34% -
401.bzip2 (avg) (percentage) -5.32% 0.940 0.890 - 790.00% -
464.h264ref -5.26% 19.000 18.000 - -59.09% -
464.h264ref (avg) -5.26% 19.000 18.000 - -59.09% -
462.libquantum (max) (percentage) -4.50% 1.110 1.060 - 2550.00% -
458.sjeng -4.35% 23.000 22.000 - -37.14% -
458.sjeng (avg) -4.35% 23.000 22.000 - -37.14% -
429.mcf (avg) (percentage) -4.17% 2.640 2.530 - 583.78% -
429.mcf -4.08% 49.000 47.000 - -31.88% -
429.mcf (avg) -4.08% 49.000 47.000 - -31.88% -
473.astar (max) -4.00% 25.000 24.000 - 140.00% -
473.astar (max) (percentage) -3.92% 0.510 0.490 - 512.50% -
456.hmmer (max) -3.45% 29.000 28.000 - -58.82% -
458.sjeng (max) -3.45% 29.000 28.000 - -20.00% -
464.h264ref (max) -3.33% 30.000 29.000 - -9.38% -
456.hmmer (max) (percentage) -2.97% 1.010 0.980 - 30.67% -
447.dealII (max) (percentage) -2.94% 0.680 0.660 - -52.52% -
458.sjeng (max) (percentage) -2.56% 0.390 0.380 - 65.22% -
444.namd (max) (percentage) -1.77% 1.130 1.110 - -78.57% -
445.gobmk (max) (percentage) -1.55% 1.290 1.270 - 111.67% -
464.h264ref (max) (percentage) -1.54% 0.650 0.640 - 137.04% -
445.gobmk (max) -1.45% 69.000 68.000 - -2.86% -
453.povray (max) (percentage) -1.19% 0.840 0.830 - -21.70% -

Added Tests - sample variability
401.bzip2 (unexplained diff vs previous)
401.bzip2 (unexplained diff vs previous) (percentage)
403.gcc (unexplained diff vs previous)
403.gcc (unexplained diff vs previous) (percentage)
429.mcf (unexplained diff vs previous)
429.mcf (unexplained diff vs previous) (percentage)
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 2.33% 224407.000 229646.000 - 0.01% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
462.libquantum (max) (percentage) 2550.00% 0.040 1.060 - -4.50% -
403.gcc (max) (percentage) 1955.00% 0.400 8.220 - 2390.91% -
473.astar (avg) (percentage) 1371.43% 0.070 1.030 - 0.00% -
462.libquantum (avg) (percentage) 972.73% 0.110 1.180 - 0.00% -
403.gcc (avg) (percentage) 829.03% 0.310 2.880 - 829.03% -
401.bzip2 (avg) (percentage) 790.00% 0.100 0.890 - -5.32% -
429.mcf (avg) (percentage) 583.78% 0.370 2.530 - -4.17% -
473.astar (max) (percentage) 512.50% 0.080 0.490 - -3.92% -
473.astar 466.67% 9.000 51.000 - 0.00% -
473.astar (avg) 466.67% 9.000 51.000 - 0.00% -
450.soplex (avg) (percentage) 464.81% 0.540 3.050 - 0.00% -
470.lbm (max) (percentage) 383.33% 0.060 0.290 - -6.45% -
470.lbm (avg) (percentage) 250.00% 0.580 2.030 - 0.00% -
401.bzip2 (max) (percentage) 220.00% 0.050 0.160 - -48.39% -
473.astar (max) 140.00% 10.000 24.000 - -4.00% -
464.h264ref (max) (percentage) 137.04% 0.270 0.640 - -1.54% -
445.gobmk (max) (percentage) 111.67% 0.600 1.270 - -1.55% -
482.sphinx3 (avg) (percentage) 110.99% 0.910 1.920 - 0.00% -
471.omnetpp (avg) (percentage) 76.67% 0.600 1.060 - 0.00% -
401.bzip2 69.23% 13.000 22.000 - -8.33% -
401.bzip2 (avg) 69.23% 13.000 22.000 - -8.33% -
458.sjeng (max) (percentage) 65.22% 0.230 0.380 - -2.56% -
471.omnetpp (max) (percentage) 57.00% 1.000 1.570 - -6.55% -
470.lbm (max) 50.00% 6.000 9.000 - 0.00% -
456.hmmer (max) (percentage) 30.67% 0.750 0.980 - -2.97% -
458.sjeng (avg) (percentage) 30.43% 0.230 0.300 - -6.25% -
429.mcf (max) (percentage) 27.78% 0.360 0.460 - -36.99% -
445.gobmk (avg) (percentage) 12.50% 0.320 0.360 - 0.00% -
464.h264ref (avg) (percentage) 10.81% 0.370 0.410 - 0.00% -
450.soplex (max) (percentage) 9.09% 1.100 1.200 - -6.25% -
453.povray (avg) (percentage) 2.74% 0.730 0.750 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
444.namd -4.50% 143421.000 136968.000 - 0.19% -
453.povray -1.60% 691522.000 680457.000 - 0.69% -
401.bzip2 -1.23% 44357.000 43812.000 - 0.00% -
464.h264ref -1.15% 399974.000 395389.000 - 0.09% -
429.mcf -1.09% 9249.000 9148.000 - 0.00% -
433.milc -1.05% 62863.000 62202.000 - -0.01% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.34% 10150.000 67.000 - -8.22% -
462.libquantum -98.58% 9659.000 137.000 - 0.74% -
482.sphinx3 -96.56% 23779.000 819.000 - 0.12% -
450.soplex -96.16% 13988.000 537.000 - 2.09% -
453.povray -92.99% 7301.000 512.000 - -0.19% -
444.namd -92.75% 9377.000 680.000 - 1.04% -
433.milc -91.43% 14590.000 1251.000 - -0.16% -
429.mcf -90.10% 18905.000 1872.000 - -0.43% -
400.perlbench -83.44% 11085.000 1836.000 - -0.86% -
401.bzip2 -81.65% 13962.000 2562.000 - 0.12% -
447.dealII -79.59% 9102.000 1858.000 - 1.31% -
471.omnetpp -74.93% 11917.000 2988.000 - 0.81% -
470.lbm -71.29% 11046.000 3171.000 - -0.63% -
456.hmmer -68.12% 9122.000 2908.000 - -0.75% -
473.astar -62.45% 13301.000 4994.000 - -0.42% -
464.h264ref -62.02% 12142.000 4611.000 - -0.52% -
445.gobmk -53.94% 11668.000 5374.000 - -0.46% -
458.sjeng -51.19% 15282.000 7459.000 - 0.05% -
483.xalancbmk -47.02% 9633.000 5104.000 - 0.71% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -98.56% 485.000 7.000 - 0.00% -
433.milc (max) -97.77% 269.000 6.000 - 0.00% -
444.namd -97.18% 213.000 6.000 - 0.00% -
444.namd (avg) -97.18% 213.000 6.000 - 0.00% -
482.sphinx3 (max) -96.95% 164.000 5.000 - 0.00% -
403.gcc -96.77% 31.000 1.000 - 0.00% -
403.gcc (avg) -96.77% 31.000 1.000 - 0.00% -
450.soplex (max) -96.08% 153.000 6.000 - 0.00% -
453.povray (max) -94.81% 77.000 4.000 - 0.00% -
453.povray -94.34% 53.000 3.000 - 0.00% -
453.povray (avg) -94.34% 53.000 3.000 - 0.00% -
482.sphinx3 -93.06% 216.000 15.000 - 0.00% -
482.sphinx3 (avg) -93.06% 216.000 15.000 - 0.00% -
400.perlbench -92.86% 56.000 4.000 - -20.00% -
400.perlbench (avg) -92.86% 56.000 4.000 - -20.00% -
433.milc -92.56% 121.000 9.000 - 0.00% -
433.milc (avg) -92.56% 121.000 9.000 - 0.00% -
447.dealII (max) -90.48% 126.000 12.000 - 0.00% -
462.libquantum -90.00% 10.000 1.000 - 0.00% -
462.libquantum (avg) -90.00% 10.000 1.000 - 0.00% -
400.perlbench (max) -89.61% 77.000 8.000 - 0.00% -
429.mcf (max) -88.24% 68.000 8.000 - -38.46% -
403.gcc (max) -87.50% 40.000 5.000 - 0.00% -
447.dealII -84.15% 82.000 13.000 - 8.33% -
447.dealII (avg) -84.15% 82.000 13.000 - 8.33% -
456.hmmer -78.75% 80.000 17.000 - 0.00% -
456.hmmer (avg) -78.75% 80.000 17.000 - 0.00% -
450.soplex -78.67% 75.000 16.000 - 0.00% -
450.soplex (avg) -78.67% 75.000 16.000 - 0.00% -
444.namd (max) (percentage) -78.57% 5.180 1.110 - -1.77% -
433.milc (max) (percentage) -72.43% 1.850 0.510 - -7.27% -
462.libquantum (max) -66.67% 3.000 1.000 - 0.00% -
471.omnetpp (max) -61.34% 119.000 46.000 - -6.12% -
483.xalancbmk (max) -61.22% 49.000 19.000 - 0.00% -
444.namd (avg) (percentage) -60.09% 2.280 0.910 - 0.00% -
464.h264ref -59.09% 44.000 18.000 - -5.26% -
464.h264ref (avg) -59.09% 44.000 18.000 - -5.26% -
456.hmmer (max) -58.82% 68.000 28.000 - -3.45% -
471.omnetpp -56.34% 71.000 31.000 - 0.00% -
471.omnetpp (avg) -56.34% 71.000 31.000 - 0.00% -
447.dealII (max) (percentage) -52.52% 1.390 0.660 - -2.94% -
483.xalancbmk -48.78% 41.000 21.000 - 0.00% -
483.xalancbmk (avg) -48.78% 41.000 21.000 - 0.00% -
445.gobmk -48.65% 37.000 19.000 - 0.00% -
445.gobmk (avg) -48.65% 37.000 19.000 - 0.00% -
400.perlbench (avg) (percentage) -47.06% 0.510 0.270 - 0.00% -
458.sjeng -37.14% 35.000 22.000 - -4.35% -
458.sjeng (avg) -37.14% 35.000 22.000 - -4.35% -
401.bzip2 (max) -33.33% 6.000 4.000 - -42.86% -
429.mcf -31.88% 69.000 47.000 - -4.08% -
429.mcf (avg) -31.88% 69.000 47.000 - -4.08% -
400.perlbench (max) (percentage) -31.43% 0.700 0.480 - 0.00% -
456.hmmer (avg) (percentage) -30.68% 0.880 0.610 - 0.00% -
483.xalancbmk (max) (percentage) -23.53% 0.510 0.390 - 0.00% -
447.dealII (avg) (percentage) -23.08% 0.910 0.700 - 0.00% -
453.povray (max) (percentage) -21.70% 1.060 0.830 - -1.19% -
458.sjeng (max) -20.00% 35.000 28.000 - -3.45% -
464.h264ref (max) -9.38% 32.000 29.000 - -3.33% -
482.sphinx3 (max) (percentage) -8.70% 0.690 0.630 - -7.35% -
433.milc (avg) (percentage) -7.23% 0.830 0.770 - 0.00% -
445.gobmk (max) -2.86% 70.000 68.000 - -1.45% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 52.63% 19.000 29.000 - 0.00% -

Added Tests - sample variability
401.bzip2 (unexplained diff vs previous)
401.bzip2 (unexplained diff vs previous) (percentage)
403.gcc (unexplained diff vs previous)
403.gcc (unexplained diff vs previous) (percentage)
429.mcf (unexplained diff vs previous)
429.mcf (unexplained diff vs previous) (percentage)
458.sjeng (unexplained diff vs previous)
458.sjeng (unexplained diff vs previous) (percentage)

Report Time: 0.35s
Execution (seconds)
Name Current %
400.perlbench 1836.000 -
401.bzip2 2562.000 -
403.gcc 67.000 -8.22%
429.mcf 1872.000 -
433.milc 1251.000 -
444.namd 680.000 1.04%
445.gobmk 5374.000 -
447.dealII 1858.000 1.31%
450.soplex 537.000 2.09%
453.povray 512.000 -
456.hmmer 2908.000 -
458.sjeng 7459.000 -
462.libquantum 137.000 -
464.h264ref 4611.000 -
470.lbm 3171.000 -
471.omnetpp 2988.000 -
473.astar 4994.000 -
482.sphinx3 819.000 -
483.xalancbmk 5104.000 -
Geometric Mean 1541.737 -
Code Size (bytes)
Name Current %
400.perlbench 894294.000 -
401.bzip2 43812.000 -
403.gcc 2479392.000 -
429.mcf 9148.000 -
433.milc 62202.000 -
444.namd 136968.000 -
445.gobmk 1849734.000 -
447.dealII 328003.000 -
450.soplex 229646.000 -
453.povray 680457.000 -
456.hmmer 112892.000 -
458.sjeng 106375.000 -
462.libquantum 16820.000 1.24%
464.h264ref 395389.000 -
470.lbm 10264.000 3.08%
471.omnetpp 491217.000 -
473.astar 23552.000 -
482.sphinx3 119661.000 -
483.xalancbmk 2522698.000 -
Geometric Mean 168651.295 -
sample variability
Name Current %
400.perlbench 4.000 -20.00%
400.perlbench (avg) 4.000 -20.00%
400.perlbench (avg) (percentage) 0.270 -
400.perlbench (max) 8.000 -
400.perlbench (max) (percentage) 0.480 -
401.bzip2 22.000 -8.33%
401.bzip2 (avg) 22.000 -8.33%
401.bzip2 (avg) (percentage) 0.890 -5.32%
401.bzip2 (max) 4.000 -42.86%
401.bzip2 (max) (percentage) 0.160 -48.39%
401.bzip2 (unexplained diff vs previous) 3.000 -
401.bzip2 (unexplained diff vs previous) (percentage) 0.120 -
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 2.880 829.03%
403.gcc (max) 5.000 -
403.gcc (max) (percentage) 8.220 2390.91%
403.gcc (unexplained diff vs previous) 6.000 -
403.gcc (unexplained diff vs previous) (percentage) 8.960 -
429.mcf 47.000 -4.08%
429.mcf (avg) 47.000 -4.08%
429.mcf (avg) (percentage) 2.530 -4.17%
429.mcf (max) 8.000 -38.46%
429.mcf (max) (percentage) 0.460 -36.99%
429.mcf (unexplained diff vs previous) 8.000 -
429.mcf (unexplained diff vs previous) (percentage) 0.430 -
433.milc 9.000 -
433.milc (avg) 9.000 -
433.milc (avg) (percentage) 0.770 -
433.milc (max) 6.000 -
433.milc (max) (percentage) 0.510 -7.27%
444.namd 6.000 -
444.namd (avg) 6.000 -
444.namd (avg) (percentage) 0.910 -
444.namd (max) 7.000 -
444.namd (max) (percentage) 1.110 -1.77%
445.gobmk 19.000 -
445.gobmk (avg) 19.000 -
445.gobmk (avg) (percentage) 0.360 -
445.gobmk (max) 68.000 -1.45%
445.gobmk (max) (percentage) 1.270 -1.55%
447.dealII 13.000 8.33%
447.dealII (avg) 13.000 8.33%
447.dealII (avg) (percentage) 0.700 -
447.dealII (max) 12.000 -
447.dealII (max) (percentage) 0.660 -2.94%
450.soplex 16.000 -
450.soplex (avg) 16.000 -
450.soplex (avg) (percentage) 3.050 -
450.soplex (max) 6.000 -
450.soplex (max) (percentage) 1.200 -6.25%
453.povray 3.000 -
453.povray (avg) 3.000 -
453.povray (avg) (percentage) 0.750 -
453.povray (max) 4.000 -
453.povray (max) (percentage) 0.830 -1.19%
456.hmmer 17.000 -
456.hmmer (avg) 17.000 -
456.hmmer (avg) (percentage) 0.610 -
456.hmmer (max) 28.000 -3.45%
456.hmmer (max) (percentage) 0.980 -2.97%
458.sjeng 22.000 -4.35%
458.sjeng (avg) 22.000 -4.35%
458.sjeng (avg) (percentage) 0.300 -6.25%
458.sjeng (max) 28.000 -3.45%
458.sjeng (max) (percentage) 0.380 -2.56%
458.sjeng (unexplained diff vs previous) 4.000 -
458.sjeng (unexplained diff vs previous) (percentage) 0.050 -
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.180 -
462.libquantum (max) 1.000 -
462.libquantum (max) (percentage) 1.060 -4.50%
464.h264ref 18.000 -5.26%
464.h264ref (avg) 18.000 -5.26%
464.h264ref (avg) (percentage) 0.410 -
464.h264ref (max) 29.000 -3.33%
464.h264ref (max) (percentage) 0.640 -1.54%
470.lbm 64.000 -
470.lbm (avg) 64.000 -
470.lbm (avg) (percentage) 2.030 -
470.lbm (max) 9.000 -
470.lbm (max) (percentage) 0.290 -6.45%
471.omnetpp 31.000 -
471.omnetpp (avg) 31.000 -
471.omnetpp (avg) (percentage) 1.060 -
471.omnetpp (max) 46.000 -6.12%
471.omnetpp (max) (percentage) 1.570 -6.55%
473.astar 51.000 -
473.astar (avg) 51.000 -
473.astar (avg) (percentage) 1.030 -
473.astar (max) 24.000 -4.00%
473.astar (max) (percentage) 0.490 -3.92%
482.sphinx3 15.000 -
482.sphinx3 (avg) 15.000 -
482.sphinx3 (avg) (percentage) 1.920 -
482.sphinx3 (max) 5.000 -
482.sphinx3 (max) (percentage) 0.630 -7.35%
483.xalancbmk 21.000 -
483.xalancbmk (avg) 21.000 -
483.xalancbmk (avg) (percentage) 0.430 -
483.xalancbmk (max) 19.000 -
483.xalancbmk (max) (percentage) 0.390 -
Geometric Mean 3.684 27.35%