Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/033b491c756f58bceaa26f1c96ec645c8d3e5e45 (+ 9705 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-aarch64-master-Oz_LTO-build/121/
View Options
Report
Filter

llvm-aarch64-master-Oz_LTO test results

Run Order Start Time Duration
Current 0121 (llvmorg-19-init-00281-g033b491c756f) 2023-11-30T17:51:50 0:00:00
Previous 0087 (llvmorg-18-init-08936-g416884544e02) 2023-10-05T13:10:32 0:00:00
Baseline 0001 (llvmorg-16-init-17110-gb1256f2345f6) 2022-12-17T20:51:03 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 30 31
Performance Improvements 28 95
Added Tests 10 10
Unchanged Tests 772 704
Total Tests 840

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
429.mcf 27.45% 1581.000 2015.000 - -89.34% -
450.soplex 13.36% 479.000 543.000 - -96.12% -
403.gcc 10.61% 66.000 73.000 - -99.28% -
401.bzip2 7.59% 2517.000 2708.000 - -80.60% -
473.astar 5.64% 4892.000 5168.000 - -61.15% -
482.sphinx3 3.75% 799.000 829.000 - -96.51% -
470.lbm 3.60% 3138.000 3251.000 - -70.57% -
447.dealII 3.27% 1833.000 1893.000 - -79.20% -
400.perlbench 2.50% 1878.000 1925.000 - -82.63% -
433.milc 2.05% 1220.000 1245.000 - -91.47% -
453.povray 1.93% 518.000 528.000 - -92.77% -
483.xalancbmk 1.63% 5221.000 5306.000 - -44.92% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
403.gcc 100.00% 1.000 2.000 - -93.55% -
403.gcc (avg) 100.00% 1.000 2.000 - -93.55% -
429.mcf (max) 28.57% 7.000 9.000 - -86.76% -
429.mcf 26.32% 38.000 48.000 - -30.43% -
429.mcf (avg) 26.32% 38.000 48.000 - -30.43% -
450.soplex 15.38% 13.000 15.000 - -80.00% -
450.soplex (avg) 15.38% 13.000 15.000 - -80.00% -
401.bzip2 9.52% 21.000 23.000 - 76.92% -
401.bzip2 (avg) 9.52% 21.000 23.000 - 76.92% -
447.dealII 8.33% 12.000 13.000 - -84.15% -
447.dealII (avg) 8.33% 12.000 13.000 - -84.15% -
482.sphinx3 7.14% 14.000 15.000 - -93.06% -
482.sphinx3 (avg) 7.14% 14.000 15.000 - -93.06% -
473.astar 6.25% 48.000 51.000 - 466.67% -
473.astar (avg) 6.25% 48.000 51.000 - 466.67% -
470.lbm 5.00% 60.000 63.000 - -1.56% -
470.lbm (avg) 5.00% 60.000 63.000 - -1.56% -
473.astar (max) 3.57% 28.000 29.000 - 190.00% -

Performance Improvements - Code Size Δ Previous Current σ Δ (B) σ (B)
464.h264ref -1.61% 394901.000 388537.000 - -2.86% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
453.povray (max) -33.33% 3.000 2.000 - -97.40% -
444.namd (max) -16.67% 6.000 5.000 - -98.97% -
470.lbm (max) -11.11% 9.000 8.000 - 33.33% -
433.milc (max) -8.33% 12.000 11.000 - -95.91% -
401.bzip2 (max) (percentage) -7.50% 0.400 0.370 - 640.00% -
433.milc (max) (percentage) -7.07% 0.990 0.920 - -50.27% -
470.lbm (max) (percentage) -6.90% 0.290 0.270 - 350.00% -
464.h264ref (max) -5.56% 18.000 17.000 - -46.88% -
453.povray (max) (percentage) -5.17% 0.580 0.550 - -48.11% -
471.omnetpp (max) -5.00% 80.000 76.000 - -36.13% -
471.omnetpp (max) (percentage) -4.85% 2.680 2.550 - 155.00% -
456.hmmer (max) -4.35% 23.000 22.000 - -67.65% -
447.dealII (max) (percentage) -4.29% 0.700 0.670 - -51.80% -
456.hmmer (max) (percentage) -3.70% 0.810 0.780 - 4.00% -
473.astar (max) (percentage) -3.39% 0.590 0.570 - 612.50% -
444.namd (max) (percentage) -3.33% 0.900 0.870 - -83.20% -
462.libquantum (max) (percentage) -3.03% 0.990 0.960 - 2300.00% -
450.soplex (max) (percentage) -2.73% 1.100 1.070 - -2.73% -
483.xalancbmk (max) (percentage) -2.56% 0.390 0.380 - -25.49% -
464.h264ref (max) (percentage) -2.50% 0.400 0.390 - 44.44% -
400.perlbench (max) (percentage) -2.08% 0.480 0.470 - -32.86% -
482.sphinx3 (max) (percentage) -2.04% 0.490 0.480 - -30.43% -
445.gobmk (max) -1.54% 65.000 64.000 - -8.57% -
403.gcc (max) (percentage) -1.12% 8.030 7.940 - 1885.00% -

Performance Improvements - Score Δ Previous Current σ Δ (B) σ (B)
nb_compile_successful 26.32% 19.000 24.000 - 26.32% -
nb_execution_successful 26.32% 19.000 24.000 - 26.32% -
nb_total_tests 17.24% 29.000 34.000 - 78.95% -

Added Tests - Code Size
410.bwaves
434.zeusmp
437.leslie3d
459.GemsFDTD
465.tonto

Added Tests - Execution
410.bwaves
434.zeusmp
437.leslie3d
459.GemsFDTD
465.tonto

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.71% 224407.000 228234.000 - -0.46% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
462.libquantum (max) (percentage) 2300.00% 0.040 0.960 - -3.03% -
403.gcc (max) (percentage) 1885.00% 0.400 7.940 - -1.12% -
473.astar (avg) (percentage) 1328.57% 0.070 1.000 - 0.00% -
462.libquantum (avg) (percentage) 918.18% 0.110 1.120 - 0.00% -
403.gcc (avg) (percentage) 829.03% 0.310 2.880 - 0.00% -
401.bzip2 (avg) (percentage) 770.00% 0.100 0.870 - 0.00% -
401.bzip2 (max) (percentage) 640.00% 0.050 0.370 - -7.50% -
473.astar (max) (percentage) 612.50% 0.080 0.570 - -3.39% -
429.mcf (avg) (percentage) 551.35% 0.370 2.410 - 0.00% -
473.astar 466.67% 9.000 51.000 - 6.25% -
473.astar (avg) 466.67% 9.000 51.000 - 6.25% -
450.soplex (avg) (percentage) 431.48% 0.540 2.870 - 0.00% -
470.lbm (max) (percentage) 350.00% 0.060 0.270 - -6.90% -
470.lbm (avg) (percentage) 234.48% 0.580 1.940 - 0.00% -
473.astar (max) 190.00% 10.000 29.000 - 3.57% -
471.omnetpp (max) (percentage) 155.00% 1.000 2.550 - -4.85% -
445.gobmk (max) (percentage) 106.67% 0.600 1.240 - -0.80% -
482.sphinx3 (avg) (percentage) 100.00% 0.910 1.820 - 0.00% -
471.omnetpp (avg) (percentage) 95.00% 0.600 1.170 - 0.00% -
401.bzip2 76.92% 13.000 23.000 - 9.52% -
401.bzip2 (avg) 76.92% 13.000 23.000 - 9.52% -
401.bzip2 (max) 66.67% 6.000 10.000 - 0.00% -
458.sjeng (max) (percentage) 52.17% 0.230 0.350 - 0.00% -
464.h264ref (max) (percentage) 44.44% 0.270 0.390 - -2.50% -
470.lbm (max) 33.33% 6.000 8.000 - -11.11% -
458.sjeng (avg) (percentage) 30.43% 0.230 0.300 - 0.00% -
429.mcf (max) (percentage) 27.78% 0.360 0.460 - 0.00% -
445.gobmk (avg) (percentage) 12.50% 0.320 0.360 - 0.00% -
464.h264ref (avg) (percentage) 5.41% 0.370 0.390 - 0.00% -
456.hmmer (max) (percentage) 4.00% 0.750 0.780 - -3.70% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
444.namd -5.18% 143421.000 135988.000 - -0.36% -
464.h264ref -2.86% 399974.000 388537.000 - -1.61% -
433.milc -1.89% 62863.000 61674.000 - -0.80% -
453.povray -1.87% 691522.000 678613.000 - -0.17% -
401.bzip2 -1.76% 44357.000 43576.000 - -0.50% -
429.mcf -1.61% 9249.000 9100.000 - -0.35% -
462.libquantum -1.51% 16989.000 16732.000 - -0.62% -
473.astar -1.47% 23789.000 23440.000 - -0.15% -
456.hmmer -1.30% 113613.000 112136.000 - -0.66% -
482.sphinx3 -1.07% 120254.000 118969.000 - -0.48% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.28% 10150.000 73.000 - 10.61% -
462.libquantum -98.61% 9659.000 134.000 - 0.75% -
482.sphinx3 -96.51% 23779.000 829.000 - 3.75% -
450.soplex -96.12% 13988.000 543.000 - 13.36% -
444.namd -92.84% 9377.000 671.000 - 0.00% -
453.povray -92.77% 7301.000 528.000 - 1.93% -
433.milc -91.47% 14590.000 1245.000 - 2.05% -
429.mcf -89.34% 18905.000 2015.000 - 27.45% -
400.perlbench -82.63% 11085.000 1925.000 - 2.50% -
401.bzip2 -80.60% 13962.000 2708.000 - 7.59% -
447.dealII -79.20% 9102.000 1893.000 - 3.27% -
471.omnetpp -74.96% 11917.000 2984.000 - -0.10% -
470.lbm -70.57% 11046.000 3251.000 - 3.60% -
456.hmmer -68.02% 9122.000 2917.000 - -0.17% -
464.h264ref -62.35% 12142.000 4572.000 - 0.11% -
473.astar -61.15% 13301.000 5168.000 - 5.64% -
445.gobmk -55.11% 11668.000 5238.000 - 0.21% -
458.sjeng -50.88% 15282.000 7507.000 - 0.89% -
483.xalancbmk -44.92% 9633.000 5306.000 - 1.63% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -98.97% 485.000 5.000 - -16.67% -
482.sphinx3 (max) -98.17% 164.000 3.000 - 0.00% -
444.namd -97.65% 213.000 5.000 - 0.00% -
444.namd (avg) -97.65% 213.000 5.000 - 0.00% -
453.povray (max) -97.40% 77.000 2.000 - -33.33% -
450.soplex (max) -96.73% 153.000 5.000 - 0.00% -
433.milc (max) -95.91% 269.000 11.000 - -8.33% -
453.povray -94.34% 53.000 3.000 - 0.00% -
453.povray (avg) -94.34% 53.000 3.000 - 0.00% -
403.gcc -93.55% 31.000 2.000 - 100.00% -
403.gcc (avg) -93.55% 31.000 2.000 - 100.00% -
482.sphinx3 -93.06% 216.000 15.000 - 7.14% -
482.sphinx3 (avg) -93.06% 216.000 15.000 - 7.14% -
433.milc -92.56% 121.000 9.000 - 0.00% -
433.milc (avg) -92.56% 121.000 9.000 - 0.00% -
400.perlbench -91.07% 56.000 5.000 - 0.00% -
400.perlbench (avg) -91.07% 56.000 5.000 - 0.00% -
447.dealII (max) -90.48% 126.000 12.000 - 0.00% -
462.libquantum -90.00% 10.000 1.000 - 0.00% -
462.libquantum (avg) -90.00% 10.000 1.000 - 0.00% -
400.perlbench (max) -88.31% 77.000 9.000 - 0.00% -
403.gcc (max) -87.50% 40.000 5.000 - 0.00% -
429.mcf (max) -86.76% 68.000 9.000 - 28.57% -
447.dealII -84.15% 82.000 13.000 - 8.33% -
447.dealII (avg) -84.15% 82.000 13.000 - 8.33% -
444.namd (max) (percentage) -83.20% 5.180 0.870 - -3.33% -
450.soplex -80.00% 75.000 15.000 - 15.38% -
450.soplex (avg) -80.00% 75.000 15.000 - 15.38% -
456.hmmer -80.00% 80.000 16.000 - 0.00% -
456.hmmer (avg) -80.00% 80.000 16.000 - 0.00% -
456.hmmer (max) -67.65% 68.000 22.000 - -4.35% -
462.libquantum (max) -66.67% 3.000 1.000 - 0.00% -
444.namd (avg) (percentage) -62.28% 2.280 0.860 - 0.00% -
464.h264ref -61.36% 44.000 17.000 - 0.00% -
464.h264ref (avg) -61.36% 44.000 17.000 - 0.00% -
483.xalancbmk (max) -59.18% 49.000 20.000 - 0.00% -
471.omnetpp -52.11% 71.000 34.000 - 0.00% -
471.omnetpp (avg) -52.11% 71.000 34.000 - 0.00% -
447.dealII (max) (percentage) -51.80% 1.390 0.670 - -4.29% -
445.gobmk -51.35% 37.000 18.000 - 0.00% -
445.gobmk (avg) -51.35% 37.000 18.000 - 0.00% -
433.milc (max) (percentage) -50.27% 1.850 0.920 - -7.07% -
453.povray (max) (percentage) -48.11% 1.060 0.550 - -5.17% -
400.perlbench (avg) (percentage) -47.06% 0.510 0.270 - 0.00% -
464.h264ref (max) -46.88% 32.000 17.000 - -5.56% -
483.xalancbmk -46.34% 41.000 22.000 - 0.00% -
483.xalancbmk (avg) -46.34% 41.000 22.000 - 0.00% -
458.sjeng -37.14% 35.000 22.000 - 0.00% -
458.sjeng (avg) -37.14% 35.000 22.000 - 0.00% -
471.omnetpp (max) -36.13% 119.000 76.000 - -5.00% -
456.hmmer (avg) (percentage) -35.23% 0.880 0.570 - 0.00% -
400.perlbench (max) (percentage) -32.86% 0.700 0.470 - -2.08% -
429.mcf -30.43% 69.000 48.000 - 26.32% -
429.mcf (avg) -30.43% 69.000 48.000 - 26.32% -
482.sphinx3 (max) (percentage) -30.43% 0.690 0.480 - -2.04% -
458.sjeng (max) -25.71% 35.000 26.000 - 0.00% -
483.xalancbmk (max) (percentage) -25.49% 0.510 0.380 - -2.56% -
447.dealII (avg) (percentage) -23.08% 0.910 0.700 - 0.00% -
445.gobmk (max) -8.57% 70.000 64.000 - -1.54% -
433.milc (avg) (percentage) -6.02% 0.830 0.780 - 0.00% -
450.soplex (max) (percentage) -2.73% 1.100 1.070 - -2.73% -
470.lbm -1.56% 64.000 63.000 - 5.00% -
470.lbm (avg) -1.56% 64.000 63.000 - 5.00% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 78.95% 19.000 34.000 - 17.24% -
nb_compile_successful 26.32% 19.000 24.000 - 26.32% -
nb_execution_successful 26.32% 19.000 24.000 - 26.32% -

Added Tests - Code Size
410.bwaves
434.zeusmp
437.leslie3d
459.GemsFDTD
465.tonto

Added Tests - Execution
410.bwaves
434.zeusmp
437.leslie3d
459.GemsFDTD
465.tonto

Report Time: 0.34s
Execution (seconds)
Name Current %
400.perlbench 1925.000 2.50%
401.bzip2 2708.000 7.59%
403.gcc 73.000 10.61%
410.bwaves 4749.000 -
429.mcf 2015.000 27.45%
433.milc 1245.000 2.05%
434.zeusmp 2048.000 -
437.leslie3d 16813.000 -
444.namd 671.000 -
445.gobmk 5238.000 -
447.dealII 1893.000 3.27%
450.soplex 543.000 13.36%
453.povray 528.000 1.93%
456.hmmer 2917.000 -
458.sjeng 7507.000 -
459.GemsFDTD 6676.000 -
462.libquantum 134.000 -
464.h264ref 4572.000 -
465.tonto 26573.000 -
470.lbm 3251.000 3.60%
471.omnetpp 2984.000 -
473.astar 5168.000 5.64%
482.sphinx3 829.000 3.75%
483.xalancbmk 5306.000 1.63%
Geometric Mean 2194.895 45.64%
Code Size (bytes)
Name Current %
400.perlbench 892927.000 -
401.bzip2 43576.000 -
403.gcc 2465052.000 -
410.bwaves 618440.000 -
429.mcf 9100.000 -
433.milc 61674.000 -
434.zeusmp 1010811.000 -
437.leslie3d 901620.000 -
444.namd 135988.000 -
445.gobmk 1843686.000 -
447.dealII 325995.000 -
450.soplex 228234.000 -
453.povray 678613.000 -
456.hmmer 112136.000 -
458.sjeng 105951.000 -
459.GemsFDTD 1655451.000 -
462.libquantum 16732.000 -
464.h264ref 388537.000 -1.61%
465.tonto 7646422.000 -
470.lbm 10224.000 -
471.omnetpp 490193.000 -
473.astar 23440.000 -
482.sphinx3 118969.000 -
483.xalancbmk 2515342.000 -
Geometric Mean 264053.502 56.71%
Score
Name Current %
nb_compile_successful 24.000 26.32%
nb_execution_successful 24.000 26.32%
nb_total_tests 34.000 17.24%
Geometric Mean 26.955 23.22%
sample variability
Name Current %
400.perlbench 5.000 -
400.perlbench (avg) 5.000 -
400.perlbench (avg) (percentage) 0.270 -
400.perlbench (max) 9.000 -
400.perlbench (max) (percentage) 0.470 -2.08%
401.bzip2 23.000 9.52%
401.bzip2 (avg) 23.000 9.52%
401.bzip2 (avg) (percentage) 0.870 -
401.bzip2 (max) 10.000 -
401.bzip2 (max) (percentage) 0.370 -7.50%
403.gcc 2.000 100.00%
403.gcc (avg) 2.000 100.00%
403.gcc (avg) (percentage) 2.880 -
403.gcc (max) 5.000 -
403.gcc (max) (percentage) 7.940 -1.12%
429.mcf 48.000 26.32%
429.mcf (avg) 48.000 26.32%
429.mcf (avg) (percentage) 2.410 -
429.mcf (max) 9.000 28.57%
429.mcf (max) (percentage) 0.460 -
433.milc 9.000 -
433.milc (avg) 9.000 -
433.milc (avg) (percentage) 0.780 -
433.milc (max) 11.000 -8.33%
433.milc (max) (percentage) 0.920 -7.07%
444.namd 5.000 -
444.namd (avg) 5.000 -
444.namd (avg) (percentage) 0.860 -
444.namd (max) 5.000 -16.67%
444.namd (max) (percentage) 0.870 -3.33%
445.gobmk 18.000 -
445.gobmk (avg) 18.000 -
445.gobmk (avg) (percentage) 0.360 -
445.gobmk (max) 64.000 -1.54%
445.gobmk (max) (percentage) 1.240 -
447.dealII 13.000 8.33%
447.dealII (avg) 13.000 8.33%
447.dealII (avg) (percentage) 0.700 -
447.dealII (max) 12.000 -
447.dealII (max) (percentage) 0.670 -4.29%
450.soplex 15.000 15.38%
450.soplex (avg) 15.000 15.38%
450.soplex (avg) (percentage) 2.870 -
450.soplex (max) 5.000 -
450.soplex (max) (percentage) 1.070 -2.73%
453.povray 3.000 -
453.povray (avg) 3.000 -
453.povray (avg) (percentage) 0.730 -
453.povray (max) 2.000 -33.33%
453.povray (max) (percentage) 0.550 -5.17%
456.hmmer 16.000 -
456.hmmer (avg) 16.000 -
456.hmmer (avg) (percentage) 0.570 -
456.hmmer (max) 22.000 -4.35%
456.hmmer (max) (percentage) 0.780 -3.70%
458.sjeng 22.000 -
458.sjeng (avg) 22.000 -
458.sjeng (avg) (percentage) 0.300 -
458.sjeng (max) 26.000 -
458.sjeng (max) (percentage) 0.350 -
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.120 -
462.libquantum (max) 1.000 -
462.libquantum (max) (percentage) 0.960 -3.03%
464.h264ref 17.000 -
464.h264ref (avg) 17.000 -
464.h264ref (avg) (percentage) 0.390 -
464.h264ref (max) 17.000 -5.56%
464.h264ref (max) (percentage) 0.390 -2.50%
470.lbm 63.000 5.00%
470.lbm (avg) 63.000 5.00%
470.lbm (avg) (percentage) 1.940 -
470.lbm (max) 8.000 -11.11%
470.lbm (max) (percentage) 0.270 -6.90%
471.omnetpp 34.000 -
471.omnetpp (avg) 34.000 -
471.omnetpp (avg) (percentage) 1.170 -
471.omnetpp (max) 76.000 -5.00%
471.omnetpp (max) (percentage) 2.550 -4.85%
473.astar 51.000 6.25%
473.astar (avg) 51.000 6.25%
473.astar (avg) (percentage) 1.000 -
473.astar (max) 29.000 3.57%
473.astar (max) (percentage) 0.570 -3.39%
482.sphinx3 15.000 7.14%
482.sphinx3 (avg) 15.000 7.14%
482.sphinx3 (avg) (percentage) 1.820 -
482.sphinx3 (max) 3.000 -
482.sphinx3 (max) (percentage) 0.480 -2.04%
483.xalancbmk 22.000 -
483.xalancbmk (avg) 22.000 -
483.xalancbmk (avg) (percentage) 0.430 -
483.xalancbmk (max) 20.000 -
483.xalancbmk (max) (percentage) 0.380 -2.56%
Geometric Mean 4.029 1.61%