Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/78b4e7c5e349d8c101b50affbd260eb109748f8f (+ 1130 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz_LTO-build/110/
View Options
Report
Filter

llvm-arm-master-Oz_LTO test results

Run Order Start Time Duration
Current 0110 (llvmorg-19-init-01166-g78b4e7c5e349) 2024-01-26T20:10:28 0:00:00
Previous 0108 (llvmorg-19-init-00036-gfb9a82b02357) 2024-01-19T10:08:22 0:00:00
Baseline 0001 (llvmorg-16-init-17036-g4f40923103d2) 2023-01-06T17:55:45 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 15 14
Performance Improvements 40 109
Added Tests 2 4
Unchanged Tests 723 653
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
483.xalancbmk 1.82% 3345.000 3406.000 - -63.67% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) 2200.00% 1.000 23.000 - -77.67% -
471.omnetpp (unexplained diff vs previous) (percentage) 1950.00% 0.040 0.820 - -16.33% -
471.omnetpp (max) 450.00% 4.000 22.000 - -97.74% -
471.omnetpp (max) (percentage) 440.00% 0.150 0.810 - -91.27% -
483.xalancbmk (max) 238.89% 18.000 61.000 - -94.95% -
483.xalancbmk (unexplained diff vs previous) 238.89% 18.000 61.000 - 0.00% -
483.xalancbmk (max) (percentage) 237.04% 0.540 1.820 - -85.89% -
483.xalancbmk (unexplained diff vs previous) (percentage) 231.48% 0.540 1.790 - 0.00% -
429.mcf 4.76% 21.000 22.000 - -92.57% -
429.mcf (avg) 4.76% 21.000 22.000 - -92.57% -
473.astar 2.70% 37.000 38.000 - -66.07% -
473.astar (avg) 2.70% 37.000 38.000 - -66.07% -
483.xalancbmk 1.54% 65.000 66.000 - -66.67% -
483.xalancbmk (avg) 1.54% 65.000 66.000 - -66.67% -

Performance Improvements - Code Size Δ Previous Current σ Δ (B) σ (B)
401.bzip2 -1.73% 33717.000 33133.000 - -1.68% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -3.39% 59.000 57.000 - -99.40% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
470.lbm (unexplained diff vs previous) (percentage) -82.50% 0.400 0.070 - -98.84% -
470.lbm (unexplained diff vs previous) -81.25% 16.000 3.000 - -99.63% -
456.hmmer (unexplained diff vs previous) (percentage) -43.75% 0.320 0.180 - -53.85% -
456.hmmer (unexplained diff vs previous) -41.67% 12.000 7.000 - -84.78% -
462.libquantum -33.33% 3.000 2.000 - -98.18% -
462.libquantum (avg) -33.33% 3.000 2.000 - -98.18% -
456.hmmer (max) -28.57% 14.000 10.000 - -99.19% -
456.hmmer (max) (percentage) -27.03% 0.370 0.270 - -97.42% -
429.mcf (max) -25.00% 4.000 3.000 - -99.87% -
470.lbm (max) -18.75% 16.000 13.000 - -98.28% -
470.lbm (max) (percentage) -17.50% 0.400 0.330 - -94.23% -
447.dealII (max) -12.50% 8.000 7.000 - -96.86% -
429.mcf (max) (percentage) -11.43% 0.350 0.310 - -98.10% -
462.libquantum (max) (percentage) -11.20% 2.500 2.220 - -65.20% -
445.gobmk (max) -8.33% 12.000 11.000 - -97.34% -
458.sjeng (max) (percentage) -8.33% 0.360 0.330 - -93.58% -
464.h264ref (max) -7.14% 14.000 13.000 - -78.33% -
458.sjeng (max) -6.90% 29.000 27.000 - -96.92% -
444.namd (max) (percentage) -6.54% 1.070 1.000 - -87.64% -
473.astar (max) (percentage) -5.88% 0.170 0.160 - -97.61% -
401.bzip2 (max) -5.88% 17.000 16.000 - -97.37% -
447.dealII (max) (percentage) -4.88% 0.410 0.390 - -83.61% -
401.bzip2 (max) (percentage) -4.76% 0.630 0.600 - -84.13% -
453.povray (max) (percentage) -4.72% 1.270 1.210 - 236.11% -
433.milc (max) (percentage) -4.71% 0.850 0.810 - 1.25% -
464.h264ref -4.35% 23.000 22.000 - -47.62% -
464.h264ref (avg) -4.35% 23.000 22.000 - -47.62% -
445.gobmk (max) (percentage) -4.00% 0.250 0.240 - -93.57% -
450.soplex (max) (percentage) -3.39% 2.360 2.280 - 38.18% -
482.sphinx3 (max) (percentage) -3.33% 0.900 0.870 - -73.72% -
456.hmmer -3.23% 62.000 60.000 - -63.41% -
456.hmmer (avg) -3.23% 62.000 60.000 - -63.41% -
456.hmmer (avg) (percentage) -2.45% 1.630 1.590 - 14.39% -
470.lbm (avg) (percentage) -1.87% 1.070 1.050 - -68.47% -
462.libquantum (avg) (percentage) -1.83% 1.640 1.610 - 109.09% -
471.omnetpp -1.67% 60.000 59.000 - -64.67% -
471.omnetpp (avg) -1.67% 60.000 59.000 - -64.67% -
471.omnetpp (avg) (percentage) -1.40% 2.150 2.120 - 32.50% -

Added Tests - sample variability
403.gcc
403.gcc (avg)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.97% 146765.000 149652.000 - 0.02% -
471.omnetpp 1.35% 316676.000 320957.000 - -0.02% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
462.libquantum (unexplained diff vs previous) (percentage) 5300.00% 0.010 0.540 - 0.00% -
453.povray (max) (percentage) 236.11% 0.360 1.210 - -4.72% -
403.gcc (avg) (percentage) 138.18% 0.550 1.310 - 0.00% -
450.soplex (avg) (percentage) 136.00% 0.500 1.180 - 0.00% -
462.libquantum (avg) (percentage) 109.09% 0.770 1.610 - -1.83% -
403.gcc (max) (percentage) 74.68% 2.330 4.070 - -0.73% -
450.soplex (max) (percentage) 38.18% 1.650 2.280 - -3.39% -
464.h264ref (avg) (percentage) 34.48% 0.290 0.390 - 0.00% -
471.omnetpp (avg) (percentage) 32.50% 1.600 2.120 - -1.40% -
453.povray (avg) (percentage) 15.00% 0.800 0.920 - 0.00% -
456.hmmer (avg) (percentage) 14.39% 1.390 1.590 - -2.45% -
433.milc (max) (percentage) 1.25% 0.800 0.810 - -4.71% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
429.mcf -3.92% 7239.000 6955.000 - -0.80% -
453.povray -2.21% 524493.000 512915.000 - -0.21% -
433.milc -2.12% 47001.000 46005.000 - -0.14% -
464.h264ref -1.81% 303767.000 298279.000 - -0.68% -
462.libquantum -1.70% 12504.000 12291.000 - 0.00% -
444.namd -1.69% 124963.000 122854.000 - -0.64% -
401.bzip2 -1.68% 33698.000 33133.000 - -1.73% -
470.lbm -1.09% 9338.000 9236.000 - 0.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.40% 9509.000 57.000 - -3.39% -
462.libquantum -98.72% 14409.000 184.000 - -0.54% -
450.soplex -96.43% 15112.000 539.000 - -0.74% -
482.sphinx3 -96.29% 26682.000 991.000 - -0.30% -
444.namd -93.50% 12400.000 806.000 - -0.62% -
453.povray -92.77% 6987.000 505.000 - -0.59% -
429.mcf -91.74% 13988.000 1155.000 - 0.96% -
433.milc -91.40% 13601.000 1170.000 - 0.43% -
401.bzip2 -82.51% 16137.000 2823.000 - -0.98% -
447.dealII -79.07% 9380.000 1963.000 - -0.51% -
471.omnetpp -73.27% 10477.000 2800.000 - -0.81% -
470.lbm -69.78% 13259.000 4007.000 - 0.07% -
456.hmmer -67.95% 11854.000 3799.000 - -0.18% -
483.xalancbmk -63.67% 9376.000 3406.000 - 1.82% -
473.astar -61.55% 12272.000 4718.000 - 0.77% -
464.h264ref -59.44% 14498.000 5881.000 - -0.98% -
445.gobmk -55.88% 11093.000 4894.000 - 0.14% -
458.sjeng -51.55% 17055.000 8264.000 - 0.11% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
429.mcf (max) -99.87% 2281.000 3.000 - -25.00% -
470.lbm (unexplained diff vs previous) -99.63% 804.000 3.000 - -81.25% -
462.libquantum (max) -99.56% 919.000 4.000 - 0.00% -
444.namd (max) -99.20% 1003.000 8.000 - 0.00% -
456.hmmer (max) -99.19% 1242.000 10.000 - -28.57% -
473.astar (max) -99.15% 822.000 7.000 - 0.00% -
403.gcc (max) -99.10% 221.000 2.000 - 0.00% -
482.sphinx3 (max) -99.09% 883.000 8.000 - 0.00% -
470.lbm (unexplained diff vs previous) (percentage) -98.84% 6.060 0.070 - -82.50% -
470.lbm (max) -98.28% 758.000 13.000 - -18.75% -
482.sphinx3 -98.28% 464.000 8.000 - 0.00% -
482.sphinx3 (avg) -98.28% 464.000 8.000 - 0.00% -
462.libquantum -98.18% 110.000 2.000 - -33.33% -
462.libquantum (avg) -98.18% 110.000 2.000 - -33.33% -
429.mcf (max) (percentage) -98.10% 16.310 0.310 - -11.43% -
471.omnetpp (max) -97.74% 972.000 22.000 - 450.00% -
473.astar (max) (percentage) -97.61% 6.700 0.160 - -5.88% -
456.hmmer (max) (percentage) -97.42% 10.480 0.270 - -27.03% -
401.bzip2 (max) -97.37% 609.000 16.000 - -5.88% -
445.gobmk (max) -97.34% 413.000 11.000 - -8.33% -
458.sjeng (max) -96.92% 876.000 27.000 - -6.90% -
447.dealII (max) -96.86% 223.000 7.000 - -12.50% -
444.namd -96.54% 318.000 11.000 - 0.00% -
444.namd (avg) -96.54% 318.000 11.000 - 0.00% -
450.soplex (max) -95.18% 249.000 12.000 - 0.00% -
483.xalancbmk (max) -94.95% 1209.000 61.000 - 238.89% -
470.lbm (max) (percentage) -94.23% 5.720 0.330 - -17.50% -
433.milc -93.97% 116.000 7.000 - 0.00% -
433.milc (avg) -93.97% 116.000 7.000 - 0.00% -
458.sjeng (max) (percentage) -93.58% 5.140 0.330 - -8.33% -
445.gobmk (max) (percentage) -93.57% 3.730 0.240 - -4.00% -
453.povray -92.73% 55.000 4.000 - 0.00% -
453.povray (avg) -92.73% 55.000 4.000 - 0.00% -
429.mcf -92.57% 296.000 22.000 - 4.76% -
429.mcf (avg) -92.57% 296.000 22.000 - 4.76% -
450.soplex -92.00% 75.000 6.000 - 0.00% -
450.soplex (avg) -92.00% 75.000 6.000 - 0.00% -
433.milc (max) -91.67% 108.000 9.000 - 0.00% -
471.omnetpp (max) (percentage) -91.27% 9.280 0.810 - 440.00% -
470.lbm -90.48% 441.000 42.000 - 0.00% -
470.lbm (avg) -90.48% 441.000 42.000 - 0.00% -
444.namd (max) (percentage) -87.64% 8.090 1.000 - -6.54% -
447.dealII -86.73% 98.000 13.000 - 0.00% -
447.dealII (avg) -86.73% 98.000 13.000 - 0.00% -
483.xalancbmk (max) (percentage) -85.89% 12.900 1.820 - 237.04% -
456.hmmer (unexplained diff vs previous) -84.78% 46.000 7.000 - -41.67% -
401.bzip2 -84.62% 104.000 16.000 - 0.00% -
401.bzip2 (avg) -84.62% 104.000 16.000 - 0.00% -
401.bzip2 (max) (percentage) -84.13% 3.780 0.600 - -4.76% -
447.dealII (max) (percentage) -83.61% 2.380 0.390 - -4.88% -
464.h264ref (max) -78.33% 60.000 13.000 - -7.14% -
471.omnetpp (unexplained diff vs previous) -77.67% 103.000 23.000 - 2200.00% -
453.povray (max) -76.00% 25.000 6.000 - 0.00% -
445.gobmk -74.32% 74.000 19.000 - 0.00% -
445.gobmk (avg) -74.32% 74.000 19.000 - 0.00% -
482.sphinx3 (max) (percentage) -73.72% 3.310 0.870 - -3.33% -
458.sjeng -69.12% 136.000 42.000 - 0.00% -
458.sjeng (avg) -69.12% 136.000 42.000 - 0.00% -
470.lbm (avg) (percentage) -68.47% 3.330 1.050 - -1.87% -
483.xalancbmk -66.67% 198.000 66.000 - 1.54% -
483.xalancbmk (avg) -66.67% 198.000 66.000 - 1.54% -
473.astar -66.07% 112.000 38.000 - 2.70% -
473.astar (avg) -66.07% 112.000 38.000 - 2.70% -
462.libquantum (max) (percentage) -65.20% 6.380 2.220 - -11.20% -
471.omnetpp -64.67% 167.000 59.000 - -1.67% -
471.omnetpp (avg) -64.67% 167.000 59.000 - -1.67% -
456.hmmer -63.41% 164.000 60.000 - -3.23% -
456.hmmer (avg) -63.41% 164.000 60.000 - -3.23% -
456.hmmer (unexplained diff vs previous) (percentage) -53.85% 0.390 0.180 - -43.75% -
482.sphinx3 (avg) (percentage) -51.15% 1.740 0.850 - 0.00% -
464.h264ref -47.62% 42.000 22.000 - -4.35% -
464.h264ref (avg) -47.62% 42.000 22.000 - -4.35% -
444.namd (avg) (percentage) -46.69% 2.570 1.370 - 0.00% -
464.h264ref (max) (percentage) -45.24% 0.420 0.230 - -4.17% -
445.gobmk (avg) (percentage) -41.79% 0.670 0.390 - 0.00% -
458.sjeng (avg) (percentage) -35.00% 0.800 0.520 - 0.00% -
447.dealII (avg) (percentage) -33.33% 1.050 0.700 - 0.00% -
433.milc (avg) (percentage) -26.74% 0.860 0.630 - 0.00% -
471.omnetpp (unexplained diff vs previous) (percentage) -16.33% 0.980 0.820 - 1950.00% -
473.astar (avg) (percentage) -11.96% 0.920 0.810 - 0.00% -
429.mcf (avg) (percentage) -9.91% 2.120 1.910 - 0.00% -
401.bzip2 (avg) (percentage) -9.23% 0.650 0.590 - 0.00% -
483.xalancbmk (avg) (percentage) -7.55% 2.120 1.960 - -0.51% -

Added Tests - sample variability
403.gcc
403.gcc (avg)
483.xalancbmk (unexplained diff vs previous)
483.xalancbmk (unexplained diff vs previous) (percentage)

Report Time: 0.35s
Execution (seconds)
Name Current %
401.bzip2 2823.000 -
403.gcc 57.000 -3.39%
429.mcf 1155.000 -
433.milc 1170.000 -
444.namd 806.000 -
445.gobmk 4894.000 -
447.dealII 1963.000 -
450.soplex 539.000 -
453.povray 505.000 -
456.hmmer 3799.000 -
458.sjeng 8264.000 -
462.libquantum 184.000 -
464.h264ref 5881.000 -
470.lbm 4007.000 -
471.omnetpp 2800.000 -
473.astar 4718.000 -
482.sphinx3 991.000 -
483.xalancbmk 3406.000 1.82%
Geometric Mean 1553.646 -
Code Size (bytes)
Name Current %
401.bzip2 33133.000 -1.73%
403.gcc 1663135.000 -
429.mcf 6955.000 -
433.milc 46005.000 -
444.namd 122854.000 -
445.gobmk 1256769.000 -
447.dealII 195882.000 -
450.soplex 149652.000 -
453.povray 512915.000 -
456.hmmer 80385.000 -
458.sjeng 79898.000 -
462.libquantum 12291.000 -
464.h264ref 298279.000 -
470.lbm 9236.000 -
471.omnetpp 320957.000 -
473.astar 15571.000 -
482.sphinx3 87440.000 -
483.xalancbmk 1610720.000 -
Geometric Mean 110732.558 -
sample variability
Name Current %
401.bzip2 16.000 -
401.bzip2 (avg) 16.000 -
401.bzip2 (avg) (percentage) 0.590 -
401.bzip2 (max) 16.000 -5.88%
401.bzip2 (max) (percentage) 0.600 -4.76%
403.gcc 0.000 -
403.gcc (avg) 0.000 -
403.gcc (avg) (percentage) 1.310 -
403.gcc (max) 2.000 -
403.gcc (max) (percentage) 4.070 -
429.mcf 22.000 4.76%
429.mcf (avg) 22.000 4.76%
429.mcf (avg) (percentage) 1.910 -
429.mcf (max) 3.000 -25.00%
429.mcf (max) (percentage) 0.310 -11.43%
433.milc 7.000 -
433.milc (avg) 7.000 -
433.milc (avg) (percentage) 0.630 -
433.milc (max) 9.000 -
433.milc (max) (percentage) 0.810 -4.71%
444.namd 11.000 -
444.namd (avg) 11.000 -
444.namd (avg) (percentage) 1.370 -
444.namd (max) 8.000 -
444.namd (max) (percentage) 1.000 -6.54%
445.gobmk 19.000 -
445.gobmk (avg) 19.000 -
445.gobmk (avg) (percentage) 0.390 -
445.gobmk (max) 11.000 -8.33%
445.gobmk (max) (percentage) 0.240 -4.00%
447.dealII 13.000 -
447.dealII (avg) 13.000 -
447.dealII (avg) (percentage) 0.700 -
447.dealII (max) 7.000 -12.50%
447.dealII (max) (percentage) 0.390 -4.88%
450.soplex 6.000 -
450.soplex (avg) 6.000 -
450.soplex (avg) (percentage) 1.180 -
450.soplex (max) 12.000 -
450.soplex (max) (percentage) 2.280 -3.39%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.920 -
453.povray (max) 6.000 -
453.povray (max) (percentage) 1.210 -4.72%
456.hmmer 60.000 -3.23%
456.hmmer (avg) 60.000 -3.23%
456.hmmer (avg) (percentage) 1.590 -2.45%
456.hmmer (max) 10.000 -28.57%
456.hmmer (max) (percentage) 0.270 -27.03%
456.hmmer (unexplained diff vs previous) 7.000 -41.67%
456.hmmer (unexplained diff vs previous) (percentage) 0.180 -43.75%
458.sjeng 42.000 -
458.sjeng (avg) 42.000 -
458.sjeng (avg) (percentage) 0.520 -
458.sjeng (max) 27.000 -6.90%
458.sjeng (max) (percentage) 0.330 -8.33%
462.libquantum 2.000 -33.33%
462.libquantum (avg) 2.000 -33.33%
462.libquantum (avg) (percentage) 1.610 -1.83%
462.libquantum (max) 4.000 -
462.libquantum (max) (percentage) 2.220 -11.20%
462.libquantum (unexplained diff vs previous) 1.000 -
462.libquantum (unexplained diff vs previous) (percentage) 0.540 -
464.h264ref 22.000 -4.35%
464.h264ref (avg) 22.000 -4.35%
464.h264ref (avg) (percentage) 0.390 -
464.h264ref (max) 13.000 -7.14%
464.h264ref (max) (percentage) 0.230 -
470.lbm 42.000 -
470.lbm (avg) 42.000 -
470.lbm (avg) (percentage) 1.050 -1.87%
470.lbm (max) 13.000 -18.75%
470.lbm (max) (percentage) 0.330 -17.50%
470.lbm (unexplained diff vs previous) 3.000 -81.25%
470.lbm (unexplained diff vs previous) (percentage) 0.070 -82.50%
471.omnetpp 59.000 -1.67%
471.omnetpp (avg) 59.000 -1.67%
471.omnetpp (avg) (percentage) 2.120 -1.40%
471.omnetpp (max) 22.000 450.00%
471.omnetpp (max) (percentage) 0.810 440.00%
471.omnetpp (unexplained diff vs previous) 23.000 2200.00%
471.omnetpp (unexplained diff vs previous) (percentage) 0.820 1950.00%
473.astar 38.000 2.70%
473.astar (avg) 38.000 2.70%
473.astar (avg) (percentage) 0.810 -
473.astar (max) 7.000 -
473.astar (max) (percentage) 0.160 -5.88%
482.sphinx3 8.000 -
482.sphinx3 (avg) 8.000 -
482.sphinx3 (avg) (percentage) 0.850 -
482.sphinx3 (max) 8.000 -
482.sphinx3 (max) (percentage) 0.870 -3.33%
483.xalancbmk 66.000 1.54%
483.xalancbmk (avg) 66.000 1.54%
483.xalancbmk (avg) (percentage) 1.960 -
483.xalancbmk (max) 61.000 238.89%
483.xalancbmk (max) (percentage) 1.820 237.04%
483.xalancbmk (unexplained diff vs previous) 61.000 238.89%
483.xalancbmk (unexplained diff vs previous) (percentage) 1.790 231.48%
Geometric Mean 3.202 -13.63%