Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/90ba33099cbb17e7c159e9ebc5a512037db99d6d (+ 1 commits)
regression LLVM-1105
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Os-build/123/
View Options
Report
Filter

llvm-arm-master-Os test results

Run Order Start Time Duration
Current 0123 (llvmorg-19-init-00059-g90ba33099cbb) 2024-01-23T06:13:58 0:00:00
Previous 0122 (llvmorg-19-init-00058-g98509c7f9792) 2024-01-21T22:02:28 0:00:00
Baseline 0001 (llvmorg-16-init-16061-ge7790fbed32b) 2022-12-29T19:41:07 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 16 27
Performance Improvements 44 98
Added Tests 1 1
Unchanged Tests 719 654
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Code Size Δ Previous Current σ Δ (B) σ (B)
401.bzip2 5.31% 38329.000 40365.000 - 2.30% -

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
458.sjeng 4.07% 8142.000 8473.000 - -48.35% -
450.soplex 3.15% 508.000 524.000 - -96.37% -
444.namd 2.54% 709.000 727.000 - -94.61% -
471.omnetpp 1.86% 2801.000 2853.000 - -73.45% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
450.soplex 20.00% 5.000 6.000 - -88.24% -
450.soplex (avg) 20.00% 5.000 6.000 - -88.24% -
471.omnetpp (max) 10.42% 48.000 53.000 - -71.96% -
471.omnetpp (unexplained diff vs previous) 8.33% 48.000 52.000 - -43.48% -
471.omnetpp (max) (percentage) 6.90% 1.740 1.860 - 5.68% -
471.omnetpp (unexplained diff vs previous) (percentage) 6.43% 1.710 1.820 - 111.63% -
458.sjeng 4.76% 42.000 44.000 - -53.68% -
458.sjeng (avg) 4.76% 42.000 44.000 - -53.68% -
450.soplex (avg) (percentage) 3.54% 1.130 1.170 - 225.00% -
471.omnetpp 1.79% 56.000 57.000 - -31.33% -
471.omnetpp (avg) 1.79% 56.000 57.000 - -31.33% -

Performance Improvements - Code Size Δ Previous Current σ Δ (B) σ (B)
444.namd -1.85% 172689.000 169489.000 - -3.24% -
450.soplex -1.60% 201502.000 198287.000 - -0.21% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -4.84% 62.000 59.000 - -99.35% -
462.libquantum -1.01% 99.000 98.000 - -99.25% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
483.xalancbmk (unexplained diff vs previous) -68.42% 19.000 6.000 - -90.16% -
483.xalancbmk (unexplained diff vs previous) (percentage) -67.92% 0.530 0.170 - -73.85% -
470.lbm (unexplained diff vs previous) -33.33% 18.000 12.000 - -97.59% -
470.lbm (unexplained diff vs previous) (percentage) -33.33% 0.570 0.380 - -89.39% -
433.milc (max) -20.00% 5.000 4.000 - -97.83% -
462.libquantum (max) (percentage) -19.84% 1.260 1.010 - 405.00% -
483.xalancbmk (max) (percentage) -19.64% 0.560 0.450 - -30.77% -
483.xalancbmk (max) -15.79% 19.000 16.000 - -73.77% -
450.soplex (unexplained diff vs previous) (percentage) -13.84% 3.540 3.050 - 435.09% -
458.sjeng (max) (percentage) -12.50% 0.240 0.210 - -47.50% -
482.sphinx3 (max) -12.50% 8.000 7.000 - -99.20% -
450.soplex (unexplained diff vs previous) -11.11% 18.000 16.000 - -80.49% -
433.milc (max) (percentage) -10.87% 0.460 0.410 - -70.07% -
458.sjeng (max) -10.53% 19.000 17.000 - -73.85% -
453.povray (max) (percentage) -10.38% 1.060 0.950 - -50.78% -
447.dealII (max) -10.00% 10.000 9.000 - -93.02% -
470.lbm (max) (percentage) -8.77% 0.570 0.520 - -90.61% -
445.gobmk (max) -8.33% 12.000 11.000 - -62.07% -
445.gobmk (max) (percentage) -8.33% 0.240 0.220 - -8.33% -
450.soplex (max) (percentage) -7.89% 3.420 3.150 - 452.63% -
401.bzip2 (max) (percentage) -7.41% 1.080 1.000 - 222.58% -
447.dealII (max) (percentage) -7.14% 0.560 0.520 - -67.09% -
401.bzip2 (max) -6.90% 29.000 27.000 - -43.75% -
429.mcf (max) -5.88% 17.000 16.000 - -76.12% -
450.soplex (max) -5.88% 17.000 16.000 - -80.49% -
464.h264ref (max) -5.88% 17.000 16.000 - -73.33% -
470.lbm (max) -5.88% 17.000 16.000 - -97.92% -
456.hmmer (max) (percentage) -4.55% 0.220 0.210 - -30.00% -
482.sphinx3 (max) (percentage) -4.26% 0.940 0.900 - -72.89% -
429.mcf (max) (percentage) -3.40% 1.470 1.420 - 153.57% -
444.namd (max) (percentage) -3.26% 0.920 0.890 - -86.39% -
464.h264ref (max) (percentage) -3.23% 0.310 0.300 - -23.08% -
473.astar (max) -3.12% 32.000 31.000 - -72.81% -
483.xalancbmk -3.12% 64.000 62.000 - 31.91% -
483.xalancbmk (avg) -3.12% 64.000 62.000 - 31.91% -
473.astar (max) (percentage) -2.99% 0.670 0.650 - -18.75% -
483.xalancbmk (avg) (percentage) -2.22% 1.800 1.760 - 252.00% -
403.gcc (max) (percentage) -1.64% 4.260 4.190 - 1002.63% -
473.astar -1.37% 73.000 72.000 - 1.41% -
473.astar (avg) -1.37% 73.000 72.000 - 1.41% -

Added Tests - sample variability
462.libquantum (max)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
401.bzip2 2.30% 39458.000 40365.000 - 5.31% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
403.gcc (max) (percentage) 1002.63% 0.380 4.190 - -1.64% -
462.libquantum (unexplained diff vs previous) (percentage) 750.00% 0.120 1.020 - 0.99% -
462.libquantum (avg) (percentage) 746.67% 0.150 1.270 - 0.00% -
403.gcc (avg) (percentage) 466.67% 0.480 2.720 - 0.00% -
450.soplex (max) (percentage) 452.63% 0.570 3.150 - -7.89% -
450.soplex (unexplained diff vs previous) (percentage) 435.09% 0.570 3.050 - -13.84% -
462.libquantum (max) (percentage) 405.00% 0.200 1.010 - -19.84% -
429.mcf (avg) (percentage) 380.00% 0.450 2.160 - 0.00% -
483.xalancbmk (avg) (percentage) 252.00% 0.500 1.760 - -2.22% -
450.soplex (avg) (percentage) 225.00% 0.360 1.170 - 3.54% -
401.bzip2 (max) (percentage) 222.58% 0.310 1.000 - -7.41% -
473.astar (avg) (percentage) 198.00% 0.500 1.490 - 0.00% -
445.gobmk (avg) (percentage) 166.67% 0.180 0.480 - 0.00% -
401.bzip2 (avg) (percentage) 159.09% 0.220 0.570 - 0.00% -
471.omnetpp (avg) (percentage) 156.41% 0.780 2.000 - 0.00% -
429.mcf (max) (percentage) 153.57% 0.560 1.420 - -3.40% -
456.hmmer (avg) (percentage) 122.22% 0.360 0.800 - 0.00% -
471.omnetpp (unexplained diff vs previous) (percentage) 111.63% 0.860 1.820 - 6.43% -
464.h264ref (avg) (percentage) 93.94% 0.330 0.640 - 0.00% -
483.xalancbmk 31.91% 47.000 62.000 - -3.12% -
483.xalancbmk (avg) 31.91% 47.000 62.000 - -3.12% -
445.gobmk 13.64% 22.000 25.000 - 0.00% -
445.gobmk (avg) 13.64% 22.000 25.000 - 0.00% -
471.omnetpp (max) (percentage) 5.68% 1.760 1.860 - 6.90% -
473.astar 1.41% 71.000 72.000 - -1.37% -
473.astar (avg) 1.41% 71.000 72.000 - -1.37% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -9.39% 26756.000 24243.000 - -0.03% -
444.namd -3.24% 175160.000 169489.000 - -1.85% -
433.milc -2.74% 74971.000 72914.000 - -0.20% -
453.povray -2.73% 593097.000 576894.000 - 0.03% -
447.dealII -2.44% 1777992.000 1734691.000 - -0.48% -
482.sphinx3 -1.36% 115736.000 114163.000 - -0.06% -
464.h264ref -1.13% 354618.000 350614.000 - -0.68% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.35% 9099.000 59.000 - -4.84% -
462.libquantum -99.25% 13126.000 98.000 - -1.01% -
482.sphinx3 -96.66% 26310.000 878.000 - -0.45% -
450.soplex -96.37% 14429.000 524.000 - 3.15% -
444.namd -94.61% 13495.000 727.000 - 2.54% -
453.povray -93.23% 6857.000 464.000 - 0.00% -
433.milc -91.51% 13482.000 1145.000 - 0.88% -
429.mcf -90.29% 11999.000 1165.000 - 0.34% -
401.bzip2 -82.67% 15683.000 2718.000 - 0.48% -
447.dealII -77.99% 8196.000 1804.000 - 0.06% -
470.lbm -77.17% 13882.000 3169.000 - 0.38% -
471.omnetpp -73.45% 10746.000 2853.000 - 1.86% -
456.hmmer -73.21% 13365.000 3580.000 - 0.70% -
473.astar -65.66% 14262.000 4898.000 - -0.12% -
464.h264ref -63.94% 15493.000 5587.000 - 0.02% -
483.xalancbmk -62.08% 9427.000 3575.000 - 0.17% -
445.gobmk -57.89% 12400.000 5222.000 - -0.74% -
458.sjeng -48.35% 16406.000 8473.000 - 4.07% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
444.namd (max) -99.32% 882.000 6.000 - 0.00% -
482.sphinx3 (max) -99.20% 873.000 7.000 - -12.50% -
482.sphinx3 -98.16% 489.000 9.000 - 0.00% -
482.sphinx3 (avg) -98.16% 489.000 9.000 - 0.00% -
444.namd -98.05% 514.000 10.000 - 0.00% -
444.namd (avg) -98.05% 514.000 10.000 - 0.00% -
470.lbm (max) -97.92% 769.000 16.000 - -5.88% -
433.milc (max) -97.83% 184.000 4.000 - -20.00% -
403.gcc -97.67% 43.000 1.000 - 0.00% -
403.gcc (avg) -97.67% 43.000 1.000 - 0.00% -
470.lbm (unexplained diff vs previous) -97.59% 497.000 12.000 - -33.33% -
453.povray (max) -96.97% 132.000 4.000 - 0.00% -
433.milc -96.32% 136.000 5.000 - 0.00% -
433.milc (avg) -96.32% 136.000 5.000 - 0.00% -
462.libquantum -94.74% 19.000 1.000 - 0.00% -
462.libquantum (avg) -94.74% 19.000 1.000 - 0.00% -
453.povray -94.62% 93.000 5.000 - 0.00% -
453.povray (avg) -94.62% 93.000 5.000 - 0.00% -
403.gcc (max) -94.12% 34.000 2.000 - 0.00% -
462.libquantum (unexplained diff vs previous) -93.75% 16.000 1.000 - 0.00% -
447.dealII (max) -93.02% 129.000 9.000 - -10.00% -
470.lbm (max) (percentage) -90.61% 5.540 0.520 - -8.77% -
483.xalancbmk (unexplained diff vs previous) -90.16% 61.000 6.000 - -68.42% -
447.dealII -90.00% 120.000 12.000 - 0.00% -
447.dealII (avg) -90.00% 120.000 12.000 - 0.00% -
470.lbm -89.44% 322.000 34.000 - 0.00% -
470.lbm (avg) -89.44% 322.000 34.000 - 0.00% -
470.lbm (unexplained diff vs previous) (percentage) -89.39% 3.580 0.380 - -33.33% -
450.soplex -88.24% 51.000 6.000 - 20.00% -
450.soplex (avg) -88.24% 51.000 6.000 - 20.00% -
444.namd (max) (percentage) -86.39% 6.540 0.890 - -3.26% -
456.hmmer (max) -82.50% 40.000 7.000 - 0.00% -
450.soplex (max) -80.49% 82.000 16.000 - -5.88% -
450.soplex (unexplained diff vs previous) -80.49% 82.000 16.000 - -11.11% -
429.mcf (max) -76.12% 67.000 16.000 - -5.88% -
458.sjeng (max) -73.85% 65.000 17.000 - -10.53% -
483.xalancbmk (unexplained diff vs previous) (percentage) -73.85% 0.650 0.170 - -67.92% -
483.xalancbmk (max) -73.77% 61.000 16.000 - -15.79% -
464.h264ref (max) -73.33% 60.000 16.000 - -5.88% -
482.sphinx3 (max) (percentage) -72.89% 3.320 0.900 - -4.26% -
473.astar (max) -72.81% 114.000 31.000 - -3.12% -
471.omnetpp (max) -71.96% 189.000 53.000 - 10.42% -
433.milc (max) (percentage) -70.07% 1.370 0.410 - -10.87% -
447.dealII (max) (percentage) -67.09% 1.580 0.520 - -7.14% -
445.gobmk (max) -62.07% 29.000 11.000 - -8.33% -
444.namd (avg) (percentage) -60.89% 3.810 1.490 - 0.00% -
401.bzip2 -55.88% 34.000 15.000 - 0.00% -
401.bzip2 (avg) -55.88% 34.000 15.000 - 0.00% -
458.sjeng -53.68% 95.000 44.000 - 4.76% -
458.sjeng (avg) -53.68% 95.000 44.000 - 4.76% -
447.dealII (avg) (percentage) -53.06% 1.470 0.690 - 0.00% -
470.lbm (avg) (percentage) -53.02% 2.320 1.090 - -0.91% -
429.mcf -52.83% 53.000 25.000 - 0.00% -
429.mcf (avg) -52.83% 53.000 25.000 - 0.00% -
453.povray (max) (percentage) -50.78% 1.930 0.950 - -10.38% -
433.milc (avg) (percentage) -48.51% 1.010 0.520 - 0.00% -
458.sjeng (max) (percentage) -47.50% 0.400 0.210 - -12.50% -
482.sphinx3 (avg) (percentage) -44.09% 1.860 1.040 - 0.00% -
401.bzip2 (max) -43.75% 48.000 27.000 - -6.90% -
471.omnetpp (unexplained diff vs previous) -43.48% 92.000 52.000 - 8.33% -
456.hmmer -41.67% 48.000 28.000 - 0.00% -
456.hmmer (avg) -41.67% 48.000 28.000 - 0.00% -
464.h264ref -31.37% 51.000 35.000 - 0.00% -
464.h264ref (avg) -31.37% 51.000 35.000 - 0.00% -
471.omnetpp -31.33% 83.000 57.000 - 1.79% -
471.omnetpp (avg) -31.33% 83.000 57.000 - 1.79% -
483.xalancbmk (max) (percentage) -30.77% 0.650 0.450 - -19.64% -
456.hmmer (max) (percentage) -30.00% 0.300 0.210 - -4.55% -
464.h264ref (max) (percentage) -23.08% 0.390 0.300 - -3.23% -
473.astar (max) (percentage) -18.75% 0.800 0.650 - -2.99% -
453.povray (avg) (percentage) -11.68% 1.370 1.210 - 0.00% -
458.sjeng (avg) (percentage) -10.34% 0.580 0.520 - 0.00% -
445.gobmk (max) (percentage) -8.33% 0.240 0.220 - -8.33% -

Added Tests - sample variability
462.libquantum (max)

Report Time: 0.36s
Execution (seconds)
Name Current %
401.bzip2 2718.000 -
403.gcc 59.000 -4.84%
429.mcf 1165.000 -
433.milc 1145.000 -
444.namd 727.000 2.54%
445.gobmk 5222.000 -
447.dealII 1804.000 -
450.soplex 524.000 3.15%
453.povray 464.000 -
456.hmmer 3580.000 -
458.sjeng 8473.000 4.07%
462.libquantum 98.000 -1.01%
464.h264ref 5587.000 -
470.lbm 3169.000 -
471.omnetpp 2853.000 1.86%
473.astar 4898.000 -
482.sphinx3 878.000 -
483.xalancbmk 3575.000 -
Geometric Mean 1451.931 -
Code Size (bytes)
Name Current %
401.bzip2 40365.000 5.31%
403.gcc 1957009.000 -
429.mcf 7842.000 -
433.milc 72914.000 -
444.namd 169489.000 -1.85%
445.gobmk 1308656.000 -
447.dealII 1734691.000 -
450.soplex 198287.000 -1.60%
453.povray 576894.000 -
456.hmmer 166438.000 -
458.sjeng 97233.000 -
462.libquantum 24243.000 -
464.h264ref 350614.000 -
470.lbm 9702.000 -
471.omnetpp 376075.000 -
473.astar 23323.000 -
482.sphinx3 114163.000 -
483.xalancbmk 2283084.000 -
Geometric Mean 162240.442 -
sample variability
Name Current %
401.bzip2 15.000 -
401.bzip2 (avg) 15.000 -
401.bzip2 (avg) (percentage) 0.570 -
401.bzip2 (max) 27.000 -6.90%
401.bzip2 (max) (percentage) 1.000 -7.41%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 2.720 -
403.gcc (max) 2.000 -
403.gcc (max) (percentage) 4.190 -1.64%
429.mcf 25.000 -
429.mcf (avg) 25.000 -
429.mcf (avg) (percentage) 2.160 -
429.mcf (max) 16.000 -5.88%
429.mcf (max) (percentage) 1.420 -3.40%
433.milc 5.000 -
433.milc (avg) 5.000 -
433.milc (avg) (percentage) 0.520 -
433.milc (max) 4.000 -20.00%
433.milc (max) (percentage) 0.410 -10.87%
444.namd 10.000 -
444.namd (avg) 10.000 -
444.namd (avg) (percentage) 1.490 -
444.namd (max) 6.000 -
444.namd (max) (percentage) 0.890 -3.26%
445.gobmk 25.000 -
445.gobmk (avg) 25.000 -
445.gobmk (avg) (percentage) 0.480 -
445.gobmk (max) 11.000 -8.33%
445.gobmk (max) (percentage) 0.220 -8.33%
447.dealII 12.000 -
447.dealII (avg) 12.000 -
447.dealII (avg) (percentage) 0.690 -
447.dealII (max) 9.000 -10.00%
447.dealII (max) (percentage) 0.520 -7.14%
450.soplex 6.000 20.00%
450.soplex (avg) 6.000 20.00%
450.soplex (avg) (percentage) 1.170 3.54%
450.soplex (max) 16.000 -5.88%
450.soplex (max) (percentage) 3.150 -7.89%
450.soplex (unexplained diff vs previous) 16.000 -11.11%
450.soplex (unexplained diff vs previous) (percentage) 3.050 -13.84%
453.povray 5.000 -
453.povray (avg) 5.000 -
453.povray (avg) (percentage) 1.210 -
453.povray (max) 4.000 -
453.povray (max) (percentage) 0.950 -10.38%
456.hmmer 28.000 -
456.hmmer (avg) 28.000 -
456.hmmer (avg) (percentage) 0.800 -
456.hmmer (max) 7.000 -
456.hmmer (max) (percentage) 0.210 -4.55%
458.sjeng 44.000 4.76%
458.sjeng (avg) 44.000 4.76%
458.sjeng (avg) (percentage) 0.520 -
458.sjeng (max) 17.000 -10.53%
458.sjeng (max) (percentage) 0.210 -12.50%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.270 -
462.libquantum (max) 0.000 -
462.libquantum (max) (percentage) 1.010 -19.84%
462.libquantum (unexplained diff vs previous) 1.000 -
462.libquantum (unexplained diff vs previous) (percentage) 1.020 -
464.h264ref 35.000 -
464.h264ref (avg) 35.000 -
464.h264ref (avg) (percentage) 0.640 -
464.h264ref (max) 16.000 -5.88%
464.h264ref (max) (percentage) 0.300 -3.23%
470.lbm 34.000 -
470.lbm (avg) 34.000 -
470.lbm (avg) (percentage) 1.090 -
470.lbm (max) 16.000 -5.88%
470.lbm (max) (percentage) 0.520 -8.77%
470.lbm (unexplained diff vs previous) 12.000 -33.33%
470.lbm (unexplained diff vs previous) (percentage) 0.380 -33.33%
471.omnetpp 57.000 1.79%
471.omnetpp (avg) 57.000 1.79%
471.omnetpp (avg) (percentage) 2.000 -
471.omnetpp (max) 53.000 10.42%
471.omnetpp (max) (percentage) 1.860 6.90%
471.omnetpp (unexplained diff vs previous) 52.000 8.33%
471.omnetpp (unexplained diff vs previous) (percentage) 1.820 6.43%
473.astar 72.000 -1.37%
473.astar (avg) 72.000 -1.37%
473.astar (avg) (percentage) 1.490 -
473.astar (max) 31.000 -3.12%
473.astar (max) (percentage) 0.650 -2.99%
482.sphinx3 9.000 -
482.sphinx3 (avg) 9.000 -
482.sphinx3 (avg) (percentage) 1.040 -
482.sphinx3 (max) 7.000 -12.50%
482.sphinx3 (max) (percentage) 0.900 -4.26%
483.xalancbmk 62.000 -3.12%
483.xalancbmk (avg) 62.000 -3.12%
483.xalancbmk (avg) (percentage) 1.760 -2.22%
483.xalancbmk (max) 16.000 -15.79%
483.xalancbmk (max) (percentage) 0.450 -19.64%
483.xalancbmk (unexplained diff vs previous) 6.000 -68.42%
483.xalancbmk (unexplained diff vs previous) (percentage) 0.170 -67.92%
Geometric Mean 3.759 -14.75%