Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/b4adb42151bbfa80be4cf6d076cbe5edf680693e (+ 850 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Os-build/141/
View Options
Report
Filter

llvm-arm-master-Os test results

Run Order Start Time Duration
Current 0141 (llvmorg-19-init-07145-gb4adb42151bb) 2024-03-27T00:06:54 0:00:00
Previous 0140 (llvmorg-19-init-06295-ge14c6fa31a30) 2024-03-20T01:52:01 0:00:00
Baseline 0001 (llvmorg-16-init-16061-ge7790fbed32b) 2022-12-29T19:41:07 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 16 27
Performance Improvements 43 100
Unchanged Tests 721 653
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
483.xalancbmk 1.66% 3547.000 3606.000 - -61.75% -
444.namd 1.10% 726.000 734.000 - -94.56% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
483.xalancbmk (unexplained diff vs previous) 391.67% 12.000 59.000 - -3.28% -
483.xalancbmk (unexplained diff vs previous) (percentage) 382.35% 0.340 1.640 - 152.31% -
471.omnetpp (unexplained diff vs previous) (percentage) 97.78% 1.350 2.670 - 210.47% -
471.omnetpp (unexplained diff vs previous) 92.11% 38.000 73.000 - -20.65% -
483.xalancbmk (max) 90.32% 31.000 59.000 - -3.28% -
471.omnetpp (max) (percentage) 89.78% 1.370 2.600 - 47.73% -
483.xalancbmk (max) (percentage) 88.64% 0.880 1.660 - 155.38% -
471.omnetpp (max) 86.84% 38.000 71.000 - -62.43% -
433.milc (unexplained diff vs previous) 20.00% 5.000 6.000 - -96.70% -
433.milc (unexplained diff vs previous) (percentage) 18.18% 0.440 0.520 - -61.48% -
444.namd 11.11% 9.000 10.000 - -98.05% -
444.namd (avg) 11.11% 9.000 10.000 - -98.05% -
483.xalancbmk 3.92% 51.000 53.000 - 12.77% -
483.xalancbmk (avg) 3.92% 51.000 53.000 - 12.77% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -6.78% 59.000 55.000 - -99.40% -
471.omnetpp -2.60% 2806.000 2733.000 - -74.57% -
462.libquantum -1.00% 100.000 99.000 - -99.25% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
450.soplex (unexplained diff vs previous) (percentage) -90.05% 1.910 0.190 - -66.67% -
450.soplex (unexplained diff vs previous) -90.00% 10.000 1.000 - -98.78% -
470.lbm (unexplained diff vs previous) -81.82% 22.000 4.000 - -99.20% -
470.lbm (unexplained diff vs previous) (percentage) -81.43% 0.700 0.130 - -96.37% -
470.lbm (max) -72.41% 58.000 16.000 - -97.92% -
470.lbm (max) (percentage) -71.74% 1.840 0.520 - -90.61% -
462.libquantum (max) -50.00% 2.000 1.000 - -96.15% -
462.libquantum (unexplained diff vs previous) -50.00% 2.000 1.000 - -93.75% -
462.libquantum (unexplained diff vs previous) (percentage) -49.50% 2.000 1.010 - 741.67% -
450.soplex (max) -20.00% 10.000 8.000 - -90.24% -
450.soplex (max) (percentage) -16.92% 1.950 1.620 - 184.21% -
403.gcc (max) -16.67% 6.000 5.000 - -85.29% -
433.milc (max) -14.29% 7.000 6.000 - -96.74% -
429.mcf (max) -13.33% 15.000 13.000 - -80.60% -
462.libquantum (max) (percentage) -12.75% 2.040 1.780 - 790.00% -
458.sjeng (max) -11.76% 17.000 15.000 - -76.92% -
429.mcf (max) (percentage) -11.19% 1.340 1.190 - 112.50% -
458.sjeng (max) (percentage) -9.52% 0.210 0.190 - -52.50% -
447.dealII (max) -6.67% 15.000 14.000 - -89.15% -
473.astar (max) -6.67% 15.000 14.000 - -87.72% -
473.astar (max) (percentage) -6.25% 0.320 0.300 - -62.50% -
444.namd (max) (percentage) -6.17% 1.620 1.520 - -76.76% -
433.milc (max) (percentage) -6.15% 0.650 0.610 - -55.47% -
456.hmmer (max) -5.88% 17.000 16.000 - -60.00% -
447.dealII (max) (percentage) -5.68% 0.880 0.830 - -47.47% -
464.h264ref (max) (percentage) -5.56% 0.540 0.510 - 30.77% -
456.hmmer (max) (percentage) -4.17% 0.480 0.460 - 53.33% -
445.gobmk (max) (percentage) -3.57% 0.280 0.270 - 12.50% -
464.h264ref (max) -3.45% 29.000 28.000 - -53.33% -
453.povray (max) (percentage) -3.09% 0.970 0.940 - -51.30% -
470.lbm -3.03% 33.000 32.000 - -90.06% -
470.lbm (avg) -3.03% 33.000 32.000 - -90.06% -
458.sjeng -2.70% 37.000 36.000 - -62.11% -
458.sjeng (avg) -2.70% 37.000 36.000 - -62.11% -
471.omnetpp -2.04% 49.000 48.000 - -42.17% -
471.omnetpp (avg) -2.04% 49.000 48.000 - -42.17% -
473.astar -1.72% 58.000 57.000 - -19.72% -
473.astar (avg) -1.72% 58.000 57.000 - -19.72% -
450.soplex (avg) (percentage) -1.57% 1.270 1.250 - 247.22% -
403.gcc (max) (percentage) -1.15% 10.450 10.330 - 2618.42% -

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
401.bzip2 2.22% 39458.000 40333.000 - -0.08% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
403.gcc (max) (percentage) 2618.42% 0.380 10.330 - -1.15% -
462.libquantum (max) (percentage) 790.00% 0.200 1.780 - -12.75% -
462.libquantum (avg) (percentage) 780.00% 0.150 1.320 - -0.75% -
462.libquantum (unexplained diff vs previous) (percentage) 741.67% 0.120 1.010 - -49.50% -
403.gcc (avg) (percentage) 689.58% 0.480 3.790 - 0.00% -
429.mcf (avg) (percentage) 366.67% 0.450 2.100 - 0.00% -
450.soplex (avg) (percentage) 247.22% 0.360 1.250 - -1.57% -
471.omnetpp (unexplained diff vs previous) (percentage) 210.47% 0.860 2.670 - 97.78% -
483.xalancbmk (avg) (percentage) 194.00% 0.500 1.470 - 0.68% -
450.soplex (max) (percentage) 184.21% 0.570 1.620 - -16.92% -
401.bzip2 (avg) (percentage) 163.64% 0.220 0.580 - 0.00% -
483.xalancbmk (max) (percentage) 155.38% 0.650 1.660 - 88.64% -
483.xalancbmk (unexplained diff vs previous) (percentage) 152.31% 0.650 1.640 - 382.35% -
401.bzip2 (max) (percentage) 145.16% 0.310 0.760 - 0.00% -
445.gobmk (avg) (percentage) 138.89% 0.180 0.430 - 0.00% -
473.astar (avg) (percentage) 138.00% 0.500 1.190 - 0.00% -
471.omnetpp (avg) (percentage) 126.92% 0.780 1.770 - 0.57% -
429.mcf (max) (percentage) 112.50% 0.560 1.190 - -11.19% -
456.hmmer (avg) (percentage) 83.33% 0.360 0.660 - 0.00% -
464.h264ref (avg) (percentage) 66.67% 0.330 0.550 - 0.00% -
456.hmmer (max) (percentage) 53.33% 0.300 0.460 - -4.17% -
471.omnetpp (max) (percentage) 47.73% 1.760 2.600 - 89.78% -
464.h264ref (max) (percentage) 30.77% 0.390 0.510 - -5.56% -
483.xalancbmk 12.77% 47.000 53.000 - 3.92% -
483.xalancbmk (avg) 12.77% 47.000 53.000 - 3.92% -
445.gobmk (max) (percentage) 12.50% 0.240 0.270 - -3.57% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -10.47% 26756.000 23955.000 - -0.03% -
444.namd -3.21% 175160.000 169537.000 - -0.03% -
447.dealII -2.86% 1777992.000 1727135.000 - 0.11% -
453.povray -2.80% 593097.000 576486.000 - -0.00% -
433.milc -2.77% 74971.000 72898.000 - -0.05% -
464.h264ref -1.42% 354618.000 349590.000 - -0.09% -
482.sphinx3 -1.38% 115736.000 114135.000 - -0.02% -
429.mcf -1.02% 7915.000 7834.000 - -0.10% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.40% 9099.000 55.000 - -6.78% -
462.libquantum -99.25% 13126.000 99.000 - -1.00% -
482.sphinx3 -96.66% 26310.000 878.000 - -0.34% -
450.soplex -96.38% 14429.000 523.000 - -0.19% -
444.namd -94.56% 13495.000 734.000 - 1.10% -
453.povray -93.22% 6857.000 465.000 - -0.64% -
433.milc -91.52% 13482.000 1143.000 - -0.52% -
429.mcf -90.34% 11999.000 1159.000 - 0.43% -
401.bzip2 -82.85% 15683.000 2690.000 - -0.85% -
447.dealII -78.29% 8196.000 1779.000 - -0.67% -
470.lbm -77.18% 13882.000 3168.000 - 0.13% -
471.omnetpp -74.57% 10746.000 2733.000 - -2.60% -
456.hmmer -73.42% 13365.000 3553.000 - -0.56% -
473.astar -65.97% 14262.000 4853.000 - -0.96% -
464.h264ref -64.07% 15493.000 5566.000 - 0.34% -
483.xalancbmk -61.75% 9427.000 3606.000 - 1.66% -
445.gobmk -57.67% 12400.000 5249.000 - 0.73% -
458.sjeng -49.90% 16406.000 8220.000 - -0.25% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
470.lbm (unexplained diff vs previous) -99.20% 497.000 4.000 - -81.82% -
450.soplex (unexplained diff vs previous) -98.78% 82.000 1.000 - -90.00% -
444.namd (max) -98.75% 882.000 11.000 - 0.00% -
482.sphinx3 (max) -98.17% 873.000 16.000 - 0.00% -
482.sphinx3 -98.16% 489.000 9.000 - 0.00% -
482.sphinx3 (avg) -98.16% 489.000 9.000 - 0.00% -
444.namd -98.05% 514.000 10.000 - 11.11% -
444.namd (avg) -98.05% 514.000 10.000 - 11.11% -
470.lbm (max) -97.92% 769.000 16.000 - -72.41% -
453.povray (max) -96.97% 132.000 4.000 - 0.00% -
433.milc (max) -96.74% 184.000 6.000 - -14.29% -
433.milc (unexplained diff vs previous) -96.70% 182.000 6.000 - 20.00% -
470.lbm (unexplained diff vs previous) (percentage) -96.37% 3.580 0.130 - -81.43% -
462.libquantum (max) -96.15% 26.000 1.000 - -50.00% -
403.gcc -95.35% 43.000 2.000 - 0.00% -
403.gcc (avg) -95.35% 43.000 2.000 - 0.00% -
433.milc -94.85% 136.000 7.000 - 0.00% -
433.milc (avg) -94.85% 136.000 7.000 - 0.00% -
462.libquantum -94.74% 19.000 1.000 - 0.00% -
462.libquantum (avg) -94.74% 19.000 1.000 - 0.00% -
453.povray -94.62% 93.000 5.000 - 0.00% -
453.povray (avg) -94.62% 93.000 5.000 - 0.00% -
462.libquantum (unexplained diff vs previous) -93.75% 16.000 1.000 - -50.00% -
447.dealII -90.83% 120.000 11.000 - 0.00% -
447.dealII (avg) -90.83% 120.000 11.000 - 0.00% -
470.lbm (max) (percentage) -90.61% 5.540 0.520 - -71.74% -
450.soplex (max) -90.24% 82.000 8.000 - -20.00% -
470.lbm -90.06% 322.000 32.000 - -3.03% -
470.lbm (avg) -90.06% 322.000 32.000 - -3.03% -
447.dealII (max) -89.15% 129.000 14.000 - -6.67% -
450.soplex -88.24% 51.000 6.000 - 0.00% -
450.soplex (avg) -88.24% 51.000 6.000 - 0.00% -
473.astar (max) -87.72% 114.000 14.000 - -6.67% -
403.gcc (max) -85.29% 34.000 5.000 - -16.67% -
429.mcf (max) -80.60% 67.000 13.000 - -13.33% -
458.sjeng (max) -76.92% 65.000 15.000 - -11.76% -
444.namd (max) (percentage) -76.76% 6.540 1.520 - -6.17% -
450.soplex (unexplained diff vs previous) (percentage) -66.67% 0.570 0.190 - -90.05% -
444.namd (avg) (percentage) -64.04% 3.810 1.370 - 0.00% -
473.astar (max) (percentage) -62.50% 0.800 0.300 - -6.25% -
471.omnetpp (max) -62.43% 189.000 71.000 - 86.84% -
458.sjeng -62.11% 95.000 36.000 - -2.70% -
458.sjeng (avg) -62.11% 95.000 36.000 - -2.70% -
433.milc (unexplained diff vs previous) (percentage) -61.48% 1.350 0.520 - 18.18% -
456.hmmer (max) -60.00% 40.000 16.000 - -5.88% -
401.bzip2 (max) -58.33% 48.000 20.000 - 0.00% -
447.dealII (avg) (percentage) -57.14% 1.470 0.630 - 0.00% -
401.bzip2 -55.88% 34.000 15.000 - 0.00% -
401.bzip2 (avg) -55.88% 34.000 15.000 - 0.00% -
433.milc (max) (percentage) -55.47% 1.370 0.610 - -6.15% -
470.lbm (avg) (percentage) -55.17% 2.320 1.040 - -0.95% -
429.mcf -54.72% 53.000 24.000 - 0.00% -
429.mcf (avg) -54.72% 53.000 24.000 - 0.00% -
464.h264ref (max) -53.33% 60.000 28.000 - -3.45% -
458.sjeng (max) (percentage) -52.50% 0.400 0.190 - -9.52% -
456.hmmer -52.08% 48.000 23.000 - 0.00% -
456.hmmer (avg) -52.08% 48.000 23.000 - 0.00% -
445.gobmk (max) -51.72% 29.000 14.000 - 0.00% -
453.povray (max) (percentage) -51.30% 1.930 0.940 - -3.09% -
447.dealII (max) (percentage) -47.47% 1.580 0.830 - -5.68% -
482.sphinx3 (avg) (percentage) -44.09% 1.860 1.040 - 0.00% -
482.sphinx3 (max) (percentage) -42.47% 3.320 1.910 - 0.00% -
471.omnetpp -42.17% 83.000 48.000 - -2.04% -
471.omnetpp (avg) -42.17% 83.000 48.000 - -2.04% -
464.h264ref -41.18% 51.000 30.000 - 0.00% -
464.h264ref (avg) -41.18% 51.000 30.000 - 0.00% -
433.milc (avg) (percentage) -36.63% 1.010 0.640 - 0.00% -
458.sjeng (avg) (percentage) -22.41% 0.580 0.450 - 0.00% -
471.omnetpp (unexplained diff vs previous) -20.65% 92.000 73.000 - 92.11% -
473.astar -19.72% 71.000 57.000 - -1.72% -
473.astar (avg) -19.72% 71.000 57.000 - -1.72% -
453.povray (avg) (percentage) -16.79% 1.370 1.140 - 0.00% -
483.xalancbmk (max) -3.28% 61.000 59.000 - 90.32% -
483.xalancbmk (unexplained diff vs previous) -3.28% 61.000 59.000 - 391.67% -

Report Time: 0.35s
Execution (seconds)
Name Current %
401.bzip2 2690.000 -
403.gcc 55.000 -6.78%
429.mcf 1159.000 -
433.milc 1143.000 -
444.namd 734.000 1.10%
445.gobmk 5249.000 -
447.dealII 1779.000 -
450.soplex 523.000 -
453.povray 465.000 -
456.hmmer 3553.000 -
458.sjeng 8220.000 -
462.libquantum 99.000 -1.00%
464.h264ref 5566.000 -
470.lbm 3168.000 -
471.omnetpp 2733.000 -2.60%
473.astar 4853.000 -
482.sphinx3 878.000 -
483.xalancbmk 3606.000 1.66%
Geometric Mean 1438.937 -
Code Size (bytes)
Name Current %
401.bzip2 40333.000 -
403.gcc 1954977.000 -
429.mcf 7834.000 -
433.milc 72898.000 -
444.namd 169537.000 -
445.gobmk 1308216.000 -
447.dealII 1727135.000 -
450.soplex 198335.000 -
453.povray 576486.000 -
456.hmmer 166422.000 -
458.sjeng 96893.000 -
462.libquantum 23955.000 -
464.h264ref 349590.000 -
470.lbm 9702.000 -
471.omnetpp 376219.000 -
473.astar 23315.000 -
482.sphinx3 114135.000 -
483.xalancbmk 2284952.000 -
Geometric Mean 162007.898 -
sample variability
Name Current %
401.bzip2 15.000 -
401.bzip2 (avg) 15.000 -
401.bzip2 (avg) (percentage) 0.580 -
401.bzip2 (max) 20.000 -
401.bzip2 (max) (percentage) 0.760 -
403.gcc 2.000 -
403.gcc (avg) 2.000 -
403.gcc (avg) (percentage) 3.790 -
403.gcc (max) 5.000 -16.67%
403.gcc (max) (percentage) 10.330 -1.15%
429.mcf 24.000 -
429.mcf (avg) 24.000 -
429.mcf (avg) (percentage) 2.100 -
429.mcf (max) 13.000 -13.33%
429.mcf (max) (percentage) 1.190 -11.19%
433.milc 7.000 -
433.milc (avg) 7.000 -
433.milc (avg) (percentage) 0.640 -
433.milc (max) 6.000 -14.29%
433.milc (max) (percentage) 0.610 -6.15%
433.milc (unexplained diff vs previous) 6.000 20.00%
433.milc (unexplained diff vs previous) (percentage) 0.520 18.18%
444.namd 10.000 11.11%
444.namd (avg) 10.000 11.11%
444.namd (avg) (percentage) 1.370 -
444.namd (max) 11.000 -
444.namd (max) (percentage) 1.520 -6.17%
445.gobmk 22.000 -
445.gobmk (avg) 22.000 -
445.gobmk (avg) (percentage) 0.430 -
445.gobmk (max) 14.000 -
445.gobmk (max) (percentage) 0.270 -3.57%
447.dealII 11.000 -
447.dealII (avg) 11.000 -
447.dealII (avg) (percentage) 0.630 -
447.dealII (max) 14.000 -6.67%
447.dealII (max) (percentage) 0.830 -5.68%
450.soplex 6.000 -
450.soplex (avg) 6.000 -
450.soplex (avg) (percentage) 1.250 -1.57%
450.soplex (max) 8.000 -20.00%
450.soplex (max) (percentage) 1.620 -16.92%
450.soplex (unexplained diff vs previous) 1.000 -90.00%
450.soplex (unexplained diff vs previous) (percentage) 0.190 -90.05%
453.povray 5.000 -
453.povray (avg) 5.000 -
453.povray (avg) (percentage) 1.140 -
453.povray (max) 4.000 -
453.povray (max) (percentage) 0.940 -3.09%
456.hmmer 23.000 -
456.hmmer (avg) 23.000 -
456.hmmer (avg) (percentage) 0.660 -
456.hmmer (max) 16.000 -5.88%
456.hmmer (max) (percentage) 0.460 -4.17%
458.sjeng 36.000 -2.70%
458.sjeng (avg) 36.000 -2.70%
458.sjeng (avg) (percentage) 0.450 -
458.sjeng (max) 15.000 -11.76%
458.sjeng (max) (percentage) 0.190 -9.52%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.320 -
462.libquantum (max) 1.000 -50.00%
462.libquantum (max) (percentage) 1.780 -12.75%
462.libquantum (unexplained diff vs previous) 1.000 -50.00%
462.libquantum (unexplained diff vs previous) (percentage) 1.010 -49.50%
464.h264ref 30.000 -
464.h264ref (avg) 30.000 -
464.h264ref (avg) (percentage) 0.550 -
464.h264ref (max) 28.000 -3.45%
464.h264ref (max) (percentage) 0.510 -5.56%
470.lbm 32.000 -3.03%
470.lbm (avg) 32.000 -3.03%
470.lbm (avg) (percentage) 1.040 -
470.lbm (max) 16.000 -72.41%
470.lbm (max) (percentage) 0.520 -71.74%
470.lbm (unexplained diff vs previous) 4.000 -81.82%
470.lbm (unexplained diff vs previous) (percentage) 0.130 -81.43%
471.omnetpp 48.000 -2.04%
471.omnetpp (avg) 48.000 -2.04%
471.omnetpp (avg) (percentage) 1.770 -
471.omnetpp (max) 71.000 86.84%
471.omnetpp (max) (percentage) 2.600 89.78%
471.omnetpp (unexplained diff vs previous) 73.000 92.11%
471.omnetpp (unexplained diff vs previous) (percentage) 2.670 97.78%
473.astar 57.000 -1.72%
473.astar (avg) 57.000 -1.72%
473.astar (avg) (percentage) 1.190 -
473.astar (max) 14.000 -6.67%
473.astar (max) (percentage) 0.300 -6.25%
482.sphinx3 9.000 -
482.sphinx3 (avg) 9.000 -
482.sphinx3 (avg) (percentage) 1.040 -
482.sphinx3 (max) 16.000 -
482.sphinx3 (max) (percentage) 1.910 -
483.xalancbmk 53.000 3.92%
483.xalancbmk (avg) 53.000 3.92%
483.xalancbmk (avg) (percentage) 1.470 -
483.xalancbmk (max) 59.000 90.32%
483.xalancbmk (max) (percentage) 1.660 88.64%
483.xalancbmk (unexplained diff vs previous) 59.000 391.67%
483.xalancbmk (unexplained diff vs previous) (percentage) 1.640 382.35%
Geometric Mean 4.260 -6.80%