Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/7075f9d926c93efefacb72caf17e93ae00b3703b (+ 3319 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Os-build/45/
View Options
Report
Filter

llvm-arm-master-Os test results

Run Order Start Time Duration
Current 0045 (llvmorg-17-init-16076-g7075f9d926c9) 2023-06-12T01:29:55 0:00:00
Previous 0041 (llvmorg-17-init-12757-g65f40cb455f8) 2023-05-27T03:41:43 0:00:00
Baseline 0001 (llvmorg-16-init-16061-ge7790fbed32b) 2022-12-29T19:41:07 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 27 41
Performance Improvements 42 87
Added Tests 2 2
Unchanged Tests 709 650
Total Tests 780

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
471.omnetpp 1.15% 2781.000 2813.000 - -73.82% -
447.dealII 1.03% 1744.000 1762.000 - -78.50% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) 966.67% 3.000 32.000 - -65.22% -
471.omnetpp (unexplained diff vs previous) (percentage) 936.36% 0.110 1.140 - 32.56% -
483.xalancbmk (unexplained diff vs previous) 540.00% 5.000 32.000 - -47.54% -
483.xalancbmk (unexplained diff vs previous) (percentage) 535.71% 0.140 0.890 - 36.92% -
401.bzip2 (unexplained diff vs previous) (percentage) 200.00% 0.180 0.540 - 74.19% -
401.bzip2 (unexplained diff vs previous) 200.00% 5.000 15.000 - -69.39% -
433.milc (unexplained diff vs previous) 125.00% 4.000 9.000 - -95.05% -
433.milc (unexplained diff vs previous) (percentage) 120.00% 0.350 0.770 - -42.96% -
401.bzip2 (max) 114.29% 7.000 15.000 - -68.75% -
401.bzip2 (max) (percentage) 103.70% 0.270 0.550 - 77.42% -
433.milc (max) 80.00% 5.000 9.000 - -95.11% -
433.milc (max) (percentage) 69.57% 0.460 0.780 - -43.07% -
471.omnetpp (max) 68.42% 19.000 32.000 - -83.07% -
483.xalancbmk (max) 68.42% 19.000 32.000 - -47.54% -
471.omnetpp (max) (percentage) 66.67% 0.690 1.150 - -34.66% -
483.xalancbmk (max) (percentage) 60.71% 0.560 0.900 - 38.46% -
429.mcf (unexplained diff vs previous) (percentage) 40.48% 0.420 0.590 - 5.36% -
429.mcf (unexplained diff vs previous) 40.00% 5.000 7.000 - -89.55% -
470.lbm (unexplained diff vs previous) (percentage) 39.39% 0.660 0.920 - -74.30% -
470.lbm (unexplained diff vs previous) 38.10% 21.000 29.000 - -94.16% -
470.lbm (max) (percentage) 35.82% 0.670 0.910 - -83.57% -
470.lbm (max) 33.33% 21.000 28.000 - -96.36% -
450.soplex (avg) (percentage) 1.41% 0.710 0.720 - 100.00% -
458.sjeng 1.10% 91.000 92.000 - -3.16% -
458.sjeng (avg) 1.10% 91.000 92.000 - -3.16% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -3.17% 63.000 61.000 - -99.33% -
444.namd -1.12% 717.000 709.000 - -94.75% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
450.soplex (unexplained diff vs previous) -16.67% 6.000 5.000 - -93.90% -
453.povray (max) -16.67% 6.000 5.000 - -96.21% -
450.soplex (unexplained diff vs previous) (percentage) -15.65% 1.150 0.970 - 70.18% -
445.gobmk (max) -15.38% 13.000 11.000 - -62.07% -
473.astar (max) -12.50% 16.000 14.000 - -87.72% -
482.sphinx3 (max) -12.50% 8.000 7.000 - -99.20% -
445.gobmk (max) (percentage) -12.00% 0.250 0.220 - -8.33% -
456.hmmer (max) -11.11% 9.000 8.000 - -80.00% -
462.libquantum (max) (percentage) -10.18% 1.670 1.500 - 650.00% -
444.namd (max) -10.00% 10.000 9.000 - -98.98% -
482.sphinx3 (max) (percentage) -9.89% 0.910 0.820 - -75.30% -
473.astar (max) (percentage) -8.82% 0.340 0.310 - -61.25% -
450.soplex (max) (percentage) -8.59% 3.610 3.300 - 478.95% -
429.mcf (max) -8.33% 24.000 22.000 - -67.16% -
464.h264ref (max) (percentage) -7.69% 0.260 0.240 - -38.46% -
453.povray (max) (percentage) -7.30% 1.370 1.270 - -34.20% -
464.h264ref (max) -7.14% 14.000 13.000 - -78.33% -
429.mcf (max) (percentage) -6.34% 2.050 1.920 - 242.86% -
450.soplex (max) -5.56% 18.000 17.000 - -79.27% -
444.namd (max) (percentage) -4.96% 1.410 1.340 - -79.51% -
458.sjeng (max) (percentage) -4.59% 1.090 1.040 - 160.00% -
458.sjeng (max) -4.55% 88.000 84.000 - 29.23% -
456.hmmer (max) (percentage) -3.85% 0.260 0.250 - -16.67% -
447.dealII (max) (percentage) -3.70% 0.540 0.520 - -67.09% -
429.mcf -3.70% 54.000 52.000 - -1.89% -
429.mcf (avg) -3.70% 54.000 52.000 - -1.89% -
483.xalancbmk (avg) (percentage) -3.68% 4.890 4.710 - 842.00% -
429.mcf (avg) (percentage) -3.25% 4.610 4.460 - 891.11% -
483.xalancbmk -2.87% 174.000 169.000 - 259.57% -
483.xalancbmk (avg) -2.87% 174.000 169.000 - 259.57% -
471.omnetpp (avg) (percentage) -2.65% 4.150 4.040 - 417.95% -
470.lbm -2.56% 78.000 76.000 - -76.40% -
470.lbm (avg) -2.56% 78.000 76.000 - -76.40% -
470.lbm (avg) (percentage) -2.02% 2.470 2.420 - 4.31% -
471.omnetpp -1.74% 115.000 113.000 - 36.14% -
471.omnetpp (avg) -1.74% 115.000 113.000 - 36.14% -
403.gcc (max) (percentage) -1.57% 5.080 5.000 - 1215.79% -
464.h264ref -1.43% 70.000 69.000 - 35.29% -
464.h264ref (avg) -1.43% 70.000 69.000 - 35.29% -
401.bzip2 (avg) (percentage) -1.28% 0.780 0.770 - 250.00% -

Added Tests - sample variability
403.gcc
403.gcc (avg)

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.70% 198708.000 202090.000 - -0.03% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
403.gcc (max) (percentage) 1215.79% 0.380 5.000 - -1.57% -
462.libquantum (avg) (percentage) 1040.00% 0.150 1.710 - 0.00% -
429.mcf (avg) (percentage) 891.11% 0.450 4.460 - -3.25% -
483.xalancbmk (avg) (percentage) 842.00% 0.500 4.710 - -3.68% -
473.astar (avg) (percentage) 666.00% 0.500 3.830 - 0.00% -
462.libquantum (max) (percentage) 650.00% 0.200 1.500 - -10.18% -
450.soplex (max) (percentage) 478.95% 0.570 3.300 - -8.59% -
471.omnetpp (avg) (percentage) 417.95% 0.780 4.040 - -2.65% -
445.gobmk (avg) (percentage) 411.11% 0.180 0.920 - 0.00% -
456.hmmer (avg) (percentage) 402.78% 0.360 1.810 - 0.00% -
464.h264ref (avg) (percentage) 284.85% 0.330 1.270 - 0.00% -
483.xalancbmk 259.57% 47.000 169.000 - -2.87% -
483.xalancbmk (avg) 259.57% 47.000 169.000 - -2.87% -
401.bzip2 (avg) (percentage) 250.00% 0.220 0.770 - -1.28% -
429.mcf (max) (percentage) 242.86% 0.560 1.920 - -6.34% -
403.gcc (avg) (percentage) 206.25% 0.480 1.470 - 0.00% -
458.sjeng (max) (percentage) 160.00% 0.400 1.040 - -4.59% -
473.astar 156.34% 71.000 182.000 - -0.55% -
473.astar (avg) 156.34% 71.000 182.000 - -0.55% -
445.gobmk 118.18% 22.000 48.000 - 0.00% -
445.gobmk (avg) 118.18% 22.000 48.000 - 0.00% -
450.soplex (avg) (percentage) 100.00% 0.360 0.720 - 1.41% -
458.sjeng (avg) (percentage) 94.83% 0.580 1.130 - 0.00% -
401.bzip2 (max) (percentage) 77.42% 0.310 0.550 - 103.70% -
401.bzip2 (unexplained diff vs previous) (percentage) 74.19% 0.310 0.540 - 200.00% -
450.soplex (unexplained diff vs previous) (percentage) 70.18% 0.570 0.970 - -15.65% -
483.xalancbmk (max) (percentage) 38.46% 0.650 0.900 - 60.71% -
483.xalancbmk (unexplained diff vs previous) (percentage) 36.92% 0.650 0.890 - 535.71% -
471.omnetpp 36.14% 83.000 113.000 - -1.74% -
471.omnetpp (avg) 36.14% 83.000 113.000 - -1.74% -
464.h264ref 35.29% 51.000 69.000 - -1.43% -
464.h264ref (avg) 35.29% 51.000 69.000 - -1.43% -
456.hmmer 33.33% 48.000 64.000 - 0.00% -
456.hmmer (avg) 33.33% 48.000 64.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) (percentage) 32.56% 0.860 1.140 - 936.36% -
458.sjeng (max) 29.23% 65.000 84.000 - -4.55% -
453.povray (avg) (percentage) 5.84% 1.370 1.450 - 0.00% -
429.mcf (unexplained diff vs previous) (percentage) 5.36% 0.560 0.590 - 40.48% -
470.lbm (avg) (percentage) 4.31% 2.320 2.420 - -2.02% -
482.sphinx3 (avg) (percentage) 1.08% 1.860 1.880 - 0.00% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
462.libquantum -5.57% 26756.000 25267.000 - 0.03% -
453.povray -2.71% 593097.000 577026.000 - -0.11% -
433.milc -2.48% 74971.000 73114.000 - -0.04% -
401.bzip2 -2.13% 39458.000 38617.000 - 0.00% -
447.dealII -1.59% 1777992.000 1749671.000 - -0.00% -
482.sphinx3 -1.18% 115736.000 114375.000 - -0.03% -
456.hmmer -1.04% 168072.000 166326.000 - 0.08% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.33% 9099.000 61.000 - -3.17% -
462.libquantum -99.25% 13126.000 98.000 - 0.00% -
482.sphinx3 -96.61% 26310.000 892.000 - 0.22% -
450.soplex -96.41% 14429.000 518.000 - -0.96% -
444.namd -94.75% 13495.000 709.000 - -1.12% -
453.povray -93.16% 6857.000 469.000 - 0.64% -
433.milc -91.34% 13482.000 1168.000 - 0.78% -
429.mcf -90.10% 11999.000 1188.000 - 0.59% -
401.bzip2 -82.42% 15683.000 2757.000 - 0.55% -
447.dealII -78.50% 8196.000 1762.000 - 1.03% -
470.lbm -77.34% 13882.000 3146.000 - -0.91% -
471.omnetpp -73.82% 10746.000 2813.000 - 1.15% -
456.hmmer -73.53% 13365.000 3538.000 - -0.08% -
473.astar -66.58% 14262.000 4767.000 - -0.27% -
464.h264ref -64.44% 15493.000 5510.000 - -0.20% -
483.xalancbmk -61.83% 9427.000 3598.000 - 0.90% -
445.gobmk -57.47% 12400.000 5274.000 - 0.46% -
458.sjeng -50.20% 16406.000 8170.000 - 0.47% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
482.sphinx3 (max) -99.20% 873.000 7.000 - -12.50% -
444.namd (max) -98.98% 882.000 9.000 - -10.00% -
482.sphinx3 -96.73% 489.000 16.000 - 0.00% -
482.sphinx3 (avg) -96.73% 489.000 16.000 - 0.00% -
470.lbm (max) -96.36% 769.000 28.000 - 33.33% -
444.namd -96.30% 514.000 19.000 - 0.00% -
444.namd (avg) -96.30% 514.000 19.000 - 0.00% -
453.povray (max) -96.21% 132.000 5.000 - -16.67% -
462.libquantum (max) -96.15% 26.000 1.000 - 0.00% -
433.milc (max) -95.11% 184.000 9.000 - 80.00% -
433.milc (unexplained diff vs previous) -95.05% 182.000 9.000 - 125.00% -
462.libquantum -94.74% 19.000 1.000 - 0.00% -
462.libquantum (avg) -94.74% 19.000 1.000 - 0.00% -
470.lbm (unexplained diff vs previous) -94.16% 497.000 29.000 - 38.10% -
450.soplex -94.12% 51.000 3.000 - 0.00% -
450.soplex (avg) -94.12% 51.000 3.000 - 0.00% -
450.soplex (unexplained diff vs previous) -93.90% 82.000 5.000 - -16.67% -
453.povray -93.55% 93.000 6.000 - 0.00% -
453.povray (avg) -93.55% 93.000 6.000 - 0.00% -
433.milc -93.38% 136.000 9.000 - 0.00% -
433.milc (avg) -93.38% 136.000 9.000 - 0.00% -
447.dealII (max) -93.02% 129.000 9.000 - 0.00% -
403.gcc (max) -91.18% 34.000 3.000 - 0.00% -
429.mcf (unexplained diff vs previous) -89.55% 67.000 7.000 - 40.00% -
473.astar (max) -87.72% 114.000 14.000 - -12.50% -
447.dealII -85.83% 120.000 17.000 - 0.00% -
447.dealII (avg) -85.83% 120.000 17.000 - 0.00% -
470.lbm (max) (percentage) -83.57% 5.540 0.910 - 35.82% -
471.omnetpp (max) -83.07% 189.000 32.000 - 68.42% -
456.hmmer (max) -80.00% 40.000 8.000 - -11.11% -
444.namd (max) (percentage) -79.51% 6.540 1.340 - -4.96% -
450.soplex (max) -79.27% 82.000 17.000 - -5.56% -
464.h264ref (max) -78.33% 60.000 13.000 - -7.14% -
470.lbm -76.40% 322.000 76.000 - -2.56% -
470.lbm (avg) -76.40% 322.000 76.000 - -2.56% -
482.sphinx3 (max) (percentage) -75.30% 3.320 0.820 - -9.89% -
470.lbm (unexplained diff vs previous) (percentage) -74.30% 3.580 0.920 - 39.39% -
401.bzip2 (unexplained diff vs previous) -69.39% 49.000 15.000 - 200.00% -
401.bzip2 (max) -68.75% 48.000 15.000 - 114.29% -
429.mcf (max) -67.16% 67.000 22.000 - -8.33% -
447.dealII (max) (percentage) -67.09% 1.580 0.520 - -3.70% -
471.omnetpp (unexplained diff vs previous) -65.22% 92.000 32.000 - 966.67% -
445.gobmk (max) -62.07% 29.000 11.000 - -15.38% -
473.astar (max) (percentage) -61.25% 0.800 0.310 - -8.82% -
483.xalancbmk (max) -47.54% 61.000 32.000 - 68.42% -
483.xalancbmk (unexplained diff vs previous) -47.54% 61.000 32.000 - 540.00% -
433.milc (max) (percentage) -43.07% 1.370 0.780 - 69.57% -
433.milc (unexplained diff vs previous) (percentage) -42.96% 1.350 0.770 - 120.00% -
464.h264ref (max) (percentage) -38.46% 0.390 0.240 - -7.69% -
401.bzip2 -38.24% 34.000 21.000 - 0.00% -
401.bzip2 (avg) -38.24% 34.000 21.000 - 0.00% -
471.omnetpp (max) (percentage) -34.66% 1.760 1.150 - 66.67% -
453.povray (max) (percentage) -34.20% 1.930 1.270 - -7.30% -
447.dealII (avg) (percentage) -31.29% 1.470 1.010 - 0.00% -
444.namd (avg) (percentage) -29.40% 3.810 2.690 - 0.00% -
433.milc (avg) (percentage) -19.80% 1.010 0.810 - -1.22% -
456.hmmer (max) (percentage) -16.67% 0.300 0.250 - -3.85% -
445.gobmk (max) (percentage) -8.33% 0.240 0.220 - -12.00% -
458.sjeng -3.16% 95.000 92.000 - 1.10% -
458.sjeng (avg) -3.16% 95.000 92.000 - 1.10% -
429.mcf -1.89% 53.000 52.000 - -3.70% -
429.mcf (avg) -1.89% 53.000 52.000 - -3.70% -

Added Tests - sample variability
403.gcc
403.gcc (avg)

Report Time: 0.47s
Execution (seconds)
Name Current %
401.bzip2 2757.000 -
403.gcc 61.000 -3.17%
429.mcf 1188.000 -
433.milc 1168.000 -
444.namd 709.000 -1.12%
445.gobmk 5274.000 -
447.dealII 1762.000 1.03%
450.soplex 518.000 -
453.povray 469.000 -
456.hmmer 3538.000 -
458.sjeng 8170.000 -
462.libquantum 98.000 -
464.h264ref 5510.000 -
470.lbm 3146.000 -
471.omnetpp 2813.000 1.15%
473.astar 4767.000 -
482.sphinx3 892.000 -
483.xalancbmk 3598.000 -
Geometric Mean 1448.638 -
Code Size (bytes)
Name Current %
401.bzip2 38617.000 -
403.gcc 1958481.000 -
429.mcf 7910.000 -
433.milc 73114.000 -
444.namd 175729.000 -
445.gobmk 1310296.000 -
447.dealII 1749671.000 -
450.soplex 202090.000 -
453.povray 577026.000 -
456.hmmer 166326.000 -
458.sjeng 96961.000 -
462.libquantum 25267.000 -
464.h264ref 352790.000 -
470.lbm 9734.000 -
471.omnetpp 376627.000 -
473.astar 23127.000 -
482.sphinx3 114375.000 -
483.xalancbmk 2287164.000 -
Geometric Mean 162937.150 -
sample variability
Name Current %
401.bzip2 21.000 -
401.bzip2 (avg) 21.000 -
401.bzip2 (avg) (percentage) 0.770 -1.28%
401.bzip2 (max) 15.000 114.29%
401.bzip2 (max) (percentage) 0.550 103.70%
401.bzip2 (unexplained diff vs previous) 15.000 200.00%
401.bzip2 (unexplained diff vs previous) (percentage) 0.540 200.00%
403.gcc 0.000 -
403.gcc (avg) 0.000 -
403.gcc (avg) (percentage) 1.470 -
403.gcc (max) 3.000 -
403.gcc (max) (percentage) 5.000 -1.57%
429.mcf 52.000 -3.70%
429.mcf (avg) 52.000 -3.70%
429.mcf (avg) (percentage) 4.460 -3.25%
429.mcf (max) 22.000 -8.33%
429.mcf (max) (percentage) 1.920 -6.34%
429.mcf (unexplained diff vs previous) 7.000 40.00%
429.mcf (unexplained diff vs previous) (percentage) 0.590 40.48%
433.milc 9.000 -
433.milc (avg) 9.000 -
433.milc (avg) (percentage) 0.810 -
433.milc (max) 9.000 80.00%
433.milc (max) (percentage) 0.780 69.57%
433.milc (unexplained diff vs previous) 9.000 125.00%
433.milc (unexplained diff vs previous) (percentage) 0.770 120.00%
444.namd 19.000 -
444.namd (avg) 19.000 -
444.namd (avg) (percentage) 2.690 -
444.namd (max) 9.000 -10.00%
444.namd (max) (percentage) 1.340 -4.96%
445.gobmk 48.000 -
445.gobmk (avg) 48.000 -
445.gobmk (avg) (percentage) 0.920 -
445.gobmk (max) 11.000 -15.38%
445.gobmk (max) (percentage) 0.220 -12.00%
447.dealII 17.000 -
447.dealII (avg) 17.000 -
447.dealII (avg) (percentage) 1.010 -
447.dealII (max) 9.000 -
447.dealII (max) (percentage) 0.520 -3.70%
450.soplex 3.000 -
450.soplex (avg) 3.000 -
450.soplex (avg) (percentage) 0.720 1.41%
450.soplex (max) 17.000 -5.56%
450.soplex (max) (percentage) 3.300 -8.59%
450.soplex (unexplained diff vs previous) 5.000 -16.67%
450.soplex (unexplained diff vs previous) (percentage) 0.970 -15.65%
453.povray 6.000 -
453.povray (avg) 6.000 -
453.povray (avg) (percentage) 1.450 -
453.povray (max) 5.000 -16.67%
453.povray (max) (percentage) 1.270 -7.30%
456.hmmer 64.000 -
456.hmmer (avg) 64.000 -
456.hmmer (avg) (percentage) 1.810 -
456.hmmer (max) 8.000 -11.11%
456.hmmer (max) (percentage) 0.250 -3.85%
458.sjeng 92.000 1.10%
458.sjeng (avg) 92.000 1.10%
458.sjeng (avg) (percentage) 1.130 -
458.sjeng (max) 84.000 -4.55%
458.sjeng (max) (percentage) 1.040 -4.59%
462.libquantum 1.000 -
462.libquantum (avg) 1.000 -
462.libquantum (avg) (percentage) 1.710 -
462.libquantum (max) 1.000 -
462.libquantum (max) (percentage) 1.500 -10.18%
464.h264ref 69.000 -1.43%
464.h264ref (avg) 69.000 -1.43%
464.h264ref (avg) (percentage) 1.270 -
464.h264ref (max) 13.000 -7.14%
464.h264ref (max) (percentage) 0.240 -7.69%
470.lbm 76.000 -2.56%
470.lbm (avg) 76.000 -2.56%
470.lbm (avg) (percentage) 2.420 -2.02%
470.lbm (max) 28.000 33.33%
470.lbm (max) (percentage) 0.910 35.82%
470.lbm (unexplained diff vs previous) 29.000 38.10%
470.lbm (unexplained diff vs previous) (percentage) 0.920 39.39%
471.omnetpp 113.000 -1.74%
471.omnetpp (avg) 113.000 -1.74%
471.omnetpp (avg) (percentage) 4.040 -2.65%
471.omnetpp (max) 32.000 68.42%
471.omnetpp (max) (percentage) 1.150 66.67%
471.omnetpp (unexplained diff vs previous) 32.000 966.67%
471.omnetpp (unexplained diff vs previous) (percentage) 1.140 936.36%
473.astar 182.000 -
473.astar (avg) 182.000 -
473.astar (avg) (percentage) 3.830 -
473.astar (max) 14.000 -12.50%
473.astar (max) (percentage) 0.310 -8.82%
482.sphinx3 16.000 -
482.sphinx3 (avg) 16.000 -
482.sphinx3 (avg) (percentage) 1.880 -
482.sphinx3 (max) 7.000 -12.50%
482.sphinx3 (max) (percentage) 0.820 -9.89%
483.xalancbmk 169.000 -2.87%
483.xalancbmk (avg) 169.000 -2.87%
483.xalancbmk (avg) (percentage) 4.710 -3.68%
483.xalancbmk (max) 32.000 68.42%
483.xalancbmk (max) (percentage) 0.900 60.71%
483.xalancbmk (unexplained diff vs previous) 32.000 540.00%
483.xalancbmk (unexplained diff vs previous) (percentage) 0.890 535.71%
Geometric Mean 4.819 -5.32%