Fields

hardware None
os None

Parameters

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/f8eceb45d0bbca092164efffc92f2e9d66b304a5 (+ 345 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz_LTO-build/181/
Filter

llvm-arm-master-Oz_LTO test results

Run Order Start Time Duration
Current 0181 (llvmorg-20-init-06108-gf8eceb45d0bb) 2024-09-15T00:38:25 0:00:00
Previous 0180 (llvmorg-20-init-05763-ga9e05a36dbbb) 2024-09-11T21:22:34 0:00:00
Baseline 0001 (llvmorg-16-init-17036-g4f40923103d2) 2023-01-06T17:55:45 0:00:00


Tests Summary

Status Group # # (B)
Performance Regressions 9 14
Performance Improvements 47 116
Added Tests 4 6
Unchanged Tests 708 632
Total Tests 768

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
447.dealII 1.82% 1920.000 1955.000 - -79.16% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
470.lbm (unexplained diff vs previous) 250.00% 4.000 14.000 0.000 -98.26% 0.000
470.lbm (unexplained diff vs previous) (percentage) 250.00% 0.100 0.350 0.000 -94.22% 0.000
444.namd (unexplained diff vs previous) (percentage) 108.33% 0.120 0.250 0.000 316.67% 0.000
444.namd (unexplained diff vs previous) 100.00% 1.000 2.000 0.000 -75.00% 0.000
456.hmmer (unexplained diff vs previous) 100.00% 1.000 2.000 0.000 -95.65% 0.000
464.h264ref (unexplained diff vs previous) 71.43% 7.000 12.000 0.000 0.00% 0.000
456.hmmer (unexplained diff vs previous) (percentage) 66.67% 0.030 0.050 0.000 -87.18% 0.000
464.h264ref (unexplained diff vs previous) (percentage) 66.67% 0.120 0.200 0.000 0.00% 0.000

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc -9.68% 62.000 56.000 - -99.41% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
429.mcf (unexplained diff vs previous) -73.33% 15.000 4.000 0.000 -81.82% 0.000
429.mcf (unexplained diff vs previous) (percentage) -73.28% 1.310 0.350 0.000 118.75% 0.000
471.omnetpp (unexplained diff vs previous) (percentage) -67.61% 2.130 0.690 0.000 -29.59% 0.000
471.omnetpp (unexplained diff vs previous) -67.21% 61.000 20.000 0.000 -80.58% 0.000
403.gcc -50.00% 2.000 1.000 - -98.08% -
403.gcc (avg) -50.00% 2.000 1.000 - -98.08% -
462.libquantum (unexplained diff vs previous) -50.00% 2.000 1.000 0.000 0.00% 0.000
462.libquantum (unexplained diff vs previous) (percentage) -50.00% 1.120 0.560 0.000 5500.00% 0.000
429.mcf (max) (percentage) -47.37% 2.660 1.400 - -91.42% -
429.mcf (max) -46.67% 30.000 16.000 - -99.30% -
473.astar (unexplained diff vs previous) -28.57% 14.000 10.000 0.000 -60.00% 0.000
473.astar (max) (percentage) -26.67% 0.300 0.220 - -96.72% -
473.astar (unexplained diff vs previous) (percentage) -26.67% 0.300 0.220 0.000 10.00% 0.000
473.astar (max) -23.08% 13.000 10.000 - -98.78% -
403.gcc (max) -16.67% 6.000 5.000 - -97.74% -
445.gobmk (max) -16.67% 12.000 10.000 - -97.58% -
453.povray (max) -16.67% 6.000 5.000 - -80.00% -
456.hmmer (max) -12.50% 8.000 7.000 - -99.44% -
471.omnetpp (max) (percentage) -12.50% 2.080 1.820 - -80.39% -
445.gobmk (max) (percentage) -12.00% 0.250 0.220 - -94.10% -
471.omnetpp (max) -11.86% 59.000 52.000 - -94.65% -
462.libquantum (max) (percentage) -10.11% 1.880 1.690 - -73.51% -
450.soplex (max) -10.00% 10.000 9.000 - -96.39% -
456.hmmer (max) (percentage) -9.09% 0.220 0.200 - -98.09% -
450.soplex (max) (percentage) -8.59% 1.980 1.810 - 9.70% -
401.bzip2 (max) -8.33% 12.000 11.000 - -98.19% -
447.dealII (max) (percentage) -7.19% 1.390 1.290 - -45.80% -
401.bzip2 (max) (percentage) -6.67% 0.450 0.420 - -88.89% -
458.sjeng (max) -6.67% 30.000 28.000 - -96.80% -
464.h264ref (max) (percentage) -6.45% 0.310 0.290 - -30.95% -
444.namd (max) (percentage) -6.33% 0.790 0.740 - -90.85% -
464.h264ref (max) -5.56% 18.000 17.000 - -71.67% -
458.sjeng (max) (percentage) -5.41% 0.370 0.350 - -93.19% -
453.povray (max) (percentage) -4.96% 1.210 1.150 - 219.44% -
470.lbm (max) -4.76% 21.000 20.000 - -97.36% -
447.dealII (max) -3.85% 26.000 25.000 - -88.79% -
470.lbm (max) (percentage) -3.70% 0.540 0.520 - -90.91% -
433.milc (max) (percentage) -3.17% 2.840 2.750 - 243.75% -
482.sphinx3 (max) (percentage) -3.13% 1.600 1.550 - -53.17% -
433.milc (max) -3.03% 33.000 32.000 - -70.37% -
444.namd (avg) (percentage) -2.47% 0.810 0.790 - -69.26% -
401.bzip2 (avg) (percentage) -2.27% 0.440 0.430 - -33.85% -
429.mcf (avg) (percentage) -1.53% 1.310 1.290 - -39.15% -
450.soplex (avg) (percentage) -1.50% 1.330 1.310 - 162.00% -
462.libquantum (avg) (percentage) -1.32% 1.510 1.490 - 93.51% -
456.hmmer (avg) (percentage) -1.25% 0.800 0.790 - -43.17% -

Added Tests - sample variability
401.bzip2 (unexplained diff vs previous)
401.bzip2 (unexplained diff vs previous) (percentage)
450.soplex (unexplained diff vs previous)
450.soplex (unexplained diff vs previous) (percentage)

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.89% 146765.000 149540.000 - 0.00% -
471.omnetpp 1.69% 316676.000 322018.000 - 0.00% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
462.libquantum (unexplained diff vs previous) (percentage) 5500.00% 0.010 0.560 0.000 -50.00% 0.000
403.gcc (avg) (percentage) 494.55% 0.550 3.270 - 0.00% -
403.gcc (max) (percentage) 355.36% 2.330 10.610 - -0.93% -
444.namd (unexplained diff vs previous) (percentage) 316.67% 0.060 0.250 0.000 108.33% 0.000
433.milc (max) (percentage) 243.75% 0.800 2.750 - -3.17% -
453.povray (max) (percentage) 219.44% 0.360 1.150 - -4.96% -
450.soplex (avg) (percentage) 162.00% 0.500 1.310 - -1.50% -
429.mcf (unexplained diff vs previous) (percentage) 118.75% 0.160 0.350 0.000 -73.28% 0.000
462.libquantum (avg) (percentage) 93.51% 0.770 1.490 - -1.32% -
453.povray (avg) (percentage) 13.75% 0.800 0.910 - 0.00% -
473.astar (unexplained diff vs previous) (percentage) 10.00% 0.200 0.220 0.000 -26.67% 0.000
450.soplex (max) (percentage) 9.70% 1.650 1.810 - -8.59% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
429.mcf -3.76% 7239.000 6967.000 - 0.00% -
462.libquantum -3.59% 12504.000 12055.000 - 0.00% -
453.povray -2.29% 524493.000 512480.000 - 0.00% -
433.milc -1.92% 47001.000 46097.000 - -0.02% -
444.namd -1.87% 124963.000 122622.000 - 0.00% -
458.sjeng -1.62% 80297.000 78994.000 - -0.13% -
464.h264ref -1.55% 303767.000 299045.000 - 0.02% -
470.lbm -1.14% 9338.000 9232.000 - 0.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.41% 9509.000 56.000 - -9.68% -
462.libquantum -98.75% 14409.000 180.000 - 0.56% -
450.soplex -96.43% 15112.000 540.000 - 0.00% -
482.sphinx3 -96.30% 26682.000 986.000 - 0.61% -
444.namd -93.42% 12400.000 816.000 - 0.25% -
453.povray -92.93% 6987.000 494.000 - -0.60% -
429.mcf -91.80% 13988.000 1147.000 - 0.35% -
433.milc -91.37% 13601.000 1174.000 - 0.34% -
401.bzip2 -82.46% 16137.000 2831.000 - 0.00% -
447.dealII -79.16% 9380.000 1955.000 - 1.82% -
471.omnetpp -72.45% 10477.000 2886.000 - 0.70% -
470.lbm -69.97% 13259.000 3982.000 - -0.35% -
456.hmmer -67.90% 11854.000 3805.000 - -0.05% -
473.astar -62.50% 12272.000 4602.000 - -0.22% -
464.h264ref -59.04% 14498.000 5939.000 - -0.20% -
445.gobmk -56.01% 11093.000 4880.000 - -0.22% -
458.sjeng -52.19% 17055.000 8154.000 - -0.29% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
462.libquantum (max) -99.67% 919.000 3.000 - 0.00% -
483.xalancbmk (max) (percentage) -99.61% 12.900 0.050 - 0.00% -
456.hmmer (max) -99.44% 1242.000 7.000 - -12.50% -
444.namd (max) -99.40% 1003.000 6.000 - 0.00% -
429.mcf (max) -99.30% 2281.000 16.000 - -46.67% -
473.astar (max) -98.78% 822.000 10.000 - -23.08% -
482.sphinx3 -98.49% 464.000 7.000 - 0.00% -
482.sphinx3 (avg) -98.49% 464.000 7.000 - 0.00% -
482.sphinx3 (max) -98.30% 883.000 15.000 - 0.00% -
470.lbm (unexplained diff vs previous) -98.26% 804.000 14.000 0.000 250.00% 0.000
401.bzip2 (max) -98.19% 609.000 11.000 - -8.33% -
462.libquantum -98.18% 110.000 2.000 - 0.00% -
462.libquantum (avg) -98.18% 110.000 2.000 - 0.00% -
444.namd -98.11% 318.000 6.000 - 0.00% -
444.namd (avg) -98.11% 318.000 6.000 - 0.00% -
456.hmmer (max) (percentage) -98.09% 10.480 0.200 - -9.09% -
403.gcc -98.08% 52.000 1.000 - -50.00% -
403.gcc (avg) -98.08% 52.000 1.000 - -50.00% -
403.gcc (max) -97.74% 221.000 5.000 - -16.67% -
445.gobmk (max) -97.58% 413.000 10.000 - -16.67% -
470.lbm (max) -97.36% 758.000 20.000 - -4.76% -
458.sjeng (max) -96.80% 876.000 28.000 - -6.67% -
473.astar (max) (percentage) -96.72% 6.700 0.220 - -26.67% -
450.soplex (max) -96.39% 249.000 9.000 - -10.00% -
456.hmmer (unexplained diff vs previous) -95.65% 46.000 2.000 0.000 100.00% 0.000
429.mcf -95.27% 296.000 14.000 - 0.00% -
429.mcf (avg) -95.27% 296.000 14.000 - 0.00% -
470.lbm -95.24% 441.000 21.000 - 0.00% -
470.lbm (avg) -95.24% 441.000 21.000 - 0.00% -
471.omnetpp (max) -94.65% 972.000 52.000 - -11.86% -
470.lbm (unexplained diff vs previous) (percentage) -94.22% 6.060 0.350 0.000 250.00% 0.000
445.gobmk (max) (percentage) -94.10% 3.730 0.220 - -12.00% -
458.sjeng (max) (percentage) -93.19% 5.140 0.350 - -5.41% -
433.milc -93.10% 116.000 8.000 - 0.00% -
433.milc (avg) -93.10% 116.000 8.000 - 0.00% -
453.povray -92.73% 55.000 4.000 - 0.00% -
453.povray (avg) -92.73% 55.000 4.000 - 0.00% -
429.mcf (max) (percentage) -91.42% 16.310 1.400 - -47.37% -
470.lbm (max) (percentage) -90.91% 5.720 0.520 - -3.70% -
444.namd (max) (percentage) -90.85% 8.090 0.740 - -6.33% -
450.soplex -90.67% 75.000 7.000 - 0.00% -
450.soplex (avg) -90.67% 75.000 7.000 - 0.00% -
401.bzip2 (max) (percentage) -88.89% 3.780 0.420 - -6.67% -
447.dealII (max) -88.79% 223.000 25.000 - -3.85% -
401.bzip2 -88.46% 104.000 12.000 - 0.00% -
401.bzip2 (avg) -88.46% 104.000 12.000 - 0.00% -
456.hmmer (unexplained diff vs previous) (percentage) -87.18% 0.390 0.050 0.000 66.67% 0.000
447.dealII -85.71% 98.000 14.000 - 0.00% -
447.dealII (avg) -85.71% 98.000 14.000 - 0.00% -
470.lbm (avg) (percentage) -83.48% 3.330 0.550 - 0.00% -
445.gobmk -82.43% 74.000 13.000 - 0.00% -
445.gobmk (avg) -82.43% 74.000 13.000 - 0.00% -
429.mcf (unexplained diff vs previous) -81.82% 22.000 4.000 0.000 -73.33% 0.000
456.hmmer -81.71% 164.000 30.000 - 0.00% -
456.hmmer (avg) -81.71% 164.000 30.000 - 0.00% -
458.sjeng -80.88% 136.000 26.000 - 0.00% -
458.sjeng (avg) -80.88% 136.000 26.000 - 0.00% -
471.omnetpp (unexplained diff vs previous) -80.58% 103.000 20.000 0.000 -67.21% 0.000
471.omnetpp (max) (percentage) -80.39% 9.280 1.820 - -12.50% -
453.povray (max) -80.00% 25.000 5.000 - -16.67% -
473.astar -79.46% 112.000 23.000 - 0.00% -
473.astar (avg) -79.46% 112.000 23.000 - 0.00% -
471.omnetpp -76.65% 167.000 39.000 - 0.00% -
471.omnetpp (avg) -76.65% 167.000 39.000 - 0.00% -
444.namd (unexplained diff vs previous) -75.00% 8.000 2.000 0.000 100.00% 0.000
462.libquantum (max) (percentage) -73.51% 6.380 1.690 - -10.11% -
464.h264ref (max) -71.67% 60.000 17.000 - -5.56% -
433.milc (max) -70.37% 108.000 32.000 - -3.03% -
444.namd (avg) (percentage) -69.26% 2.570 0.790 - -2.47% -
464.h264ref -69.05% 42.000 13.000 - 0.00% -
464.h264ref (avg) -69.05% 42.000 13.000 - 0.00% -
473.astar (unexplained diff vs previous) -60.00% 25.000 10.000 0.000 -28.57% 0.000
458.sjeng (avg) (percentage) -58.75% 0.800 0.330 - 0.00% -
445.gobmk (avg) (percentage) -58.21% 0.670 0.280 - 0.00% -
482.sphinx3 (avg) (percentage) -54.02% 1.740 0.800 - 0.00% -
482.sphinx3 (max) (percentage) -53.17% 3.310 1.550 - -3.13% -
447.dealII (max) (percentage) -45.80% 2.380 1.290 - -7.19% -
473.astar (avg) (percentage) -43.48% 0.920 0.520 - 0.00% -
456.hmmer (avg) (percentage) -43.17% 1.390 0.790 - -1.25% -
429.mcf (avg) (percentage) -39.15% 2.120 1.290 - -1.53% -
401.bzip2 (avg) (percentage) -33.85% 0.650 0.430 - -2.27% -
483.xalancbmk (avg) (percentage) -32.08% 2.120 1.440 - 0.00% -
464.h264ref (max) (percentage) -30.95% 0.420 0.290 - -6.45% -
471.omnetpp (unexplained diff vs previous) (percentage) -29.59% 0.980 0.690 0.000 -67.61% 0.000
447.dealII (avg) (percentage) -29.52% 1.050 0.740 - 0.00% -
464.h264ref (avg) (percentage) -24.14% 0.290 0.220 - 0.00% -
433.milc (avg) (percentage) -19.77% 0.860 0.690 - 0.00% -
471.omnetpp (avg) (percentage) -15.00% 1.600 1.360 - -0.73% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 110.53% 19.000 40.000 - 0.00% -
nb_compile_successful 89.47% 19.000 36.000 - 0.00% -
nb_execution_successful 88.89% 18.000 34.000 - 0.00% -

Added Tests - sample variability
401.bzip2 (unexplained diff vs previous)
401.bzip2 (unexplained diff vs previous) (percentage)
450.soplex (unexplained diff vs previous)
450.soplex (unexplained diff vs previous) (percentage)
464.h264ref (unexplained diff vs previous)
464.h264ref (unexplained diff vs previous) (percentage)


Report Time: 0.35s
Name Current %
401.bzip2 2831.000 -
403.gcc 56.000 -9.68%
429.mcf 1147.000 -
433.milc 1174.000 -
444.namd 816.000 -
445.gobmk 4880.000 -
447.dealII 1955.000 1.82%
450.soplex 540.000 -
453.povray 494.000 -
456.hmmer 3805.000 -
458.sjeng 8154.000 -
462.libquantum 180.000 -
464.h264ref 5939.000 -
470.lbm 3982.000 -
471.omnetpp 2886.000 -
473.astar 4602.000 -
482.sphinx3 986.000 -
Geometric Mean 1478.041 -
Name Current %
401.bzip2 33373.000 -
403.gcc 1655999.000 -
429.mcf 6967.000 -
433.milc 46097.000 -
444.namd 122622.000 -
445.gobmk 1255297.000 -
447.dealII 195558.000 -
450.soplex 149540.000 -
453.povray 512480.000 -
456.hmmer 80265.000 -
458.sjeng 78994.000 -
462.libquantum 12055.000 -
464.h264ref 299045.000 -
470.lbm 9232.000 -
471.omnetpp 322018.000 -
473.astar 15607.000 -
482.sphinx3 87564.000 -
Geometric Mean 94470.602 -
Name Current %
401.bzip2 12.000 -
401.bzip2 (avg) 12.000 -
401.bzip2 (avg) (percentage) 0.430 -2.27%
401.bzip2 (max) 11.000 -8.33%
401.bzip2 (max) (percentage) 0.420 -6.67%
401.bzip2 (unexplained diff vs previous) 0.000 -
401.bzip2 (unexplained diff vs previous) (percentage) 0.000 -
403.gcc 1.000 -50.00%
403.gcc (avg) 1.000 -50.00%
403.gcc (avg) (percentage) 3.270 -
403.gcc (max) 5.000 -16.67%
403.gcc (max) (percentage) 10.610 -
429.mcf 14.000 -
429.mcf (avg) 14.000 -
429.mcf (avg) (percentage) 1.290 -1.53%
429.mcf (max) 16.000 -46.67%
429.mcf (max) (percentage) 1.400 -47.37%
429.mcf (unexplained diff vs previous) 4.000 -73.33%
429.mcf (unexplained diff vs previous) (percentage) 0.350 -73.28%
433.milc 8.000 -
433.milc (avg) 8.000 -
433.milc (avg) (percentage) 0.690 -
433.milc (max) 32.000 -3.03%
433.milc (max) (percentage) 2.750 -3.17%
444.namd 6.000 -
444.namd (avg) 6.000 -
444.namd (avg) (percentage) 0.790 -2.47%
444.namd (max) 6.000 -
444.namd (max) (percentage) 0.740 -6.33%
444.namd (unexplained diff vs previous) 2.000 100.00%
444.namd (unexplained diff vs previous) (percentage) 0.250 108.33%
445.gobmk 13.000 -
445.gobmk (avg) 13.000 -
445.gobmk (avg) (percentage) 0.280 -
445.gobmk (max) 10.000 -16.67%
445.gobmk (max) (percentage) 0.220 -12.00%
447.dealII 14.000 -
447.dealII (avg) 14.000 -
447.dealII (avg) (percentage) 0.740 -
447.dealII (max) 25.000 -3.85%
447.dealII (max) (percentage) 1.290 -7.19%
450.soplex 7.000 -
450.soplex (avg) 7.000 -
450.soplex (avg) (percentage) 1.310 -1.50%
450.soplex (max) 9.000 -10.00%
450.soplex (max) (percentage) 1.810 -8.59%
450.soplex (unexplained diff vs previous) 0.000 -
450.soplex (unexplained diff vs previous) (percentage) 0.000 -
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.910 -
453.povray (max) 5.000 -16.67%
453.povray (max) (percentage) 1.150 -4.96%
456.hmmer 30.000 -
456.hmmer (avg) 30.000 -
456.hmmer (avg) (percentage) 0.790 -1.25%
456.hmmer (max) 7.000 -12.50%
456.hmmer (max) (percentage) 0.200 -9.09%
456.hmmer (unexplained diff vs previous) 2.000 100.00%
456.hmmer (unexplained diff vs previous) (percentage) 0.050 66.67%
458.sjeng 26.000 -
458.sjeng (avg) 26.000 -
458.sjeng (avg) (percentage) 0.330 -
458.sjeng (max) 28.000 -6.67%
458.sjeng (max) (percentage) 0.350 -5.41%
462.libquantum 2.000 -
462.libquantum (avg) 2.000 -
462.libquantum (avg) (percentage) 1.490 -1.32%
462.libquantum (max) 3.000 -
462.libquantum (max) (percentage) 1.690 -10.11%
462.libquantum (unexplained diff vs previous) 1.000 -50.00%
462.libquantum (unexplained diff vs previous) (percentage) 0.560 -50.00%
464.h264ref 13.000 -
464.h264ref (avg) 13.000 -
464.h264ref (avg) (percentage) 0.220 -
464.h264ref (max) 17.000 -5.56%
464.h264ref (max) (percentage) 0.290 -6.45%
464.h264ref (unexplained diff vs previous) 12.000 71.43%
464.h264ref (unexplained diff vs previous) (percentage) 0.200 66.67%
470.lbm 21.000 -
470.lbm (avg) 21.000 -
470.lbm (avg) (percentage) 0.550 -
470.lbm (max) 20.000 -4.76%
470.lbm (max) (percentage) 0.520 -3.70%
470.lbm (unexplained diff vs previous) 14.000 250.00%
470.lbm (unexplained diff vs previous) (percentage) 0.350 250.00%
471.omnetpp 39.000 -
471.omnetpp (avg) 39.000 -
471.omnetpp (avg) (percentage) 1.360 -
471.omnetpp (max) 52.000 -11.86%
471.omnetpp (max) (percentage) 1.820 -12.50%
471.omnetpp (unexplained diff vs previous) 20.000 -67.21%
471.omnetpp (unexplained diff vs previous) (percentage) 0.690 -67.61%
473.astar 23.000 -
473.astar (avg) 23.000 -
473.astar (avg) (percentage) 0.520 -
473.astar (max) 10.000 -23.08%
473.astar (max) (percentage) 0.220 -26.67%
473.astar (unexplained diff vs previous) 10.000 -28.57%
473.astar (unexplained diff vs previous) (percentage) 0.220 -26.67%
482.sphinx3 7.000 -
482.sphinx3 (avg) 7.000 -
482.sphinx3 (avg) (percentage) 0.800 -
482.sphinx3 (max) 15.000 -
482.sphinx3 (max) (percentage) 1.550 -3.13%
483.xalancbmk (avg) (percentage) 1.440 -
483.xalancbmk (max) (percentage) 0.050 -
Geometric Mean 2.047 -36.07%