Runs:

Compare To:

Machine Info

Fields

hardware None
os None

Parameters

Run Info

Fields

NameValue

Parameters

NameValue
git_llvm https://github.com/llvm/llvm-project/commit/f1e78f776908f2bc1759eae25381f576f62728a2 (+ 3535 commits)
test_url https://ci.linaro.org/job/tcwg_bmk-code_size-spec2k6--llvm-arm-master-Oz_LTO-build/150/
View Options
Report
Filter

llvm-arm-master-Oz_LTO test results

Run Order Start Time Duration
Current 0150 (llvmorg-19-init-13479-gf1e78f776908) 2024-05-18T10:47:25 0:00:00
Previous 0149 (no-change) 2024-04-29T13:10:25 0:00:00
Baseline 0001 (llvmorg-16-init-17036-g4f40923103d2) 2023-01-06T17:55:45 0:00:00

Tests Summary

Status Group # # (B)
Performance Regressions 15 16
Performance Improvements 38 112
Unchanged Tests 715 640
Total Tests 768

Run-Over-Run Changes Detail

Performance Regressions - Execution Δ Previous Current σ Δ (B) σ (B)
403.gcc 7.14% 56.000 60.000 - -99.37% -
471.omnetpp 2.83% 2829.000 2909.000 - -72.23% -
482.sphinx3 1.22% 983.000 995.000 - -96.27% -

Performance Regressions - sample variability Δ Previous Current σ Δ (B) σ (B)
471.omnetpp (unexplained diff vs previous) 344.44% 18.000 80.000 0.000 -22.33% 0.000
471.omnetpp (unexplained diff vs previous) (percentage) 329.69% 0.640 2.750 0.000 180.61% 0.000
471.omnetpp (max) 290.48% 21.000 82.000 - -91.56% -
471.omnetpp (max) (percentage) 267.53% 0.770 2.830 - -69.50% -
433.milc (unexplained diff vs previous) 233.33% 3.000 10.000 0.000 -23.08% 0.000
433.milc (unexplained diff vs previous) (percentage) 226.92% 0.260 0.850 0.000 750.00% 0.000
429.mcf (unexplained diff vs previous) (percentage) 17.31% 0.520 0.610 0.000 281.25% 0.000
429.mcf (unexplained diff vs previous) 16.67% 6.000 7.000 0.000 -68.18% 0.000
471.omnetpp 2.17% 46.000 47.000 - -71.86% -
471.omnetpp (avg) 2.17% 46.000 47.000 - -71.86% -
433.milc (avg) (percentage) 1.59% 0.630 0.640 - -25.58% -
471.omnetpp (avg) (percentage) 1.23% 1.630 1.650 - 3.12% -

Performance Improvements - Code Size Δ Previous Current σ Δ (B) σ (B)
401.bzip2 -1.36% 33549.000 33093.000 - -1.80% -

Performance Improvements - Execution Δ Previous Current σ Δ (B) σ (B)
447.dealII -1.23% 1950.000 1926.000 - -79.47% -

Performance Improvements - sample variability Δ Previous Current σ Δ (B) σ (B)
462.libquantum (unexplained diff vs previous) -80.00% 5.000 1.000 0.000 0.00% 0.000
470.lbm (unexplained diff vs previous) (percentage) -80.00% 0.100 0.020 0.000 -99.67% 0.000
462.libquantum (unexplained diff vs previous) (percentage) -79.93% 2.690 0.540 0.000 5300.00% 0.000
470.lbm (unexplained diff vs previous) -75.00% 4.000 1.000 0.000 -99.88% 0.000
456.hmmer (max) -20.00% 25.000 20.000 - -98.39% -
462.libquantum (max) -16.67% 6.000 5.000 - -99.46% -
456.hmmer (max) (percentage) -16.67% 0.660 0.550 - -94.75% -
456.hmmer (unexplained diff vs previous) -16.00% 25.000 21.000 0.000 -54.35% 0.000
456.hmmer (unexplained diff vs previous) (percentage) -15.38% 0.650 0.550 0.000 41.03% 0.000
401.bzip2 (max) -13.04% 23.000 20.000 - -96.72% -
464.h264ref (max) -11.11% 18.000 16.000 - -73.33% -
450.soplex (max) (percentage) -10.26% 3.120 2.800 - 69.70% -
429.mcf (max) -10.00% 10.000 9.000 - -99.61% -
462.libquantum (max) (percentage) -9.94% 3.520 3.170 - -50.31% -
464.h264ref (max) (percentage) -9.68% 0.310 0.280 - -33.33% -
445.gobmk (max) (percentage) -8.82% 0.340 0.310 - -91.69% -
401.bzip2 (max) (percentage) -8.64% 0.810 0.740 - -80.42% -
447.dealII (max) -8.33% 24.000 22.000 - -90.13% -
433.milc (max) -6.67% 15.000 14.000 - -87.04% -
453.povray (max) (percentage) -6.31% 1.110 1.040 - 188.89% -
445.gobmk (max) -6.25% 16.000 15.000 - -96.37% -
450.soplex (max) -6.25% 16.000 15.000 - -93.98% -
433.milc (max) (percentage) -5.93% 1.350 1.270 - 58.75% -
447.dealII (max) (percentage) -4.80% 1.250 1.190 - -50.00% -
444.namd (max) (percentage) -4.65% 0.860 0.820 - -89.86% -
482.sphinx3 (max) (percentage) -4.38% 1.600 1.530 - -53.78% -
458.sjeng (max) -4.35% 23.000 22.000 - -97.49% -
473.astar (max) -3.70% 27.000 26.000 - -96.84% -
470.lbm -3.45% 29.000 28.000 - -93.65% -
470.lbm (avg) -3.45% 29.000 28.000 - -93.65% -
429.mcf (max) (percentage) -3.41% 0.880 0.850 - -94.79% -
473.astar (max) (percentage) -3.39% 0.590 0.570 - -91.49% -
470.lbm (avg) (percentage) -2.74% 0.730 0.710 - -78.68% -
403.gcc (max) (percentage) -1.54% 6.500 6.400 - 174.68% -
429.mcf (avg) (percentage) -1.31% 1.530 1.510 - -28.77% -
462.libquantum (avg) (percentage) -1.24% 1.610 1.590 - 106.49% -

Run-Over-Baseline Changes Detail

Performance Regressions - Code Size Δ (B) Baseline Current σ (B) Δ σ
450.soplex 1.87% 146765.000 149516.000 - -0.06% -
471.omnetpp 1.81% 316676.000 322402.000 - 0.42% -

Performance Regressions - sample variability Δ (B) Baseline Current σ (B) Δ σ
462.libquantum (unexplained diff vs previous) (percentage) 5300.00% 0.010 0.540 0.000 -79.93% 0.000
433.milc (unexplained diff vs previous) (percentage) 750.00% 0.100 0.850 0.000 226.92% 0.000
403.gcc (avg) (percentage) 387.27% 0.550 2.680 - 0.00% -
429.mcf (unexplained diff vs previous) (percentage) 281.25% 0.160 0.610 0.000 17.31% 0.000
453.povray (max) (percentage) 188.89% 0.360 1.040 - -6.31% -
471.omnetpp (unexplained diff vs previous) (percentage) 180.61% 0.980 2.750 0.000 329.69% 0.000
403.gcc (max) (percentage) 174.68% 2.330 6.400 - -1.54% -
450.soplex (avg) (percentage) 162.00% 0.500 1.310 - 0.00% -
462.libquantum (avg) (percentage) 106.49% 0.770 1.590 - -1.24% -
450.soplex (max) (percentage) 69.70% 1.650 2.800 - -10.26% -
433.milc (max) (percentage) 58.75% 0.800 1.270 - -5.93% -
456.hmmer (unexplained diff vs previous) (percentage) 41.03% 0.390 0.550 0.000 -15.38% 0.000
453.povray (avg) (percentage) 13.75% 0.800 0.910 - 0.00% -
471.omnetpp (avg) (percentage) 3.12% 1.600 1.650 - 1.23% -

Performance Improvements - Code Size Δ (B) Baseline Current σ (B) Δ σ
429.mcf -4.03% 7239.000 6947.000 - 0.00% -
462.libquantum -3.05% 12504.000 12123.000 - 0.00% -
453.povray -2.14% 524493.000 513268.000 - 0.04% -
401.bzip2 -1.80% 33698.000 33093.000 - -1.36% -
433.milc -1.73% 47001.000 46189.000 - 0.00% -
464.h264ref -1.58% 303767.000 298965.000 - 0.30% -
444.namd -1.51% 124963.000 123078.000 - 0.10% -
458.sjeng -1.27% 80297.000 79274.000 - -0.01% -
470.lbm -1.26% 9338.000 9220.000 - 0.00% -

Performance Improvements - Execution Δ (B) Baseline Current σ (B) Δ σ
403.gcc -99.37% 9509.000 60.000 - 7.14% -
462.libquantum -98.72% 14409.000 185.000 - -0.54% -
450.soplex -96.41% 15112.000 543.000 - 0.18% -
482.sphinx3 -96.27% 26682.000 995.000 - 1.22% -
444.namd -93.51% 12400.000 805.000 - 0.50% -
453.povray -92.90% 6987.000 496.000 - 0.61% -
429.mcf -91.86% 13988.000 1139.000 - -0.61% -
433.milc -91.34% 13601.000 1178.000 - 0.86% -
401.bzip2 -82.43% 16137.000 2836.000 - -0.74% -
447.dealII -79.47% 9380.000 1926.000 - -1.23% -
471.omnetpp -72.23% 10477.000 2909.000 - 2.83% -
470.lbm -69.77% 13259.000 4008.000 - 0.02% -
456.hmmer -67.87% 11854.000 3809.000 - -0.55% -
473.astar -62.06% 12272.000 4656.000 - -0.26% -
464.h264ref -59.37% 14498.000 5890.000 - 0.49% -
445.gobmk -55.95% 11093.000 4886.000 - 0.45% -
458.sjeng -51.89% 17055.000 8206.000 - -0.42% -

Performance Improvements - sample variability Δ (B) Baseline Current σ (B) Δ σ
470.lbm (unexplained diff vs previous) -99.88% 804.000 1.000 0.000 -75.00% 0.000
470.lbm (unexplained diff vs previous) (percentage) -99.67% 6.060 0.020 0.000 -80.00% 0.000
483.xalancbmk (max) (percentage) -99.61% 12.900 0.050 - 0.00% -
429.mcf (max) -99.61% 2281.000 9.000 - -10.00% -
462.libquantum (max) -99.46% 919.000 5.000 - -16.67% -
444.namd (max) -99.40% 1003.000 6.000 - 0.00% -
403.gcc (max) -98.64% 221.000 3.000 - 0.00% -
456.hmmer (max) -98.39% 1242.000 20.000 - -20.00% -
482.sphinx3 (max) -98.30% 883.000 15.000 - 0.00% -
482.sphinx3 -98.28% 464.000 8.000 - 0.00% -
482.sphinx3 (avg) -98.28% 464.000 8.000 - 0.00% -
462.libquantum -98.18% 110.000 2.000 - 0.00% -
462.libquantum (avg) -98.18% 110.000 2.000 - 0.00% -
403.gcc -98.08% 52.000 1.000 - 0.00% -
403.gcc (avg) -98.08% 52.000 1.000 - 0.00% -
470.lbm (max) -97.89% 758.000 16.000 - 0.00% -
444.namd -97.80% 318.000 7.000 - 0.00% -
444.namd (avg) -97.80% 318.000 7.000 - 0.00% -
458.sjeng (max) -97.49% 876.000 22.000 - -4.35% -
473.astar (max) -96.84% 822.000 26.000 - -3.70% -
401.bzip2 (max) -96.72% 609.000 20.000 - -13.04% -
445.gobmk (max) -96.37% 413.000 15.000 - -6.25% -
429.mcf (max) (percentage) -94.79% 16.310 0.850 - -3.41% -
456.hmmer (max) (percentage) -94.75% 10.480 0.550 - -16.67% -
458.sjeng (max) (percentage) -94.55% 5.140 0.280 - -3.45% -
429.mcf -94.26% 296.000 17.000 - 0.00% -
429.mcf (avg) -94.26% 296.000 17.000 - 0.00% -
450.soplex (max) -93.98% 249.000 15.000 - -6.25% -
433.milc -93.97% 116.000 7.000 - 0.00% -
433.milc (avg) -93.97% 116.000 7.000 - 0.00% -
470.lbm -93.65% 441.000 28.000 - -3.45% -
470.lbm (avg) -93.65% 441.000 28.000 - -3.45% -
470.lbm (max) (percentage) -93.01% 5.720 0.400 - -2.44% -
453.povray -92.73% 55.000 4.000 - 0.00% -
453.povray (avg) -92.73% 55.000 4.000 - 0.00% -
445.gobmk (max) (percentage) -91.69% 3.730 0.310 - -8.82% -
471.omnetpp (max) -91.56% 972.000 82.000 - 290.48% -
473.astar (max) (percentage) -91.49% 6.700 0.570 - -3.39% -
450.soplex -90.67% 75.000 7.000 - 0.00% -
450.soplex (avg) -90.67% 75.000 7.000 - 0.00% -
447.dealII (max) -90.13% 223.000 22.000 - -8.33% -
444.namd (max) (percentage) -89.86% 8.090 0.820 - -4.65% -
447.dealII -87.76% 98.000 12.000 - 0.00% -
447.dealII (avg) -87.76% 98.000 12.000 - 0.00% -
401.bzip2 -87.50% 104.000 13.000 - 0.00% -
401.bzip2 (avg) -87.50% 104.000 13.000 - 0.00% -
433.milc (max) -87.04% 108.000 14.000 - -6.67% -
401.bzip2 (max) (percentage) -80.42% 3.780 0.740 - -8.64% -
453.povray (max) -80.00% 25.000 5.000 - 0.00% -
470.lbm (avg) (percentage) -78.68% 3.330 0.710 - -2.74% -
445.gobmk -78.38% 74.000 16.000 - 0.00% -
445.gobmk (avg) -78.38% 74.000 16.000 - 0.00% -
458.sjeng -76.47% 136.000 32.000 - 0.00% -
458.sjeng (avg) -76.47% 136.000 32.000 - 0.00% -
456.hmmer -75.00% 164.000 41.000 - 0.00% -
456.hmmer (avg) -75.00% 164.000 41.000 - 0.00% -
473.astar -74.11% 112.000 29.000 - 0.00% -
473.astar (avg) -74.11% 112.000 29.000 - 0.00% -
464.h264ref (max) -73.33% 60.000 16.000 - -11.11% -
471.omnetpp -71.86% 167.000 47.000 - 2.17% -
471.omnetpp (avg) -71.86% 167.000 47.000 - 2.17% -
471.omnetpp (max) (percentage) -69.50% 9.280 2.830 - 267.53% -
429.mcf (unexplained diff vs previous) -68.18% 22.000 7.000 0.000 16.67% 0.000
444.namd (avg) (percentage) -62.65% 2.570 0.960 - 0.00% -
464.h264ref -61.90% 42.000 16.000 - 0.00% -
464.h264ref (avg) -61.90% 42.000 16.000 - 0.00% -
456.hmmer (unexplained diff vs previous) -54.35% 46.000 21.000 0.000 -16.00% 0.000
482.sphinx3 (max) (percentage) -53.78% 3.310 1.530 - -4.38% -
482.sphinx3 (avg) (percentage) -52.87% 1.740 0.820 - 0.00% -
458.sjeng (avg) (percentage) -51.25% 0.800 0.390 - 0.00% -
445.gobmk (avg) (percentage) -50.75% 0.670 0.330 - 0.00% -
462.libquantum (max) (percentage) -50.31% 6.380 3.170 - -9.94% -
447.dealII (max) (percentage) -50.00% 2.380 1.190 - -4.80% -
447.dealII (avg) (percentage) -39.05% 1.050 0.640 - 0.00% -
464.h264ref (max) (percentage) -33.33% 0.420 0.280 - -9.68% -
483.xalancbmk (avg) (percentage) -32.08% 2.120 1.440 - 0.00% -
473.astar (avg) (percentage) -31.52% 0.920 0.630 - 0.00% -
429.mcf (avg) (percentage) -28.77% 2.120 1.510 - -1.31% -
401.bzip2 (avg) (percentage) -26.15% 0.650 0.480 - 0.00% -
433.milc (avg) (percentage) -25.58% 0.860 0.640 - 1.59% -
433.milc (unexplained diff vs previous) -23.08% 13.000 10.000 0.000 233.33% 0.000
471.omnetpp (unexplained diff vs previous) -22.33% 103.000 80.000 0.000 344.44% 0.000
456.hmmer (avg) (percentage) -22.30% 1.390 1.080 - -0.92% -

Performance Improvements - Score Δ (B) Baseline Current σ (B) Δ σ
nb_total_tests 110.53% 19.000 40.000 - 0.00% -
nb_compile_successful 89.47% 19.000 36.000 - 0.00% -
nb_execution_successful 88.89% 18.000 34.000 - 0.00% -

Report Time: 0.33s
Execution (seconds)
Name Current %
401.bzip2 2836.000 -
403.gcc 60.000 7.14%
429.mcf 1139.000 -
433.milc 1178.000 -
444.namd 805.000 -
445.gobmk 4886.000 -
447.dealII 1926.000 -1.23%
450.soplex 543.000 -
453.povray 496.000 -
456.hmmer 3809.000 -
458.sjeng 8206.000 -
462.libquantum 185.000 -
464.h264ref 5890.000 -
470.lbm 4008.000 -
471.omnetpp 2909.000 2.83%
473.astar 4656.000 -
482.sphinx3 995.000 1.22%
Geometric Mean 1487.738 -
Code Size (bytes)
Name Current %
401.bzip2 33093.000 -1.36%
403.gcc 1656563.000 -
429.mcf 6947.000 -
433.milc 46189.000 -
444.namd 123078.000 -
445.gobmk 1255889.000 -
447.dealII 195962.000 -
450.soplex 149516.000 -
453.povray 513268.000 -
456.hmmer 80341.000 -
458.sjeng 79274.000 -
462.libquantum 12123.000 -
464.h264ref 298965.000 -
470.lbm 9220.000 -
471.omnetpp 322402.000 -
473.astar 15551.000 -
482.sphinx3 87396.000 -
Geometric Mean 94486.582 -
sample variability
Name Current %
401.bzip2 13.000 -
401.bzip2 (avg) 13.000 -
401.bzip2 (avg) (percentage) 0.480 -
401.bzip2 (max) 20.000 -13.04%
401.bzip2 (max) (percentage) 0.740 -8.64%
403.gcc 1.000 -
403.gcc (avg) 1.000 -
403.gcc (avg) (percentage) 2.680 -
403.gcc (max) 3.000 -
403.gcc (max) (percentage) 6.400 -1.54%
429.mcf 17.000 -
429.mcf (avg) 17.000 -
429.mcf (avg) (percentage) 1.510 -1.31%
429.mcf (max) 9.000 -10.00%
429.mcf (max) (percentage) 0.850 -3.41%
429.mcf (unexplained diff vs previous) 7.000 16.67%
429.mcf (unexplained diff vs previous) (percentage) 0.610 17.31%
433.milc 7.000 -
433.milc (avg) 7.000 -
433.milc (avg) (percentage) 0.640 1.59%
433.milc (max) 14.000 -6.67%
433.milc (max) (percentage) 1.270 -5.93%
433.milc (unexplained diff vs previous) 10.000 233.33%
433.milc (unexplained diff vs previous) (percentage) 0.850 226.92%
444.namd 7.000 -
444.namd (avg) 7.000 -
444.namd (avg) (percentage) 0.960 -
444.namd (max) 6.000 -
444.namd (max) (percentage) 0.820 -4.65%
445.gobmk 16.000 -
445.gobmk (avg) 16.000 -
445.gobmk (avg) (percentage) 0.330 -
445.gobmk (max) 15.000 -6.25%
445.gobmk (max) (percentage) 0.310 -8.82%
447.dealII 12.000 -
447.dealII (avg) 12.000 -
447.dealII (avg) (percentage) 0.640 -
447.dealII (max) 22.000 -8.33%
447.dealII (max) (percentage) 1.190 -4.80%
450.soplex 7.000 -
450.soplex (avg) 7.000 -
450.soplex (avg) (percentage) 1.310 -
450.soplex (max) 15.000 -6.25%
450.soplex (max) (percentage) 2.800 -10.26%
453.povray 4.000 -
453.povray (avg) 4.000 -
453.povray (avg) (percentage) 0.910 -
453.povray (max) 5.000 -
453.povray (max) (percentage) 1.040 -6.31%
456.hmmer 41.000 -
456.hmmer (avg) 41.000 -
456.hmmer (avg) (percentage) 1.080 -
456.hmmer (max) 20.000 -20.00%
456.hmmer (max) (percentage) 0.550 -16.67%
456.hmmer (unexplained diff vs previous) 21.000 -16.00%
456.hmmer (unexplained diff vs previous) (percentage) 0.550 -15.38%
458.sjeng 32.000 -
458.sjeng (avg) 32.000 -
458.sjeng (avg) (percentage) 0.390 -
458.sjeng (max) 22.000 -4.35%
458.sjeng (max) (percentage) 0.280 -
462.libquantum 2.000 -
462.libquantum (avg) 2.000 -
462.libquantum (avg) (percentage) 1.590 -1.24%
462.libquantum (max) 5.000 -16.67%
462.libquantum (max) (percentage) 3.170 -9.94%
462.libquantum (unexplained diff vs previous) 1.000 -80.00%
462.libquantum (unexplained diff vs previous) (percentage) 0.540 -79.93%
464.h264ref 16.000 -
464.h264ref (avg) 16.000 -
464.h264ref (avg) (percentage) 0.280 -
464.h264ref (max) 16.000 -11.11%
464.h264ref (max) (percentage) 0.280 -9.68%
470.lbm 28.000 -3.45%
470.lbm (avg) 28.000 -3.45%
470.lbm (avg) (percentage) 0.710 -2.74%
470.lbm (max) 16.000 -
470.lbm (max) (percentage) 0.400 -
470.lbm (unexplained diff vs previous) 1.000 -75.00%
470.lbm (unexplained diff vs previous) (percentage) 0.020 -80.00%
471.omnetpp 47.000 2.17%
471.omnetpp (avg) 47.000 2.17%
471.omnetpp (avg) (percentage) 1.650 1.23%
471.omnetpp (max) 82.000 290.48%
471.omnetpp (max) (percentage) 2.830 267.53%
471.omnetpp (unexplained diff vs previous) 80.000 344.44%
471.omnetpp (unexplained diff vs previous) (percentage) 2.750 329.69%
473.astar 29.000 -
473.astar (avg) 29.000 -
473.astar (avg) (percentage) 0.630 -
473.astar (max) 26.000 -3.70%
473.astar (max) (percentage) 0.570 -3.39%
482.sphinx3 8.000 -
482.sphinx3 (avg) 8.000 -
482.sphinx3 (avg) (percentage) 0.820 -
482.sphinx3 (max) 15.000 -
482.sphinx3 (max) (percentage) 1.530 -4.38%
483.xalancbmk (avg) (percentage) 1.440 -
483.xalancbmk (max) (percentage) 0.050 -
Geometric Mean 3.613 -