Benchmark timings for mkin

Johannes Ranke

Last change 1 July 2022 (rebuilt 2022-07-01)

Source: vignettes/web_only/benchmarks.rmd

Each system is characterized by the operating system type, the CPU type, the mkin version, and, as in June 2022 the current R version lead to worse performance, the R version. A compiler was available, so if no analytical solution was available, compiled ODE models are used.

Every fit is only performed once, so the accuracy of the benchmarks is limited.

Test cases

Parent only:

FOCUS_C <- FOCUS_2006_C
FOCUS_D <- subset(FOCUS_2006_D, value != 0)
parent_datasets <- list(FOCUS_C, FOCUS_D)

t1 <- system.time(mmkin_bench(c("SFO", "FOMC", "DFOP", "HS"), parent_datasets))[["elapsed"]]
t2 <- system.time(mmkin_bench(c("SFO", "FOMC", "DFOP", "HS"), parent_datasets,
    error_model = "tc"))[["elapsed"]]

One metabolite:

SFO_SFO <- mkinmod(
  parent = mkinsub("SFO", "m1"),
  m1 = mkinsub("SFO"))
FOMC_SFO <- mkinmod(
  parent = mkinsub("FOMC", "m1"),
  m1 = mkinsub("SFO"))
DFOP_SFO <- mkinmod(
  parent = mkinsub("FOMC", "m1"),
  m1 = mkinsub("SFO"))
t3 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D)))[["elapsed"]]
t4 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D),
    error_model = "tc"))[["elapsed"]]
t5 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D),
    error_model = "obs"))[["elapsed"]]

Two metabolites, synthetic data:

m_synth_SFO_lin <- mkinmod(parent = mkinsub("SFO", "M1"),
                           M1 = mkinsub("SFO", "M2"),
                           M2 = mkinsub("SFO"),
                           use_of_ff = "max", quiet = TRUE)

m_synth_DFOP_par <- mkinmod(parent = mkinsub("DFOP", c("M1", "M2")),
                           M1 = mkinsub("SFO"),
                           M2 = mkinsub("SFO"),
                           use_of_ff = "max", quiet = TRUE)

SFO_lin_a <- synthetic_data_for_UBA_2014[[1]]$data

DFOP_par_c <- synthetic_data_for_UBA_2014[[12]]$data

t6 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a)))[["elapsed"]]
t7 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c)))[["elapsed"]]

t8 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a),
    error_model = "tc"))[["elapsed"]]
t9 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c),
    error_model = "tc"))[["elapsed"]]

t10 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a),
    error_model = "obs"))[["elapsed"]]
t11 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c),
    error_model = "obs"))[["elapsed"]]
mkin_benchmarks[system_string, paste0("t", 1:11)] <-
  c(t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11)
save(mkin_benchmarks, file = benchmark_path)
# Hide rownames from kable for results section
rownames(mkin_benchmarks) <- NULL

Results

Benchmarks for all available error models are shown. They are intended for improving mkin, not for comparing CPUs or operating systems. All trademarks belong to their respective owners.

Parent only

Constant variance (t1) and two-component error model (t2) for four models fitted to two datasets, i.e. eight fits for each test.

OS CPU R mkin t1 t2
Linux Ryzen 7 1700 NA 0.9.48.1 3.610 11.019
Linux Ryzen 7 1700 NA 0.9.49.1 8.184 22.889
Linux Ryzen 7 1700 NA 0.9.49.2 7.064 12.558
Linux Ryzen 7 1700 NA 0.9.49.3 7.296 21.239
Linux Ryzen 7 1700 NA 0.9.49.4 5.936 20.545
Linux Ryzen 7 1700 NA 0.9.50.2 1.714 3.971
Linux Ryzen 7 1700 NA 0.9.50.3 1.752 4.156
Linux Ryzen 7 1700 NA 0.9.50.4 1.786 3.729
Linux Ryzen 7 1700 NA 1.0.3 1.881 3.504
Linux Ryzen 7 1700 NA 1.0.4 1.867 3.450
Linux Ryzen 7 1700 4.1.3 1.1.0 1.791 3.289
Linux Ryzen 7 1700 4.2.1 1.1.0 1.842 3.453
Linux i7-4710MQ 4.2.1 1.1.0 1.959 4.116
Linux i7-4710MQ 4.1.3 1.1.0 1.877 3.906

One metabolite

Constant variance (t3), two-component error model (t4), and variance by variable (t5) for three models fitted to one dataset, i.e. three fits for each test.

OS CPU R mkin t3 t4 t5
Linux Ryzen 7 1700 NA 0.9.48.1 3.764 14.347 9.495
Linux Ryzen 7 1700 NA 0.9.49.1 4.649 13.789 6.395
Linux Ryzen 7 1700 NA 0.9.49.2 4.786 8.461 5.675
Linux Ryzen 7 1700 NA 0.9.49.3 4.510 13.805 7.386
Linux Ryzen 7 1700 NA 0.9.49.4 4.446 15.335 6.002
Linux Ryzen 7 1700 NA 0.9.50.2 1.402 6.174 2.764
Linux Ryzen 7 1700 NA 0.9.50.3 1.430 6.615 2.878
Linux Ryzen 7 1700 NA 0.9.50.4 1.397 7.251 2.810
Linux Ryzen 7 1700 NA 1.0.3 1.430 6.344 2.798
Linux Ryzen 7 1700 NA 1.0.4 1.415 6.364 2.820
Linux Ryzen 7 1700 4.1.3 1.1.0 1.310 6.279 2.681
Linux Ryzen 7 1700 4.2.1 1.1.0 3.802 21.247 8.461
Linux i7-4710MQ 4.2.1 1.1.0 3.334 19.521 7.565
Linux i7-4710MQ 4.1.3 1.1.0 1.578 8.058 3.339

Two metabolites

Constant variance (t6 and t7), two-component error model (t8 and t9), and variance by variable (t10 and t11) for one model fitted to one dataset, i.e. one fit for each test.

OS CPU R mkin t6 t7 t8 t9 t10 t11
Linux Ryzen 7 1700 NA 0.9.48.1 2.623 4.587 7.525 16.621 8.576 31.267
Linux Ryzen 7 1700 NA 0.9.49.1 2.542 4.128 4.632 8.171 3.676 5.636
Linux Ryzen 7 1700 NA 0.9.49.2 2.723 4.478 4.862 7.618 3.579 5.574
Linux Ryzen 7 1700 NA 0.9.49.3 2.643 4.374 7.020 11.124 5.388 7.365
Linux Ryzen 7 1700 NA 0.9.49.4 2.635 4.259 4.737 7.763 3.427 5.626
Linux Ryzen 7 1700 NA 0.9.50.2 0.777 1.236 1.332 2.872 2.069 2.987
Linux Ryzen 7 1700 NA 0.9.50.3 0.858 1.264 1.333 2.984 2.113 3.073
Linux Ryzen 7 1700 NA 0.9.50.4 0.783 1.282 1.486 3.815 1.958 3.105
Linux Ryzen 7 1700 NA 1.0.3 0.763 1.244 1.457 3.054 1.923 2.839
Linux Ryzen 7 1700 NA 1.0.4 0.785 1.252 1.466 3.091 1.936 2.826
Linux Ryzen 7 1700 4.1.3 1.1.0 0.744 1.227 1.288 3.553 1.895 2.738
Linux Ryzen 7 1700 4.2.1 1.1.0 3.018 4.165 5.036 10.844 6.623 9.722
Linux i7-4710MQ 4.2.1 1.1.0 2.522 3.792 4.143 11.268 5.935 8.728
Linux i7-4710MQ 4.1.3 1.1.0 0.907 1.535 1.589 4.544 2.302 3.463