Benchmark timings for mkin

Johannes Ranke

Last change 17 February 2023 (rebuilt 2023-10-30)

Source: vignettes/web_only/benchmarks.rmd

Each system is characterized by the operating system type, the CPU type, the mkin version, and, as in June 2022 the current R version lead to worse performance, the R version. A compiler was available, so if no analytical solution was available, compiled ODE models are used.

Every fit is only performed once, so the accuracy of the benchmarks is limited.

The following wrapper function for mmkin is used because the way the error model is specified was changed in mkin version 0.9.49.1.

if (packageVersion("mkin") > "0.9.48.1") {
  mmkin_bench <- function(models, datasets, error_model = "const") {
    mmkin(models, datasets, error_model = error_model, cores = 1, quiet = TRUE)
  }
} else {
  mmkin_bench <- function(models, datasets, error_model = NULL) {
    mmkin(models, datasets, reweight.method = error_model, cores = 1, quiet = TRUE)
  }
}

Test cases

Parent only:

FOCUS_C <- FOCUS_2006_C
FOCUS_D <- subset(FOCUS_2006_D, value != 0)
parent_datasets <- list(FOCUS_C, FOCUS_D)


t1 <- system.time(mmkin_bench(c("SFO", "FOMC", "DFOP", "HS"), parent_datasets))[["elapsed"]]
t2 <- system.time(mmkin_bench(c("SFO", "FOMC", "DFOP", "HS"), parent_datasets,
    error_model = "tc"))[["elapsed"]]

One metabolite:

SFO_SFO <- mkinmod(
  parent = mkinsub("SFO", "m1"),
  m1 = mkinsub("SFO"))
FOMC_SFO <- mkinmod(
  parent = mkinsub("FOMC", "m1"),
  m1 = mkinsub("SFO"))
DFOP_SFO <- mkinmod(
  parent = mkinsub("FOMC", "m1"), # erroneously used FOMC twice, not fixed for consistency
  m1 = mkinsub("SFO"))
t3 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D)))[["elapsed"]]
t4 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D),
    error_model = "tc"))[["elapsed"]]
t5 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D),
    error_model = "obs"))[["elapsed"]]

Two metabolites, synthetic data:

m_synth_SFO_lin <- mkinmod(parent = mkinsub("SFO", "M1"),
                           M1 = mkinsub("SFO", "M2"),
                           M2 = mkinsub("SFO"),
                           use_of_ff = "max", quiet = TRUE)

m_synth_DFOP_par <- mkinmod(parent = mkinsub("DFOP", c("M1", "M2")),
                           M1 = mkinsub("SFO"),
                           M2 = mkinsub("SFO"),
                           use_of_ff = "max", quiet = TRUE)

SFO_lin_a <- synthetic_data_for_UBA_2014[[1]]$data

DFOP_par_c <- synthetic_data_for_UBA_2014[[12]]$data

t6 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a)))[["elapsed"]]
t7 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c)))[["elapsed"]]

t8 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a),
    error_model = "tc"))[["elapsed"]]
t9 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c),
    error_model = "tc"))[["elapsed"]]

t10 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a),
    error_model = "obs"))[["elapsed"]]
t11 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c),
    error_model = "obs"))[["elapsed"]]

Results

Benchmarks for all available error models are shown. They are intended for improving mkin, not for comparing CPUs or operating systems. All trademarks belong to their respective owners.

Parent only

Constant variance (t1) and two-component error model (t2) for four models fitted to two datasets, i.e. eight fits for each test.

OS CPU R mkin t1 t2
Linux Ryzen 7 1700 NA 0.9.48.1 3.610 11.019
Linux Ryzen 7 1700 NA 0.9.49.1 8.184 22.889
Linux Ryzen 7 1700 NA 0.9.49.2 7.064 12.558
Linux Ryzen 7 1700 NA 0.9.49.3 7.296 21.239
Linux Ryzen 7 1700 NA 0.9.49.4 5.936 20.545
Linux Ryzen 7 1700 NA 0.9.50.2 1.714 3.971
Linux Ryzen 7 1700 NA 0.9.50.3 1.752 4.156
Linux Ryzen 7 1700 NA 0.9.50.4 1.786 3.729
Linux Ryzen 7 1700 NA 1.0.3 1.881 3.504
Linux Ryzen 7 1700 NA 1.0.4 1.867 3.450
Linux Ryzen 7 1700 4.1.3 1.1.0 1.791 3.289
Linux Ryzen 7 1700 4.2.1 1.1.0 1.842 3.453
Linux i7-4710MQ 4.2.1 1.1.0 1.959 4.116
Linux i7-4710MQ 4.1.3 1.1.0 1.877 3.906
Linux i7-4710MQ 4.2.1 1.1.1 1.644 3.172
Linux Ryzen 7 1700 4.2.1 1.1.1 1.770 3.377
Linux Ryzen 7 1700 4.2.1 1.1.2 1.957 3.633
Linux Ryzen 7 1700 4.2.2 1.2.0 2.140 3.774
Linux Ryzen 7 1700 4.2.2 1.2.2 2.187 3.851
Linux Ryzen 9 7950X 4.2.2 1.2.0 1.288 1.794
Linux Ryzen 9 7950X 4.2.2 1.2.2 1.276 1.804
Linux Ryzen 9 7950X 4.2.2 1.2.3 1.370 1.883
Linux Ryzen 9 7950X 4.2.3 1.2.3 1.406 1.948
Linux Ryzen 9 7950X 4.3.0 1.2.4 1.386 1.960
Linux Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz 4.3.1 1.2.5 2.369 3.632
Linux Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz 4.3.1 1.2.6 2.856 4.960

One metabolite

Constant variance (t3), two-component error model (t4), and variance by variable (t5) for three models fitted to one dataset, i.e. three fits for each test.

OS CPU R mkin t3 t4 t5
Linux Ryzen 7 1700 NA 0.9.48.1 3.764 14.347 9.495
Linux Ryzen 7 1700 NA 0.9.49.1 4.649 13.789 6.395
Linux Ryzen 7 1700 NA 0.9.49.2 4.786 8.461 5.675
Linux Ryzen 7 1700 NA 0.9.49.3 4.510 13.805 7.386
Linux Ryzen 7 1700 NA 0.9.49.4 4.446 15.335 6.002
Linux Ryzen 7 1700 NA 0.9.50.2 1.402 6.174 2.764
Linux Ryzen 7 1700 NA 0.9.50.3 1.430 6.615 2.878
Linux Ryzen 7 1700 NA 0.9.50.4 1.397 7.251 2.810
Linux Ryzen 7 1700 NA 1.0.3 1.430 6.344 2.798
Linux Ryzen 7 1700 NA 1.0.4 1.415 6.364 2.820
Linux Ryzen 7 1700 4.1.3 1.1.0 1.310 6.279 2.681
Linux Ryzen 7 1700 4.2.1 1.1.0 3.802 21.247 8.461
Linux i7-4710MQ 4.2.1 1.1.0 3.334 19.521 7.565
Linux i7-4710MQ 4.1.3 1.1.0 1.578 8.058 3.339
Linux i7-4710MQ 4.2.1 1.1.1 1.230 5.839 2.444
Linux Ryzen 7 1700 4.2.1 1.1.1 1.308 5.758 2.558
Linux Ryzen 7 1700 4.2.1 1.1.2 1.503 6.147 2.803
Linux Ryzen 7 1700 4.2.2 1.2.0 1.554 6.193 2.843
Linux Ryzen 7 1700 4.2.2 1.2.2 1.585 6.335 3.003
Linux Ryzen 9 7950X 4.2.2 1.2.0 0.792 2.378 1.245
Linux Ryzen 9 7950X 4.2.2 1.2.2 0.784 2.355 1.233
Linux Ryzen 9 7950X 4.2.2 1.2.3 0.770 2.011 1.123
Linux Ryzen 9 7950X 4.2.3 1.2.3 0.793 2.109 1.178
Linux Ryzen 9 7950X 4.3.0 1.2.4 0.779 2.080 1.106
Linux Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz 4.3.1 1.2.5 1.823 5.555 2.404
Linux Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz 4.3.1 1.2.6 1.761 5.405 2.462

Two metabolites

Constant variance (t6 and t7), two-component error model (t8 and t9), and variance by variable (t10 and t11) for one model fitted to one dataset, i.e. one fit for each test.

OS CPU R mkin t6 t7 t8 t9 t10 t11
Linux Ryzen 7 1700 NA 0.9.48.1 2.623 4.587 7.525 16.621 8.576 31.267
Linux Ryzen 7 1700 NA 0.9.49.1 2.542 4.128 4.632 8.171 3.676 5.636
Linux Ryzen 7 1700 NA 0.9.49.2 2.723 4.478 4.862 7.618 3.579 5.574
Linux Ryzen 7 1700 NA 0.9.49.3 2.643 4.374 7.020 11.124 5.388 7.365
Linux Ryzen 7 1700 NA 0.9.49.4 2.635 4.259 4.737 7.763 3.427 5.626
Linux Ryzen 7 1700 NA 0.9.50.2 0.777 1.236 1.332 2.872 2.069 2.987
Linux Ryzen 7 1700 NA 0.9.50.3 0.858 1.264 1.333 2.984 2.113 3.073
Linux Ryzen 7 1700 NA 0.9.50.4 0.783 1.282 1.486 3.815 1.958 3.105
Linux Ryzen 7 1700 NA 1.0.3 0.763 1.244 1.457 3.054 1.923 2.839
Linux Ryzen 7 1700 NA 1.0.4 0.785 1.252 1.466 3.091 1.936 2.826
Linux Ryzen 7 1700 4.1.3 1.1.0 0.744 1.227 1.288 3.553 1.895 2.738
Linux Ryzen 7 1700 4.2.1 1.1.0 3.018 4.165 5.036 10.844 6.623 9.722
Linux i7-4710MQ 4.2.1 1.1.0 2.522 3.792 4.143 11.268 5.935 8.728
Linux i7-4710MQ 4.1.3 1.1.0 0.907 1.535 1.589 4.544 2.302 3.463
Linux i7-4710MQ 4.2.1 1.1.1 0.678 1.095 1.149 3.247 1.658 2.472
Linux Ryzen 7 1700 4.2.1 1.1.1 0.696 1.124 1.321 2.786 1.744 2.566
Linux Ryzen 7 1700 4.2.1 1.1.2 0.861 1.295 1.507 3.102 1.961 2.852
Linux Ryzen 7 1700 4.2.2 1.2.0 0.913 1.345 1.539 3.011 1.987 2.802
Linux Ryzen 7 1700 4.2.2 1.2.2 0.935 1.381 1.551 3.209 1.976 3.013
Linux Ryzen 9 7950X 4.2.2 1.2.0 0.445 0.591 0.660 1.190 0.814 1.100
Linux Ryzen 9 7950X 4.2.2 1.2.2 0.443 0.586 0.661 1.176 0.803 1.097
Linux Ryzen 9 7950X 4.2.2 1.2.3 0.418 0.530 0.591 1.006 0.716 0.949
Linux Ryzen 9 7950X 4.2.3 1.2.3 0.432 0.549 0.609 1.052 0.743 0.989
Linux Ryzen 9 7950X 4.3.0 1.2.4 0.410 0.526 0.553 1.249 0.712 0.948
Linux Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz 4.3.1 1.2.5 0.798 1.096 1.217 3.173 1.634 2.271
Linux Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz 4.3.1 1.2.6 0.813 1.136 1.220 3.114 1.598 2.255