Benchmark timings for mkin

Each system is characterized by its CPU type, the operating system type and the mkin version. Currently only values for one system are available. A compiler was available, so if no analytical solution was available, compiled ODE models are used.

Test cases

Parent only:

FOCUS_C <- FOCUS_2006_C
FOCUS_D <- subset(FOCUS_2006_D, value != 0)
parent_datasets <- list(FOCUS_C, FOCUS_D)

t1 <- system.time(mmkin_bench(c("SFO", "FOMC", "DFOP", "HS"), parent_datasets))[["elapsed"]]
t2 <- system.time(mmkin_bench(c("SFO", "FOMC", "DFOP", "HS"), parent_datasets,
    error_model = "tc"))[["elapsed"]]

One metabolite:

SFO_SFO <- mkinmod(
  parent = mkinsub("SFO", "m1"),
  m1 = mkinsub("SFO"))
FOMC_SFO <- mkinmod(
  parent = mkinsub("FOMC", "m1"),
  m1 = mkinsub("SFO"))
DFOP_SFO <- mkinmod(
  parent = mkinsub("FOMC", "m1"),
  m1 = mkinsub("SFO"))
t3 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D)))[["elapsed"]]

## Warning in mkinfit(models[[model_index]], datasets[[dataset_index]], ...):
## Shapiro-Wilk test for standardized residuals: p = 0.0165

## Warning in mkinfit(models[[model_index]], datasets[[dataset_index]], ...):
## Shapiro-Wilk test for standardized residuals: p = 0.0499

## Warning in mkinfit(models[[model_index]], datasets[[dataset_index]], ...):
## Shapiro-Wilk test for standardized residuals: p = 0.0499

t4 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D),
    error_model = "tc"))[["elapsed"]]
t5 <- system.time(mmkin_bench(list(SFO_SFO, FOMC_SFO, DFOP_SFO), list(FOCUS_D),
    error_model = "obs"))[["elapsed"]]

Two metabolites, synthetic data:

m_synth_SFO_lin <- mkinmod(parent = mkinsub("SFO", "M1"),
                           M1 = mkinsub("SFO", "M2"),
                           M2 = mkinsub("SFO"),
                           use_of_ff = "max", quiet = TRUE)

m_synth_DFOP_par <- mkinmod(parent = mkinsub("DFOP", c("M1", "M2")),
                           M1 = mkinsub("SFO"),
                           M2 = mkinsub("SFO"),
                           use_of_ff = "max", quiet = TRUE)

SFO_lin_a <- synthetic_data_for_UBA_2014[[1]]$data

DFOP_par_c <- synthetic_data_for_UBA_2014[[12]]$data

t6 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a)))[["elapsed"]]
t7 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c)))[["elapsed"]]

## Warning in mkinfit(models[[model_index]], datasets[[dataset_index]], ...):
## Shapiro-Wilk test for standardized residuals: p = 0.000174

t8 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a),
    error_model = "tc"))[["elapsed"]]
t9 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c),
    error_model = "tc"))[["elapsed"]]

t10 <- system.time(mmkin_bench(list(m_synth_SFO_lin), list(SFO_lin_a),
    error_model = "obs"))[["elapsed"]]
t11 <- system.time(mmkin_bench(list(m_synth_DFOP_par), list(DFOP_par_c),
    error_model = "obs"))[["elapsed"]]

mkin_benchmarks[system_string, paste0("t", 1:11)] <-
  c(t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11)
save(mkin_benchmarks, file = "~/git/mkin/vignettes/web_only/mkin_benchmarks.rda")

Results

Currently, we only have benchmark information on one system, therefore only the mkin version is shown with the results below. Timings are in seconds, shorter is better. All results were obtained by serial, i.e. not using multiple computing cores.

Benchmarks for all available error models are shown.

Parent only

Constant variance (t1) and two-component error model (t2) for four models fitted to two datasets, i.e. eight fits for each test.

mkin version	t1 [s]	t2 [s]
0.9.48.1	3.610	11.019
0.9.49.1	8.184	22.889
0.9.49.2	7.064	12.558
0.9.49.3	7.296	21.239
0.9.49.4	5.936	20.545
0.9.50.2	1.714	3.971
0.9.50.3	1.752	4.156

One metabolite

Constant variance (t3), two-component error model (t4), and variance by variable (t5) for three models fitted to one dataset, i.e. three fits for each test.

mkin version	t3 [s]	t4 [s]	t5 [s]
0.9.48.1	3.764	14.347	9.495
0.9.49.1	4.649	13.789	6.395
0.9.49.2	4.786	8.461	5.675
0.9.49.3	4.510	13.805	7.386
0.9.49.4	4.446	15.335	6.002
0.9.50.2	1.402	6.174	2.764
0.9.50.3	1.430	6.615	2.878

Two metabolites

Constant variance (t6 and t7), two-component error model (t8 and t9), and variance by variable (t10 and t11) for one model fitted to one dataset, i.e. one fit for each test.

mkin version	t6 [s]	t7 [s]	t8 [s]	t9 [s]	t10 [s]	t11 [s]
0.9.48.1	2.623	4.587	7.525	16.621	8.576	31.267
0.9.49.1	2.542	4.128	4.632	8.171	3.676	5.636
0.9.49.2	2.723	4.478	4.862	7.618	3.579	5.574
0.9.49.3	2.643	4.374	7.020	11.124	5.388	7.365
0.9.49.4	2.635	4.259	4.737	7.763	3.427	5.626
0.9.50.2	0.777	1.236	1.332	2.872	2.069	2.987
0.9.50.3	0.858	1.264	1.333	2.984	2.113	3.073

Johannes Ranke

2020-10-15

Test cases

Results

Parent only

One metabolite

Two metabolites