Benchmarking estimation speed against other packages

{logitr} is faster than most other packages with similar functionality. To demonstrate this, a benchmark was conducted by estimating the same preference space mixed logit model using the following R packages:

• {logitr}
• {mixl}
• {mlogit}
• {gmnl}
• {apollo}

The benchmark can be viewed at this Google Colab notebook:

https://colab.research.google.com/drive/1vYlBdJd4xCV43UwJ33XXpO3Ys8xWkuxx?usp=sharing

Benchmarks will always vary for every run of a benchmarking code, even when run on the same machine due to variations in background processes. Thus, if you run this code yourself on a different machine, your results may vary, though the overall order and trends in terms of each package’s relative speed should be similar to those from the Colab notebook.

Comparing run times

The {logitr} package includes a runtimes data frame that is exported from the Google Colab notebook used to conduct the benchmark. The tables below summarize the run times for each package and how many times slower they are relative to {logitr}.

library(logitr)
library(dplyr)
library(tidyr)
library(kableExtra) # For tables

numDraws <- unique(runtimes$numDraws)
logitr_time <- runtimes %>%
    filter(package == "logitr") %>%
    rename(time_logitr = time_sec)
time_compare <- runtimes %>%
    left_join(select(logitr_time, -package), by = "numDraws") %>%
    mutate(mult = round(time_sec/ time_logitr, 1)) %>%
    select(-time_logitr)
# Compare raw times
time_compare %>%
    select(-mult) %>%
    pivot_wider(names_from = numDraws, values_from = time_sec) %>% 
    kbl()

package	50	200	400	600	800	1000
logitr	2.752860	8.930408	13.72697	24.13735	33.47056	39.22012
mixl (1 core)	10.640267	49.667703	80.22922	158.23673	229.17401	271.35824
mixl (2 cores)	8.928487	41.577738	66.05436	129.84954	184.54557	230.91617
mlogit	11.926502	19.901097	87.58429	60.38106	100.63138	98.41650
gmnl	10.553787	31.379011	69.74575	121.93843	99.23701	141.36727
apollo (1 core)	17.287605	44.118732	84.29395	129.33302	164.37400	198.21812
apollo (2 cores)	21.911355	53.223971	82.69647	120.27101	163.80518	196.84896

# Compare how many times slower compared to logitr
time_compare %>%
    select(-time_sec) %>%
    pivot_wider(names_from = numDraws, values_from = mult) %>% 
    kbl()

package	50	200	400	600	800	1000
logitr	1.0	1.0	1.0	1.0	1.0	1.0
mixl (1 core)	3.9	5.6	5.8	6.6	6.8	6.9
mixl (2 cores)	3.2	4.7	4.8	5.4	5.5	5.9
mlogit	4.3	2.2	6.4	2.5	3.0	2.5
gmnl	3.8	3.5	5.1	5.1	3.0	3.6
apollo (1 core)	6.3	4.9	6.1	5.4	4.9	5.1
apollo (2 cores)	8.0	6.0	6.0	5.0	4.9	5.0

The code below plots the relative run times from the Colab notebook.

library(ggplot2)
library(ggrepel)

plotColors <- c("black", RColorBrewer::brewer.pal(n = 5, name = "Set1"), "gold")
benchmark <- runtimes %>% 
    ggplot(aes(x = numDraws, y = time_sec, color = package)) +
    geom_line() +
    geom_point() +
    geom_text_repel(
        data = . %>% filter(numDraws == max(numDraws)),
        aes(label = package),
        hjust = 0, nudge_x = 40, direction = "y",
        size = 4.5, segment.size = 0
    ) +
    scale_x_continuous(
        limits = c(0, 1200),
        breaks = numDraws,
        labels = scales::comma) +
    scale_y_continuous(limits = c(0, 300), breaks = seq(0, 300, 100)) +
    scale_color_manual(values = plotColors) +
    guides(
        point = guide_legend(override.aes = list(label = "")),
        color = guide_legend(override.aes = list(label = ""))) +
    theme_bw(base_size = 18) +
    theme(
        panel.grid.minor = element_blank(),
        panel.grid.major.x = element_blank(),
        legend.position = "none",
        axis.line.x = element_blank(),
        axis.ticks.x = element_blank()
    ) +
    labs(
        x = "Number of random draws",
        y = "Computation time (seconds)"
    )

benchmark