Matching large lists

How long matching takes

scripts/benchmark-scale.R in the taxify repository matches lists of accepted species names drawn from World Flora Online (WFO), as they are and with one letter changed in every tenth name. Each list runs in a new R session, after one call that loads the backbone. The run recorded in scripts/benchmark-scale-results.json used taxify 0.7.0 and vectra 0.13.0 under R 4.6.0, on Intel(R) Core(TM) i9-14900K with 32 logical cores, on 2026-10-13. A misspelt name goes through fuzzy matching after every exact stage has missed it, so a list with misspellings takes longer:

Loading the backbone took about 2.1 s and is paid once per session; every later call reuses it.

Lists that need no fuzzy matching

A list taken from a curated database or from an earlier taxify run is spelt as the backbone spells it. fuzzy = FALSE skips fuzzy matching, and the names that find no exact match come back as none:

checked <- c("Quercus robur", "Fagus sylvatica", "Betula pendula",
             "Pinus sylvestris", "Acer pseudoplatanus", "Qurecus petraea")
exact <- taxify(checked, backbone = "wfo", fuzzy = FALSE, verbose = FALSE)
table(exact$match_type)
#> 
#> exact  none 
#>     5     1

For a list with some misspellings, the fuzzy stage can run on the unmatched names alone; its rows replace the none rows of the first result, matched by input name:

missed <- exact$input_name[exact$match_type == "none"]
second <- taxify(missed, backbone = "wfo", verbose = FALSE)
rows <- match(second$input_name, exact$input_name)
exact[rows, ] <- second
exact[, c("input_name", "matched_name", "match_type")]
#>            input_name        matched_name match_type
#> 1       Quercus robur       Quercus robur      exact
#> 2     Fagus sylvatica     Fagus sylvatica      exact
#> 3      Betula pendula      Betula pendula      exact
#> 4    Pinus sylvestris    Pinus sylvestris      exact
#> 5 Acer pseudoplatanus Acer pseudoplatanus      exact
#> 6     Qurecus petraea     Quercus petraea      fuzzy
#> Sources: WFO 2026.06 | cite() for full citations

The backbone stays loaded between the two calls, so the second one does not load it again.

Ordering the backbones

In a fallback chain, the first backbone receives every name and each later one only the names still unmatched. A chain that leads with the backbone holding most of the list sends fewer names to the others, and a smaller backbone is faster to search. The sizes:

bb <- list_backbones(verbose = FALSE)
bb <- bb[bb$code == "taxon", c("name", "n_names", "size_mb")]
bb[order(bb$n_names), ]
#>           name n_names size_mb
#> 18     avilist   40572       8
#> 19        lpsn   44885      12
#> 14   reptiledb   50043       9
#> 17         mdd   61862      10
#> 12    fishbase  102703      18
#> 13 sealifebase  134031      27
#> 9      euromed  146922      33
#> 11   algaebase  172351      34
#> 10    fungorum  315037      68
#> 5         itis 1000552     197
#> 15        lcvp 1337891     241
#> 16        wcvp 1448984     326
#> 8        worms 1565652     290
#> 1          wfo 1664014     785
#> 6         ncbi 2952424     524
#> 7          ott 3690217     728
#> 2          col 5432444    1997
#> 4         gbif 6404001    1622
#> 3        colxr 8479599    1393

The order also decides the accepted name of a name both backbones hold, as described in Choosing and combining backbones.

Long lists in parts

A single call matches a list of any length, and loses everything if the session ends during the run. Splitting the list into parts and writing each result to disk lets a run resume from the last finished part. The example uses a short list and parts of three names, written to a temporary directory; a real run uses parts of tens of thousands:

all_names <- c(checked, "Fraxinus excelsior", "Tilia cordata",
               "Sorbus aucuparia")
parts <- split(all_names, ceiling(seq_along(all_names) / 3))
out_dir <- file.path(tempdir(), "taxify-parts")
dir.create(out_dir, showWarnings = FALSE)

for (i in seq_along(parts)) {
  file <- file.path(out_dir, sprintf("part_%04d.rds", i))
  if (file.exists(file)) next
  res <- taxify(parts[[i]], backbone = "wfo", verbose = FALSE)
  saveRDS(res, file)
}

files <- list.files(out_dir, pattern = "\\.rds$", full.names = TRUE)
matched <- do.call(rbind, lapply(files, readRDS))
nrow(matched)
#> [1] 9

A part that already has a file is skipped, so running the loop again after an interruption continues where it stopped. Each name is matched on its own, so the result does not depend on how the list is split. Attaching published data inside the loop, before saveRDS(), keeps only one part in memory at a time.

Memory

scripts/benchmark-memory.R measures the memory of the R process, including what the backbone occupies outside R’s own objects, which gc() does not report. Each stage runs in a new session. The run recorded in scripts/benchmark-memory-results.json used taxify 0.7.0 under R 4.6.0 on 2026-10-13, matching 5,000 names. File is the size on disk of the backbone that run read, in megabytes of 10^6 bytes, so it differs from the sizes list_backbones() reports from the manifest. The other values are megabytes above an R session with taxify loaded; the fuzzy column is the case in which every name needs fuzzy matching:

A loaded backbone takes between 1.9 and 2.6 times its file size here, for its indexes and its text, and stays loaded for the session. Fuzzy matching adds a working buffer that grows with the number of unmatched names, and is released after each call.

taxify_clear_cache() releases the loaded backbones, for example before fitting a model on the matched result. The files stay on disk, and the next call loads them again.

Downloads before a run

A run on a server or a cluster can download what it needs first, so that the matching step does not depend on the network:

taxify_download(c("col", "wfo"))
taxify_download_enrichment(c("iucn", "zanne"))
Sys.setenv(TAXIFY_OFFLINE = "1")

With TAXIFY_OFFLINE set, or options(taxify.offline = TRUE), taxify uses the files on disk and does not check for newer versions. taxify_refresh_manifest() makes the next call in a long session check again.

Where the files are

taxify_data_dir() returns the directory that holds every backbone and dataset. It is shared by all R projects of a user, so a backbone is downloaded once per machine. The taxify.data_dir option or the TAXIFY_DATA_DIR environment variable sets another directory, for example one shared by several users of a server; the files are only read during matching, so several sessions can use them at once. taxify_store() lists the builds on disk. Deleting a backbone’s directory removes it, and the next call that needs it downloads it again.

Where next

• Choosing and combining backbones: versions, pinning and the fallback chain.

• How names are matched: the fuzzy stage and its threshold.