--- title: "Matching large lists" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Matching large lists} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- This article shows how long `taxify()` takes on lists from a thousand to a hundred thousand names, how much memory a backbone takes, and how to run a list too long for one call in parts that can resume after an interruption. Installation is covered in [Get started](taxify.html). 1. **Turn off** fuzzy matching for checked lists with `fuzzy = FALSE`, or run it on the unmatched names only. 2. **Order** the backbones so the one that resolves most names comes first. 3. **Split** a very long list into parts and write each result to disk. 4. **Download** backbones and datasets before a run, and free memory after it. ``` r library(taxify) ``` ## How long matching takes `scripts/benchmark-scale.R` in the taxify repository matches lists of accepted species names drawn from World Flora Online (WFO), as they are and with one letter changed in every tenth name. Each list runs in a new R session, after one call that loads the backbone. The run recorded in `scripts/benchmark-scale-results.json` used taxify 0.7.0 and [vectra](https://gillescolling.com/vectra/) 0.13.0 under R 4.6.0, on Intel(R) Core(TM) i9-14900K with 32 logical cores, on 2026-10-13. A misspelt name goes through fuzzy matching after every exact stage has missed it, so a list with misspellings takes longer: | Names| All spelt correctly (s)| One in ten misspelt (s)| |-------:|-----------------------:|-----------------------:| | 1,000| 0.9| 3.3| | 10,000| 1.8| 11.1| | 100,000| 14.1| 36.8| Loading the backbone took about 2.1 s and is paid once per session; every later call reuses it. ## Lists that need no fuzzy matching A list taken from a curated database or from an earlier taxify run is spelt as the backbone spells it. `fuzzy = FALSE` skips fuzzy matching, and the names that find no exact match come back as `none`: ``` r checked <- c("Quercus robur", "Fagus sylvatica", "Betula pendula", "Pinus sylvestris", "Acer pseudoplatanus", "Qurecus petraea") exact <- taxify(checked, backbone = "wfo", fuzzy = FALSE, verbose = FALSE) table(exact$match_type) #> #> exact none #> 5 1 ``` For a list with some misspellings, the fuzzy stage can run on the unmatched names alone; its rows replace the `none` rows of the first result, matched by input name: ``` r missed <- exact$input_name[exact$match_type == "none"] second <- taxify(missed, backbone = "wfo", verbose = FALSE) rows <- match(second$input_name, exact$input_name) exact[rows, ] <- second exact[, c("input_name", "matched_name", "match_type")] #> input_name matched_name match_type #> 1 Quercus robur Quercus robur exact #> 2 Fagus sylvatica Fagus sylvatica exact #> 3 Betula pendula Betula pendula exact #> 4 Pinus sylvestris Pinus sylvestris exact #> 5 Acer pseudoplatanus Acer pseudoplatanus exact #> 6 Qurecus petraea Quercus petraea fuzzy #> Sources: WFO 2026.06 | cite() for full citations ``` The backbone stays loaded between the two calls, so the second one does not load it again. ## Ordering the backbones In a fallback chain, the first backbone receives every name and each later one only the names still unmatched. A chain that leads with the backbone holding most of the list sends fewer names to the others, and a smaller backbone is faster to search. The sizes: ``` r bb <- list_backbones(verbose = FALSE) bb <- bb[bb$code == "taxon", c("name", "n_names", "size_mb")] bb[order(bb$n_names), ] #> name n_names size_mb #> 18 avilist 40572 8 #> 19 lpsn 44885 12 #> 14 reptiledb 50043 9 #> 17 mdd 61862 10 #> 12 fishbase 102703 18 #> 13 sealifebase 134031 27 #> 9 euromed 146922 33 #> 11 algaebase 172351 34 #> 10 fungorum 315037 68 #> 5 itis 1000552 197 #> 15 lcvp 1337891 241 #> 16 wcvp 1448984 326 #> 8 worms 1565652 290 #> 1 wfo 1664014 785 #> 6 ncbi 2952424 524 #> 7 ott 3690217 728 #> 2 col 5432444 1997 #> 4 gbif 6404001 1622 #> 3 colxr 8479599 1393 ``` The order also decides the accepted name of a name both backbones hold, as described in [Choosing and combining backbones](backbones.html). ## Long lists in parts A single call matches a list of any length, and loses everything if the session ends during the run. Splitting the list into parts and writing each result to disk lets a run resume from the last finished part. The example uses a short list and parts of three names, written to a temporary directory; a real run uses parts of tens of thousands: ``` r all_names <- c(checked, "Fraxinus excelsior", "Tilia cordata", "Sorbus aucuparia") parts <- split(all_names, ceiling(seq_along(all_names) / 3)) out_dir <- file.path(tempdir(), "taxify-parts") dir.create(out_dir, showWarnings = FALSE) for (i in seq_along(parts)) { file <- file.path(out_dir, sprintf("part_%04d.rds", i)) if (file.exists(file)) next res <- taxify(parts[[i]], backbone = "wfo", verbose = FALSE) saveRDS(res, file) } files <- list.files(out_dir, pattern = "\\.rds$", full.names = TRUE) matched <- do.call(rbind, lapply(files, readRDS)) nrow(matched) #> [1] 9 ``` A part that already has a file is skipped, so running the loop again after an interruption continues where it stopped. Each name is matched on its own, so the result does not depend on how the list is split. Attaching published data inside the loop, before `saveRDS()`, keeps only one part in memory at a time. ## Memory `scripts/benchmark-memory.R` measures the memory of the R process, including what the backbone occupies outside R's own objects, which `gc()` does not report. Each stage runs in a new session. The run recorded in `scripts/benchmark-memory-results.json` used taxify 0.7.0 under R 4.6.0 on 2026-10-13, matching 5,000 names. `File` is the size on disk of the backbone that run read, in megabytes of 10^6 bytes, so it differs from the sizes `list_backbones()` reports from the manifest. The other values are megabytes above an R session with taxify loaded; the fuzzy column is the case in which every name needs fuzzy matching: |Backbone | File (MB)| Loaded| Exact match| Fuzzy match| |:--------|---------:|------:|-----------:|-----------:| |COL | 2,056| 4,283| 5,161| 7,096| |GBIF | 1,701| 4,491| 5,453| 7,223| |WFO | 775| 1,477| 1,728| 3,043| A loaded backbone takes between 1.9 and 2.6 times its file size here, for its indexes and its text, and stays loaded for the session. Fuzzy matching adds a working buffer that grows with the number of unmatched names, and is released after each call. `taxify_clear_cache()` releases the loaded backbones, for example before fitting a model on the matched result. The files stay on disk, and the next call loads them again. ## Downloads before a run A run on a server or a cluster can download what it needs first, so that the matching step does not depend on the network: ``` r taxify_download(c("col", "wfo")) taxify_download_enrichment(c("iucn", "zanne")) Sys.setenv(TAXIFY_OFFLINE = "1") ``` With `TAXIFY_OFFLINE` set, or `options(taxify.offline = TRUE)`, taxify uses the files on disk and does not check for newer versions. `taxify_refresh_manifest()` makes the next call in a long session check again. ## Where the files are `taxify_data_dir()` returns the directory that holds every backbone and dataset. It is shared by all R projects of a user, so a backbone is downloaded once per machine. The `taxify.data_dir` option or the `TAXIFY_DATA_DIR` environment variable sets another directory, for example one shared by several users of a server; the files are only read during matching, so several sessions can use them at once. `taxify_store()` lists the builds on disk. Deleting a backbone's directory removes it, and the next call that needs it downloads it again. ## Where next - [Choosing and combining backbones](backbones.html): versions, pinning and the fallback chain. - [How names are matched](fuzzy-matching.html): the fuzzy stage and its threshold.