--- title: "Requesting GBIF data for a species list" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Requesting GBIF data for a species list} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- This article shows how to request occurrence records from the Global Biodiversity Information Facility (GBIF) for a species list. GBIF identifies each taxon in its backbone by an integer, the taxon key. `taxify()` matches the names to keys offline, `gbif_request()` asks GBIF for the records under those keys, and `gbif_fetch()` downloads them with the input names attached. It ends with the records and the DOI to cite them by. Installation is covered in [Get started](taxify.html). Sending a request needs the [rgbif](https://docs.ropensci.org/rgbif/) package and, for a download, a free GBIF account; the examples that contact GBIF are shown without their output. 1. **Match** the names against the GBIF backbone with `taxify()`. 2. **Check** the keys behind each name with `taxify_ids()`, and the size of the request with `dry_run = TRUE`. 3. **Request** the records with `gbif_request()` and download them with `gbif_fetch()`. 4. **Cite** the download with `cite()`. ``` r library(taxify) ``` ## Matching the names to keys The list has four plants: ``` r spp <- c("Quercus robur", "Pinus sylvestris", "Bellis perennis", "Morus alba") matched <- taxify(spp, backbone = "gbif", verbose = FALSE) matched[, c("input_name", "accepted_name", "accepted_id", "n_ids")] #> input_name accepted_name accepted_id n_ids #> 1 Quercus robur Quercus robur 2878688 3 #> 2 Pinus sylvestris Pinus sylvestris 5285637 6 #> 3 Bellis perennis Bellis perennis 3117424 1 #> 4 Morus alba Morus alba 5361889 1 #> Sources: GBIF 2023.08 | cite() for full citations ``` `accepted_id` holds the GBIF key of each accepted taxon. `n_ids` counts the keys of the name: *Quercus robur* has three, *Pinus sylvestris* six. ## Several keys for one name A name has several keys when the GBIF backbone holds several records of it: doubtful duplicates beside the accepted record, or the same name used by different authors for different taxa. `taxify_ids()` lists one row per name and key, with the number of records GBIF held under each key when the backbone was built: ``` r ids <- taxify_ids(matched, verbose = FALSE) ids[, c("input_name", "accepted_id", "taxonomic_status", "n_occurrences", "role")] #> input_name accepted_id taxonomic_status n_occurrences role #> 1 Quercus robur 2878688 ACCEPTED 1725528 pick #> 2 Quercus robur 7586523 DOUBTFUL 9 same_name #> 3 Quercus robur 7911626 DOUBTFUL 2 same_name #> 4 Pinus sylvestris 5285637 ACCEPTED 1204947 pick #> 5 Pinus sylvestris 7718215 DOUBTFUL 0 same_name #> 6 Pinus sylvestris 8116613 DOUBTFUL 0 same_name #> 7 Pinus sylvestris 9079676 DOUBTFUL 0 same_name #> 8 Pinus sylvestris 8251433 DOUBTFUL 0 same_name #> 9 Pinus sylvestris 7393648 DOUBTFUL 0 same_name #> 10 Bellis perennis 3117424 ACCEPTED 1107989 pick #> 11 Morus alba 5361889 ACCEPTED 51610 pick ``` `role` says what each key is to the name. `pick` is the key in `accepted_id`. `same_name` is another record of the same taxon, here GBIF's doubtful records of *Quercus robur* and *Pinus sylvestris*. A third role, `other_taxon`, is a record of the name that leads to a different species. *Absinthium vulgare* was published by Lamarck and again by Dulac, and GBIF files the two under different species: ``` r abs <- taxify("Absinthium vulgare", backbone = "gbif", verbose = FALSE) taxify_ids(abs, verbose = FALSE)[ , c("accepted_name", "n_occurrences", "role", "pick_basis")] #> accepted_name n_occurrences role pick_basis #> 1 Artemisia absinthium 109694 pick priority #> 2 Artemisia vulgaris subsp. vulgaris 35385 other_taxon priority ``` `pick_basis` says what decided the pick: here `priority`, Lamarck's name of 1779 before Dulac's of 1867. The count of an accepted key includes the records of its synonyms and of the taxa below it, which is what a request by that key returns. ## What a request covers By default (`keys = "taxon"`) a request covers the `pick` and `same_name` keys and leaves out `other_taxon` keys. `dry_run = TRUE` returns the keys a request would send, and the message estimates the number of records from the counts above, without contacting GBIF: ``` r gbif_request(matched, dry_run = TRUE) #> 4 name(s): 4 picked key(s) and 7 other key(s) of the same taxa requested. #> 11 GBIF key(s) from 4 matched name(s). About 4,090,085 occurrence record(s) across them. #> [1] 2878688 7586523 7911626 5285637 7718215 8116613 9079676 8251433 7393648 #> [10] 3117424 5361889 ``` `keys = "pick"` sends only the picked key of each name, and `keys = "all"` adds the other taxa: ``` r gbif_request(matched, keys = "pick", dry_run = TRUE) #> 4 name(s): 4 picked key(s) and 0 other key(s) of the same taxa requested. Left out under keys = "pick": 7 same-taxon key(s) (Quercus robur, Pinus sylvestris). #> 4 GBIF key(s) from 4 matched name(s). About 4,090,074 occurrence record(s) across them. #> [1] 2878688 5285637 3117424 5361889 ``` For this list the two choices differ by eleven records under the doubtful *Quercus robur* keys, so sending every key of the taxon costs little and keeps those records. ## Results matched against another backbone The default chain of `taxify()` starts with the Catalogue of Life Extended Release (COL XR), whose IDs are not GBIF keys. COL XR records the GBIF key of each record, so `gbif_request()` reads the key from there and, with the GBIF backbone installed, adds GBIF's other records of the same taxon: ``` r default <- taxify(spp, verbose = FALSE) gbif_request(default, dry_run = TRUE) #> 4 name(s): 4 picked key(s) and 7 other key(s) of the same taxa requested. Left out under keys = "taxon": 6 other-taxon key(s) (Quercus robur, Pinus sylvestris). #> 11 GBIF key(s) from 4 matched name(s). About 4,090,085 occurrence record(s) across them. #> [1] 2878688 7911626 7586523 5285637 8116613 8251433 7393648 9079676 7718215 #> [10] 3117424 5361889 ``` The request is the same as from the GBIF match, and the message counts the other-taxon keys it leaves out and names the input names they belong to: COL XR lists *Pinus sylvestris* and *Quercus robur* as synonyms of other species under other authors. A row whose backbone records no GBIF key is matched again against GBIF by its accepted name. `attr(x, "taxa")$key_source` records where each key came from (`backbone`, `crosswalk` or `rematch`). ## Setting up rgbif A request needs rgbif: ``` r install.packages("rgbif") ``` `method = "search"` needs nothing further. `method = "download"`, the default, needs a GBIF account, which is free at [gbif.org](https://www.gbif.org/). rgbif reads the account from three environment variables, which its documentation recommends keeping in `.Renviron`: ``` r usethis::edit_r_environ() ``` ``` GBIF_USER=yourname GBIF_PWD=yourpassword GBIF_EMAIL=name@example.org ``` Restart R after saving so the file is read. `rgbif::occ_download_list(limit = 5)` lists the account's past downloads and succeeds only when the credentials are right. When a variable is missing, `gbif_request()` stops before contacting GBIF and names it. ## Sending the request A download has no limit on the number of records and receives a DOI. `gbif_request()` submits it and returns at once, since GBIF prepares the file on its own servers; `gbif_fetch()` waits for it, downloads it, imports the records and adds the input names: ``` r dl <- gbif_request(matched) recs <- gbif_fetch(dl) ``` `method = "search"` returns records at once, without an account, up to `limit` records per key. GBIF's search limits how deep it pages, so it suits a look at a few taxa: ``` r recs <- gbif_request(matched, method = "search", keys = "pick", limit = 50) ``` The records carry a `taxon_key_requested` column with the key that returned each row. A download query is limited to 12,000 characters, about 1,187 keys. Above 1,000 keys `gbif_request()` splits them into several downloads, each with its own DOI, submitted through rgbif's queue, which runs at most three at a time. `gbif_fetch()` waits for all of them and returns the records in one table. ## Linking records to the requested names A record's `taxonKey` is the key of the taxon it was identified as, which can lie below the requested key: a record of *Pinus nigra* subsp. *salzmannii* carries that subspecies' key when *Pinus nigra* was requested. `gbif_fetch()` therefore links each record through `gbif_backmatch()`. It reads the record's keys from the most specific outwards (taxon, accepted taxon, species, genus, family) and takes the first one that was requested. The record then carries the requested key, the input name and the accepted name. A record that carries no requested key keeps `NA` in those three columns, and the message counts such records. For records fetched another way, `gbif_backmatch()` takes the request, a `taxify_ids()` table or the `taxify()` result: ``` r recs <- gbif_fetch(dl, backmatch = FALSE) recs <- gbif_backmatch(recs, matched) ``` ## Citing the download A download is cited by its DOI, which GBIF issues once the file is ready. `cite()` on the records lists the DOI next to the GBIF backbone and taxify: ``` r cite(recs) cite(recs, file = "refs.bib") ``` A download still being prepared is listed without a DOI. `file` writes the same entries as BibTeX. ## Where next - [Exploring the backbone](exploring.html): `taxify_ids()`, `id2name()` and the other functions that read the backbone. - [Matching large lists](large-scale.html): long species lists.