--- title: "The tests in detail" description: > What each test checks, the fields it reads and the arguments that change it. output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{The tests in detail} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>", fig.width = 6.5, fig.height = 4, dev = "svglite", fig.ext = "svg", dev.args = list(pointsize = 14)) has_geosphere <- requireNamespace("geosphere", quietly = TRUE) has_ggplot <- requireNamespace("ggplot2", quietly = TRUE) ``` This article explains how each test checks a record, which fields it needs, and which arguments control its behaviour. It covers: - Eleven coordinate tests adapted from [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/) (Zizka et al. [2019](https://doi.org/10.1111/2041-210X.13152)). Their buffers use CoordinateCleaner’s defaults. The article [Compatibility with CoordinateCleaner](coordinatecleaner.html) explains differences in their results. - Eight additional tests using fields from the Global Biodiversity Information Facility (GBIF). - Two additional tests comparing a record’s location on land or at sea with the habitat associated with its genus. The remaining tests are covered in separate articles: - Four fossil tests: [Cleaning fossil records](fossils.html). - Geographic outliers, two dataset tests, and `aohi`: [Outliers and dataset tests](outliers.html). - Species ranges (`range`): [Custom reference points and polygons](gazetteers.html). Each test also has its own help page, such as [`?test_capital`](https://gillescolling.com/OccurrenceCleaner/reference/test_capital.html) and [`?test_sea`](https://gillescolling.com/OccurrenceCleaner/reference/test_sea.html). The full list of tests and the integer that stores their results are at the end of the article. Installation is covered in [Get started](OccurrenceCleaner.html). ## Coordinate validity `invalid` flags a coordinate that is missing, not a number, or outside [-180, 180] x [-90, 90]. Text columns are coerced to numbers first, as in [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/)'s `cc_val`: ```{r invalid} library(OccurrenceCleaner) x <- data.frame(species = "a", decimalLongitude = c("16.37", "abc", NA, "200", "-180"), decimalLatitude = c("48.21", "10", "10", "10", "-90")) f <- flag_occurrences(x, tests = c("invalid", "zero", "sea")) f[, c("decimalLongitude", "decimalLatitude", "flag_invalid", "flag_zero", "flag_sea")] ``` A record that fails `invalid` gets no other spatial flag: there is no point to test, so `zero` and `sea` stay `FALSE` for the second to fourth rows. CoordinateCleaner instead returns `NA` or "clean" for such rows depending on the test; [Compatibility with CoordinateCleaner](coordinatecleaner.html) lists the cases. The boundary values (-180, -90) are valid. ## Zero and equal coordinates `zero` flags records with a longitude or latitude of exactly zero, and records within 0.5 degrees of (0, 0), measured as a planar distance in degrees. `equal` flags records whose absolute longitude equals their absolute latitude, a common data-entry error where one value was copied into both fields: ```{r zero-equal} x <- data.frame(species = "a", decimalLongitude = c(0, 25.3, 0.3, 0.4, 10, -10, 10.0001), decimalLatitude = c(42.1, 0, 0.3, 0.4, 10, 10, 10)) f <- flag_occurrences(x, tests = c("zero", "equal")) cbind(x[, 2:3], dist_deg = round(sqrt(x[, 2]^2 + x[, 3]^2), 3), f[, c("flag_zero", "flag_equal")]) ``` The point at (0.3, 0.3) is 0.42 degrees from the origin and flagged, the one at (0.4, 0.4) is 0.57 degrees away and not. The radius is set with `buffers = list(zero = ...)` in degrees: ```{r zero-buffer} flag_occurrences(x[3:4, ], tests = "zero", buffers = list(zero = 1))$flag_zero flag_occurrences(x[3:4, ], tests = "zero", buffers = list(zero = 0))$flag_zero ``` With `zero = 0` only exact zeros are flagged, which is what [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/) 3.0.1 does in practice; [Compatibility with CoordinateCleaner](coordinatecleaner.html) explains why its documented 0.5-degree buffer is not applied there. ## Proximity to reference points `capital`, `centroid`, `institution` and `gbif_headquarters` flag records within a fixed distance of a reference point: 10 km of a country capital, 1 km of a country or province centroid, 100 m of a biodiversity institution, and 1 km of the GBIF headquarters in Copenhagen (12.58 E, 55.67 N). Records end up at such points when a collector, a database or a georeferencing tool assigns the coordinates of the place that holds or names the specimen instead of where it was found. The distance is the geodesic distance on the WGS84 ellipsoid, and the buffer is closed: a record at exactly the buffer distance is flagged. Candidates are found with a kd-tree on the unit sphere, and each candidate is decided with the Vincenty inverse formula. The GBIF headquarters, a single fixed point, show the edge directly. Points due north of it at 990 m to 1,010 m: ```{r gbif-edge, eval = has_geosphere} d <- c(990, 999, 999.9, 1000.1, 1001, 1010) p <- geosphere::destPoint(c(12.58, 55.67), b = 0, d = d) x <- data.frame(species = "a", decimalLongitude = p[, 1], decimalLatitude = p[, 2]) data.frame(distance_m = d, flag = flag_occurrences(x, tests = "gbif_headquarters")$flag_gbif_headquarters) ``` The flag switches between 999.9 m and 1000.1 m, at the same distance in every direction. [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/) draws each buffer as a polygon with vertices on the circle, so its edge lies between 99.7 % and 100 % of the buffer depending on the bearing; records in that thin band are the only difference between the two packages for these tests. Buffers are changed per test, in metres: ```{r buffers, eval = has_geosphere} flag_occurrences(x, tests = "gbif_headquarters", buffers = list(gbif_headquarters = 995))$flag_gbif_headquarters ``` A buffer of zero flags only records at the reference coordinates themselves. The size of the capital buffer matters more than any other, because 10 km around a capital covers whole cities with real records in them. Records spread evenly over the 40 km around Vienna show how the flagged share grows with the buffer: ```{r capital-sweep, eval = has_geosphere && has_ggplot, fig.height = 3.5} set.seed(1) m <- 3000 p <- geosphere::destPoint(c(16.37, 48.21), b = runif(m, 0, 360), d = 40000 * sqrt(runif(m))) x <- data.frame(species = "a", decimalLongitude = p[, 1], decimalLatitude = p[, 2]) radii <- c(1, 2, 5, 10, 15, 20, 30) * 1000 share <- vapply(radii, function(r) mean(flag_occurrences(x, tests = "capital", buffers = list(capital = r))$flag_capital), numeric(1)) library(ggplot2) ggplot(data.frame(km = radii / 1000, share = share), aes(km, share)) + geom_line(linewidth = 1) + geom_point(size = 2.5) + scale_y_continuous(labels = function(v) paste0(100 * v, " %")) + labs(x = "capital buffer (km)", y = "records flagged") + theme_minimal(base_size = 17) + theme(panel.background = element_rect(fill = "transparent", colour = NA), plot.background = element_rect(fill = "transparent", colour = NA)) ``` For records spread evenly over a disc, the share grows with the square of the buffer. `r if (exists("share")) paste0(round(100 * share[radii == 10000], 1), " %")` of the records lie within the default 10 km, close to the 6.25 % of the area that a 10 km circle covers in a 40 km disc (10^2^/40^2^); with 3,000 records the sampling error of that share is about 0.4 percentage points. At 30 km the share rises above the disc ratio (56 %) because the eastern edge of the disc lies within 30 km of Bratislava, 56 km from Vienna, and those records are flagged for the second capital. On real data the share depends on how the records are distributed around the capital, which the per-test tally of `tally_flags()` reports directly. ## Land and sea `sea` flags records outside the land polygons of Natural Earth. The land layer comes at three scales: 1:110 million (the default, as in [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/)'s `cc_sea()`; its `clean_coordinates()` uses 1:50 million), 1:50 million and 1:10 million, set with `land_scale`. The test is a planar point-in-polygon test in longitude and latitude, through a grid index whose boundary cells are resolved exactly. The scale decides the result for records near the coast. The GBIF headquarters in Copenhagen are on land at 1:110 million and in the sea at 1:50 million: ```{r sea-scale} x <- data.frame(species = "a", decimalLongitude = c(12.58, 16.37, -30), decimalLatitude = c(55.67, 48.21, 40)) data.frame(x[, 2:3], sea_110 = flag_occurrences(x, tests = "sea")$flag_sea, sea_50 = flag_occurrences(x, tests = "sea", land_scale = 50)$flag_sea) ``` The 1:10 million layer is downloaded once on first use (`download_reference()`) and cached. The reference-data article measures how many coastal records change between scales. ## Habitat at sea and on land `sea` flags every record off the land, whales and corals included. `sea_nonmarine` flags only the records off the land whose genus does not live in the sea, and `land_marine` the records on land whose genus lives only in the sea. The habitats come from the Interim Register of Marine and Nonmarine Genera (IRMNG; Rees 2026), and `reference_habitat()` lists the genera with the rule that classes them. Genera that IRMNG gives both habitats or no flags, such as gulls, seals or mangroves, are not tested. The genus is the first word of the `genus` column, so `occ_columns(genus = "species")` reads it from a binomial, and a `kingdom` column, when present, tells apart genus names used in two kingdoms (*Asterias* is a starfish and a plant genus): ```{r habitat} x <- data.frame(kingdom = c("Plantae", "Animalia", "Animalia", "Animalia"), genus = c("Quercus", "Asterias", "Asterias", "Larus"), decimalLongitude = c(-30, 16.37, -30, -30), decimalLatitude = c(40, 48.21, 40, 40)) f <- flag_occurrences(x, tests = c("sea", "sea_nonmarine", "land_marine")) f[, c("genus", "flag_sea", "flag_sea_nonmarine", "flag_land_marine")] ``` The oak in the Atlantic fails both `sea` and `sea_nonmarine`; the starfish in Vienna fails `land_marine`; the starfish and the gull in the Atlantic fail only `sea`. The land is the layer of `sea`, so `land_scale` moves coastal records here too, and `buffers = list(sea_nonmarine = m, land_marine = m)` leaves records within `m` metres of the coast untested. Fossils of a living marine genus lie on land as well, and `land_marine` flags them. A rule that includes `basis_fossil` excludes fossil specimens in any case; a study that keeps its fossils should leave `land_marine` out of the rule, or apply the rule to the fossil records separately. ## Urban areas `urban` flags records inside the Natural Earth 1:50 million urban-area polygons. It runs whenever coordinates are present. Records in cities are often correct, so a rule can leave `urban` out; as a separate column it can remove garden and street-tree records of plants. [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/)'s `clean_coordinates()` leaves this test out by default: ```{r urban} x <- data.frame(species = "a", decimalLongitude = c(16.37, 16.51, 15.3), decimalLatitude = c(48.21, 48.25, 47.5)) flag_occurrences(x, tests = c("urban", "capital"))[, c("decimalLongitude", "decimalLatitude", "flag_urban", "flag_capital")] ``` The first point is central Vienna. The second is in the north-east of the city, 11 km from the gazetteer's point for Vienna (16.365 E, 48.202 N): still inside the urban polygon, but outside the capital buffer. The third is in rural Styria. ## Country codes `country_mismatch` flags records whose coordinates fall outside the country named by their country code, using the Natural Earth 1:50 million country polygons. The record's code is compared with the polygon's ISO alpha-2 code (as GBIF's `countryCode` gives it), its ISO alpha-3 code (as [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/) expects), and Natural Earth's own `ADM0_A3` code, which is filled where ISO has none. Matching is case-insensitive and ignores surrounding spaces: ```{r country} x <- data.frame(species = "a", decimalLongitude = c(16.37, 16.37, 16.37, 16.37, 16.37, -30), decimalLatitude = c(48.21, 48.21, 48.21, 48.21, 48.21, 40), countryCode = c("AT", "AUT", " at ", "DE", NA, "PT")) f <- flag_occurrences(x, tests = "country_mismatch") f[, c("countryCode", "flag_country_mismatch")] ``` A record without a code is flagged, and so is a record at sea whatever its code, as in CoordinateCleaner. Both are often what a rule should remove, but in a table where many records lack a code the test flags all of them; the per-test tally shows how many before the test goes into a rule. The column is named through `occ_columns(country = ...)`. ## Duplicates `duplicate` flags every record that has the same longitude, latitude and species as an earlier record in the input. The first occurrence is kept unflagged, so removing the flagged records leaves one record per distinct key: ```{r dupl} x <- data.frame(species = c("a", "a", "a", "b", "a"), decimalLongitude = c(16.37, 16.37, 16.37, 16.37, 16.3701), decimalLatitude = 48.21, eventDate = c("2020-05-01", "2020-05-01", "2021-06-03", "2020-05-01", "2020-05-01")) f <- flag_occurrences(x, tests = "duplicate") f$flag_duplicate ``` Two arguments change what counts as the same record. `duplicate_by` adds columns that must also match, as [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/)'s `additions`, so repeat visits on different dates are kept apart. `duplicate_cell` bins coordinates to a grid of that size in degrees first, so records in the same cell count as duplicates: ```{r dupl-args} flag_occurrences(x, tests = "duplicate", duplicate_by = "eventDate")$flag_duplicate flag_occurrences(x, tests = "duplicate", duplicate_cell = 0.01)$flag_duplicate ``` With `eventDate` as an addition only the second record is a duplicate. With a 0.01-degree cell the fifth record, about 7 m east of the others, joins them. The species column (`occ_columns(species = ...)`) can be a name or a taxon key; a key avoids treating two spellings of one name as different species. ## GBIF issue flags GBIF attaches issue names to each record while processing it. Four tests group them: `issue_coordinate` for georeferencing problems, `issue_taxon` for taxon matching, `issue_date` for dates and `issue_rounded` for rounded coordinates. The `issue` column can be a string of names separated by `;` (or `,`, `|`, space) or a list column of character vectors; case does not matter: ```{r issues} x <- data.frame(issue = c("COORDINATE_ROUNDED;GEODETIC_DATUM_ASSUMED_WGS84", "TAXON_MATCH_FUZZY", "recorded_date_unlikely", "", NA, "PRESUMED_SWAPPED_COORDINATE;COUNTRY_COORDINATE_MISMATCH")) iss <- c("issue_coordinate", "issue_taxon", "issue_date", "issue_rounded") f <- flag_occurrences(x, tests = iss) cbind(issue = x$issue, f[paste0("flag_", iss)]) ``` Each test flags a record if its GBIF `issue` field contains any of the names listed below. ```{r issue-groups, echo = FALSE, results = "asis"} groups <- OccurrenceCleaner:::.issue_groups for (g in names(groups)) { cat("**`", g, "`**\n\n", paste0("- `", groups[[g]], "`", collapse = "\n"), "\n\n", sep = "") } ``` `GEODETIC_DATUM_ASSUMED_WGS84` records GBIF’s assumption about the coordinate reference system. It is excluded from these groups. Rounding has its own test so it can be considered separately when choosing a cleaning rule. GBIF sets `COORDINATE_ROUNDED` when it rounds coordinates to five decimal places. To filter by the reported uncertainty in a record’s location, use `coordinateUncertaintyInMeters`. ## Basis of record and establishment means Four tests read two Darwin Core fields. `basis_fossil` and `basis_living` flag fossil and living specimens (`basisOfRecord` of `FOSSIL_SPECIMEN` or `LIVING_SPECIMEN`), which are usually not evidence of a wild population at the recorded place. `establishment_introduced` flags `establishmentMeans` of introduced, naturalised or invasive, and `establishment_managed` flags managed or cultivated individuals: ```{r basis} x <- data.frame(basisOfRecord = c("HUMAN_OBSERVATION", "LIVING_SPECIMEN", "FOSSIL_SPECIMEN", "PRESERVED_SPECIMEN"), establishmentMeans = c(NA, "MANAGED", "NATIVE", "Naturalised")) f <- flag_occurrences(x, tests = c("basis_fossil", "basis_living", "establishment_introduced", "establishment_managed")) f[, -(1:3)] ``` Whether introduced records should be removed depends entirely on the question: a native-range model drops them, an invasion study keeps only them. The flag is there so the choice is made in the rule. ## Choosing columns Use `occ_columns()` to map your column names to the fields a test needs. Here, `X` and `Y` hold longitude and latitude, and `taxon` holds the species: ```{r columns} x <- data.frame(taxon = "a", X = 16.37, Y = 48.21) flag_occurrences( x, tests = c("capital", "duplicate"), columns = occ_columns(lon = "X", lat = "Y", species = "taxon") )$flags ``` By default, `occ_columns()` uses Darwin Core names such as `decimalLongitude`, `decimalLatitude`, and `species`. It looks for an exact match first, then tries a case-insensitive match. The `needs` column of `flag_tests()` lists the fields each test requires. An explicitly requested test raises an error if a required column is missing. When `tests` is omitted, the package selects the tests supported by the available columns. ## All tests and the flag word `flag_tests()` lists every test: its definition with its default buffer, the fields it needs, and its [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/) counterpart in the `equivalent` column. The table also has a `source` column, left out here, and a `bit` column, explained below: ```{r layout} catalogue <- flag_tests()[, c("bit", "test", "definition", "needs", "equivalent")] ``` ```{r layout-table, echo = FALSE} knitr::kable(catalogue, row.names = FALSE) ``` Tests with an entry in the `equivalent` column follow [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/)’s record tests and default buffers. Differences are documented in [Compatibility with CoordinateCleaner](coordinatecleaner.html). The remaining tests use GBIF’s `issue`, `basisOfRecord` and `establishmentMeans` fields, or compare the land polygons with the habitat of the record’s genus (`sea_nonmarine`, `land_marine`). Every test owns one bit of an integer, the `bit` column. A record's flag word is the sum of `2^bit` over the tests it fails, so a word of 10 (`2^1 + 2^3`) means `zero` and `capital`. `flag_mask()` turns a set of tests into such a word and `has_flag()` asks whether a word carries any test of a set: ```{r mask} flag_mask(c("zero", "capital")) has_flag(c(0L, 2L, 8L, 10L, 16L), c("zero", "capital")) ``` The layout carries a version number, `attr(flag_tests(), "layout_version")`. A change to it (a new bit, a redefined test) increases the version, and flags written to disk record the version they were made with, so a stored flag word is never read against the wrong layout. Without `tests`, `flag_occurrences()` runs every test whose columns are in the data, and `range` when range polygons are given. `tests` names a subset instead. `test_plan()` lists, for a table, which tests would run and the missing input of each other test: ```{r defaults} x <- data.frame(species = "a", decimalLongitude = 16.37, decimalLatitude = 48.21) p <- test_plan(x) p$test[p$run] ``` ## References Rees T (compiler) (2026). The Interim Register of Marine and Nonmarine Genera. Available from at VLIZ. Darwin Core archive of 2026-10-06, CC BY 4.0. Zizka A, Silvestro D, Andermann T, Azevedo J, Duarte Ritter C, Edler D, Farooq H, Herdean A, Ariza M, Scharn R, Svanteson S, Wengström N, Zizka V, Antonelli A (2019). CoordinateCleaner: standardized cleaning of occurrence records from biological collection databases. *Methods in Ecology and Evolution* 10:744-751. [doi:10.1111/2041-210X.13152](https://doi.org/10.1111/2041-210X.13152)