---
title: "Choosing a cleaning rule"
description: >
  What each test and each rule removes, by taxon, record age and coordinate precision, and how to apply the rule you choose.
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Choosing a cleaning rule}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", fig.width = 6.5, fig.height = 4.5,
                      dev = "svglite", fig.ext = "svg", dev.args = list(pointsize = 14))
```

`flag_occurrences()` stores each test's result alongside each record. All records remain in the table. A cleaning rule specifies which tests to use when filtering: a record is excluded if any test in the rule flags it. Installation is covered in [Get started](OccurrenceCleaner.html).

This article compares how individual tests and cleaning rules affect the dataset, then applies a chosen rule:

1. Flag the records with every test of interest using `flag_occurrences()`.

2. Compare the share of records flagged by each test and rule using `summary()` and `plot()`.

3. Compare these shares by taxon, record age and coordinate precision using `tally_flags()` with `by` and `bins`.

4. Select records using the chosen rule with `keep_unflagged()`.

## Example

### Example records

The example records are terrestrial: 3,000 random points on land in Europe, drawn by rejecting points the `sea` test flags. Each of 60 species belongs to one kingdom, and each record has a year and the GBIF fields a download carries. Coordinates are rounded to four decimals.

The simulation gives records from before 1970 a larger coordinate uncertainty and assigns GBIF's `COORDINATE_ROUNDED` issue to 30% of them, against 3% of later records. The age and uncertainty patterns later in the article follow from these two choices. It then moves 150 records to within 3 km of five capitals, 40 to (0, 0) and 60 into the Atlantic, rounded to four decimals like the rest, and appends 120 exact copies of earlier records.

<details>
<summary>Simulation code</summary>

```{r}
library(OccurrenceCleaner)

set.seed(42)
n <- 3000
lon <- runif(3 * n, -10, 30)
lat <- runif(3 * n, 36, 60)
sea <- flag_occurrences(data.frame(decimalLongitude = lon, decimalLatitude = lat),
                        tests = "sea")$flag_sea
kingdoms <- sample(c("Animalia", "Plantae", "Fungi"), 60, TRUE, c(0.5, 0.4, 0.1))
sp <- sample(60, n, TRUE)
x <- data.frame(decimalLongitude = round(lon[!sea][1:n], 4),
                decimalLatitude = round(lat[!sea][1:n], 4),
                kingdom = kingdoms[sp],
                species = paste("species", sp),
                year = sample(1850:2024, n, TRUE, prob = exp(seq(-3, 0, length.out = 175))))
```

```{r}
old <- x$year < 1970
x$coordinateUncertaintyInMeters <- round(rlnorm(n, ifelse(old, 8, 4), 1))
x$coordinateUncertaintyInMeters[sample(n, 600)] <- NA
x$basisOfRecord <- ifelse(old, "PRESERVED_SPECIMEN", "HUMAN_OBSERVATION")
x$basisOfRecord[sample(n, 15)] <- "FOSSIL_SPECIMEN"
x$issue <- ifelse(runif(n) < ifelse(old, 0.3, 0.03), "COORDINATE_ROUNDED", "")
x$issue[sample(n, 40)] <- "TAXON_MATCH_FUZZY;GEODETIC_DATUM_ASSUMED_WGS84"
```

```{r}
capitals <- data.frame(lon = c(16.37, 2.35, 13.40, -3.70, 12.50),
                       lat = c(48.21, 48.86, 52.52, 40.42, 41.90))
i <- sample(n, 250)
cap <- i[1:150]; orig <- i[151:190]; atl <- i[191:250]
j <- sample(5, 150, TRUE)
x$decimalLongitude[cap] <- round(capitals$lon[j] + runif(150, -0.02, 0.02), 4)
x$decimalLatitude[cap] <- round(capitals$lat[j] + runif(150, -0.02, 0.02), 4)
x[orig, c("decimalLongitude", "decimalLatitude")] <- 0
x$decimalLongitude[atl] <- round(runif(60, -25, -15), 4)
x$decimalLatitude[atl] <- round(runif(60, 40, 55), 4)
x <- rbind(x, x[sample(n, 120), ])
```

</details>

```{r}
nrow(x)
head(x[, c("species", "kingdom", "decimalLongitude", "decimalLatitude", "year",
           "coordinateUncertaintyInMeters", "issue")], 3)
```

### Flag once

The tests are the CoordinateCleaner tests that need only coordinates and a species, the GBIF issue flags on coordinates and rounding, and the fossil test on the basis of record. Without `tests`, `flag_occurrences()` would run every test the columns allow; naming them here keeps the comparison to these twelve.

```{r}
cc_tests <- c("invalid", "zero", "equal", "capital", "centroid", "institution",
              "gbif_headquarters", "sea", "duplicate")
tests <- c(cc_tests, "issue_coordinate", "issue_rounded", "basis_fossil")
f <- flag_occurrences(x, tests = tests)
setdiff(names(f), names(x))
```

The result is `x` with a `flags` column and one logical `flag_<test>` column per test.

### What each test removes

`summary()` turns the flags into shares of records. Without a `rules` argument it uses one rule, `all`, holding every test that was run.

```{r}
summary(f)
```

For each test of the rule, `flagged` is the number of records carrying that test and `pct` the same as a percentage of all records. `only` counts the records for which that test is the sole flag among the rule's tests: dropping that test from the rule would keep exactly these records. The row `(any)` is the rule itself, the records that carry at least one of its tests.

A record can fail several tests at once. The 44 records at (0, 0), the 40 moved there and 4 of the copies, are flagged by `zero`, `equal` (|0| = |0|) and `sea`, so none of them is counted under `only`. The per-test shares therefore add up to more than the rule's total. The rule's `pct` minus the sum of the `pct_only` shares is the share of records flagged by more than one of its tests. `pct` says how often a test flags a record; `pct_only` says how much of the rule's loss would disappear without that test.

The same numbers as a plot: the full bar is `pct`, the dark part `pct_only`.

```{r, fig.height = 6}
plot(f)
```

### Comparing rules

Candidate rules go into `rules` as a named list. This example compares three, with names that label them for this article:

- `strict` uses all twelve tests.

- `coordinatecleaner` uses the eight tests of this set that [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/)'s `clean_coordinates()` runs by default: `invalid`, `zero`, `equal`, `capital`, `centroid`, `institution`, `gbif_headquarters` and `sea`. That default also runs an outlier test, which this example leaves out, and does not run the duplicate test.

- `minimal` uses `invalid`, `zero` and `sea`: coordinates outside the valid range, on either zero axis or near (0, 0), and in the sea. For terrestrial records such as these, the three screen for misplaced coordinates; for marine species, `sea` would remove valid records.

```{r}
rules <- list(strict = tests,
              coordinatecleaner = setdiff(cc_tests, "duplicate"),
              minimal = c("invalid", "zero", "sea"))
s <- summary(f, rules = rules)
s[s$test == "(any)", c("rule", "n", "flagged", "pct")]
```

The selection above keeps one row per rule. `plot()` draws one rule at a time. This plot and the kingdom comparison below use `coordinatecleaner`, the rule applied at the end of this article:

```{r}
plot(s, rule = "coordinatecleaner")
```

The records at (0, 0) carry several flags, so they contribute zero to every `only` count. The capital test is the sole reason for most of its records.

### By group

The arguments of `tally_flags()` pass through `summary()`, so `by` splits the shares by any column of the records. Each percentage is out of the records in its own group.

```{r}
s <- summary(f, rules = rules["coordinatecleaner"], by = "kingdom")
s[s$test %in% c("capital", "sea", "(any)"), ]
```

With grouping, `plot()` shows a grid of tests by groups, shaded by the share each test flags in each group, with the rule itself in the bottom row.

```{r, fig.height = 5.5}
plot(f, rules = rules["coordinatecleaner"], by = "kingdom")
```

### Record age and coordinate precision

Year and coordinate uncertainty stay raw columns of the records and are cut into classes only when tallying. `bins` takes the break points and adds a `<column>_bin` grouping column. Each class includes its lower break and excludes its upper one, so `1950-1970` holds the years 1950 to 1969. Values below the first break form `<1900`, values from the last break on form `>=2000`, and missing values form the class `missing`. Each percentage is out of the records in its class.

This section reads the shares under `strict`, the only one of the three rules that includes `issue_rounded`, so the share of records with GBIF's rounding issue can be followed across age and uncertainty classes.

```{r}
t <- tally_flags(f, rules = rules,
                 bins = list(year = c(1900, 1950, 1970, 2000)))
s <- summary(t)
s[s$rule == "strict" & s$test %in% c("issue_rounded", "(any)"),
  c("year_bin", "test", "n", "pct")]
```

The higher `issue_rounded` share before 1970 reproduces the rate the simulation assigned to older records.

`tally_flags()` returns the counts; `summary()` on its result gives the percentages, and `plot()` the grid. The tally is the one to keep when the counts will be summed or merged later.

```{r, fig.height = 7.5}
plot(t, rule = "strict")
```

The same works for coordinate uncertainty, `coordinateUncertaintyInMeters`, a distance in metres describing how uncertain the stated location is:

```{r}
t <- tally_flags(f, rules = rules["strict"],
                 bins = list(coordinateUncertaintyInMeters = c(100, 1000, 10000)))
s <- summary(t)
s[s$test %in% c("issue_rounded", "(any)"),
  c("coordinateUncertaintyInMeters_bin", "test", "n", "pct")]
```

The simulation drew larger uncertainties for older records, so the share with the rounding issue is highest in the classes from 1 km up.

### Where the removed records are

On a flagged data frame, `plot(type = "map")` draws the records over the Natural Earth land, coloured by whether the first rule of `rules` removes them. The map returns to `coordinatecleaner`, the rule applied below.

```{r, fig.height = 5}
plot(f, type = "map", rules = rules["coordinatecleaner"])
```

### Applying the rule

Choose the tests by what your analysis needs. Suppose the example records are used to map where each species occurs on a 10 km grid. Coordinates that may point to a capital, a country centroid, an institution, GBIF's headquarters or the sea can place a record in the wrong cell, so the example applies `coordinatecleaner`. Repeated records of a species at the same coordinates fall in the same cell and leave the map unchanged, so `duplicate` stays out of the rule.

Other analyses add other tests:

- `basis_fossil` when the analysis concerns living populations.

- `duplicate` when each combination of species and coordinates should count once, for example in a presence-only distribution model.

- `issue_coordinate` to exclude records with GBIF's georeferencing warnings.

`issue_rounded` is left out. GBIF sets `COORDINATE_ROUNDED` when the original coordinate was modified by rounding to five decimals ([GBIF issue definition](https://gbif.github.io/gbif-api/apidocs/org/gbif/api/vocabulary/OccurrenceIssue.html#COORDINATE_ROUNDED)). A precision threshold can be taken from coordinate uncertainty instead.

`keep_unflagged()` keeps the records that carry no flag from the rule. `has_flag()` gives the logical vector behind it, `TRUE` for every record carrying at least one test of the rule.

```{r}
clean <- keep_unflagged(f, rules$coordinatecleaner)
nrow(clean)
```

A cut on record age or coordinate uncertainty is a filter on the raw columns, with an explicit choice for missing values. This one keeps records from 1950 on, and records whose coordinate uncertainty is at most 10 km or not stated:

```{r}
u <- clean$coordinateUncertaintyInMeters
keep <- !is.na(clean$year) & clean$year >= 1950 & (is.na(u) | u <= 10000)
sum(keep)
```

The removed records keep their flags. Another rule built from the tests already run reads the same `f`. Adding a test that was not run, or changing buffers, `settings`, reference data or the land scale, needs a new call to `flag_occurrences()`.

To compare with `clean_coordinates()` output, both packages need the same tests, buffers and reference data. At its defaults, `clean_coordinates()` adds its outlier test and draws the sea test from the 1:50 million land, while `flag_occurrences()` uses the 1:110 million land and its own gazetteers; `land_scale = 50` and `reference = "CoordinateCleaner"` align the last two. [Compatibility with CoordinateCleaner](coordinatecleaner.html) maps the remaining arguments.

## How the shares are computed

`flag_occurrences()` returns its input with class `occ_flags`, and `summary()` returns a data frame of class `occ_summary`. `summary()` and `plot()` dispatch on these classes; rows of a summary are selected as in any data frame.

All counts come from the integer flag word $w$ of each record, in which each test $t$ has one bit. Write $m_t$ for the mask with only the bit of $t$ set, $m_T$ for the mask of a rule's tests $T$ (`flag_mask(T)`), and $\mathbin{\&}$ for bitwise AND. A record is

* flagged by test $t$ when $w \mathbin{\&} m_t \neq 0$,

* removed by the rule when $w \mathbin{\&} m_T \neq 0$,

* removed by $t \in T$ alone when $w \mathbin{\&} m_T = m_t$.

A removed record carries either one test of the rule or several. The rule's removed count minus the sum of the `only` counts over $t \in T$ is therefore the number of records flagged by more than one test of the rule.

`tally_flags()` counts these per group in one pass over the flags, which is also how it runs on on-disk output part by part (see [Large tables on disk](on-disk.html)).