---
title: "Get started"
description: >
  Flag seven records, filter them with a cleaning rule and compare two rules.
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Get started}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

This article flags a small set of species records, removes the records a chosen cleaning rule flags, and counts what two different rules exclude. It assumes basic R and data frames. The coordinate tests follow [CoordinateCleaner](https://ropensci.github.io/CoordinateCleaner/) (Zizka et al. [2019](https://doi.org/10.1111/2041-210X.13152)), with the same definitions and default distances. The documented differences are listed in [Where the results differ](coordinatecleaner.html#where-the-results-differ).

1. **Flag** the records with `flag_occurrences()`.

2. **Inspect** the flags with `print()` and `summary()`.

3. **Filter** out flagged records, using `keep_unflagged()`.

4. **Compare** rules with `tally_flags()`.

## Installation

```{r install, eval = FALSE}
# install.packages("pak")
pak::pak("gcol33/OccurrenceCleaner")
```

Needs a C++ compiler; [Installation](https://gillescolling.com/OccurrenceCleaner/#installation) shows how to set one up.

## Flagging a data frame

The example has seven records of four tree species, each with an identifier. Coordinates are longitude and latitude in decimal degrees. The column names follow Darwin Core, the standard set of field names for biodiversity data.

```{r}
library(OccurrenceCleaner)

x <- data.frame(
  id = 1:7,
  species = c("Quercus robur", "Quercus robur", "Fagus sylvatica",
              "Fagus sylvatica", "Pinus nigra", "Pinus nigra",
              "Picea abies"),
  decimalLongitude = c(14.05, 0, 16.37, 16.37, 200, -30, 13.06),
  decimalLatitude = c(47.85, 0, 48.21, 48.21, 45, 40, 47.08)
)
```

`flag_occurrences()` runs the tests named in `tests`. The first three need no reference data:

- `invalid`: a coordinate is missing, not numeric, or outside the range of longitude and latitude.

- `zero`: longitude or latitude is exactly zero, or the point lies within 0.5 degrees of (0, 0).

- `duplicate`: the record has the same species and coordinates as an earlier record.

```{r}
f <- flag_occurrences(x, tests = c("invalid", "zero", "duplicate"))
f
```

The result is `x` with the same rows in the same order and its columns unchanged, plus the flag columns described below. The printed display also shows a `flagged_by` column naming the tests that flag each record; the object itself has no such column. Record 2 has zero coordinates, record 4 repeats record 3, and record 5 has a longitude of 200 degrees.

A record without a flag has passed only the tests that were run. Record 3 lies in Vienna but carries no flag, because no test against capitals has run yet. A record can also stay unflagged when a test ran but could not judge it: a fossil without an age, or a species with too few records for the outlier test. So `FALSE` in a flag column means that the test did not flag the record. It does not mean that the record was examined and found correct.

Each test also has a logical column, `flag_<test>`, which is `TRUE` where the test flagged the record. The `flags` column combines the results of all tests for a record into one integer; [The tests in detail](tests.html) explains the encoding.

```{r}
f$flag_duplicate
```

## Filtering records

A cleaning rule is the set of tests whose flags exclude a record. `keep_unflagged()` keeps the records that no test in the rule flags. It also checks that every named test was run on the flagged table, and stops with an error if one was not:

```{r}
rule <- c("invalid", "zero", "duplicate")
cleaned <- keep_unflagged(f, rule)
cleaned[, c("id", "species", "decimalLongitude", "decimalLatitude")]
```

Record 3 is kept and record 4 is excluded: the `duplicate` test flags only the repeats, so one copy of each record remains.

## Running every test the columns allow

Without `tests`, `flag_occurrences()` runs every test whose columns are in the data. Running a test only adds its flags, so the default checks for every potential problem the columns allow. Which records to exclude is a separate choice, made afterwards with a cleaning rule.

With coordinates and species, the default adds tests against reference points and polygons: country capitals, country centroids, biodiversity institutions, the headquarters of the Global Biodiversity Information Facility (GBIF), urban areas, the land, and the Artificial Hotspot Occurrence Inventory (Park et al. 2023), a list of coordinates that recur in GBIF records for reasons such as georeferencing defaults. It also runs the test for geographic outliers within a species. The printed header names the columns whose tests could not run, such as a country code or GBIF's reported issues, and `test_plan()` lists those tests with the reason for each.

```{r}
f <- flag_occurrences(x)
f
```

Records 3 and 4 lie in Vienna, within 10 km of a capital and inside an urban area. Record 6 is in the Atlantic. Record 2, at (0, 0), is also at sea, has equal longitude and latitude, and lies within 10 km of an inventory point. The outlier test flags no record. Records 1 and 7 pass every test run here.

The `sea` test compares each record with land polygons from Natural Earth, at the 1:110 million scale by default. Near coasts and on small islands a finer scale (`land_scale = 50` or `10`) can change the flag; [Reference data](reference-data.html) maps the difference.

A flagged record may still be valid: a beech can grow within 10 km of a capital, so you decide which tests suit your analysis. A rule for species distribution modelling might exclude fossil and living specimens along with coordinate problems, and another study can choose a different set.

`summary()` counts the records each test flags, and in the `only` column the records that test alone flags. Without a rule, it counts under one rule named `all` that lists every test that was run. It only counts: `f` still contains all seven records.

```{r}
summary(f)
```

A rule excludes each record once, however many of its tests flag it. Record 2 is flagged by four tests and record 4 by three, so the per-test counts add up to 11 while the `(any)` row excludes 5 records.

## Comparing cleaning rules

Several rules can be compared on the same flags without running the tests again. Here the strict rule adds the `capital` and `sea` tests to the mild one. `tally_flags()` counts, per species, the records each rule removes:

```{r}
strict <- c("invalid", "zero", "capital", "sea", "duplicate")
mild <- c("invalid", "zero", "duplicate")
t <- tally_flags(f, by = "species",
                 rules = list(strict = strict, mild = mild))
t
```

Under the strict rule both beech records go; under the mild rule one stays. `tally_flags()` also cuts numeric columns such as the record year into bins, which [Choosing a cleaning rule](rules.html) shows.

## Changing a test's distance

The distances are part of the tests, so changing one means flagging again. The `capital` test flags records within 10 km of a capital by default. With 200 m, the Vienna records pass it:

```{r}
f200 <- flag_occurrences(x, buffers = list(capital = 200))
f200[3:4, ]
```

The capital distance is given in metres, as are the other proximity distances; the `zero` distance is in degrees. The help page of `flag_occurrences()` lists the default of each test.

## Flagging other data

Other data usually arrives as a file with its own column names. The example below writes the records to a CSV file with the columns `taxon`, `lon` and `lat`, then reads it back as any CSV file would be read:

```{r}
csv <- tempfile(fileext = ".csv")
write.csv(data.frame(taxon = x$species, lon = x$decimalLongitude,
                     lat = x$decimalLatitude),
          csv, row.names = FALSE)
y <- read.csv(csv)
```

Columns are found by their Darwin Core names regardless of capitalisation. The file has none of them, so the call stops:

```{r, error = TRUE}
flag_occurrences(y)
```

The message names the coordinate columns it looked for and points to `occ_columns()`, which maps other names onto the fields the tests read. For another file only the `read.csv()` path and these names change:

```{r}
cols <- occ_columns(lon = "lon", lat = "lat", species = "taxon")
flag_occurrences(y, columns = cols)
```

With the three columns mapped, the same 12 tests run as on `x`. A mapping that leaves out a column gives no error: the tests that need that column do not run. Here the species column is left out:

```{r}
cols_xy <- occ_columns(lon = "lon", lat = "lat")
print(flag_occurrences(y, columns = cols_xy), n = 3)
```

The header counts 10 tests instead of 12 and lists `species` among the missing inputs: the `duplicate` and `outlier` tests did not run. Adding `species = "taxon"`, as in `cols`, runs them again.

## Fixing common errors

Two more errors stop a call before anything runs. An unknown test name is one:

```{r, error = TRUE}
flag_occurrences(x, tests = "capitals")
```

`flag_tests()` lists the valid names; the capital test is called `capital`:

```{r}
fc <- flag_occurrences(x, tests = "capital")
fc$flag_capital
```

A rule that names a test which was not run also stops with an error, because the rule would otherwise exclude no records for that test:

```{r, error = TRUE}
f3 <- flag_occurrences(x, tests = c("invalid", "zero"))
has_flag(f3, c("zero", "sea"))
```

The fix is to run `flag_occurrences()` with the test, or to drop it from the rule. Here the `sea` test is added:

```{r}
f3 <- flag_occurrences(x, tests = c("invalid", "zero", "sea"))
has_flag(f3, c("zero", "sea"))
```

## Checks based on GBIF metadata

Some tests read GBIF fields instead of the coordinates. `basis_fossil` flags records whose basis of record is a fossil specimen. The example adds a `basisOfRecord` column to a copy of the records, `xb`, and marks record 7 as a fossil:

```{r}
basis <- c(rep("HUMAN_OBSERVATION", 6), "FOSSIL_SPECIMEN")
xb <- data.frame(x, basisOfRecord = basis)
fb <- flag_occurrences(xb, tests = "basis_fossil")
fb$flag_basis_fossil
```

[Cleaning GBIF data](gbif-workflow.html) uses these tests on a GBIF download, including the issue warnings and establishment means.

## Flagging a table on disk

`flag_occurrences()` also takes a path to a table on disk, such as the directory of Parquet files of a GBIF snapshot. Each file is flagged and written separately to `out`, and `keep` names the input columns copied next to the flags.

The example writes the seven records to a directory `src` as two files. The files use the `.vtr` format of vectra, the column-store package OccurrenceCleaner reads and writes tables on disk with:

```{r}
src <- tempfile("records")
dir.create(src)
vectra::write_vtr(x[1:4, ], file.path(src, "part-1.vtr"))
vectra::write_vtr(x[5:7, ], file.path(src, "part-2.vtr"))
```

The call is the same as for a data frame, with the path and an output directory. The identifier is looked for under its GBIF name `gbifID` by default, so `columns` maps it to `id`:

```{r}
out <- flag_occurrences(src, out = tempfile("flags"),
                        columns = occ_columns(id = "id"),
                        keep = "species", verbose = FALSE)
out
```

For a path, the result describes the output directory instead of containing the records. The files in it store the identifier, the `keep` columns and the flags of every record, and `tally_flags()` reads them like a data frame. [Large tables on disk](on-disk.html) continues from here: tallying from disk, duplicates across files, joining flags back to records and resuming an interrupted run.

## Where next

- [Cleaning GBIF data](gbif-workflow.html): a GBIF download from start to the retained records.

- [Choosing a cleaning rule](rules.html): what each test and rule removes, by taxon, record age and coordinate precision.

- [Large tables on disk](on-disk.html): a whole GBIF snapshot, resuming, and reading the flags.

- [Cleaning fossil records](fossils.html): the age tests for Paleobiology Database records.

- [The tests in detail](tests.html): the definition and required columns of every test; `flag_tests()` lists them in R.

- [Reference data](reference-data.html): where the capitals, centroids, institutions and land polygons come from, their versions, and how the land scale changes the `sea` test.

- [Compatibility with CoordinateCleaner](coordinatecleaner.html): where the two packages agree, how that is tested, and where results differ.