Temporal and grouped data

The same interface handles temporal and grouped data. BORG auto-detects common column names (date, time, site, patient_id, etc.), but you can always specify them explicitly.

sim_temporal <- borg_simulate(n = 300, type = "temporal", seed = 7)
result_temporal <- borg(
  sim_temporal$data,
  time = "time",
  target = "y",
  v = 5
)
cat("Recommended CV:", result_temporal$diagnosis@recommended_cv, "\n")
#> Recommended CV: temporal_block

For temporal data, BORG runs a Ljung-Box test to detect autocorrelation and estimates the decorrelation lag (the number of time steps after which observations become approximately independent). The recommended CV strategy ensures no future data leaks into past folds, either through temporal blocking or a purged CV scheme with an appropriate embargo period.

grouped_data <- data.frame(
  site = rep(1:30, each = 10),
  x1 = rnorm(300),
  measurement = rnorm(300)
)
result_grouped <- borg(grouped_data, groups = "site", target = "measurement", v = 5)
cat("Recommended CV:", result_grouped$diagnosis@recommended_cv, "\n")
#> Recommended CV: random

For grouped data, BORG computes the intraclass correlation coefficient (ICC), which measures how much variance is explained by group membership. An ICC of 0.5 means half the variance in the response comes from between-group differences. With high ICC, random CV creates optimistic estimates because the same group’s observations appear in both train and test. BORG switches to leave-group-out folds automatically.

When multiple dependency types coexist (spatial coordinates plus a group column), BORG detects the combined structure and recommends a strategy that respects all of them. For instance, spatial-temporal data gets a spatial-temporal blocking strategy that separates folds in both space and time.

Exporting folds to other frameworks

By default, borg_cv() returns a list of index vectors. Set the output argument to convert folds into the format your framework expects.

cv_rsample <- borg_cv(
  sim$data,
  diagnosis = result$diagnosis,
  coords = c("x", "y"),
  v = 5,
  output = "rsample"
)
class(cv_rsample)
#> [1] "borg_rset"   "manual_rset" "rset"        "tbl_df"      "tbl"        
#> [6] "data.frame"

Supported formats: "rsample" (tidymodels), "caret" (trainControl), "mlr3" (Resampling object). The conversion preserves the fold structure exactly; no resampling or re-blocking occurs. You get the same train/test splits regardless of output format. This makes it straightforward to integrate BORG’s blocked folds into an existing modelling pipeline without changing the downstream code. See vignette("frameworks") for framework-specific examples with caret’s train(), tidymodels’ tune_grid(), and mlr3’s resample().

Overlapping indices

The most basic mistake: some row indices appear in both the training and test set. This means the model is tested on data it already saw during training.

bad_risk <- borg(sim$data, train_idx = 1:200, test_idx = 150:300)
#> Error in `validate_existing_split()`:
#> ! BORG HARD VIOLATION: train_idx and test_idx overlap (51 shared indices: 150, 151, 152, 153, 154...)

borg() refuses to continue when it finds overlapping indices. It throws an error with the number of shared indices and the first few values. This fail-fast behaviour prevents you from accidentally computing metrics on a contaminated split.

For a non-stopping approach that returns a BorgRisk object instead of throwing, use borg_inspect():

bad_risk <- borg_inspect(sim$data, train_idx = 1:200, test_idx = 150:300)
bad_risk
#> BorgRisk Assessment
#> ===================
#> 
#> Status: INVALID (1 hard violation) — Resistance is futile
#>   Hard violations:  1
#>   Soft inflations:  0
#>   Train indices:    200 rows
#>   Test indices:     151 rows
#>   Inspected at:     2026-06-15 19:48:14
#> 
#> --- HARD VIOLATIONS (must fix) ---
#> 
#> [1] index_overlap
#>     Train and test indices overlap (51 shared indices). This invalidates evaluation.
#>     Source: train_idx/test_idx
#>     Affected: 51 indices (first 5: 150, 151, 152, 153, 154)
#>     Fix: Recreate train/test split with non-overlapping indices

The BorgRisk object reports is_valid = FALSE and lists the overlap as a hard violation. Metrics computed from this split are meaningless.

Target leakage

Target leakage occurs when a feature is derived from (or highly correlated with) the response variable. The classic example: including days_since_diagnosis as a predictor when the outcome is has_disease. The feature is only available after the outcome is known and would not exist at prediction time.

risk_leak <- borg(
  sim_leak$data,
  train_idx = 1:140,
  test_idx = 141:200,
  target = "y"
)
risk_leak
#> BorgRisk Assessment
#> ===================
#> 
#> Status: INVALID (1 hard violation) — Resistance is futile
#>   Hard violations:  1
#>   Soft inflations:  0
#>   Train indices:    140 rows
#>   Test indices:     60 rows
#>   Inspected at:     2026-06-15 19:48:14
#> 
#> --- HARD VIOLATIONS (must fix) ---
#> 
#> [1] target_leakage_direct
#>     Feature 'leaked_feature' has correlation 0.993 with target 'y'. Likely derived from outcome.
#>     Source: data.frame$leaked_feature
#>     Fix: Remove features derived from the target variable

The leaky feature (leaked_feature) was flagged because its correlation with the response exceeds 0.99. Removing it before modelling would fix the problem. BORG also flags “proxy leakage” (correlation between 0.95 and 0.99) as a soft warning, since these features might be legitimate strong predictors that require domain knowledge to judge.

Preprocessing leakage

Preprocessing leakage is subtler than the examples above. When you fit a scaler, PCA, or imputer on the entire dataset before splitting, the parameters (means, standard deviations, principal components) incorporate information from the test set. The test data’s distribution influences how training data is transformed. This creates a feedback loop that inflates apparent performance.

BORG inspects preprocessing objects directly. If you scaled the data before splitting, the test set’s statistics contaminated the scaler, and BORG will flag this.

full_scaled <- scale(sim$data[, c("x1", "x2", "x3")])
risk_pp <- borg_inspect(
  full_scaled,
  train_idx = 1:210,
  test_idx = 211:300,
  data = sim$data
)
risk_pp
#> BorgRisk Assessment
#> ===================
#> 
#> Status: VALID (no hard violations)
#>   Hard violations:  0
#>   Soft inflations:  0
#>   Train indices:    210 rows
#>   Test indices:     90 rows
#>   Inspected at:     2026-06-15 19:48:15
#> 
#> No risks detected.

The correct pattern: fit the scaler on training data only, then apply the same transformation to both sets. This way, the test set is transformed using parameters it did not influence. BORG validates this pattern too: pass the correctly fitted preprocessing object and BORG confirms no leakage occurred.

train_subset <- sim$data[1:210, c("x1", "x2", "x3")]
mu <- colMeans(train_subset)
s <- apply(train_subset, 2, sd)
scaled_correct <- scale(sim$data[, c("x1", "x2", "x3")], center = mu, scale = s)

risk_pp_good <- borg_inspect(
  scaled_correct,
  train_idx = 1:210,
  test_idx = 211:300,
  data = sim$data
)
cat("Valid:", risk_pp_good@is_valid, "\n")
#> Valid: TRUE

Accessing risk details

Every BorgRisk object stores a list of individual risks with their type, severity, description, and (where applicable) the affected rows or features. The is_valid slot gives a quick binary answer: TRUE if no hard violations were found, FALSE otherwise. For programmatic access, convert the object to a data frame.

df <- as.data.frame(bad_risk)
df[, c("type", "severity", "description")]
#>            type       severity
#> 1 index_overlap hard_violation
#>                                                                        description
#> 1 Train and test indices overlap (51 shared indices). This invalidates evaluation.

Area of Applicability

A model trained on data from lowland forests should not be trusted when predicting in alpine meadows. The training data simply does not cover that part of environmental space. borg_aoa() formalizes this intuition by computing a dissimilarity index (DI) for every new prediction location relative to the training data, following Meyer & Pebesma (2021). Points with DI above a threshold fall outside the area of applicability (AOA). Predictions at those points should be flagged or excluded from downstream analysis.

train_data <- sim$data[train_idx, ]
test_data <- sim$data[test_idx, ]

aoa <- borg_aoa(
  train = train_data,
  new = test_data,
  predictors = c("x1", "x2", "x3", "x4", "x5"),
  coords = c("x", "y")
)
cat("Inside AOA:", sum(aoa$aoa), "/", nrow(aoa), "\n")
#> Inside AOA: 76 / 90
autoplot(aoa)

The DI threshold is derived from the distribution of distances within the training set. Points beyond the threshold are in regions of predictor space that the training data does not cover. The autoplot() method shows a map (if coordinates are available) or a histogram of DI values, with the threshold marked.

For a complementary view, borg_extrapolation() identifies which specific variables are outside their training range at each prediction point, and borg_transferability() estimates how well the model transfers to distinct geographic regions. See vignette("aoa-extrapolation") for the full treatment.

Block-permutation variable importance

Standard permutation importance shuffles a variable’s values row by row and measures the drop in predictive performance. On autocorrelated data, this can be misleading: even after permuting one variable, nearby observations still share information through the remaining (non-permuted) spatially structured predictors. The result is that importance scores are deflated for the permuted variable and inflated for correlated neighbours.

borg_importance() fixes this by permuting entire spatial blocks instead of individual rows. All observations within a block are permuted together, breaking the local spatial signal.

# Fit a model first
model <- lm(y.1 ~ x1 + x2 + x3 + x4 + x5, data = train_data)

imp <- borg_importance(
  model = model,
  data = train_data,
  target = "y.1",
  coords = c("x", "y"),
  n_rep = 5
)
autoplot(imp)

The lollipop chart ranks variables by their block-permutation importance. Error bars reflect variability across repeated permutations (n_rep). Variables with importance near zero contribute nothing to prediction and can safely be dropped.

BORG also provides borg_forward_selection() for stepwise variable selection using blocked CV at each step, and borg_best_subset() for exhaustive search over small predictor sets. See vignette("feature-selection") for the full treatment.

Residual diagnostics

After fitting a model, check whether the residuals still carry spatial structure. If they do, the model is missing a spatially structured predictor, or the CV strategy did not adequately separate spatial neighbours. Either way, error estimates are unreliable.

resid_check <- borg_check_residuals(
  model,
  data = train_data,
  coords = c("x", "y")
)
resid_check
#> BORG Residual Autocorrelation Check
#>   Moran's I: 0.1615 (p = 5.97e-164)
#>   Assessment: MILD
#>   WARNING: Residuals have significant spatial autocorrelation.
#>   The model has not fully captured the spatial process.
autoplot(resid_check)

The variogram shows how residual semivariance changes with distance. A flat variogram (constant semivariance at all distances) means residuals are spatially independent, meaning the model has captured the spatial signal. An increasing variogram signals unmodelled spatial structure: nearby residuals are more similar than distant ones, which means the model is systematically missing a spatial pattern.

borg_check_residuals() also reports Moran’s I for the residuals and classifies the spatial signal as “clean”, “mild”, or “strong”. See vignette("diagnostics") for spatial residual checks combined with posterior predictive checks and formal model comparison.

Calibration

Discrimination (does the model rank observations correctly?) gets most of the attention. Calibration (are the predicted values accurate in absolute terms?) is equally important but rarely checked. A model can have high AUC but still predict probabilities that are systematically too high or too low.

preds <- predict(model, newdata = test_data)
cal <- borg_calibration(
  predicted = preds,
  actual = test_data$y.1,
  n_bins = 8
)
cal
#> BORG Calibration Diagnostics
#> ============================
#> 
#>   Type: regression
#>   N: 90 | Bins: 8
#>   Expected Calibration Error (ECE): 0.0235
#>   Maximum Calibration Error (MCE): 0.0444
#>   Calibration MSE: 2.3271
#>   Reliability slope: 0.911 (ideal: 1.000)
#>   Reliability intercept: -0.216 (ideal: 0.000)
#>   Assessment: well_calibrated
autoplot(cal)

The reliability diagram plots observed values against predicted values, binned into groups. A perfectly calibrated model falls on the 1:1 diagonal. Points above the diagonal indicate underprediction; points below indicate overprediction. The expected calibration error (ECE) summarizes the average deviation from the diagonal across all bins. Lower is better; ECE below 0.05 is generally considered well-calibrated.

borg_calibration() also reports the reliability slope (ideal: 1.0) and intercept (ideal: 0.0), plus the Brier score for classification or calibration MSE for regression. Three binning strategies are available: "uniform" (equal bin width), "quantile" (equal bin count), and "spatial" (bins that respect spatial structure). The spatial binning strategy groups predictions by their geographic location, which matters when calibration varies across the study area. vignette("diagnostics") covers calibration diagnostics in full detail.

Null test

Before interpreting a model’s predictive skill, verify that it beats random guessing. borg_null_test() permutes the response variable and re-evaluates performance many times to build a null distribution. This is worth checking whenever the signal is weak or the sample is small.

nt <- borg_null_test(
  data = sim$data,
  folds = result$folds,
  formula = y.1 ~ x1 + x2 + x3 + x4 + x5,
  n_null = 20
)
nt
#> BORG Null Model Significance Test
#>   Empirical RMSE: 1.7226
#>   Null mean: 2.0762 (SD 0.0206)
#>   Z-score: -17.14
#>   P-value: 0.0476 (20 permutations)
#>   Significant: YES

If the observed metric falls within the null distribution, the model has no predictive skill beyond chance. The p-value counts what fraction of permuted performances equal or exceed the observed performance. Use n_null = 100 or higher for publication-quality p-values; 20 is enough for a quick sanity check.

The null test uses the same blocked folds you pass in, so the test respects the spatial (or temporal, or grouped) structure of the data. A standard permutation test with random folds would itself be subject to the same inflation problem BORG is designed to solve.

Extrapolation detection

A lighter-weight alternative to AOA, borg_extrapolation() flags prediction points that fall outside the training data’s range in one or more variables. This is a univariate check (AOA uses multivariate dissimilarity), so it catches obvious range violations without the computational cost of a full DI computation.

extrap <- borg_extrapolation(
  train = train_data,
  new = test_data,
  predictors = c("x1", "x2", "x3", "x4", "x5"),
  coords = c("x", "y")
)
cat("Extrapolating:", sum(extrap$extrapolating), "/", nrow(extrap), "\n")
#> Extrapolating: 5 / 90

Each row reports which variables (if any) are outside the training range at that prediction point, plus its Mahalanobis distance from the training centroid. Because the Mahalanobis distance accounts for correlations between predictors, a point within range on every individual variable can still be flagged if its combination of values is unusual relative to the training distribution.

Use borg_extrapolation() as a quick pre-check before the full AOA analysis. No extrapolating points in any individual variable means the AOA will likely classify most points as inside the area of applicability. Many out-of-range points signal unreliable predictions before you invest computation time in the full multivariate analysis.

Geographic distance analysis

How well do the CV folds mimic the distance structure the model will face at prediction time? If nearest-neighbour distances in CV test-to-train splits are much shorter than prediction-to-train distances, the CV estimate is still too optimistic, even after spatial blocking. borg_geodist() visualizes both distance distributions and runs a Kolmogorov-Smirnov test for a formal check.

gd <- borg_geodist(
  data = sim$data,
  folds = result$folds,
  coords = c("x", "y")
)
gd
#> BORG Distance Distribution Diagnostic
#>   Geographic: CV median=0.12
#>   Feature:    CV median=1.14
autoplot(gd)

A close match (small KS statistic, overlapping density curves) means the CV folds are representative of the actual prediction task. A large gap means the model will face harder predictions in deployment than the CV folds suggest, and the KNNDM strategy (borg_cv(..., strategy = "knndm")) is specifically designed to minimize this gap when prediction locations are known.

Interactive maps

For spatial data, borg_leaflet() creates an interactive Leaflet map where points are coloured by fold assignment. You can toggle layers to see training vs test points for each fold, which is useful for spotting spatial patterns in the blocking that a static scatter plot might miss.

borg_leaflet(result, data = sim$data, coords = c("x", "y"))

Leaflet maps are most useful with real geographic coordinates (longitude/latitude). The simulated data here uses arbitrary x/y values, so the map places points at the origin. With real data, you get a full basemap with zoom and pan.

For static publication figures, use autoplot() instead. For interactive exploration during analysis, borg_leaflet() is faster to iterate with because you can zoom into specific regions and hover over individual points to see their fold assignment and risk status.

Methods text for publications

Writing the model evaluation paragraph of a Methods section is tedious and error-prone. You need to describe the dependency type, the statistical test used to detect it, the CV strategy and its parameters, the number of folds, and the literature references for each choice. summary() on a borg_result generates this paragraph automatically. Three citation styles are available: APA (default), Nature (superscript numbers), and Ecology (inline author-year).

methods_text <- summary(result)
#> 
#> Spatial autocorrelation was detected in the data (Moran's I = 0.21, p < 0.001) with an estimated autocorrelation range of 0.5 units. Model performance was evaluated using spatial block cross-validation (k = 5 folds). Cross-validation strategy was determined using the BORG package (version 0.3.1) for R.

summary(result, style = "nature")
summary(result, style = "ecology")

If you also ran borg_compare_cv(), pass the comparison object to include empirical inflation estimates in the methods text. This adds a sentence along the lines of “Random cross-validation inflated RMSE by X% compared to spatial blocking (paired t-test, p = Y).”

summary(result, comparison = comparison)

The returned string is a character vector that you can write directly to a file or paste into a manuscript. All references to BORG functions, statistical tests, and methodological citations are included.

When to use BORG

Use BORG any time you evaluate a predictive model on data that has spatial, temporal, or grouped structure. This includes species distribution models, remote-sensing-based land cover classification, clinical models with repeated patient measurements, environmental time series forecasting, and any situation where observations are not independent.

If your data is genuinely IID (no spatial coordinates, no time column, no grouping variable), BORG will detect "none" as the dependency type and recommend random CV. In this case, there is no penalty from using BORG; it simply confirms that random CV is appropriate.

When NOT to use BORG

BORG is designed for tabular prediction problems with a response variable and structured dependencies between observations. The following fall outside its scope:

• Image or text data where spatial/temporal structure is internal to each observation rather than between observations.
• Unsupervised learning (no target variable to evaluate).
• Reinforcement learning or online learning settings.

Rules of thumb

• Minimum 100 observations for stable spatial blocking. Below this, folds become too small and fold-to-fold variance dominates the CV estimate.
• Minimum 20 clusters for leave-group-out CV. With fewer groups, each fold removes a large fraction of the data, inflating variance.
• 5 repeats minimum for borg_compare_cv(). The paired t-test needs enough observations to distinguish a real signal from noise. Use 10 or more for publication.
• Check power before blocking. If borg_power() reports sufficient = FALSE, consider whether the cost of biased error estimates (from random CV) outweighs the cost of noisy estimates (from blocked CV with low power).
• Always simulate first. Use borg_simulate() with parameters similar to your real data (same n, similar autocorrelation strength) to calibrate your expectations. If BORG cannot detect the simulated dependency, it will also miss it in your real data.
• Include the validation certificate in your supplementary materials. It documents every diagnostic test and CV decision, which makes peer review faster and more transparent.

Core

Evaluation

Spatial analysis

Feature selection and importance

Reporting

Simulation