summary() on a tobs_fit returns tulpa's summary.tulpa_fit object
(tulpa >= 0.7.3) with the skew and rhat / ess_bulk / ess_tail
columns added to its coefficient table; coef(summary(fit)) is that table,
the data frame summary() returned before.
A random intercept or uncorrelated random slope on the joint
nested-Laplace route is read as the effect it is, sigma * u: tulpa's iid
block is non-centred, and the BLUPs, ranef() SDs, and the draws behind
predict(), waic(), loo() and cpo() had used the unit-scale latent
u (gcol33/tulpaObs#415). The BLUP SDs now also carry the within-cell
posterior variance. On the full MOT data a detection random intercept on
habitat had p_waic of 2-4e5 and ranef() SDs of ~1e-5 from this. Applies to
occu_cover() and cover().
occu_cover() on the joint nested-Laplace route fits a temporal(time, type = "rw1" | "rw2" | "ar1" | "iid") term in the occurrence formula as one temporal
block on psi (detection and cover are not touched). Before, the term was
dropped without a word and the fit carried no time structure on psi. The
block's SD is reported as sigma_temporal (and rho_temporal for AR1), its
per-interval effects as fit$re$temporal and in ranef(), and predictions,
WAIC, LOO and CPO include it. predict(type = "trajectory" | "change") moves
the temporal level with time.col, also when the term is indexed by another
column that pairs one-to-one with it in the fitted data (a time.sc
trajectory over a temporal(yr.interval) term); a time or newdata row with
no fitted level is an error. A replicated term (group =), tau.shape /
tau.rate, two temporal terms, a correlated | spatial bar, the latent cover
aggregate, and every other occurrence-formula term the joint route does not
carry are refused by name (#414).
An AR1 temporal block lays its precision and correlation axes as their tensor; the defaults (three precision nodes, two correlation nodes) were handed to the engine as unequal paired columns, which it refused (#414).
Every argument a user types is now dotted, matching n.iter, max.iter and
the rest of control. The old spellings are gone, not aliased (#406).
occu_cover() scripts must change: occu_cover(cover_aggregate = ) is
now occu_cover(cover.aggregate = ).
occu_cover() scripts must change: the cover response passed to
tobs() is now y.pos = (was y_pos =). The same key applies to
occu_multiscale_cover(), ms_occu_cover(), tobs(by = ) batches and
occu_cover_inputs(y.pos = ).
Other tobs() extras: y_rem -> y.rem (gdistremoval()), site_map ->
site.map (ms_int_occu()), det_visit_formula / det_visit_data ->
det.visit.formula / det.visit.data (ms_abun()).
Latent-count ceiling: K_max -> K.max on abun(), ms_abun(),
dyn_abun(), distance(), ms_distance(), removal(),
royle_nichols() and distsamp_open().
occu_ttd(surveyLength = ) -> occu_ttd(survey.length = ).
predict(): n_points, time_col, X_det.0 -> n.points, time.col,
X.det.0 on a tobs_fit; include_RE, time_col -> include.RE,
time.col on a cover_fit. tobs_marginal_effect(n_points = ) ->
n.points.
Priors: occu_priors() takes p.intercept, p.slope,
beta.occ.intercept, beta.occ.slope; cover_priors() takes
occ.intercept, occ.slope, pos.intercept, pos.slope. A named list
passed as priors = uses the same keys. The returned prior objects keep
their element names.
Data helpers: occu_aggregation_scan(cell_sizes, block_lengths) ->
cell.sizes, block.lengths; tobs_format_ms(species_names = ) ->
species.names.
obs_family(): class_long, default_engine, control_keys,
control_groups -> class.long, default.engine, control.keys,
control.groups.
Every simulate_*() argument is dotted (n_occ_covs -> n.occ.covs,
beta_occ -> beta.occ, n_species -> n.species, sigma_logr ->
sigma.logr, T_seasons -> T.seasons, and so on). The elements of the
returned list keep their names (sim$y_pos, sim$truth$beta_occ).
occu_cover() scripts must change: the areal terms take group.var
(was group_var): icar(graph = adj, group_var = "cell") is now
icar(graph = adj, group.var = "cell"). The same applies to bym2(),
car(), car_proper() and spatial(model = ...) on every family (#407).
The other formula-term arguments are dotted too (#407):
bym2(): scale_factor -> scale.factor.gp(): sigma2_prior_U, sigma2_prior_alpha, prior_range ->
sigma2.prior.U, sigma2.prior.alpha, prior.range.multiscale_gp(): nn_local, nn_regional -> nn.local,
nn.regional; range_local_lower, range_local_upper,
range_regional_lower, range_regional_upper -> range.local.lower,
range.local.upper, range.regional.lower, range.regional.upper;
sigma2_local_prior_U, sigma2_local_prior_alpha,
sigma2_regional_prior_U, sigma2_regional_prior_alpha ->
sigma2.local.prior.U, sigma2.local.prior.alpha,
sigma2.regional.prior.U, sigma2.regional.prior.alpha.spde(): max_edge, prior_range, prior_sigma -> max.edge,
prior.range, prior.sigma.svc(): sigma2_prior_scale, prior_range -> sigma2.prior.scale,
prior.range.re(): sigma_scale -> sigma.scale.temporal(): tau_shape, tau_rate -> tau.shape, tau.rate.latent(): n_factors, sigma_prior_rate -> n.factors,
sigma.prior.rate.share() and grid() are unchanged.
The abundance vignette's ceiling check now refits with
abun(K.max = 200); it previously passed the ceiling to tobs(), which
ignored it.
ranef() returns one layout on every family and engine: arm, group,
level, term, estimate, std.error. It used to return four: the
documented one from the single-species Laplace / NUTS paths, species, arm, term, estimate from the community families, arm, var, group, term, level, blup, blup_sd from a joint occu_cover fit and arm, group, blup, blup_sd
from a sampled count fit. group is now the grouping variable as written in
the formula (observer, not the positional p1), level its level (the
species on a community fit, under group = "species"), and std.error is
NA where the fitter reports no posterior SD.
ranef() on a community fit reports std.error, the per-species posterior
SD of each deviation, from the species' posterior covariance
fit$ms_community$Cinv[[s]] through an index map the builder records
(fit$ms_community$blup_idx). Every route fills it: the community
Laplace-EM families, ms_abun() on its default EM (which now returns the
per-species covariance) and joint AGHQ routes (the log_r / logit_omega
deviations included), the spatial routes by the law of total covariance
over the outer grid, and the NUTS and Polya-Gamma Gibbs routes from the
sampled deviations. A spatial-factor ms_occu_cover() fit now reports its
per-species deviations at all (#413).
The occu_cover() escape hatches control$engine = "v3_nested" (pure-R
outer-BFGS nested Laplace, lognormal only) and "v2_joint" (joint Laplace
MAP over the field) are removed (#402). The spatial nested_laplace path has
one fitter, the joint engine; control$engine accepts "joint" only and
any other value errors at dispatch.
An ordinal cover() fit with a | bar or a single-arm (to) field now
reports the latent SD as sigma_pos; it was NA on those two routes (#395).
control knobs splatted into the joint fitters are read by exact key. A
shorter key no longer picks up a longer one when it is unset:
n.threads.outer no longer sets the inner n.threads, sigma.grid.pos.field
no longer sets sigma.grid, and the same for re.sigma.grid, phi.grid,
n.quad, re.aghq and omega (#397).
The occu_cover() joint fitter's cover-dispersion pre-fit reads the same
values on a compact (tobs_data(compact = TRUE)) and a dense fit; on a
compact fit it used to find no rows and take its fallback start (#397).
abun() and removal() with a grouped random effect under
method = "nuts" warm-start the coefficient block on the Laplace metric; it
silently fell back to a unit metric (#401).
dyn_abun() and fp_occu() accept control = list(sigma.logr = ), the
prior scale of the random-effect log-SD. Under method = "nuts" the
dyn_abun() negative-binomial log_r takes this prior (default 1.5) instead
of sigma.beta (default 10), so its draws change (#401, #408).
Negative-binomial dispersion is clamped at |log r| <= 30 before exp() in
every NUTS target and pointwise log-likelihood, and the NUTS targets check
their input shapes before sampling (#408).
The compiled count kernels compute lgamma / digamma through tulpa's
portable math, which is safe inside OpenMP regions. Values move at the
1e-11 level against R's lgammafn (#408).
WAIC / LOO / CPO on a beta occu_multiscale_cover() fit score the cover
density the fit uses. A different clamp in the scoring kernel put pointwise
values up to 213 log units off the fitted density (#410).
WAIC / LOO / CPO on a cover() fit evaluate the positive-arm density
through tulpa's own per-observation kernel (tulpa/family_density.h), so
the lognormal, gaussian and beta arms score the function the fit maximised.
The beta arm now holds mu inside the fit's [1e-15, 1 - 1e-15]; values
move only where |eta| > 34.5 on the positive arm (gcol33/tulpa#950).
Requires tulpa >= 0.7.2.
dyn_abun() pairs a field with its engine like removal(), distance()
and fp_occu(): an areal or temporal() term under method = "laplace"
errors, as does method = "nested_laplace" with no structured term. Both
used to fit, the first on the nested-Laplace field fitter and the second as a
plain Laplace fit. dyn_abun(mixture = "zip" / "zinb") rejects an svc()
term instead of fitting without the surface (#412).
distance() with a field under method = "nuts" honours a user-set
control$sigma.logr; it was ignored (#411).
The README, API.md and the community and random-effects vignettes list the
engines each family accepts as tobs() enforces them, and every family
constructor's help page has an "Engines" section (#394, #396).
occu_cover or occu_multiscale_cover
fit with a field is accumulated in the engine
(tulpa::tulpa_joint_inner_vcov_mixture(), gcol33/tulpa#947) instead of
from one (betas + field) square block per outer-grid cell. Those blocks
grew with the grid size times the latent dimension squared: a full-data
25 km fit (878 solved cells, 4479 latents) needed about 141 GB and died
with std::bad_alloc. The covariance and the per-cell fixed-effect blocks
are the same numbers (relative difference 7e-16 on an ICAR fit), and a cell
of zero weight is no longer extracted.occu_cover, the cover hurdle and occu_multiscale_cover no longer default
control$hessian to "fisher" for a beta positive arm; an unset one takes
the engine's "auto" (tulpa 0.7.0 at 2510f4b4 and later): the observed
Hessian where it factors, the expected information where it does not. On
the full 25 km Calluna occu_cover fit (32 threads) the inner iterations
per cell went from a median of 48 to 5, no cell stops at max.iter (three
did, carrying 1.7% of the posterior mass), and the fit takes 21 against
67.6 min at a peak of 18 against 14 GB. Hyperparameter summaries agree with
the unscreened dense grid within 0.06 posterior SD.
control$hessian = "fisher" restores the expected-information step.LikelihoodSpec behind the single-season nested-Laplace
state refine now sets ll_eta_weights_fn (tulpa ABI 47), so the Laplace
loop reads the log-likelihood, score and Fisher weight of a site from one
inverse link. Value, score and weight come from one occ_site_terms()
kernel shared by the split callbacks; occ_nested_likelihood_eval() reads
the registered callbacks and test-occ-nested-likelihood-fused.R holds the
fused pass bit-identical to the split pair and its score to finite
differences. Fits are unchanged.control$prune / control$prune.tol as set, NULL when unset, so the
engine's per-door default decides on all of them; the cover hurdle route no
longer carries its own prune = FALSE beside it.count()
nested-Laplace fit errored with "non-conformable arguments" when the screen
dropped a cell: the per-cell fixed-effect modes and Hessians run parallel to
the grid weights with an empty slot for a cell never solved, and the mixture
read collapsed the slots. A batch.backend = "fused" occu_cover() batch
fell silently to the looped backend, since the fused solve walks one shared
grid and cannot screen; the batch now runs its species unscreened, and a
batch that asks for both the fused solve and the screen is refused..tobs_joint_copy_amps() gives every draw branch and the
simplified-Laplace evaluator one source for a copied block's per-arm
amplitudes. On the ic-fullmodel fixture WAIC reads 798 (p_waic 53) in place
of 13000 (p_waic 6123).converged = FALSE or warns that it
stopped short when only the cheap-pass screen skipped cells.predict(type = "trajectory", mc.floor =)
(default 0.001, on the quantity's own scale) measures each bound's Monte
Carlo error against max(SD, mc.floor): where occupancy saturates near 0 or
1 the posterior SD goes to about 0, and an error relative to it would be
unstable. The check still runs on every unit, time and quantity, since the
units needing the most draws are the few data-poor ones a subsample would
miss. attr(, "mc_binding") is a one-row data frame giving the cell,
time, quantity and bound whose error is mc_se_max, so a run that
nsim.max stopped shows where.Warning: stack imbalance in '.Call', ...: OpenMP workers in the inner
covariance extraction made the first calls into those stubs, and each
first call runs R_GetCCallable(), which touches the R protect stack.predict(type = "trajectory", nsim = "auto") chooses the draw count by
Monte Carlo precision (#386). Draws are taken until the Monte Carlo
standard error of every reported median and interval bound, over every
unit, time and quantity, is at most mc.tol (default 0.05) of that
quantity's posterior SD, or nsim.max (default 10000) draws are reached. A
quantile's standard error is read from the draws with the Hall-Sheather
sparsity estimate; because that error times sqrt(n) is a property of the
posterior, each round's estimate is pooled with the earlier rounds', and a
round grows the count at most 4x, so one noisy small round cannot set the
draw count. Each round redraws the whole table, so memory is that of one run
at the final count and every quantity still shares one draw set. On
Gaussian-shaped quantities at level = 0.95, where 2,900 draws meet 0.05,
the rule stopped at 4,000-5,500 draws for 3 units and about 6,500 for 2,000
units: stopping on the worst of many noisy estimates errs toward more draws.
Every trajectory, fixed nsim included, now reports attr(, "mc_se_max")
and attr(, "nsim_used"); "auto" adds attr(, "mc_tol") and
attr(, "nsim_capped") (TRUE when nsim.max stopped it short of
mc.tol). An integer nsim draws exactly as before. Available on
occu_cover(), the nested-Laplace cover() hurdle and the joint occu()
SVC route; nsim = "auto" on any other type is an error.confint(), coef(), vcov(), summary() and tidy() read the
nested-Laplace grid mixture exactly (#385). On a fit whose posterior is the
Gaussian mixture its outer grid defines (verified on joint occu_cover() and
the areal abun(), ms_abun(), count(), distance() and fp_occu()
routes), the fixed-effect estimate and SD are the mixture's moments and
the bounds invert its CDF, as confint() on the engine fit already did.
They were read off the 1000 Monte Carlo draws the fit carries, which moved a
bound on a weakly identified coefficient by up to a quarter of its SD and the
estimate by up to 0.12 on a production fit. confint() passes through
interval_source ("mixture_cdf") and retained_mass; a hyperparameter
asked for by name is still read off the draws. The mixture is read only when
its moments equal the fit's reported means / sds, so a fit it does not
describe keeps the draw read.fit$sds could disagree with vcov() and with the draws. Where
the fit carries its covariance the SDs are now the diagonal of the
transformed covariance.predict(type = "trajectory") predicts occupancy and cover at any times.
On occu_cover(), nested-Laplace cover() and the joint occu() route,
times = c(t1, ..., tK) gives a long table, one row per unit and time, with
the posterior mean, median and level interval of occupancy psi,
conditional cover cover_cond and expected cover cover_exp (occupancy only
on occu()). The times are any values of the time covariate, between or
beyond the fitted ones, and all of them share one draw set. aggregate = TRUE
averages over the units per draw first and returns the overall trajectory, one
row per time. Each time's draws are summarised before the next is evaluated,
so memory does not grow with the number of times.Imports: and LinkingTo:. tulpa
0.6.4 moves TULPA_ABI_VERSION to 46 (the SMC entry gained the tempered
population WBIC reads), so tulpaObs is rebuilt against it; no tulpaObs route
calls the SMC or VI entries, and its fits are unchanged.predict(type = "change") names occupancy psi on every route and gives
every level an interval. The occu_cover() / nested-Laplace cover()
change table named occupancy p_T<k> / delta_p while the joint occu()
route named it psi_T<k> / delta_psi; both now use psi. Every level
column (psi_T<k>, cover_cond_T<k>, cover_exp_T<k>) now carries .sd /
.lwr / .upr (cover levels previously had none), and every delta carries
.lwr / .upr / .prob_pos, including the delta_cover_from_occ /
delta_cover_from_ab split. The joint occu() route now accepts
times = c(t1, ..., tK) trajectories as well. Both routes build the table
through one helper, so the two layouts cannot drift. Breaking: code reading
p_T1 / delta_p must read psi_T1 / delta_psi.predict(weights = ) pools prediction rows per cell. On occu_cover(),
nested-Laplace cover() and the joint occu() SVC route, newdata may carry
several rows per cell (for example one per habitat level) and weights (a
numeric vector or a newdata column name) gives each row's share of its
cell. Every quantity is pooled per posterior draw before it is summarised or
differenced: occupancy, detection and expected cover as weighted means,
conditional cover as pooled expected cover over pooled occupancy. One call
therefore returns a habitat-marginal change table whose occupancy and cover
columns share one draw set, where averaging cover over a habitat mix
previously took a second predict() call with its own draws.predict(newdata = ) uses the new data (#371). On occu / dyn_occu /
int_occu, abun / removal, distance, fp_occu and dyn_abun it returned the
in-sample fit and ignored the frame. newdata is now expanded through the
predicted arm's fitted formula (fitted factor levels and transforms,
coefficients matched by name). Combining newdata with X.0 / X_det.0 /
terms, or asking for type = "state" with any of them, is an error, and a
formula variable missing from newdata is named.newdata, each row gets its fitted group effect (times the
slope covariate for a random slope), and 0 for a level the fit did not see.
Without the column, and always through X.0, the prediction stays
population-level.cover() + (1 | g) reports its random effect (#373). ranef()
returns the group effects under the factor labels, and the SD is reported
as fit$sigma_re.control$max.iter, on every family; the per-fitter warnings (which
fired on four families, once per species on the community routes, and on
fp_occu only under verbose) are gone. The top-level fit$converged is
removed: convergence() / converged() read fit$convergence on every
family. control$max.iter / control$tol now reach the dyn_abun() and
plain fp_occu() routes.tol; all of them converge by 287 iterations. The ms_occu() areal / SPDE
community EM takes the same 300: at 100, 54 of 180 simulated fits stopped
early, at 300 one does. Fits that converge sooner are unchanged.convergence(fit)$re_boundary. It is reported as one
"boundary (singular) fit" message per fit, the way lme4 reports it, since it
describes the data rather than a failed fit.sigma_<g>_<coef>, cor_<g>_<ci>_<cj> and
re_<g>_<coef>[k] on the natural scale, where it listed the sampler's
log_sigma_* / chol_* / z_* coordinates in summary() / coef() /
tidy().n.threads.grad,
sigma.logr, dispersion.re and sigma.ld.init are admitted only by the
NUTS fitters that read them, and the pg_gibbs route no longer admits the
NUTS-only knobs (adapt.delta, max.treedepth, n.threads, ...), which it
used to drop. The error for a wrong-method knob names only the methods the
family supports.control$max.grid.cells is an opt-in ceiling, unset
by default.Imports: and LinkingTo:.occu_cover() estimates a
starting residual variance for the positive arm from a regression on the
cells carrying a positive record. With every positive in one cell, the cell
factor there had one level and model.matrix() stopped with "contrasts can
be applied only to factors with 2 or more levels", before any fit. That
step now falls back to the covariate-only residual.control$max.grid.cells sets the engine's cell-count ceiling on the
dense outer grid, on occu_cover(), occu_multiscale_cover(), cover()
and occu() spatial. Two copied field blocks (an intercept and a trend
field, each shared onto the positive arm) at the default field-SD and copy
axes cross to 50 x 50 = 2500 cells, past the engine's 2048 default, and no
route forwarded the engine's knob. With tulpa >= 0.6.1 the refusal names the
per-block axes that produced the count.ranef() returns the grouping factor's own labels in the level column for
a formula random effect (1 | g), where it returned an integer index (#370).Imports: and LinkingTo:. That
release changes the ABI (45), so a tulpaObs built against 0.6.0 refuses to
run against it until rebuilt.Imports: and LinkingTo:. The draws
and diagnostics this version reports read engine surface that release
carries: the corrected fixed-effect block the subspace debias writes to
$draws, the $subspace_debias record ($idx / $declined) that says
whether a correction completed rather than declined, and the outer-grid
facts diagnostic_summary() reports.detection = ~ effort + (1 | observer)
with visits = a per-visit frame, is a random effect on the visit's
detection logit: the classic case is an observer who changes between visits
to the same site. method = "laplace" fits it in the random-effect EM on
one Bernoulli detection row per observed visit, weighted by the site's
occupancy weight; the per-group AGHQ debias does not factor over visit rows
and records fit$aghq$declined = "visit_rows". method = "nuts" samples it
as extra parameters of the occupancy likelihood under tulpa's own
random-effect term prior (<tulpa/re_term_prior.h>, tulpa >= 0.5.1). On
250 sites x 4 visits x 10 observers (true SD 1) both recover the SD (Laplace
0.90, NUTS 0.96, realised 1.00) and the observer effects (correlation 0.97
with the truth), and agree with each other. The nested-Laplace route refuses
the term; other families with visit covariates refuse a bar over visit rows
(#362). A site-level random effect alongside visit covariates, refused on
the Laplace path until now, is fitted the same way. simulate_occu() gains
n_visit_groups / sigma_visit.p_visit_* name and the names after it were shifted by the block's size.tanh map, which agrees with tulpa's
partial-correlation map for two coefficients and not from three; it now calls
tulpa::build_L_from_raw().method = "laplace_gibbs" / "laplace_mi" with a formula random effect
returned the plain fit labelled with the correction's method.model.matrix() gave a factor visit covariate all k dummies, or kept an
intercept column, beside the site intercept.- 1, which is how every
visit formula arrives; the #362 refusal did not reach a bar written through
tobs(visits =) before.occu_cover() defaults phi.grid.pos to a three-node band about the
residual pre-fit and, measured, integrates it as written -- placing it costs
a pilot plus a second full grid solve (178s against 80s on
test-occu-cover-joint.R) and buys nothing on a span that already brackets
the truth. The only way to ask the engine for that was to leave the axis
unmarked, which declares it a PIN THE USER WROTE, and a fit with a defaulted
sigma.grid then came back
outer_grid_recenter_declined == "axis_pinned". It is now declared with
tulpa::auto_grid(place = FALSE): the same integration, reported as ours.
Needs tulpa (>= 0.5.0). .tobs_mark_auto() / .tobs_num_auto() carry both
halves of a declaration, and occu_multiscale_cover() keeps a stated
auto_grid() marker across its SD-to-variance conversion instead of dropping
it in as.numeric().tulpa:::,
get(, envir = asNamespace("tulpa")) and getFromNamespace() now call
tulpa::tulpa_rpg(), tulpa_joint_inner_vcov_blocks(),
tulpa_iter_progress(), tulpa_grid_axis(), tulpa_grid_log_quad(),
tulpa_theta_matrix(), tulpa_normalise_weights_safe(),
tulpa_batched_pareto_k(), tulpa_spde_precision_Q(),
tulpa_spde_log_hyperprior(), tulpa_joint_grid_batch(),
tulpa_joint_axis_specs_from_grid() and the four tulpa_hyper_*() doors
(gcol33/tulpa#826-#834). An unexported symbol carries no stability contract:
the engine had already renamed .tulpa_iter_progress to the exported
tulpa_iter_progress(), so the eight progress reaches would have errored on
any engine past the pin. Imports/Remotes move to tulpa 0.4.12, the first
version carrying all of them, and test-no-new-tulpa-internals.R now asserts
the surface is empty rather than capping it at an audited ceiling.waic() returns loo's own waic object (#333). It was a
tulpa_criteria, so loo::loo_compare(waic(f1), waic(f2)) could not read
it. It is now built by loo::waic() on the same (group-folded) pointwise
matrix, as loo() already was through loo::loo(), and carries the scored
unit in attr(, "loo_unit"). Read the numbers from $estimates
(waic(fit)$estimates["waic", "Estimate"]); loo deprecates the old top-level
$waic / $p_waic / $elpd_waic fields, and the $elpd alias is gone.
waic() and loo() accept only group = in ... and refuse anything else,
which they had been dropping.
One layout for the stats accessors on every fit (#332). Breaking:
coef() is a named numeric vector, named <arm>_<term> as vcov(),
confint() and summary() name the same coordinates (psi_(Intercept),
p_det_cov1). One arm, without the prefix: coef(fit, arm = "psi")
(replaces coef(fit)$psi). Cover and occu_categorical() fits follow the
same rule, so their names change from presence:(Intercept) to
presence_(Intercept). An arm is assigned only where it declares the
term, which also fixes occu_multi(), whose pairwise coefficients
f_sp1_sp2_* had been listed under f_sp1.tidy() has arm and term columns on every family (arm NA for a
coordinate that enters no arm, such as log_r).glance() has one column set: nobs, df, logLik, n_fixed,
n_samples, n_divergent, mean_accept, converged.summary() is the same estimate / std.error / interval data frame on
every method; cover() no longer has a separate summary.cover_fit object
under laplace and a mean / sd table under nuts.simulate() always returns a list of nsim responses (sim_1, ...)
carrying the "seed" attribute of stats::simulate(), including at
nsim = 1, where most families had returned the bare response. A supplied
seed no longer moves the caller's random number stream. simulate() now
works on dyn_occu(), int_occu(), t_occu(), occu_cover(),
occu_multiscale_cover(), cover() and occu_categorical(), which had
refused.residuals() keeps its list(occ, det) contract: det is scored against
z * p, which is not one of the fitted() quantities.logLik() and nobs() count the same observations (#331). The "nobs"
attribute of logLik() came from the engine's fit$N, which the detection
families record as sites, while nobs() counts observed response entries
(e.g. 60 against 240 on a 60-site, 4-visit occu() fit). logLik.tobs_fit()
now takes the attribute from nobs(), so stats::BIC() on a logLik object
uses the same n as nobs() on 26 fits over 11 families.
Every fit states what its draws are (#326). tobs() stamps
fit$draws_kind from the route table: "chain" for nuts / pg_gibbs,
"iid" for every Laplace route. tulpa read an unstamped fit as a chain, so
tulpa::diagnostics() computed Rhat / ESS on a Laplace fit's i.i.d.
approximation draws and tulpa::check_diagnostics() printed a convergence
pass on them. Those fits now get the approximation-reliability table and a
"not applicable" result.
laplace_gibbs / laplace_mi pool calibrated SEs (#334), and dyn_occu()
fits under both (#327). The corrections' hard latent draws went through the
EM's M = 1000 pseudo-binomial encoding, so each refit's within-imputation
variance was ~0 and the pooled occupancy SEs came out at about a third of the
exact-marginal SEs (0.057 against 0.207 on a 150-site occu() fixture). A draw
is now encoded as complete data through the family's hard_encode, and the
pooled SEs agree with the laplace route's on occu(), int_occu() and
dyn_occu(). On dyn_occu() the correction steps had stopped with
nrow(X) != length(y); a draw now samples each site's whole occupancy path
by forward-filter backward-sample, and a site with no transition from the
origin state contributes no transition trial.
The goodness-of-fit doors honour their generics' arguments (#328).
test_dispersion(), test_outliers(), test_zero_inflation() and
check_model() take tulpa's nsim beside n.samples and a seed that is
scoped to the call; test_dispersion() takes alternative (default
"greater", the previous p-value). An unrecognised argument, or a non-NULL
observed, is now an error on these and on pit_residuals() instead of being
dropped into ....
ranef() on cover() / occu_categorical() fits returns the empty frame,
and ppc() / pit_residuals() refuse occu_categorical() cleanly (#329).
Those fits carry no model$model_type, and the family-handler lookup failed
with argument is of length zero.
logLik() is finite on jsdm() / ms_count() sampler fits, t_occu() and
ms_abun() Laplace fits (#330). The fill at the tobs() tail now reaches
the community count kernel, t_occu() gains a pointwise log-likelihood (one
observation per surveyed site-season, which also gives it waic() / loo()),
and ms_abun() Laplace / nested-Laplace fits record N and replicate their
log marginal into log_prob, as abun() does. A fit with nothing to fill from
now reports NA with attr(, "declined") rather than NaN.
fit$hyper_summary$sd reports read 2x
to 6x below the posterior's; it now reads each cell over its own row's box
(gcol33/tulpa#746). Unrefined fits and every reported mean and interval are
unchanged.occu_cover() joint fits report the engine's own read of each
hyperparameter axis. fit$hyper_axis names the engine column each public
hyperparameter was read off (NA for a quantity no single axis determines,
such as an MCAR correlation or a correlated-slope SD), and fit$hyper_summary is a
data frame of parameter, mean (the grid-weighted mean means reports),
sd / sd_source (the engine's theta_sd and the estimator that produced
it), lwr / upr (theta_ci_lo / theta_ci_hi), sd_grid (the
grid-weighted SD over the outer cells) and axis. A random-effect slope SD
divided by its covariate scale takes the same division. A quantity reported
through a monotone map of its axis takes its quantiles through the map and its
SD by the delta method: the cover dispersion reported as an SD through the
square root (.cover_phi_engine_to_sd()), a field SD read off a precision
axis (b<k>.tau, as a cover- or detection-arm field carries) through
1 / sqrt(tau).
A joint occu_cover() fit reports a hyperparameter's SD as the engine's
per-axis SD (#325). The grid-weighted SD measures the outer grid's
resolution, and reads 0 where the weight sits on one cell: a 25 km fit at
grid ESS 1 reported sigma, alpha and phi_pos at +- 1e-4 against engine SDs
of 0.145, 0.0129 and 0.0947. sds now carries hyper_summary$sd for every
hyperparameter read off an engine axis, and the grid-weighted sd_grid for a
quantity no single axis determines or an axis with no finite engine SD. vcov() carries each hyperparameter's row and column to
that SD, keeping the correlations the grid measured, and the draws are taken
from it; a hyperparameter with no spread over the grid is uncorrelated with
the rest. The field_sd companions follow their sigma. The occupancy-only
joint route (occu() with a spatially-varying-coefficient bar) reads its
field SDs off precision axes the same way and now carries hyper_axis,
hyper_summary and the same reported SD. The interval in
hyper_summary is the engine's weighted-quantile read, which on an axis
whose weight has collapsed onto one cell spans that cell's box and is wider
than sd suggests.
A fused-batch occu_cover species fit is the object its own tobs()
call returns. The fused backend used to collect each species' prepared
engine call, run the batched kernel and assemble a plain list per species by
hand, which carried neither the class nor the fields of a joint fit, so
.tobs_joint_draws() stopped with "Got: list" and predict() / fitted()
could not read the fit. Each species is now fitted by the same single-species
tobs() call the looped backend makes (.tobs_batch_species_call()), run
through tulpa's tulpa_joint_grid_batch(): the species' main outer-grid
solves are answered by one fused solve and every other step runs as it does
alone. The .batch_collect branches of the joint builder are gone, so the
fused route also carries refinement, placement and the single-block backend
the ordinary fit uses. The species share one design only uncompressed, so
the no-detection compression is off inside the batch; the latent cover RE,
and any request tulpa declines, fall back to the looped backend. On the
24-site, 3-species fixture the fused fits match independent uncompressed fits
to 1.4e-14 on log-marginals, 1.7e-15 on weights, 4.4e-16 on means and 1.0e-14
on draws at one seed; test-occu-cover-batch.R restores the draws check on
the pinned-dispersion block and adds draws, predict(), fitted() and
summary() against the independent fits.
Outer-grid weights rebuilt in tulpaObs now take the grid's own cell
measure, refinement slice cells included. A refinement pass appends slice
cells to an outer hyperparameter grid: new levels on one axis at one
combination of the others. Such a grid is not a tensor product, and tulpa
0.4.0 measures it cell by cell from the per-cell refining_axis tag.
.tobs_grid_weights(), the helper every post-processor that rebuilds weights
from theta_grid goes through, called the measure without that tag and so
applied the tensor-product rule: on a 3 x 3 grid with two sigma slices in one
row the weights moved by up to 0.035. It now passes refining_axis, uses a
driver result's stored log_quad when it carries one, and reads a
single-axis grid stored as a bare vector through its axis name. A
log_marginal whose length does not match the grid is an error instead of a
silent fall-back to equal weights.
Every fixed-grid driver in tulpaObs (the areal count, community occupancy /
N-mixture, SPDE N-mixture and spatial ms_abun() routes) builds a tensor
grid with no refinement, so their weights do not change; on the areal count
route they equal the engine's own weights to the bit, as before.
The converged-cell weights of a joint fit keep the measure when the engine
left none usable. .tobs_joint_ok_cells() fell back to a softmax of
log_marginal when fit$weights was absent, the wrong length or all
non-positive, which weighs every node equally and gives a slice cell the mass
of a whole column. It now rebuilds them through .tobs_grid_weights().
The areal-BFGS product grid is weighted by its cell measure. The
per-family areal fits (removal, distance, fp_occu(), dyn_abun(),
dyn_int_occu(), the SVC surfaces) normalised the grid log-marginals with equal node weights. The grid
is now named from each block's hyperparameters (b<k>.-prefixed with more
than one block) and weighted like every other outer grid. The default ICAR
tau grid is evenly spaced on log tau and does not move; the defaults that are
not evenly spaced on their integration coordinate, or reach a bounded axis's
edge, do: per-cell weight relative to equal weights ranges 0.69 to 1.06 on
the proper-CAR (tau, rho) grid, 0.71 to 1.19 on the BYM2 (sigma, rho) grid
and 0.86 to 1.14 on the temporal AR1 (tau, rho) grid.
The fused occu_cover batch backend forwards the fit's hyperpriors and
copy-scale prior to the batched engine. tulpa's batched joint driver now
folds the hyperprior into each species' log-marginal and weights the grid
with its cell measure, the same construction as the multi-block driver, so
control$batch.backend = "fused" passes prior.sigma / prior.alpha /
prior.phi, copy.atom.mass and copy.slab through and carries the
engine's log_quad into the per-species fit. The fused backend is reachable
only for a configuration without a positive-arm dispersion axis, which every
occu_cover fit has carried since 0.2.2, so every current fit still falls
back to the looped backend.
?tobs_terms: refining an axis changes how densely its nodes sit, and the
reported span stays the declared span unless a refinement point lands past
it. A stated c(0.2, 0.5) is only densified and spans [0.127, 0.791] with
refinement on as with it off; the page previously gave a tighter span for the
refined fit, which tulpa no longer reports.
The SPDE routes of abun(), ms_abun() and ms_occu() fit again on tulpa
0.4.0. They added tulpa's pc_prior_log_density() to each grid
log-marginal, and tulpa removed it when the SPDE hyperprior became a density
on (log range, log sigma), the coordinates the outer grid is measured in.
All three now take .spde_log_hyperprior(), so the PC prior carries the
Jacobian of that change of coordinates; test-spde-nmix.R (18) and
test-ms-occu-spatial-spde.R (11) pass, and errored before.
Requires tulpa 0.4.0: .nl_grid_log_quad(refining =) is new there, and
tulpa::CovType::MATERN is now MATERN32 (same code, Matern 3/2).
fitted() on an occu() fit with an areal field returned a flat psi
(gcol33/tulpaObs#318). The field was never recorded as an eta offset on the
single-season occupancy routes, so every door that rebuilds eta from the
design -- fitted(), residuals() and the pointwise log-likelihood behind
waic() / loo() / dic() / cpo() -- read log psi = X beta while
predict(type = "state") carried the field. On a 6x6 lattice whose field the
fit recovered at cor 0.79, sd(fitted(fit)$psi) was 0 under both
nested_laplace and pg_gibbs, and elpd_waic scored the field fit below
the intercept-only fit (-67.1 vs -65.5). tobs() now derives the offset from
the field a fit reports wherever its fitter did not record one, and
.tobs_eta_draws() adds it for every family whose predictor it builds, so the
same fit reads fitted()$psi == plogis(intercept + field) and elpd_waic
-45.8. Fits without a field are unchanged.
A pg_gibbs fit reported no convergence diagnostics
(gcol33/tulpaObs#319). Split-Rhat and ESS were computed and left at
fit$rhat / fit$ess, where summary() and print() do not look, on all
seven families that advertise the engine (t_occu() has no other). The
shared finalizer now writes the same per-parameter record the NUTS paths
write, so summary() carries rhat / ess_bulk / ess_tail, print()
prints the Convergence: line and its warning, and converged is decided on
the split-Rhat < 1.01 rule rather than < 1.1.
logLik(), AIC() and BIC() were NaN on every pg_gibbs fit
(gcol33/tulpaObs#320). The finalizer stored log_lik and every log_prob as
NA. A sampled fit with no finite logLik() now gets the marginal
log-likelihood at the posterior mean from its family's own pointwise kernel,
evaluated after any field offset is in place. A fit that already reports a
value keeps it: NUTS still reads mean(log_prob).
glance()$converged was NA on every sampled fit
(gcol33/tulpaObs#321). The inherited method reads a top-level converged,
which sampled fits do not carry, so a NUTS chain that had not mixed glanced as
NA while converged() said FALSE. glance() now reports converged().
Control knobs whose name prefixes another knob were read by partial
match (gcol33/tulpaObs#322). control$n.threads in cover() returned
n.threads.outer when only that was set, putting the outer-grid width on the
inner per-observation loops. The same $ read sat on sigma.grid,
phi.grid and the sbc() check for alpha.grid (which returned NULL when
both alpha.grid and alpha.grid.trend were set, so the formula copy was
written beside a stated scale). All are exact-key reads now, and a test walks
every function in the namespace so a new read of that shape fails the suite.
Holding an outer-grid axis at exactly the nodes stated takes two knobs, not
one (gcol33/tulpaObs#312). ?tobs_terms said control$adaptive.grid = FALSE
holds a stated axis at the nodes the formula wrote. Refinement reaches an axis
from two independent passes, and the second -- the var-of-means consistency
pass -- runs whatever adaptive.grid is set to, so a fit turning off only the
first still integrates nodes placed inside the stated range: a
share(spatial(), alpha = grid(c(0.25, 0.5, 1, 1.5))) fit came back
integrating five alpha nodes, the fifth at 0.7303. A genuinely fixed outer
grid is adaptive.grid = FALSE AND var.of.means.consistency = FALSE, which
is what the joint fitter's own comment says and what
test-cover-hurdle-adaptive-grid.R already asserts on the cover() path.
The stated nodes are a BOUND either way -- refinement never places a node past
an end node, which is what #301 established -- so no fit's numbers move here:
the documentation and the one test that read the promise the other way are
what change.
The engine behaviour arrived in tulpa 0.2.14 (gcol33/tulpa#658), the version 0.2.0 pinned. Measured on one fixture with the engine swapped underneath a fixed tulpaObs binary in a private library: 0.2.14 and 0.3.0 place the same fifth node, to the digit.
r-universe can build tulpaObs binaries again. Each binary job runs
R CMD check with no TULPAOBS_FAST set, so it reaches the
test-share-formula-api.R block that the package's own gate skips
(R-CMD-check.yaml sets TULPAOBS_FAST=1), and 13 of 14 binary jobs were
failing on that single assertion -- [ FAIL 1 | WARN 21 | SKIP 810 | PASS 6244 ] -- while the source job passed and published.
The 0.1.4 section below was never tagged or released, so everything in it
reaches a user here for the first time, including its four breaking changes:
share() replacing copy() as the cross-arm formula verb, and a bare areal
term no longer coupling the arms on cover(), occu_cover() or
occu_multiscale_cover().
The cover dispersion is reported as an SD on every route
(gcol33/tulpaObs#306, gcol33/tulpaObs#309). The joint nested-Laplace path --
occu_cover()'s spatial default -- reported the cover arm's dispersion as the
residual VARIANCE where laplace, the non-spatial NUTS path and cover() all
report the SD, and it consumed a stated control$phi.grid.pos, documented in
SD, as variances. tulpa's R-level phi is per family: for the gaussian arm
that lognormal and gaussian both compile to it is the residual variance,
converted to the kernel's SD parameterization by the engine itself, while
truncated_gaussian and interval_gaussian carry the SD at both levels. The
axis was built from an SD pre-fit, so it explored variances over a span meant
for SDs and the read-out sites returned it unchanged.
On a null-field control, where the residual SD is the whole cover-arm spread and nothing can absorb it:
simulated cover SD oracle laplace joint (before) joint (after)
0.40 0.5053 0.4213 0.1849 0.4308
0.70 0.7818 0.7353 0.5657 0.7561
occu_multiscale_cover() carried it further than a label:
.occu_mscale_cover_sigma_pos() reads that entry to build the lognormal
conditional mean, so fitted() and predict() applied the exp(sigma^2 / 2)
correction at the square of the dispersion -- about 9% on every predicted
cover value from a spatial multiscale fit. The conversion now lives in one
pair of functions rather than at each of the four decode sites, and a stated
phi.grid keeps its documented SD surface. The beta arm passes through
unconverted: its phi is a precision on both levels, so converting it would
be the mirror defect.
The reported dispersion's SCALE is now asserted against a simulated truth
(gcol33/tulpaObs#310). The defect above passed the whole cover / occu_cover /
ms_occu_cover / multiscale suite byte-identically, 2612 assertions, twice: the
recovery tests score arm coefficients and the mean model does not see the
dispersion scale, while the one test reading the entry scored it at
tol <- if (p == "phi_pos") 5 else 0.15, which a factor-of-sigma error passes
at every sigma its fixture produces. Two reads that do not depend on the
per-seed level are asserted instead -- the ratio across two truths at one seed
(1.757 / 1.775 / 1.806 against 0.70 / 0.40 = 1.75, where the variance scale
gives 3.06) and that the estimate is nearer its truth than its truth squared,
which needs no tolerance at all. The beta arm is scored the same way as a
control.
A standard error of exactly zero is a failed solve, not a value
(gcol33/tulpaObs#307). sqrt(pmax(diag(V), 0)) was the idiom at all 17 sites
reporting marginal standard errors. A diagonal entry the solve did not produce
as positive -- an indefinite or singular marginal Hessian, or a block that
never left its starting value -- was floored to 0, and 0 survives as a value:
the fit returns converged = TRUE, with an ordinary point estimate beside an
SE claiming infinite information from finite data. A Wald interval collapses
onto the estimate and a standardized residual is infinite, which is how the
ms_abun(negbin) coverage loop rendered a degenerate interval as "the errors
are not normal" -- a true failure pointing at the wrong quantity, since
shapiro.test() returns NaN on an infinity while dropping NA and NaN.
All 17 sites go through one .tobs_sds_from_vcov(), which returns NA for a
non-positive diagonal and is the previous expression to the bit wherever the
Hessian is positive definite. NA is the honest value and also the quiet one --
every na.rm = TRUE reader drops the pathological row and lets the survivors
report a clean number -- so the helper warns, naming the coordinates, at the
one place that knows the solve failed. A NULL block does not warn: that is "no
covariance available", an explicit contract of the callers.
share(residual = ) carries an arm deviation beside the copy.
share(spatial()) puts the occurrence field on the cover arm as alpha * w:
one amplitude, the same pattern, which is the whole model wherever the two
arms really do respond to one surface. Where they do not, the copy cannot
bend, the amplitude drifts toward whichever arm carries more information, and
a cover-arm pattern the occurrence field lacks is not fitted at all. A free
field placed in the positive formula fits any pattern, but costs 2K latent and
leaves the amplitude weakly identified, a free field being able to reproduce
alpha * w for any alpha.
residual = is the middle: the cover arm carries alpha * w + delta with
delta orthogonal to the shared component, so the two halves cannot trade
against each other. residual = "full" is one latent per field node and
compiles to the same blocks a placed positive-arm bar produces (asserted with
expect_identical), reported as an amplitude and a deviation rather than as
one surface. residual = r takes the r smoothest Laplacian modes scaled to
the ICAR covariance and projected off the shared field -- latent K + r
rather than 2K, orthogonality exact by construction, and Sorbye-Rue column
scaling so a rank-r deviation's SD reads on a full field's scale and a rank
sweep is comparable. The deviation SD is pinned, a free one being an outer
axis per block, which forces integration = "grid".
A posterior SBC refit carries the copy-amplitude axis the fit asked for
(gcol33/tulpaObs#305, gcol33/tulpaObs#311). A fit retained neither end of its
request: fit$model$control is empty, so the alpha.grid / alpha.n a
share() spec compiles into was gone, and fit$model$formulas$pos is ~1
with the term stripped. .tobs_sbc_pos_formula() could only synthesise a bare
share(spatial()), so a replicate was analysed under a different model than
the observed data whenever the observed fit pinned its axis -- the engine
default 0, 0.1, ... 16.432 against a stated 0, 0.25, 0.75, 1.5.
What is carried is the REQUEST, never the realised axis:
control$adaptive.grid refines a defaulted axis against the data and only
densifies a stated one, so handing back the nodes a fit realised asks for
something different from what it asked for. Restating them instead was tried
and reverted, having broken the same equivalence test in the other direction
(sigma 0.481 against 0.619 on identical data), so the distinction is written
at both ends.
A decoupled block's pinned alpha is not an estimate on any route, and the
fused batch field is on the scale it is read on (gcol33/tulpaObs#304). Both
are confined to the fused batch backend, which is not the default. It reported
an alpha in means / sds on a formula with no share(), because the
single-block branch of .occu_cover_jc_postprocess() read the amplitude
without the guard its two siblings carry: harmless on the single-block driver,
where a decoupled block builds no amplitude axis at all, but the fused path
reaches that branch with a multi-block engine fit, where the axis is present
and pinned at 0. Its spatial_field was also about 2.3x large and not by a
constant -- the multi-block driver returns a copied block's latent whitened,
where the single-block driver returns it already multiplied by that cell's
sigma, so the field was read one factor short per grid cell, and the
grid-weighted mean of a per-cell error is not a ratio. The per-cell modes
differ by exactly 1 / sigma at every grid point. Everything else already
agreed: sigma axis, weights, log_marginal, and the coefficients to 7.7e-10.
The cover phi axis default declares itself engine-placed. Every computed
outer-grid default in the package carries the auto_grid() marker except the
four phi_grid_pos defaults, which are built from a pre-fit and were handed
over unmarked. The marker is how the engine separates an axis this package
placed from one the user pinned, so the auto-recenter has never been able to
move the phi axis onto its mode, on any cover route, and nothing said so. The
marker is set only when the user named nothing, leaving a stated grid a pin.
Byte-neutral today, and load-bearing under per-axis refinement control, where
an unmarked axis is treated as a stated bound and frozen.
The test suite's assertion and skip counts are recorded per file, and a run
that drifts fails (gcol33/tulpaObs#302). tests/expected-counts.csv holds
assertions as a floor and skips as a ceiling per (file, tier), so a block that
stops executing cannot report a clean tail. The recorder takes its numbers
from a measured run, refuses a run carrying any failure, and is add-only:
differences on an existing row are reported for a person rather than
overwritten, because "asserted fewer" is a leaner environment and a dead block
identically, and auto-taking the minimum would erase exactly the finding the
manifest exists for. Smoke floors come from the leanest box and full floors
from CI, deliberately. The on-request recovery tier can also now run a named
subset of files.
The sampled areal hypers changed what they reach, on every such fit. Two separate movements, both in this version, and neither announced itself in a fitted object -- so a number carried across this boundary is not comparable.
The sampled proper-CAR / bym2 rho now stops at the field's admissible
interval instead of a clamp at 1 - 1e-4. tulpa declares every rho* axis
evenly spaced while the default node set c(0.5, 0.8, 0.95, 0.99) is built
logit-spaced, so the half-node-step support came back as [0.35, 1.01] --
1.01 is not a correlation -- and the proper-CAR loading scales its j-th
eigen-column by (1 - rho lambda_j)^-1/2, which is numerically intrinsic at
lambda_max = 1. Measured on the coupled fixture: divergences 17 / 82 / 14 /
96 across four seeds fall to 0, and the field_sd posterior mean falls from
10.9-22.0 at its worst draws. Upstream cause filed as gcol33/tulpa#657; the
downstream guard is belt-and-braces once that lands.
Separately, the outer grid's declared SPAN (gcol33/tulpaObs#301) moved a
REPORTED summary. field_sd is weakly identified and right-skewed, so a flat
prior over a wider log span raises its posterior mean: 1.18 against 1.011
recorded for the same fixture, ~17%, with interval coverage unchanged at
0.95-0.97. The calibration claim is intact; the point summary is not
comparable across the boundary. It also reversed which across-seed summary is
more stable -- the median now is, where the mean was on the narrower span --
because the mean is the summary the span moves.
BREAKING: the coupling-amplitude control keys are no longer user surface
(gcol33/tulpaObs#295). control$alpha.grid, control$alpha.n,
control$alpha.grid.trend and control$alpha.n.trend are the wire format the
formula compiles into, and a fit that sets one is now refused with the
formula spelling to use:
control$alpha.grid -> share(spatial(), alpha = grid(c(...)))
control$alpha.n -> share(spatial(), alpha = grid(n = <k>))
control$alpha.grid.trend -> share(spatial(), terms = list(trend = grid(c(...))))
control$alpha.n.trend -> share(spatial(), terms = list(trend = grid(n = <k>)))
Two front doors for one request meant a guard layer to keep them consistent,
an asymmetry in which one composes with what, and two spellings a reader had
to know were the same model. share() says what the model IS; the keys say
how the engine reads an axis, and only one of those is the user's business.
Every case is expressible: on a two-block field
share(spatial(), terms = list(intercept = grid(g1), trend = grid(g2)))
reproduces alpha.grid + alpha.grid.trend exactly (verified to the reported
digit before the keys were closed). A block the formula does not name is
decoupled, per the reading settled in #297.
control$prior.alpha is unaffected: it is one of three regularizing
hyperpriors on the outer-grid axes, and retiring one member alone would break
the family.
BREAKING: a missing share() decouples the arms on cover() too
(gcol33/tulpaObs#297). The third and last door now reads a bare areal term
the way occu_cover() always has and occu_multiscale_cover() moved to in
55d1219: a shared field that no share() couples is pinned at alpha = 0
and rides the presence arm alone. Structure nobody wrote is not in the model.
A shared-formula cover() fit that writes no coupling therefore changes: the
presence field no longer reaches the cover arm. Restore the previous fit by
writing the coupling it was getting implicitly, + share(spatial()) in the
formula, which #298 made expressible. A fit that already states
control$alpha.grid[.trend] is unaffected, and so is a correlated | bar --
that spelling is copy-only by definition, so its coupling is written by the
bar itself.
The two spellings still agree exactly when both state the coupling: per-arm
positive = ~ ... + share(spatial()) and shared
~ ... + spatial(~ 1 || cell, graph = adj) + share(spatial()) stay
byte-identical across every numeric fit field.
cover() accepts share() in the single shared formula
(gcol33/tulpaObs#298). It previously died inside the strip with
attempt to select less than one element in get1index, naming no term the
user wrote, because the coupling verb was lifted on the per-arm path only.
The shared branch now strips it first and translates it through the same
promotion the per-arm branch uses. This is what makes the decoupling above a
default rather than a capability loss: without it a shared-formula model
would have no way to state a coupling at all.
A structured term on a cover arm reaches the arm rejector
(gcol33/tulpaObs#296). .occu_cover_extract_pos_copies() parsed the whole
positive formula against data = NULL to pull the coupling out, so
positive = ~ x + re(cell) died as object cell not found, pointing at the
user's data instead of at the unsupported feature. The strip now evaluates
the share() calls alone and leaves every other term in place, so the arm
rejector's own message is what speaks. Reaches occu_multiscale_cover(),
which adopted the same strip in #294.
BREAKING: share() replaces copy() as the cross-arm formula verb. The
term that carries one arm's latent field onto another is now
share(spatial(), ...). One concept, one verb: the old name is deleted
rather than aliased, and writing copy() in a formula raises an error naming
its replacement instead of falling through to the fixed-effects design and
dying there as a missing function. The whole argument surface is unchanged --
alpha = (scalar, grid(), grid(n = )), per-component terms =, and
prior = on the coefficient -- so a fit is restored by changing the verb and
nothing else. copy remains the ENGINE word: the tulpa-side copy = spec
key, the internal constructor and class, and the prose describing INLA's own
copy= mechanism all keep it. The rename is what makes room for the
arm-specific deviation (share(spatial(), residual = )), which lands next:
once an arm's field is a copy PLUS a deviation, "copy" no longer describes
it.
predict(type = "change") takes more than two times. times = c(t1, t2)
is unchanged, down to the column names. times = c(t1, ..., tK) widens the
same table into a trajectory: a level column per step (p_T1..p_TK,
cover_cond_T*, cover_exp_T*) and a _T<k>-suffixed delta per step, each
differenced against t1 and each carrying the occupancy / abundance
decomposition, its .lwr / .upr at level and its .prob_pos. Every step
is evaluated on ONE draw set, so the steps share a posterior and their deltas
are jointly valid -- which K separate two-time calls would not be. With two
times the delta columns stay unsuffixed, there being only one step to name.
BREAKING: a missing share() decouples the arms on
occu_multiscale_cover() (gcol33/tulpaObs#297). The three cover doors gave
a bare areal term opposite readings: occu_cover() pinned the amplitude at
alpha = 0 -- structure nobody wrote is not in the model, which is what makes
share() mean something -- while occu_multiscale_cover() and cover()
integrated the engine's default axis and coupled the arms. Same formula,
different model, no message either way. occu_cover()'s reading is the one
kept, and occu_multiscale_cover() now matches it.
A multiscale fit that writes no share() therefore changes: the occurrence
field no longer reaches the cover arm. Restore the previous fit by writing the
coupling the model was getting implicitly, share(spatial()) in the positive
formula, which #294 made expressible. A fit that already states
control$alpha.grid[.trend] is unaffected.
The pin reaches the engine as a real one-node axis rather than a dropped copy
spec: the joint driver offers the (sigma, alpha) parameterization only to a
copied block, so removing the spec moves the shared field onto b<k>.tau and
re-priors it.
cover() is NOT changed and still couples by default. It has no formula
spelling for the amplitude at all under a shared formula -- share() there
dies with an internal error (gcol33/tulpaObs#298) -- so decoupling it by
default would make the coupling unreachable except through the control key.
It moves once #298 gives it one.
occu_multiscale_cover() can name the copy coefficient it is built
around (gcol33/tulpaObs#294). The model carries one: alpha is in means,
in draws, in the fitted-surface function, and the simulator draws the cover
arm as ... + alpha * sigma * f[plot_cell]. Its cover formula could not say
so -- the raw formula reached .occu_cover_reject_structured(), whose list
ends in "copy", because this door skipped the copy-stripping step
occu_cover() does first -- so control$alpha.grid[.trend] /
alpha.n[.trend] was the only way in, and the documented
share(alpha = grid(...)) spelling named a term the front door rejected.
share(spatial()) now works in the positive formula, with the stated-nodes,
grid(n = ) resolution and fixed-scalar forms occu_cover() takes, and is
translated through the same shared helper. It is refused under
method = "laplace" / "nuts", which are the non-spatial path (iid cells,
field fixed at 0) and so have no field for an amplitude to scale.
A fit that writes no share() is unchanged: it still integrates the engine's
default amplitude axis. That is NOT what occu_cover() does with the same
absence -- there it pins alpha = 0 -- so the shared translation runs only
when a copy is actually written; routing every fit through it flips this
door's default from coupled to decoupled. Verified against the pre-change
resolved axis on the same fixture. The disagreement between the doors is
gcol33/tulpaObs#297 and is a modelling call, not a cleanup.
Everything a cross-arm copy states about its coefficient is now written in
the formula. share() stated the amplitude's NODES, and the other two things
a coupling has -- how finely the axis is read, and the prior on the
coefficient itself -- existed only as control$alpha.n[.trend] and
control$prior.alpha, so a fit that wanted either had to leave the formula
for the dotted knobs the formula API was built to replace. prior.alpha was
not documented at the front door at all.
grid(n = 9) states a RESOLUTION where grid(c(...)) states nodes: the
engine re-reads its OWN axis with that many slab nodes, atom and slab bounds
unchanged, so sharpening the axis never restates the prior structure it
carries. It is accepted wherever nodes are, terms = list(...) included, so
one field can take stated nodes on its intercept block and a resolution on its
trend block. share(prior = list("pc.prec", c(4, 0.01))) carries the
hyperprior on the copy coefficient (prior_alpha in the joint driver), shape
checked where it is written.
The control keys stay as the lower-level spelling and the representation the
formula compiles into; a fit writing one request in both places is refused,
and a bare share(spatial()) still composes with control$alpha.n -- the
translation writes only what the copy stated.
A prior on the copy coefficient is refused on a fit that copies two
blocks. The engine bakes one prior_alpha onto the FIRST block carrying a
(sigma, alpha) axis pair, so a fit with an intercept and a weighted trend
field would have had one coupling regularized and the other flat, silently.
Filed upstream as gcol33/tulpa#655; refused here in the meantime rather than
applied to block 1.
print() on a share() term printed nothing at all for the common
share(spatial()) form: it read x$ref, which a selector copy does not carry,
and sprintf() on a NULL returns character(0). It now names the selector,
the amplitude and the prior.
The header on .occu_cover_apply_copy_coupling() described a naming-keyed
back-compat path -- an occurrence field with no name = and no share()
leaving control$alpha.grid untouched -- that has not existed since the copy
selector became type-carrying at 0.0.45. Decoupled is the default on every
path (gcol33/tulpaObs#290).
A control key is read exactly, not by prefix. .cover_joint_control()
read the caller's control list with $, which matches on unique prefix:
prune is a prefix of prune.tol, so a fit that only sized the screening
tolerance was turned into one that screened. Every key another key extends is
now read with [[.
The var-of-means consistency pass is reachable from cover(). The two
halves a control needs are both in place: cover() declares
var.of.means.consistency and var.of.means.min.ess, and
.cover_joint_control() names them when it builds the engine's control list.
The pass places slice points around the modal cell of any axis whose marginal
has collapsed onto too few nodes to carry a spread, so the reported axis SD is
a spread rather than a floor at zero. It runs whatever adaptive.grid is set
to, and on a declared axis as well, so a fit that must integrate the nodes it
was given and nothing else sets it FALSE.
A screened cell is no longer reported as a non-converged one. A cell the
engine's cheap pass dropped is never solved, so it holds the same non-finite
log-marginal a failed inner Newton leaves. .tobs_joint_ok_cells() separates
the two on the engine's prune_mask, and reads both the count and the
denominator over the cells that were actually solved. On the 563-cell test
fixture 539 cells are screened, and reporting them as failures told the caller
the fit had collapsed.
The outer-grid cost controls are reachable from occu_cover(). The
engine's cheap-pass screen (prune, prune_tol) and its two placement knobs
(auto_recenter, recenter_pilot) were built for exactly this fit -- a copy
model whose outer posterior collapses onto a handful of cells, each costing a
full inner Newton on an areal field -- and none of them could be set. Only
cover() declared the prune keys, and occu_cover_joint.R builds the control
list it hands tulpa_nested_laplace_joint() term by term rather than
forwarding control, so a key reaches the engine only if the family declares
it AND the builder names it; neither half held for any of the four. Setting
control$prune on an occu_cover() fit failed at the front door with
"'prune' is not a known control option", with no near-match hint, which is
what a missing declaration looks like as opposed to a misspelling. The
forwarding reads dots[["prune"]], not dots$prune: prune is a unique
prefix of prune.tol, so the partial match turns a fit that sizes the
tolerance into one that screens. Each new key is tested through the
provenance field the engine writes only when the knob fires (prune_mask /
prune_fallback_triggered, outer_grid_pilot,
outer_grid_recenter_declined), since a value-only test passes just as well
when a control is dropped on the floor. auto.recenter takes TRUE / FALSE
here; the joint path recentres on the whole grid's collapsed-edge regime
rather than a per-axis rail, so the per-axis policy names the standalone path
accepts are refused with an error.
The parallel test-worker crash is an R bug, not one of ours
(gcol33/tulpaObs#289). A testthat parallel worker died with 0xC0000005 in
a different file each run, aborting the tier. The fault was captured from
inside the worker with a vectored exception handler -- Windows Error Reporting
writes nothing for processx children and LocalDumps does not override that,
verified with admin -- and lands in wcsstr, reading a page-aligned address,
called from R with no tulpaObs or tulpa frame anywhere on the stack.
R has exactly three wcsstr calls, all in R_is_redirection_tty()
(src/main/sysutils.c), all on FILE_NAME_INFO.FileName, which is documented
as NOT NUL-terminated -- R's own comment says so and passes it to wcsstr
regardless, so the search runs past the name and faults when it crosses into
an unmapped page. Its one caller asks whether fd 0 and fd 1 are msys/cygwin
ptys during Windows startup, and a testthat worker is Rterm with both wired
to processx pipes, which is why workers die at the very start of their file,
why the file varies, and why serial runs never show it.
Fixed upstream 2026-08-11, r-source a2066dd40, "Applied buffer overread patch
from PR19104". NO released R carries it yet: 4.6.1 is 2026-06-24 and 4.6.0 is
2026-04-24. Until the next release, do not run the parallel tier while
anything is building the package -- that is what takes the rate from rare to
roughly one run in five (4 crashes in 18 build-concurrent rounds against 0 in
40 without) -- or run serially. Ruled out en route, each by measurement rather
than argument: the binaries, resource exhaustion, out-of-range vector or Eigen
indexing (both packages rebuilt with -UNDEBUG -D_GLIBCXX_ASSERTIONS, whole
tier, zero assertions), file ordering, and callr.
The two parallel test mechanisms are not the same one, and only one of them
has shown a worker crash (gcol33/tulpaObs#289). test_dir() reads
Config/testthat/parallel and dispatches files to testthat's own callr worker
pool, and test_check() is test_dir(load_package = "installed"), so
R CMD check takes that pool too. .github/scripts/run-tests.R takes neither:
it holds a parallel::makeCluster pool and hands each worker one file via
test_file(), whose parallel formal defaults FALSE. TESTTHAT_PARALLEL
is read only by find_parallel(), which nothing on that script's path calls,
so the variable set in smoke.yaml and full-recovery.yaml reaches nothing
and R-CMD-check.yaml was the only job that could reach the callr pool -- with
the variable pinned false, which is why no CI job ever has. It is now false
on push and pull request, so the gate stays deterministic, and true on
workflow_dispatch, which is the deliberate three-platform run. CLAUDE.md's
claim that the recipe is "what run-tests.R does" and that it is "safe at any
worker count" is corrected: what gcol33/tulpaObs#151 established is that a
worker no longer recompiles the DLL, not that the pool is sound. A clean tier
read while gcol33/tulpaObs#289 stands means running serially.
The cross-arm copy axis takes a resolution, not a set of nodes
(gcol33/tulpaObs#287). The copy coefficient's outer axis carries prior
structure -- the atom at alpha = 0 that gives the no-coupling model
posterior mass, and the log-spaced slab above it -- so stating its nodes
restates that structure, and a fit that only wanted the axis integrated more
finely had no way to ask. It was also the one outer axis a coupled fit could
not raise: it does not densify when the donor sigma axis does, so on
informative data the outer grid's quadrature effective sample size saturates
on it while every other axis tracks the request. control$alpha.n (and
alpha.n.trend for the weighted blocks) states a RESOLUTION instead: the
engine re-reads its own axis with that many slab nodes, atom and slab bounds
unchanged, through the alpha_n field tulpa 0.2.6 added. One resolver,
.tobs_alpha_axis() (R/joint_substrate.R), feeds every joint route --
cover(), occu_cover(), occu_multiscale_cover(), MCAR and coupled-trend
blocks included -- and the NUTS warm fit takes the axis's realised nodes,
since the sampled alpha's flat prior takes their span as its support. A
block takes one spelling: alpha.grid beside alpha.n, or a copy()
stating an amplitude beside alpha.n, is refused in the dispatcher where the
message can name the knobs as typed; a bare copy(spatial()) composes with
it, since that copy asks for the default axis.
21 slab nodes on that axis restore a coupled fit's calibration
(gcol33/tulpaObs#288). Posterior SBC on the coupled occu_cover() fixture,
n.sim = 300, at two visit counts and three copy-axis resolutions, donor
field-SD and cover-dispersion axes pinned so the copy axis is the only thing
moving. At the engine's declared resolution the field SD goes from calibrated
at 3 visits per site (uniformity p = 0.17) to clearly too narrow at 10
(9.1e-05), and the copy amplitude with it; at control$alpha.n = 21 every
scored quantity is calibrated at both, smallest p 0.138 and 0.107 against a
1e-3 gate, with the mis-scaled controls still failing everywhere. Thirteen
slab nodes are not enough -- the field SD still reads 4.5e-03 -- so the
declared five are about four times too coarse, not slightly. The axis's
resolution does not only decide the copy amplitude: the cover dispersion
fails at the default too. And resolution is what is identified, not
placement: a stated 22-node axis reproduces alpha.n = 21 to every digit.
The default is left where it is -- the price is a 2.3-3x longer fit, the
outer grid being a tensor, and a point estimate is not affected the way an
interval is -- so ?occu_cover and ?cover now say when to set it.
Issue references dropped from user-facing help. The community-mean and
dispersion-interval notes on ms_abun(), simulate_ms_abun() and
nmix_laplace_re() cited tracker numbers for measurements the text already
states in full. The rendered help names the fixture and the measurement
instead.
Every outer-grid weight this package rebuilds carries the grid's own
quadrature element. A post-processor that renormalised
exp(log_marginal - max) weighs each node equally, which is the measure the
engine integrated only when the nodes are evenly spaced and the axis carries
no declared prior. tulpa 0.2.1 makes the outer grid a quadrature rule for a
declared prior, so the two parted. .tobs_grid_weights()
(R/joint_postprocess_shared.R) is the one place a consumer weight is now
formed: it reads the fit's own cell widths through tulpa:::.nl_grid_log_quad()
and normalises through tulpa:::.nl_normalise_weights_safe(), falling back to
the softmax only where a driver result carries no theta_grid to measure.
Callers moved: the areal count post-processor, ms_abun()'s
(tau, rho) product grid, ms_occu_spatial(),
nmix_laplace_re_spatial(), nmix_laplace_spatial() (both the tau and the
(sigma, rho) reads), nmix_laplace_spde()'s (range, sigma) grid with its
PC prior, and the occu_cover() NUTS warm start, which now reconciles through
the shared .tobs_joint_ok_cells() rather than its own flat softmax.
test-outer-grid-weight-arithmetic.R recomputes both weights from the raw
nodes rather than from the code that produced them, so a consumer that
rebuilds them its own way and drops the prior doing it fails there.
occu_cover()'s NUTS route takes the measure the grid declares, and the
span it integrates. control$copy.slab left unset resolved to "flat",
which was the grid's measure before the engine let an axis declare one; it now
resolves through tulpa:::.hyper_check_copy_slab(), so both engines target the
same model on that axis unless the caller asks otherwise and "flat" is the
alternative rather than the default. The sampler's axis bounds read the fit's
axis_support — the span the outer quadrature integrates, which reaches half a
node step past the outermost node — instead of the range of the nodes
themselves, recomputing it through tulpa:::.hyper_axis_support() for a fit
that does not carry one.
control$copy.slab and control$copy.atom.mass are forwarded, and the
sampler declares its own hyperpriors. The cross-arm copy scale's continuum
measure and the prior probability of the no-coupling point mass at alpha = 0
were fixed at the engine defaults. HyperCoord now carries a prior code and
its parameters, and hyper_coord_backward writes
log p(u) = log p_t(t) + log(dt/du) for either shape; the exponential is
declared on alpha's natural scale and so carries the log-link Jacobian, with
its natural-scale score joining the data score in the same chain rule. A
gradient sampler cannot visit the point mass, so the sampled copy amplitude is
the posterior conditional on a coupled field and copy.atom.mass reaches only
the warm nested-Laplace fit.
control$var.of.means.tolerance is control$var.of.means.min.ess. The
joint fitters forwarded a var_of_means_tolerance the engine no longer
carries: tulpa 0.2.1 triggers the post-integration consistency pass off the
axis marginal's quadrature effective sample size rather than off one SD
estimator's ratio to another, so every joint occu_cover() and occu() fit
died in tulpa_check_control() on an unknown knob. Unset now means the
engine's own default instead of a copy of it restated here.
Engine pin moved to tulpa 0.2.1 (Imports floor and the Remotes tag
together), which is where the declared-prior quadrature these paths read lives.
occu_multiscale_cover()'s remaining two divergences from its occu_cover()
twin (#262). The MAR-cover divergence closed in 0.0.245 was one of three the
issue filed together; these two were still open:
.occu_cover_coupled_arm_priors() defaults the cover arm to
cover_priors() even with no explicit spec, because the cover intercept
trades off against the shared field's level over the cells it can see (only
the detected ones) and the field's sum-to-zero constraint does not pin that
direction -- left at the engine's flat 1e-4 ridge, the intercept SD runs
away and blows up predict()'s conditional cover through Jensen.
.occu_mscale_cover_arm_priors() resolved a NULL cover spec instead of
the default, so every multiscale joint fit ran the cover arm at that flat
ridge unconditionally. It now resolves cover_priors() the same way.log_disp under no prior at all. Both sibling
NUTS targets (occu_cover_nuts.cpp, cover_nuts.cpp) add a weak
N(0, sigma_logdisp^2) prior to keep the sampled dispersion proper;
occu_multiscale_cover_nuts.cpp left it flat -- sigma_logdisp appeared
nowhere in the file. mscale_cover_nuts_eval() now takes the same prior
term, sigma.logdisp = 5 is threaded through
.tobs_fit_occu_multiscale_cover_nuts() /
.tobs_occu_mscale_cover_nuts_logpost() and the two exported entry points
(cpp_occu_mscale_cover_nuts, cpp_occu_mscale_cover_nuts_joint_logpost,
both now require the extra argument), and reported in fit$nuts like its
siblings.Three joint routes each bypassed part of the shared joint-postprocess
spine (#258). .tobs_joint_ok_cells() (R/joint_postprocess_shared.R)
is the declared gate for a joint nested-Laplace fit: drop outer-grid cells
whose inner Newton failed, error when none survived, warn when some did,
and reconcile the engine's weight vector when it collapsed to all-NaN.
Only occu_joint and occu_cover() called it. cover()'s joint route
skipped the gate entirely -- .cover_joint_beta_moments() aggregated betas
with the raw engine weights and never inspected log_marginal, so a fit
with a failed cell reported converged = TRUE, no warning, and betas
averaged in a NaN mode. occu_multiscale_cover()'s route reimplemented the
cell filter but dropped the weights reconciliation, so predict() / WAIC
on a fit with one failed cell handed tulpa::tulpa_posterior_draws() an
all-NaN weight vector -- the exact condition the reconciliation exists to
fix. cover()'s joint route was also the only one that did not splice
pareto_k onto the top-level fit. .cover_joint_beta_moments() now runs
every cover joint fitter through .tobs_joint_ok_cells() and returns the
reconciled fit plus a converged flag the decode reports; the
multiscale-cover route now calls the shared gate instead of its own copy;
the cover decode splices .tobs_promote_pareto_k() beside the outer-grid
record like its three siblings.
laplace_sla on cover() reaches the scores it was computed for (#268).
Both cover routes built skew-normal pseudo-draws at fit time and stored them
on the fit as draws_occ / draws_pos. Nothing read them:
.tobs_cover_eta_draws() is the only per-draw eta source for ploglik / PIT /
PPC / LOO / WAIC / predict(), and it rebuilt Gaussian draws from the arm
Hessians instead, so method = "laplace_sla" and "nested_laplace_sla"
produced numerically identical scores to method = "laplace". The correction
is now imposed where the draws are consumed, by
.tobs_cover_sla_arms(): the separate-Laplace route reshapes the per-arm
Gaussian resample, and the joint route reshapes the posterior bundle's arm
coefficient blocks with the field blocks untouched, so each keeps whatever
covariance its own source carried (the full per-arm V, or the grid mixture
plus the shared field). Measured on the beta-cover fixture, WAIC moves
174.26 -> 173.44. The two stored draw matrices are gone with
.sla_build_cover_hurdle_draws(); a fit carries sla_ref instead.
A cover simplified-Laplace gamma is a skewness of the marginal it was
differenced against (#283). The gamma comes out of
.sla_compute_cover_hurdle() / .sla_compute_cover_hurdle_joint() at
fits$m_*$mode against enc$*_data$X, i.e. in the SCALED parameterization
encode_cover_hurdle() autoscales each arm into, and it was handed the
NATURAL-scale beta_occ / se_occ to build the skew-normals from. Skewness
is invariant under a positive rescale of one coordinate, so a scaled slope is
unaffected, but the natural intercept is a combination of scaled coordinates
and its skewness is a different number: on a centered covariate the reported
intercept gamma was -0.0221 where the natural-scale value is -0.0164. The
marginals the gamma corrects now travel with it as fit$sla_ref, and the
reshaping happens in those coordinates.
A random-effect fit that asked for the correction says it did not get it
(#268). R/laplace.R's RE branch passed approx = "gaussian_laplace" to
build_laplace_fit() rather than the caller's approx, so method = "laplace_sla" with a formula random effect returned Gaussian marginals and
reported sla_status = "off" -- the one status print() suppresses, and the
one that says the correction was never requested. It now records
"fallback_gaussian (random-effect M-step)", the form every other decline in
the package takes. .validate_re_laplace() drops the approx formal it
never read.
The two cover SLA evaluators no longer overflow on a displaced beta
(#268). .sla_gamma_fd() steps the coefficients +-2h along a Sigma column,
so the Bernoulli arms are evaluated at displaced values that can saturate the
linear predictor, where log1p(exp(eta)) overflows to Inf and takes the
arm log-likelihood to -Inf and the finite difference with it. The
occurrence arm (both the separate and the joint evaluator) and the beta arm's
plogis() now take .tobs_clamp_eta(), and the log-normalizer goes through
the new shared .tobs_log1pexp(). The same data got a correction under
int_occu and a silent fallback_gaussian (FD gamma ... non-finite) under
cover().
Two documentation defects in the same layer (#268).
.sla_replace_draws() documented a three-regime fallback keyed on a constant
(SN_GAMMA_MAX) that does not exist anywhere in the package, and an
sla_fallback attribute the loop sets only on a moment-match error; it now
describes the two regimes the loop implements. .sla_compute_dyn_occu()
declined a spatial term with "does not support spatial yet", which is false
for the state arm -- .validate_spatial_laplace() blocks only a
detection-arm SPDE for the dynamic model, so a state-arm spde() does reach
the evaluator, where the decline is the same deliberate one its two siblings
phrase through .sla_spatial_reason(). It now uses that.
occu_multiscale_cover() admits a detected visit with an unrecorded cover
(#262). A recorded presence whose cover was not written down is ordinary in
the releve data this family exists for. occu_cover() has always kept such a
visit and dropped only its cover factor; the multiscale builder refused the
data outright (y_pos must be finite at every detected visit), and had it not,
three kernels disagreed about what a non-finite cover meant -- the two fit
kernels would have carried the NaN into the likelihood while the family's own
pointwise-loglik kernel already gated it. The builder now runs the same
.occu_cover_validate_pos_values() as the two-level family, carrying a missing
cover as NA, and the gate is in the cell-coupling spec, the NUTS target and
the R marginal. Cover is missing at random: the visit keeps its detection, and
because a detected plot's density factorises, the occupancy / availability /
detection arms are unmoved. Data that fits today is unaffected -- the gate
cannot fire on a finite cover.
The detected-plot score behind that gate is now written once
(mscale_det_plot_block() in src/occu_coupling_shared.h), where the
cell-coupling spec and the NUTS target each derived it (#270). They reach the
cover arm through one of two adapters (PosPolicyAccess for the spec's
compile-time policy, PosCodeAccess for the sampler's runtime code), so
neither gives up its own dispatch, and want_logdisp keeps the beta arm's
digamma work out of the grid-integrated paths that do not differentiate the
dispersion.
residuals() has one return shape, and an unregistered family is an error
rather than an empty answer (#261). Three shapes were in use against one
documented contract: list(occ = , det = ), a bare matrix (abun(),
removal(), distance(), dyn_abun(), fp_occu(), where $occ ERRORED),
and list(mu = , det = ) (count(), ms_count(), where $occ was NULL).
Every family now returns list(occ = , det = ): occ the state-level series,
det the observation-level one, either NULL where the family has no such
level. The count families fill det (or occ for the one-row-per-observation
GLMMs) and no longer answer $occ with an error.z their fitted() does not carry -- NA for
every site on an int_occu() fit, and list(occ = numeric(0), det = NULL)
returned as a SUCCESS for ms_nmix / ms_distance / ms_occu_cover. Four of
them are now registered: integrated (the multi-source ever-detected
indicator its own fitted() z branch conditions on), ms_nmix and
ms_distance (the per-species Poisson / negbin marginal their single-species
siblings score), and ms_occu_cover with ms_occu_cover_spatial aliased onto
it (the community state-level surface). t_occu, occu_cover and
occu_multiscale_cover now raise the "no handler registered for model type"
error nobs() already raised, instead of returning an empty vector.test-residuals-contract.R.check_model(fit, coords = ) keys its Moran's I panel on the computed
statistic rather than on coords being supplied, and passes the residual it
already computed into the panel instead of recomputing it. A fit whose
residuals() throws no longer prints the whole report and then errors,
losing the plot and the invisible return.ms_abun(), ms_distance() and ms_occu_cover() have a real newdata
predictor (#256). The three fell through predict.tobs_fit() to the generic
single-season occupancy path, which ignores newdata (returning the in-sample
fitted()) and, given X.0, reports plogis() of process 1. For ms_abun()
and ms_distance() process 1 is the LOG-link abundance arm, so
predict(fit, X.0 = X) returned plogis(X b_lambda) labelled as an occupancy
probability. Each now predicts its own arms: "abundance" / "detection",
"lambda" / "sigma", and "occupancy" / "detection" / "cover_cond" /
"cover_exp" -- the vocabulary the spatial-factor ms_occu_cover() twin
already answers to..TOBS_MS_PREDICT_ARMS): the design formula, coefficient block, fitted()
slot and inverse link of every response arm. The predictor, the default
response type, the shared-handler alias and the terms refusal are all read
off it, so a family gains the whole surface by gaining a row rather than by a
fourth near-duplicate handler.newdata for all six families, not only for spatial_field: a latent-factor
ms_occu() fit (occu_factor_offset) was predicting at new rows with the
factor contribution silently dropped.fitted() and predict() read, rather than the same three-way branch twice.predict() on a non-spatial ms_occu_cover() fit returns
the requested arm's matrix (occupancy by default), where it used to return the
whole fitted() list. Every other community family already reported one arm.ms_dyn_occu forward marginal, and a dead site is refused rather than
dropped (#259). .ms_dyn_occu_fwd_ll_vec() summed ll[is.finite(ll)], so a
site whose forward mass underflowed left the species' log-likelihood finite
over the surviving sites -- a value the EM line search accepts. The scalar
.ms_dyn_occu_forward_ll() the laplace route used instead returned -Inf
for the same data, so the two kernels documented as equivalent disagreed on
exactly the step that matters. The vectorised kernel now sums every site, the
scalar one is deleted, and the laplace route and the SBC registry entry
drive the same kernel the NUTS, spatial and warm-start routes already did,
off the per-species detection sufficient statistics built once
(.ms_dyn_occu_emit_stats()) rather than rebuilt per call. The kernel is also
defined at one season, which the 2:n_seasons loop was not.
test-ms-dyn-occu.R checks it against an independent per-site filter, at the
underflowing site and at n_seasons = 1.test-ms-abun-nb-rs.R's coverage gate asserts what is measured (#250
item 1). expect_gt(mean(covered), 0.8) carried no measurement and did not
pass. #250's preferred replacement was coverage conditional on the dispersion
variance being recovered, from a 39-fit pool at 8 and 36 species where fits
returning sigma_log_r >= 0.30 covered 33/34 against 2/5 below. Re-measured
at the 18 species the fixture actually uses (19 fits, same seeds): one fit
sits below 0.30, cor(|err|, sigma_log_r) is -0.08 and cor(se, sigma_log_r)
is +0.98, so the error does not grow as the SD shrinks and only the interval
does. The conditional form has nothing to condition on there and is not what
is asserted. What is: the estimator is unbiased (+0.017, 0.42 SE of the mean),
z is normal (Shapiro p = 0.515), and the interval scale is the whole of the
miss (15/19 at the reported SE, 19/19 at 1.28x). The last assertion pins the
defect and so goes red when gcol33/tulpaObs#280 lands, which is when the block
should be rewritten as a nominal-coverage gate.?ms_abun and NOTES_measurements.md no longer state the conditional
account as general. Both recorded the 8-and-36-species split without the
group count it belongs to. The measurement is kept and scoped, and the S = 18
re-measurement is recorded beside it.converged = TRUE before #281, which is the
difference between the 16/20 previously recorded and 15/19 now.sigma_log_r / sigma_omega was
invisible from the fit: the optimizer converged, the point estimate was
ordinary, and the community mean the component scales reported an interval
that shrank with it. .tobs_aghq_variance_boundary() reads the component's
own curvature instead of cutting on sigma_hat, which is a number with
nothing behind it and does not transfer between fixtures. A 1x1 covariance
block's integration coordinate is log(sigma), so tulpa_re_aghq()'s
re_par_se is SE(log sigma) directly and the delta method reduces the Wald
statistic for H0: sigma = 0 to 1 / SE(log sigma), free of the component's
own scale. The critical value is qnorm(1 - alpha): sigma = 0 is a boundary
point, where the one-sided null is 0.5 chi^2_0 + 0.5 chi^2_1 (Self & Liang
1987; Stram & Lee 1994) and P(W > c) = 1 - Phi(c), so the boundary-aware
value and the ordinary one-sided normal quantile coincide. ms_abun() fits
carry the record on sigma_log_r_boundary / sigma_omega_boundary and raise
one warning naming every component that failed. Unblocked by gcol33/tulpa#418,
which added re_par_se / re_par_layout in tulpa v0.1.18.Two simulators drew from a different model than the likelihood they mirror
(#257). Both matter beyond simulate(), because SBC replicates route
through them.
cpp_simulate_dyn_abun() read the survival and recruitment designs at the
per-site row stride unconditionally. omega and gamma are INTERVAL-level --
the transition into season t uses interval t-1's rate, and
.tobs_interval_arm_design() returns n_sites * (T-1) site-major rows
whenever the arm carries a season-varying covariate -- so every non-intercept
column was read from the wrong memory region and the interval variation the
forward HMM integrates was dropped, the rates coming out constant across
seasons. The reads stayed in bounds, so nothing errored. Both shapes are now
read exactly as src/dyn_abun_laplace.cpp reads them, and a design of neither
shape is refused rather than indexed on an assumption. The interval lookup
draws nothing, so a constant-rate fit simulates byte-identically.
cpp_simulate_distance() hardcoded a 64-node Gauss-Legendre rule and had no
quad_order parameter, while quad.order is a user-facing knob every
likelihood path reads from model$quad_order. A fit at any other order was
simulated from per-bin detection probabilities it was never fit against, and
an ms_distance SBC run at a non-default order compared ranks against a
different generative pi. The rule is (cutpoints, transect, quad_order), so
the order is now an argument and each of the three call sites passes the one
its model carries; simulate_ms_distance() gains a quad_order argument
(default 64) for the same reason.
A failed per-group AGHQ solve is a field on the fit, not a warning to parse
(#281). tulpa::tulpa_re_aghq() reports, per group, whether that group's
posterior mode search and precision factorization succeeded; a group it marks
failed comes back with NA BLUPs, so every per-group quantity read off it is
NA and every community-level quantity summing over groups is missing that
group's information. Every AGHQ consumer here took the result unconditionally,
which is how ms_abun(mixture = "negbin") returned its community dispersion
block at the values it STARTED from -- mu_log_r a hair below log(r_init),
sigma_log_r exactly sigma_logr_init, one NA in r_s, converged = TRUE
and the tightest log_r standard error of twenty seeds -- with a runtime
warning as the only signal and the failing index recoverable only by parsing
its message text.
The status now travels on the fit. convergence(fit) carries group_ok,
groups_failed and groups_failed_names (species, for the community
models), and converged is FALSE whenever any group failed. One predicate
reads it (.tobs_aghq_group_status()), so occupancy, N-mixture, removal,
distance, fp_occu(), dyn_abun() and the community N-mixture answer the
same way. The occupancy EM declines the AGHQ refinement on a failed group and
keeps the EM result, which is what that pass already promised on any other
failure; the community spatial-Newton loop refuses the node rather than
handing NA coefficients to the shared-field solve.
convergence(fit)$n_iter is no longer NA on a community N-mixture fit
reported as converged: the EM path reports its iteration count and the joint
AGHQ path the optimizer's evaluation count (tulpa 0.1.23), which is what the
package's other stats::optim-driven fitters already report there.
The error raised when the AGHQ engine returns no community fit named a singular Hessian as the cause. The engine declines for three reasons, and on two of them the cause is a per-species solve that failed -- which its own warning names. (#281)
The fitted field now reaches every post-fit door on the five count families
(#254). abun(), removal(), distance(), fp_occu() and dyn_abun()
estimate a latent areal / temporal / continuous surface that is part of the
arm's linear predictor, but every reader rebuilt eta from
model$X_processes and the coefficients alone. fitted() returned
exp(X beta) for a fit whose log-mean is X beta + f; residuals() and
predict() inherited that, and WAIC / LOO / DIC / CPO scored a model the fit
never made. On a 25-cell ICAR N-mixture the reported per-site abundance
correlated 0.32 with the truth where the fit's own predictor correlates 0.91.
Nothing warned.
The offset is now recorded once per ARM on model$field_eta_offset, in that
process's own design row layout (R/field_offset.R) -- written by the shared
areal-BFGS field attach, the shared NUTS field attach, and the two C++ areal
branches, then moved onto the fit's model at the ONE dispatch tail the seven
observation-family branches now share (the model swap there is what dropped
whatever a fitter set on its own autoscaled copy). fitted() / residuals()
/ predict(), the five pointwise-log-likelihood kernels, and simulate()
(hence test_dispersion() / test_zero_inflation() / test_outliers() /
check_model()) all read it through one accessor. The deterministic and the
sampled (fixed-hyper NUTS) field paths, a detection-arm field with its
per-pass expansion (#114), and a temporal-only or svc() block all ride the
same slot. simulate() carries it as a design column with its coefficient
pinned at 1, so the C++ simulators are untouched and a field-free fit is
byte-identical. A fit with no field carries no offset and is unchanged.
logLik() on a SAMPLED field fit was the same omission one level up: the
field is not one of the coefficient draw columns, so each family's
posterior-mean marginal evaluation ran it at offset 0 and log_prob --
what logLik() / AIC() / BIC() / glance() read -- described a model
without the field while the criteria scored one with it (on a 25-cell
car_proper N-mixture, -232.11 against the field-aware -199.89). It is
re-evaluated through the same pointwise kernel the criteria use, so the two
cannot disagree. The deterministic grid-integrated field paths already
evaluate their marginal with the field in the predictor and keep their own
grid-weighted value.
count() reads the shared field -> per-site contribution helper now
(.tobs_spatial_field_offset(), was .count_spatial_field_offset()); its own
count_field_offset slot is unchanged.
A detection = ~ icar(...) term is no longer fitted as a spatially-varying
ABUNDANCE (#255). removal(), fp_occu() and dyn_abun() rejected a
detection-arm field under NUTS with a pointer to nested_laplace; three
sibling fitters had no such check and loaded the term on the state arm
instead -- .tobs_fit_distance_nuts_spatial(),
.tobs_fit_abun_nuts_spatial() and .tobs_fit_nmix_spatial(), so abun()
routed no arm on EITHER method. Asking for a spatially-varying detection
probability returned a spatially-varying abundance, silently: on a 25-cell
fixture the field a "detection" term produced recovered the ABUNDANCE truth
surface at cor 0.90. The check is one shared helper
(.tobs_reject_det_arm_spatial()), called by the six single-species areal
fitters and both ms_abun() areal fitters, so a ninth cannot omit it.
abun() / ms_abun() say so explicitly (no detection arm is wired on either
method); the four observation families keep their pointer to
method = "nested_laplace", where that field is fitted.
Six R sites that stated one fact twice, or guarded on a field nothing sets (#274). Three were defects rather than duplication:
control$sigma.grid / rho.grid / tau.grid now reach the multi-block
nested-Laplace route. .tobs_block_from_spatial() / _temporal() /
_re() read their outer-grid overrides off the TERM object
(spatial$sigma_grid), and no constructor or binder ever wrote those
fields, so twelve guards were dead and the route had no grid override at
all -- while the control validator ADMITTED the names, so a user setting
one had it silently dropped and cover() / occu_cover() drove the same
engine with a working knob. The grids arrive as an argument now, and
rho.grid / tau.grid / range.grid join the allowlist beside
sigma.grid.occu_categorical) called neither. On ms_abun(), whose binder stores
X_det_visit and appends its columns to the detection coefficient names
while formulas$det keeps only the site-level arm, that omission drew
theta from a posterior carrying visit-level coefficients and refit a model
that had none, with nothing in the ranks saying so. Both checks are one
call now, made by all twenty specs.community_latent.R, replacing three that disagreed on the ridge scale,
the tier count and the failure value. The relative one guarded its second
tier on a finite mean diagonal and reused the same value unguarded on its
third, so a non-finite diagonal gave max(NaN, 1e-6) = NaN as the ridge --
in exactly the near-singular case the retry exists for. The field Newton
also applied its step unconditionally, so a non-finite step poisoned the
field, made max(abs(step)) < tol NA so the loop never broke early, and
reached the tau M-step with no error and no flag; it holds the step now,
as the factor Newton beside it already did.The rest is duplication removed: eight family dispatchers restated a method
rejection .tobs_validate_family_method() already makes before dispatch
(each of those families lists exactly one method, so the branches were
unreachable); .map_engine() kept a second nested_laplace family list that
had drifted from the registry in both directions -- carrying two families
that dispatch directly and never reach it, missing seven the registry does
list -- and now reads the registry; and the positive-arm log density had
three boundary policies across R (log_safe, a raw log, and a clamp on the
RESPONSE), so at a cover of exactly 1 the NUTS oracle and the sampler it is
asserted byte-for-byte against differed by -Inf against a finite value. All
three now use the policy of the C++ kernel they mirror.
API.md, ?tobs_terms and three diagnostic doors match the package again
(#278). README calls API.md the full reference; it was 25 exports stale,
named a tobs_check that does not exist, omitted pg_gibbs entirely and the
occurrence / positive / by formals, listed 8 of 31 simulators, and
inverted n.iter (documented as total iterations at a default of 2000; it is
the post-warmup draws KEPT, default 1000). Its export list and simulator
table are regenerated from NAMESPACE, and
.github/scripts/check-api-md.R (a step in the smoke and tarball workflows)
now fails when the export list, the simulator table, the tobs() signature
or a sampler default drifts from the package. The three ?tobs_terms
examples all failed on two counts each -- two positional arguments filled
formula and family in order, and the two-sided formula was resolved
against a response occu() does not carry -- and are now a runnable areal
fit plus corrected \dontrun{} sketches. The ?tobs sigma.beta bullet
gave one number where the table resolves two.
Three behaviour halves came with it. test_outliers() returned
$n_outliers on the single-season branch and $observed / $ratio on the
count branch, so no caller could read the statistic by name across families;
the names are now the same everywhere. pit_residuals.tobs_fit() accepts the
nsim / seed / observed arguments tulpa::test_uniformity() forwards --
none was a prefix of n.samples, so test_uniformity(fit, nsim = 2000, seed = 7) ran at 250 draws with an unseeded draw selection and gave a
different KS statistic on every call; the caller's RNG stream is restored
afterwards. And tobs_check_id() gated its pre-fit checks on
model_type %in% c("single", "community"), a type no family assigns, so
every family except single-season occupancy skipped all of them and still
printed "No identifiability issues detected."; the gate is now the four
families whose response IS a site-by-visit grid, and elsewhere it says the
checks did not run.
residuals() scores a negative-binomial count fit at its own variance
(#260). abun(), removal() and distance() computed the Pearson and
deviance residual from a Poisson variance whatever the fitted mixture, so a
negbin fit was scored at mu where its marginal variance is
mu + mu^2 / r. The negative binomial is closed under binomial thinning with
the same size, so those three per-cell marginals are NB(r, lambda * pi) and
now share one helper with count(), .tobs_count_residual().
dyn_abun() deliberately does NOT use that helper: its estimated size
describes the INITIAL abundance, and N_t for t > 1 is a survival-thinning
plus recruitment convolution that is not negative binomial. Its Pearson
residual uses the exact moment recursion
Var[N_t] = omega^2 Var[N_{t-1}] + omega (1 - omega) E[N_{t-1}] + gamma,
Var[y] = p^2 Var[N_t] + p (1 - p) E[N_t], validated against 150k-draw Monte
Carlo (max relative error 0.0043 Poisson, 0.0088 negbin). Under a Poisson
initial abundance Var[N_t] collapses to E[N_t] at every season, which is
why every Poisson path is byte-identical rather than merely close.
control$n.thin and control$n.threads reach every NUTS route (#273).
Both were documented in ?tobs, admitted by the control validator and
carried in the engine table, and exactly one route -- the occupancy families
-- read either one, so a user thinning a long community chain got the
unthinned chain and no warning. Every chain loop now runs through
.tobs_nuts_run_parallel() / .tobs_nuts_thin_chain(), and thinning applies
to the per-iteration divergent / accept_prob / treedepth series as well
as to the draws, so a diagnostic cannot end up misaligned with the draws it
describes. n.threads = 1L joins the NUTS table. Two more instances of the
same mechanism: .tobs_fit_model() had no sigma.logr formal, so
control$sigma.logr fell into ... for abun() / removal() /
distance() under NUTS while a hardcoded 1.5 won; and its
n.threads = 1L literal suppressed the fitter-side filler. Both are NULL
sentinels resolved from the table now.
The community NUTS gradient targets take a thread cap (#265). The four
per-species gradient loops (ms_occu(), ms_count() / jsdm(),
ms_abun(), ms_dyn_occu()) opened #pragma omp parallel with no
num_threads clause, so every leapfrog step spawned OMP_NUM_THREADS
threads with no ceiling. New control$n.threads.grad caps the region; the
default 0 leaves the count with OpenMP, so the shipped behaviour is
unchanged. It is deliberately not n.threads, which on a sampler means whole
chains run in parallel. The per-species reduction is serial and order-fixed,
so the chain is identical at any thread count -- asserted to the bit.
Issue-tracker references removed from source, documentation and tests
(#249). 954 of them: comments, roxygen (23 of which rendered into 16
shipped .Rd files), 29 inside stop() messages that printed to users, and
197 in the bare tulpaObs#NNN spellings. A comment states what the code does
or why the domain demands it; a tracker pointer is neither, and it does not
survive a renumbering. Where a reference sat beside a fact worth keeping the
fact stays; where the sentence stopped making sense without it the sentence
states the reason directly. NEWS.md keeps its references, which is what a
changelog citation is for, and
.github/scripts/check-issue-refs.R (a step in all three workflows) fails on
any that reappear elsewhere.
control$sigma.logr is reachable on the NUTS routes. The prior SD on the
community-mean log-dispersion mu_log_r was read from control by the
ms_count() and jsdm() dispatchers, but the name was in no group of the
control allowlist, so tobs() rejected it with "'sigma.logr' is not a known
control option" before either read saw a value. The ms_abun() NUTS branches
did not take it from control at all -- the spatial one passed a literal
1.5 beside a sigma.beta that was read from control, and the non-spatial
one omitted it. All three now behave the way sigma.beta already does. Unset
is unchanged: control[["sigma.logr"]] is NULL and resolves to the engine
table's 1.5, which is the value the spatial branch hardcoded. ?tobs now
documents it, and sigma.beta, in the NUTS control block. Note the default is
informative on the dispersion scale, so raise it to compare the sampler
against a maximum-likelihood target.
Scalar nuisance AGHQ blocks are floored at 3 nodes (#234). BEHAVIOUR
CHANGE on every ms_abun(mixture = "negbin" / "zip" / "zinb") Laplace fit.
control$n.quad.scalar defaulted to 2 and was clamped into [2, n.quad], so
the negative-binomial dispersion block log_r and the zero-inflation block
logit_omega integrated at two Gauss-Hermite nodes regardless of n.quad --
the family's own documented n.quad = 5 included. A 2-node rule places two
nodes and has no freedom left to represent curvature, so where the 1-D
posterior is not near-Gaussian the marginal it returns can come out
arbitrarily sharp: on one seed of the test-ms-abun-nb-rs.R fixture the
reported SE on mu_log_r read 0.0110 against 0.1886 at three nodes, a 17x
collapse, with the fit converged, the point estimate ordinary and no warning
emitted. Every downstream Wald CI, confint(), coverage check and SBC rank
built on that coordinate inherited it. The rule is converged at three nodes --
3, 5 and 9 agree to ten decimal places on that block -- so n.quad.scalar now
defaults to 3 and takes 3 as a hard floor rather than a default, and the floor
overrides a smaller n.quad (a coefficient block may legitimately run at the
plain-Laplace order; a block whose marginal SE is reported may not). The
per-block quadrature saving is mostly preserved: the tensor grid grows 1.5x on
that one axis at n.quad = 5, where integrating it at the full n.quad would
be 2.5x. It is also a point-estimate fix, not only a rescue of the pathological
seed -- across the nineteen fixture seeds that never collapsed, every SE moves
under the floor (median -4.6%) and mean bias on mu_log_r drops 77%, from
+0.0482 to +0.0109. Measurements in NOTES_measurements.md. The residual is a
multiplicative understatement of the mu_log_r SE with no bias left, tracked
separately in #235; test-ms-abun-nb-rs.R:94 measures 0.800 against its
> 0.8 assertion and stays red until that one is resolved.
BYM2 uses the Riebler marginal-variance scale factor (#232). BEHAVIOUR
CHANGE on every bym2() fit. .bym2_scale() returned the geometric mean of
the non-zero eigenvalues of the intrinsic precision Q = D - W; the constant
the BYM2 parameterization is defined against is the geometric mean of
diag(Q+), the generalized-inverse diagonal, which is what
INLA::inla.scale.model() applies (Riebler et al. 2016, section 3.2). The two
are not close and do not even move the same way with graph size: on 5x5 /
10x10 / 20x20 lattices the old constant gives 2.651 / 2.832 / 2.985 where the
correct one gives 0.516 / 0.645 / 0.765. The structured component was
therefore scaled by roughly sqrt(2.773 / 0.605) ~ 2.1 in the wrong
direction, and sigma absorbed it: over 8 seeds on a 35-cell graph at truth
sigma = 1, rho = 0.7, the old constant recovered sigma = 1.429 (high on
8 of 8 seeds, range 1.13 to 1.91) against 0.921 now. Field correlation is
unchanged (0.896 against 0.897) -- the shape of the surface was never wrong,
only the variance it was reported at. rho is mildly worse in the mean
(-0.105 against +0.026) but per-seed scatter puts the standard error near
0.065, so at 8 seeds neither shift is decisive; it is reported, not claimed.
Verified against INLA::inla.scale.model() to 6 digits on three lattices and
against the tulpa engine's own constant to machine precision.
The two loading conventions for scale_factor are named (#232). The
package's own R paths write sigma * sqrt(rho / s) and the tulpa engine
writes sigma * sqrt(rho) * scale_factor, so the engine's argument means
1 / sqrt(s). That reciprocal was inlined at each of the three engine
boundaries. .bym2_engine_scale() (R/spatial.R) now performs it, and
.bym2_scale_from_Q() is the single definition both conventions read;
.tobs_latent_bym2_scale() in R/community_latent.R was a fourth copy of the
eigenvalue formula and is deleted.
One block-offset walk behind the six community NUTS layouts (#231). The
six community samplers pack the same vector -- community means, then
species-major deviations, then a log-Cholesky block per arm, then whatever
trails -- and each rebuilt that walk privately, so an arm added to one family
had to be re-derived rather than declared. .ms_ocs_layout()
(R/ms_occu_cover_spatial_nuts.R) takes the arm widths and the trailing
declaration and returns the slices; a family aliases them onto its own
spelling, so the builder never learns a family's names. All three trailing
shapes in use are the same declaration: a shared vector (ms_dyn_occu's
gamma / eps globals), a shared scalar (ms_occu_cover's log-dispersion), and
a per-species vector (ms_count's gaussian log-variances). Behaviour is
unchanged: the R logpost and gradient oracles stay byte-exact against their
C++ twins on all seven community NUTS files (249 assertions, 0 failures).
One registry entry builder behind the eight community SBC families (#230).
Each community row in .TOBS_SBC_REGISTRY restated the same names / draws
/ simulate / loglik_many construction around a per-family kernel.
.tobs_sbc_community_entry() (R/sbc.R) builds the row from the arm
declaration plus that kernel; spec / data / pool / refit stay the
family's own, since what a community family constructs and how it pools is not
shared. R/sbc.R loses 590 lines. Ranks are unchanged: run at the acceptance
blocks' own fixtures on both sides of the refactor, all eight families return
a bit-identical per-replicate PIT table (2952 rows over 369 quantities, 8
replicates each, n.draws = 1000, n.ref = 200, seed = 0), and the wall
times track to within 2 percent. This is an equivalence measurement, not the
rank-uniformity blocks in test-sbc-registry.R -- those score whether a
posterior is calibrated, which is a different question and is tier 3.
One preamble behind the count-marginal spatial fitters (#229). The three
areal N-mixture wrappers, the SPDE wrapper and the three removal wrappers fit
the same count marginal over the same designs and so validated the same
arguments; each inlined the coercions, dimension and adjacency checks, the
beta warm starts and the latent-N truncation, and the seven copies of the
K_max boundary warning were identical character for character.
.count_spatial_prep() (R/nmix_laplace_spatial.R) now resolves all of it,
taking the family's truncation rule (.count_K_floor_max_y() for the
N-mixture, .count_K_floor_site_total() for removal, where the depleting
passes at a site sum) and the names of its latent warm starts; the boundary
warning is .count_spatial_warn_boundary().
The areal outer-grid defaults are settled per axis, not per wrapper
(#229). The copies had drifted where the code did not show which value was
chosen and which was inherited. Two removal defaults MOVE, both to the
N-mixture spelling: removal_laplace_car_proper()'s tau_grid from 9 nodes
to 7 and its rho_grid from seq(0.1, 0.95, length.out = 6) to
c(0.1, 0.3, 0.5, 0.75, 0.95). The 2D proper-CAR grid is a product over both
axes, so 7 x 5 = 35 inner Newton solves is already four times the 1D ICAR
grid's 9 -- a defensible reason for the coarser tau axis, and the reason it
belongs to the axis rather than to whichever wrapper the caller reached.
.count_spatial_default_grid() is now the only place any of them is written.
An explicit tau_grid / rho_grid / sigma_grid is unaffected.
Two range checks now apply at both count doors (#229). The removal
proper-CAR wrapper did not validate rho_grid at all, so rho = 1 reached
the dense Cholesky behind log|Q(rho)| at a rank-deficient Q; the removal
BYM2 wrapper checked only scale_factor, leaving sigma_grid <= 0 and
rho_grid outside [0, 1] to the kernel. Both now go through
.count_spatial_check_grid(), which the N-mixture doors already had in
inlined form. test-count-spatial-prep.R.
AGENTS.md added, the Codex-facing counterpart of CLAUDE.md. Same
contract: rules and interfaces here, the measurements they rest on in
NOTES_measurements.md. Listed in .Rbuildignore alongside CLAUDE.md, so
R CMD check does not report a non-standard file at top level.sbc() registry test reads ms_abun()'s ranked quantity, not its
community mean. .sbc_reg_means() carries one branch per community
family because a community fit's fit$means is the community MEAN alone
(P-length) while its SBC design ranks the per-species REALIZED coefficient
mu + b_s (S x P, species-major). ms_abun() was registered without its
branch, so it fell through to the fit$means short-circuit and returned a
2-length vector against the registry's 40 -- the exact failure the comment
above that helper describes. Branch added, mirroring ms_distance()'s:
as.vector(t(cbind(coef_lambda, coef_p))) against
.tobs_sbc_ms_abun_names()$cols. All eight community families now have one.The sbc() registry's end-to-end composition test slices the pooled
response at whatever rank the family brings. Its slice_site() helper
enumerated ranks -- 3D array, else 2D matrix -- so ms_dyn_occu(), whose
y is 4D [site x visit x season x species], fell to the 2D branch and
errored with "incorrect number of dimensions". .tobs_sbc_pool_4d_species()
was right: every registered family pools on the first axis and leaves the
rest alone, so the helper now indexes axis 1 at any rank and the
rank-enumeration cannot go stale again the next time a family arrives with
a new response shape.
man/dot-tobs_community_em.Rd regenerated, clearing an R CMD check
codoc WARNING. .tobs_community_em() gained re_aghq/n_quad/
re_aghq_maxdim (with their roxygen) when the AGHQ variance debias was
generalized onto it, but document() was not re-run, so the Rd's \usage
had been three arguments behind the function since then.
ms_abun() registered for sbc() (gcol33/tulpaObs#220, the 27th and
final family of the original registration scope). Needed TWO upstream
tulpa engine changes before it could even be attempted:
tulpa#398 (blup_cross): exposes tulpa_re_aghq()'s per-group
mode/theta cross-Hessian.tulpa_re_aghq() also
only ever exposed the per-RE-TERM DIAGONAL of a group's posterior
covariance (blup_var). ms_abun()'s lambda (abundance) and p
(detection) arms share a per-species grouping factor with real
cross-arm posterior covariance -- the same lambda/p identifiability
ridge tobs()'s penalized-EM exists to break for occupancy psi/p --
so drawing a species' abundance and detection deviations
independently would repeat the #226 bug one level deeper (inside a
species instead of between the community mean and a species). Fixed
upstream by adding blup_cov_g/blup_cross_g (the full per-group
joint covariance/cross-Hessian, unsliced by RE term), validated
against a closed-form joint-Hessian construction on a toy model with
deliberately collinear RE terms.ms_community$Cinv/Bf now thread off blup_cov_g/blup_cross_g
(R/ms_abun.R), matching the convention every other sbc()-registered
community family uses. Only reachable via the AGHQ/joint_fd engine
(control = list(optimizer = "joint_fd", n.quad > 1)) -- the default
n.quad = 1 Laplace-EM path (a different, faster engine entirely) does
not expose Cinv/Bf at all, and errors with a pointer rather than
silently drawing an independent (mu, b_s).
Registered at reduced n.sim = 15 (distsamp_open's own precedent for the
identical cost problem, see 0.0.227 below): a n.sim = 100 run at
standard settings was projected to take >12h; the actual n.sim = 15 run
completed in 39.9 min. Posterior calibrates cleanly (min p_unif 3.9e-3,
0/40 quantities below 1e-3); narrow control's minimum p_unif is 2.9e-12
(comfortably below the 1e-3 pass bar every sibling family uses), though
one narrow-arm quantity did not fail as comprehensively as siblings'
(max p_unif 1.3e-2 there, vs ~1e-6 to 1e-15 elsewhere) -- not
re-investigated given the ~40min cost of a re-run, and does not weaken
the primary (posterior) calibration signal.
All 27 originally-scoped families are now registered for sbc().
Engine pin raised to tulpa 0.0.198 (Imports floor and Remotes tag
together, as .github/scripts/check-engine-pin.R requires). That release
carries both engine changes the ms_abun() registration above depends on:
blup_cross (gcol33/tulpa#398) and the full per-group joint covariance
blup_cov_g/blup_cross_g. The floor and the tag had also drifted apart
from each other (0.0.197 vs v0.0.196), which the pin check fails on.
ms_distance() and ms_dyn_occu() registered for sbc() (gcol33/tulpaObs#220,
the 26th and 27th registered families -- the last two of the original
registration scope besides ms_abun(), which stays deferred on
gcol33/tulpa#398).
ms_distance(): community binned distance sampling, S=20/N=150 fixture
(cutpoints = c(0,25,50,75,100), halfnorm/line). A multi-seed (0, 1)
reproducibility probe found a DIFFERENT coefficient dipping moderately low
each time (seed 0: sp15_sigma_(Intercept) at 6.3e-4; seed 1:
sp19_lambda_abund_cov1 at 2.7e-4) -- the false-positive signature (~6%
expected across 60 tested quantities per run), not the same-coefficient-
pinned-low-every-seed signature that would indicate a real calibration bug.
Registered at bad.factor = 3.0 (the 1.75 posterior arm is identical --
bad.factor only rescales the narrow control, verified by comparing both
arms' completed runs): posterior min p_unif 6.3e-4 (1/60 below 1e-3),
narrow max p_unif 3.0e-15.
ms_dyn_occu(): community dynamic (HMM-forward) occupancy, S=20/N=150/
4 seasons/3 visits fixture. Two of the four arms (psi1, p) carry a
per-species deviation; the other two (gamma, eps, shared colonization/
extinction) are community globals with no RE of their own -- draws()
conditions each species' psi1/p deviation on the FULL (mu, global) draw via
the (P+G) x P Bf[[s]] cross-Hessian block (verified by inspecting
dim(Bf[[1]]) directly, not assumed square). Registered at the standard
bad.factor = 1.75: posterior min p_unif 4.1e-3 (0/42 below 1e-3), narrow
max p_unif 6.5e-6 -- both arms tested (1.75 and 3.0), byte-identical
posterior result at both (confirms bad.factor only touches the narrow
arm).
Both acceptance runs (n.sim = 100) took multi-hour wall time (ms_distance
~121min for the bad.factor=3 arm; ms_dyn_occu ~513min/~516min per arm) --
run via detached Windows Scheduled Tasks (the hardware-nodes pattern),
which were themselves killed by an unrelated external event partway through
the first attempt and relaunched, skipping the already-completed
bad.factor=1.75 arm for ms_distance to avoid redundant compute.
jsdm() registered for sbc() (gcol33/tulpaObs#220, the 25th
registered family). jsdm() is ms_count(response = "bernoulli") under
the hood (.dispatch_jsdm routes through the same community Laplace-EM
as ms_count(), gcol33/tulpaObs#121), so its data/draws/simulate
adapters reuse ms_count()'s exact generic helpers unchanged; only
spec (constructs jsdm(), not ms_count(response=)) and loglik_many
(the Bernoulli kernel .ms_count_ll_bern, not .ms_count_ll_pois) are
new. Shares ms_count()'s S=20 fixture scale, resolved the same way for
the same reason (gcol33/tulpaObs#226) -- 3 seeds, min p_unif range
0.0093-0.0108, 0 quantities below 1e-3 out of 40 possible across all 3
runs. bad.factor=3.0 rather than the 1.75 every sibling family uses:
a refit here is cheap (~35s for the full n.sim=100 run) so the standard
budget did not need reducing, but at 1.75 the narrow arm only reached
max p_unif ~0.001 (borderline); 3.0 (same cost) pushed every quantity to
p_unif 0.distsamp_open() acceptance-tier SBC finalized (gcol33/tulpaObs#220,
closing out the last item from the original registration scope). CONTRACT
tier had been verified earlier; the acceptance run needed its own tuning.
A single refit on the standard fixture measured ~100-130s, and the full
posterior-SBC experiment (which refits on a pooled/augmented dataset)
measured 758.5 sec/sim -- at the standard n.sim=100 every sibling family
uses that is ~21h, which is exactly what the first attempt ran into before
being killed with no output. Registered instead at n.sim=15 (~4.3h). At
that reduced simulation count the standard bad.factor=1.75 under-powered
the narrow (mis-scaled) control arm -- only 3 of 6 quantities failed
clearly, lambda_(Intercept) and sigma_det_cov1 showed no signal at all.
bad.factor=4.0 (same n.sim, same cost) pushed every quantity below
1e-5, matching the comprehensive-failure bar every other family's narrow
arm hits. Posterior arm: p_unif in [0.011, 0.904] across all six
coefficients, byte-identical between the two tuning runs (bad.factor
only touches the narrow arm). 24th family registered.ms_count() registered for sbc() (gcol33/tulpaObs#220, the 23rd
registered family) -- completes #226 part 2 for all three families it was
originally filed against (ms_occu, ms_int_occu, ms_count). Originally
the worst of the three on a small fixture (multi-seed 0/1/2 pinned
sp3_mu_(Intercept) at p_unif ~9.6e-7-9.9e-7 every time, no detection arm
to dilute the effect). At S=20 (matching ms_occu's own resolved scale),
the plain Laplace-EM -- no debiasing -- calibrates cleanly: 5 seeds, min
p_unif range 0.0016-0.086, 0 quantities below 1e-3 out of 41 possible
across all 5 runs, no reproducible failing coefficient. Registered with
the fixture and test explicitly pinned to S=20. No explicit pool slot
needed for this family's registry entry -- y is a plain 2D [site x species] matrix, so the shared generic default pooling applies.ms_int_occu() registered for sbc() (gcol33/tulpaObs#220, the 22nd
registered family) -- the species-count scoping that resolved ms_occu()
in 0.0.224 transfers directly. Originally found to share ms_occu()'s
exact failure signature (a probe at three seeds on a small fixture found
sp3_p2_(Intercept) stuck at p_unif ~0.0029-0.0030, reproducible). At S=14
(matching this family's own existing recovery-test fixture size, N=140, 2
detection sources), the plain Laplace-EM -- no debiasing -- calibrates
cleanly: 5 seeds, min p_unif range 0.0013-0.052, 0 quantities below 1e-3
out of 43 possible across all 5 runs, no reproducible failing coefficient.
Registered with the fixture and test explicitly pinned to S=14. ms_count
(same .tobs_community_em() engine, same original failure signature) is
the remaining untested candidate for this fix.ms_occu() registered for sbc() (gcol33/tulpaObs#220, the 21st
registered family) -- resolves #226 part 2 for this family, not with a code
fix but a species-count scope finding. A direct comparison against
method="nuts"'s exact joint posterior (Rhat 1.011, ESS 523, 0
divergences) showed the S=5 fixture used to diagnose #226 had a genuinely
non-Gaussian Laplace-EM posterior (Vf/Cinv 3-15x too narrow, point
estimates measurably off) -- not fixable by any Gaussian-correction
machinery (AGHQ debiasing, Vf/Cinv reprojection), both tried and both ruled
out as the cause in 0.0.223. At S=20 (a community size typical of real
ecological data), the SAME plain Laplace-EM -- no debiasing at all --
calibrates cleanly: 5 seeds, posterior min p_unif range 0.0017-0.032, 0
quantities below 1e-3 out of 81 possible across all 5 runs, no reproducible
failing coefficient. Registered with the fixture and test explicitly
pinned to S=20 (tests/testthat/test-sbc-registry.R, comments warn against
shrinking it for speed). ms_int_occu/ms_count (same
.tobs_community_em() engine, same original S=5-style failure signature)
are likely fixable the identical way but each needs its own species-count
check, not an assumption this transfers automatically.ms_occu(), found actively harmful, reverted
(gcol33/tulpaObs#226 part 2). Enabling the shared engine's AGHQ debias by
default (matching ms_occu_cover()'s own precedent) made posterior SBC
calibration WORSE, not better: seed 0 of a 5-species fixture went from a
passing baseline to min p_unif 4.6e-5 at n.sim=100, then collapsed to 5.8e-10
(6/21 quantities below 1e-3) at n.sim=400. Two hypotheses tested and BOTH
ruled out: (1) insufficient quadrature nodes -- Sigma was already converged
comparing n.quad=5 vs 9, identical SBC result; (2) Vf/Cinv/Bf
mutual inconsistency (Vf computed pre-debias, Cinv reprojected post-debias)
-- a real bug, fixed exactly (validated to 2.8e-14 against a direct
reconstruction) in both R/community_em.R's shared engine and
R/ms_occu_cover.R's own implementation, but fixing it left ms_occu's
failure essentially unchanged (seed 0 at n.sim=400: 1.7e-10 -> 5.8e-10,
same coefficient sp1_psi_x, same magnitude of failure). Root cause of
#226 part 2 remains genuinely unidentified for ms_occu -- ms_occu.R's
own AGHQ opt-in reverted to keep it OFF (.tobs_community_em() still
defaults re_aghq = FALSE, and ms_occu() no longer sets it), restoring
the pre-experiment, tested baseline (test-ms-occu.R 30/30). The Vf-
consistency fix is KEPT on both engines regardless (correct on its own
terms, validated, no regression on ms_occu_cover()'s existing 19/19 test
suite) even though it did not resolve ms_occu's issue..tobs_community_em()
(gcol33/tulpaObs#226 part 2 groundwork): ms_occu_cover()'s own
.ms_occu_cover_aghq_sigma() was already ~95% generic (it took the
per-species log-likelihood as a first-class sp_ll(s, theta, global)
callback, exactly .tobs_community_em()'s own convention) -- extracted
into R/community_em.R as .tobs_cem_aghq_sigma()/
.tobs_cem_reproject_cinv(), shared by every .tobs_community_em()
consumer (ms_occu, ms_int_occu, ms_count, ms_dyn_occu,
ms_distance), not just ms_occu_cover(). Wired into
.tobs_community_em() as new opt-in parameters re_aghq/n_quad/
re_aghq_maxdim, default re_aghq = FALSE -- every existing caller stays
byte-identical unless it explicitly opts in, so this commit alone changes
no family's default behavior. ms_occu_cover() itself refactored to call
the shared functions instead of its own copy (no-copy-paste); its test
suite unaffected (19/19 pass, unchanged). Opting individual families in
(starting with ms_occu, the actual #226 part-2 target) is a separate,
deliberately smaller follow-up step so each can be regression-checked on
its own.ms_occu_cover() registered for sbc() (gcol33/tulpaObs#220, the 20th
registered family): the occ+p+pos community adapter added in 0.0.220
cleared acceptance after a 9-seed check (seeds 0-8, not just the usual
3) -- every seed's posterior min p_unif comfortably above 1e-3 (range
0.0024-0.087), no coefficient repeating as the minimum more than twice out
of 17 possible across the 9 runs (chance level, not the reproducible-bug
signature #226 taught this design to watch for), and the mis-scaled
narrow control rejecting hard every time. An apparent decreasing trend
across the first 3 seeds (0.087 -> 0.012 -> 0.0044) turned out to be
small-sample noise -- seeds 4 and 7 bounced back up to 0.068 and 0.028,
and by seed 8 there was no trend left to see. bad.factor = 1.75 (the
package's established default for this test) separated cleanly throughout.ms_occu_cover() gains a CONTRACT-verified sbc() adapter
(gcol33/tulpaObs#220): spec/data/pool/draws/simulate/refit/loglik_many,
the occ+p+pos analogue of ms_occu's (reverted) adapter -- same joint
mu/b_s draw (#226 part 1) and safe to attempt because this family's Cinv
now stays consistent with Sigma under AGHQ debiasing (#226 part 2 fix,
0.0.219). draws() ranks S blocks of per-species realized coefficients
(psi_/p_/pos_, species-major) plus one trailing shared dispersion
coordinate (log_sigma_pos/log_phi, no per-species deviation).
loglik_many reuses .occu_cover_sp_ll, the exact per-species two-state
marginal .tobs_fit_ms_occu_cover() already optimizes. Verified against
the real sbc() dispatcher (not just the adapter functions in isolation):
refit-on-observed-data reproduces the fit exactly, pooling creates disjoint
groups, draws/simulate/loglik_many all compose and stay finite (8/8 checks).
Found and fixed a generalization gap in the shared test's .sbc_reg_means()
helper along the way: it assumed fit$means IS what a family's SBC design
ranks, true for single-species families but wrong for COMMUNITY ones (whose
fit$means is the P-length community mean, not the S x P per-species
theta) -- the same gap flagged earlier for ms_int_occu but never fixed
since that registration was reverted first. Acceptance (posterior SBC
uniformity) not yet run -- v1 scope non-spatial laplace, positive = "lognormal" only, matching occu_cover/cover's own v1 scope.ms_occu_cover()'s AGHQ variance debias never reprojected Cinv
(gcol33/tulpaObs#226 part 2, found while scoping that issue's remaining
gap). .ms_occu_cover_aghq_sigma() debiases the community covariance
Sigma, but Cov(b_s|y) depends on Sigma only through Sinv in
C_s = Htt_s + Sinv, and nothing re-solved that per-species block at the
debiased Sinv -- fit$ms_community$Cinv silently kept its pre-debias
(attenuated) value even on an AGHQ-debiased fit, contradicting the fit's
own reported Sigma. Fixed by reprojecting: Htt_s is recoverable
without re-deriving the Newton solve (solve(Cinv_old_s) - Sinv_old), then
Cinv_new_s = solve(Htt_s + Sinv_new) -- exact, no new approximation
(Bf_s, being pure likelihood curvature with no Sinv term, needs no
update). Round-trip and reprojection identities validated to machine
precision (~1e-15) on synthetic SPD inputs; on a real fit (P=4, AGHQ
engaged), Cinv now measurably differs from the AGHQ-off fit where before
it was byte-identical. test-ms-occu-cover.R unaffected (19/19 pass)..tobs_community_em()'s
Newton solve already carries a Bf_s cross-Hessian block between the
community mean and each species' deviation, discarded after every solve --
now returned and exposed on the fit (fit$ms_community$Bf, alongside the
existing Cinv) for ms_occu/ms_int_occu/ms_count/ms_dyn_occu/
ms_distance/ms_occu_cover. A new shared .tobs_sbc_community_b_draws()
draws b_s | mu ~ N(blup_s - Cinv_s %*% t(Bf_s) %*% (mu_draw - means), Cinv_s) -- derived by block-inverting the joint (mu, b_s) arrowhead
precision and validated to machine precision against a direct construction
of that matrix. Confirmed the diagnosis: the original failure was one
SPECIFIC coefficient pinned at p_unif ~1e-5 on every seed; after this fix,
10 seeds each fail (if at all) on a DIFFERENT coefficient at n.sim=100 --
the multiple-testing-noise signature replacing the reproducible-bug one.
Not sufficient alone, though: one of those 10 seeds does not regress
toward the pack at n.sim=400 (it gets WORSE -- min p_unif 5.4e-6 -> 8.2e-11,
spreading from 2 to 6 affected coefficients across three species), the
signature of a real remaining effect. Most likely the documented Laplace
small-cluster community-VARIANCE attenuation (ms_occu_cover() already has
an AGHQ debias step for exactly this; the plain .tobs_community_em()
consumers do not) compounding through Cinv_s. ms_occu/ms_int_occu/
ms_count stay OUT of .TOBS_SBC_REGISTRY pending that debias; Bf/Cinv
are kept regardless since they are correct and independently useful.sbc() refits now show progress/ETA by default. Every .tobs_sbc_spec_*
callback had its own copy of list(verbose = FALSE, progress = FALSE),
silencing every refit inside an sbc() run unless the caller explicitly
passed fit.control = list(progress = TRUE). Found the hard way: a
distsamp_open() 100-sim acceptance run (2 control arms x n.ref=200,
~130-170s/refit measured directly) ran 11+ hours with zero console output,
indistinguishable from a hang without separately probing CPU time. The 15
duplicated literals are now one .tobs_sbc_default_control()
(verbose = FALSE, progress = TRUE); fit.control = still overrides it,
so the automated test suite's fast fixtures are unaffected wherever they
set their own control, and any sbc() call as expensive as
distsamp_open()'s now reports where it is instead of going dark.ms_distance() gained a simulate() handler (#227): reads
ms_community$coef_lambda/coef_sigma (the deterministic per-species
matrices, matching ms_occu()/ms_count()'s own handlers) and draws a
replicate through cpp_simulate_distance -- the same kernel the
likelihood integrates against, as simulate_ms_distance()'s own
docstring already notes for the standalone generator. key = "halfnorm"
only for now; the hazard key's log-shape is a community global scalar
not carried by ms_community, left as a follow-up. Unblocks registering
ms_distance() for sbc() -- not attempted yet in this pass, and would
need the same multi-seed check every other community family in this
design has needed given #226's track record (3 for 3 so far).ms_count() is a THIRD family confirmed to share #226's bug.
CONTRACT-verified, but multi-seed (0, 1, 2) posterior SBC found
sp3_mu_(Intercept) pinned at p_unif ~9.6e-7-9.9e-7 every time (several
other coefficients also suspiciously low, e.g. sp3_mu_x ~5e-6) --
the same systematic Cov(mu, b_s) bias diagnosed for ms_occu and
confirmed for ms_int_occu, and WORSE here: ms_count has no detection
arm to dilute it, just the single mu arm. Not registered; the adapter
functions (R/sbc.R section 6n) are kept as CONTRACT-verified groundwork.
#226 updated to note all three affected families.occu_multiscale_cover() registered for sbc() (#220, multiarm-S3
group -- completes it, alongside cover()): the standard single-block fit
shape (unlike cover()'s two-block one), with the exchangeable unit the
CELL rather than the plot, so pooling/site-labeling tracks cell indices.
Checked at three configurations (n_cells=40 seeds 0/1, n_cells=120
seed 0) before registering -- all consistent, no anomaly like the
ms_occu/ms_int_occu near-misses (#226). Measured (n_cells=40, seed 0):
posterior min p_unif 0.093 (psi_(Intercept)), narrow max 5.5e-6.ms_int_occu() was live in .TOBS_SBC_REGISTRY with an unverified
acceptance test -- reverted, confirmed to share #226's bug. Found while
scoping the next community family: its registry entry and adapter (community
analogue of int_occu(), same "rank a fixed species set" design as
ms_occu) had already been written in an earlier pass, registered as
active, but its acceptance test still carried a bad.factor placeholder
comment -- it had never actually been run to completion, so sbc() on an
ms_int_occu fit was silently offering an unverified calibration check. A
single seed happened to cross the 1e-3 bar (p_unif 0.0029) and an earlier
pass concluded from that alone that it was fine, unlike ms_occu. Running
it at three different sbc() seeds (0, 1, 2) found the SAME coefficient
(sp3_p2_(Intercept)) pinned at p_unif ~0.0029-0.0030 every time --
reproducibility that rules out sampling noise (which would vary run to
run) and points at the same systematic bias #226 diagnosed for ms_occu
(the independent mu/b_s draw construction dropping Cov(mu, b_s)),
just proportionally smaller for this fixture. Reverted the registry entry
and acceptance test; the adapter functions (R/sbc.R section 6l) are kept
as CONTRACT-verified groundwork, matching ms_occu's treatment. #226
updated to note both families are now confirmed affected, not just one
suspected.cover() registered for sbc() (#220, multiarm-S3 group): a
tobs_multiarm_fit with the same two-independent-Laplace-Gaussian-block
shape as occu_categorical (presence, positive) -- .tobs_sbc_data_vector/
.tobs_sbc_pool_vector (renamed from the occu_categorical-only spelling,
now shared by both) handle the plain length-N vector response unchanged.
Two small additions to the fit object, both independently useful: raw
encoding$data/encoding$y (needed to rebuild a refit call) and the full
per-arm coefficient covariance V_occ/V_pos (previously only the
diagonal se_occ/se_pos was stored -- drawing a joint sample of an arm's
coefficients needs the off-diagonal terms, a lesson directly from the
ms_occu near-miss in #226). X_occ/X_pos are always rebuilt fresh via
model.matrix() from the arm formula against natural-scale data, never
read off the stored autoscaled encoding$occ_data$X/pos_data$X -- mixing
those with the already-unscaled beta_occ/beta_pos would reproduce the
exact #225 bug class. v1 scope: non-spatial laplace,
positive = "lognormal" only (simulate_cover()'s own generator only
covers lognormal/gaussian); sigma_pos is held fixed rather than scored,
since no SE is reported for it anywhere in the package. Checked at two
sample sizes (N=200, N=600) before registering, given the ms_occu lesson
about not trusting a single passing run -- both consistent, no anomaly.
Verified to the #207 bar: 100-sim posterior SBC uniform on all four
coefficients (min p_unif 0.124 at bad.factor = 1.75), a mis-scaled
control rejects hard (max 1.8e-8).t_occu() registered for sbc() (#220, closes the multi-season/
multi-source group): a pg_gibbs family whose fit$draws is already the
real pooled cross-chain posterior sample (.tobs_pg_finalize_fit()), so no
Gibbs-aware draws() was needed. It has no simulate() handler at all
(simulate_t_occu() draws a fresh truth, not a replicate at a given theta)
and no .tobs_pointwise_loglik dispatch, so both were written: simulate
draws a fresh AR1 year-effect sequence at the theta's own (sigma, rho);
loglik_many is a Laplace approximation to the AR1 year effect's marginal
(the effect is not part of theta and has to be integrated out), using the
same Louis (1982) mixture identity already established elsewhere in this
codebase for an additive-logit-offset 2-state mixture. FD-validated
(gradient and Hessian match finite differences to ~1e-8) and cross-checked
against a brute-force dense-grid integral at T = 2 (agrees to grid
resolution, ~0.02 nats, after the validation script's own missing
normalizing-constant bug was found and fixed -- the package function was
correct throughout). Verified to the #207 bar: 100-sim posterior SBC
uniform on all five coefficients (min p_unif 0.081 at bad.factor = 1.75),
a mis-scaled control rejects hard (max 1.2e-4).ms_occu()'s per-species posterior covariance is now exposed
(fit$ms_community$Cinv, R/ms_occu.R): .tobs_community_em() already
computed Cov(b_s|y) per species internally (a Louis 1982 block from its
own Newton solve) but never surfaced it. Independently useful for anyone
wanting a per-species coefficient's own uncertainty beyond the point BLUP.ms_occu()'s sbc() registration attempt found a real calibration gap,
filed as #226, not registered. Following the community-group design
(rank a fixed species set's own realized coefficients mu + b_s), the
adapter draws mu ~ N(means, vcov) and each species' b_s ~ N(blup_s, Cinv_s) independently. CONTRACT-verified (refit reproduces exactly,
simulate()/pooling shapes correct via the family's own validated
cpp_simulate_ms_occu kernel, loglik_many finite), but posterior SBC
is not uniform: min p_unif ~1e-5 on one species' occupancy intercept at
N=80/5 species, and worse (more coefficients affected) at N=300 -- the
opposite of what Laplace-EM small-sample attenuation predicts, ruling out
that documented caveat. Leading hypothesis: Cinv_s is Cov(b_s|y)
conditional on mu at its mode, so independent draws implicitly assume
Cov(mu, b_s) = 0, which is almost certainly wrong in a hierarchical model
where only mu + b_s is informed by data. The adapter functions
(R/sbc.R section 6j) are kept as CONTRACT-verified groundwork, commented
as not wired into the registry pending #226 -- likely affects
ms_dyn_occu/ms_int_occu too, sharing the same community EM engine.dyn_int_occu() registered for sbc() (#220): the product of the
multi-season and multi-source shapes -- model$y is a named list of S
per-source 3D [n_sites x max_visits_s x T] arrays sharing the site axis.
Neither the named-matrices pool (int_occu()/gdistremoval()) nor the
3D-season pool (dyn_occu()/dyn_abun()/distsamp_open()) fits alone, so
a new .tobs_sbc_pool_named_3d composes both: site-axis stack within each
source's own 3D array. simulate() wraps the family's own
.tobs_simulate_dyn_int_occu() handler, which already reproduces each
source's own NA/missingness pattern, so partial season overlap needs no
special-casing. fit$means/fit$draws are the standard shape and
.tobs_pointwise_loglik already dispatches for this model_type, so
draws/loglik_many are the shared generic ones. Verified to the #207
bar: 100-sim posterior SBC uniform on all five coefficients (min p_unif
0.033 at bad.factor = 1.75), a mis-scaled control rejects hard (max
2.5e-5).rbind() on two zero-column data frames silently collapses to 0 rows,
0 columns -- base R drops the row count when there is no column to align
on. Every SBC pool helper built its refit data via
rbind(obs$cells, rep$cells), so an intercept-only (~ 1) fixed-effects
fit on ANY already-registered family -- not just the one that surfaced
it -- would fail its refit with a bogus "y has N sites but data has 0
rows" the moment someone ran sbc() on it. Fixed with one shared
.tobs_sbc_rbind_cells() guard, applied at all five call sites
(.tobs_sbc_pool, .tobs_sbc_pool_3d_season, .tobs_sbc_pool_named_matrices,
.tobs_sbc_pool_occu_categorical, the new .tobs_sbc_pool_named_3d).dyn_abun()'s sbc() acceptance-tier bad.factor = 1.75 (written
pending measurement in 0.0.207) is confirmed: its 100-sim posterior SBC
run finished uniform on all five coefficients (min p_unif 0.087), a
mis-scaled control rejects hard (max 1.6e-7). No retuning needed.occu_multi() registered for sbc() (#220, multi-response group):
model$y is a list of S per-species detection matrices sharing one site
axis, the same shape int_occu() already pools (.tobs_sbc_pool_named_matrices
reused unchanged). simulate() is custom -- unlike int_occu(), species
are not independently observed given the shared latent state: the joint
occupancy state is one draw from the log-linear multi-species model
(first + second-order natural parameters), then each species is
independently detected given its own realized state. Wraps the family's
own .tobs_simulate_occu_multi() handler, matching how gdistremoval()'s
adapter reuses its own simulate(). fit$means/fit$draws are the
standard .tobs_bfgs_marginal_fit() shape and .tobs_pointwise_loglik
already dispatches for this model_type, so draws/loglik_many are the
shared generic ones. Verified to the #207 bar: 100-sim posterior SBC
uniform on all eight coefficients (min p_unif 0.019 at bad.factor = 1.75),
a mis-scaled control rejects hard (max 4.3e-6).distsamp_open()'s sbc() adapter is written and CONTRACT-verified
(refit reproduces the observed fit; simulate()/pooling shapes correct);
constant-dynamics, Poisson only for v1, sharing dyn_abun's 3D response
and site-axis pooling. Its acceptance-tier bad.factor is still a
placeholder pending its 100-sim measurement (a long-running fit, same as
dyn_abun's still-pending acceptance run) -- to be finalized in a
follow-up commit.test-sbc-registry.R) generalized
past a third implicit assumption: the observed-slice-equals-pooled-input
check compared y including dimnames, which a family's own stored
response may carry (site/bin/season labels) but a freshly pooled array
never does (distsamp_open surfaced this; dyn_abun/dyn_occu never
carried dimnames to expose it). Both sides are now compared by value only.occu_categorical() registered for sbc() (#220, multiarm-S3 group):
a tobs_multiarm_fit with no fit$means/fit$draws -- the presence and
class arms are two independent Laplace-Gaussian blocks fit as separate
GLMs, not a joint MVN pseudo-draw matrix. draws() samples the two blocks
independently and stacks them into one theta row; loglik_many() scores
the two-arm likelihood directly off the encoding, since this family has no
.tobs_pointwise_loglik dispatch to reuse. The response is also a new
shape for the registry: a plain length-N vector (one nominal-class
observation per unit, no visit/season axis), which the generic
cross-family CONTRACT test's y-shape branching did not previously
handle. Verified to the #207 bar: 100-sim posterior SBC uniform on all
eight coefficients (min p_unif 0.057 at bad.factor = 1.75), a
mis-scaled control rejects hard (max 2.8e-5).test-sbc-registry.R, "each registered
family composes its callbacks end to end") generalized past its two
implicit assumptions -- every fit has fit$means, every pooled response
has nrow/dim -- via a .sbc_reg_means() accessor and a third y-shape
branch for a plain vector response, so future families with either shape
are covered by the existing test rather than needing a bespoke one.control$batch.backend = "fused") could converge one species to a
different mode than its own independent fit; root-caused and fixed
upstream in gcol33/tulpa#397 (the batched joint driver's dense path lacked
the PD escalation the sparse path and the single-species driver already
had). The previously-skip()-blocked bit-identity gate in
test-occu-cover-batch.R (gcol33/tulpaObs#224) is un-skipped and passing.gdistremoval() registered for sbc() (#220, multi-response group):
its response is two matrices (yDist band counts, yRem period counts)
that must stay row-consistent -- pooled together on the site axis via the
same list-of-matrices pool int_occu() already uses (now shared as
.tobs_sbc_pool_named_matrices()). Its own simulate() handler serves as
the replicate generator directly. Verified to the #207 bar: 100-sim
posterior SBC uniform on all six coefficients (min p_unif 0.449 at
bad.factor = 2.0), a mis-scaled control rejects hard (2.0e-9).int_occu()'s detection-arm intercepts and SEs were systematically
wrong (breaking: reported values change; #225, root-caused from the
SBC calibration failure filed there). Two compounding bugs, neither an
SBC adapter issue:
int_occu()'s per-source detection design matrix lost its column
names when padded to full-site width (R/occu.R), so the covariate
autoscaler's unscale step could not find the intercept column by name.
It correctly unscaled the detection slope back to natural units but
silently left the detection intercept in standardized-covariate units,
on every fit with an autoscaled detection covariate.model_type == "integrated" never received the exact-marginal Newton
debiasing that single-season occu() and dyn_occu() already have
(the EM's pseudo-binomial M-step leaves a small discretisation
residual their refine corrects); a new .tobs_integrated_marginal_refine()
(R/int_occu_marginal.R) closes that gap the same way.int_occu() with a single source now matches occu() on
identical data to full optimizer precision (all four coefficients and
both SDs bit-identical, previously off by ~0.06 on the detection
intercept even at full EM convergence with priors disabled). The
int_occu() SBC acceptance test (blocked in 0.0.203) now passes
cleanly: posterior min p_unif 0.171 (was failing at 8e-6).int_occu() registered for sbc() (#220, multi-source group,
full-overlap only): its response is a list of one detection matrix per
source, a third response shape needing its own spec/data/pool/
simulate/refit set, pooling on the site axis and leaving the
per-source axis alone. The CONTRACT tier (refit reproduces the observed
fit exactly, pooling creates disjoint groups) passes cleanly, confirming
the adapter itself is correct. The acceptance tier does not: the
per-source detection intercepts fail posterior SBC's uniformity check,
and get WORSE at larger N rather than better -- not the small-sample-noise
signature #219 found for royle_nichols(), and not reproduced by a naive
bias check (which shows a comparable or larger discrepancy on plain
occu(), whose own SBC is known to pass, so that check isn't diagnostic
here). Filed as #225 rather than shipped as a false pass; the affected
test is skip()'d with a pointer, not deleted or asserted green.dyn_occu() registered for sbc() (#220, multi-season group,
constant-rate only): its response is 3D [n_sites x max_visits x n_seasons], so unlike every family registered so far it needed a bespoke
spec/data/pool/simulate/refit set -- pooling on the site axis
and leaving the season axis alone (the design the issue itself states),
with a hand-written HMM forward-simulator for the replicate generator.
draws and the log_lik rank statistic reuse the shared machinery
unchanged. Verified to the #207 bar: 100-sim posterior SBC uniform on all
five coefficients (min p_unif 0.090), a mis-scaled control rejects hard
(1.5e-4 at bad.factor = 1.75). Season-varying detection/colonization/
extinction fits are refused with a pointer, not silently mis-scored.occu_cover()'s fused batch backend not bit-identical to
independent fits): narrowed to a tulpa engine bug in the fused Newton
solve (tulpa_nl_joint_batch) that follows the DATA, not the batch slot
or the areal kind -- reproduces on icar too, not bym2-specific, and
the divergent species stays divergent regardless of which batch position
it occupies. Filed upstream as gcol33/tulpa#397 with a two-order
reproduction. control$batch.backend = "fused" is opt-in, not the
default; the default "looped" backend is unaffected and gained its own
dedicated regression test (previously only verified transitively via a
comparison against the now-known-unreliable fused output). The affected
assertions are skip()'d with a pointer to #397, not deleted or
weakened.int_occu()'s fitted()$z ignored the data, and predict() had no
out-of-sample detection prediction (#223, found while fixing #218):
fitted(fit)$z on an int_occu() fit reported the prior marginal
(psi) rather than a real posterior, for every fit. Now a proper
multi-source Bayes update: a site is occupied with certainty if ANY
source detected it, otherwise its posterior pools every source's
non-detection evidence.predict(fit, X.0 = , type = "detection" / "both") silently returned
only occupancy regardless of type, for occu() AND int_occu(). A
new X_det.0 argument (a plain matrix for occu(), a list of one
matrix per source for int_occu(), matching fitted()$p's per-source
shape) now drives real out-of-sample detection prediction; requesting
"detection"/"both" without it errors rather than answering with
occupancy. No design matrix at all still means in-sample (fitted()),
unchanged.sbc() on every family was broken (found while starting #220):
sbc.tobs_fit()'s internal call into the engine used
tulpa::sbc(experiment = "posterior", ...), but the currently pinned
tulpa (>= 0.0.196) names that parameter object; the stale name fell
through to ... unmatched, leaving object at its default
("prior_predictive") and erroring there on EVERY family, not just a
newly registered one. Every skip_if_fast()-gated SBC acceptance test was
silently never exercising this path in the fast tier, so it went
undetected. Fixed to tulpa::sbc(object = "posterior", ...).double_observer() registered for sbc() (#220, partial): the
independent-observer protocol needed no new adapter (both p1/p2 read
one detection formula slot, same shape as every other simple registry
entry); the dependent protocol's primary assignment threads through as
extra. Verified to the same bar as #207: 100-sim posterior SBC uniform
on every coefficient (min p_unif 0.057), a deliberately mis-scaled control
rejects hard (1.3e-8). The remaining groups #220 asks for (multi-season,
community, multi-response shapes, multiarm S3) are not attempted here --
the community group in particular needs a modelling decision (rank
community means/covariance with fresh species draws, vs. a fixed species
set's coefficients) that is the user's call, not a default to pick
silently.pit_residuals() on single-season occu() was degenerate at 1 for every
detected site (breaking: reported values and test_uniformity() results
change) (#222). cpp_single_pit treated the site-level detected/all-zero
event's CDF as pinned at 1 for a detected site and added a 1/n_draws
jitter instead of the randomized-PIT interpolation, so test_uniformity()
rejected on correctly specified models. Rewritten as cpp_single_pit_cdf
(deterministic CDF limits, no RNG) + tulpa::tulpa_pit(), the same
construction cover()/occu_cover() already used correctly. Verified: KS
test on the package's own diagnostics-vignette fixture moves from
rejecting (p < 2.2e-16, ~58% of PIT values piled at 1) to p = 0.28; a
20-seed calibration loop shows no seed rejecting at alpha = 0.05.occu_cover()'s joint nested_laplace path reported the areal field's
raw amplitude, not its scale (#221). sigma there is the amplitude
against the unscaled intrinsic ICAR precision Q = D - W, while the #204
NUTS path's field_sd states the geo-mean marginal SD (Sorbye-Rue) -- the
only field-scale summary comparable across backends and to
simulate_occu_cover()'s own sigma convention. The two differed by
sqrt(scale_q) (~2.1 for a 30-node chain graph), so comparing a fit
against a simulator, or the two backends against each other, by reading
sigma was silently off by that factor. The joint path now also reports
field_sd (fit$spatial$field_sd_mean / field_sd_sd, and
field_sd_trend_mean / _sd for a coupled trend field) in the NUTS
convention. Kept out of fit$means/fit$vcov deliberately: field_sd is
a fixed multiple of sigma, not an independent grid axis, so folding it
into the sampled parameter vector makes the joint vcov exactly singular.spde() field on occu()/int_occu() was fit and
reported (fit$spatial_field_det) but not applied in fitted()/predict()
(#218). Two sites with the same detection covariates and different
detection fields got identical fitted p. fitted.tobs_fit() now adds the
projected field to the detection linear predictor, mirroring how the
state-arm field already worked; predict() with no newdata already
delegates to fitted(), so it inherits the fix. Fixing this exposed that
int_occu()'s fitted()/predict() used only the FIRST detection
source's coefficients for every source (silently, pre-existing) --
fitted(fit)$p on an int_occu() fit is now a named list, one detection
probability vector per source (matching y's names), each with its own
field when present. fitted()$z on an integrated fit still reports the
prior marginal rather than a real posterior, and out-of-sample
predict(newdata=) still has no detection prediction for occu()/
int_occu() at all -- both pre-existing, tracked separately (#223).royle_nichols()'s SBC joint log_lik arm rejects
while every coefficient marginal passes. No code defect -- the
optim(hessian=TRUE) Gaussian pseudo-posterior is a fair approximation to
each 1-D coefficient marginal (profile-likelihood CIs 11-23% narrower than
Wald at N=100, narrowing further at N=1000) but cannot represent the curved
ridge the nonlinear p_site = 1-(1-r)^N link induces between lambda and
r; the joint SBC statistic is doing exactly what it exists to catch. Not
something a tolerance change should paper over. Details in
NOTES_measurements.md.occu_cover()'s fused batch backend
(control$batch.backend = "fused") is not bit-identical to independent
per-species fits on a bym2() shared field -- the fused outer grid carries
one fewer cell. Pre-existing, unrelated to #221 (reproduces with that
change reverted); tracked separately (#224).Every diagnostic is now one verb, dispatched on the fit (breaking). The
package exported a parallel tobs_* spelling for diagnostics that already had
an owner elsewhere, so the same concept carried two names and neither could
dispatch. Each is now an S3 method on the generic of whichever package owns
the concept, and the generic is re-exported here, so attaching tulpaObs is
enough to reach it:
| was | now | generic from |
| --- | --- | --- |
| tobs_waic(fit) | waic(fit) | loo |
| -- | loo(fit) | loo |
| tobs_cpo(fit) | cpo(fit) | tulpa |
| tobs_dic(fit) | dic(fit) | tulpa |
| tobs_sbc(fit) | sbc(fit) | tulpa |
| tobs_pit_residuals(fit) | pit_residuals(fit) | tulpa |
| tobs_test_uniformity(pit) | test_uniformity(pit) | tulpa |
| tobs_test_dispersion(fit) | test_dispersion(fit) | tulpa |
| tobs_test_outliers(fit) | test_outliers(fit) | tulpa |
| tobs_test_zero_inflation(fit) | test_zero_inflation(fit) | tulpa |
| tobs_ppc(fit) | ppc(fit) | tulpaObs |
| tobs_check(fit) | check_model(fit) | tulpa |
The old names are removed rather than aliased. Results are unchanged: each
method is the previous function body, and test_uniformity() runs the same
Kolmogorov-Smirnov test on the same PIT vector.
loo(fit) is new, and returns a genuine psis_loo object, so
[loo::loo_compare()] and the model-weight machinery read a tobs_fit
directly. It builds the same pointwise log-likelihood matrix the other
criteria use and hands it to PSIS with relative effective sample sizes off the
chain layout -- the builder tobs_stack() already used per member.
tobs_check() is now check_model(fit), and it draws the panel its
generic promises. [tulpa::check_model()] errored on a tobs_fit before
this: its default method builds the panel from fitted() and residuals()
as numeric vectors, and a latent-state fit has neither (fitted() returns
list(psi, p, z)), so the response could not be resolved. The method reads
the same quantities through the family's own doors instead -- pit_residuals(),
ppc(), test_dispersion(), moran_i() -- printing the roll-up report as
before and drawing a QQ-uniform, PPC, dispersion and (given coords)
correlogram panel beside it. plot = FALSE prints the report alone. Nothing
is simulated twice: the panel plots what the report already computed, which is
why test_dispersion() now also returns the simulated variances in $sim.
ppc() deliberately did not fold into tulpa::pp_check(). That generic
draws a graphical check while ppc() returns a Bayesian p-value, and a method
has to honour its generic's contract, so the discrepancy check became its own
generic.
loo moves from Suggests to Imports (it now owns two of the doors), and
requires tulpa >= 0.0.196 for the generics.
The information criteria score a random effect the fit carried
(gcol33/tulpaObs#215). On the grid-integrated (nested_laplace) route,
occu_cover() scored every observation-arm random effect at the population
mean in WAIC, LOO, CPO, PPC and PIT, and an occupancy-arm one likewise. Both
now reach the criteria through the same per-visit and per-site offset path the
sampled route uses. This moves existing numbers: elpd_waic -17.939 ->
-6.406 on a detection-arm RE at sigma_re_p 1.10 over 216 visits, -125.94 ->
-64.41 on a cover-arm RE over 400 visits, and -209.525 -> -200.180 on an
occupancy-arm RE at sigma_re 1.57. A fit carrying no random effect is
identical to the bit. The occupancy term also carries its grouping variable
and factor levels now, so ranef() labels it like the observation arms and
predict(newdata = ) matches on it.
A bare areal term on occu_cover() loads on occurrence only under nuts,
as it already did under nested_laplace (gcol33/tulpaObs#217). The same
input previously fit two different models: the deterministic backend decoupled
the cover arm while the sampler estimated a copy amplitude off the default
axis (mean 0.78 on a 36-cell fixture). A term's process is the formula it sits
in, and copy() is what couples it to the other arm. This changes fitted
values for existing no-copy spatial NUTS fits. Declaring copy(spatial())
reproduces the previous numbers to every printed digit on all three coupled
recovery fixtures.
occu_cover() samples a second (spatially-varying-coefficient) areal field
under nuts (gcol33/tulpaObs#214). The sampler carried one field block, so a
trend surface was confined to the grid-integrated path. Each block now carries
its own basis, site-to-node map, per-site design weight and its own sampled
(sigma, rho, alpha). Reported as fit$trend_field / trend_fields, with
per-block suffixed hyperparameters in hyper_draws. Both surfaces recover with
zero divergences.
A per-source detection formula can carry a structured term
(gcol33/tulpaObs#216). An integrated detection-arm spde() field was wired in
the fitter and unreachable from tobs(). Per-source field correlation with
truth 0.87-0.93; the fields are reported in fit$spatial_field_det. Gating
this uncovered a silent misplacement: under method = "nested_laplace" a
detection-arm term was fit against the occupancy arm, on the single-season path
as well as the integrated one. Both now error with a pointer.
Simulation-based calibration runs on eight more families
(gcol33/tulpaObs#207): occu, count, abun, royle_nichols, occu_ttd,
fp_occu, removal and distance, each verified end to end against a
deliberately mis-scaled control. A structured term and a visit-level
observation design are refused rather than approximated.
The SBC replicate field is drawn at the width the joint engine fits
(gcol33/tulpaObs#213). The generator normalised its ICAR draw to geo-mean
marginal SD 1 while the joint nested-Laplace engine carries the amplitude
against the raw Q = D - W, so both reported field SDs were off by one common
factor while their ratio stayed clean. Ranks return inside the band. Note that
the joint path reports sigma as a raw amplitude while the sampled path
reports a geo-mean marginal SD; compare a fit against a simulator on the field,
not on the reported scalar.
An areal graph with more than one connected component says so at fit time
(gcol33/tulpaObs#212). icar(), bym2(), car(), car_proper() and the
spatial() umbrella now report the component count and the component sizes
when the adjacency is disconnected. One shared check (.tobs_check_graph())
covers every areal entry point; the traversal is a few lines of base R, no new
dependency. A single-node component shows up in those sizes; the areal cover
paths already reject an isolated node with a more specific error naming it, and
that guard is unchanged.
The CONSTRAINT itself was already right and is unchanged. An intrinsic field
carries one constant null direction per connected component, and the engine
has augmented the precision with one sum-to-zero per component since tulpa
eb7e532 (shipped in 0.0.195), reading component MEMBERSHIP from the
adjacency rather than from a contiguous equal-split assumption. Measured
against the engine's own log_prior_icar on a disconnected graph: the pooled
log-prior equals the sum of the independent per-component log-priors exactly;
raising one component's level and lowering another's, which leaves the GLOBAL
sum at zero, is charged tau * (n1 + n2) * c^2 / 2 where a single global
constraint would charge nothing; and the log-prior is invariant under a
permutation of the node ordering, which is what pins membership rather than
only the count. What #212 read as a missing constraint was the reporting
convention: fit$spatial_field is demeaned globally per field block
(.occu_cover_demean_fields()), which makes two equal-size components' means
exactly equal and opposite whatever the prior did. The RAW field's component
means are not (they sum to -4.5e-04 on the fixture that motivated the issue).
A single-component graph, the overwhelmingly common case, is silent and
bit-identical: identical() holds on the coefficient means, SDs, vcov, field,
log-marginal, grid modes, weights and grid of a connected-graph occu_cover
fit run with and without the check. test-graph-components.R.
Simulation-based calibration runs on a real observation-process family
(gcol33/tulpaObs#207, consumer of gcol33/tulpa#380). New exported verb
tobs_sbc() adapts a fitted tobs_fit into the callback contract
tulpa::sbc()'s POSTERIOR experiment reads (Talts et al. 2018; Sailynoja,
Schmitt, Buerkner and Vehtari 2026, Algorithm 2), so calibration is measured
on the coupled occu_cover a deliverable ships rather than on a toy fixture.
The truth is drawn from the fit's own posterior at the observed data, a
replicate is simulated at that truth, the model is refitted on both data sets
together, and the truth's rank under that augmented posterior is scored
against tulpa's exact simultaneous band.
Three premises are carried rather than assumed. The replicate is drawn on
FRESH cells whose areal graph is block-diagonal against the observed one --
occu_cover couples its two arms through one shared field and theta carries
no per-cell value, so re-observing the observed cells would break the
factorization -- and group_ids() is supplied, so tulpa::sbc() VERIFIES the
observable half of that instead of taking it on trust. The per-cell field is
not scored; the arm fixed effects, the field SD on each arm, the copy scale
alpha and the dispersion are. alpha and the cover-arm field SD are read
per draw off the outer grid through .tobs_joint_draws(), whose cells are
sampled by their own normalized weight, so what is ranked is each one's
grid-marginalized posterior rather than a ratio of component modes.
Measured on a 50-cell, 6-visit chain fixture at n.sim = 100 (204.6 s): the
six arm coefficients and alpha are uniform, smallest p-value 0.133, against
3.3e-16 for the same quantities on a deliberately mis-scaled control arm --
alpha alone reads 0.52 as reported and 3.3e-16 once its width is distorted.
Two reads leave the band on the correct fit and are reported rather than
tuned away: the occurrence-arm field SD puts 81 of 100 ranks in the top decile
and the derived cover-arm SD 87 of 100. The mechanism is the fresh cells --
the pooled graph has two components and the fit applies ONE global
sum-to-zero rather than one per component (measured: the two blocks' field
means come back exactly equal and opposite, and the pooled precision under a
single constraint is singular at reciprocal condition 5.6e-18), so the
between-block level has no prior and the shared SD absorbs it. alpha, a
ratio of the two SDs, is untouched, and so are the slopes. Numbers in
NOTES_measurements.md; the constraint itself is gcol33/tulpaObs#212.
A family is registered by one entry in the internal registry -- a replicate
generator, a refit call, a draw reader, optionally a joint statistic. The
pooling, the grouping labels, the arm construction, the mis-scaled controls
and the fixed-quantity guard are written once and shared. occu_cover is the
registered and verified family; a fit whose family has no entry errors naming
the roster rather than producing a result nothing checked.
control$phi.grid.pos is what makes the cover dispersion a scored
quantity. Left off it the joint engine fixes the dispersion per data set, so
the observed fit and each refit hold it at different values; tobs_sbc()
warns, names the consequence, and drops it from the scored set rather than
ranking a truth against a point mass.
.occu_cover_eta_from_par() takes per-arm offsets (off_occ, off_p,
off_pos), and .occu_cover_draw_icar_field() is the one unit-scale ICAR
draw simulate_occu_cover() and the SBC replicate generator share. Both
default to the previous behaviour exactly, and the extracted field draw keeps
the inline code's arithmetic as well as its RNG order -- checked
BYTE-IDENTICAL at 20, 47 and 100 nodes (max|diff| = 0), because writing
z * (1 / sqrt(lambda)) instead of z / sqrt(lambda) moves the draw by one
ULP and that does reach simulate_occu_cover(seed = )'s output.
Engine pin raised to tulpa 0.0.195 (Imports floor and Remotes tag
together, as .github/scripts/check-engine-pin.R requires). sbc() was
exported at tulpa commit 5d6c0d3, first tagged in v0.0.194, and the pooling
premise it reports was fixed in f7ac31d; r-universe serves 0.0.195, which is
also what the check demands be installed.
Five cover-hurdle fixtures re-anchored on the axis the engine integrates
(gcol33/tulpaObs#196, #197, #198, #199, #200). Each carried a number -- a
tolerance, a coverage floor, a recovery band, a node count -- stated against
control$sigma.pos.grid, the retired axis gcol33/tulpaObs#192 established
nothing had ever read. Every one of them is now measured on the grid the fit
runs today, and every rewritten assertion is recorded with the perturbation
that breaks it.
test-sla-cover-joint.R's vanishing-sigma test (#196) justified a 1e-1
skewness tolerance by "a 3-point (sigma, sigma_pos) grid at scale ~0.02" --
the literal nodes of the removed pin. Measured, that fit ran a 138-cell grid
whose copy axis reached 16.4 after refinement. It now pins sigma.grid,
alpha.grid and adaptive.grid = FALSE, so the tensor is exactly 63 cells
and the largest cover-arm amplitude any cell reaches is 0.03, with the donor
and copy truths both inside their axis and on no node. The skewness tolerance
is gone: measured over ten seeds it does not separate the vanishing regime
from sigma = 0.5, alpha = 1 (0.128 against 0.127), so it was never evidence
of the collapse. What replaces it is the cover-arm posterior itself, in units
of the separate path's own standard error -- 0.2024 at vanishing amplitude
against a smallest 0.6594 in the control, so the assertion holds on all ten
seeds of one regime and fails on all ten of the other. No translation of the
old nodes was attempted; c(0, 0.01, 0.02) in sigma_pos is not one alpha
grid across a three-node donor axis, which is why #192 refused to translate.
The same file's large-N band (#196) moves from 0.2 to 0.02: at N = 1000 the
measured maxima over seeds 104-108 are 0.00002-0.00200 (occurrence) and
0.00199-0.01292 (cover). The same fixture at N = 120 measures 0.128 on the
cover arm.
test-cover-hurdle-nested-joint-recovery.R (#197) stated a phi axis of "5
log-spaced points in [2, 300]"; the beta arm's default has 7 and, per
git log -S, never had 5. Both of its pinned axes also sat with the truth on
the MIDDLE node, unrecorded. Every fit now pins the truth off-node on both
axes and the file records what the centred placement measured: 0.0384 against
0.0346 mean relative error on sigma_pos, identical 0.933 slope coverage,
0.0298 against 0.0246 on phi_pos -- centring moved nothing here, and the
file says so instead of leaving it to be inferred. The bands tighten onto the
re-measured numbers (0.15/0.25 -> 0.08/0.20 on sigma_pos, 0.12/0.10 ->
0.05/0.05 on the slopes, 0.10 -> 0.06 on phi_pos), and phi_pos_sd's
span-derived < 100 becomes the measured < 8 (2.1975 at n_pos 316, 11.65 at
n_pos 42).
test-cover-hurdle-bpos0-coverage.R's 0.80 floor (#198) was measured with
sigma and rho each on the middle node of a symmetric three-node axis. All
three axes now hold their truth off-node, and the file records both the
geometry and the two numbers the floor sits between: 0.95 (19/20) as it
stands, 0.55 with the demean stripped out of the simulator -- the regime the
test exists to catch.
Five bands in test-cover-hurdle-multi-block.R (#199) asserted that a
posterior-weighted mean lies inside its own pinned node span, which it does by
construction; two of them were drawn exactly at the pin. They are replaced by
one paired fit -- one grid, two simulated truths -- and one ordering per block,
so the spatial SD, the copy coefficient, the AR1 precision, the observer RE SD
and the beta precision each have to follow their own truth. Every ordering
holds on five seeds, and each fails when its own component of the second truth
is put back to the first's. The test also asserts every block axis the driver
placed is the one asked for, and pins alpha.grid rather than inheriting it.
The outer-tensor comment in R/cover_hurdle_joint.R named sigma_occ and
sigma_pos as axes of the grid (#200). The tensor is
sigma x [rho] x alpha x phi_pos; sigma_occ / sigma_pos are C++ kernel
columns the backend materializes from (sigma, alpha). A second comment in
the same file named them as what the multi-block copy carries.
vignettes/cover-hurdle-vs-inla.Rmd knits again (gcol33/tulpaObs#201). Its
joint chunk passed control$sigma.pos.grid, which the cover() front door
refuses outright, so the vignette -- and with it R CMD check -- stopped
there. It carries the copy axis on control$alpha.grid and reads
theta_mean["sigma"], the name the outer grid actually uses.
The same vignette's joint fixture put its cover-arm intercept at 0.4
(gcol33/tulpaObs#202), so the lognormal median sat at exp(0.4) = 1.49 and
104 of its 140 positive responses were censored at the cover ceiling of 1
before the fit saw them; its cover-arm slope came back at 30% of truth. The
intercept moves to -1.5, which leaves 7 censored, and the recovery table now
reads 0.654 / -0.472 against truths 0.7 / -0.5 with alpha at 1.204 and
sigma at 0.653.
The adaptive-grid fixture places its truth back on the axis the engine
integrates (gcol33/tulpaObs#194). test-cover-hurdle-adaptive-grid.R was
written around a truth sitting at the top node of control$sigma.pos.grid,
an axis of the retired (sigma_occ, sigma_pos) parameterization that no
cover() route read (gcol33/tulpaObs#192). Its four tests therefore integrated
the package default copy axis with alpha_true = 1.5 between the 1.282 and
3.0 nodes -- inside the grid, not at its edge -- so they passed on a
configuration they were not designed to discriminate. Every fit now pins
control$alpha.grid = c(0, 0.5, 1.0, 1.5), the axis the engine grids the
copy coefficient on, restoring the INLAabun Demo 3 placement the adaptive
path was built against.
The coverage gap the file used to assert does not open at that placement, and
the file no longer claims it does. Measured over 20 seeds on tulpa 0.0.163,
fixed and adaptive both cover 20/20: with the truth exactly ON the top node
the upper side of the fixed arm's quantile interval cannot miss, since
gcol33/tulpa#353 gives a grid's outer cell its own half-width of support and
before that the quantile clamped at the node. What separates the two reads is
the upper edge itself -- the fixed arm reports the axis's own geometry
(mean 1.733, SD 0.011, range 1.689-1.738), the adaptive arm follows the data
(mean 2.270, SD 0.373, range 1.825-3.226), and is higher on 20/20 seeds. The
tests assert that separation, plus that the boundary trigger fires on the copy
axis, plus non-regression on coverage. NOTES_measurements.md carries the
numbers. Each of the four was shown to fail against a perturbation of the
behaviour it checks.
The cover-arm coupling axis reaches the engine on the field the engine
reads (gcol33/tulpaObs#192, #195). control$sigma.pos.grid named the
cover-arm field amplitude when the joint engine parameterized the copy as
(sigma_occ, sigma_pos). It parameterizes it as (sigma, alpha) with
sigma_pos = alpha * sigma, and has done since that axis was retired, so
nothing has read the knob on any cover() route: .cover_build_multi_prior()
wrote it to copy$sigma_pos_grid, a field tulpa's .resolve_one_copy_spec()
does not look at, and the single-block and coupled-trend routes never took the
value at all. Measured on simulate_cover_multi_block(N = 400, seed = 7001),
c(0.4, 0.8, 1.2) against c(11, 12, 13) gave a bit-identical
log_marginal and beta_pos. control$alpha.grid was unread on the
multi-block route too, so that route had no user control over the copy axis
whatsoever. It is now carried on the copy spec's alpha_grid, matching the
single-block and coupled-trend routes, and control$sigma.pos.grid is
refused with a message naming control$alpha.grid. NO FITTED NUMBER MOVES
for a caller who did not pin alpha.grid on a multi-block fit: the package
default .tobs_default_alpha_grid() is node-for-node the engine default the
route was falling back to. A caller who DID pin either knob was getting the
default axis and now gets the one they asked for.
The prior_from_spec() tau_grid -> sigma_grid translation in
fit_cover_hurdle_joint_nested() is removed (gcol33/tulpaObs#193). It was
guarded on a field tulpa::prior_from_spec() does not return: every areal
branch of the engine's spec converter builds a fresh list of type,
spatial_idx, n_spatial_units, adj_row_ptr, adj_col_idx,
n_neighbors (plus scale_factor / rho_bounds) and no grid field, and has
since the earliest version in that repo -- so the branch was dead when it was
written rather than dead from a regression (gcol33/tulpa#363 tracks the
roxygen claim it was written against). control$tau.grid is a separate,
live knob and is untouched.
Every outer-grid axis this package chooses now reaches the engine declared
as a default (gcol33/tulpaObs#190, #191). The engine decides axis
PROVENANCE, not field presence: an axis marked with tulpa::auto_grid() (or
one whose nodes are exactly the engine's own default) may be recentred onto
the hyperparameter mode, and anything else is a deliberate pin it never
moves (gcol33/tulpa#293, #297). Nine sites chose a default and wrote it out
through as.numeric() / sort() / 1 / sigma^2, each of which drops the
marker, so the engine read a tulpaObs default as a user pin. Now marked, on
the vector actually written into the block: the cover() multi-block AR1
(tau_grid, rho_grid), IID-temporal (sigma_grid), RW1 / RW2
(tau_grid) and per-group RE (sigma_grid) blocks; the multi-block copy
axis (sigma_pos_grid), whose mark .cover_fit_joint() applied and
.cover_build_multi_prior() then coerced away one call later; the
standalone occu() joint path's per-block tau_grid; the occu_cover() NUTS
warm fit's rho_car_grid; and the spec-supplied tau_grid the cover()
single-block path translates to a field SD.
Measured against tulpa 0.0.160 on the cover() multi-block, cover()
single-block, occu() SVC and occu_cover() NUTS warm paths, defaults and
pins alike: every fit's coefficients, hyperparameter means and outer-grid
placement are unchanged to every digit reported. No rescue reads these axes
yet -- the single-block joint rescue reads a sigma_grid the cover()
single-block path leaves absent (and therefore already recentre-eligible),
and the multi-block rescue is scoped to a copy block's sigma_grid, which
.cover_build_multi_prior() already marked. What the marks buy is that the
axes stay recentre-eligible as that scope widens, and that
gcol33/tulpa#352's unread-axis check keeps dropping them silently instead of
refusing the fit.
Grid defaults written in more than one place are single-sourced in
R/joint_substrate.R alongside the existing .tobs_default_*_grid()
helpers: .tobs_default_rho_car_grid(), .tobs_default_temporal_tau_grid()
(AR1 and RW1 / RW2 shared one literal), .tobs_default_temporal_rho_grid(),
.tobs_default_temporal_sigma_grid(), .tobs_default_re_sigma_grid() and
.tobs_default_occu_joint_sigma_grid(). Each returns its vector already
declared, and .tobs_num_auto() is the one spelling of "coerce to numeric,
carry the source's own provenance" the eight re-marking sites shared.
tulpa (>= 0.0.136), which makes a joint nested-Laplace fit
return the same numbers twice. test-occu-cover-parallel-coupling.R
asserts that two identical parallel occu_cover() fits are bit-identical,
and it failed once under a loaded parallel suite while passing 200+ paired
fits in isolation. The cause was upstream: the coupled-cell scatter's chunk
count, which sets its summation order, was derived from a count of solves in
flight, and the outer width was clamped against a live free-memory reading,
so both moved with whatever else the machine was doing (gcol33/tulpa#300).
Measured from here: forcing the chunked path moves means by 1.4e-15 and
sds by 1.19e-07, and the fixture's default path was bit-identical to the
forced serial scatter -- so the chunk count demonstrably changes the answer,
and whether it changed was race-decided. Both inputs are now read from the
problem rather than the machine, with no parallelism given up. Nothing in
this package changed; the engine's exported headers are unchanged from
0.0.135, so no rebuild is needed.tulpa (>= 0.0.135). The engine's exported headers
(inst/include/tulpa/) are unchanged from 0.0.132, so this package's
compiled backend links against the same ABI and needs no rebuild; the
bump carries the engine's own nested-Laplace and diagnostics work
(gcol33/tulpa#294 through #299), including the per-block declaration of
required fields checked at dispatch. The Imports floor, the Remotes
tag, and the installed engine are checked against each other by
.github/scripts/check-engine-pin.R on every CI job, so the three
cannot drift apart silently.One answer per control knob (#188). engine_defaults.R was read only
by the 14 community dispatch branches, while 29 fitters restated the same
sampler knobs in their own formals. For everything routed through
.tobs_fit_model() -- occu(), dyn_occu(), int_occu(), abun(),
removal(), distance(), fp_occu(), dyn_abun(), count() -- those
formals were dead: that entry forwards explicit values, so
abun(method = "nuts") ran 2050 iterations and kept 2000 rather than its
own documented n.iter = 1000. Every sampler knob is now a NULL formal
filled by .tobs_fill_sampler() as the fitter's first statement, making
the table the only answer, and test-engine-defaults.R asserts
structurally that no fitter carries a literal.
Two divergences were deleted as proven artifacts: n.iter 2000 -> 1000 on
the occu / cover / occu_cover NUTS paths (commit 8975470 changed
n.iter from a total to a kept-post-warmup count on exactly those paths
and left the literal behind, so a default that had always kept 1000
silently began keeping 2000), and occu(method = "pg_gibbs") 2000/1000 ->
the shared 3000/1500, which had been keeping 1000 draws where every ms_*
sibling kept 1500. Three were recorded rather than aligned, as
.TOBS_SINGLE_SPECIES_NUTS / _LAPLACE: the .tobs_fit_model() entry
keeps sigma.beta = 10, adapt.delta = 0.8 and seed = 42, each the
value its recovery and coverage tests were calibrated against. n.iter
means opposite things under the two samplers -- a kept count under nuts
(the run is n.iter + n.warmup), a total under pg_gibbs (warmup taken
out of it) -- which is now stated in the table and pinned by a test that
recomputes the kept count from the profile.
sd.load and re.lkj join the table; n.quad deliberately does not
(#189). sd.load (1.0) and re.lkj (1.5) are laplace rows. n.quad is
not one number: .TOBS_NQUAD_ROUTES / .tobs_n_quad() enumerate the seven
marginals that name it, from 1 on the closed-form lognormal cover latent to
15 on its beta counterpart, and ?tobs now lists them per route instead of
stating a single default most routes never use.
Defaulted outer grids declare themselves (#186). The engine's
auto-recentering rescue decides axis provenance, not field presence
(gcol33/tulpa#293): it moves an axis that is absent, auto_grid()-marked,
or exactly equal to its own default, and treats anything else as a user
pin. This package writes a grid on every joint fit, so an undeclared
default read as a pin and the rescue went inert. The
.tobs_default_sigma_grid() / _alpha_grid() / _bym2_rho_grid() helpers
now return tulpa::auto_grid(...), and .tobs_mark_auto() re-applies the
marker wherever a site reshapes a defaulted grid, since sort(), [,
c(), as.numeric() and expand.grid() all drop the attribute. Verified
end to end: a defaulted axis reports declined = "grid_not_collapsed", a
control$sigma.grid one "axis_pinned".
Outer-grid placement is visible in glance() (#187).
.tobs_promote_outer_grid() lifts outer_grid_placement,
_recenter_attempts, _prior_added and _recenter_declined to the
tobs_fit top level and into glance(), so an inert recenter shows up in
a batch summary rather than only under a manual $joint_fit inspection.
It is not gated on the grid having moved -- a declined recenter is exactly
the case worth seeing. glance.tobs_multiarm_fit() gets the columns too,
since it is terminal for cover_fit and never reaches
glance.tobs_fit().
Pinned to tulpa (>= 0.0.131) so the auto-recentering outer grid also
engages at this package's own defaults (gcol33/tulpa#291, #292). The
0.0.130 rescue that 0.0.183 pinned for read pareto_k_mode_u / cov_u,
fields only the full outer Pareto-k diagnostic populates -- and
control$diagnose.k defaults to FALSE on every route reaching the joint
driver -- occu_cover(), occu_multiscale_cover(), and occu()'s
SVC-bar joint path (#101) -- so the driver's rescue never fired on a
default fit. It now computes the same (mode, covariance) itself
whenever the grid has collapsed onto an edge, independent of the
diagnostic. A collapsed default occu_cover() fit therefore re-centers and
refits where before it stayed railed.
The companion upstream change adds the same rescue to fit_st_nested()'s
tau_spatial x tau_temporal [x rho] tensor grid, which reaches the
observation families carrying an areal + temporal field on one arm
(removal(), distance(), fp_occu(), dyn_abun()).
Both are behaviour-preserving where the grid was not collapsed, and both
decline whenever a grid-construction knob was set explicitly. The
Remotes: gcol33/[email protected] tag moves with the floor, as
check-engine-pin.R asserts: r-universe builds default-branch HEAD on its
own poll schedule and currently serves 0.0.130, so the tag is what resolves
the pinned engine.
tobs_cpo(), tobs_ppc() and tobs_pit_residuals() now work on a compact
(ragged) occu_cover() fit (#185). method = "nested_laplace" defaults
compact = TRUE, so the ragged carrier is the standard representation for the
joint engine, and a compact fit stores no padded [n_sites x max_visits]
y / valid grid. The pointwise-log-likelihood path already read the ragged
carrier, but the PPC and the PIT / LOO-PIT CDF limits still derived their
per-site detection summary as rowSums(model$y * model$valid), which errored
with 'x' must be an array of at least two dimensions. LOO / CPO / LOO-PIT and
the posterior predictive check were therefore unavailable for any real-scale
occu_cover() fit while tobs_waic() worked.
All three now read one .occu_cover_visit_view(): the one-row-per-valid-visit
structure a compact fit stores, and the flattening of a dense fit's padded grid
(.occu_cover_dense_ragged(), the site-major visit-ascending order the
pointwise kernel already used). n_valid and any_det are derived there, so
there is one definition of each. Because both layouts now feed one kernel per
diagnostic, the dense and compact builds of the same data agree to the bit:
measured 0.000e+00 across elpd_waic, elpd_loo, the LOO-PIT, the PIT
residuals, and the PPC fit.y / fit.y.rep / bayesian.p.
tobs_ppc() no longer returns NA when a detected visit has a missing
cover (#185). The likelihood gates the cover density on detected AND finite (missing-at-random cover), but the PPC kernel scored the NA into the
Freeman-Tukey / chi-squared sum, so fit.y came back NA for every draw and
bayesian.p with it. Verified directly against the pre-change kernel: one NA
cover at one detected visit took fit.y from 1.788 to NA on all draws.
The replicate is now drawn at every valid visit (so the RNG stream does not
depend on which visits happened to be detected) and scored only where a cover
was observed.
Internal: the three ragged occu_cover() kernels -- pointwise
log-likelihood, PPC, CDF limits -- assemble their per-draw predictors from one
shared Arms view (src/occu_cover_ragged.h) instead of each writing out the
column-major design / draw index arithmetic. The pointwise kernel is
byte-identical after the change (test-occu-cover-ploglik-cpp.R scores it
against its R oracle). The PPC kernel now takes the arm designs and the
coefficient draws rather than precomputed [n_sites x (S * max_visits)]
predictor blocks, so its memory is O(V) in the observations rather than
O(padded grid x draws).
Pinned to tulpa (>= 0.0.130) for the auto-recentering outer grid
(gcol33/tulpa#289, #290). Every nested-Laplace family built its outer
hyperparameter grid from a fixed default axis, so a fit whose field-SD
posterior mode sat above the top node collapsed every outer weight onto
that boundary node and reported it only as
pareto_k_regime = "collapsed_edge". The motivating case was a tulpaObs
one: on the 78 real EVA occu_cover() fits, 10 railed the 5.0 sigma
ceiling (measurement upstream, in gcol33/tulpa#289). The fixed grid is now
a starting axis rather than a ceiling -- a collapsed fit re-centers using
the mode-Hessian its own outer Pareto-k diagnostic already computes, and
refits.
It reaches this package through the joint driver occu_cover() and
cover() already fit against (single-block icar/bym2 backends and
multi-block copy blocks) and through the standalone .NL_REGISTRY path.
An explicit sigma_grid / tau_grid still wins; auto-recenter engages
only where the grid was left at its default, and is a no-op when the mode
already sits inside the old axis. car_proper (its rho axis is
unguessable) and MCAR are out of scope upstream, so the correlated
cover-field routes keep the old behaviour.
Restored the exact Remotes: gcol33/[email protected] tag pin.
be6b2de dropped the tag on the expectation that r-universe would serve
the floor. r-universe builds default-branch HEAD on its own poll schedule,
not tags or releases, and currently serves tulpa 0.0.125 -- so an Imports
floor above that resolves an engine below the floor and
check-engine-pin.R fails, which is the gcol33/tulpaObs#150 skew
reopening. The tag pin installs the pinned engine directly. Cost, as
before: the Imports floor and the Remotes tag now have to move
together, which check-engine-pin.R asserts.
Folds in the unreleased 0.0.182, which moved the floor to
tulpa (>= 0.0.129) for the outer-grid regime split
(gcol33/tulpa#276) that the cover reliability CSV reports.
Pinned to tulpa (>= 0.0.117) for the inner-Laplace skewness
diagnostic (gcol33/tulpa#272). pareto_k scores only the OUTER
hyperparameter-grid integration around a fixed inner Laplace, but it read
as a whole-fit verdict; the inner Gaussian approximation to the latent-field
conditional posterior was a separate, unscored layer. The motivating case
was a tulpaObs one: an occu_cover() batch flagged 42/78 species
"unreliable" on outer k-hat alone when their point estimates, governed by
the healthy inner layer, were fine. tulpa 0.0.117 adds gamma_3 (the
leading-order Edgeworth skewness estimate of Rue, Martino & Chopin 2009
Sec 3.2.3), computed with one extra deterministic Newton solve at the
fitted MAP grid cell.
It reaches a tulpaObs fit exactly where that fit routes through one of
tulpa's own single-arm nested-Laplace kernels, surfacing as inner_skew /
inner_skew_idx / inner_skew_dropped on the inner fit. Measured on this
release: occu() + icar() reports it (inner_skew -0.038 / 0.091,
inner_skew_dropped 0), as does count() + icar() (-0.057 / 0.049) --
both small, i.e. the inner Gaussian approximation is healthy there.
abun() + icar() does NOT report it, because that fit runs tulpaObs's own
in-tree nmix_laplace_icar() rather than a tulpa kernel -- which by the
same routing argument should also exclude the areal fits that go through
the in-tree count-spatial and areal-BFGS drivers, though only abun() was
measured here. Upstream, the diagnostic declines to
NA rather than a silently-wrong 0 for a coupled multi-process spec,
which is what occu_cover()'s joint fit is -- the coupled-arm cubic term
is tracked in gcol33/tulpa#273.
Default on (control$diagnose_skew); no tulpaObs-side change needed, and
the areal recovery fixture is unmoved (occu() + icar() seed 1: slope
0.596 against truth 0.5, field cor 0.965, versus that test's 8-seed
medians of 0.943 cor / 0.9189 field sd). tulpa 0.0.116 in between was a
stale-test fix with no engine change.
spOccupancy and withr declared in Suggests. Both were already
used under :: in the test suite -- spOccupancy::PGOcc as the external
reference implementation in test-occu-pg-gibbs.R, withr::local_options
in test-occu-cover-pareto-k.R -- without being declared, which
R CMD check reports as "unstated dependencies in 'tests'". This is a
declaration fix, not a new dependency: both are on CRAN, both stay in
Suggests, and the spOccupancy block is already gated by
skip_if_not_installed(). Restores the tarball check to
0 errors / 0 warnings; the one remaining NOTE is the deliberate
-Wa,-mbig-obj flag in src/Makevars.win.
Pinned to tulpa (>= 0.0.115) (gcol33/tulpa#270). The 0.0.114 pin
below (needed for #169's set_spatial_adjacency() fix) exposed a
separate engine bug: occu_cover()'s coupled ICAR joint Hessian dropped
10592-124160 nonzero contributions whenever the field met either a
correlated detection-arm random slope or its own detection-arm beta
under the rank-1 s2z fold path (fields above 256 units) --
HessianPatternGuard (new since 0.0.101) caught the pattern builder
missing a cross-arm entry for a latent block reached by only one side of
a coupled arm pair. Fixed in tulpa 0.0.115; no tulpaObs-side change
needed.
ICAR/BYM2 occupancy NUTS fits work again (bugfix, #169).
populate_spatial() (the shared ModelData builder cpp_occu_fit's
unified NUTS entry point uses) assigned n_spatial_units /
adj_row_ptr / adj_col_idx / n_neighbors directly for ICAR/BYM2,
leaving spatial_partition at its default (0 nodes) and
n_spatial_components at its default (1) -- the exact case tulpa's
ABI 39->40 set_spatial_adjacency() was added to prevent
(tulpa_compute_param_layout() now rejects a partition that doesn't
describe its adjacency instead of silently pinning nothing). Every
ICAR/BYM2 occupancy NUTS fit through this path errored at first use.
Now calls set_spatial_adjacency(). Requires tulpa (>= 0.0.114).
Distance-sampling marginal sums truncate per site instead of to a
shared ceiling (bugfix, #168). Each site's binned-distance
marginal-count sum now caps at K_hi = min(K_max, K_lo + headroom)
instead of always summing to the shared K_max, mirroring
nmix_kernel.h's per-site truncation. A verified-widening safety
layer ports nmix's approach: after fitting under a capped headroom,
the score at the fitted coefficients is compared under both the
capped and uncapped truncation, and disagreement widens the headroom
4x and re-fits, escalating to fully uncapped if needed. Wired into
distance_laplace(), .tobs_fit_distance_spatial(),
.tobs_fit_ms_distance(), and .tobs_fit_distance_nuts().
distance_laplace()'s quad_xptr parameter is now documented
(codoc fix; the parameter itself shipped in 0.0.179 for #165).
macOS R CMD check builds again (tulpa (>= 0.0.101),
Remotes: gcol33/[email protected]). tulpa/portable_math.h returns
std::pair while including only <cmath> and <limits>; libstdc++ supplies
<utility> transitively, Apple clang's libc++ does not, so
count_grouped_oracle.cpp failed with no template named 'pair' in namespace 'std' and the whole macOS job died before producing a check directory. The
header now includes <utility> itself -- a header-only fix with no behaviour
on any platform, so nothing in this package moves with it.
Worth recording how long it hid: R-CMD-check.yaml runs ubuntu alone on push
and adds windows and macOS only on the weekly cron, so a macOS-only compile
error is invisible to every push and surfaces once a week at most. The two
platforms that pass are the two that share libstdc++.
The cover-hurdle occurrence reduction no longer shifts the reported
log-marginal (bugfix). control$aggregate.occ (default ON) collapses the
occurrence arm to one Binomial row per group of exchangeable Bernoulli plots.
That leaves the gradient, the Hessian, the mode, the SEs and every posterior
weight pointwise unchanged, but the binomial log-density carries
lchoose(n, y), which is zero on a Bernoulli row and non-zero on the group
replacing it -- so fit$log_marginal came back on a different additive scale
depending on an internal performance switch (measured 74.7 and 72.4 nats on an
18-cell x 10-plot fixture). The constant is now subtracted, putting the
reported value back on the scale of the observed per-plot data. Estimates are
unaffected in either direction; only the reported marginal moves, and only
when the reduction runs. A model comparison between two fits made with the same
setting was already valid and is unchanged.
TULPAOBS_PROGRESS=0 silences the console progress bar process-wide. The
bar defaults on and is set per fit from control$progress, which tobs()
writes to the scoped tulpa.nl_progress option on every call -- so a caller
that cannot reach the individual fits could not turn it off at all. A test
suite is exactly that caller: the weekly recovery job's log is mostly
per-iteration bars from hundreds of fits, which buries the failure output the
log exists to show. Setting the environment variable (also false / no /
off) flips the default for the whole process. An explicit
control$progress still wins in both directions, and the progress.file
heartbeat is untouched -- it is the only liveness signal on a detached run,
so silencing the console must not take it along.
Two recovery blocks that had never run are now green, and one of them was
hiding a real bug. Both sit behind skip_if_fast() and skip_on_cran(), so
neither smoke nor R CMD check reaches them and the weekly full-recovery job
has never completed a run -- the cover-hurdle aggregation block was byte
identical to the commit that introduced it and had simply never executed. It
found the log-marginal shift above. The second, test-dyn-int-occu-recovery.R,
failed on the test runner rather than the model: the runner passes
package = "tulpaObs" so that oracle files can reach internals through :::,
which also puts the namespace on the symbol-lookup chain ahead of the search
path and shadows unmarked's S4 logLik() with the S3 generic. Both
unmarked calls are now namespace-qualified.
The cover trend-field test asserts the amplitude ratio, not the amplitude.
sigma and sigma_trend are ICAR conditional-scale hyperparameters, and the
map onto a field's marginal SD is an engine parameterisation: both moved by the
same factor (-33% and -32% over 12 seeds, sd 0.031) while the occurrence slope,
its coverage and alpha_trend stood still. Two independently simulated fields
with different truths shifting together is a convention change, not a recovery
regression. The test now asserts the convention-free estimand -- the ratio of
the two amplitudes, 1.129 against a simulated 1.143 -- and keeps an absolute
bracket only as a gross-regression guard, labelled as such.
Engine floor moved to tulpa (>= 0.0.100) (Remotes: gcol33/[email protected]),
which carries exact dispersion and closed-form outer curvature for
zero-inflated and hurdle mixtures. gcol33/tulpaMesh stays at @v0.1.3.
n.iter is the post-warmup sample count on every NUTS path (bugfix). The
observation-family and community samplers already ran n.iter + n.warmup
iterations and kept n.iter post-warmup draws, but the single-season
occupancy path (occu() / dyn_occu() / int_occu()) and the cover paths
(cover() / occu_cover()) instead treated n.iter as the total including
warmup, so they kept only n.iter - n.warmup draws -- and returned zero draws
(all-NaN summaries) when n.iter == n.warmup. All paths now agree:
n.iter = kept samples, total run per chain = n.iter + n.warmup. The ?tobs
control docs are corrected to match. Existing calls keep more draws (and run
n.warmup more iterations) than before on the occu/cover paths.NUTS calibration coverage and areal-field exposure from the read-only test-coverage audit (closes #139, #142; #137/#140 test gaps also closed).
The single-season occupancy NUTS path now exposes its areal field
(#142). occu() with an icar() / bym2() term under method = "nuts"
sampled the field but left the field parameters unnamed (param[k]) and
fit$spatial_field NULL, so it could only be smoke-tested. The engine
already exports the block offsets on its parameter layout; occu_fit.cpp now
emits them as spatial_layout and names the columns
(spatial_field[i] / spatial_theta[i] / log_tau_spatial /
log_sigma_spatial / logit_rho_spatial), and fit$spatial_field carries
the centred per-cell surface for icar / car_proper (the intrinsic level is
confounded with the intercept, so it is centred as the nested-Laplace summary
is). The bym2 field is the Riebler rho-mix of two blocks scaled by the graph
scale factor, so its columns are named but its reconstruction is left to the
named draws. Recovery-tested: icar field correlation ~0.81 across seeds
(test-occu-areal-nuts-recovery.R).
Observation-family, abundance, community, and single-season NUTS paths get
20-seed CI-coverage tests (#139). The family paths (distance / removal /
fp_occu / dyn_abun / abun) and community paths (ms_count /
ms_int_occu / ms_dyn_occu / ms_abun) had only single-seed recovery;
dyn_occu / int_occu NUTS had no recovery test at all. All now carry a
=20-seed 95% CI-coverage study on the calibrated interval the NUTS path exists to get right (
test-family-nuts-coverage.R). The single-seasoncpp_occu_fitpath reports no per-parameter SD, so those blocks score the credible interval off the draws.
Consistency, single-source, and WAIC-boundary fixes from the read-only audit (closes #141, #135, #133).
The Polya-Gamma Gibbs conjugate machinery is a single shared engine
(#135). The six PG-Gibbs fitters (occu(), t_occu(), ms_occu(),
ms_int_occu(), ms_dyn_occu(), ms_count() / jsdm()) re-inlined the same
conjugate coefficient draw, the community mean + Inverse-Gamma variance loop,
the hyperprior constants, and the post-sampling summary / fit assembly. These
now live once in R/pg_gibbs_shared.R (.tobs_pg_draw_beta,
.tobs_pg_community_update, .tobs_pg_hyper, .tobs_pg_summarize,
.tobs_pg_finalize_fit); each fitter keeps only its latent-state step and
design layout. Behaviour is byte-identical (verified: every family's fit is
bit-for-bit unchanged at a fixed seed).
The #116 observation families give the friendly method error (#141).
royle_nichols(), occu_ttd(), occu_multi(), double_observer(),
gdistremoval(), and distsamp_open() already sat in .tobs_family_methods,
so an unsupported method = was already rejected with a "Supported: laplace"
pointer; a regression test now pins that contract.
occu_cover() WAIC / LOO pointwise density matches the fit kernel at the
cover boundary (#133). The WAIC / LOO pointwise log-density
(src/occu_cover_ploglik.cpp) is now the fit kernel
src/occu_coupling_shared.h::pos_log_density itself, and the R positive-arm
density (.occu_cover_pos_logdens, via the new .tobs_log_safe) drops the eta
clamp and guards log(0). A detected visit with cover exactly 0 or 1 now
yields a finite pointwise density (the number the model was fit with) instead
of -Inf / NaN, and the density no longer clamps the predictor. The
three-level src/occu_ms_cover_ploglik.cpp gains the missing
isfinite(cover) guard at a detected plot, so a missing-at-random cover drops
out instead of poisoning the pointwise sum. Non-spatial fits are numerically
unchanged.
Three inline-variant scope notes graduated to working features.
distsamp_open() gains alternative population dynamics
(dynamics = "notrend" / "trend" / "autoreg" / "ricker" / "gompertz").
Following unmarked::distsampOpen(), the transition N_t | N_{t-1} can now be
the density-independent "constant" (default, Binomial(N_{t-1}, omega) + Poisson(gamma)), the stationary "notrend" (recruitment tied to
(1 - omega) * lambda, no free gamma arm), the exponential "trend"
(Poisson(N_{t-1} * gamma), no survival), the "autoreg" density-dependent
recruitment (Binomial(N_{t-1}, omega) + Poisson(gamma * N_{t-1})), or the
carrying-capacity forms "ricker" / "gompertz" (an estimated K reported as
K, a growth rate reported as r). "constant" / "notrend" fit with the
exact analytic gradient; the density-dependent forms use the same exact
forward-HMM marginal through a value-only kernel
(compute_dyn_abun_site_dyn() in src/dyn_abun_kernel.h, validated against a
brute-force reference to < 4e-15) with a numeric gradient. The abundance and
distance-scale arms recover on every dynamics; survival / recruitment / K / r
sit on the usual short-series ridge, and the density-regulated forms grow
toward the carrying capacity, so bound K_max (the forward is cubic in the
truncation) when abundance is high. simulate_distsamp_open() gains
dynamics; NUTS / areal / season-varying detection stay follow-ups.double_observer() gains the dependent (removal-style) protocol
(type = "dependent"). A primary observer records what it detects and a
secondary observer records only the primary's misses (two cell counts:
primary-detected and secondary-only). A single fixed primary observer gives
two cell means for three parameters (lambda, p1, p2) -- a ridge -- so the
two detections are not separately identified; observer role-swapping breaks it,
with observer 1 primary at some sites and observer 2 primary at others (a
per-site primary indicator in {1, 2}), giving four cell means that identify
all three. The marginal is again a product of independent thinned Poissons, fit
by optim BFGS with an observed-information vcov. A single-observer primary
warns (no role-swapping). simulate_double_observer(type = "dependent") draws
the two cells with an alternating primary; the independent three-cell protocol
is unchanged.count() gains a continuous NNGP Gaussian-process field via gp().
count(...) + gp(lon, lat, prior_range = c(r0, alpha)) on the abundance formula
routes to tulpa's single-block nngp nested-Laplace kernel: the GP marginal
variance and range are integrated on the kernel's own outer grid and the field
is Schur-folded out, so the fit reports grid-integrated fixed effects plus the
GP hyperparameter posterior (fit$gp_hyper, sigma2 / phi_gp). The per-cell
field is integrated out on this path (fit$spatial_field is NULL); use
spde() for a reconstructed continuous field map. Poisson / binomial only (a
negbin size / gaussian residual variance is not jointly identified against a
per-node field, gated with a pointer), as for the areal icar() / car_proper()
path; multiscale_gp() (two-scale) is not hosted by the nested-Laplace engine
and errors with a pointer to gp() / spde().distsamp_open() gains negative-binomial and zero-inflated initial
abundance (mixture = "negbin" / "zip" / "zinb"). The open-population
distance marginal already runs through the dyn_abun() forward-HMM kernel,
which carries use_nb / eta_logr, so the negative-binomial initial abundance
is threaded straight through with an analytic log_r score (FD-validated,
cor = 1, max|an - fd| = 6e-8). Zero inflation is a pure-R additive layer
over the composed per-site marginal (the abun() / dyn_abun() ZIP recipe): a
structural-zero site is never occupied across any primary period, so all its
band counts are zero and the marginal is
omega * 1{all zero} + (1 - omega) * L_open, with omega an intercept-only
structural-zero probability reported as zi_logit / zi_omega (distinct from
omega, the survival arm). The ZI structural-zero share recovers cleanly (a
single-seed probe recovered 0.30 against a truth of 0.30); the abundance and
distance-scale arms recover on every mixture. simulate_distsamp_open() gains
mixture / size / zi. Poisson fits are byte-identical. As with the base
family the survival / recruitment ridge (and the NB size) is weakly identified
at short series; the forward is cubic in the truncation K_max, so a high
abundance is slow. NUTS / areal / season-varying detection stay a follow-up.distsamp_open() WAIC / pointwise log-likelihood. The per-site
.dso_site_loglik() read ev$log_lik (the summed dataset scalar) instead of
ev$log_lik_site (the per-site vector), adding the whole-dataset HMM total to
each site's band term. WAIC / LOO / tobs_ploglik() are now per-site correct.New family distsamp_open() -- open-population distance sampling (#116, the
final item; closes #116). The unmarked distsampOpen model: a Dail-Madsen
open N-mixture (as [dyn_abun()]) observed by distance sampling at each primary
period -- the open-population counterpart of the single-season
gdistremoval(). N_1 ~ Poisson(lambda), N_t = Binomial(N_{t-1}, omega) + Poisson(gamma) (apparent survival omega, recruitment gamma); at each period
the detected birds are distance-sampled into bins. The band allocation is
conditional on the period total detected, so it factors out of the abundance
HMM (the gdistremoval() trick): the marginal reuses the validated dyn_abun
forward kernel with the detection probability set to the overall distance
detection pdist (eta_p = logit(pdist)), plus the per-period band
multinomials -- no new HMM kernel. Four site-level arms: log abundance
(formula), log distance scale (detection), logit survival (omega = ~ ...),
log recruitment (gamma = ~ ...). Maximised by optim BFGS with an ANALYTIC
gradient (one kernel call per evaluation -- the dyn_abun kernel returns
grad_eta_*, and the sigma block chains through pdist via a distance-integral
finite difference; FD-validated to 4e-7) and an observed-information vcov. The
K_max truncation defaults to the detection-corrected abundance scale
(max(period total) / pdist + headroom), since the forward recursion is cubic
in K_max. simulate_distsamp_open(), full S3 (fitted / predict /
residuals / WAIC), y = a [n_sites x n_bins x n_seasons] array of
distance-band counts per primary period. Recovery-tested: 20/20 seeds converge,
every parameter's 95% coverage >= 0.90 (pooled 0.933), max coefficient bias
~0.068 at n_seasons = 4 (test-distsamp-open.R). Scope for the first ship:
half-normal key, line / point transect, Poisson initial abundance, constant
Dail-Madsen dynamics, site-level arms; NB / ZIP, other dynamics, and
season-varying detection are follow-ups.
This closes #116 -- every enumerated model family (Royle-Nichols, ZIP/ZINB, continuous SPDE fields on the count families, generalized multinomial / double-observer, time-to-detection occupancy, community distance sampling, multi-species co-occurrence occupancy, joint distance+removal, and now open-population distance sampling) is shipped and recovery-tested.
gdistremoval() -- joint distance + removal sampling (#116, the
last open item). The unmarked gdistremoval model (Amundson et al. 2014): a
single-season point count where the detected birds are cross-classified by a
distance band (distance sampling) AND the removal period of first detection
(removal sampling). Site abundance N ~ Poisson(lambda); the total detected is
a binomial thinning of N, and Poisson is closed under binomial thinning, so
the marginal is closed-form -- ysum ~ Poisson(lambda * pdist * prem) with the
band and period counts as two conditional multinomials (the double_observer()
Poisson-multinomial pattern, here with a half-normal distance multinomial and a
depleting-removal multinomial pi_k = r(1-r)^{k-1}). Three site-level arms: log
abundance (formula), log distance scale (detection), logit per-period
removal capture (removal = ~ ...). Maximised by optim BFGS with an
observed-information vcov. The half-normal band integrals are closed-form for
both line and point transects (matching numeric integration to machine
precision). simulate_gdistremoval(), full S3 (fitted / predict /
residuals / WAIC), y = distance-band counts + y_rem = removal-period
counts (per-site totals must match). Recovery-tested: 20/20 seeds converge,
coefficients unbiased (max bias 0.011), pooled 95% coverage ~0.975
(test-gdistremoval.R). Scope for the first ship: half-normal key, line / point
transect, Poisson abundance, constant removal capture, availability fixed at 1
(a single primary period does not identify availability separately). Hazard-rate
key, NB / ZIP abundance, and a multi-period availability arm are follow-ups
(each closed-form under the same thinning). This closes #116.ms_occu_cover() NUTS
(#115, B7 DoD). The community-model NUTS Definition of Done asks for community
covariance recovery at nominal 95% coverage over >= 20 seeds, not only the
mean and the de-attenuation direction. Added a 20-seed test
(test-ms-occu-cover-nuts.R) that reconstructs, per fit, the posterior for the
six community SDs (sqrt(diag(Sigma_arm)) for the occ / p / pos arms, 2 coords
each, from the sampled log-Cholesky coordinates via .ms_ocs_chol_unpack) and
checks each central 95% interval against the simulated truth. Measured: pooled
coverage 0.908 over the six SD components x 20 seeds (every component >= 0.85),
0 divergences on every seed. This is the joint-cover analogue of the ms_occu
covariance-coverage sweep (0.0.167); the covariance sampling machinery
(.ms_ocs_* log-Cholesky pack / unpack + prior) is byte-shared across all four
community NUTS families, and the two direct measurements (a two-state
single-season marginal and the heavier three-arm joint-cover marginal) bracket
the family likelihoods. Closes the last #115 item -- both B2 (community
occupancy NUTS for ms_occu / ms_dyn_occu / ms_int_occu) and B7
(ms_occu_cover NUTS + dispersion-RE) are shipped and validated to the DoD.count() + spde(); #116).
A continuous-mesh Matern field via spde(lon, lat, ...) on the abundance
formula now fits under method = "nested_laplace", alongside the areal
icar / car_proper / bym2 fields. The latent lives on the mesh nodes
(fit$spatial_field, length n_mesh) and the barycentric projector
fit$spatial$tulpa_spec$A (n_sites x n_mesh) maps it onto the sites, so
fitted() / predict() add the projected per-site contribution
A %*% mesh_field. Three touch points: the count dispatch gate accepts
spde; .tobs_nested_attach_field_summary reconstructs the mesh field from
the grid-averaged block modes (the same icar-style summary, one field node per
mesh node); .count_spatial_field_offset A-projects the mesh field to sites.
The areal (icar/car_proper/bym2) path is byte-identical -- those fits carry no
projector (tulpa_spec$A is NULL), so the A-projection branch is skipped.
Poisson only (as the areal count field: a per-node field is not jointly
identified with a negbin size / gaussian residual variance under the
fixed-dispersion nested loop). Recovery-tested: the mesh field projects to the
sites with cor > 0.9 against the simulated truth and the abundance slope
recovers (test-count-spatial.R, gated skip_if_no_tulpamesh()).ms_occu() NUTS (#115).
The community-model NUTS Definition of Done asks for community-covariance
recovery, not only the mean. Added a 20-seed test that reconstructs, per fit,
the posterior for the community SDs (sqrt(diag(Sigma_arm)) from the sampled
log-Cholesky coordinates) and checks their 95% CIs against the simulated truth.
Measured coverage: sd_psi[1] 0.85, sd_psi[2] 0.95, sd_p[1] 0.90 (pooled
~0.90), at/above the 0.85 rubric floor -- the calibration the Laplace-EM
variance lower bound cannot provide (test-ms-occu-nuts.R).count(...) + bym2(graph)
under method = "nested_laplace" now fits and reconstructs its field, joining
the icar / car_proper areal support. The generic nested-Laplace field summary
(.tobs_nested_attach_field_summary) gained a bym2 branch: it reads the block's
sigma / rho grid axes and the (phi | theta) mode slices and reconstructs the
rho-mixed unit field z = sqrt(rho / scale) * phi + sqrt(1 - rho) * theta
(Riebler 2016), grid-averaged. Measured field correlation ~0.93 with the
simulated (structured) truth over three seeds; the abundance slope recovers
(test-count-spatial.R). The same summary path drives occu() etc., so a bym2
areal field on those families is now reconstructed (fit$spatial_field) rather
than dropped. icar / car_proper reconstruction is unchanged. spde() / gp() on
the count families remain follow-ups.ms_occu_cover() NUTS (#115 B7).
ms_occu_cover(..., method = "nuts", control = list(dispersion.re = TRUE))
promotes the shared community cover log-dispersion to a per-species random
effect log_disp_s = mu_ld + sigma_ld * z_ld_s -- a fourth 1-D community arm
(the ms_abun log_r_s analogue), sampled jointly with the coefficient arms.
It reuses the same non-centered machinery (the 1x1 log-Cholesky factor is
sigma_ld) and the extracted per-cell sweep, so the shared-dispersion path is
byte-identical. The C++ FullGradFn is byte-exact against the FD-validated R
oracle (lp diff 0); 0 divergences. fit$ms_dispersion carries
sigma_log_disp (the community dispersion SD) and log_disp_species (the
per-species posterior-mean log-dispersions). test-ms-occu-cover-nuts.R. (A
simulated per-species dispersion spread for a full sigma_ld recovery sweep is
a simulator follow-up; on shared-dispersion data sigma_ld is small, as
expected.)ms_occu_cover(),
#115 part B7). ms_occu_cover(..., method = "nuts") (non-spatial) now samples
the exact joint posterior -- the community means, the per-species
occupancy / detection / cover deviations, the three independent per-arm
community covariances (occ / p / pos), AND the shared community log-dispersion
-- over the per-(species, cell) two-state occu_cover marginal via an in-tree C++
FullGradFn (src/ms_occu_cover_nuts.cpp), warm-started at the Laplace-EM mode.
The joint-cover analogue of the ms_occu / ms_int_occu community NUTS targets
(per-arm non-centered blocks b = C z + a shared log-dispersion scalar); it
wraps the existing cover-density kernels (occu_coupling_shared.h,
pos_log_density / pos_grad_eta / pos_grad_logdisp) in the community loop
rather than reimplementing them. Non-centered, so 0 divergences, and it
removes the documented Laplace community-variance attenuation (the
fit$ms_community$var_attenuation caveat) -- the sampled per-arm community SD
de-attenuates toward truth where the Laplace estimate collapses at small
per-species n. The R oracle is FD-validated and the C++ port is byte-exact
against it (test-ms-occu-cover-nuts.R: R-oracle vs FD, C++ vs oracle,
b_from_z round-trip, community-mean recovery + de-attenuation). With this the
community-model NUTS family (#115 B2) is joined by the joint-cover route (B7).
Non-spatial lognormal / beta / gaussian; negbin / dispersion-RE remain
follow-ups. The shared spatial-factor NUTS route (a field on the occupancy arm)
is unchanged.ms_int_occu(), #115).
ms_int_occu(..., method = "nuts") now samples the exact joint posterior --
the community means, the per-species occupancy / per-source detection
deviations, and the D + 1 independent per-arm community covariances -- over the
multi-source two-state per-(species, site) marginal via an in-tree C++
FullGradFn (src/ms_int_occu_nuts.cpp), warm-started at the community
Laplace-EM mode. The multi-source generalisation of the ms_occu / ms_dyn_occu
NUTS targets (per-arm non-centered blocks b = C z, no shared globals), reusing
the shared .ms_ocs_ epilogue helpers (#128). Non-centered, so 0 divergences,
and it removes the Laplace-EM small-cluster attenuation of the community
variance (the sampled community SD de-attenuates vs the Laplace lower bound).
The R oracle .tobs_ms_int_occu_nuts_logpost is FD-validated and the C++ port
is byte-exact against it (test-ms-int-occu-nuts.R: R-oracle vs FD, C++ vs
oracle, b_from_z round-trip, community-mean recovery + de-attenuation).
Non-spatial only..ms_ocs_* helpers
(#128). The tail half of every in-tree community-NUTS fitter (ms_occu,
ms_dyn_occu, ms_abun non-spatial + spatial, ms_count) carried
near-byte-identical copies of the warm-start metric, the multi-chain harness,
and the posterior-summary code. These now live once in the .ms_ocs_
namespace: .ms_ocs_b_idx() (per-species z-block indices), .ms_ocs_pd()
(symmetrise + jitter to PD), .ms_ocs_fd_metric(grad_fn, theta) (FD-Hessian
inverse mass, taking the per-family gradient closure), .ms_ocs_sig_mean()
(community-covariance posterior mean), and .ms_ocs_run_chains(run_chain, n_chains) (the multi-chain runner). The five families call them instead of
private twins; net ~180 fewer duplicated lines. Behaviour-preserving: the
refactor only relocates identical code and factors the gradient closure out of
the metric, so the NUTS draws are byte-identical on a fixed seed (verified per
family) and every recovery test is unchanged. The per-family _b_from_z
reconstruction stays local (the scalar dispersion / Gaussian-variance arms make
it genuinely family-specific), and ms_count's log-phi prior likewise.nested_laplace areal path on distance, removal, fp_occu, and
dyn_abun was recovery-tested for icar but only smoke-tested (finite vcov,
non-null field) for bym2 / car_proper. bym2 is a distinct code path -- it
reconstructs the rho-mixed unit field z = sqrt(rho) * phi + sqrt(1 - rho) * theta, which the icar recovery never exercises. Each family now has a field
correlation + FE slope recovery block for bym2 (all four) and car_proper
(removal upgraded from smoke, fp_occu / dyn_abun newly covered; distance
proper-CAR is already recovered via the hazard-key variant). Measured field
correlations: count families recover strongly (distance ~0.92, removal ~0.89,
dyn_abun ~0.82), the weakly-identified occupancy fp_occu field lower (~0.37 --
one binary site per node) but well above chance. Reuses the existing icar
fixtures; only the term and the field reconstruction differ.p) arm RE on abun(), the cover (pos) arm RE on occu_cover(), and the
uncorrelated detection random slope (0 + x | g) on occu_cover() were
shipped with label/shape smoke tests only. Each now has a multi-seed recovery
block asserting the RE SD recovers (band, measured from the sibling paths) and
the per-group BLUPs correlate with the simulated offsets. To support the cover
arm, simulate_occu_cover() gained re_pos_groups / sigma_re_pos, which
inject a per-visit random intercept on the positive-cover linear predictor
(mirroring the existing re_det_groups on the detection arm) and return
truth$b_pos_re / truth$sigma_re_pos..cover_arm_to_slot,
.occu_cover_joint_coef_names, .occu_cover_ppc_cover, .se_from_hessian);
the .tobs_ppc_occu_cover() PPC computes its cover discrepancy inline, so the
extracted twin was orphaned. Kept the third-derivative helper .l3_poisson_log
for family symmetry with its exercised .l3_binomial_logit /
.l3_gaussian_identity siblings and added its unit test (#129).
test-sla-int-occu.R gained the skip_if_fast() / skip_on_cran() guards its
four integrated-occupancy SLA fits were missing, so the fast pass no longer runs
them (#132). The SPDE recovery suite's skip_if_not_installed("tulpaMesh")
gates became skip_if_no_tulpamesh(), which fails loudly under
TULPAOBS_REQUIRE_SPDE=1 instead of silently skipping the whole SPDE surface
when tulpaMesh is absent on a CI runner (#132). Also refreshed the stale
ms_dyn_occu method-coverage assertions to reflect its new NUTS route (0.0.158).ms_dyn_occu(), #115).
ms_dyn_occu(..., method = "nuts") now samples the exact joint posterior --
the community means, the per-species first-season / detection deviations, the
two independent per-arm community covariances, and the shared colonisation /
extinction transition globals -- over the per-(species, site) HMM-forward
marginal via an in-tree C++ FullGradFn (src/ms_dyn_occu_nuts.cpp,
R/ms_dyn_occu_nuts.R), warm-started at the community Laplace-EM mode with a
diagonal Laplace metric. Non-centered per-species blocks
b_{s,arm} = C_arm z_{s,arm} break the covariance/deviation funnel; the C++
gradient is byte-exact against the R oracle. Removing the Gaussian
approximation de-attenuates the community variance components the Laplace-EM
under-reports on the binary arms. Non-spatial only (a structured term still
routes to nested_laplace). test-ms-dyn-occu-nuts.R (R-oracle vs
finite-difference gradient, C++ byte-exact, community-mean recovery + 0
divergences, S3 methods, variance de-attenuation, spatial gate).ms_abun(mixture = "zip" / "zinb") now puts a weak Penalized-
Complexity prior on the per-species structural-zero random-effect SD
sigma_omega (new control$omega.sigma.prior = c(U, alpha), default
c(1, 0.05); set NULL to restore pure ML). sigma_omega is the softest
AGHQ direction and at few species can collapse to the boundary, flattening the
marginal Hessian and attenuating the recovered SD; the prior adds curvature
there without biasing an identified fit (measured: a collapse-prone seed's
sigma_omega recovers 0.07 -> 0.18 toward a 0.3 truth, identified seeds
unmoved). Consumes tulpa 0.0.85's tulpa_re_aghq(sigma_prior = ...).K_max is below the largest observed count. The
community N-mixture marginal sums the latent N only to K_max, so a
user-supplied K_max < max(y) made the per-(species,site) marginal
structurally -Inf and the joint optimum singular -- previously surfaced as an
opaque "singular marginal Hessian". It now fails early with a message pointing
at K_max. The default (max(y) + 100) never trips it.log_lik); the default zero-inflated n_quad
drops to 3 (from 5), making a default zinb fit tractable; and the scalar
nuisance blocks (the NB dispersion log_r, the structural-zero omega)
integrate at fewer quadrature nodes than the correlated abundance / detection
blocks via tulpa 0.0.85's per-block n_quad (control$n.quad.scalar, default
2). Measured on the test fixtures: NB ~24% faster, ZINB ~33% faster, with
lambda / r / omega unchanged.tulpa (>= 0.0.85) and pin Remotes: gcol33/[email protected].tulpa (>= 0.0.84) and pin Remotes: gcol33/[email protected] so an
install pulls the engine build in which a joint occu_cover fit with a
checkpoint and diagnose_k = TRUE no longer aborts with a fingerprint
mismatch from the Pareto-k diagnostic re-solve (tulpa#161).Zero-inflated counts for the community N-mixture ms_abun(mixture = "zip" / "zinb") (spAbundance-style structural absence; #116). Structural zeros are a
per-species random effect logit_omega_s ~ N(mu_omega, sigma_omega): a share
omega_s of a species' sites is never occupied. It mirrors the per-species NB
dispersion log_r_s design, integrated jointly with the abundance / detection
(and, under zinb, dispersion) coefficients by the community AGHQ oracle. The
per-site marginal wraps the Royle marginal in a structural-zero mixture in
NMixCommunityOracle -- an all-zero site is log(omega_s + (1 - omega_s) exp(L_i)), a detection site rules out N = 0 -- with the score / observed-
information / Fisher composed in closed form from the plain-marginal kernel
outputs (no marginal-kernel change). mu_omega joins the coefficient surface
as logit_omega; ms_zi carries the community-mean omega and the
per-species omega_s; ranef() gains the structural-zero deviation. A shared
field / latent() factor on a zip / zinb fit errors rather than silently
dropping the structural-zero share. simulate_ms_abun(mixture = "zip" / "zinb") generates the truth; 20-seed community-mean recovery holds at the
coverage floor. This completes the ZIP / ZINB axis across abun() /
ms_abun() / dyn_abun().
Fixed simulate() on a zero-inflated abun() fit dropping the structural
zeros: the posterior-predictive count draws ignored zi_omega, so a zip /
zinb fit's replicates had no excess zeros. The structural-zero Bernoulli is
now drawn per site (byte-identical stream when no zero-inflation).
Partial season overlap for dyn_int_occu() (spOccupancy tIntPGOcc; #122).
A detection source that does not observe a (site, season) marks it NA, so a
staggered survey where sources cover different seasons is expressed by
NA-padding each source to the common [n_sites x max_visits x T] grid: the
forward drops an absent source from that season's emission (nvalid = 0) and
marginalises a (site, season) unobserved by every source (e0 = e1 = 1).
simulate_dyn_int_occu(source_seasons = list(1:4, 3:6)) generates staggered
coverage; 20-seed recovery holds. Two boundary anchors pin the reduction to its
neighbours: one source (the others all NA) reproduces dyn_occu(), and one
season of data (T = 2, season 2 all NA) reproduces int_occu(), both to
~1e-3 -- the #122 definition-of-done anchors.
Fixed the non-spatial dyn_int_occu() variance-covariance sign: the observed
information was built from the Jacobian of the log-likelihood gradient (-I),
so solve() returned the negative vcov and every standard error was clamped to
zero, silently breaking confint() and interval coverage. It is now the
Jacobian of the negative-log-likelihood gradient (+I); SEs are proper and
95% Wald intervals cover at ~0.90 over 20 seeds. The spatial fitter was
unaffected.
royle_nichols() (unmarked occuRN with
observation covariates; #116). Detection was site-level only; passing visits
now lets logit r_ij carry visit-level covariates. The latent-N marginal
generalises from the (k_i, n_i) sufficient-statistic form to the full
per-visit product inside the same Poisson sum,
prod_j [1 - (1 - r_ij)^N]^{y_ij} [(1 - r_ij)^N]^{1 - y_ij}, which reduces
exactly to the site-level form when r_ij is constant (verified to machine
precision). fitted() / predict(type = "detection") / simulate() return the
per-visit [n_sites x max_visits] detection grid; the site-level path is
byte-identical. simulate_royle_nichols(beta_r_visit =) draws a visit-level
detection covariate and returns visits for tobs(..., visits =). Recovery
over 20 seeds: max rel-bias ~0.04, min 95% CI coverage ~0.95
(test-royle-nichols.R).dyn_int_occu() + a spatial(~ 1 + w || cell, graph) bar (spOccupancy
svcTIntPGOcc; #122). Alongside the shared intercept field (stIntPGOcc), a
covariate-weighted (spatially-varying-coefficient) field loads on the
first-season occupancy logit. The areal-BFGS nested-Laplace driver
(R/areal_bfgs.R) already accepts a list of field blocks and scatters the
per-site psi1 score w1 - psi1 to each; the weighted block is the ICAR field
with a w-weighted loading (eta += w * z[map], the field z reported
unweighted). .areal_field_car() gained an optional weight (byte-identical
when absent), and .tobs_areal_field_blocks() expands a bar into an intercept
block plus one weighted block per covariate. fit$spatial_field is the
intercept surface, fit$trend_field(s) the varying-coefficient surface(s);
both recover by correlation (intercept ~0.85, trend ~0.72;
test-dyn-int-occu-areal-recovery.R). simulate_dyn_int_occu(trend =). icar
only in v1.t_occu()
(spOccupancy tPGOcc; #124). Per-(site, season) Bernoulli occupancy with a
shared AR1 year effect on the occupancy logit and NO colonization / extinction
transition (that is dyn_occu()); use it for an occupancy trend over years
with a temporal random effect. Given the year effects the seasons factorise,
so the fit is an exact Polya-Gamma Gibbs sampler (method = "pg_gibbs", the
engine spOccupancy uses): draw the latent state, then the joint
(beta_occ, eta) as one Gaussian Markov random field update with the AR1
precision as the year-effect prior, then beta_p, then the AR1
hyperparameters (sigma^2 conjugate Inverse-Gamma, rho on a grid). The
year-effect surface (fit$temporal_field), the occupancy / detection
coefficients, and the innovation SD sigma recover cleanly (20-seed
test-t-occu.R, year-effect correlation ~0.97, Rhat ~1.00); the AR1
correlation rho is a weakly-identified parameter of a short time series that
recovers only as the number of seasons grows (an identifiability property of
the AR1, not the sampler -- given even the true year effects rho_hat climbs
from ~0.08 at 8 seasons to ~0.59 at 200, dev_notes/_probe_tpg_rho.R), so it
is reported but not asserted tightly. y is a 3D array
[n_sites x n_seasons x max_visits] or a list of per-season matrices;
simulate_t_occu(). v1 = site-level occupancy / detection covariates,
pg_gibbs only.ms_int_occu(method = "pg_gibbs") (#115, #126). The community integrated
extension of msPGOcc: one latent occupancy state per (species, site) observed by
D detection sources, with per-species occupancy and per-source detection
coefficients under Gaussian community hyperpriors. Conditional on the
Polya-Gamma auxiliaries each sweep samples, per species, the single latent z
(occupied if any source detects, else Bernoulli on the pooled
occupied-undetected mass across sources), the PG-conjugate beta_psi_s, and the
D per-source beta_pd_s (each at that species' occupied, covered sites), then
the conjugate community mean and Inverse-Gamma community variance per coordinate
per arm. This gives a calibrated community-variance posterior (sd_psi ~0.53 vs
a 0.5 truth) where the community Laplace-EM attenuates it; the community means
recover and split-Rhat ~ 1.0. ms_community matches the Laplace fit's layout
(Sigma_/sd_/coef_/blup_<arm>), so coef() / ranef() / fitted()
work unchanged. test-ms-int-occu-pg-gibbs.R. Completes the community-occupancy
Polya-Gamma trio (ms_occu / ms_dyn_occu / ms_int_occu).Shared areal field on multi-season integrated occupancy: dyn_int_occu() +
~ 1 + icar(graph = adj) under method = "nested_laplace" (the spOccupancy
stIntPGOcc model, #122). The field sits on the first-season occupancy (psi1)
arm; because psi1 only sets the initial mixing weight of each site's HMM, the
exact per-site field gradient is the Fisher-identity score w1 - psi1 (the
smoothed season-1 occupancy). Fit through the shared areal-BFGS nested-Laplace
driver (the same recipe as fp_occu / dyn_abun), one field unit per site; the
colonization / extinction / per-source detection arms carry fixed effects only.
The interior field recovers (cor ~0.8), transition rates recover.
simulate_dyn_int_occu(field =); test-dyn-int-occu-areal-recovery.R. icar()
only in v1 (bym2 / car_proper, a varying-coefficient bar, and NUTS are
follow-ups).
dyn_int_occu() now fits with analytic forward-backward gradients (the
Fisher-identity score of the multi-source colext HMM: w1 - psi1 on occupancy,
smoothed pairwise transition joints on colonization / extinction, occupancy-
weighted binomial score per detection source) instead of a finite-difference
BFGS with a numeric Hessian. Faster (a fit that took seconds of numeric
differencing now completes in well under a second) and the observed-information
vcov comes from the FD-Jacobian of the analytic gradient; the non-spatial fit is
otherwise unchanged (recovery regression tests pass).
dyn_occu() (#124): a detection covariate supplied
as a [n_sites x T] matrix column of data (one column per primary season)
drives per-season detection, logit p_it = X_it beta_p, fit with
detection = ~ det_cov. The E-step emission reads the season's detection
probability, the M-step encodes one detection row per (site, season) at that
season's covariate, and the exact season-varying HMM-forward marginal refine
calibrates the coefficients (recovered with ~95% coverage over 20 seeds). A
plain per-site detection covariate keeps the byte-identical constant-detection
path. Detection unrolling shares the period-agnostic
.tobs_period_arm_design() with the season-varying colonization / extinction
path (the #80 recipe); gated under method = "nuts" with a pointer.
simulate_dyn_occu(beta_det_season =); test-dyn-occ.R. fitted()$z is now
period-varying-aware (also corrects the smoothed state for season-varying
colonization / extinction, which the previous per-site indexing got wrong).Zero-inflated open N-mixture: dyn_abun(mixture = "zip" / "zinb") (#116). A
structural-zero site is never occupied across any season, so all its counts are
zero; the observed per-site marginal is the two-component mixture
omega * 1{all y = 0} + (1 - omega) * L_DailMadsen, the same additive
structural-zero layer the static abun() ZIP uses (nmix_zip.R) but over the
exact forward-HMM open-population marginal. Pure-R over the C++ per-site
marginal; the Poisson / negbin paths are byte-identical. Analytic-gradient BFGS
(the Dail-Madsen per-site eta gradients weighted by the structural-zero
posterior, plus the mixture score for the ZI logit), observed-information vcov
from the FD-Jacobian of the analytic gradient. The structural-zero share
recovers with nominal coverage over 20 seeds; the ZI logit is named zi_logit
(dyn_abun's omega_* is the SURVIVAL arm). simulate_dyn_abun(zi =) injects
structural zeros. Non-spatial laplace, intercept-only omega; a field / RE / NUTS
stay Poisson / negbin with a pointer. test-dyn-abun-zip.R. This completes the
ZIP / ZINB mixture axis across abun() and dyn_abun().
Spatially-varying coefficient on community dynamic occupancy:
ms_dyn_occu() + spatial(~ 1 + w || cell, graph) (the svcTMsPGOcc model,
#123). An intercept field plus a shared covariate-weighted field on the
first-season occupancy arm, both recovering (cor ~0.90 / ~0.89). No new code
was needed: the psi1 oracle already returns per-site/per-species score+curv, and
the K-field weighted-ICAR block-coordinate solve is the same machinery the
community count SVC (svcMsAbund) uses -- the weighted bar flows through the
existing dynamic-spatial dispatch unchanged. simulate_ms_dyn_occu(trend =)
adds the varying-coefficient surface; test-ms-dyn-occu.R. Completes the
spatial multi-season community occupancy line (stMsPGOcc + svcTMsPGOcc);
bym2() / car_proper() fields and a NUTS sampler remain follow-ups.
Spatial Polya-Gamma Gibbs for single-season occupancy: occu() + icar() under
method = "pg_gibbs" (#126; the spOccupancy spPGOcc engine). An intrinsic
areal (ICAR) field on the occupancy logit, logit(psi_i) = X_i beta + f_i,
f ~ ICAR(tau). Conditional on the Polya-Gamma auxiliaries the joint
(beta, f) update is a single Gaussian Markov random field draw (dense
(p + n) x (p + n) precision t(W) Omega W + blkdiag(B0inv, tau Q),
W = [X | I]); tau has a conjugate Gamma full conditional; the field is
centred to sum-to-zero each sweep with its level moved into the intercept (so
eta is preserved -- omitting that leaks a systematic shift). The field
recovers (cor ~ 0.74), the intercept and detection recover, split-Rhat
~ 1.00. As with any spatial occupancy model, a site-level occupancy covariate
spatially-confounds with the saturated one-node-per-site field, so the
recovery target is the field + intercept + detection. icar only in v1
(bym2 / car_proper are follow-ups). test-occu-pg-gibbs-spatial.R.
This completes the Polya-Gamma engine (#126): PGOcc + spPGOcc (occu),
msPGOcc (ms_occu), tMsPGOcc (ms_dyn_occu), and the community Bernoulli /
binomial GLMM (jsdm() / ms_count("binomial")).
jsdm() and
ms_count(response = "binomial") with method = "pg_gibbs" (#126). The
community logistic GLMM has no latent state and no detection sub-model -- y
is the observed k-of-n (n = 1 for jsdm's presence/absence) -- so this is the
simplest of the PG engines: per-species coefficients with Gaussian community
hyperpriors, each an exactly conjugate Gaussian update conditional on the
Polya-Gamma auxiliaries, plus the conjugate community mean and near-Jeffreys
Inverse-Gamma community variance. One fitter serves both front doors (jsdm =
bernoulli, ms_count("binomial") = k-of-n). Samples the exact community
posterior, so the community variance recovers where the Laplace-EM attenuates
(jsdm sd_mu 0.674 / 0.439 vs truth 0.7 / 0.5, vs Laplace 0.635 / 0.401);
community means + per-species coefficients recover, split-Rhat ~ 1.00. Only the
logistic responses are routed here -- Poisson / negbin / gaussian reject
pg_gibbs with a pointer. Non-spatial in v1 (the sfMsPGBinom / lfJSDM
spatial-factor PG variants are follow-ups). R/ms_count_pg_gibbs.R;
test-ms-count-pg-gibbs.R.ms_dyn_occu() with
method = "pg_gibbs" (#115, #126; the spOccupancy tMsPGOcc engine). Combines
the community PG machinery (msPGOcc) with a two-state HMM forward-filter
backward-sample (FFBS) latent-occupancy step: each sweep FFBS-samples the
per-species occupancy path z, then does the PG-augmented conjugate coefficient
updates -- per-species season-1 occupancy psi1 and detection p (with
Gaussian community hyperpriors), and the SHARED community-level colonization
gamma and extinction eps from the aggregated 0-origin / 1-origin
transitions across all species -- plus the conjugate community mean and
Inverse-Gamma community variance for the psi1 / p arms. Samples the exact
community posterior, so the community variance recovers where the Laplace-EM
attenuates. The shared gamma / eps recover tightly (informed by every species),
the community season-1 occupancy mean and SD recover, and split-Rhat ~ 1.00. v1:
constant transitions, site-level detection, no structured terms.
R/ms_dyn_occu_pg_gibbs.R; test-ms-dyn-occu-pg-gibbs.R.ms_occu() with method = "pg_gibbs"
(#115, #126; the spOccupancy msPGOcc engine). The hierarchical extension of
the single-species PGOcc engine: per-species occupancy / detection coefficients
with Gaussian community hyperpriors, each PG-augmented conjugate-Gaussian
update wrapped in conjugate community-mean and near-Jeffreys Inverse-Gamma
community-variance draws (a diagonal per-arm community covariance, as
spOccupancy uses). Conditional on the Polya-Gamma auxiliaries every coefficient
update is exactly conjugate, so this samples the exact community posterior --
and, unlike the community Laplace-EM (whose variance components carry a
documented small-cluster attenuation), it recovers the community VARIANCE:
across seeds the PG sd_psi / sd_p land on the true community SDs (e.g.
0.667 / 0.417 vs truth 0.6 / 0.4) where the Laplace-EM attenuates below them
(0.595 / 0.342). The SD is reported as the posterior median (robust to the
variance-component skew at moderate species counts). Community means and
per-species coefficients recover; split-Rhat ~ 1.00. v1: single-season
community occupancy, site-level detection, no structured terms (the shared-field
sfMsPGOcc is a follow-up). R/ms_occu_pg_gibbs.R; test-ms-occu-pg-gibbs.R.method = "pg_gibbs"
(#126, the spOccupancy PGOcc engine). A REAL Gibbs chain over the exact
occupancy posterior via Polya-Gamma data augmentation (Polson, Scott & Windle
2013) -- conditional on the PG auxiliaries, both the occupancy and detection
logistic coefficient updates are exactly conjugate Gaussian. This is distinct
from method = "laplace_gibbs", which is a stochastic-EM variance correction
(mode-finds + Rubin pooling, no PG augmentation, stationary distribution not
the posterior). Each sweep samples the latent occupancy z, then draws
omega ~ PG and the conjugate Gaussian coefficients on each arm, using tulpa's
tested Polson-Scott-Windle sampler (tulpa:::cpp_rpg). Reports split-Rhat and
bulk-ESS (a real MCMC). Validated: on a well-identified model the PG-Gibbs
posterior matches the Laplace observed-Fisher fit (means within 1 SE, SDs
within 20%), Rhat ~ 1.00 / ESS ~ 700-900, and 95% credible intervals cover at
the nominal rate over 15 seeds. v1: single-season occu(), site-level
detection, no structured terms (the PG-spatial extensions --
pg_binomial_{icar,bym2,rsr,gp} in tulpa -- are the documented follow-up).
R/occu_pg_gibbs.R; test-occu-pg-gibbs.R.dyn_int_occu() (#122, spOccupancy
tIntPGOcc). The product of the two shipped families -- a dynamic
(multi-season HMM) occupancy state (psi1, colonization gamma, extinction
eps) whose per-season emission pools SEVERAL detection sources (integrated
occupancy). Pooling sources across seasons is how colonization / extinction
estimates come out of individually-sparse opportunistic data. The latent
occupancy sequence integrates out by the two-state HMM forward recursion (the
dyn_occu exact-marginal forward generalised with a per-season emission pooled
over sources), maximised by optim with an observed-information vcov -- pure R,
no new C++. y is a list of S [sites x visits x seasons] detection arrays;
the state formula models logit psi1, colonization = ~ / extinction = ~
the transitions (required, as in dyn_occu()), detection the shared
per-source design (each source carries its own coefficients). Full fitted /
predict / residuals / simulate / WAIC surface; simulate_dyn_int_occu().
Recovers psi1 / gamma / eps and per-source detection over 12 seeds. v1:
full site/season overlap, constant transitions, site-level detection,
non-spatial laplace; partial overlap, season-varying rates, an areal psi1
field (stIntPGOcc), a weighted bar (svcTIntPGOcc), and NUTS are documented
follow-ups. test-dyn-int-occu.R.double_observer() (#116, \pkg{unmarked}
multinomPois with the independent double-observer pi-function). Site
abundance N ~ Poisson(lambda) surveyed by two independent observers with
detection p1 / p2; each individual is recorded by observer 1 only, 2 only,
or both. By Poisson-multinomial thinning the three observable cell counts are
independent Poissons (n_c ~ Poisson(lambda * pi_c)), so the marginal is
closed form with NO latent-abundance summation -- the simplest of the
self-contained pure-R families. y is an N x 3 cell-count matrix
(observer-1-only, observer-2-only, both); the state formula models
log lambda, detection the shared per-observer design (observers carry
separate coefficients). optim BFGS mode-find with observed-information vcov
(R/double_observer.R); full fitted / predict / residuals / simulate /
WAIC surface; simulate_double_observer(). Recovers abundance + both detection
probabilities with calibrated slope coverage. Site-level detection, non-spatial
laplace in v1. test-double-observer.R.occu_multi() (#116, \pkg{unmarked}
occuMulti; Rota et al. 2016). The joint occupancy state of S species at a
site, z in {0,1}^S, follows a log-linear model whose natural parameters are
first-order (per species) and second-order (per species pair -- the
interaction: positive = co-occur more than independent, negative = avoid); each
species is then detected conditional on presence. The latent state integrates
out by enumerating the 2^S states, so -- like royle_nichols() / occu_ttd()
-- the exact marginal is maximised by optim with an observed-information vcov
(R/occu_multi.R, no new C++). y is a list of S N x J detection matrices
(or a 3D array) with species =; the state formula is the shared occupancy
covariate design (each natural parameter carries its own coefficients),
detection the shared site-level per-species design. Full fitted
($psi = marginal per-species occupancy) / predict / residuals / simulate
/ WAIC surface; simulate_occu_multi(). The log-linear natural-parameter
covariate slopes are only weakly identified individually (they trade off), but
the marginal occupancy and the interaction recover cleanly -- the recovery test
targets those (interaction sign + magnitude over 15 seeds, positive and
negative). Shared design, first + second order, site-level detection, laplace
in v1 (per-parameter formulas, higher-order terms, visit-level detection, areal
fields are follow-ups). test-occu-multi.R.occu_ttd() (#116, \pkg{unmarked} occuTTD). A
survey records the TIME to first detection rather than a 0/1 outcome; at an
occupied site the time-to-detection is exponential with rate lambda (constant
hazard) over a survey of length surveyLength, censored if it reaches the end.
The state formula models logit psi, detection the log lambda rate. The
latent occupancy state integrates out in closed form (two states), so -- like
royle_nichols() -- the exact marginal is maximised by optim with an
observed-information vcov (R/occu_ttd.R, .ttd_site_loglik over per-site
sufficient statistics). y is an N x J matrix of detection times (a value
>= surveyLength is a non-detection, NA a survey not conducted).
simulate_occu_ttd(), full fitted / predict / residuals / simulate /
WAIC surface. Site-level rate, exponential TTD, non-spatial laplace in v1
(Weibull shape, visit-varying rate, areal fields, NUTS are follow-ups).
test-occu-ttd.R (constructor + gates, S3, 15-seed recovery with calibrated
rate-slope coverage).abun(mixture = "zip") / "zinb" (#116).
A structural-zero share omega of sites have N = 0 regardless of lambda
(L_i = omega * 1{all y_i = 0} + (1 - omega) * L_royle_i). Implemented as a
pure-R additive layer over the shared per-site Royle marginal
(nmix_site_marginal()), so no marginal-kernel change and the plain
Poisson / negbin paths are byte-identical; .tobs_fit_nmix_zip
(R/nmix_zip.R) mode-finds by L-BFGS-B over
[beta_lambda | beta_p | logit_omega | (log_r)] with box bounds only on the
pathological logit_omega / log_r corners (betas unbounded, no interior
bias). logit_omega is a model coordinate in coef/vcov/summary (like the
negbin log_r); fit$zi_omega reports the structural-zero probability. ZIP
recovers betas + omega cleanly with calibrated slope coverage; ZINB is weakly
identified (structural zeros vs NB overdispersion both absorb zeros) and
recovers in a higher-count regime. Non-spatial laplace only in v1 -- a
nuts / nested_laplace / structured-term request errors at the dispatcher with a
pointer rather than silently dropping the zero-inflation. simulate_abun(mixture = "zip"/"zinb", omega =). test-abun-zip.R.ms_occu_cover(response = "gaussian") and
occu_multiscale_cover(response = "gaussian"), the delta-normal magnitude
(mu = eta, shared residual SD sigma_pos, no log Jacobian) for a
pre-transformed / unbounded positive response. The density
(GaussianPositive) already shipped for the single-species arms (#112); this
threads the response enum, the OccuMultiscaleCoverGaussianCoupling typedef,
and a shared positive-code (0 lognormal / 3 beta / 4 gaussian) through
the community Laplace-EM, the multiscale nested-Laplace coupling, the
non-spatial Laplace / NUTS marginal, the WAIC ploglik kernel, and the simulate
draws (.occu_cover_pos_code, replacing the binary is_beta branches). The
gaussian residual SD recovers to lognormal parity (0.392 vs 0.392 at a matched
config); the shared-field spatial-factor ms_occu_cover() variant stays
lognormal-only and now rejects the new arm explicitly.detection = ~ icar() (or car_proper() / bym2()) term now loads a
spatially-varying detection surface on the detection arm of removal(),
distance(), fp_occu(), and dyn_abun(), alongside the existing
abundance/occupancy-arm areal fields. Each routes through the shared areal-BFGS
driver with a det_arm flag that sends the field offset to the detection scale
and reads the detection-arm gradient: the capture logit for removal() (per-
pass design, so the per-pass score is rowsum-summed to a per-site field
gradient), the true-positive logit eta_p11 for fp_occu(), the per-site
detection logit applied across every season for dyn_abun(), and the detection
scale eta_sigma for distance() (working under both the half-normal and the
hazard key, with the log-shape threaded as a global in the same evaluation).
fit$spatial_field_arm labels which arm carries the field. Poisson and negbin;
the detection-arm field is nested_laplace only (NUTS carries a field on the
abundance/occupancy arm alone). Recovery-tested in test-obs-det-field.R.distance() with a grouped random effect on the abundance arm now fits under
the hazard-rate key as well as the half-normal. The scalar log-shape is not a
per-site design column, so instead of a second global parameter it is profiled
over the AGHQ log-marginal -- an outer optimisation over the shape, each
candidate a full AGHQ fit at a fixed shape the DistanceGroupedOracle carries
-- and the profile log-shape row/column is inserted into the AGHQ variance-
covariance matrix (profile standard error, zero off-diagonal). Poisson and
negbin.temporal() term (AR1 / RW1 / RW2 / iid) on the
abundance/occupancy arm of removal(), distance(), fp_occu(), and
dyn_abun() now samples under method = "nuts" via the same non-centered
fixed-hyper field block the areal NUTS path uses, with the field map keyed by
period and a temporal whitened loading. Recovery-tested (AR1 field + slope, 0
divergences) in test-obs-nuts-temporal.R; a simultaneous areal-and-temporal
field under NUTS remains gated to nested_laplace with a pointer.spOccupancy
stMsPGOcc model. A shared icar() field on the first-season occupancy
formula of ms_dyn_occu() -- ~ 1 + icar(graph = adj) under
method = "nested_laplace" -- fits a spatial multi-season community occupancy
model. Because the first-season occupancy only sets the initial mixing weight
of each species' HMM, the per-(species, site) marginal is linear in it, the
same two-component mixture the single-season community field uses, so the field
routes through the shared block-coordinate driver with the HMM-forward
conditional likelihoods in place of the single-season emission. The forward-
backward kernels are vectorised over sites and the community EM is driven by an
analytic Fisher-identity gradient (validated against finite differences), so
the fit is fast. The shared field recovers cleanly (cor ~0.94 on an interior
field) and the community transition-dynamics coverage clears the 0.85 floor.
simulate_ms_dyn_occu() gained a field = argument. icar() only; a NUTS
sampler and bym2() / car_proper() fields remain follow-ups (gated with a
pointer).dyn_occu() (#124). A covariate
supplied as an [n_sites x (T-1)] matrix column of data now drives an
interval-indexed colonization or extinction rate --
colonization = ~ season_cov / extinction = ~ season_cov -- the recipe
dyn_abun() already uses for season-varying survival / recruitment (#80),
ported to the colonization-extinction HMM forward. The forward-backward E-step
uses a per-interval transition matrix (constant rates broadcast, so an existing
site-level or constant fit is byte-identical to before), the transition M-step
becomes a weighted logistic on the smoothed per-interval transitions, and the
exact-marginal refine gains a season-varying HMM-forward marginal that
calibrates the standard errors. simulate_dyn_occu() gained beta_gamma /
beta_epsilon to draw season-varying truth. Recovery and ~95% Wald coverage
hold across 20 seeds; the season-varying rate is gated under method = "nuts"
(the compiled forward reads one rate per site) with a pointer to laplace.k-of-n count GLMM without replicates (#125), the spOccupancy
svcPGBinom family. count(response = "binomial") with a per-site trials =
argument (default 1, i.e. Bernoulli) fits detection-free binomial data through
the same count() front door as the Poisson / negbin / Gaussian responses --
logit link, no dispersion (the variance is pinned by n). A plain areal field
(icar() / car_proper()) is supported (svcPGBinom): unlike the negbin /
Gaussian responses, the binomial is identified against a per-node field, at
trials = 1 too. ms_count(response = "binomial") extends the community
Laplace-EM the same way (community svcPGBinom; NUTS and a shared field /
latent factor are gated to laplace non-spatial for now). simulate_count() /
simulate_ms_count() gained a binomial draw with trials =; fitted values are
expected successes n * p, predict(newdata=) the per-trial probability, and
WAIC / residuals score the binomial pmf. The non-spatial fit matches
glm(..., family = binomial) to ~1e-3 and runs ~100x faster than
spOccupancy::svcPGBinom(). jsdm() stays the Bernoulli (trials = 1) alias.Identity-Gaussian positive arm for the cover hurdle (#112). cover(response = "gaussian") and occu_cover(response = "gaussian") fit the delta-normal
hurdle -- a Bernoulli presence process times a Gaussian magnitude on the
positive scale -- the lognormal arm without the log transform or its Jacobian
(residual y - eta, mean mu = eta, draw y = eta + sigma z). This is for a
signed / unbounded positive magnitude, NOT cover fractions on (0, 1) (use
beta / beta_oi for those). Recovery and 95% interval coverage hold at the
lognormal bar (20 seeds, pooled coverage 0.94-0.95). WAIC/LOO carry through; the
Freeman-Tukey PPC (non-negativity assumed) and NUTS on the joint path are gated
with a pointer. simulate_cover() / simulate_cover_joint() /
simulate_occu_cover() gained the Gaussian draw.
Intrinsic ICAR / BYM2 areal fields now sample under NUTS for the observation
families and ms_abun (#113), via the #71 sum-to-zero reparameterisation
(whitened loading drops the precision null-space, so z is auto-centred and
n_raw = n_units - 1 for ICAR / 2n - 1 for BYM2). Previously
car_proper-only. The shared field-block C++ (nuts_field_block.h) generalised
to a non-square loading byte-identically to the square path; a single R helper
.tobs_nuts_field_loading() is the one source of the whitened loading. Per
family the field posterior mean is sum-to-zero centred and correlates with
truth at 0 divergences (recovery in test-count-spatial-nuts.R).
Observation-family spatial/temporal breadth (#114). Four structured-term corners on the observation families:
nested_laplace on removal(), distance(),
fp_occu(), dyn_abun(): a temporal() term on its own (no companion areal
field) runs the shared areal-BFGS driver with a single AR1/RW1/RW2/iid block
on the family's structured arm. Recovers a known AR1 truth (cor 0.90-0.99),
slope on truth; per-family multi-seed recovery tests.dyn_abun() NUTS + temporal(): a fixed-hyper non-centered AR1/RW1/RW2/iid
field on the initial-abundance arm rides the SAME NUTS field block as the
areal field (nuts_field_block.h), with field_map = period index and a
temporal whitened loading fixed at the nested-Laplace temporal-only estimate.
0 divergences, temporal field cor ~1.0. Areal + temporal under NUTS stays
gated to nested_laplace.distance() detection-arm areal field: a field in the detection= formula
(detection = ~ icar(graph)) loads on the per-site detection scale (log
sigma) instead of the abundance arm -- a spatially-varying detection scale,
via the per-site sigma gradient the kernel already exposes. Half-normal key;
field recovers (cor ~0.97); recovery + gate tests.distance() hazard-key areal field on the abundance arm was already wired and
recovery-tested (test-distance.R), confirming that corner.
A temporal term under nuts on the other observation families, and a temporal
term on the count() family, remain gated with a pointer.temporal() now errors on the Laplace engine instead of being dropped.
.tobs_laplace() consumes the spatial and re terms but has no temporal
channel, and .tobs_fit_model() never passed one, so
tobs(~ x + temporal(year), family = occu(), method = "laplace") -- the
DEFAULT method for occu() / dyn_occu() / int_occu() -- silently fitted a
model without the field the formula asked for, with no warning. The term is
carried by nested_laplace (grid-integrated, composing with an areal field and
re() blocks) and by nuts (populate_temporal); the gate points at both.
This is the failure the svc() gate directly above it already guarded against.
Repaired 25 malformed sprintf() gate messages across 15 files. A multi-line
message written as comma-separated literals inside sprintf() without a
paste0() wrapper leaves only the FIRST literal as the format string; the rest
silently become ... arguments. 17 of the 25 errored at call time rather than
reporting anything: where a string literal landed on a %d (every
"spatial term has %d units but the model has %d sites" gate -- abun,
ms_abun, removal, distance, fp_occu, dyn_abun, ms_occu, and the
shared areal_bfgs driver -- plus the distance() / ms_distance()
cutpoints length check and the community icar node-count check), and one
invalid %q directive in the .tobs_family_methods dispatcher fallback. The
remaining 8 truncated mid-sentence, dropping the value the user needed: the
abun() / ms_abun() continuous-field gates stopped before naming spde(),
the package's only continuous-field escape hatch for that family, and the
unsupported-areal-type gate never interpolated the type it had rejected.
Every gate now emits its full message with the offending value interpolated.
tests/testthat/test-gate-messages.R covers both: the message text on the
repaired gates, and a package-wide AST scan asserting no sprintf() call in
R/ has a format/argument arity mismatch or an invalid directive. Gates fire
from error paths, so no fitting test exercised any of them -- which is how 25
broken messages accumulated unnoticed.
Requires tulpa >= 0.0.82 (ABI 34) and rebuilds against it.
BREAKING: the continuous NNGP fields gp() and svc() now require
prior_range = c(r0, alpha), a PC prior on the spatial range encoding
P(range < r0) = alpha -- the contract spde() has always had. tulpa 0.0.82
ships the range anchors unset and refuses a NNGP block without them
(gcol33/tulpa#144), and neither package invents a default: the range is in
coordinate units, so no value suits every dataset, and the range is weakly
identified by the likelihood, so a default would be a prior doing real work
on the posterior rather than a convenience. phi_prior_lower /
phi_prior_upper are gone (they parameterized the Uniform-behind-a-wall
prior that #144 removed). On unit-square coordinates
prior_range = c(0.1, 0.05) reads as "a 5% chance the range is under 0.1".
svc() on single-season occu() under NUTS is now recovery-validated rather
than shape-validated (gcol33/tulpaObs#118, #119). This is the measurement the
engine fixes could not make upstream: svc() is a tulpaObs term, so no fit
reachable from tulpa's own R side exercises the NNGP SVC path. Over seeds
1/2/3/11 at N = 150, J = 6, p = 0.6, truth phi = 0.25, sigma = 1.3:
divergences fell from 72-83% of post-warmup draws to 0 on every seed, and
phi from ~4 (the old Uniform prior's mean) to 0.14-0.23. Two upstream causes
contributed -- the range prior (tulpa#144) and the SVC marginal-SD prior,
which was improper on the coordinate it is sampled on and left sigma
unbounded above; fixing the latter is also what pulled phi onto its truth,
the two being the two ends of the GP ridge.
Surface correlation with the known truth did not improve (0.73 mean over
seeds, against 0.66 before). The calibration test used to predict it would,
and no longer asserts on it: sampler health and surface accuracy are separate
axes, and the latter is bounded by the information at these settings. The
skip()ped calibration assertions are enabled.
Occupancy with an areal field under method = "nested_laplace" estimates the
field precision. The state block was encoded at M = 1000 pseudo-trials per
site whenever the latent block was areal, but a state row carries one binary
occupancy observation, so this overstated its information roughly 1000-fold.
The field prior was swamped, between-cell binomial noise was read as a real
field, and the state slope inflated with the field through the logistic
conditional-to-marginal factor. Any nested latent block now encodes the state
arm at M = 1, the site's actual information content. On dyn_occu() +
icar() this moves the fitted field from sd 1.35 to 0.164 where the truth is
0, and from 2.46 to 0.917 where the truth is 1.0; the season-1 slope moves from
0.64 to 0.53 against a truth of 0.5, and interval coverage from 0.83 to 0.95.
The encoding is monotone in M and no arm regresses, single-season included.
dyn_occu() and int_occu() under method = "nested_laplace" report state
standard errors. Both fell through to a path that finds no H_beta on a
nested-Laplace fit and returns NA, so every such fit reported NA state SEs
(and therefore no intervals) while the detection, colonization and extinction
arms were finite. The Louis identity that the single-season arm already used is
not family-specific -- the state arm's complete-data score is
x_i (z_i - psi_i) in all three -- so it now covers them, reading the season-1
smoothed weight column for dynamic and the shared per-site weight for
integrated.
int_occu() + an areal field additionally reached neither of the two fixes
above: its M-step had no latent-block branch (so a nested areal block, which
arrives with an empty spatial slot, took the M = 1000 path), and the driver
never handed it the latent prior, leaving its field-aware E-step unreachable.
Its field now recovers at 0.976 against a truth of 1.0 and shrinks to 0.178
against a truth of 0, with finite SEs on every seed.
These combinations are now recovery-tested rather than smoke-tested
(test-occu-areal-recovery.R, test-int-occu-areal-recovery.R,
test-dyn-occu-areal-recovery.R, test-dyn-occu-svc-recovery.R). The previous
tests asserted a class, a type string and a field length on fixtures containing
no field, and passed throughout. Note that a null-field fixture cannot test this
path at all: the outer grid runs over tau in [0.3, 30], so a true field sd of
0 lies outside it and the marginal can only report the smallest field the grid
expresses -- "correctly shrunk" and "pinned at the grid edge" give the same
number. The new tests use a field the grid represents in its interior and assert
the grid mode is off both boundaries.
Community N-mixture latent factors: ms_abun() now takes latent(n) on the
abundance formula (the spAbundance lfMsNMix analogue), giving residual
species co-occurrence through per-site factors with per-species loadings, and
composes with a shared field (icar() / car_proper() / bym2() / spde())
for the spatial-factor case. The latent abundance still marginalises in closed
form per species-site, so the whole latent structure sits on log lambda and
the family reduces to one working oracle over the Royle marginal, driving the
shared block-coordinate engine added in 0.0.127. That oracle is the Louis
(1982) block nmix_site_marginal() already exposes -- no new kernel. The
residual species correlation recovers at ~0.99, and with a shared field the
field and factors separate cleanly (field ~0.98, residual ~0.94). Poisson only:
a negative-binomial size is a second per-site dispersion and is not identified
against a per-site latent structure. A plain shared field with no factors keeps
its existing C++ path.
Community distance sampling: new ms_distance() family (the spAbundance msDS
analogue) -- per-species binned distance sampling with Gaussian community
hyperpriors on the abundance and detection-scale coefficients, with latent(n)
factors (lfMsDS) and a shared field (sfMsDS). y is a
[n_sites x n_bins x n_species] array or a named list of per-bin count
matrices. The latent abundance marginalises per species-site exactly as for
distance(), so the family adds no C++: the existing distance kernel already
returns the per-site marginal pieces the community EM and the latent driver
need, and the driver's oracle is the same Louis formula the community N-mixture
uses. Under the hazard-rate key the log-shape is shared across species. The
community means are unbiased over seeds with ~0.9 Wald coverage, and the
residual species correlation recovers at ~0.99. Poisson only; no NUTS path yet.
simulate_ms_distance() draws through the same quadrature the likelihood
integrates against, so simulated data come from the pi the model is fit
against.
abun() and ms_abun() now reject a non-numeric K_max. K_max is the first
argument, so abun("negbin") -- reaching for the mixing distribution -- bound
the string to K_max, coerced it to NA, and surfaced much later as an
unrelated comparison error inside a kernel. Write abun(mixture = "negbin").
The shared community factor update now rejects a non-finite Newton step,
holding the previous iterate instead. A marginal that sums over a latent count
can return a non-finite curvature once an iterate wanders far enough out, and
the resulting NaN propagated into the factors and surfaced only later, as a
non-finite standard deviation in the rescale. The guard is a no-op when every
step is finite, so existing factor fits are unchanged.
Caught up with the current tulpa: the nested-Laplace fitters no longer pass
the removed verbose control, the joint paths pass k_samples (renamed from
diagnose_draws), and the AGHQ calls pass max_iter (renamed from maxit).
Against the current tulpa these had broken the areal count() path, the
varying-coefficient abundance fit, and every occu_cover() nested-Laplace
joint fit. The user-facing control names are unchanged.
The shared community Laplace-EM gained two optional arguments, both defaulting
to the previous behaviour byte-identically: sp_info (an analytic per-species
observed information, for families whose kernel already exposes it -- the
finite-difference fallback spends 2(P + G) full marginal sweeps per species
per Newton step rediscovering it) and init_b / init_Sigma (a warm start, so
a block-coordinate caller re-entering the EM once per outer pass resumes
instead of restarting cold).
One shared latent-structure engine for the community families
(R/community_latent.R). The block coordinate ascent, the areal Newton, the
latent-factor update and the field hyperparameter grids used to be duplicated
across ms_count_spatial.R, ms_count_factor.R and ms_occu_field.R; they
differed only in the per-(site, species) working score and curvature with
respect to an additive offset on the structured arm. A family now supplies one
callback -- working(eta) -> list(score, curv) (Poisson gives (y - mu, mu),
the occupancy two-state marginal its own pair, Bernoulli (y - psi, psi (1 - psi))) -- and gets the field, the factors and the SVC generalisation
from the shared driver. The refactor is behaviour-preserving: every existing
recovery test passes unchanged (the inner factor loops are exactly, not
approximately, equivalent, since site i's working weights depend only on
zeta[i, ] and species s's only on lambda[s, ]).
Community latent-factor and spatial-factor occupancy (ms_occu() +
latent(n), with or without a shared field; the spOccupancy lfMsPGOcc /
sfMsPGOcc analogues, gcol33/tulpaObs#119). Residual species co-occurrence on
the occupancy arm via Q per-site factors with per-species loadings. Composed
with a shared field the loadings are centred across species, so the field owns
the shared spatial mean and the factors the between-species residual. The
loadings / factors are identified only up to rotation, so the reported target
is the residual species covariance lambda lambda'.
jsdm() is now the community GLMM (gcol33/tulpaObs#121): per-species
coefficients under a Gaussian community covariance -- the spOccupancy lfJSDM
/ sfJSDM model class -- rather than shared fixed effects with a scalar
per-species intercept. A JSDM is exactly the ms_count() community model with
a logit link, so it now shares that binder, community Laplace-EM, latent
driver, NUTS target and S3 surface. latent(n) gives lfJSDM; a shared areal
field alongside it gives sfJSDM. method = "nuts" samples the exact joint
community posterior over the Bernoulli response through the family-aware
in-tree C++ target (byte-exact against the R oracle). The former single-block
correction routes (laplace_sla / laplace_gibbs / laplace_mi) do not apply
to the community EM and are no longer offered.
Continuous Matern (spde()) fields on the community latent driver. The mesh
nodes carry the field and the barycentric projector A maps them onto sites,
which is the same linear-map slot the areal group_var incidence uses, so the
driver's t(A) diag(w) A + tau Q solve applies unchanged. The precision is
Q(kappa) = kappa^4 C0 + 2 kappa^2 G1 + G1 C0^-1 G1 scaled by the driver's
tau, with kappa (the Matern range) chosen on a grid by the field marginal
exactly as proper-CAR's rho is; it is proper, so the field carries no
sum-to-zero constraint. ms_occu() and ms_count() gain continuous fields
alongside jsdm().
Single-species spatially-varying-coefficient abundance (count() +
spatial(~ 1 + w || cell, graph); the spAbundance svcAbund analogue,
gcol33/tulpaObs#120). No new engine: the multi-block prior already emitted one
weighted latent block per resolved field for the count model, and the nested
field summary already looped them into spatial_field + trend_fields. The
fit records the per-site contribution sum_k W[i,k] f_k[i] so fitted() adds
the full weighted field rather than the intercept field alone.
ms_count() gains predict() and residuals(), which it never had (only
coef / ranef / fitted / simulate).
Bug fix: tobs_waic() / tobs_loo() / tobs_cpo() scored the community
occupancy families without a shared areal field. community_ploglik.R
built the occupancy predictor from the coefficients alone, so the shipped
ms_occu() + icar() path was compared as a fixed-effect-only model. The
field (and now the factor) offsets enter the pointwise log-likelihood.
Bug fix: control = list(max.outer = ...) was read by the ms_count() /
ms_occu() dispatchers but rejected by control validation, so the block
coordinate outer cap was unreachable.
Bug fix: the areal count fit recorded the intercept field as its per-site offset, which is the whole contribution only when there is no SVC field.
ms_count() +
method = "nuts"; gcol33/tulpaObs#117): the Poisson sampler generalises to
negative-binomial and Gaussian. The negative binomial adds a per-species
dispersion random effect log_r_s ~ N(mu_log_r, sigma_log_r^2) as a second
community arm (the same structure the negbin Laplace-EM fits); the Gaussian
adds S free per-species residual variances log_phi_s (matching the Laplace
outer loop, which estimates each phi_s with no community prior), each with a
weakly-informative prior. One family-aware in-tree C++ FullGradFn
(src/ms_count_nuts.cpp) drives all three, byte-exact against the R oracle
.tobs_ms_count_nuts_logpost; warm-started at the community Laplace-EM mode,
0 divergences, NUTS == Laplace on the means, and the dispersion recovered
(fit$ms_dispersion carries r_s / variance). This removes the Laplace-EM's
mild negbin slope attenuation and returns calibrated non-Gaussian community
intervals. test-ms-count-nuts.R adds the negbin / Gaussian oracle
cross-check and recovery.skip_if_fast: the tulpa NUTS engine takes
n_iter as the total (warmup + sampling) count, so passing n.iter alone
returned zero post-warmup draws; the fitter now passes n.iter + n.warmup and
surfaces the real per-draw diagnostics (divergent / treedepth / epsilon)
instead of NA placeholders, and calls .tobs_nuts_rhat_ess() with its correct
single argument.NA) site x species entries
(ms_count() + method = "nuts"), matching the Laplace-EM path. An NA in the
response matrix marks that (species, site) as unobserved and drops it from the
per-(species, site) data sum, so a species keeps only its observed sites (ragged
coverage). The C++ FullGradFn and the R oracle mask the missing entries
identically, so the joint log-posterior + gradient stay byte-exact with NA
present; the reported N counts only observed entries. test-ms-count-nuts.R
adds the byte-exact NA cross-check (all three responses) and a NUTS-with-NA run
that agrees with the Laplace-EM fit on the same ragged data.msAbund NUTS; gcol33/tulpaObs#117): ms_count() +
method = "nuts" samples the exact joint posterior of the non-spatial
community Poisson GLMM -- the community means, the per-species coefficient
deviations, and the community covariance -- via a new in-tree C++ FullGradFn
(src/ms_count_nuts.cpp) over tulpa's NUTS engine, warm-started at the
community Laplace-EM mode. The reduced counterpart of the ms_abun NUTS: no
detection arm, no latent-N marginalisation, so the per-(species, site)
contribution is a plain Poisson log-likelihood. NON-CENTERED (b_s = C z_s,
z_s ~ N(0, I)) so the community covariance enters only the data term. The
joint log-posterior + gradient are byte-exact vs the R oracle
(.tobs_ms_count_nuts_logpost, checked to 1e-7); NUTS recovers the community
means and agrees with the Laplace-EM mode with 0 divergences. Poisson;
negbin / gaussian community NUTS are follow-ups.svcMsPGOcc analogue; gcol33/tulpaObs#118): a
varying-coefficient bar spatial(~ 1 + w || cell, graph) on the ms_occu()
occupancy formula now fits an intercept field plus one spatially-varying-
coefficient field per covariate,
logit psi_{s,i} = X_i (mu + b_s) + f0_{u(i)} + w_i f1_{u(i)}. Fit by block
coordinate ascent (R/ms_occu_field.R): the community occupancy Laplace-EM
with the shared field as a psi offset, alternated with a two-state-marginal
occupancy field update (.ms_occu_field_solve, a Newton over the per-(species,
site) occupancy marginal + the ICAR prior). Both fields recover cleanly (field
correlations ~0.96). icar() only; the plain single intercept field keeps the
in-tree C++ community-spatial path (sfMsPGOcc), so that route is unchanged.
Recovery-tested in test-ms-occu-field.R.bym2() (the Riebler scaled BYM2) now joins icar() and
car_proper() as a shared-field kind for ms_count() (gcol33/tulpaObs#117). The
field is the combined phi = a v + b u of a structured ICAR part v and an
unstructured iid part u, with a = sigma sqrt(rho/scale),
b = sigma sqrt(1 - rho) and the Riebler scale factor; (sigma, rho) are chosen
over a small grid by the field marginal (reported as fit$spatial_hyper$sigma
/ rho). Fit by a two-component joint Newton in the block-coordinate field step
(.ms_count_bym2_solve). The single shared intercept field only (no
varying-coefficient bar / group_var; use icar()/car_proper() there).
Recovery-tested in test-ms-count-spatial.R. With this, the community count
field supports the intrinsic (icar), proper (car_proper), and scaled BYM2
(bym2) areal priors.group_var (sites > field cells) now works for
ms_count() shared fields (gcol33/tulpaObs#117). When several sites share a
spatial cell -- icar(graph = cell_adj, group_var = "cell") -- the field has
one node per graph cell and a site->cell incidence aggregates the per-site
working residuals; fit$spatial_field is the per-cell field. The shared field
solve carries an optional incidence map M (identity when one node per site,
so the one-node-per-site routes are byte-compatible). Recovery-tested in
test-ms-count-spatial.R.car_proper() (a proper CAR) now joins icar() as a
shared-field kind for ms_count() (gcol33/tulpaObs#117). The field is
sum-to-zero deviations (the intercept owns the level); icar() fixes the
dependence at the intrinsic limit, car_proper() estimates the dependence
strength rho over a small grid by the field marginal likelihood (reported as
fit$spatial_hyper$rho). Recovery-tested in test-ms-count-spatial.R. The
field solve is shared with the shared-field / SVC / spatial-factor routes.ms_count(): a shared areal field icar() and latent factors
latent(n) now compose on one formula (the spatial-factor sfMsAbund;
gcol33/tulpaObs#117), log mu_{s,i} = X_i (mu + b_s) + f_{u(i)} + sum_q lambda_{s,q} eta_{q,i}. The block coordinate ascent runs the field update and
the factor update in the same loop; when both are present the factor loadings
are centred across species (sum_s lambda_{s,q} = 0) so the shared field owns
the spatial mean and the factors own the between-species residual co-occurrence
-- both recover cleanly (field correlation ~0.99, residual correlation ~0.99).
The shared-field-only, factor-only, and combined routes are now one fitter
(.tobs_fit_ms_count_latent, R/ms_count_spatial.R): single source of truth,
with the previous fitters as thin special cases. Recovery-tested in
test-ms-count-factor.R.Community latent-factor ms_count() (the spAbundance lfMsAbund analogue,
Poisson; gcol33/tulpaObs#117): a latent(n_factors) term on the abundance
formula models residual species co-occurrence with Q per-site latent factors +
per-species loadings, log mu_{s,i} = X_i (mu + b_s) + sum_q lambda_{s,q} eta_{q,i}. Fit by block coordinate ascent (R/ms_count_factor.R): the
community Laplace-EM with the factor term as a per-observation offset,
alternated with a Poisson factor update (alternating Newton on the factors and
the loadings, with a unit-variance anchor). The loadings / factors are
identified only up to rotation, so the recoverable target is the residual
species covariance Sigma_res = lambda lambda' (reported as
fit$ms_factor$residual_cov / residual_cor, with loadings and factors);
the residual correlation recovers cleanly (correlation with truth ~0.95).
fitted() / WAIC are factor-aware. Poisson; not composed with a shared areal
field yet. Recovery-tested in test-ms-count-factor.R.
With this, every tier of the spAbundance relative-abundance branch is covered:
abund (count()), spAbund (count() + areal), msAbund (ms_count()),
sfMsAbund (ms_count() + icar()), svcMsAbund (ms_count() + SVC bar),
and lfMsAbund (ms_count() + latent()).
ms_count() (the spAbundance svcMsAbund analogue, Poisson;
gcol33/tulpaObs#117, #118): a varying-coefficient areal bar
spatial(~ 1 + w || cell, graph) on the abundance formula now fits an
intercept field plus one shared spatially-varying-coefficient field per
covariate, log mu_{s,i} = X_i (mu + b_s) + f0_{u(i)} + w_i f1_{u(i)}. The
block-coordinate field step (R/ms_count_spatial.R) generalises to K
covariate-weighted ICAR fields solved jointly (a K x K sparse block system with
a per-field tau M-step); the intercept field is fit$spatial_field, the
varying-coefficient field(s) fit$trend_field(s). Both recover cleanly (field
correlations ~0.98). Recovery-tested in test-ms-count-spatial.R. Together with
the shared-field (sfMsAbund) case this closes the actionable community-SVC
item of #118 for the abundance branch.ms_count(): a shared areal field icar() on the abundance
formula now fits under method = "nested_laplace" (the spAbundance sfMsAbund
analogue, Poisson; gcol33/tulpaObs#117). One shared field across all species,
log mu_{s,i} = X_i (mu + b_s) + f_{u(i)}. Fit by block coordinate ascent
(R/ms_count_spatial.R): the community Laplace-EM with the field as a per-site
offset, alternated with a self-contained Poisson-ICAR field update (an analytic
sparse Newton + a closed-form tau M-step) -- pure R, no new C++. The field is
informed by every species at each site, so it recovers cleanly (field
correlation ~0.98) alongside the community means; fitted() and WAIC are
field-aware. Poisson + icar() only (an overdispersed community count is not
identified against a per-site field; bym2()/car_proper(), group_var, and
the varying-coefficient bar are follow-ups). Recovery-tested in
test-ms-count-spatial.R (community-mean recovery + pooled coverage + field
recovery over 20 seeds).count() (negbin / Gaussian): the gate now states, and its error points
out, that an overdispersed areal count with one field node per site is
fundamentally not identified -- the latent field absorbs all extra-Poisson
variation, so the field-integrated marginal likelihood is monotone in the
dispersion toward the Poisson limit (verified over a dispersion grid). No
estimator recovers the dispersion in this design; the message points to
abun() (an N-mixture on replicated counts) for overdispersion with a spatial
signal, or a Poisson areal count().ms_count() family: the community / multispecies relative-abundance GLMM
(the spAbundance msAbund model, gcol33/tulpaObs#117). Per-species GLMM on an
observed count / continuous response with Gaussian community hyperpriors on the
coefficients -- no detection, no latent state -- the abundance analogue of
ms_occu() without the occupancy layer, and the community version of
count(). Poisson / negative-binomial (per-species dispersion RE) / Gaussian.
y is an n_sites x n_species matrix (or a named list of count vectors);
species names the columns. Reuses the shared community Laplace-EM
(R/community_em.R) -- pure R, no new C++ -- with a count sp_ll / sp_grad.
Full S3 (coef / vcov / confint / ranef / fitted / simulate / WAIC)
plus simulate_ms_count(). Recovery-tested in test-ms-count.R: community-mean
recovery + pooled 95% coverage over 20 seeds (Gaussian and Poisson unbiased;
the negbin slope carries a mild first-order-Laplace attenuation) + dispersion
recovery. Non-spatial Laplace; the community-spatial (sfMsAbund) and NUTS
variants are documented follow-ups.count(): a plain areal field -- icar() or car_proper() -- on the
abundance formula now fits under method = "nested_laplace" (the spAbundance
spAbund analogue, gcol33/tulpaObs#117). The count response is observed
directly, so the fit is a single tulpa::tulpa_nested_laplace() call over the
count block with the areal field as its latent GMRF prior: the field
hyperparameters integrate on the nested outer grid, and the fixed effects plus
their covariance are grid-integrated (law of total covariance). fitted() is
field-aware in-sample. Poisson only -- with one field node per site a negbin
size / gaussian residual variance is not jointly identified with the field
under the fixed-dispersion loop, so those error with a pointer; bym2() (mixed
structured/unstructured field) and the improper car() are likewise gated to
icar() / car_proper(). Recovery-tested (fixed-effect coverage + field
recovery over 20 seeds) in test-count-spatial.R.New count() family: a GLMM on an observed count / continuous response
directly, with no detection process and no latent state -- the abundance
analogue of jsdm(), and the relative-abundance model of spAbundance
(abund). Poisson / negative-binomial (log link) or Gaussian (identity), one
value per site, supplied via y = or a two-sided formula left-hand side. The
negative-binomial size and the Gaussian residual variance are estimated by an
outer dispersion loop (reported in fit$count_dispersion). Full S3 surface
(coef / vcov / confint / fitted / predict / residuals / summary /
WAIC) plus simulate_count(). Non-spatial Laplace for this release; areal
(spAbund), community (msAbund), and NUTS are the documented follow-ups
(gcol33/tulpaObs#117). Recovery-tested (coefficient coverage + dispersion
recovery over 20 seeds) in test-count.R.
svc() on a family that does not consume it now errors with a pointer instead
of silently dropping the term (gcol33/tulpaObs#118). The continuous NNGP svc()
spatially-varying coefficient is wired only for single-season occu() under
method = "nuts"; on the count families (and occu() under laplace /
nested_laplace) it was extracted and discarded, fitting a model missing the term
the user asked for. The recovery-tested route for a spatially-varying
coefficient -- a weighted areal bar, spatial(~ 1 + w || cell, graph) with
method = "nested_laplace" -- is unchanged.
joint
everywhere, completing the rename the 0.0.111 message removal left partway.
control$engine = "joint" is the documented value for the occu_cover() /
occu_multiscale_cover() and standalone-occupancy spatial fits (the old
"joint_coupled" string still routes to the same fitter), and the internal
fitters, helpers, source files (occu_cover_joint.R, occu_joint.R,
occu_multiscale_cover_joint.R), and the stored engine label all read joint
now. No fitting behaviour changes.tobs() no longer messages when the cover-hurdle state formula is given
as formula = rather than occurrence = (both are accepted; occurrence
reads symmetrically with detection / positive), and occu_cover() no
longer messages on control$engine = "joint_coupled" (the documented engine
name; the internal "joint" default routes to the same fitter). Nothing
changes beyond dropping the two messages.tobs() are the bare names only (colonization / extinction,
omega / gamma, p10 / certainty); the <x>_formula spellings and the
simulate_dyn_ms_occu() / simulate_int_ms_occu() simulator aliases are gone,
along with the deprecation-warning helper and the tulpaObs-deprecated page.
There is no public release depending on the old names, so this is a clean hard
rename rather than a deprecation cycle.tobs() on bare process / symbol
names, matching detection, positive, and availability: col_formula ->
colonization, ext_formula -> extinction (dynamic occupancy),
omega_formula -> omega, gamma_formula -> gamma (open N-mixture), and
fp_formula -> p10, b_formula -> certainty (false-positive occupancy;
certainty is the b arm, renamed off b to avoid clashing with the by
argument). The old spellings still work but emit a one-time deprecation warning.simulate_dyn_ms_occu() -> simulate_ms_dyn_occu() and
simulate_int_ms_occu() -> simulate_ms_int_occu() (matching ms_dyn_occu()
/ ms_int_occu()). The old names remain as deprecated pass-throughs.cover_fit
(all cover() Laplace / nested-Laplace joint paths) and
occu_categorical_fit. These fits carry independent per-arm coefficient blocks
rather than a flat $model / $draws layout, so the inherited tobs_fit
methods that branch on object$model$model_type errored (nobs, residuals,
simulate crashed with "argument is of length zero"; vcov, confint,
logLik, tidy errored in the tulpa summarizer; fitted returned
silently-wrong values). A new intermediate class tobs_multiarm_fit supplies
arm-aware coef, vcov (exact block-diagonal, since arms are fit
independently), confint, logLik, nobs, glance, tidy, and summary,
and turns fitted / residuals / simulate into informative refusals
pointing at predict() / simulate_<family>(). The NUTS cover() path (which
does carry $draws) defers to the tulpa draw-based methods unchanged.>= 0.0.79, which makes integration = "grid_adaptive" decline to
the dense tensor before any inner solve on a small outer grid (default
control$adaptive_grid_min_cells = 48), so the adaptive integrator is never
slower than dense on the small hyperparameter grids typical of shared-trend
occupancy fits.>= 0.0.78 (Remotes gcol33/[email protected]), which adds the
integration = "grid_adaptive" outer-grid integrator. Occupancy / cover joint
fits (occu_cover(), cover()) forward control$integration unchanged, so
"grid_adaptive" reaches the joint driver: on a sharply-peaked hyperparameter
posterior (a strongly-identified field SD / Beta precision, the fine-grid
regime) it evaluates only the mass-concentrated outer cells for a posterior
that matches the dense tensor, and declines back to the tensor otherwise.tulpa dependency to the memory-safe v0.0.77 (the Imports floor and
the Remotes ref), which sizes the joint outer-grid thread budget from the
measured CHOLMOD factor rather than a fixed per-thread guess. The earlier
Remotes pin (@v0.0.70) conflicted with installing gcol33/tulpa at HEAD, so
a fresh pak::pak(c("gcol33/tulpa", "gcol33/tulpaObs")) could not solve the
dependency graph; the two refs are consistent again.Missing-at-random cover on occu_cover():
A detected visit may now carry a missing cover value (y_pos = NA). The
detection and occupancy arms keep the visit; only the Beta / lognormal cover
factor drops out, so a species recorded as present but with the cover not
measured contributes its detection without fabricating a cover. Missingness is
taken as at-random given the model covariates: the cover likelihood is the
product of the cover density over the observed detected visits only.
The cover density gates on detected & is.finite(cover) uniformly across the
builders (dense and compact), the joint nested-Laplace cell coupling, the
non-spatial Laplace / NUTS fits, the aggregated and latent cover arms, and the
WAIC / PSIS-LOO pointwise log-likelihood. With no missing cover the build and
fit are byte-identical to before (the NA sentinel is never introduced).
Performance (occu_cover joint fit):
Detection-pattern compression on the coupled nested-Laplace fit. Within a site, all-undetected visits that share a detection design row enter the occupancy mixture only through prod_v (1 - p_v) = prod_u (1 - p_u)^{w_u}, so they collapse to one row of multiplicity w_u carried in the p arm's existing n_trials slot; detected visits stay individual (each keeps its own per-visit cover, so the cover arm and its alignment are untouched). This is exact sufficient statistics, not an approximation: the compressed fit matches the uncompressed fit to floating point (equivalence test in test-occu-cover-compact.R; on the MOT-scale detection design the 923k plots reduce to ~298k unique rows, and the large all-undetected cells collapse 30-65x). On by default for the single-species path; off for the batched fused solve (per-species detection differs, so nodet rows are not exchangeable across species). getOption("tulpaObs.compress_nodet", TRUE) forces the uncompressed build for an equivalence check. No tulpa engine change (the weight rides the coupled arm's n_trials, already plumbed to CellResponse).
New family:
royle_nichols() -- Royle-Nichols occupancy (Royle & Nichols 2003;
unmarked::occuRN): occupancy with abundance-induced detection heterogeneity,
N_i ~ Poisson(lambda_i), detect ~ Bernoulli(1 - (1 - r_i)^{N_i}). The latent
N marginalises in closed form (a Poisson sum), so the exact marginal is
maximised with an observed-information vcov. Site-level detection, laplace
only for this first ship (R/royle_nichols.R). Parameter recovery validated
over seeds (< 5% bias, ~0.92-1.00 CI coverage); full fitted() / predict() /
residuals() / tobs_waic() / tobs_dic() / tobs_cpo() and a
simulate_royle_nichols() generator.Post-fit coverage: the S3 and diagnostic surface now spans the family set, closing the gaps where whole family clusters had no prediction or goodness-of-fit.
tobs_waic() / tobs_dic() / tobs_cpo() now work on the community-occupancy
families (ms_occu(), ms_dyn_occu(), ms_int_occu()), which previously
errored. The per-(species, site) marginal is scored over the community-mean
pseudo-draws with the per-species BLUP deviations plugged in; the two-state
marginal reproduces the C++ single-season kernel column-for-column
(R/community_ploglik.R).predict() and residuals() now work on the community-occupancy families
(per-species occupancy / detection, and per-species occupancy residuals)
instead of raising a "not implemented" error.jsdm() gains fitted() / predict() / residuals() (per-species occupancy
probability and presence residuals); previously fitted() errored.abun(), removal(), distance(), dyn_abun()) gain
tobs_test_dispersion() / tobs_test_zero_inflation() / tobs_test_outliers(),
scored on the per-site total count -- the natural overdispersion / excess-zero
unit for an N-mixture-type model. They were previously gated to single-season
occupancy.tobs_predict_spatial(), predict(terms = ), and tobs_marginal_effect() now
apply the state process's inverse link, so a count fit returns the abundance
intensity instead of a silently mis-linked logit-of-log-lambda.
tobs_marginal_effect(process = "abundance") is now available.tobs_richness() now accepts all three community-occupancy families (not only
ms_occu()); for the dynamic family it reports first-season richness.simulate_jsdm(), completing the per-family simulator set.foray_counts, meadow_cover,
peatland_occu) with a data-raw/make_datasets.R builder and LazyData,
so the family help pages and vignettes can run on shipped data.family_cover_hurdle.R,
laplace.R, occu_cover.R, occu_cover_joint_coupled.R, tobs.R,
ms_occu_cover_spatial.R) into focused modules (dispatch, arms,
postprocess, decode, diagnostics, helpers, callbacks, simulate). No
user-facing behaviour change; the fast smoke suite is unchanged.LinkingTo: gcol33/[email protected].cover() gains per-component copy(spatial(), terms = list(intercept = , trend = )), matching occu_cover()'s grammar: the presence field's intercept
and weighted-trend blocks can now couple onto the positive arm with
independent amplitude grids (control$alpha.grid / control$alpha.grid.trend)
instead of one shared amplitude. The whole-field forms
(copy(spatial()), copy(spatial(), alpha = )) are unchanged and stay
byte-identical. Per-component terms = must address every field block, and a
trend component named against an intercept-only field errors.cover() copy-coupling amplitude handoff (the alpha ->
control$alpha.grid[.trend] path) is now covered end-to-end -- whole-field
copy(alpha = ) byte-matches the shared-formula + control spelling,
per-component terms = with equal grids byte-matches the whole-field form,
and a decoupled terms = byte-matches the explicit control grids while
differing from the coupled fit. The occu_cover cover-arm INTERCEPT field SD
(the base per-cell cover-arm map, the direct deliverable) now has a
parameter-recovery test, alongside the existing trend-field recovery.?tobs disambiguates the two response arguments -- the long-frame
pivot response = (a column of data) versus the family constructor's
response = (the positive-part distribution), which can coexist in one call.tobs() documents the positive argument (the cover-hurdle positive-arm
formula) under its own name. The roxygen entry carried the stale response
label left over from the 0.0.95 family-constructor rename, so the generated
help documented a response argument absent from the signature and left the
real positive formal undocumented. Documentation only; no change to fitting.glance() now reports pareto_k_is_ess as the numeric importance-sampling
effective sample size the engine computes (matching fit$pareto_k_is_ess at
the top level), instead of coercing it to a logical. The field was documented
and consumed as a boolean "the k-hat column is a quad-ESS fallback" flag, but
the engine has no such fallback: pareto_k is always the outer k-hat (or NA)
and pareto_k_is_ess is always the IS-ESS on the PSIS-smoothed weights, so
as.logical() discarded the number (a healthy IS-ESS of, e.g., 110 became
TRUE). pareto_k_is_ess / control$k.samples is the relative IS efficiency.to = is fully decoupled from the spatial-field call surface in both cover
families. Placement now tags each field's arm on the evaluated spec directly,
instead of round-tripping the arm through a deparsed formula, so to = is no
longer an argument the bar form reads: the spatial(~ ... || node) bar takes
only graph and by, and an explicit to = is an unknown argument. The arm
is chosen by placement (write the field in that arm's formula) and shared
across arms with copy(), exactly as in 0.0.96; fits are byte-identical (the
cover placement-equals-shared and occu_cover() recovery suites are
unchanged). The bespoke to = deprecation guard is removed.copy() is now wired into the cover() hurdle engine, so a shared spatial
field across the two cover arms is written the same way as in occu_cover():
place the field in the presence formula and copy(spatial()) in the
positive formula. copy(spatial()) estimates the coupling amplitude on the
default grid (byte-identical to the previous both-arm spelling);
copy(spatial(), alpha = grid(c(...))) integrates it over a supplied grid and
copy(spatial(), alpha = 0.5) fixes it.
to = on a spatial field is retired from both cover families. An arm is now
chosen by placement (write the field in that arm's formula) and a field is
shared across arms with copy(); a field in the single shared formula
reaches both arms. The || / | (independent / correlated MCAR) axis is
unchanged. Every routing to = expressed is byte-identical under the new
spelling: shared intercept/trend via a shared-formula field or copy(),
arm-specific and correlated single-arm fields via placement, MCAR both-arm via
a shared-formula | bar or copy(). A user-supplied to = now errors with a
pointer to placement / copy().
positive -> response: cover(response = "beta"),
occu_cover(response = "beta"), ms_occu_cover(), occu_multiscale_cover().
This removes the name clash with the positive = ~ x arm formula on a
tobs() call, where positive otherwise meant both the positive-cover arm
formula and the positive-arm response distribution. Positional calls
(cover("beta")) are unchanged; a named positive = now errors. The stored
family field and the whole fitting engine are unchanged, so fits are
byte-identical.cover() per-arm formulas now carry spatial fields by placement, completing
"arm = formula" for both hurdle families: a spatial() term in presence or
positive becomes an arm-specific field on that arm, byte-identical to the
shared-formula to = spelling (the field is routed through the same machinery
and indexed onto the arm's rows by the fitter). Fixed effects and fields both
follow placement; copy() reuses a named field across arms. temporal() /
re() in a per-arm formula are still declared on the shared formula. Closes
gcol33/tulpaObs#111.cover() per-arm formulas (arm = formula): cover(presence = ~ x, positive = ~ t) gives the presence and positive hurdle arms their own fixed effects (two
independent designs), matching occu_cover()'s per-arm formulas. The single
shared formula stays the back-compat spelling (byte-identical; the full cover
test suite is unchanged). First cut: per-arm formulas carry fixed effects only;
declare fields on the shared formula with to = (per-arm field placement is
gcol33/tulpaObs#111).
Bugfix: an lme4 random-effect bar on the occu_cover() detection or
positive-cover formula (detection = ~ det_cov1 + (1 | habitat), and the
(x | g) / (0 + x | g) slope spellings) is fitted again. The arm-field
lifting introduced in 0.0.90 round-tripped each arm formula through
terms() / reformulate(), which dropped the parentheses off a bar
((1 | g) -> 1 | g); R then re-parsed ~ det_cov1 + 1 | g as
(det_cov1 + 1) | g, so the downstream RE parse saw no bar and silently
dropped the random effect. The lift now re-parenthesizes bar terms it keeps.
The re(g) spelling was unaffected. Restores gcol33/tulpaObs#102, #103.
occu_cover() detection-arm spatial field: a spatial-field term on the
detection formula (detection = ~ 1 + spatial(~ 0 + time || cell, graph = adj), or the explicit to = "detection" spelling) now fits a
spatially-structured detection probability. The non-copied field block scatters
onto the detection predictor by riding the detection arm with field_coef = 1
while the shared occupancy field is kept off detection by the spatial_idx = 0
sentinel -- the same mechanism the detection random effect uses
(gcol33/tulpaObs#102), so no engine change was needed (closes gcol33/tulpa#140).
Recovery of the detection field SD is tested across seeds.occu_cover() field placement: a spatial-field term written in the positive
formula (positive = ~ t + spatial(~ 0 + time || cell, graph = adj)) declares
an independent cover-arm field by placement, byte-identical to the explicit
to = "positive" spelling (which is retained). The intake lifts the term onto
the occurrence formula with the arm tag; copy() stays on its arm's formula.simulate_occu_cover(det_field = TRUE) injects a known
detection field. A detection-arm field (to = "detection") is reserved but
rejected at parse: the joint C++ substrate scatters fields onto the occupancy
and cover arms only, so a detection field is unidentified until the substrate
is wired (gcol33/tulpa#140).occu_cover() now accepts control$sigma.grid.pos.field, the SD grid for the
independent cover-arm field (to = "positive", #110). The joint engine already
read it and defaulted it to control$sigma.grid; it was missing from the
family's control whitelist, so passing it errored. Setting a coarse field grid
now keeps the added axis from multiplying the outer-grid cost.cpp_occu_cover_ppc_agg): the detection replicate plus
the aggregated (mean / median) or shared cover-RE (latent) cover replicate,
drawing from R's RNG stream in the former order (byte-identical). With this,
every posterior-predictive / PIT diagnostic that draws from posterior
components -- occu_cover (all cover modes), cover, single-season -- is a C++
kernel. The remaining R generators (simulate()) are left in R by design: they
select a posterior draw with sample.int, whose index sampler is not
byte-reproducible in C++, so porting them would change results for no runtime
gain -- the same boundary the package already draws for Monte Carlo generation..tobs_loo_pit_from_limits) moved its
per-observation PSIS loop into tulpa's C++ cpp_psis_loo_pit (PSIS columns
parallel, the jitter in index order), byte-identical to the former R loop.
Requires tulpa >= 0.0.64.cpp_cover_pit_cdf,
deterministic) and PPC (cpp_cover_ppc); single-season PPC (cpp_single_ppc)
and randomized PIT residuals (cpp_single_pit). The replicate draws come from
R's RNG stream via the R:: samplers in the SAME order as the former R loops
(the posterior-draw selection sample.int stays in R), so under a fixed seed
every result is byte-identical. All four positive families for the cover PIT.occu_cover() posterior diagnostics moved their per-draw loops into C++:
the deterministic detection-summary CDF limits (the randomized-PIT / LOO-PIT
building block, cpp_occu_cover_cdf_limits, parallel over draws) and the
posterior predictive check (cpp_occu_cover_ppc, cover_aggregate = "none").
The PPC draws the latent state, detection replicate, and cover replicate from
R's RNG stream via the R:: samplers in the SAME order as the former R loop, so
under a fixed seed the discrepancy is byte-identical; the aggregated cover
modes keep their R path.ms_occu_cover_spatial) WAIC / DIC / LOO pointwise log-likelihood moved its
per-draw loop into C++ (cpp_ms_ocs_ploglik). The former R loop unpacked the
NUTS draw (community mean, per-species deviation, shared fields W, occupancy
loadings L, optional cover loadings Lpos, log-dispersion), assembled each
species' predictors with the shared-factor offset W L[s,] on occupancy (and
W Lpos[s,] on cover), and evaluated the dense occu_cover per-cell marginal;
the kernel does all of that, parallel over draws. Both constrained and
unconstrained loading parameterisations canonicalise to one packed layout.
Byte-close (~1e-15) to the R oracle, thread-count invariant. With this, every
family's pointwise log-likelihood in the WAIC / DIC / LOO path is a C++
kernel.ms_abun) WAIC / DIC / LOO pointwise log-likelihood
moved its per-draw loop into C++ (cpp_ms_nmix_ploglik_batch). The former R
loop reconstructed each species' deviation b = C z from the non-centered
NUTS draw (a log-Cholesky factor per arm) and called the per-species Royle
marginal in R; the kernel now does the log-Cholesky reconstruction and the
per-(species, site) marginal (compute_nmix_site) internally, parallel over
draws. Byte-identical to the former loop (same kernel), thread-count invariant.occu_multiscale_cover) builds its per-cell pointwise log-likelihood in a
C++ OpenMP kernel (cpp_occu_ms_cover_ploglik), parallel over draws. This was
the last pure-R marginal in the criteria path (the cell -> plot -> visit
three-level mixture plus the per-detected-visit cover density, formerly an
apply() over draws). The kernel mirrors the R reference
.occu_ms_cover_nonspatial_ll draw for draw (~1e-14) and is thread-count
invariant; both cover families and both visit-block layouts. WAIC / DIC / LOO
for this family now honour n.threads.cpp_nmix_ploglik_batch),
removal (cpp_removal_ploglik_batch), false-positive occupancy
(cpp_fp_occu_ploglik_batch), binned distance sampling
(cpp_distance_ploglik_batch), and open N-mixture / dyn_abun
(cpp_dyn_abun_ploglik_batch). The per-site marginal was already C++
(compute_*_site); each kernel now loops draws over it directly, parallel over
draws, so the result is byte-identical to the former R loop (0 difference,
same kernel) and thread-count invariant. n.threads reaches these through
.tobs_ploglik_from_draws.[n_sites x max_visits]) occu_cover() no-aggregation
path is flattened to the ragged form and shares the 0.0.72 kernel, so every
occu_cover() fit (compact or dense) builds its pointwise log-likelihood in
parallel.cpp_occu_single_ploglik),
multi-season dynamic HMM (cpp_occu_dynamic_ploglik), and multi-source
integrated (cpp_occu_integrated_ploglik) -- moved their per-observation
marginal out of R. The R functions keep the (BLAS) linear-predictor
assembly and the draw-invariant count gathering; the marginal runs in the
kernel. n.threads is plumbed through .tobs_ploglik_from_draws.
Each kernel mirrors its former R loop (reproduced as the test oracle),
agreeing to libm rounding (~1e-15) and thread-count invariant.cover() hurdle pointwise log-likelihood (the WAIC / DIC / LOO input) now
runs in a C++ OpenMP kernel parallel over posterior draws
(cpp_cover_hurdle_ploglik), covering all four positive families (lognormal,
lognormal_trunc, ordinal, beta). Same treatment as the occu_cover() ragged
path in 0.0.72: tobs_waic() / tobs_dic() / tobs_cpo() on a cover_fit
honour n.threads. The kernel mirrors the R reference .tobs_cover_hurdle_ll
(retained for the posterior-mean plug-in and the tests), agreeing to libm
rounding (~1e-14) and thread-count invariant.occu_cover() fit now build the
pointwise log-likelihood in a C++ OpenMP kernel that parallelises over
posterior draws (cpp_occu_cover_ploglik_ragged). This was the dominant
serial cost on large single-species fits: the draw loop scales with
n.draws x total plots and previously ran single-threaded in R. tobs_waic(),
tobs_dic(), and tobs_cpo() gain an n.threads argument (default: all but
four logical cores, matching the occu_cover fit's own outer-grid default). The
kernel mirrors .occu_cover_site_ll_ragged draw for draw and accumulates each
site's visit sums in visit order, so it agrees with the R path to libm rounding
(~1e-13) and is thread-count invariant. On ~200k visits x 200 draws the pointwise
build drops from 17.9 s (R) to 0.76 s (16 threads); the dense and aggregated
(mean / median / latent) cover paths are unchanged.occu_cover() can now give the cover (positive) arm its OWN independent
spatial field, decoupled from the occupancy field's alpha copy
(gcol33/tulpaObs#110). An arm-specific spatial() bar with a single
to = "positive" on the occurrence formula --
spatial(~ 1 + time || cell, graph = adj, to = "positive") -- adds a
non-copied ICAR block on the cover arm (intercept + per-covariate trend
fields, each with its own precision integrated on the outer grid). It composes
with the shared occupancy field: occupancy still drives psi and, via the alpha
copy, delta_cover_exp, while the independent cover field carries a
cover-specific structure the alpha copy cannot express -- so delta_cover_cond
is spatially varying instead of collapsing to a global slope when alpha -> 0.
Reported as sigma_pos_field / sigma_pos_field_<col> with the per-cell
posterior in fit$pos_field / fit$pos_field_table. simulate_occu_cover()
gains pos_field / sigma_pos_int / sigma_pos_trend to simulate it. Scope:
per-visit cover (cover_aggregate = "none"); ICAR only (bym2/car read as
ICAR); not composed with the correlated | MCAR field, the latent cover RE,
or the batched fused path.|, free-Sigma MCAR) intercept + slope spatial field can now sit
on a single cover arm: spatial(~ 1 + time.sc | cell, graph = adj, to = "presence") puts a free 2x2 cross-coefficient Sigma on the occurrence arm
alone (the occupancy intercept and time-slope fields covary), with no
cross-arm copy. Previously the | form was copy-only and required both arms.
The single-arm field uses the 0-sentinel spatial_idx on the other arm and
carries no alpha (reported as NA); sigma_mcar / rho_mcar recover the
field SDs and their correlation (gcol33/tulpaObs#109).cover(positive = "beta_oi") -- a one-inflated Beta cover family. Plots
recorded at exactly full cover (y = 1) are modelled as a genuine point mass
instead of being clamped to 1 - 1e-6 (which biases the interior precision).
With a constant inflation probability the likelihood factorizes: pi is the
share of positive plots at the ceiling (a binomial proportion, reported as
pi_one with its SE), and the interior Beta is fit on the (0, 1) plots.
predict() returns the one-inflated conditional cover pi + (1 - pi) * mu.
Works on the non-spatial and nested-Laplace (areal) paths
(gcol33/tulpaObs#108).model = "bym2":
spatial(~ 1 || cell, graph = adj, to = "positive", model = "bym2") now puts a
BYM2 (structured ICAR + iid) field on the cover arm, previously restricted to
icar / car / car_proper. The block fits as a non-copied length-2 latent over
the paired (sigma, rho) grid, and the joint-draw projection reconstructs the
rho-mixed unit field so predict() / WAIC see the full mix. Recovers the
field and the structured fraction (gcol33/tulpaObs#107).cover() / occu_cover() spatial fits accept control$prior.phi, a
regularizing hyperprior on the cover-arm dispersion grid (the Beta precision
under positive = "beta", the log-scale SD under lognormal). Forwarded to
tulpa's new prior_phi, it re-weights the phi.grid axis by the chosen
density (list("pc.prec", c(U, alpha)) / list("half_normal", scale))
instead of an implicit flat prior (gcol33/tulpa#139). Requires tulpa >= 0.0.62.tulpa import floor raised to >= 0.0.61, the version the engine is built and
tested against (the CCD outer-integration path and the adjacency() graph
front door). The previous >= 0.0.57 floor could pair this release with a
tulpa too old for those, so an install that did not also upgrade tulpa would
resolve to a broken combination. Metadata only; no code change.predict(type = "change") for the joint cover-family (occu_cover()) and the
rerouted standalone occu() SVC fit now reports the per-cell change certainty,
not just the change. The change table gains, per cell: the start / end
occupancy (p_T1 / p_T2, or psi_T1 / psi_T2 for occu()) with their own
.sd / .lwr / .upr interval, and a .prob_pos column per headline delta
(delta_p, delta_cover_cond, delta_cover_exp; delta_psi for occu())
giving the directional posterior probability P(delta > 0). All are taken over
the grid-integrated draws, so they carry the joint posterior rather than a
plug-in of the means, and they are pure additions (existing columns unchanged).
This makes the two-point change prediction a complete drop-in for a per-cell
occupancy-trend summary (start, end, change, direction certainty) without a
hand-rolled trend pass downstream.tobs_data() gains type = "positive": a positive-real (0, Inf) response
for the lognormal / gamma cover arm, alongside type = "cover" (a [0, 1]
proportion, the beta arm). Both share the floor-to-NA absence policy; only
"cover" enforces the upper bound (the shared validation lives in one
.tobs_floor_continuous() helper). The long-frame builders pick the storage
type from the family's positive distribution, so a lognormal / gamma cover
that exceeds 1 now round-trips through occu_cover_inputs(), the single-fit
long-frame path, and by= instead of being rejected by the [0, 1] check.
occu_cover_inputs() exposes this as positive = (default "beta").occu_cover() fit now accepts a long / plot-level frame directly,
the same contract the by= batch path already supported for many species:
pass site = , visit = , response = (the 0/1 detection column) and
y_pos = (the cover column), plus any visit-level det.covs = , with a long
data, and tobs() builds the paired occurrence / cover arms and the
site-level design internally. This removes the hand-built tobs_data() x2 plus
the manual occurrence/cover alignment check from user scripts. The arms default
to the compact (ragged) layout on the nested-Laplace route (no per-site visit
cap); set control$compact = FALSE for the dense grid. The fit is byte-
identical to the hand-built route (asserted in test-occu-cover-long.R).occu_cover_inputs() exposes that builder: it returns the y / y_pos /
visits / site_data bundle from a long frame so the arms can be inspected
before fitting. The long -> arms construction (.occu_cover_response_pair())
is now single-sourced and shared by the by= batch loop and the single-fit
path, rather than re-implemented per call site.tobs(by = ) for occu_cover() now builds its per-species response arms and
the shared visit grid compactly on the nested-Laplace route (its default
looped backend), instead of allocating B dense [n_sites x max_visits]
response matrices plus a dense visit grid. The padded-grid memory was the
reason the batch could not run on the uncapped EVA data the single-fit path
handles; the looped batch now scales the same way (memory is O(observations),
not the padded grid). Each species' fit is unchanged to the compact-vs-dense
tolerance (asserted in test-occu-cover-by.R). The non-joint laplace route
stays dense (its engine reads the padded grid), as does the opt-in fused
backend (control$batch.backend = "fused"), which stacks dense per-species
columns by design; control$compact overrides the default.tobs_waic() / tobs_dic() for occu_cover() now build the pointwise
log-likelihood in memory-adaptive draw-chunks. The heavy transient was the two
[n_plots x n_draws] per-visit predictor matrices (about 15 GB at 1000 draws on
the full no-cap EVA data); .occu_cover_ploglik_core() now processes the draws
in blocks sized to a fraction of free RAM (/proc/meminfo on Linux, the
optional ps package elsewhere, a 4 GB default otherwise), so the peak stays
bounded on a memory-tight node. WAIC is a sum over draws, so the result is
byte-identical to the unchunked computation regardless of chunk size (asserted
in test-occu-cover-compact.R). The full draw count is kept, so WAIC precision
is unchanged -- callers no longer need to trade draws for memory.occu_cover() path now carries an observation-arm random
effect ((1 | g) on the detection or positive-cover formula), so a
random-habitat detection fit runs uncapped like the fixed-effects spec. The RE
group codes (and any random-slope design) are resolved over the V valid visits
directly -- site-level groupings broadcast via the visit's site -- so the
compact fit is byte-identical to the dense fit, BLUPs included (asserted in
test-occu-cover-compact.R). Removes the earlier guard that errored on
compact + observation RE.Remotes: no longer pins gcol33/[email protected]. That tag predates
the tulpa (>= 0.0.57) import floor (added in 0.0.60), so a fresh
pak::pak("gcol33/tulpaObs") resolved tulpa to v0.0.50 and failed the version
constraint. The gcol33/tulpa remote now tracks the default branch (where
tulpa and tulpaObs are released together) so it cannot go stale against the
import floor again; gcol33/tulpaMesh stays pinned at @v0.1.3 to match
tulpa's own remote (a differing tulpaMesh ref across the two would itself be a
resolver conflict).tobs_data(compact = TRUE) builds a compact (ragged) tobs_data: the response
is stored as one row per valid site-visit (a tobs_ragged carrier) and each
detection covariate as a length-V vector, instead of a padded
[n_sites x max_visits] matrix. The joint nested-Laplace occu_cover() engine
consumes the valid visits directly (it compacts the dense grid to exactly these
rows anyway), so a fit on compact input is byte-identical to the dense fit on
the same data -- now asserted in tests/testthat/test-occu-cover-compact.R
(means, sds, predict(), and WAIC all match). Because the compact layout never
materialises the padded grid, its memory is the number of observations rather
than n_sites x max_visits, so a site with tens of thousands of visits no
longer needs a per-site visit cap before the data can be built. Scoped to the
joint nested-Laplace path; other engines, an observation-arm random effect, and
cell-aggregated cover read the dense grid and error clearly on compact input.occu_cover cell-coupling spec declares its dense cross-Hessian pairs
(dense_cross_pairs()), so the joint engine (tulpa >= 0.0.57) no longer
reserves a J x J slab for the all-undetected (p, p) block (it is the rank-1
self-cross) or the always-zero (p, pos) / (pos, pos) blocks. A cell with J
visits is now O(J), not O(J^2), which is what lets the uncapped compact path
fit a grid with a cell holding tens of thousands of plots without running out of
memory. The DESCRIPTION tulpa floor moves to >= 0.0.57.tobs() exposes the single-batch + bootstrap outer Pareto-k controls for the
joint-coupled families (occu_cover(), cover(), occu(), multiscale),
forwarding them to the engine (tulpa >= 0.0.50, gcol33/tulpa#127):
control$diagnose.draws (the precision knob; legacy k.samples accepted as an
alias), control$k.bootstrap, control$k.tail.points, control$k.conf.bands.
Replaces the removed k.batches / k.adapt / k.batches.max (#123/#124). The
fit's $joint_fit carries pareto_k_se_boot, pareto_k_ci_low /
pareto_k_ci_high, pareto_k_se_formula, pareto_k_tail_points /
pareto_k_tail_points_requested, pareto_k_band_confident, and the top-level
diagnose_draws / diagnose_cost_ratio. For a tighter k raise diagnose.draws,
not k.bootstrap.tobs() exposes control$k.adapt + control$k.batches.max for the
joint-coupled families (occu_cover(), cover(), occu(), multiscale),
forwarding them to the engine's adaptive batched outer Pareto-k
(gcol33/tulpa#124). With k.adapt = TRUE the batch count grows from
k.batches until the reliability band resolves (the k-hat band lands in one
band) or the k.batches.max cap is reached; default off (k.adapt = FALSE).
Requires tulpa >= 0.0.49.tobs() exposes control$k.batches for the joint-coupled families
(occu_cover(), cover(), occu(), multiscale), forwarding it to the engine's
batched outer Pareto-k (gcol33/tulpa#123). With k.batches > 1 the fit reports
the median outer k-hat over that many independent importance batches plus the
observed pareto_k_lo / pareto_k_hi range (the diagnostic's Monte Carlo
spread, not a posterior CI); the band is classified off the median. Default
1L (single batch), so existing fits are unchanged. Requires tulpa >= 0.0.48.New occu_categorical() family (gcol33/tulpaObs#106): a presence + nominal
(unordered) K-class hurdle. Each unit is absent (y = 0) or present in one of
K classes (y in 1..K); presence is a Bernoulli arm and the class given
present is a baseline-category multinomial logit (the last class the baseline),
the two arms factorising the likelihood exactly. This is the categorical
counterpart of cover() (presence + magnitude) and the K-class generalisation
of fp_occu() (its K = 2 confusion case): for an ordered class response use
cover(positive = "ordinal"); this family is for classes with no ordering
(colour morph, microhabitat use, classifier label). The multinomial math is
the FD-validated tulpa kernel (multinomial_logit.h); the non-spatial Laplace
fit is the vectorised R Newton over the same closed forms. Ships with
simulate_occu_categorical(), a predict() method (presence, conditional, and
joint class probabilities), and parameter-recovery tests. Non-spatial Laplace
for this first ship; spatial fields / NUTS (the native multi-process
likelihood) and the latent-class misclassification variant are documented
follow-ups. Requires tulpa (>= 0.0.47).
New cover(positive = "lognormal_trunc") positive arm (consumer of
gcol33/tulpa#122): an upper-truncated lognormal for bounded cover -- a Gaussian
on log-cover upper-truncated at log(1) = 0, so it cannot place mass above
cover = 1 the way "lognormal" can. It rides the joint nested-Laplace path on
the engine's new truncated_gaussian family via a per-plot truncation ceiling,
threaded through encode / decode, the pointwise log-likelihood, the PIT CDF,
and posterior-predictive replication (inverse-CDF truncated-normal draws), with
the truncated-lognormal conditional cover mean in prediction. Requires
method = "nested_laplace". Recovery-tested against bounded cover data, and
verified close to the plain lognormal fit under negligible truncation.
Requires tulpa (>= 0.0.47).
tobs_waic() / tobs_cpo() gain loo.unit = c("obs", "cell"): the
cross-validation unit. The default "obs" is the family's pointwise unit (one
plot for cover(), one site for occu_cover()) and is byte-identical to the
previous call. "cell" switches to leave-one-group-out cross-validation
(LOGO-CV): the fit's own per-observation cell map is supplied to
tulpa::tulpa_criteria(group = ), so each spatial cell is one fold instead of
each plot / site, without the caller hand-building the map. Implemented for
cover() (the areal field node via spi_full, when plots are grouped with
group_var) and occu_cover() (the site_cell map); a non-spatial fit has no
cells and errors with a pointer to loo.unit = "obs". The per-family
column -> cell map is the single .tobs_loo_cell_map() (each family's
pointwise builder fixes its own column order). Equivalent to passing
group = the cell map directly. Requires tulpa (>= 0.0.45). (tulpaObs#105)New cover(positive = "ordinal", breaks = ...): an interval-censored Gaussian
positive arm for Braun-Blanquet (ordinal cover-class) data, the
measure-invariant counterpart of the lognormal / beta arms. The cover is
recorded only as the ordinal class it falls in, so the latent log-cover is
Normal(eta, sigma^2) censored to the observed class band and the likelihood
is the class probability MASS -- an ordered probit with KNOWN thresholds, no
free cutpoints and no change-of-variable Jacobian. breaks are the interior
class boundaries on the (0, 1) cover-fraction scale (open outer classes added
automatically); class bands are lower-closed so a representative sitting on its
own boundary (the myscale class a rep at 0.03) maps to its own class. Wired
on the joint nested-Laplace engine only (the per-observation (lower, upper)
bounds are consumed by tulpa 0.0.44's built-in interval_gaussian family); the
single-Laplace and NUTS paths error with a pointer, and the simplified-Laplace
skew correction no-ops (sla_status = "ordinal_unsupported"). sigma_pos
carries the latent log-cover SD, integrated on the same outer phi grid as the
lognormal sigma. WAIC / PIT / PPC are the genuine discrete-class diagnostics.
Recovery-tested from censored class data (test-cover-ordinal.R); shared
positive-arm family / phi-grid logic extracted to .cover_pos_family_grid().
Requires tulpa (>= 0.0.44).
Response / site / visit input handling now has a single source of truth in
R/inputs.R, replacing the near-identical site-count cross-check each family
binder hand-rolled (occu, abun, removal, distance, fp_occu,
dyn_abun, dyn_occu, jsdm, the ms_* community families, and the cover /
occu_cover / multiscale families):
.tobs_check_site_count() is the one check that the site dimension of y
matches nrow(data). The historical messages are preserved -- a 2D
response counts "rows", a 3D / per-source response "sites", and the cover
hurdle's response vector "values". (The integrated families' deliberate
partial-coverage site maps are unchanged: they are not a mismatch.)tobs() accepts a tobs_data frame in place of the (data, y, visits)
triple. The frame's occ.covs / y / det.covs are unpacked through the
same pipeline raw arguments take (det.covs is already the named-matrix
shape visits consumes), so a frame fit is byte-identical to the
equivalent raw fit. Passing y = / visits = alongside a frame errors.tobs() records canonical response totals on the fit (fit$dims:
n_sites, max_visits, n_sources) and, under control$verbose, reports
them once before dispatch.test-inputs-frame.R.occu_cover() outer Pareto-k diagnostic re-solves are validated against
the faster tulpa 0.0.43 path (gcol33/tulpa#118, now the dependency floor):
test-occu-cover-pareto-k.R pins that the fast default (Shamanskii reuse,
loosened inner tol, near-neighbour batch order, per-cell nearest-grid-mode
warm start) reports the SAME k-hat as the byte-for-byte exact diagnostic
(tulpa.kdiag.refresh = 1, tol = 0, no reorder, no per-cell warm), to a
few 1e-4, and that the k-hat agrees with loo::psis on the diagnostic's
actual importance ratios.control$diagnose.k), since it reports k-hat only and does not move the
betas, SDs, or field.occu_cover_joint_coupled.R / CLAUDE.md: the
binding per-solve cost is the per-Newton-iteration Hessian/gradient SCATTER
(the beta arm's per-observation digamma/trigamma fill, 73-83%), NOT the sparse
Cholesky factorize (a flat ~0.5 ms, 8-12%, not super-linear up to ~1100 cells).tobs_fit itself (gcol33/tulpaObs#104). It was only at
fit$joint_fit$pareto_k, so a diagnostic script reading fit$pareto_k
directly got NULL.
occu_cover(), occu() spatial, and occu_multiscale_cover() joint-coupled
fits promote pareto_k, pareto_k_is_ess, pareto_k_scope, and
pareto_k_proposal_source to the fit top level (shared
.tobs_promote_pareto_k()), and glance.tobs_fit() surfaces pareto_k,
pareto_k_is_ess, and pareto_k_proposal_source.pareto_k_proposal_source (the tulpa 0.0.41 mode-Hessian proposal,
gcol33/tulpa#116, now the dependency floor) reads "mode_hessian" -- the
importance proposal is curvature-backed, so the k-hat stays trustworthy
even when a sharp posterior collapses the integration grid to ~1 cell -- or
"grid_moment", the regime to watch, when it comes from the grid-weighted
node covariance as the grid concentrates.control$diagnose.k defaults
OFF (gcol33/tulpaObs#101), so with it off no field or glance() column is
added.occu_cover() (and every other) nested-Laplace joint fit gets a usable
outer-grid progress signal on a detached / redirected run, via the tulpa
0.0.40 reporter fix (gcol33/tulpa#115, now the dependency floor):
1/N) -- the master thread emits from inside the parallel region;ETA >=) until a parallel cell has finished.
The detached-run heartbeat file (control$progress.file, gcol33/tulpaObs#43)
is unchanged and remains the robust signal where a console flush is buffered
away.occu_cover() are verified end to end
against simulated truth (crossed-intercept, nested, and correlated-slope
parameter recovery; predict() shrink-to-mean for unseen levels), closing
gcol33/tulpaObs#103.occu_cover() random slopes on the detection / positive-cover arms now
fit on a meaningful Sigma grid regardless of the covariate's scale. Each slope
covariate is standardized to unit SD before fitting (mirroring the
fixed-effect design autoscaling), and the reported slope BLUPs and SDs are
back-transformed to the covariate's natural units (correlation is scale-free).
The correlated-slope free-Sigma (miid) grid now uses a principled compact
default -- symmetric correlation nodes including 0 and reaching strong +/-, and
log-spaced SD nodes -- so the marginal correlation is no longer forced into a
lop-sided range; widen it with control$re.logchol.grid.p /
re.logchol.grid.pos. simulate_occu_cover() gains re_det slope-covariate
scale control (slope_sd) for non-unit-scale recovery tests.occu_cover() supports random effects -- intercepts AND slopes -- on the
detection and positive-cover arms on the shared-field nested-Laplace
path, with the usual lme4 bar or re() spelling
(gcol33/tulpaObs#102, #103). The grouping is per visit (one code per
site-visit), distinct from the per-site occupancy-arm random intercept (#56),
so a many-level categorical visit covariate (EUNIS habitat, observer) is
partially pooled.
(1 | habitat) + (1 | observer))
and nested ((1 | region/site)): each term is one iid latent block,
named sigma_re_p / sigma_re_pos (suffixed by the grouping variable when
several share an arm).(x || g), (0 + x | g)) is one per-row weighted iid
block per coefficient; a correlated slope ((1 + x | g)) one multivariate
free-Sigma miid block. A slope term reports an [n_groups x n_coefs]
BLUP matrix, a per-coefficient sigma, and (correlated) a cor matrix, all
marginalized over the grid.
BLUPs are in fit$re (a per-term list keyed by arm or "<arm>:<var>") and via
ranef(). predict() gains type = "detection" and sums each term's offset
(weighting a slope by its covariate column in newdata), shrinking unseen /
held-out levels to the population mean. Several crossed terms or a correlated
slope grow the outer grid, so set control$integration = "ccd"; the free-Sigma
grid uses a compact default, widened with control$re.logchol.grid.p /
re.logchol.grid.pos. simulate_occu_cover() gains a re_det argument for
crossed / nested / slope detection random-effect truth. As with the
occupancy-arm RE the grid-integrated variances carry the binary / small-cluster
inner-Laplace attenuation (a lower bound); the BLUPs recover the per-group
structure. Needs tulpa >= 0.0.39.occu_cover() and occu_multiscale_cover() joint nested-Laplace fits
(method = "nested_laplace") now default the outer Pareto-k accuracy
diagnostic OFF (control$diagnose.k = FALSE), matching occu_joint_coupled().
Profiling traced the joint-fit runtime to this diagnostic: it re-solves the
inner Laplace k.samples (200) times on the full areal field, each a
super-linear sparse factorization, and accounted for 84-90% of wall time
across field sizes -- the binding limit on per-species fits at fine spatial
resolution (gcol33/tulpaObs#101). The diagnostic only reports the k-hat value;
the fitted coefficients, SEs, and spatial field are byte-identical with it on
or off. Re-enable it with control$diagnose.k = TRUE (and size the importance
batch with control$k.samples).tobs_data(type = "cover") gains cover.floor (default 0): a cover value at
or below the floor is stored as NA rather than as a positive observation,
because the cover hurdle's positive arm is positive-only and a cover of 0 is
an absence handled by the occurrence arm. This stops a 0 padded across
unsampled cells (instead of left NA) from entering the positive arm as a
fabricated zero, which, spread over a grid, flattens the spatial field. The
conversion is reported with a one-line message; cover.floor = -Inf keeps
every value verbatim.Formula-native cross-arm coupling for occu_cover(). The cover (positive) arm
carries a scaled copy of the occurrence arm's spatial effect, selected
structurally with a constructor and no required name:
copy(spatial(), alpha = grid(c(0.25, 0.5, 1.0, 1.5))). The selector is
type-carrying and reorder-stable: copy(spatial(cell_idx)) disambiguates by
grouping variable when several spatial effects are present, and an integer
position is rejected. A per-component amplitude is
copy(spatial(), terms = list(intercept = grid(g0), time.sc = grid(g1))),
keyed by the field's own block names and required to address every block. The
coupling coefficient alpha (= sigma_pos / sigma_occ, the INLA copy=
analogue) is marginalised over the grid; a scalar fixes it. Coupling is
formula-native and explicit: a block with no copy() is decoupled (the field
rides occupancy only), and decoupling is structural rather than a magic
alpha of 0. engine = "joint" replaces engine = "joint_coupled", and
occurrence = / positive = read symmetrically with detection =
(formula = stays as a deprecated alias). The fit is byte-identical to the old
control-driven path (max abs difference 0).
Full-model field-folded information criteria for occu_cover(). tobs_waic(),
tobs_dic() and tobs_cpo() now fold the spatial intercept and trend fields and
the per-visit detection process into the pointwise log-likelihood, so WAIC / DIC /
CPO / LOO are numerically comparable to the INLA and spOccupancy criteria.
tobs_cpo() additionally returns a LOO-PIT ($pit) for calibration checking and
a per-observation LOO Pareto-k ($failure). Cross-checked against a brute-force
pointwise evaluation (max abs difference < 1e-8).
build: the tulpa dependency floor is raised to tulpa (>= 0.0.38) and the
Remotes install reference to gcol33/[email protected].
tulpa dependency floor is raised to tulpa (>= 0.0.37) and the
Remotes install reference to gcol33/[email protected], version-matching tulpaObs
to the current tulpa release.tulpaMesh dependency floor is raised to tulpaMesh (>= 0.1.3) and
the stale Remotes install reference is corrected to gcol33/[email protected]
(the current tulpaMesh release).tulpa dependency floor is raised to tulpa (>= 0.0.36), locking
tulpaObs to the matching tulpa release (the two are ABI-coupled via
LinkingTo: tulpa). The Remotes install reference stays at
gcol33/[email protected].Remotes install reference for tulpa is updated to
gcol33/[email protected] (the current tulpa release). The Imports floor stays at
tulpa (>= 0.0.34), the minimum API tulpaObs requires.predict() on an occu_cover() joint fit now propagates the visit-level
positive-arm covariates from newdata into the conditional cover, instead of
holding them at the reference (gcol33/tulpaObs#95). The positive arm splits a
site design (intercept) from its visit-level covariate design at fit time; the
predict handler rebuilt only the site design and zero-padded the visit columns,
so a positive covariate supplied in newdata (e.g. the time axis of a
type = "change" map) never entered the cover linear predictor and
delta_cover_cond came out flat at zero. The handler now rebuilds the
visit-level positive design from newdata with the same builder and column
order as the fit. The model retains its visit-level formulas
(formulas$pos_visit, formulas$det_visit) to support this. The occupancy /
occurrence arm and the cover() hurdle were unaffected.tobs_data() now preserves factor / character visit-level detection
covariates as categorical. A column named in det.covs that is a factor or
character is reshaped into a tagged character site x visit matrix (carrying its
level set) instead of being coerced to numeric, and the detection / positive
visit design expands it to k - 1 dummies for a k-level factor, with the first
level (sorted-unique first value for a character column) the reference. Numeric
det.covs follow the existing double-matrix path unchanged. The visit design
builder now adds k - 1 contrast coding for any visit-level factor (previously a
no-intercept visit formula expanded a factor to full k-dummy coding, collinear
with the site-level intercept).occu_cover() spatial NUTS (method = "nuts" with a car_proper() field on
the occupancy arm, gcol33/tulpaObs#74) errored against tulpa (>= 0.0.34). The
fixed-hyper warm start passed the single-block copy= argument to
tulpa_nested_laplace_joint(), which the single-block joint path no longer
accepts; the copy coefficient is now declared on the cover arm as
field_coef = list(name = "alpha", grid = ...), the same convention the
nested-Laplace joint_coupled path uses. Recovery, 95% interval coverage, and
the calibration of the sampled coefficient SDs to the nested-Laplace SEs are
restored.abun() points to dyn_abun(); random effects with visit-level detection
covariates on abun() name the two ways to proceed; and the multi-block
nested-Laplace random-effect path lists the supported models (iid, ar1,
rw1, rw2).simulate_occu() documents its actual return value (y, data, truth); the
coords element it never produced is removed from the help.status = "experimental" to
"working" after parameter-recovery and CI-coverage validation against
simulated truth across seeds (gcol33/tulpaObs#96-100):
occu_cover() (#96): 95% Wald CI coverage holds near nominal on every path
-- non-spatial laplace / nuts and the shared-field nested_laplace
(joint_coupled) engine -- for both the beta and lognormal positive arms
(measured pooled coverage 0.92-0.96). The recovery gates move from the 0.80
experimental floor to the 0.85 working floor (pooled), and the beta arm and
the shared-field paths gain explicit coverage gates.occu_multiscale_cover() (#97): the four-arm recovery (cell psi, plot theta,
visit p, cover) is validated on the nested_laplace and non-spatial
laplace engines (pooled coverage ~0.95), and the coupled SVC-trend field
recovers its shape. The availability / detection identifiability reduction is
now surfaced: a fit with no within-plot replication (single releves) emits a
message that theta and p collapse to the identified product theta * p, and
the reduction is tested (psi and theta * p recover, the levels separately do
not).ms_occu_cover() (#98): community-mean 95% CIs are gated at the 0.85 working
floor (measured ~0.92). The community-variance AGHQ debias cap
(re.aghq.maxdim, default 4) is documented as a hard scope limit -- the
tensor AGHQ is exponential in the total RE dimension -- with the EM variance
above the cap explicitly tested as a lower bound. The reduced-rank
spatial-factor (JSDM) path's loading / association recovery and per-species
map calibration remain validated.ms_dyn_occu() (#99): community-mean coverage gated at 0.85 (measured ~0.98;
the shared colonisation / extinction dynamics cover at ~1.0) and the
per-species first-season occupancy / detection variance components recover
the realised spread.ms_int_occu() (#100): the shared occupancy mean and the per-source
detection components recover across seeds and more than one source; the
community-mean coverage gate is tightened (measured ~0.89). For both
ms_dyn_occu() and ms_int_occu(), a NUTS sampler and an areal-field path
are a deliberate follow-up, not part of the working surface.feat(tobs): tobs() gains a by = "<species_col>" argument for per-species
batched fitting. Given a long / plot-level data frame and a species column,
tobs() splits by species, builds each species' response onto one shared
site x visit grid (via tobs_data()), and routes the per-species responses
through the batched-independent driver, returning a tobs_batch. Scoped to
occu_cover() and cover(). The split only reorganises the input: each
species' fit is identical to the hand-built multi-response batch and to an
independent single-species fit (equivalence tested to 1e-10).
tobs_batch_fit() renamed to tobs_get(): it extracts one species' fit from a
tobs_batch, and the old name read like a fitter.
fix(cover): arm-specific spatial fields (single-arm spatial(~ ... || node, to = "presence" / "positive"), method = "nested_laplace") are now projected
at predict() time. The fields were estimated correctly (sigma > 0 on each
arm) but every per-cell prediction came back flat, because the per-arm field
block stored the fit-time per-observation node map and the accumulator skipped
it whenever its length did not match the design (always true at predict, where
the design has one row per cell). Arm-specific blocks now mirror the
shared-field path: a block carries only its per-arm amplitude (membership) and,
for a trend field, its covariate column name; the node map and per-cell weight
are supplied by the consumer through .tobs_joint_arm_eta -- predict() passes
the newdata cell map and column, the pointwise-log-likelihood consumer rebuilds
the per-observation map and weight from armspec_blocks
(.tobs_armspec_obs_units / .tobs_armspec_obs_wfun). The log-likelihood /
WAIC / PPC consumer is numerically unchanged; the shared-copy and standalone
paths are unaffected (gcol33/tulpaObs#95). Also fixes the spurious
"this fit has 1 time-varying (trend) field(s); pass time_col" error raised by
predict() on an intercept-only arm-specific fit: the positional "blocks 2.."
trend convention no longer applies to arm-specific fits, whose blocks each
carry their own weight column name.
occu() / occu_cover() joint_coupled is no longer
materialised as a dense V x V block per site (V = visits at that site). The
block is analytically the rank-1 a p p^T (the cell density depends on the
visits only through the scalar P0 = prod_v (1 - p_v)), so nodet_mixture_block
/ occu_nodet_block (src/occu_coupling_shared.h) now emit the per-site
(a, p) into the engine's rank-1 self-cross descriptor (needs tulpa >= 0.0.34)
and fold the rank-1 diagonal into the stored detection diagonal. Occupancy data
is sparse, so nearly every site hits this branch; the per-iteration
Gauss-Newton scatter -- the profiled inner-solve bottleneck at EVA scale
(~99.9% of inner-solve wall time) -- drops from O(sum_s V_s^2) to O(sum_s V_s)
(gcol33/tulpaObs#94; measured ~17x faster scatter at 32 visits/site on an
occu_cover joint_coupled fit, the gap widening with visit count). The fit is
numerically unchanged: a deterministic
occu_cover joint_coupled fit (lognormal + beta arms) matches the former dense
path to <= 1.4e-14 on coefficients, SDs, the spatial field, and logLik. Shared
by the occu_only, occu_cover, and occu_cover_latent cell-coupling specs;
occu_multiscale_cover keeps the dense path (its no-detection (p, p) block is a
nested mixture, not a single rank-1).occu() / occu_cover() / occu_multiscale_cover() joint_coupled summaries
no longer runs a serial-R solve(Qk, E) over the full betas + field latent
per outer-grid cell. .joint_inner_vcov_block() now calls the engine's
parallel selected-inversion primitive (tulpa:::cpp_joint_inner_vcov_blocks,
needs tulpa >= 0.0.33): the betas block and betas x field cross are solved
directly, the field marginal variances come from one Takahashi pass, and the
cells run concurrently over n.threads.outer (gcol33/tulpaObs#93,
gcol33/tulpa#112, #113). The SD summary is numerically unchanged; the betas
block and field marginal variances match the former dense path to machine
precision. fit$joint_vcov keeps its betas covariance and field marginal
variances; its field x field off-diagonal now carries only the between-cell
mode-dispersion term (the within-cell field cross-covariance, read by neither
the summary nor the Q_k-direct predict() draws, is no longer formed).cover(engine = "nested_laplace") hurdle path is
migrated off the copy= argument the joint single-block fitter dropped in the
(sigma, alpha) reparam (it now hard-errors). The positive arm declares
field_coef = list(name = "alpha", grid = alpha_grid) and the top-level copy
is removed, mirroring occu_cover()'s single-block path; the multi-block /
MCAR / trend branches (which still route through the copy-taking multi-block
dispatch) are unchanged. Restores the single-field cover() nested-Laplace
fits that errored at fit time (test-cover-hurdle-nested-joint.R).n_visits (the count of valid visits)
while indexing a max_visits-strided response that stores missing visits in
place as a -1 sentinel, so an interleaved or leading NA terminated the loop
early and silently dropped the trailing valid visits, corrupting the
likelihood, gradient, and posterior. The loops now stride the full
max_visits dimension and skip the sentinels via the existing guard;
n_visits is kept only for the "no surveys" early-out. Only method = "nuts"
with non-trailing missing visits was affected (method = "laplace" always
iterated the full dimension). test-occu-interleaved-na.R asserts that
leading-vs-trailing NA encodings of identical data give identical fits under
both methods (single and dynamic occupancy)..tobs_clamp_eta() helper over one .TOBS_ETA_BOUND constant, replacing ~20
identical local cl <- function(e) pmin(pmax(e, -30), 30) definitions and the
inline copies across the package; the C++ community-field kernel gains the
matching clamp_eta() / kEtaBound (#89)..tobs_logsumexp2() (a max-shifted log1p form), replacing
the byte-identical block copied across the three occu_cover paths (#90)..tobs_build_visit_X(). occu(), abun(), and removal() previously inlined
their own builders and kept the visit (Intercept) column, which duplicated
the site-level detection intercept; all four families now drop it through the
one helper, so an intercept-bearing visit formula no longer makes the stacked
detection design collinear (#91).by = "factor" argument on a cover spatial bar replicates the
areal field across the factor's levels -- the graph becomes the block-diagonal
I_L (x) Q (L disjoint copies) with each observation offset into its level's
copy, sharing one precision across levels (one outer-grid axis). It composes with
all three bar forms: shared (||), correlated (|, MCAR), and arm-specific, via
tulpa::tulpa_bar_field_replicate(). Fixes the shared-|| path, where the
replication updated the adjacency but left the cached n_spatial at the base node
count, so the replicated index was rejected as out of range; n_spatial now
tracks the replicated graph, and the intentional L-component disconnection no
longer emits the generic connectivity warning. test-cover-spatial-bar-by.R
covers all three bar forms plus per-level field recovery.logLik() (and AIC() / BIC() / glance()) now return a
finite value for dyn_occu fits (#87). The EM+Laplace packer left the
log-likelihood unpopulated; build_laplace_fit now evaluates the exact
HMM-forward marginal at the fixed-effect mode, and the dynamic exact-marginal
refine moves the EM mode onto colext's MLE. tidy() / glance() are
re-exported so they resolve after library(tulpaObs).cover() stored its verdict at fit$converged while every other family uses
fit$convergence = list(converged, n_iter), so a mixed-family QC pass read NA
for one location. cover_fit now carries the same record, and convergence() /
converged() are exported as the documented accessors that normalise both
layouts.P(z_{t-1}=0, z_t=1 | y) / P(z_{t-1}=1, z_t=0 | y),
and psi1 / detection use the smoothed marginals P(z_t | y). Previously the
E-step used only forward-filtered occupancy and a marginal-product
approximation (1 - w_{t-1}) w_t for the transition events, which converged to a
biased fixed point about 3.4 log-likelihood below unmarked::colext on the same
data (inflated colonization / extinction). The fit now matches colext's MLE to
within the EM pseudo-count discretisation; a colext coefficient-equivalence
gate is added to test-dyn-int-occu-recovery.R (#86). The shared single-species
dynamic E-step also feeds the nested-Laplace and simplified-Laplace dynamic
paths; the community dynamic family (ms_dyn_occu) has a separate E-step and is
unchanged.test(refimpl): removal_laplace and distance_laplace now gate head-to-head
against unmarked::multinomPois and unmarked::distsamp (coefficients to
~5e-3, byte-identical log-likelihoods), extending the N-mixture-vs-pcount
gold standard to the removal and distance families; plus a CI-runnable
community-mean recovery smoke gate for ms_abun (gcol33/tulpaObs#83).
test(recovery): multi-seed point recovery for dyn_occu (psi1, gamma, epsilon,
p) with an independent R forward-recursion anchor for the dynamic-occupancy HMM
marginal, and multi-seed recovery for single-source int_occu (gcol33/tulpaObs#84).
NOTE: both families' deterministic Laplace standard errors are overconfident
(the occupancy-intercept SE is ~an order of magnitude too small), so the gates
assert point recovery; the SE-calibration gap is a separate kernel issue.
docs: DESCRIPTION gains URL / BugReports; internal issue tokens stripped
from rendered help; two non-ASCII characters removed from code comments; and
the family front doors abun(), cover(), distance(), ms_occu(), and
occu_cover() gain runnable \donttest{} fit-and-summary examples
(gcol33/tulpaObs#85).
feat(dyn_abun): a grouped random intercept on the detection (p) arm now
fits on both engines (#82), alongside the existing initial-abundance (lambda)
arm RE. Put the bar on the detection formula, e.g.
tobs(~ x, detection = ~ (1 | site), family = dyn_abun(), y = y). Unlike the
initial-abundance arm -- where the predictor enters only the season-1 initial
distribution, so the data-conditional weights are precomputed once and each
quadrature node is an O(K) dot -- the detection predictor enters every season's
observation pmf, so each AGHQ node re-evaluates the full exact HMM-forward
marginal through a closed-form second-order eta_p forward-mode pass
(compute_dyn_abun_p_curv / cpp_dyn_abun_p_loglik); NUTS adds a non-centered
p-arm offset routed through the kernel's existing detection gradient. One
grouping factor, on lambda OR p (a random effect on both arms in one fit is
rejected; the AGHQ path integrates one arm at a time); survival / recruitment
never carry random effects. Poisson and negative-binomial initial abundance.
ranef() / coef() surface the detection RE as sigma_p<t>_* (AGHQ) and
log_sigma_p_* (NUTS), with AGHQ debias on sigma_p.
occu() varying-coefficient (SVC) spatial bar
spatial(~ 1 + x || cell, graph = adj) now fits through the joint direct-grid
engine, single-arm (occupancy + detection, no cover arm), instead of the EM
fixed-point nested-Laplace path (#81). The EM path oscillated and did not
converge on real EVA-scale occupancy data -- a large-amplitude field, sparse
detection, and a rich detection model drove the M-step to bounce and the fit
was truncated at the iteration cap with converged = FALSE. The joint engine
integrates the field hyperparameters on a direct outer grid with no
fixed-point iteration, so it cannot oscillate; it is the same engine
occu_cover() uses, with the cover arm removed. The occupancy mixture runs
through a new occu_only cell-coupling spec that reuses the occupancy /
detection derivatives of the occu_cover specs (single source of truth). The
reroute is scoped to the SVC case: a plain single intercept field, a correlated
MCAR bar, and temporal / random-effect structure stay on the EM path. occu()
still reports sigma / sigma_trend (the field SDs marginalized over the
grid), spatial_field / trend_field, and predict(type = "occupancy" | "change") now reads the shared areal field at each cell. Requires
tulpa >= 0.0.30.spatial(~ 1 + x || cell, graph = adj) (a cell-indexed spatial intercept field plus a spatial trend
field weighted by a per-site covariate) now fits on a standalone occu()
nested-Laplace model, not only on occu_cover() (#67). The two-term spelling
icar(graph, group_var = "cell") + icar(graph, weight = x, group_var = "cell")
fits the same structure. This is the occupancy-only analogue of the
occu_cover() coupled trend, with no cover arm and no coupling alpha -- the
apples-to-apples match for an occupancy-only spatially-varying-coefficient
model. Each field becomes one areal latent block on the existing multi-block
nested-Laplace path: the intercept field is a plain icar block, and the trend
field rides the same graph with a per-site svc_weight so its contribution is
weight_i * z[cell_i]. The areal blocks are cell-indexed (one field node per
graph cell, many sites per cell via group_var), so the field count can be far
smaller than the site count. occu() reports sigma for the intercept field
and sigma_trend for the trend field (the field SD marginalized over the outer
grid), alongside spatial_field and trend_field. Requires tulpa >= 0.0.30
(the single-arm driver gained the per-observation svc_weight). The correlated
| bar (a free-Sigma MCAR field) stays on occu_cover() and errors with a
pointer on occu(). Recovery-tested in test-occu-spatial-svc-recovery.R.occu() now points
to both the occu_cover() joint path and the standalone occu() nested-Laplace
path; the term is rejected only on the engines that cannot carry it (the NUTS
sampler and the single-Laplace path).|) free-Sigma MCAR spatial coefficient fields on
occu_cover() (#63). spatial(~ 1 + x | cell, graph = adj) on the occupancy
formula declares the intercept and x-slope Besag fields as a separable MCAR with
a free 2x2 cross-covariance Sigma (the within-arm covariance among the fields,
integrated over the outer mode-centred CCD in log-Cholesky coordinates), then
copies the whole correlated field onto the cover arm with one estimated
amplitude alpha. Reports sigma_mcar / rho_mcar / alpha_mcar, marginalized
over the grid. The independent || spelling (#61) is unchanged. Scoped to
icar, the standard (non-latent) cover path, and at least two coefficient
fields; correlated | does not compose with a per-group occupancy RE or the
v2/v3 escape engines, which error with a pointer. This required a tulpa engine
change (the coupled per-cell scatter now handles INDEXED_MULTI blocks);
requires tulpa >= 0.0.29. Recovery-tested in test-occu-cover-spatial-mcar.R.| bar in an occu_cover() / occu_multiscale_cover() formula
previously fell through to the independent || expansion silently -- the
free cross-covariance was dropped with no warning. It now routes to the MCAR
field (occu_cover()) or errors with a pointer (occu_multiscale_cover()),
never a silent wrong model.cover() /
occu_cover() (#61). spatial(~ 1 + time || cell, graph = adj, to = c("presence", "positive")) is a compact spelling of the existing two
weighted-areal-term coupled trend: the intercept column is the unweighted
shared field, each covariate column a weight-scaled coefficient field, both on
the bar node index, presence-anchored and copied to the positive arm with an
estimated coupling alpha. It desugars to exactly the two-term form (~ time + icar(graph, group_var = "cell") + icar(graph, weight = time, group_var = "cell")), so the two spellings give the same fit; the bare spatial() /
weight = forms are unchanged. to = validates against the family arm set,
is order-free (presence anchor regardless of order), and defaults to both
arms. Built on tulpa::tulpa_is_spatial_bar() / tulpa::tulpa_bar_field_specs()
(tulpa#93). The correlated | bar (#64) and the arm-specific single-arm ||
bar (#65) have their own entries below.cover() / occu_cover() now emit an informative message when a bare | / ||
formula bar groups by the same factor an areal term uses as its graph-node
group_var (#62): the bar is a random effect, not a spatial field, so the
message points to spatial(~ ... || cell, graph = adj) (or the two-term
icar(graph, group_var) + icar(graph, weight, group_var) form) for a spatial
field. RE bars still fit as random effects; the message is suppressible and
silent when the bar's factor is unrelated to any spatial term.y = (#66). cover() is the first such
family: tobs(cover.flat ~ predictors, data = dat, family = cover()) is
equivalent to the one-sided ~ predictors form with y = dat$cover.flat.
A new response property on obs_family() ("vector" vs the default
"matrix") declares eligibility; matrix / array / list response families
(occu(), abun(), the ms_* families, ...) keep y = and a two-sided
formula for those errors. Supplying the response on both the LHS and y =
errors.|) on the cover hurdle wires
the within-arm coefficient fields as a separable MCAR with a free 2x2 Sigma
(#64). spatial(~ 1 + time | cell, graph = adj) makes the intercept and slope
Besag fields correlated (free cross-covariance, integrated over the outer CCD
grid in log-Cholesky coords), then copies the whole correlated field onto the
positive arm with one estimated amplitude alpha. The fit reports sigma_mcar
(per-field SDs), rho_mcar (cross-correlation), and alpha_mcar. ||
(independent fields) is unchanged. nested_laplace engine, intrinsic-CAR
(icar) only; the simplified-Laplace correction over a correlated MCAR field is
not yet wired (records sla_status). Requires tulpa >= 0.0.28.||) spatial bar with a single-arm to = fits
an arm-specific separate latent field -- on that arm only, with its own
precision and NO cross-arm copy (#65). spatial(~ 1 + w || cell, graph = adj, to = "positive") (or "presence") places a non-copied areal field on the named
arm; two separate single-arm calls give independent per-arm fields with no
coupling between them. This is the free counterpart to the shared, presence-
anchored, copied to = c("presence", "positive") field (#61): there one field
is copied across arms with an estimated alpha, here each arm carries its own
field, with that field's precision integrated on the outer nested-Laplace grid.
No engine change -- a per-arm spatial_idx = 0 sentinel makes the other arm's
rows skip the block (the engine's l_b > 0 scatter guard). nested_laplace
engine; intrinsic-CAR / proper-CAR (icar / car / car_proper) only (the
bym2 phi+theta mix is deferred). The fit reports sigma_armspecific (per-field
SDs). Arm-specific fields do not compose with a shared field, a correlated |
bar, a weighted trend, or temporal() / re() in the same formula (those
couple the arms), and at most one field targets each arm. A single-arm
correlated | bar stays copy-only and errors.presence (the y > 0 Bernoulli
arm) and positive (the y | y > 0 arm) consistently across summary(),
print(), and the to = argument (formula label == output label),
replacing the earlier occurrence / cover headings.control$integration = "ccd" mode-finds the field hyperparameters
(tau, rho, sigma, range) and places a CCD at the marginal-likelihood
mode, scaled by the outer posterior covariance, reusing the engine's exported
CCD primitives (tulpa::ccd_grid() / ccd_to_theta() / ccd_weights()) and
surfacing the outer PSIS Pareto-k (fit$spatial_pareto_k). It declines to the
fixed tensor grid when the outer curvature is ill-conditioned (a weakly-
identified axis) and for a single positive hyperparameter, where the 1D grid is
already cheap. The default control$integration = "grid" keeps the fixed tensor
grid: each outer node is a full inner Laplace/EM solve, so the mode-find adds
cost without a node-count saving on these already-coarse grids, and the CCD is
most useful when a multi-axis hyperparameter posterior is well identified.
Wired across the areal-BFGS families (distance(), dyn_abun(), fp_occu()),
the community N-mixture Newton areal path, and the SPDE community path.tulpa (>= 0.0.25) and update the Remotes pin (the CCD primitives and
the mode-centred-CCD machinery ship in tulpa 0.0.25).control$trend is removed. A spatially-varying trend is model
structure, so it is now declared in the formula as a second, weighted areal
term on the same graph as the intercept field --
icar(graph, weight = time, group_var) (equivalently
spatial(graph, model = "icar", weight = time, group_var)). The weighted
term's contribution to each arm's predictor is weight_i * z[cell_i], coupled
onto the cover arm with its own scale (fit$alpha_trend / fit$sigma_trend)
integrated over the outer grid (control$alpha.grid.trend, defaulting to
control$alpha.grid). Requires method = "nested_laplace"; a leftover
control$trend now errors with a migration pointer (#59).tulpa (>= 0.0.18) and update the Remotes pin. The committed s2z
log-determinant guard test relies on the engine fix shipped in tulpa 0.0.18.removal(), distance(), dyn_abun(),
fp_occu(), abun(), and occu_cover() (#51), with the spatial fitters
unified onto a shared field-spec areal driver.removal(),
distance(), dyn_abun(), and fp_occu() (occupancy and detection arms)
(#51), plus NUTS sampling of a single intercept random effect on each arm
including abun() (#51).fitted() and predict()
(#53 part 1), a non-spatial Laplace path (#53 part 2), spatially-varying trend
fields (#53 part 3), and AGHQ debias of the community variance components
(#56).tulpaMesh moves from Suggests to Imports (the spde() term depends on
it for mesh construction). tulpaMesh::fem_matrices() is re-exported so the
mesh-assembly entry point is reachable directly from tulpaObs.control$aggregate.occ and
control$aggregate.pos, both default ON and byte-identical to the full
per-plot fit). Corrected the spatial-occupancy WAIC interpretation (the areal
field is not folded into the WAIC score), the ms_occu ranef() description
(it returns the per-species deviations), and the abun() backend list (laplace,
nested_laplace, nuts).control$aggregate.pos now defaults ON for the beta positive
arm (tulpaObs#49). The grouped sufficient-statistic collapse is byte-identical
to the full per-plot beta arm on the single-block, coupled-trend and
multi-block paths (test-cover-hurdle-aggregate-pos.R), with a multi-seed
parameter-recovery suite behind the both-arms-aggregated default
(test-cover-hurdle-aggregate-recovery.R); set control$aggregate.pos = FALSE
for the full per-plot arm. An explicit aggregate.pos = TRUE still errors on a
non-beta positive arm; the default leaves a non-beta arm untouched.tulpa (>= 0.0.16) and update the Remotes pin so a fresh install
resolves the grouped beta sufficient-statistic engine that aggregate.pos
(0.0.19) depends on; the prior >= 0.0.13 pin pre-dated that interface.fix(occu_cover): the joint-coupled nested-Laplace parameter-surface
covariance now carries the exact beta-hyperparameter cross-covariance and the
full hyper-hyper covariance via the law of total covariance (the
hyperparameters are the grid coordinates, so the within-cell term is zero and
the cross-covariance is purely between-grid). Previously the hyperparameter
block was diagonal, under-propagating the covariance of any derived quantity
mixing a regression coefficient with sigma / alpha. Predicted occupancy
and cover were unaffected (functions of the betas only). (tulpaObs#46)
feat(cover): control$aggregate.occ (exact Binomial sufficient-statistic
reduction of the cover-hurdle occurrence arm) now defaults to TRUE, backed
by a multi-seed parameter-recovery suite on simulated beta-trend data
(test-cover-hurdle-aggregate-recovery.R): the aggregated fit recovers truth
with nominal 95% CI coverage on both arms' coefficients and the beta
precision, and is byte-identical to the full per-plot fit. Set
aggregate.occ = FALSE for the full occurrence arm. (tulpaObs#48)
feat(cover): control$aggregate.pos (opt-in, default FALSE) adds the exact
grouped-beta sufficient-statistic reduction of the positive (cover) arm. Plots
sharing the positive design row and every per-observation latent component are
collapsed to one row carrying (n, sum log y, sum log(1 - y)); tulpa's
built-in beta likelihood reads those sufficient statistics, leaving the
log-likelihood, gradient and Fisher Hessian pointwise unchanged. The fit is
byte-identical to the full per-plot beta arm on the single-block and
coupled-trend paths, alone and combined with aggregate.occ. Beta only (a
lognormal positive arm errors with a pointer). (tulpaObs#49)
docs(ms_occu_cover): the community joint fit now flags that its community
VARIANCE components carry Laplace small-cluster attenuation (the community
MEANS do not), via print(), a machine-readable
fit$ms_community$var_attenuation marker, and ?ms_occu_cover, so the
reported between-species spread is not read as unbiased. (tulpaObs#47)
ms_abun() NUTS recovery assertions. The
fitted()$lambda dimension check used the wrong site count (40 instead of
the fixture's 30), and the per-species detection-coefficient recovery bar
(0.80) was tighter than the arm actually recovers -- the realized
correlation is 0.72 (abundance coefficients recover at 0.97, with zero
divergences), so the bar is now 0.65. No change to the sampler or the
model.R CMD check --as-cran ERRORs and WARNINGs (now only
the GitHub-ecosystem CRAN-incoming WARNING remains). The namespace now
imports every stats/utils/methods generic and function it uses
(nobs, simulate, update, model.matrix, glm, optim, ...), which
fixes the namespace-load failure that blocked the whole test suite; the two
vignettes that failed to build are fixed (stale summary() column names;
predict() returns a tobs_prediction, so the point estimate is $mean);
non-ASCII characters and lost-brace Rd math are removed; the tobs_test_*
goodness-of-fit helpers are documented; model.matrix-style globals and the
tulpaObs::: self-reference are cleaned up.-D_GLIBCXX_ASSERTIONS -g from
Makevars.win -- they triggered a spurious GCC 14 -Warray-bounds warning in
std::string and bloated the shared object; the precompiled header now also
rebuilds when Makevars.win changes.control$progress = FALSE. Requires tulpa (>= 0.0.13).z_s and
reconstructs the deviation per arm as b_{s,arm} = C_arm z_{s,arm} (C_arm
the log-Cholesky factor of Sigma_arm). The community covariance leaves the
b-prior (z ~ N(0, I)) and enters only the data term through b = C z,
breaking the centered b/Sigma funnel that saturated the NUTS treedepth.control$progress.file is set, the channel that survives a
detached run). The cover() / occu_cover() outer-grid paths flip progress
ON by default (no longer tied to verbose); set control$progress = FALSE
to silence the console bar.gcol33/[email protected] for the shared progress
reporter.gcol33/[email protected], which carries the fix for
the threaded outer-grid nested-Laplace data race behind gcol33/tulpaObs#42
(the coupled cover-arm dispersion was read lock-free across n.threads.outer
threads). The threaded beta cover() / occu_cover() EVA-scale fits are now
reproducible and crash-free; see the 0.0.14 entry and
dev_notes/issue42_root_cause.md.cover(positive = "beta") and
occu_cover(positive = "beta") fits with n.threads.outer > 1 could
intermittently crash (native memory corruption) or hang at MOTIVATE/EVA scale.
Root cause: in tulpa's threaded sparse joint outer-grid driver, the coupled
(cover) arm's per-cell dispersion -- the beta precision on the phi.grid.pos
axis -- was read lock-free from the shared arms during the inner Newton solve
while a concurrent grid cell's prep_at_grid rewrote it under the phi-sync
critical; every non-coupled arm already read a thread-local snapshot, but the
coupled arm did not. Fixed in tulpa (nested_laplace_joint_multi.{h,cpp}) by
snapshotting the coupled arms' dispersion under that critical and reading the
per-thread snapshot in the coupled scatter / log-lik. Verified: a 220-region
BYM2 beta cover fit is now identical serial vs n.threads.outer = 6 to ~1e-10
and finishes cleanly. The fix is in the tulpa dependency (root cause is there,
compiled into tulpaObs.dll via the header-only joint driver); see
dev_notes/issue42_root_cause.md.dyn_abun()
(gcol33/tulpaObs#37). Latent abundance evolves across primary seasons:
N_1 ~ Poisson(lambda); for t >= 2, N_t = Binomial(N_{t-1}, omega) + Poisson(gamma); observed via Binomial(N_t, p) over secondary visits. Unlike
the static abun(), the latent abundance sequence is not closed form -- it is
summed out by an exact HMM forward recursion over the abundance states, with
analytic gradients from forward-mode differentiation of the scaled forward
algorithm. Direct maximum-likelihood / Laplace fit (analytic-gradient BFGS,
observed-information covariance) and a NUTS path over the same marginal
(method = "nuts", WAIC / LOO from the draws). Four site-level arms: initial
abundance lambda (formula), detection p (detection), apparent survival
omega (omega_formula), recruitment gamma (gamma_formula). The response
is a 3D array [n_sites x max_visits x n_seasons]. simulate_dyn_abun(), full
S3. Recovery / 95% coverage / NUTS recovery + WAIC, plus a correctness anchor
(the C++ forward log-lik against an independent R forward recursion, exact to
1e-9), an FD gradient check, and a C++ <-> R oracle cross-check in
test-dyn_abun.R. Poisson initial abundance + constant recruitment this round;
negative binomial, season-varying dynamics, and spatial / RE not yet wired.
Per-site math src/dyn_abun_kernel.h; NUTS via the shared src/nuts_engine.h.fp_occu() (gcol33/tulpaObs#40).
The Miller et al. (2011) confirmed-detection design: each visit yields a state
y in {0, 1, 2} (no detection / ambiguous detection / certain detection), with
certain detections (state 2) only possible at occupied sites, which makes the
model robustly identifiable. Four site-level logit arms -- occupancy psi
(formula), true detection p11 (detection), false-positive p10
(fp_formula), certain-classification b (b_formula). The latent occupancy
marginalises in closed form (two states); the Laplace fit maximises the exact
marginal with an analytic gradient (BFGS) and an observed-information covariance
(the inverse of the negative finite-difference Jacobian of the analytic
gradient at the mode), and a NUTS path (method = "nuts") samples the same
marginal (WAIC / LOO from the draws). simulate_fp_occu(), full S3. Recovery /
95% coverage / false-positive-arm covariate / NUTS recovery + WAIC, a
correctness anchor (the two-state marginal against a direct computation, and the
certain-detection identity), an FD gradient check, and a C++ <-> R oracle
cross-check in test-fp_occu.R. Per-site math src/fp_occu_kernel.h; NUTS via
the shared src/nuts_engine.h driver.distance() (gcol33/tulpaObs#38). Latent
N ~ Poisson(lambda) (or negative binomial) in a covered region, observed
through a half-normal or hazard-rate detection function over distance bins. The
per-bin detected counts are multinomial over (bin 1, ..., bin B, undetected)
with cell probabilities pi_b = integral_bin g(x; sigma) f(x) dx (line- or
point-transect distance density f), integrated by Gauss-Legendre quadrature;
the latent N is summed out in closed form (truncation K_max). Direct
Laplace fit (method = "laplace", Poisson or negbin, half-normal or
hazard-rate with an estimated scalar shape) and a NUTS path over the same
marginal (method = "nuts", WAIC / LOO from the draws). The abundance formula
is tobs()'s formula; the site-level log sigma model is detection; the
response is an n_sites x n_bins count matrix; the bin edges and transect
geometry travel with the family (distance(cutpoints =, transect =)).
simulate_distance(), full S3. Recovery / 95% coverage / hazard-shape / NB
dispersion / point-transect / NUTS recovery + WAIC, a closed-form correctness
anchor (the Poisson distance marginal equals independent per-bin Poissons by
thinning), and an analytic-observed-information vs finite-difference-Hessian
check (the Louis curvature, including the second-derivative bin quadrature) in
test-distance.R.accumulate_count_moments / fill_nb_dispersion, src/nmix_kernel.h) for
the abundance / NB-dispersion math; the detection arm (site-level log sigma,
optional scalar hazard shape, bin integrals + first/second eta-derivatives by
quadrature) is src/distance_quad.h / src/distance_kernel.h. The tulpa NUTS
engine plumbing is factored into a shared driver (src/nuts_engine.h) now used
by both the count-marginal families and distance(). Byte-identical for the
existing families (full test-abun.R / test-abun-re.R / test-removal.R
suites unchanged).removal() (gcol33/tulpaObs#39). Sequential
depletion: latent N ~ Poisson(lambda) (or negative binomial) observed
through K ordered removal passes, where pass k removes
Binomial(N - sum_{l<k} y_l, p_k) of the individuals still present. The
depleting-binomial product is the multinomial-removal likelihood, and the
latent N is summed out in closed form (truncation K_max), so the fit is a
direct Laplace approximation (method = "laplace", Poisson or negbin) with a
NUTS path over the same marginal (method = "nuts", WAIC / LOO from the
draws). simulate_removal(), full S3 (fitted/predict/simulate/
residuals/coef/vcov/confint/logLik). Recovery / 95% coverage / NB
dispersion / NUTS recovery, plus a closed-form correctness anchor (the Poisson
removal marginal equals independent Poissons) in test-removal.R.accumulate_count_moments, fill_nb_dispersion
in src/nmix_kernel.h), and the non-spatial count-marginal Laplace driver and
NUTS machinery are now shared headers (src/marginal_count_laplace.h,
src/marginal_count_nuts.h) instantiated by both abun() and removal() --
one source of truth for the count-marginal fit. Byte-identical for abun()
(full test-abun.R / test-abun-re.R recovery suites unchanged).R CMD Rd2pdf / document().
simulate_cover() had a bare %*% in its generative-model \describe block
(the % opened an Rd comment and mangled the \item entries); the criteria
page (tobs_waic() / tobs_dic() / tobs_cpo()) linked to the unexported
.tobs_loglik_at_mean(), which has no Rd topic. Both now render as inline
code.fix(occu_cover): the beta latent cover spec honours the engine's Expected
(Fisher) curvature request, so the outer-grid corner cells converge instead of
returning a non-finite log_marginal (tulpaObs#35). The latent marginal's
observed information E_pi[sneg] - Var_pi(s) can go indefinite at extreme
hyperparameter cells (large sigma_u driving the beta mean toward 0/1), where
it stalled the inner Newton; under hessian = "fisher" (the beta default) the
inner step now uses the always-positive marginal Fisher information
E_pi[sum_j fisher_beta(eta + u)]. The reported log_marginal / SEs are
unchanged (the final mode-pass re-evaluates with observed curvature); the fix
recovers the ~20% of grid mass previously discarded at the corners and removes
the fragile NaN path for beta latent. The lognormal latent path is exactly
quadratic (observed == expected) and is unchanged. Ground-truth check (Expected
curvature == the brute-force Fisher marginal integral, PSD) in
test-occu-cover-latent.R.
fix(occu_cover): GOF (tobs_waic / tobs_dic / tobs_cpo) now scores the
cover term at the granularity the fit used (tulpaObs#34). The pointwise
log-likelihood evaluated the positive-arm density at every detected visit even
for a cover_aggregate = "mean" / "median" / "latent" fit, so it scored a
likelihood the model was never fit to: p_waic grew super-linearly in the
visits-per-site count and the LOO Pareto-k fraction went pathological. The
cover term is now evaluated once per occupancy unit at the aggregated cover
(mean / median) or via the per-unit cover-RE marginal (latent), matching the
fitter. The pointwise log-likelihood also reads the dispersion the spec held
fixed (sigma_pos / beta precision) instead of a bare unit default, so the
WAIC / DIC / LOO of every spatial occu_cover() fit is on the fitted
dispersion scale. Regression test in test-occu-cover-aggregate.R.
fix(occu_cover): predict() / WAIC grid sampling no longer fails when the
outer grid has non-converged cells. tulpa_posterior_draws() samples the grid
mixture by fit$weights; when a corner of the grid carries a non-finite
log_marginal (e.g. the beta latent spec's Gauss-Hermite arm not converging)
the engine's normalized weights could collapse to all-zero, leaving the sampler
with no positive-weight cell ("nothing to sample"). The joint-coupled fitter now
falls back to the same finite-cell softmax weights it uses for the reported
posterior moments, so predict() and WAIC stay consistent with the point
estimates. Covered by the beta latent predict test in
test-occu-cover-latent.R.
feat(occu_cover): latent cover-per-unit (cover_aggregate = "latent"). The
principled counterpart of mean / median aggregation: instead of collapsing a
unit's detected covers to one number, the cover arm carries a per-unit cover
random effect u_i ~ N(0, sigma_u^2) shared across the unit's detected visits
and integrates it out per unit, keeping every detected visit. The lognormal
arm integrates in closed form (compound-symmetry sufficient statistics); the
beta arm uses adaptive Gauss-Hermite quadrature (control$n.quad, default 15).
Because the cover predictor is unit-level the per-unit marginal is a scalar
function of one eta, so it slots into the one-row-per-unit layout with no
within-arm Hessian coupling. The within-unit dispersion is pre-fit from the
within-unit spread and held fixed; sigma_u is integrated on the outer grid
(reported as the phi_pos hyperparameter; control$sigma.u.grid). Same gates
as aggregation (cell-level positive design, shared-field spatial path,
joint_coupled engine). New stateful _latent cell-coupling specs
(FD-checked vs brute-force numerical integration in test-occu-cover-latent.R);
sigma_u + field + coefficient recovery in the same file.
feat(occu_cover): cell-aggregated cover (cover_aggregate, tulpaObs#33). On
the shared-field spatial path the cover arm carried one observation per
detected visit, so a cell with many detected plots drove the shared ICAR field
far more than the single occupancy observation for that cell and the
detection-corrected occupancy surface flattened. cover_aggregate = "mean"
(the new default on the spatial path) / "median" collapses the cover arm to
one response per occupancy unit (the mean / median cover over its detected
visits), so occupancy and cover inform the field with comparable weight;
"none" keeps the per-visit arm. Aggregation reads a cell-level positive
design from data; a visit-level positive covariate keeps the per-visit arm
(the bare default falls back, an explicit request errors). New _agg
cell-coupling specs evaluate the cover density once per cell (FD-checked in
test-occu-cover-coupling.R); recovery + gates in
test-occu-cover-aggregate.R.
fix(occu_cover): regularise the cover (pos) arm intercept by default on the
joint spatial path (tulpaObs#32). The cover intercept was left at the engine's
flat 1e-4 ridge while occupancy / detection carried the occu_priors()
defaults, so on a shared field it could float along the field-level confound to
a huge posterior SD (occupancy stayed tight) and blow up predict()'s
conditional cover via Jensen. It now carries the weakly-informative
cover_priors() intercept prior by default, like the load-bearing detection
prior; priors = FALSE / "none" still disables it.
feat(occu_aggregation_scan): suggest a spatial cell size and yearly clustering
that make a single-season occupancy model identifiable. Single-visit plot data
carries no within-unit replication, so psi and p are confounded until records
are pooled into (cell, year-block) buckets; the scan scores candidate (cell
size x year block) pairs by structural replication ("count") or the curvature
of the constant-model occupancy likelihood ("info", smallest eigenvalue /
posterior SEs of the 2x2 (logit psi, logit p) information), with a plot()
method.
feat(occu_cover, cover): forward the integration control key to the tulpa
joint backend (gcol33/tulpaObs#31). control$integration = "ccd" / "grid"
now reaches tulpa_nested_laplace_joint(), so the coupled cover-hurdle fit
can select CCD outer integration over the latent + phi hyperparameter axes.
feat(ms_dyn_occu, ms_int_occu): community (multispecies) dynamic and integrated
occupancy families. ms_dyn_occu() is the community version of dyn_occu()
(per-species first-season occupancy + detection coefficient random effects,
shared community-wide colonisation / extinction); ms_int_occu() is the
community version of int_occu() (multiple detection sources share one latent
occupancy state per species, per-species occupancy + per-source detection RE).
Both fit by a shared community Laplace-EM (R/community_em.R): the latent state
marginalizes in closed form (HMM forward for dynamic, two-state mixture for
integrated), the per-species coefficient deviations are integrated by a
joint-Newton mode-find with the RE blocks Schur-folded, and a closed-form
M-step updates the per-arm community covariance. Parameter-recovery + 95% CI
coverage tests (test-ms-dyn-occu.R, test-ms-int-occu.R).
fix(ms_occu): community single-season occupancy is now a correct community
model (gcol33/tulpaObs#30). The previous ms_occu() did not fit per-species
random effects in either engine: the Laplace route collapsed to a pooled GLM
over the stacked species rows (no species RE), and the NUTS route forced one
shared species intercept onto both the occupancy and detection arms. ms_occu()
now uses the shared community Laplace-EM with independent per-arm Gaussian
community covariances (the spOccupancy msPGOcc model), recovering the
per-species occupancy and detection coefficients its own simulate_ms_occu()
generates. ranef(), per-species fitted(), and tobs_richness() read the
per-species structure; recovery + coverage tests in test-ms-occu.R. The
legacy generic-engine community path (build_community_callbacks,
.tobs_build_community, the community model_type in src/occu_fit.cpp, and
the community entries in the Laplace / nested-Laplace switches) is removed.
ms_occu() is Laplace-only; a correct community NUTS / areal-spatial path
needs independent per-arm RE blocks in the sampler and is deferred.
feat(occu_multiscale_cover): three-level occupancy + cover hurdle family
(gcol33/tulpaObs#29). A cell-level occupancy gate (psi), a plot-level
availability gate (theta), per-visit detection (p) and the cover hurdle
(pos), for vegetation data where a site's "visits" are spatially distinct
plots aggregated into a (cell, period) rather than temporal revisits
(Nichols et al. 2008; Mordecai et al. 2011). Where occu_cover() conflates
within-cell prevalence into the detection arm (Kendall & White 2009), the
explicit middle level separates them. Both z (over cells) and a (over
plots) marginalize in closed form, so the joint marginal log-likelihood is
exact -- a new four-arm cell-coupling spec
(src/cell_coupling_occu_multiscale_cover.h, the nested two-state mixture)
drives tulpa::tulpa_nested_laplace_joint() over the shared (sigma, alpha)
field grid. Spatial-only (method = "nested_laplace"); a single shared areal
field. The no-detection occupancy-mixture math is now shared with
occu_cover() via nodet_mixture_block (src/occu_coupling_shared.h). Adds
occu_multiscale_cover() and simulate_occu_multiscale_cover(). Tests:
test-occu-multiscale-cover-coupling.R (FD-checks every closed-form
derivative, both families, branches A/B, Expected curvature, 2-level
reduction), test-occu-multiscale-cover-recovery.R (parameter recovery + CI
coverage + field shape). fitted() / predict() for the family are pending.
feat(ms_occu_cover): community (multispecies) joint occupancy-detection +
cover family, the community version of occu_cover(). Per-species coefficient
random effects with Gaussian community hyperpriors on all three arms
(occupancy psi, detection p, positive cover), so rare species borrow
strength from common ones through the shared community means and covariances.
The latent presence z integrates out per species-cell in closed form (the
same two-state mixture as occu_cover()); the per-species deviations are
integrated by a Laplace-EM (arrowhead joint Newton with the per-species RE
blocks Schur-folded, closed-form community-covariance M-step, Louis 1982
Schur-complement community-mean SEs). Beta + lognormal positive arms,
non-spatial Laplace only (method = "nested_laplace" errors: the per-group RE
on a shared coupled field needs upstream engine support). Adds
ms_occu_cover() and simulate_ms_occu_cover(). Tests: test-ms-occu-cover.R
(community-mean recovery + 15-seed CI coverage + per-species coefficient
recovery). Status "experimental"; NUTS / negbin / per-species dispersion RE /
AGHQ variance-component debias pending.
Requires tulpa (>= 0.0.7) and tulpaMesh (>= 0.1.2).
control$checkpoint = list(path =, resume =) is passed verbatim to tulpa::tulpa_nested_laplace_joint() from
both the occu_cover() joint-coupled path and the cover() hurdle path, so a
full-field fit killed by a reboot or OOM resumes from the last completed outer
grid cell instead of restarting. "checkpoint" is on the occu_cover + cover
control allowlist and documented as a Checkpoint/resume section on both
families. Tests: test-occu-cover-checkpoint.R.Requires tulpa (>= 0.0.6) and tulpaMesh (>= 0.1.2).
control$diagnose.k / control$k.samples to the
joint engine. The outer Pareto-k diagnostic re-solves the inner Laplace at
k.samples sampled hyperparameters; at field scale the draws stall at extreme
sigma and the diagnostic costs ~50x the grid integration while returning NA
for the multi-block ICAR config (gcol33/tulpa#51). control$diagnose.k = FALSE
skips it for a production fit; small fits keep the engine default.group_var on the icar() / bym2() term decouples the
occupancy units (sites) from the field nodes (cells), so many sites share one
areal field node. A site = cell x time-period then carries a per-site trend
weight, giving a detection-corrected occupancy trend on a shared cell field
(the field stays length n_cells while psi / p / cover run over n_sites).
Joint_coupled engine only; the v2/v3 escape hatches reject it. Recovery in
test-occu-cover-group-var.R.(sigma, alpha) copy convention -- the occupancy arm
scales block k by sigma, the positive arm by alpha * sigma
(gcol33/tulpaObs#15).cover() accepts the trend, alpha.grid, and
alpha.grid.trend control knobs for the trend-field integration.Requires tulpa (>= 0.0.5): the joint cover-hurdle path links against the
engine's cell_coupling.h / model_data.h (ABI 32).
perf(occu_cover): speed up the beta positive arm in the joint cover-hurdle
cell-coupling spec (gcol33/tulpa#46, lever 3). The per-observation
digamma/trigamma now use tulpa's portable, inlinable, OpenMP-safe
implementations instead of the R:: math-library calls; the score and
curvature share their digamma terms in a single pass; and the spec honours
the engine's CellDerivs::grad_only request, skipping the trigamma entirely
on a factor-reuse inner-Newton step. The joint_coupled engine now also
exposes control$n.threads.outer (the engine's parallel sparse outer grid)
and control$force.sparse. End-to-end invariance of the beta cover fit to
inner.refresh and n.threads.outer is covered in
tests/testthat/test-occu-cover-joint-reuse.R.
feat(occu_cover): joint occupancy-detection + cover-hurdle family
(occu_cover("lognormal") / occu_cover("beta"), gcol33/tulpa#32). A site's
occupancy/detection arm and a positive-cover arm (lognormal or beta) are fit
jointly, with the two linear predictors sharing a spatial field through a
cell-coupling spec. method = "nested_laplace" routes through the
joint_coupled engine by default; the engine integrates the shared field's
hyperparameters and the coupling coefficient on the outer grid. coef(),
vcov(), ranef(), fitted() and simulate() carry both arms.
feat(occu_cover): predict() for the joint fit (gcol33/tulpaObs#22). Samples
the grid-integrated joint latent via tulpa::tulpa_posterior_draws() and
marginalizes each derived quantity per draw, returning a tobs_prediction
with per-unit draw matrices and the change-column contract (delta_p,
delta_cover_cond, delta_cover_exp) with .lwr / .upr at the requested
level.
feat(ms_abun): per-species negative-binomial dispersion (gcol33/tulpaObs#14).
ms_abun(mixture = "negbin") gives each species its own overdispersion
log r_s ~ N(mu_log_r, sigma_log_r), partially pooled across the assemblage;
fit$ms_dispersion reports the per-species r_s with the community
mu_log_r / sigma_log_r, and ranef() carries a logr arm.
feat(spde): native SPDE (Matern-via-mesh) continuous spatial fields on the
occupancy state and detection arms and on the single-species / community
N-mixture abundance arm, with the (range, sigma) hyperparameters integrated on
the outer grid under a PC prior. Requires tulpaMesh for mesh construction.
feat(ms_abun): opt-in exact-Newton inner solver for the areal shared-field
community N-mixture (control$inner_solver = "newton", default "em",
gcol33/tulpaObs#12) -- an accuracy/validation alternative to the EM M-step on
the same outer field-hyperparameter grid; both return the same tobs_fit
shape and ms_community$optimizer records which ran.
feat(abun): grouped random effects on the single-species N-mixture
(gcol33/tulpaObs#13). abun() now accepts (1 | g) (and the slope /
uncorrelated / correlated variants the formula RE machinery already
understands) on either the abundance or the detection predictor of a
single-species fit. .tobs_fit_nmix_re() warm-starts the betas with the
no-RE Laplace fit and refines through .tobs_nmix_re_aghq(), a thin
make_site callback over nmix_site_marginal() driven by
tulpa::tulpa_re_aghq(). control$n.quad = 1 is the joint Laplace
(production); n.quad > 1 is the AGHQ debias of the small-cluster sigma
attenuation. Poisson and negbin (the global log_r is carried as the
trailing theta coordinate, jointly estimated with the betas). Gated: RE
combined with an areal spatial term, RE with visit-level detection
covariates, and RE shared across both arms (each errors with a pointer
rather than silently dropping the requested structure). coef(),
vcov(), and ranef() carry the RE component. tests/testthat/test-abun-re.R
covers the structural surface, the capability gates, and (under
NOT_CRAN=true) sigma recovery, fixed-effect CI coverage, and the NB+RE
path.
fix(build): clean-slate compile against tulpa restored. The
in-tree N-mixture move (commit c8b6912) left two casing mismatches
(using tulpaObs::NMix_* for functions defined as tulpaObs::nmix_*) in
src/nmix_spatial.cpp / src/nmix_spatial_bym2.cpp, and a broken include
guard in src/nmix_spatial_assemble.h (#ifndef TULPA_NMIX_SPATIAL_ASSEMBLE_H
vs #define TULPAOBS_NMIX_SPATIAL_ASSEMBLE_H) that re-included the file and
redefined its templates on the second pass. All three fixed; cold parallel
build is ~13 s (R CMD INSTALL -j8, rtools45). Requires tulpa >= 0.0.3.
refactor: the N-mixture observation model (single-species, areal-spatial,
community) has moved from the tulpa engine into tulpaObs as a
consumer-side LikelihoodSpec, restoring the principled
engine/model-package boundary. No user-facing API change: abun() and
ms_abun() continue to be the public surface. The native
NMixCommunityOracle is now a tulpaObs XPtr<tulpa::REGroupOracle> that
reaches the engine through <tulpa/aghq_oracle.h>; the community fit
drives it through tulpa::tulpa_re_aghq(). Bumps the tulpa pin to
v0.0.3.
feat(ms_abun): community / multispecies N-mixture (ms_abun(), the
spAbundance msNMix model) now fits under method = "laplace". Per-species
abundance and detection coefficients are random effects with Gaussian
community hyperpriors (beta_lambda_s ~ N(mu_lambda, Sigma_lambda),
beta_p_s ~ N(mu_p, Sigma_p)); the latent abundances integrate out in closed
form per species-site, and the fit is a C++ Laplace-EM (nmix_laplace_re(),
driving a native NMixCommunityOracle) -- per-species coefficient
modes, a closed-form covariance M-step, and fixed-effect SEs from the marginal
observed-information Schur complement (with the Var[N|y] rank-1 correction).
y is a 3D array [n_sites x max_visits x n_species] or a named list of
count matrices; pass species =. coef() returns the community means;
ranef() the per-species coefficient deviations; vcov() / confint() the
community-mean covariance; fitted() / simulate() the per-species
lambda / p / counts. simulate_ms_abun() +
tests/testthat/test-ms-abun.R cover community-mean recovery, 95% CI
coverage over 20 seeds, per-species coefficient recovery, and the S3 surface.
Poisson only for now (a global negative-binomial size and an areal-spatial
community field are upstream-pending). Requires tulpa >= 0.0.2.
feat(abun): abun(mixture = "negbin") now fits. The negative-binomial
abundance mixture (Var(N) = lambda + lambda^2 / r) is wired through to
tulpa's N-mixture kernel (mixture = "NB") on both the non-spatial Laplace
path and the areal-spatial nested-Laplace path (icar() / bym2() /
car_proper()). Non-spatially the log size log_r is estimated jointly with
the coefficients and reported with a standard error (the trailing vcov
coordinate); spatially the size r is integrated over the outer
hyperparameter grid and reported as a posterior mean / sd. coef() /
confint() / vcov() report the abundance and detection arms; the dispersion
is surfaced via fit$nmix_dispersion. simulate() and simulate_abun() draw
N ~ NegBin(mu, r) under the NB mixture. Matches unmarked::pcount(mixture = "NB"); test-abun.R adds NB point recovery, dispersion recovery, 95% CI
coverage, and a spatial NB fit. Requires tulpa >= 0.0.2.
refactor(re): the AGHQ quadrature engine moved to tulpa (tulpa::tulpa_re_aghq,
requires tulpa >= 0.0.2). R/re_aghq.R previously reimplemented the
Gauss-Hermite nodes, the log-Cholesky covariance parametrization, and the LKJ
penalty -- all of which tulpa already owns -- which duplicated generic
inference machinery across the package boundary. It is now a thin wrapper that
supplies only the family-specific occupancy / detection per-site marginal (the
make_site callback) and delegates the quadrature, mode-finding, joint
optimization, and marginal Hessian to the engine. Behaviour is unchanged
(the recovery numbers are identical); the engine is reusable by other families
and is recovery-tested in tulpa. A failed refine now warns (it kept the EM
result silently before).
feat(re): random effects on the detection predictor now fit under
method = "laplace" (gcol33/tulpaObs#11 follow-up). Previously a (1 | g) /
(x | g) term on the detection formula errored toward NUTS; the
variance-component EM (R/em_laplace_re.R) now partitions the RE terms into an
occupancy and a detection arm (.tobs_re_split_arms()), fits each arm's RE
block in its own M-step (occupancy as the M-inflated pseudo-binomial, detection
as a genuine weighted binomial at natural prior scale), and folds both RE
posterior modes back into psi and p at the E-step. The AGHQ debias
(R/re_aghq.R) is generalized to either arm: when the random effect enters the
detection predictor b moves p (not psi), so the per-group marginal uses
the binomial-in-p site derivatives (finite-difference verified,
dev_notes/probe_re_det_aghq_deriv.R). Detection-arm RE parameters are named
for the detection process (sigma_p<t>_*, re_p<t>_*). Measured recovery
(dev_notes/probe_re_det_*.R, 24 seeds, per-group n ~ 10): the raw nAGQ = 1
EM attenuates the detection sigma by ~70% (only occupied sites inform p);
the AGHQ refine restores it to within ~1% of truth (0.805 at truth 0.80) with
88-96% fixed-effect CI coverage. Supports the same iid-intercept /
uncorrelated-slope / correlated-slope forms as the occupancy arm. A single RE
shared across BOTH predictors, RE + visit-level detection, RE + spatial, and
non-single families still point to NUTS.
feat(re): adaptive Gauss-Hermite (AGHQ) debias of the random-effect variance
components under method = "laplace", on by default (R/re_aghq.R,
.tobs_re_aghq()). The variance-component EM integrates the RE block b by
Laplace (the glmer nAGQ = 1 regime), which attenuates sigma / the RE
correlation toward zero for binary occupancy at small per-group sample size.
After the EM converges, the per-group marginal int prod_i L_i(eta_i + Z_i b) N(b; 0, Sigma) db -- reusing the exact closed-form occupancy site marginal
(z integrated out) -- is refined by n.quad-point adaptive Gauss-Hermite
quadrature centred at the EM mode, and (beta, chol Sigma) are re-optimized on
it; the fixed-effect SEs are read from the exact-marginal Hessian. Measured
recovery (dev_notes/probe_re_aghq*.R): at per-group n = 8 the EM attenuates
sigma by ~18% (bias -0.16 at truth 0.9), AGHQ cuts that to ~4% (matching
NUTS); correlated-slope sigmas recover to ~1% on the seed average. Controls:
re.aghq (default TRUE; FALSE keeps the raw nAGQ = 1 EM), n.quad
(default 9; n.quad = 1 is the plain Laplace marginal), and re.lkj (default
1.5). Applies to a single grouping factor with RE dimension <= 3 (the
recovery-tested forms); crossed / nested groupings fall back to the EM. A
weakly-identified correlated random slope's correlation is regularized off
the +-1 boundary by a default LKJ(re.lkj = 1.5) penalty on the block's
correlation matrix -- (eta - 1) log det R, maximized at independence,
leaving the marginal SDs untouched and O(1) against the O(n_groups)
likelihood; re.lkj = 1 disables it. On the recovery sim (truth rho = 0.61)
this removes every +-1 boundary hit while keeping rho near-unbiased (bias
-0.00 at per-group n = 12, +0.02 at n = 25), where unregularized ML
over-estimates rho and hits the boundary. The fit is then a MAP and the
reported SEs come from the penalized (posterior-precision) Hessian. NUTS
remains available for a full posterior treatment of the correlation. Recovery
tests in tests/testthat/test-re-laplace-recovery.R.
feat(nested): calibrated credible intervals for NA-response prediction.
predict(fit, type = "state") now returns psi_lower / psi_upper (95%)
alongside psi for single-season nested-Laplace fits. The intervals are
calibrated by refining the EM field with one exact-marginal pass
(.tobs_occu_state_marginal_fit()): integrating out the latent occupancy
state makes each site a Bernoulli on D = 1{>=1 detection} with mean
q * plogis(eta), where q = 1 - (1 - p)^J is the per-site detection
probability (a held-out site has no visits, so q = 0 and it is interpolated
by the field). This is fit through tulpa's generic family = "bernoulli" with
a per-observation probability scale det_prob = q, so the converged Hessian is
the marginal curvature and the per-cell predictive variance
(tulpa_nested_laplace()$fitted_eta_var) is calibrated directly. The per-row
eta posterior is a Gaussian mixture over the hyperparameter grid; psi is its
Gauss-Hermite mean and the interval is the mixture-CDF quantiles, both per the
marginalise-derived-quantities rule. The EM's M-inflated pseudo-binomial
(which weights the data ~M times the prior and whose unit-trial Hessian is the
complete-data information) is kept only for the mode and detection estimate;
reading variance or grid weights off it under-covers and collapses the
hyperparameter grid. Held-out coverage measures ~1.0 (calibrated, slightly
conservative) with cor ~0.88 / MAE ~0.11 on a 10x10 icar/bym2 grid; recovery
test in tests/testthat/test-nested-laplace-prediction.R. Older tulpa without
fitted_eta_var reports psi with NA interval columns. Other model types
keep the EM occ fit (their NA-response mapping is not yet wired).
feat(nested): nested Laplace generalised beyond single-season occupancy.
method = "nested_laplace" now fits int_occu(), ms_occu(), and
dyn_occu() as well as occu() -- the multi-block latent prior (spatial /
temporal / iid) is attached to the state ("occ") M-step block of the same
per-model-type callbacks the single-Laplace path uses, so there is one set of
callbacks rather than a build_*_callbacks_nested duplicate. The block's
per-row spatial_idx maps each state row to its site, so a community model
shares one site-level field across the species at a site, and integrated /
dynamic models carry a spatial / temporal field on the shared psi / psi1
predictor. The driver .tobs_em_nested_laplace() is now a thin wrapper over
.tobs_laplace(latent_prior = ). The registry (.tobs_family_methods) lists
nested_laplace for these families; calibrated recovery for the multi-block
engine remains the same follow-up tracked for single-season occupancy. Smoke
tests in tests/testthat/test-nested-laplace-families.R.
feat(nested): INLA-style NA-response prediction. A single-season occupancy
site whose detection history is all-missing (all NA) is held out of the
likelihood (n_trials = 0) but kept in the latent field, so its occupancy is
interpolated from neighbours (.tobs_heldout_sites()). predict(fit, type = "state") returns the marginalised per-site psi posterior -- the weighted mean
over the hyperparameter grid of plogis(eta), integrated rather than plugged
in at the mode -- with the held-out rows flagged. The E-step is now
field-aware (psi_i = plogis(X_i beta + field[idx(i)])) so the field tracks
the data instead of converging to the fixed-effect-only fixed point. The
marginalised psi is read from the engine's per-cell fitted linear predictor
(tulpa_nested_laplace()$fitted_eta), so it is exact for every latent prior
-- including bym2, whose predictor mixes structured and unstructured
components with hyperparameter-dependent scales that the engine reconstructs
with the right d_fac (older tulpa without fitted_eta falls back to
mode reconstruction, exact for the d_fac = 1 priors only). Calibrated
predictive intervals need the latent field's per-cell posterior variance and
are deferred to engine support -- only the marginalised mean is reported.
Recovery tests (icar + bym2) in
tests/testthat/test-nested-laplace-prediction.R.
fix(api): backend coverage is now enforced from a single source of truth.
.tobs_family_methods (in R/tobs.R) declares the methods each working
family supports, and tobs() validates the resolved method against it,
erroring with a pointer to the supported set. This removes the silent
nested_laplace -> single-Laplace downgrade that dyn_occu() / ms_occu() /
int_occu() / jsdm() previously hit (.map_engine() emitted only a
message() and then stamped fit$method <- "nested_laplace" on a fit that was
actually single-Laplace -- a provenance bug). The nested-Laplace engine is
wired only for single-season occupancy and the cover hurdle joint path; the
cover hurdle has no NUTS likelihood or EM-correction engine, so its
nuts / laplace_gibbs / laplace_mi rejections (previously scattered across
.dispatch_cover()) now flow through the same central check. Tests in
tests/testthat/test-family-method-coverage.R.
feat(formula): single-verb spatial(..., model = ...) umbrella over the
areal (icar, bym2, car, car_proper) and continuous (gp,
multiscale_gp, spde) spatial terms, mirroring temporal(time, type = ...)
and INLA's f(i, model = ...). spatial(graph = adj, model = "bym2") is
identical to bym2(graph = adj) and spatial(lon, lat, model = "spde") to
spde(lon, lat); the specific constructors still work. Dispatches through the
term registry (single source of truth), forwarding coords / graph = /
per-model arguments and id unchanged. Named arguments are validated against
the target constructor's formals, so a typo'd or wrong-model argument
(spatial(lon, lat, model = "gp", graph = adj)) errors at the call site
rather than being silently absorbed as a coordinate by the continuous terms'
....
feat(re): correlated random slopes (1 + x | g) now fit under the default
method = "laplace" (gcol33/tulpaObs#11), not only NUTS. The variance-
component EM (R/em_laplace_re.R) carries a full per-term RE covariance
Sigma (diagonal for (x || g), full for (1 + x | g)) and updates it with
Sigma_k <- mean_g [b_g b_g' + Cov(b_g | y)], consuming the per-group
posterior covariance from tulpa::tulpa_laplace(return_re_cov = TRUE). Because
the M-step fits the package's M-inflated pseudo-binomial (n = M, prior
Sigma/M), the natural-scale covariance is M times the block tulpa returns
(the inflated Hessian is M times the natural one). The estimated off-diagonal
is reported as a cor_<g>_<ci>_<cj> correlation alongside the sigma_ marginal
SDs. This removes the previous .validate_re_laplace() rejection that routed
(1 + x | g) to NUTS; the duplicate R-side Schur for the RE posterior variance
is dropped in favour of the engine's cov_blocks. The variance components
still carry the Laplace approximation's small-cluster bias for binary data
(the glmer nAGQ=1 regime, not Breslow-Clayton PQL) -- NUTS is the calibrated
route for the covariance. Recovery
test in tests/testthat/test-re-laplace-recovery.R (sigmas, correlation, and
BLUPs vs simulated truth).
feat(priors): cover_priors() adds opt-in Gaussian fixed-effect priors to the
cover hurdle, penalising both arms -- the occurrence (Bernoulli) intercept /
slopes and the positive-part (beta or lognormal) intercept / slopes. The
penalty threads through tulpa::tulpa_laplace() for the occurrence and
lognormal arms and through tulpa::tulpa_laplace_beta(beta_prior = ) for the
beta arm. Priors are opt-in (priors = NULL/FALSE fits unpenalised); an
sd = Inf component is a no-op. occu_priors() is rejected for cover() with
a pointer to cover_priors(), and the prior errors (no silent drop) when
combined with a spatial term or with method = "nested_laplace". Recovery
tests in tests/testthat/test-cover-priors.R.
break(api): abun(), ms_abun(), and dyn_abun() rename the latent-mixture
argument family = to mixture = ("poisson" / "negbin", after
unmarked::pcount()), removing the collision with the family-object concept
that tobs(family = ) already owns.
break(api): the exported tobs_priors() constructor (and its print method)
are removed -- it was wired to no fitting path. Use the family-group prior
builders occu_priors() (occupancy group) and cover_priors() (cover).
break(control): cover-hurdle control = list(...) keys are renamed from
underscores to the package's dotted convention (max_iter -> max.iter,
prior_sigma -> prior.sigma, prior_alpha -> prior.alpha,
sigma_pos_grid -> sigma.pos.grid, etc.), matching every other
user-facing control key.
fix(print): print.tobs_fit and print.tobs_family label the default route
"default method" instead of "engine", matching the method = argument users
actually type.
feat(re): formula random effects are now fit by the default engine = "laplace" instead of being silently dropped (gcol33/tulpaObs#11). A
variance-component EM (R/em_laplace_re.R) wraps tulpa's fixed-sigma Laplace
in the occupancy missing-data EM (feeding the random-effect mode back into
psi) and an EM/REML update for the variance components, fitting iid intercept
RE ((1 | g)) and uncorrelated random slopes ((x || g), (0 + x | g),
(1 + x || g)) on the occupancy predictor of a single-season model. The
variance-component estimates carry the Laplace approximation's small-cluster
bias for binary data (the glmer nAGQ=1 regime, not Breslow-Clayton PQL);
engine = "nuts" remains the calibrated route for the covariance.
Forms the deterministic path cannot fit -- correlated slopes (a
Cholesky-factored covariance, (1 + x | g)), random effects on the detection
predictor, RE combined with a spatial term, RE with visit-level detection
covariates, or RE on a non-single family -- now error with a pointer to
engine = "nuts" rather than being silently dropped.
feat(re): random-effect parameters are named and summarised. Under NUTS the
log_sigma / chol / z columns are labelled (replacing param[i]), and
ranef() reconstructs the per-group BLUPs on the natural scale
(b_{g,c} = sigma_c (L z_g)_c, marginalising over the draws). The
deterministic path reports the variance-component sigma and the Schur
posterior standard errors. ranef() is re-exported from tulpa, and
coef() now includes the visit-level detection coefficients (p_visit_*)
that summary() already reported.
feat(formula): lme4 bar syntax now supports the slope-only, multi-slope, and nested/crossed grouping forms (gcol33/tulpaObs#10):
(1 + x + z | g) stacks several random slopes into one correlated block
(re() accepts a covariate matrix; build_re_spec() derives n_coefs
from the slope-matrix width instead of hardcoding two).(0 + x | g) is a slope-only block with no group intercept (threaded
through tulpa's new re_has_intercept flag; requires tulpa >= ABI 22).(1 | g:h) groups over the interaction factor; (1 | g/h) expands to one
re() per implied grouping factor (g, g:h), distributing the LHS
slopes across each. || makes the block's covariance diagonal.
Recovery tests cover a 3x3 correlated intercept+2-slope block and a
slope-only block under NUTS (tests/testthat/test-re-bar-recovery.R).fix(re): the correlated-slope Cholesky factor is now sized k*(k-1)/2
(strictly-lower triangle) to match tulpa's tanh-Cholesky prior, replacing an
oversized k*(k+1)/2 that left k unconstrained parameters per term.
fix(data): tobs_data() output composes with tobs(visit_data = ...)
(gcol33/tulpaObs#8). A det.covs named list of [N, J] matrices is now
reshaped internally to the visit-level detection design instead of erroring
with object '<covariate>' not found, giving the visit-level-detection path
a clean public route. Recovery test in
tests/testthat/test-issue8-visit-detection.R.