subsampling LOO estimates with diff-est-srs-wor start

[avehtari]

Work in progress.

Tested with

• family="normal" and stats %in% c("elpd", "mlpd", "gmpd", "mse", "rmse")
• family="bernoulli" and stats %in% c("acc","pctcorr")
• family="poisson" usings trad and latent approaches
• tests pass expect some plot tests fail with svglite: ... Graphics API version mismatch

Notes

• n_loo matters only if validate_search = TRUE (with FALSE there is no speed advantage)
• in cv_varsel if nloo<n and fast PSIS-LOO result is not yet available, fast PSIS-LOO result is computed
• in cv_varsel if nloo<n, fast PSIS-LOO result is stored in slot $summaries_fast
• the subsampling indices are stored in slot $loo_inds
• the actual subsampling estimating happens in summary_funs.R get_stat()
• removed some NA checking and need to recheck if those need to put back
• improved quantiles for "mse" if the value is close to 0 (can't get negative lq)
• "auc" is not supported (complicated)

Next

• add support for incrementally increasing nloo?

tagging @n-kall

[avehtari]

I wanted to use R2, and as I had rewrote summary stats anyway, added R2 and made all mse, rmse, and R2 to use only normal approximation with as much shared computation as possible

With the added R2 support, this PR will close also #483

[n-kall]

I can take a look

[fweber144]

I have added some comments, but I'm not done with the review yet.

Besides, I think documentation needs to be updated (at least re-roxygenized, but also progressr should be mentioned), the vignettes perhaps as well, and I haven't run R CMD check (including the unit tests) yet. The NEWS file would also need to be updated.

[avehtari]

I checked 5 comments in which you asked me to check something, and all others are good, but I replied to one

[avehtari]

Noticed 6th comment which is also ok.

[fweber144]

I checked 5 comments in which you asked me to check something, and all others are good, but I replied to one

Sorry, I cannot find your reply. Do you remember to which comment it referred?:

• subsampling LOO estimates with diff-est-srs-wor start #496 (comment)
• subsampling LOO estimates with diff-est-srs-wor start #496 (comment)
• subsampling LOO estimates with diff-est-srs-wor start #496 (comment)
• subsampling LOO estimates with diff-est-srs-wor start #496 (comment)
• subsampling LOO estimates with diff-est-srs-wor start #496 (comment)
• subsampling LOO estimates with diff-est-srs-wor start #496 (comment)

[avehtari]

A comment for commit de1cf4d (you may need to click a few times "Show more")

[fweber144]

So that would be #496 (comment), but I cannot see a reply from you there :/ Am I looking at the wrong place?

[avehtari]

I clicked the link in your last comment, and I can see the discussion with my comment. I'm copying my comment here

I would drop "exact", or at least drop quotes, as quotes imply it's not really exact which is confusing

[fweber144]

That's strange, I still cannot see any comment from you when clicking that link. But thanks for copying, I just pushed commit 7e61368 which drops the quotes.