resampling fails with autotuner when trying to set automatically the fallback learner #1249
Labels
Comments
|
The code seems to work if one only sets library(mlr3)
library(mlr3learners)
library(mlr3extralearners)
library(mlr3proba)
library(mlr3spatiotempcv)
library(mlr3tuning)
#> Loading required package: paradox
library(progressr)
data("lsl", "study_mask", package = "spDataLarge")
task = mlr3spatiotempcv::as_task_classif_st(
mlr3::as_data_backend(lsl),
target = "lslpts",
id = "ecuador_lsl",
positive = "TRUE",
coordinate_names = c("x", "y"),
crs = "EPSG:32717",
coords_as_features = FALSE
)
lrn_ksvm = mlr3::lrn("classif.ksvm", predict_type = "prob", kernel = "rbfdot",
type = "C-svc")
lrn_ksvm$encapsulate(method = "try",
fallback = lrn("classif.featureless",
predict_type = "prob"))
perf_level = mlr3::rsmp("repeated_spcv_coords", folds = 2, repeats = 2)
# two spatially disjoint partitions
tune_level = mlr3::rsmp("spcv_coords", folds = 2)
# define the outer limits of the randomly selected hyperparameters
search_space = paradox::ps(
C = paradox::p_dbl(lower = -12, upper = 15, trafo = function(x) 2^x),
sigma = paradox::p_dbl(lower = -15, upper = 6, trafo = function(x) 2^x)
)
# use 50 randomly selected hyperparameters
terminator = mlr3tuning::trm("evals", n_evals = 50)
tuner = mlr3tuning::tnr("random_search")
at_ksvm = mlr3tuning::auto_tuner(
learner = lrn_ksvm,
resampling = tune_level,
measure = mlr3::msr("classif.auc"),
search_space = search_space,
terminator = terminator,
tuner = tuner
)
rr_spcv_svm = mlr3::resample(task = task,
learner = at_ksvm$learner,
# outer resampling (performance level)
resampling = perf_level,
store_models = FALSE,
encapsulate = "evaluate")
#> INFO [01:49:58.444] [mlr3] Applying learner 'classif.ksvm' on task 'ecuador_lsl' (iter 1/4)
#> INFO [01:49:58.869] [mlr3] Calling train method of fallback 'classif.featureless' on task 'ecuador_lsl' with 190 observations {learner: <LearnerClassifFeatureless/LearnerClassif/Learner/R6>}
#> INFO [01:49:58.883] [mlr3] Applying learner 'classif.ksvm' on task 'ecuador_lsl' (iter 2/4)
#> INFO [01:49:58.903] [mlr3] Calling train method of fallback 'classif.featureless' on task 'ecuador_lsl' with 160 observations {learner: <LearnerClassifFeatureless/LearnerClassif/Learner/R6>}
#> INFO [01:49:58.913] [mlr3] Applying learner 'classif.ksvm' on task 'ecuador_lsl' (iter 3/4)
#> INFO [01:49:58.921] [mlr3] Calling train method of fallback 'classif.featureless' on task 'ecuador_lsl' with 188 observations {learner: <LearnerClassifFeatureless/LearnerClassif/Learner/R6>}
#> INFO [01:49:58.930] [mlr3] Applying learner 'classif.ksvm' on task 'ecuador_lsl' (iter 4/4)
#> INFO [01:49:58.937] [mlr3] Calling train method of fallback 'classif.featureless' on task 'ecuador_lsl' with 162 observations {learner: <LearnerClassifFeatureless/LearnerClassif/Learner/R6>}Created on 2025-01-24 with reprex v2.1.1 Session infosessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.4.2 (2024-10-31)
#> os macOS Sequoia 15.2
#> system aarch64, darwin24.1.0
#> ui unknown
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz Europe/Berlin
#> date 2025-01-24
#> pandoc 3.2 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> backports 1.5.0 2024-05-23 [1] RSPM
#> bbotk 1.5.0 2024-12-17 [1] RSPM (R 4.4.2)
#> checkmate 2.3.2 2024-07-29 [1] RSPM (R 4.4.1)
#> cli 3.6.3 2024-06-21 [1] RSPM
#> codetools 0.2-20 2024-03-31 [2] CRAN (R 4.4.2)
#> colorspace 2.1-1 2024-07-26 [1] RSPM
#> crayon 1.5.3 2024-06-20 [1] CRAN (R 4.4.1)
#> data.table 1.16.4 2024-12-06 [1] RSPM (R 4.4.2)
#> dictionar6 0.1.3 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> digest 0.6.37 2024-08-19 [1] RSPM
#> distr6 1.8.4 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> dplyr 1.1.4 2023-11-17 [1] RSPM
#> evaluate 1.0.3 2025-01-10 [1] RSPM (R 4.4.2)
#> fastmap 1.2.0 2024-05-15 [1] RSPM
#> fs 1.6.5 2024-10-30 [1] RSPM
#> future 1.34.0 2024-07-29 [1] RSPM (R 4.4.1)
#> future.apply 1.11.3 2024-10-27 [1] RSPM (R 4.4.2)
#> generics 0.1.3 2022-07-05 [1] RSPM (R 4.4.1)
#> ggplot2 3.5.1 2024-04-23 [1] RSPM (R 4.4.1)
#> globals 0.16.3 2024-03-08 [1] RSPM (R 4.4.1)
#> glue 1.8.0 2024-09-30 [1] RSPM
#> gtable 0.3.6 2024-10-25 [1] RSPM (R 4.4.2)
#> htmltools 0.5.8.1 2024-04-04 [1] RSPM
#> kernlab 0.9-33 2024-08-13 [1] RSPM (R 4.4.1)
#> knitr 1.49 2024-11-08 [1] RSPM (R 4.4.2)
#> lattice 0.22-6 2024-03-20 [2] CRAN (R 4.4.2)
#> lgr 0.4.4 2022-09-05 [1] RSPM (R 4.4.1)
#> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0)
#> listenv 0.9.1 2024-01-29 [1] RSPM (R 4.4.1)
#> magrittr 2.0.3 2022-03-30 [1] RSPM
#> Matrix 1.7-1 2024-10-18 [2] CRAN (R 4.4.2)
#> mlr3 * 0.22.1.9000 2025-01-23 [1] Github (mlr-org/mlr3@54e6aaf)
#> mlr3extralearners * 0.9.0 2024-09-18 [1] Github (mlr-org/mlr3extralearners@1c297f9)
#> mlr3learners * 0.9.0 2024-11-23 [1] RSPM (R 4.4.2)
#> mlr3measures 1.0.0 2024-09-11 [1] RSPM (R 4.4.1)
#> mlr3misc 0.16.0 2024-11-28 [1] RSPM (R 4.4.2)
#> mlr3pipelines 0.7.1 2024-11-14 [1] RSPM (R 4.4.2)
#> mlr3proba * 0.6.8 2024-09-18 [1] https://mlr-org.r-universe.dev (R 4.4.1)
#> mlr3spatiotempcv * 2.3.2 2024-11-29 [1] RSPM (R 4.4.2)
#> mlr3tuning * 1.3.0 2024-12-17 [1] RSPM (R 4.4.2)
#> mlr3viz 0.10.1 2025-01-16 [1] RSPM (R 4.4.2)
#> munsell 0.5.1 2024-04-01 [1] RSPM (R 4.4.1)
#> ooplah 0.2.0 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> palmerpenguins 0.1.1 2022-08-15 [1] RSPM (R 4.4.1)
#> paradox * 1.0.1 2024-07-09 [1] RSPM (R 4.4.1)
#> parallelly 1.41.0 2024-12-18 [1] RSPM (R 4.4.2)
#> param6 0.2.4 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> pillar 1.10.1 2025-01-07 [1] RSPM (R 4.4.2)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.4.0)
#> progressr * 0.15.1 2024-11-22 [1] RSPM (R 4.4.2)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.4.0)
#> Rcpp 1.0.14 2025-01-12 [1] RSPM (R 4.4.2)
#> reprex 2.1.1 2024-07-06 [1] RSPM (R 4.4.1)
#> rlang 1.1.5 2025-01-17 [1] RSPM (R 4.4.2)
#> rmarkdown 2.29 2024-11-04 [1] RSPM (R 4.4.2)
#> rstudioapi 0.17.1 2024-10-22 [1] RSPM (R 4.4.2)
#> scales 1.3.0 2023-11-28 [1] RSPM (R 4.4.1)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.1)
#> set6 0.2.6 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> survival 3.8-3 2024-12-17 [2] RSPM (R 4.4.2)
#> tibble 3.2.1 2023-03-20 [1] RSPM
#> tidyselect 1.2.1 2024-03-11 [1] RSPM (R 4.4.1)
#> uuid 1.2-1 2024-07-29 [1] RSPM (R 4.4.1)
#> vctrs 0.6.5 2023-12-01 [1] RSPM
#> withr 3.0.2 2024-10-28 [1] RSPM (R 4.4.2)
#> xfun 0.50 2025-01-07 [1] RSPM (R 4.4.2)
#> yaml 2.3.10 2024-07-26 [1] RSPM
#>
#> [1] /opt/homebrew/lib/R/4.4/site-library
#> [2] /opt/homebrew/Cellar/r/4.4.2_2/lib/R/library
#>
#> ────────────────────────────────────────────────────────────────────────────── |
|
Hey Jannes, thanks for reporting this bug. Since mlr3 version 0.21.0 we actually want to enforce that encapsulation can only be set together with a fallback. I think nobody has remembered the at_ksvm$encapsulate(method = "evaluate", fallback = lrn("classif.featureless", predict_type = "prob"))The inner encapsulation protects the tuning process and the outer encapsulation protects the train and predict step of the final model. |
|
Second option would be to throw an error when no default fallback learner can be found. |
|
@be-marc thank you for reply!! I swear I tried exactly this last night and it didn't work... but now it works (excellent!) library(mlr3)
library(mlr3learners)
library(mlr3extralearners)
library(mlr3proba)
library(mlr3spatiotempcv)
library(mlr3tuning)
#> Loading required package: paradox
library(progressr)
data("lsl", "study_mask", package = "spDataLarge")
task = mlr3spatiotempcv::as_task_classif_st(
mlr3::as_data_backend(lsl),
target = "lslpts",
id = "ecuador_lsl",
positive = "TRUE",
coordinate_names = c("x", "y"),
crs = "EPSG:32717",
coords_as_features = FALSE
)
lrn_ksvm = mlr3::lrn("classif.ksvm", predict_type = "prob", kernel = "rbfdot",
type = "C-svc")
lrn_ksvm$encapsulate(method = "try",
fallback = lrn("classif.featureless",
predict_type = "prob"))
perf_level = mlr3::rsmp("repeated_spcv_coords", folds = 2, repeats = 2)
# two spatially disjoint partitions
tune_level = mlr3::rsmp("spcv_coords", folds = 2)
# define the outer limits of the randomly selected hyperparameters
search_space = paradox::ps(
C = paradox::p_dbl(lower = -12, upper = 15, trafo = function(x) 2^x),
sigma = paradox::p_dbl(lower = -15, upper = 6, trafo = function(x) 2^x)
)
# use 50 randomly selected hyperparameters
terminator = mlr3tuning::trm("evals", n_evals = 50)
tuner = mlr3tuning::tnr("random_search")
at_ksvm = mlr3tuning::auto_tuner(
learner = lrn_ksvm,
resampling = tune_level,
measure = mlr3::msr("classif.auc"),
search_space = search_space,
terminator = terminator,
tuner = tuner
)
at_ksvm$encapsulate(method = "try",
fallback = lrn("classif.featureless",
predict_type = "prob"))
rr_spcv_svm = mlr3::resample(task = task,
learner = at_ksvm$learner,
# outer resampling (performance level)
resampling = perf_level,
store_models = FALSE,
encapsulate = "evaluate")
#> INFO [12:18:31.788] [mlr3] Applying learner 'classif.ksvm' on task 'ecuador_lsl' (iter 1/4)
#> INFO [12:18:32.219] [mlr3] Calling train method of fallback 'classif.featureless' on task 'ecuador_lsl' with 165 observations {learner: <LearnerClassifFeatureless/LearnerClassif/Learner/R6>}
#> INFO [12:18:32.233] [mlr3] Applying learner 'classif.ksvm' on task 'ecuador_lsl' (iter 2/4)
#> INFO [12:18:32.253] [mlr3] Calling train method of fallback 'classif.featureless' on task 'ecuador_lsl' with 185 observations {learner: <LearnerClassifFeatureless/LearnerClassif/Learner/R6>}
#> INFO [12:18:32.263] [mlr3] Applying learner 'classif.ksvm' on task 'ecuador_lsl' (iter 3/4)
#> INFO [12:18:32.270] [mlr3] Calling train method of fallback 'classif.featureless' on task 'ecuador_lsl' with 162 observations {learner: <LearnerClassifFeatureless/LearnerClassif/Learner/R6>}
#> INFO [12:18:32.280] [mlr3] Applying learner 'classif.ksvm' on task 'ecuador_lsl' (iter 4/4)
#> INFO [12:18:32.288] [mlr3] Calling train method of fallback 'classif.featureless' on task 'ecuador_lsl' with 188 observations {learner: <LearnerClassifFeatureless/LearnerClassif/Learner/R6>}Created on 2025-01-24 with reprex v2.1.1 Session infosessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.4.2 (2024-10-31)
#> os macOS Sequoia 15.2
#> system aarch64, darwin24.1.0
#> ui unknown
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz Europe/Berlin
#> date 2025-01-24
#> pandoc 3.2 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> backports 1.5.0 2024-05-23 [1] RSPM
#> bbotk 1.5.0 2024-12-17 [1] RSPM (R 4.4.2)
#> checkmate 2.3.2 2024-07-29 [1] RSPM (R 4.4.1)
#> cli 3.6.3 2024-06-21 [1] RSPM
#> codetools 0.2-20 2024-03-31 [2] CRAN (R 4.4.2)
#> colorspace 2.1-1 2024-07-26 [1] RSPM
#> crayon 1.5.3 2024-06-20 [1] CRAN (R 4.4.1)
#> data.table 1.16.4 2024-12-06 [1] RSPM (R 4.4.2)
#> dictionar6 0.1.3 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> digest 0.6.37 2024-08-19 [1] RSPM
#> distr6 1.8.4 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> dplyr 1.1.4 2023-11-17 [1] RSPM
#> evaluate 1.0.3 2025-01-10 [1] RSPM (R 4.4.2)
#> fastmap 1.2.0 2024-05-15 [1] RSPM
#> fs 1.6.5 2024-10-30 [1] RSPM
#> future 1.34.0 2024-07-29 [1] RSPM (R 4.4.1)
#> future.apply 1.11.3 2024-10-27 [1] RSPM (R 4.4.2)
#> generics 0.1.3 2022-07-05 [1] RSPM (R 4.4.1)
#> ggplot2 3.5.1 2024-04-23 [1] RSPM (R 4.4.1)
#> globals 0.16.3 2024-03-08 [1] RSPM (R 4.4.1)
#> glue 1.8.0 2024-09-30 [1] RSPM
#> gtable 0.3.6 2024-10-25 [1] RSPM (R 4.4.2)
#> htmltools 0.5.8.1 2024-04-04 [1] RSPM
#> kernlab 0.9-33 2024-08-13 [1] RSPM (R 4.4.1)
#> knitr 1.49 2024-11-08 [1] RSPM (R 4.4.2)
#> lattice 0.22-6 2024-03-20 [2] CRAN (R 4.4.2)
#> lgr 0.4.4 2022-09-05 [1] RSPM (R 4.4.1)
#> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.4.0)
#> listenv 0.9.1 2024-01-29 [1] RSPM (R 4.4.1)
#> magrittr 2.0.3 2022-03-30 [1] RSPM
#> Matrix 1.7-1 2024-10-18 [2] CRAN (R 4.4.2)
#> mlr3 * 0.22.1.9000 2025-01-23 [1] Github (mlr-org/mlr3@54e6aaf)
#> mlr3extralearners * 0.9.0 2024-09-18 [1] Github (mlr-org/mlr3extralearners@1c297f9)
#> mlr3learners * 0.9.0 2024-11-23 [1] RSPM (R 4.4.2)
#> mlr3measures 1.0.0 2024-09-11 [1] RSPM (R 4.4.1)
#> mlr3misc 0.16.0 2024-11-28 [1] RSPM (R 4.4.2)
#> mlr3pipelines 0.7.1 2024-11-14 [1] RSPM (R 4.4.2)
#> mlr3proba * 0.6.8 2024-09-18 [1] https://mlr-org.r-universe.dev (R 4.4.1)
#> mlr3spatiotempcv * 2.3.2 2024-11-29 [1] RSPM (R 4.4.2)
#> mlr3tuning * 1.3.0 2024-12-17 [1] RSPM (R 4.4.2)
#> mlr3viz 0.10.1 2025-01-16 [1] RSPM (R 4.4.2)
#> munsell 0.5.1 2024-04-01 [1] RSPM (R 4.4.1)
#> ooplah 0.2.0 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> palmerpenguins 0.1.1 2022-08-15 [1] RSPM (R 4.4.1)
#> paradox * 1.0.1 2024-07-09 [1] RSPM (R 4.4.1)
#> parallelly 1.41.0 2024-12-18 [1] RSPM (R 4.4.2)
#> param6 0.2.4 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> pillar 1.10.1 2025-01-07 [1] RSPM (R 4.4.2)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.4.0)
#> progressr * 0.15.1 2024-11-22 [1] RSPM (R 4.4.2)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.4.0)
#> Rcpp 1.0.14 2025-01-12 [1] RSPM (R 4.4.2)
#> reprex 2.1.1 2024-07-06 [1] RSPM (R 4.4.1)
#> rlang 1.1.5 2025-01-17 [1] RSPM (R 4.4.2)
#> rmarkdown 2.29 2024-11-04 [1] RSPM (R 4.4.2)
#> rstudioapi 0.17.1 2024-10-22 [1] RSPM (R 4.4.2)
#> scales 1.3.0 2023-11-28 [1] RSPM (R 4.4.1)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.4.1)
#> set6 0.2.6 2024-09-18 [1] https://raphaels1.r-universe.dev (R 4.4.1)
#> survival 3.8-3 2024-12-17 [2] RSPM (R 4.4.2)
#> tibble 3.2.1 2023-03-20 [1] RSPM
#> tidyselect 1.2.1 2024-03-11 [1] RSPM (R 4.4.1)
#> uuid 1.2-1 2024-07-29 [1] RSPM (R 4.4.1)
#> vctrs 0.6.5 2023-12-01 [1] RSPM
#> withr 3.0.2 2024-10-28 [1] RSPM (R 4.4.2)
#> xfun 0.50 2025-01-07 [1] RSPM (R 4.4.2)
#> yaml 2.3.10 2024-07-26 [1] RSPM
#>
#> [1] /opt/homebrew/lib/R/4.4/site-library
#> [2] /opt/homebrew/Cellar/r/4.4.2_2/lib/R/library
#>
#> ────────────────────────────────────────────────────────────────────────────── |
|
OK, works on my end too. Didn't at first then I realized that you had to set learner = at_ksvm$learner, whereas the original code (https://r.geocompx.org/spatial-cv.html) set learner=at_ksvm. And it even works faster so as james_m indicated (geocompx/geocompr#1150) it was probably using the default with the initial fix. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The following code used to work but now fails with
#> Error in assert_learner(fallback, task_type = self$task_type): Assertion on 'fallback' failed: Must inherit from class 'Learner', but has class 'NULL'.even though we have set a fallback learner:Created on 2025-01-24 with reprex v2.1.1
Session info
This happens when mlr3::resample() calls
set_encapsulation()and the latter runs:However,
default_fallback(at_ksvm)is indeedNULL(which results in the observed error), I think what probably would be needed here isdefault_fallback(at_ksvm$learner)though this would not respect any predefined fallback learner by the user before.In any case, thank you so much for your work and thank you for your help!
The text was updated successfully, but these errors were encountered: