lime/keras image classification: Input must be a vector, not a superpixel_list object.

[iacobus42]

Using this tutorial on the RStudio AI blog: https://blogs.rstudio.com/ai/posts/2018-03-09-lime-v04-the-kitten-picture-edition/

Everything runs fine until

explanation <- explain(img_path, explainer, n_labels = 2, n_features = 20)

Running either explanation or plot_image_explanation(explanation) returns the error message:

> plot_image_explanation(explanation)
Error: Input must be a vector, not a `superpixel_list` object.
Run `rlang::last_error()` to see where the error occurred.

Using rlang::last_error() and rlang::last_trace() suggest a possible conflict with the vctrs package.

> rlang::last_error()
<error/vctrs_error_scalar_type>
Input must be a vector, not a `superpixel_list` object.
Backtrace:
 1. lime::plot_image_explanation(explanation)
 7. vctrs:::stop_scalar_type(...)
 8. vctrs:::stop_vctrs(msg, "vctrs_error_scalar_type", actual = x)

> rlang::last_trace()
<error/vctrs_error_scalar_type>
Input must be a vector, not a `superpixel_list` object.
Backtrace:
    █
 1. ├─lime::plot_image_explanation(explanation)
 2. │ ├─...[]
 3. │ └─tibble:::`[.tbl_df`(...)
 4. │   └─tibble:::tbl_subset_row(xo, i = i, i_arg)
 5. │     └─base::lapply(unclass(x), vec_slice, i = i)
 6. │       └─vctrs:::FUN(X[[i]], ...)
 7. └─vctrs:::stop_scalar_type(...)
 8.   └─vctrs:::stop_vctrs(msg, "vctrs_error_scalar_type", actual = x)

str(explanation) looks complete and explain() doesn't cause any errors - only when attempting to plot/print/view the results.

My session info is below:

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Pop!_OS 20.04 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /home/jacob/.local/share/r-miniconda/envs/r-reticulate/lib/libmkl_rt.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] magick_2.3      lime_0.5.1      keras_2.2.5.0   forcats_0.5.0   stringr_1.4.0  
 [6] dplyr_0.8.5     purrr_0.3.3     readr_1.3.1     tidyr_1.0.2     tibble_3.0.0   
[11] ggplot2_3.3.0   tidyverse_1.3.0

loaded via a namespace (and not attached):
 [1] httr_1.4.1        jsonlite_1.6.1    foreach_1.5.0     modelr_0.1.6      shiny_1.4.0.2    
 [6] assertthat_0.2.1  cellranger_1.1.0  pillar_1.4.3      backports_1.1.5   lattice_0.20-40  
[11] glue_1.4.0        reticulate_1.15   digest_0.6.25     promises_1.1.0    rvest_0.3.5      
[16] colorspace_1.4-1  htmltools_0.4.0   httpuv_1.5.2      Matrix_1.2-18     pkgconfig_2.0.3  
[21] broom_0.5.5       haven_2.2.0       xtable_1.8-4      scales_1.1.0      whisker_0.4      
[26] later_1.0.0       gower_0.2.1       farver_2.0.3      generics_0.0.2    ellipsis_0.3.0   
[31] withr_2.1.2       cli_2.0.2         magrittr_1.5      crayon_1.3.4      readxl_1.3.1     
[36] mime_0.9          fs_1.3.2          fansi_0.4.1       nlme_3.1-144      xml2_1.2.5       
[41] tools_3.6.3       hms_0.5.3         lifecycle_0.2.0   munsell_0.5.0     reprex_0.3.0     
[46] glmnet_3.0-2      compiler_3.6.3    rlang_0.4.5       grid_3.6.3        iterators_1.0.12 
[51] rstudioapi_0.11   rappdirs_0.3.1    htmlwidgets_1.5.1 labeling_0.3      base64enc_0.1-3  
[56] gtable_0.3.0      codetools_0.2-16  curl_4.3          abind_1.4-5       DBI_1.1.0        
[61] R6_2.4.1          tfruns_1.4        lubridate_1.7.4   tensorflow_2.0.0  utf8_1.1.4       
[66] fastmap_1.0.1     zeallot_0.1.0     shinythemes_1.1.2 shape_1.4.4       stringi_1.4.6    
[71] Rcpp_1.0.4        vctrs_0.2.4       dbplyr_1.4.2      tidyselect_1.0.0 

[BavoDC]

I got the same error and was able to fix it using:

explanation <- as.data.frame(explanation)
plot_image_explanation(explanation)

Found out that the class of the object explanation was also tbl_df and tbl instead of just data.frame (solved it thanks to the example from the help-file of plot_image_explanation).

[yufree]

I got the same error and now the as.data.frame is not working for the tbl.

The example of plot_image_explanation is a data frame and it still work. However, the object from explain is tbl and will throw the following error for as.data.frame:

Error in `vec_size()`:
! `x` must be a vector, not a <superpixel_list> object.
---
Backtrace:
     ▆
  1. ├─base::as.data.frame(explanation)
  2. ├─tibble:::as.data.frame.tbl_df(explanation)
  3. │ ├─base::`[<-`(`*tmp*`, unname, value = `<named list>`)
  4. │ └─tibble:::`[<-.tbl_df`(`*tmp*`, unname, value = `<named list>`)
  5. │   └─tibble:::tbl_subassign(x, i, j, value, i_arg, j_arg, substitute(value))
  6. │     └─tibble:::vectbl_recycle_rhs_rows(...)
  7. │       ├─base::withCallingHandlers(...)
  8. │       └─vctrs::vec_recycle(value[[j]], nrow)
  9. └─vctrs:::stop_scalar_type(`<fn>`(`<sprpxl_l>`), "x", `<fn>`(vec_size()))
 10.   └─vctrs:::stop_vctrs(...)
 11.     └─rlang::abort(message, class = c(class, "vctrs_error"), ..., call = vctrs_error_call(call))

I wonder if anyone has insights on it? Thanks!

[zixuanzhang]

I figured that we can create a data frame from explanation object in the following code. We need to take care of the nested list object (eg. feature_value, data, prediction). So here is the workaround so that explanation_df is the input of plot_image_explanation() function:

explanation_df <- data.frame(model_type = explanation$model_type,
                             case = explanation$case,
                             label = explanation$label,
                             label_prob = explanation$label_prob,
                             model_r2 = explanation$model_r2,
                             model_intercept = explanation$model_intercept,
                             model_prediction = explanation$model_prediction,
                             feature = explanation$feature,
                             feature_value = NA,
                             feature_weight = explanation$feature_weight,
                             feature_desc = explanation$feature_desc,
                             data = NA,
                             prediction = NA,
                             stringsAsFactors = FALSE)

## this will coerce superpixel_list and bitmap_list to list, but still works for plotting
for (i in 1:nrow(explanation_df)){
  explanation_df$feature_value[i] <- explanation$feature_value[i]
  explanation_df$data[i] <- explanation$data[i]
  explanation_df$prediction[i] <- explanation$prediction[i]
}

plot_image_explanation(explanation_df)

[jtr13]

Thanks for this... I want to experiment with lime on images but got stuck installing tensorflow. Might anyone have a link to the explanation data frame to use with lime thus bypassing the modeling step? Much appreciated.