Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sync_table_2 regex does not properly escape a string replacement #80

Open
ChemiKyle opened this issue Aug 31, 2022 · 0 comments
Open

sync_table_2 regex does not properly escape a string replacement #80

ChemiKyle opened this issue Aug 31, 2022 · 0 comments

Comments

@ChemiKyle
Copy link
Contributor

ChemiKyle commented Aug 31, 2022

dplyr::rename_with(., ~ gsub(".x", "", .x), dplyr::ends_with(".x"))

The gsub call in this line treats . as a wildcard, not a literal . character.

To reproduce:

library(tidyverse)
library(redcapcustodian)

conn <- DBI::dbConnect(RSQLite::SQLite(), dbname = ":memory:")

table_names <- c(
  "redcap_projects"
)

for (table_name in table_names) {
  rcc.billing::create_and_load_test_table(
    table_name = table_name,
    conn = conn,
    load_test_data = T,
    is_sqllite = T
  )
}

redcap_projects <- tbl(conn, "redcap_projects") %>%
  collect()


updated_projects <- redcap_projects %>%
  mutate(status = 1)

# throws error
project_update_sync_activity <- sync_table_2(
  conn = rc_conn,
  table_name = "redcap_projects",
  source = updated_projects,
  source_pk = "project_id",
  target = redcap_projects,
  target_pk = "project_id"
)

# successful
project_update_sync_activity <- sync_table_2(
  conn = rc_conn,
  table_name = "redcap_projects",
  source = updated_projects %>%
    select(-contains("x")),
  source_pk = "project_id",
  target = redcap_projects,
  target_pk = "project_id"
)

Erroneous diagnosis:

dbx::dbxUpdate(
conn = conn,
table = table_name,
records = update_records,
where_cols = target_pk
)

While attempting to make an ETL, I've come across an odd limitation in sync_table_2 while investigating a failure to update redcap_projects. On my machine, update_records has a hard upper limit of 133 columns, at 134 I get Error: no such column:.

@ChemiKyle ChemiKyle changed the title dbx::dbxUpdate seems to have a column count limit sync_table_2 regex does not properly escape a string replacement Aug 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant