Quarto Surveys

[jhelvy]

Description

I do a lot of online survey work, and I am not satisfied with any of the platforms out there. Very few have the capability of defining surveys that allow you to run code in the background (formr is the only one I know of), and I haven't seen any where you can define the survey content using markdown. I think Quarto is extremely well-positioned to fill a massive gap in online survey development.

I wrote up some ideas in this blog post, though it's a little out-dated now. In general, my thought is to use {shiny} functions to define forms, and Quarto to render everything together into a web page that displays the survey content. The only piece missing is some sort of backend that writes the form responses to a database. As a simple starting point, we could just use Google sheets to store responses. I've built shiny apps where I've used Google sheets as a hacky database, and it worked surprisingly well. The whole thing could just render to a shiny app actually, with Quarto just defining the UI. The goal would be for a user to use a simple syntax to define and preview their survey content, things like using --- to define a page break (similar to new slides in Quarto revealjs), etc. Below is a really simple demo of what I have in mind, which renders to this

Ultimately this may be a separate package that uses Quarto (like I wrote in my blog post, maybe a {surveydown} R package). But I wanted to see if this might spark other ideas on ways to possible build something like this before I spent more time on it.

---
name: "surveydown demo"
author: "John Paul Helveston"
format: html
self-contained: true
execute: 
  echo: false
---

```{r}
#| include: false

library(shiny)

# Basic question function

question <- function(
  name,
  type,
  required = FALSE,
  label    = "label",
  option   = NULL,
  dependence = NULL,
  dependence_value = NULL
) {
  
  output <- NULL

  if (type ==  "select") {

    output <- shiny::selectizeInput(
        inputId = name,
        label = label,
        choices = option,
        options = list(
          placeholder = '',
          onInitialize = I('function() { this.setValue(""); }')
        )
      )

  } else if (type == "mc") {

    output <- shiny::radioButtons(
        inputId = name,
        label = label,
        selected = base::character(0),
        choices = option
      )

  } else if (type == "text") {

    output <- shiny::textInput(
        inputId = name,
          label = label,
          placeholder = option
    )

  } 

  return(output)

}
```

# Welcome!

This is a demo survey! Click the button below to go to the next page.

---

This is a screener.

You must choose "Red" in the question below, otherwise you will be screened out (meaning you will be sent to the end of the survey)

```{r}
question(
  name     = 'color',
  type     = 'mc',
  required = TRUE,
  label    = "Do you want to take the red pill or the blue pill?",
  option   = c('Red', 'Blue')
)
```

---

Here are some more questions:

```{r}
question(
  name     = 'animal',
  type     = 'text',
  required = FALSE,
  label    = "What's your favorite animal?"
)

question(
  name     = 'education',
  type     = 'select',
  required = TRUE,
  label    = "What is the highest level of education you have attained?",
  option   = c(
    "Did not attend high school" = "hs_no",
    "Some high school"           = "hs_some",
    "High school graduate"       = "hs_grad",
    "Some college"               = "college_some",
    "College"                    = "college_grad",
    "Graduate Work"              = "grad",
    "Prefer not to say"          = "no_response"
  )
)
```

---

# Fine

Thank you for taking our survey!

[mcanouil]

You could use/leverage webr (https://github.com/r-wasm/webr/) and shinylive (https://github.com/posit-dev/shinylive) via their Quarto extension: https://github.com/coatless/quarto-webr and https://github.com/quarto-ext/shinylive.

Note that Observable could possibly be use for that purpose.
As well as some JavaScript library (e.g., https://surveyjs.io/) could be ported as a Quarto extension.

PS: what's the feature request exaclty? It seems to be more of a "Show and Tell" thing, isn't it?

[jimjam-slam]

Re. using Google Sheets as a survey back-end, I've done this before using an Observable notebook I wrote to poll a Google Sheet regularly (I wrote the backend functionality as an Observable notebook so that I could pull it into Quarto docs and share it easily!).

Here's the notebook: https://observablehq.com/@jimjamslam/live-google-sheet

And here's an example of a RevealJS presentation where I've used it to collect and chart a live poll:

• Code for the part of the talk where I collect and chart the results: https://github.com/jimjam-slam/talk-runapp-quarto-2022/blob/ecf6e2760fbaeef9c94ccdbf4eee35c34ff3f41d/index.qmd#L158C1-L214C4
• The slides: https://runapp2022.talks.jamesgoldie.dev/#/poll-what-tools-do-you-use

I put up a QR code to the Google Form (but you could write some code with the Google Sheets API to populate a sheet yourself if you preferred), users fill it in to populate the sheet, and the liveGoogleSheet() function returns a promise that updates periodically (I think I set it to every 15 seconds in this talk) with the results. Not something you'd necessarily want to use for, say, > 1000 responses, but it works great for talks 🙂

(not sure what's happened to the chart layout since I gave the talk, they were definitely full-size before 😅)

[mcanouil]

(not sure what's happened to the chart layout since I gave the talk, they were definitely full-size before 😅)

A bug introduced recently which should be fixed with the latest pre-release or the next one.

[andrewpbray]

@jhelvy Cool idea!

One thought would be to encapsulate the functionality that you describe, potentially using the tools suggested by @mcanouil , into a project extension. I'd gladly use project: type: survey =)

[jhelvy]

Great stuff folks!

@mcanouil Yeah I wasn't really sure how to classify this. It's not quite "Show and Tell" because I don't really have anything to show yet. I mostly just wanted to put the idea out there to get feedback from folks who may have done something similar. So it's not quite a feature request either, but more like a request for advice and ideas.

There are a lot of ways I think you could put something like this together. It sounds straightforward enough in premise, but actually implementing it smoothly would take a lot of careful thinking. One basic survey concept is flow control (e.g. using responses from certain questions to control skip forward logic, like screening someone out of the survey by skipping to the end. It's not so obvious how one might implement that. One thought was to have multiple "surveys" as separate .qmd files and then tie them together with a _quarto.yml file that controls things like sequencing and flow logic, as well as maybe other global options and perhaps the link up to a Google sheet to store responses. Kind of similar to a Quarto website in architecture. @andrewpbray I think this is in line with what you're saying about project extensions.

@jimjam-slam Amazing examples here! This gives me a little more hope for the idea of using a Google sheet as the database. Like, I'm sure we could come up with some super fancy db solution, but the users I have in mind here are people who want to focus their time on designing the survey experiment and as little time as possible setting up a db. They also want to be able to download their data as a csv when it's all done. That's it. Google sheets is something simple that everyone understands, plus most surveys aren't capturing thousands of responses, so data sizes tend to be rather management - certainly within the limits of a Google sheet.

I haven't worked much at all with Observable or javascript, so I'm not sure really where to start with doing something porting https://surveyjs.io/ as a Quarto extension. Though for sure I've seen surveyjs before and it looks absolutely beautiful. Anyone out there with javascript skills want to see how that might work?

Again, my goal would be to make the UI as simple as possible for a user to design their survey. Ideally they can insert code chunks to define different types of questions (like my demo above).

[jimjam-slam]

The main challenge with writing to a Google Sheet through anything other than a Google Form is that you need a key to do it—and if that key is sitting client-side (ie. in the web page itself), anyone can use it. So you could:

1. try to scope the key so that it only has access to this particular document (and you're not opening your entire Google Drive to the public), accepting that someone may use the key to tamper with the Sheet outside the parameters of your form... or read other people's responses! Or,
2. put the key on a server (eg. a Shiny server), and then the form responses go to the server and the server uploads them to Google Sheets.

Since there's already a {googlesheets4} R package for interacting with the Sheets API, a Shiny-based form might make more sense than trying to do it all with JS/OJS if you're willing to wear any server costs!

[jimjam-slam]

Also worth highlighting that the makers of Observable have actually published a survey library that could be pulled into a Quarto doc! I would assume it has the same limitations outlined above, but definitely worth looking into!

EDIT: this relies on you providing a CSV of survey responses—it's more about analysis and visualising survey data.

[jhelvy]

@jimjam-slam Yes, this is actually also why I'm not so sure the webr approach would work, because I can't think of a way to keep the response data private. Data security is absolutely critical for this kind of application as most users would want the responses to be private, so that is why I was originally thinking about a Shiny-based survey application. This also isn't specific to Google sheets either - ANY database backend used would need to be kept secure, and I don't know of a way to do that without having a server. Something like surveyjs could still be used for setting up the forms though.

The nice thing about using shiny is it's relatively easy to host on a server. You don't have to be a web mastermind to learn how to get it working, and for most people they could probably just use https://shinyapps.io/. Plus, if you wanted to include some more complex functionality (e.g. showing respondents how their responses compare to all others), you could do it by just writing some R code. I was envisioning a kind of shiny wrapper package, one where you start with a set of Quarto files that renders into a shiny survey app. The UI and server for the shiny app could be built from the quarto files. It's a different use case for shiny - nothing at all like a dashboard. It's a one-way application where the researcher is asking for data from the user, not the other way around.

[jhelvy]

Alright, with a little GPT help I'm starting to get a somewhat working prototype. Amazing that this is actually working, but it's far from complete. My biggest challenge right now is I'm trying to incorporate page breaks in shiny, and it's not beautiful. Just like in revealjs slides, I want to use the --- symbol as a page break. My general strategy is to first parse the qmd file for --- symbols to split the content into pages, then try to render the content in each "page", then conditionally display that content using a little javascript show / hide logic. It should work, but I feel like I'm re-inventing the wheel here. How does revealjs break content up into multiple slides? What I'm trying to do must be quite similar, so it'd be great if I could use existing functionalities in Quarto to handle this. If anyone has thoughts on this please let me know!

[mcanouil]

It depends what you want to do exactly, you can use the fact that it appears in AST as HorizontalRule and do whatever you want in a LUA Filter.
This way you let Quarto/Pandoc deal with the parsing step.

[jhelvy]

Okay I'll try playing with a LUA filter. I've been able to otherwise get a working prototype if I ignore the need for multiple pages. With that demo I'm first rendering the quarto doc to html, then I have some code to build a shiny app that basically takes the rendered html as the UI. I'm not sure how such a setup would work with the LUA filter - do you think it could work the same way? In the end, I think all I really need is a way to insert some sort of section div or something so that the javascript can know what to display with show and hide. I suppose I could just insert some html to mark each new section (page), but I want the user experience to be more seemless by being able to just insert --- symbols.

[mcanouil]

With a Lua filter, you will basically intercept "HorizontalRule" and replace it with whatever you want, such as raw HTML, then Pandoc will read and render that. Since "HorizontalRule" will no longer be "HorizontalRule", Pandoc won't render it.

Now this would also mean that "HorizontalRule" from --- is no longer possible in your template.

[jhelvy]

Okay I'm getting the lua filter to work now! Amazing - I have zero experience with lua filters, but I just worked with ChatGPT to explain how they work in detail, and then we co-wrote a filter to replace the --- with section tags. I'm now getting the javascript working to show and hide the sections based on the tags...we live in a wonderful time for software development! Once I get a full working demo I'll post it here.

[mrossde]

Hello, thats a really great idea to have survey function in quarto. im keen on hearing how your progress is. Regards, Mario

[jhelvy]

Hi everyone, writing with some updates on this idea.

First, I managed to come up with a working version of this general idea in R here. If you look at the survey.qmd file, you'll see that I set it up as a Quarto qmd file with server: shiny. Works great! The pages are divided using fences, like ::: {#page-1}, ::: {#page-2}, etc. I created a simple function, sd_question() for defining a few types of question forms (it's just shiny inputs), and a sd_next() function for inserting a next button. Finally, I wrote a sd_server() function that handles writing the data and page navigation. For now it just writes responses to a local csv file, but eventually I'll try to connect this to a database.

So, the idea can work. And if I compile these functions into a {surveydown} package, it should be readily usable by others. It needs some love for sure though. Could use some CSS, and the database part is still empty. I actually was able to get a version to work with writing to googlesheets, but I'm not super happy with it. I'm just too worried about multiple users and conflicting writing, etc. I'm thinking of connecting to something like supabase since it's open source.

If anyone has thoughts / ideas on how to improve this little demo, please let me know! Eventually I'd like to create the package, but I'm waiting until I'm happy with the overall structure, and until I have a functioning database backend.

One other thing that looked super promising is the multi-page survey wizard python shiny app. It's very pro looking, but I'm not sure if something like this could be built out in python using Quarto with server: shiny. I've never worked with shiny for python, but if anyone has and thinks this could be done, I'd be curious to know what strategies you have.

[jhelvy]

Okay, actually once I figured this out with manually-inserted page divs, I chatGPT-ed my way into a lua filter that just automatically inserts those divs on --- breaks, which achieves my original goal. It also inserts "next" buttons. Now the demo survey looks like this, which is much easier to write. And it works!

[gadenbuie]

The tricky bit with using --- is that you can't attach attributes to the horizontal rules, e.g. this won't work:

--- {#page-one}

Have you looked at using headings to determine pages? You could use level-2 headings for each page in the same way that quarto slides use level-2 headings to create new slides, e.g.

## Welcome

<!-- question: name -->
<!-- question: email -->

## Demographics

<!-- question: age -->
<!-- question: zip code -->

What's nice about this approach is that it ensures each page has a unique identifier (e.g. #welcome) and are contained together in a <section> element. You could also attach other attributes to each heading/page that you could use for control flow, etc.

## Welcome {on-success="next" on-failure="exit"}

What's probably more useful is being able to add logic that decides which page sd_next() activates next, in which case you'd use the page id to activate a specific page.

[jhelvy]

Good suggestion about level-2 headings. The only thing I don't like about that is if the text of those headings will also get displayed on the html page. I'd like the survey designer to be able to fully control what text is displayed without having to necessarily link that text to flow control logic.

I do like the idea of having a name for each page (rather than a number). That could be handy in some areas, like skip logic. But in my latest iteration of this in R, I've managed to get the --- page break working with appropriate next button logic to go on to the next page without having to name or number the pages or next buttons. I'd like for the user to not have to keep track of the exact page number because that can change over time and can introduce a bunch of magic numbers as the survey grows while being developed (similar to making a xaringan slide deck).

I've also enabled the user to define skip logic (control the behavior of the next button based on responses to questions) using a somewhat messy observe, combined with a set of user-provided skip logics, defined in a data frame (e.g. like this). So far this all seems to be working. It may not be the best architecture, but it at least demonstrates a proof of concept that someone can design a survey using --- to break pages and control skip logic based on input responses.

I'm now thinking if I can re-define some of this using an intermediary json object (per your suggestion offline).

[gadenbuie]

I'd like the survey designer to be able to fully control what text is displayed without having to necessarily link that text to flow control logic.

Good news! This can also be made to work with level-2 headings (with a slightly unusual syntax):

## {#welcome}

<!-- a page with id "welcome" but no official title -->

We use this syntax in learnr for a similar effect.

I'd like for the user to not have to keep track of the exact page number because that can change over time and can introduce a bunch of magic numbers...

Totally agree, I wouldn't recommend using page numbers either. Unique IDs are much better 😄

[jhelvy]

What if I used other headers inside the page, like this:

## {#welcome}

# title

More text

## {#page2}

Would I get a section with id "welcome" then another section with id "title"?

If I were to use this approach to denote pages, then ideally all of the content from the first ## {#welcome} up until ## {#page2} should be within the "welcome" section.

[jhelvy]

Okay that makes sense. That's actually what I sort of had in my original first working version of this, where I used fences to separate pages like ::: {#page-1}. Except in that version you had to use a consistent numbering (page-1, page-2, etc.). By adding a class like .survey-page I should be able to make it work with just id names for the page divs.

Still, I'm not yet sure if there is a need to attach attributes to page divs. So far I like the ease of being able to separate survey pages with just --- and using a lua filter to handle the page div and next button creation. From a user perspective, it's really simple, and they can just focus on the content and questions rather than a lot of structural elements. That UI is 100% inspired by xaringan, and I've always loved how easy it is to make slides that way. I suppose being able to add custom classes to certain pages to control css could be useful; with the lua filter approach, each page div is the same and has the same class.

[andrewpbray]

Another approach would be to designate the level-N heading as your page separator by default but the user could opt out of that by using a class like ## Section 2 {.not-a-page}.

[gadenbuie]

@jhelvy I also really like xaringan's syntax! It took some getting used to Quarto's approach, but now that I'm more comfortable with it there are some drawbacks to using ---.

1. First, you can't add attributes without inventing new syntax, which is what remark.js does

   ---
   class: center middle
   background-image: url(cat.jpg)

   You might not want attributes now, but I doubt you want to close that door entirely.

2. The --- were a source of constant confusion in xaringan -- granted because the parser was finicky and --- and --- were considered different tokens.

3. Using sections and fenced divs are the de facto standard in Quarto because they're so well supported by pandoc. There's value in using familiar syntax in the ecosystem and there's developer value in having access to the robust section/div structures in Lua.

What if you used --- for pauses within a section? For places where you don't want to reveal new questions until previous questions have been answered, but you don't need any more logic than just move forward. That would give you the ability to code very simple surveys with --- separators, but for places where continuation/skip logic is required you could use sections instead?

[jhelvy]

Actually I'm already starting to see where fences may win out. One thing I've been experimenting with is trying to extract some of the overall structure and save it as a json object -- things like the total number of pages, the question IDs, etc. Having that stored somewhere is immensely useful for writing the server as it can make some of the complex logic a lot easier (e.g. skip forward in the survey, conditionally display questions based on prior responses, etc.). But it's hard to extract all of that. My current strategy is trying to use a lua filter to grab it and save it as json, but the AST is hard to work with when you don't have fences.

Right now without fences, it's coded up where it just loops over everything when anything is trigger (like a next button), but with named objects it can be a lot more efficient and probably easier to control. So I'm going to experiment with fences more again.

One other thought is it would be really helpful to have fences for both pages and questions, something like this:

## {#welcome .sd-page}

::: {#has_pet .sd-question}

```{r}
sd_question(
  name = "has_pet",
  type = "mc",
  label = "Do you have a pet?",
  option = c(
    "No" = "no",
    "Yes" = "yes"
  )
)
```

:::

The annoying thing with this though is that I have to write the name twice - once in the fence, and again in the name argument to the sd_question() function. It's tolerable, but annoying. Not sure if there's a way to perhaps avoid this.

[jhelvy]

BTW, a working R version using fences for pages is now up on a new branch here.

[jhelvy]

Thanks everyone for the thoughts on this earlier this year. We worked over the summer on building this out, an I'm pretty happy with how it came out: https://surveydown.org/

We've pulled together Quarto, shiny, and supabase in an R package called {surveydown} that achieves a pretty nice platform for building reproducible, markdown-based surveys.