Statcan Dialogue Dataset

💻Code	📄Paper	🌐Homepage	🤗Huggingface	🐦Tweets	📺Video

The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents
Xing Han Lu, Siva Reddy, Harm de Vries
EACL 2023

This repository contains the code for our project. For more detailed information, instructions on requesting access, API user guide and documentation, please visit the website. You can find useful links above. A quickstart is provided below.

Quickstart

First, you need to request access (please go to the main webpage for more information).

Then, you can install our Python library to load the dataset:

pip install statcan-dialogue-dataset

Now, inside Python, you can use various functions to work with the data:

import statcan_dialogue_dataset as sdd

# Download the dataset (from dataverse)
sdd.download_dataverse("your_api_token")
# Alternatively, Download from huggingface
sdd.download_huggingface("your_api_token")

# Extract ZIP into ~/.statcan_dialogue_dataset/
sdd.extract_task_data_zip(remove_zip=True)

# Load task specific split for a specific language
train_ret = sdd.load_task_data(task="retrieval", split="train", lang="en")

# Load all task conversations
task_conversations = sdd.load_task_conversations()

# Load table metadata
table = sdd.metadata.Table.from_code(10100002)
# Get subjects or surveys
subjects = table.get_subjects()
surveys = table.get_surveys()

For more information, head to the Core API Reference page.

Reproduce paper results

The full eacl_code directory contains the code used to produce the results in the paper. To reproduce the results, start by reading the README.md file in the eacl_code directory.

StatCan resources

You may find the following external resources useful:

• Web Data Service User Guide
• Web Data Service API
• Developer homepage

Citation

If you use our dataset, please cite our conference paper:

@inproceedings{lu-etal-2023-statcan,
    title = "The {S}tat{C}an Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents",
    author = "Lu, Xing Han  and
      Reddy, Siva  and
      de Vries, Harm",
    booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2304.01412",
    pages = "2799--2829",
}