Skip to content

developer36/Out-of-Scope-Intent-Detection-with-Supervised-Deep-Metric-Learning

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Out-of-Scope-Intent-Detection-with-Supervised-Deep-Metric-Learning

Introduction

Out-of-Scope(OOS) intents in dialogue systems is a challenging technique with practical applications. As for OOS intent detection, it not only ensures the accuracy of classifying known intents but detecting OOS intents is also crucial. Current related models are limited in learning decision boundaries or setting the threshold of confidence score, which all neglect that a well-formed intent representation is a key point. Meanwhile, text extractors trained by traditional cross- entropy loss merely focus on reducing the error rate of the class to which the sample is classified. In this paper, we propose an effective feature extraction method based on deep metric learning to construct the triplet network with prior knowledge. With the constructed triplet loss, mining hard samples, which refers to the far-apart intents between the same class and close intent representations among different classes, can further obtain discriminative intent representations. In addition, we also introduce adversarial training to make intent representations more robust. Experiments on three public datasets prove the effectiveness of our proposed method of learning discriminative intent representations. This repository provides the official PyTorch implementation of the research paper Out-of-Scope Intent Detection with Supervised Deep Metric Learning (Accepted by IJCNN2023).

Dependencies

We use anaconda to create python environment:

conda create --name python=3.6

Install all required libraries:

pip install -r requirements.txt

Run

sh run.sh

Experiment

Dataset

To demonstrate the improvements of the proposed approach, we perform our experiments on three public dialogue datasets.

  • Banking: Banking is a fine-grained intent detection dataset in the banking domain.
  • OOS: OOS is commonly used in text classification and open intent detection with a relative large number intent classes which is 150.
  • Snips: Snips is collected by personal voice assistant which contains 7 types of user intents across different domains. You can The detailed information is shown in Table.
Dataset Classes Training Validation Test
Snips 7 13,084 700 700
Banking 77 9,003 1,000 3080
OOS 150 15,000 2,000 6,000

Result

Macro-f1 score of unknown(open) intents and known intents with three different proportions of known classes on Banking, OOS, and Snips datasets.

Banking OOS Snips
KIR Methods Open Known Open Known Open Known
25% MSP 42.17 51.75 62.83 52.81 0.00 56.71
DOC 78.17 65.68 91.05 75.79 22.57 60.53
SEG 54.93 51.81 64.29 49.07 59.90 71.48
(K+1)-way 82.77 68.83 91.89 76.68 65.78 71.51
ADB 84.24 70.24 91.96 77.12 70.17 73.56
Ours 87.18 73.62 93.31 79.62 70.82 74.62
50% MSP 45.91 73.14 64.04 72.82 10.03 78.24
DOC 72.42 78.25 87.26 83.79 49.46 83.51
SEG 43.73 63.66 59.94 62.34 25.73 77.48
(K+1)-way 74.44 79.01 88.55 84.83 70.63 86.59
ADB 78.84 81.10 88.35 84.88 72.32 86.54
Ours 83.55 83.56 90.40 86.53 75.46 87.45
75% MSP 48.15 85.16 65.81 84.04 10.12 86.52
DOC 63.79 83.95 83.30 87.91 55.23 90.20
SEG 38.41 70.31 45.46 48.21 18.73 84.69
(K+1)-way 63.48 85.63 83.45 88.13 70.67 90.76
ADB 66.26 86.09 85.09 88.99 71.45 91.16
Ours 71.06 87.21 86.61 89.65 74.28 91.37

“Open” and “Known” denote the macro f1-score over open class and known classes respectively.

If you are insterested in this work, and want to use the codes or results in this repository, please star this repository and cite by:

@article{Out-Of-Scope Intent Detection with Supervised Deep Metric Learning, 
      title={Out-Of-Scope Intent Detection with Supervised Deep Metric Learning}
}

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published