Skip to content

Latest commit

 

History

History
43 lines (33 loc) · 1.93 KB

README.md

File metadata and controls

43 lines (33 loc) · 1.93 KB

TUR2SQL: A Cross-Domain Turkish Dataset For Text-to-SQL

Results

If you submit papers on TUR2SQL, please let us know so we can merge your results into the results table.

Execution Accuracy (EX) and Logical Form Accuracy (LFA) of LLMs on the TUR2SQL development set

Model Logical Form Accuracy (LFA) Execution Accuracy (EX)
SQLNet 42.92 44.49
ChatGPT 93.61 98.79
T5 (With Schema Context) [1] 98.24 98.24
T5 (Without Schema Context) [1] 05.83 05.83
SQLCoder (With Schema Context) [1] 96.21 96.21
SQLCoder (Without Schema Context) [1] 97.13 97.13

Execution Accuracy (EX) and Logical Form Accuracy (LFA) of LLMs on the TUR2SQL test set

Model Logical Form Accuracy (LFA) Execution Accuracy (EX)
SQLNet 39.03 40.19
ChatGPT 86.72 98.38
T5 (With Schema Context) [1] 98.38 98.38
T5 (Without Schema Context) [1] 05.18 05.18
SQLCoder (With Schema Context) [1] 97.04 97.04
SQLCoder (Without Schema Context) [1] 96.48 96.48

References

[1] Demirkıran, Ferhat et al. "Enhancing Text-to-SQL Conversion in Turkish: An Analysis of LLMs with Schema Context." 2024 9th International Conference on Computer Science and Engineering (UBMK). IEEE, 2024.

TUR2SQL dataset consists of 10,809 pairs of natural language statements and their corresponding SQL queries. The generation of TUR2SQL is detailed in our UBMK-2023 paper:

@inproceedings{tur2sql,
    title     	= {TUR2SQL: A Cross-Domain Turkish Dataset For Text-to-SQL},
    author    	= {Kanburoğlu, Ali Buğra and Tek, F. Boray},
    booktitle   = {2023 8th International Conference on Computer Science and Engineering (UBMK)},
    year      	= {2023},
    pages       = {206-211},
    doi         = {10.1109/UBMK59864.2023.10286686}
}