Skip to content

Commit

Permalink
Allow to skip the token output in the table
Browse files Browse the repository at this point in the history
  • Loading branch information
@thomaskrause
thomaskrause committed Oct 8, 2024
1 parent a4aca1f commit a228c60
Show file tree
Hide file tree
Showing 5 changed files with 42 additions and 35 deletions.
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- Export token in `table` exporter instead of ignoring them.
- Export token in `table` exporter instead of ignoring them. You can disable
exporting the token with the `skip_token` parameter.

## [0.17.0] - 2024-09-27

Expand Down
14 changes: 7 additions & 7 deletions src/exporter/snapshots/annatto__exporter__table__tests__core.snap
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ source: src/exporter/table.rs
expression: export.unwrap()
---
---- test_doc.csv:
norm::lemma id_norm::lemma norm::pos id_norm::pos dipl::sentence id_dipl::sentence norm::norm id_norm::norm dipl::dipl id_dipl::dipl
I exmaralda/test_doc#a_norm_T286-T0 PRON exmaralda/test_doc#a_norm_T286-T0 1 exmaralda/test_doc#a_dipl_T286-T4 I exmaralda/test_doc#t_norm_T286-T0 I'm exmaralda/test_doc#t_dipl_T286-T1
be exmaralda/test_doc#a_norm_T0-T1 VERB exmaralda/test_doc#a_norm_T0-T1 1 exmaralda/test_doc#a_dipl_T286-T4 am exmaralda/test_doc#t_norm_T0-T1 I'm exmaralda/test_doc#t_dipl_T286-T1
in exmaralda/test_doc#a_norm_T1-T2 ADP exmaralda/test_doc#a_norm_T1-T2 1 exmaralda/test_doc#a_dipl_T286-T4 in exmaralda/test_doc#t_norm_T1-T2 in exmaralda/test_doc#t_dipl_T1-T2
New York exmaralda/test_doc#a_norm_T2-T4 PRON exmaralda/test_doc#a_norm_T2-T4 1 exmaralda/test_doc#a_dipl_T286-T4 New York exmaralda/test_doc#t_norm_T2-T4 New exmaralda/test_doc#t_dipl_T2-T3
New York exmaralda/test_doc#a_norm_T2-T4 PRON exmaralda/test_doc#a_norm_T2-T4 1 exmaralda/test_doc#a_dipl_T286-T4 New York exmaralda/test_doc#t_norm_T2-T4 York exmaralda/test_doc#t_dipl_T3-T4

annis::tok id_annis::tok norm::lemma id_norm::lemma norm::pos id_norm::pos dipl::sentence id_dipl::sentence norm::norm id_norm::norm dipl::dipl id_dipl::dipl
exmaralda/test_doc#T286 I exmaralda/test_doc#a_norm_T286-T0 PRON exmaralda/test_doc#a_norm_T286-T0 1 exmaralda/test_doc#a_dipl_T286-T4 I exmaralda/test_doc#t_norm_T286-T0 I'm exmaralda/test_doc#t_dipl_T286-T1
exmaralda/test_doc#T0 be exmaralda/test_doc#a_norm_T0-T1 VERB exmaralda/test_doc#a_norm_T0-T1 1 exmaralda/test_doc#a_dipl_T286-T4 am exmaralda/test_doc#t_norm_T0-T1 I'm exmaralda/test_doc#t_dipl_T286-T1
exmaralda/test_doc#T1 in exmaralda/test_doc#a_norm_T1-T2 ADP exmaralda/test_doc#a_norm_T1-T2 1 exmaralda/test_doc#a_dipl_T286-T4 in exmaralda/test_doc#t_norm_T1-T2 in exmaralda/test_doc#t_dipl_T1-T2
exmaralda/test_doc#T2 New York exmaralda/test_doc#a_norm_T2-T4 PRON exmaralda/test_doc#a_norm_T2-T4 1 exmaralda/test_doc#a_dipl_T286-T4 New York exmaralda/test_doc#t_norm_T2-T4 New exmaralda/test_doc#t_dipl_T2-T3
exmaralda/test_doc#T3 New York exmaralda/test_doc#a_norm_T2-T4 PRON exmaralda/test_doc#a_norm_T2-T4 1 exmaralda/test_doc#a_dipl_T286-T4 New York exmaralda/test_doc#t_norm_T2-T4 York exmaralda/test_doc#t_dipl_T3-T4
exmaralda/test_doc#T4
25 changes: 12 additions & 13 deletions src/exporter/snapshots/annatto__exporter__table__tests__custom.snap
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,15 @@ source: src/exporter/table.rs
expression: export.unwrap()
---
---- website_example.csv:
'lemma';'id_lemma';'upos';'id_upos';'xpos';'id_xpos';'Case';'id_Case';'Number';'id_Number';'in_Pointing__dep_deprel';'in_Pointing__dep';'sent_id';'id_sent_id';'text';'id_text';'Person';'id_Person';'Tense';'id_Tense';'out_Pointing__dep_deprel';'out_Pointing__dep';'SpaceAfter';'id_SpaceAfter';'PronType';'id_PronType'
'they';'valid/website_example#t5_1';'PRON';'valid/website_example#t5_1';'PRP';'valid/website_example#t5_1';'Nom';'valid/website_example#t5_1';'Plur';'valid/website_example#t5_1';'nsubj';'valid/website_example#t5_68';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
'buy';'valid/website_example#t5_68';'VERB';'valid/website_example#t5_68';'VBP';'valid/website_example#t5_68';'n/a';'n/a';'Plur';'valid/website_example#t5_68';'n/a';'n/a';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'3';'valid/website_example#t5_68';'Pres';'valid/website_example#t5_68';'punct;obj;conj;nsubj';'valid/website_example#t9_66;valid/website_example#t8_76;valid/website_example#t7_32;valid/website_example#t5_1';'n/a';'n/a';'n/a';'n/a'
'and';'valid/website_example#t6_67';'CONJ';'valid/website_example#t6_67';'CC';'valid/website_example#t6_67';'n/a';'n/a';'n/a';'n/a';'cc';'valid/website_example#t7_32';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
'sell';'valid/website_example#t7_32';'VERB';'valid/website_example#t7_32';'VBP';'valid/website_example#t7_32';'n/a';'n/a';'Plur';'valid/website_example#t7_32';'conj';'valid/website_example#t5_68';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'3';'valid/website_example#t7_32';'Pres';'valid/website_example#t7_32';'cc';'valid/website_example#t6_67';'n/a';'n/a';'n/a';'n/a'
'book';'valid/website_example#t8_76';'NOUN';'valid/website_example#t8_76';'NNS';'valid/website_example#t8_76';'n/a';'n/a';'Plur';'valid/website_example#t8_76';'obj';'valid/website_example#t5_68';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'No';'valid/website_example#t8_76';'n/a';'n/a'
'.';'valid/website_example#t9_66';'PUNCT';'valid/website_example#t9_66';'.';'valid/website_example#t9_66';'n/a';'n/a';'n/a';'n/a';'punct';'valid/website_example#t5_68';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
'I';'valid/website_example#t14_1';'PRON';'valid/website_example#t14_1';'PRP';'valid/website_example#t14_1';'Nom';'valid/website_example#t14_1';'Sing';'valid/website_example#t14_1';'nsubj';'valid/website_example#t14_57';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'1';'valid/website_example#t14_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
'have';'valid/website_example#t14_57';'VERB';'valid/website_example#t14_57';'VBP';'valid/website_example#t14_57';'n/a';'n/a';'Sing';'valid/website_example#t14_57';'n/a';'n/a';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'1';'valid/website_example#t14_57';'Pres';'valid/website_example#t14_57';'punct;obj;nsubj';'valid/website_example#t17_54;valid/website_example#t16_38;valid/website_example#t14_1';'n/a';'n/a';'n/a';'n/a'
'no';'valid/website_example#t15_64';'DET';'valid/website_example#t15_64';'DT';'valid/website_example#t15_64';'n/a';'n/a';'n/a';'n/a';'det';'valid/website_example#t16_38';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'Neg';'valid/website_example#t15_64'
'clue';'valid/website_example#t16_38';'NOUN';'valid/website_example#t16_38';'NN';'valid/website_example#t16_38';'n/a';'n/a';'Sing';'valid/website_example#t16_38';'obj';'valid/website_example#t14_57';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'n/a';'n/a';'n/a';'n/a';'det';'valid/website_example#t15_64';'No';'valid/website_example#t16_38';'n/a';'n/a'
'.';'valid/website_example#t17_54';'PUNCT';'valid/website_example#t17_54';'.';'valid/website_example#t17_54';'n/a';'n/a';'n/a';'n/a';'punct';'valid/website_example#t14_57';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'

'annis::tok';'id_annis::tok';'lemma';'id_lemma';'upos';'id_upos';'xpos';'id_xpos';'Case';'id_Case';'Number';'id_Number';'in_Pointing__dep_deprel';'in_Pointing__dep';'sent_id';'id_sent_id';'text';'id_text';'Person';'id_Person';'Tense';'id_Tense';'out_Pointing__dep_deprel';'out_Pointing__dep';'SpaceAfter';'id_SpaceAfter';'PronType';'id_PronType'
'They';'valid/website_example#t5_1';'they';'valid/website_example#t5_1';'PRON';'valid/website_example#t5_1';'PRP';'valid/website_example#t5_1';'Nom';'valid/website_example#t5_1';'Plur';'valid/website_example#t5_1';'nsubj';'valid/website_example#t5_68';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
'buy';'valid/website_example#t5_68';'buy';'valid/website_example#t5_68';'VERB';'valid/website_example#t5_68';'VBP';'valid/website_example#t5_68';'n/a';'n/a';'Plur';'valid/website_example#t5_68';'n/a';'n/a';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'3';'valid/website_example#t5_68';'Pres';'valid/website_example#t5_68';'punct;obj;conj;nsubj';'valid/website_example#t9_66;valid/website_example#t8_76;valid/website_example#t7_32;valid/website_example#t5_1';'n/a';'n/a';'n/a';'n/a'
'and';'valid/website_example#t6_67';'and';'valid/website_example#t6_67';'CONJ';'valid/website_example#t6_67';'CC';'valid/website_example#t6_67';'n/a';'n/a';'n/a';'n/a';'cc';'valid/website_example#t7_32';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
'sell';'valid/website_example#t7_32';'sell';'valid/website_example#t7_32';'VERB';'valid/website_example#t7_32';'VBP';'valid/website_example#t7_32';'n/a';'n/a';'Plur';'valid/website_example#t7_32';'conj';'valid/website_example#t5_68';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'3';'valid/website_example#t7_32';'Pres';'valid/website_example#t7_32';'cc';'valid/website_example#t6_67';'n/a';'n/a';'n/a';'n/a'
'books';'valid/website_example#t8_76';'book';'valid/website_example#t8_76';'NOUN';'valid/website_example#t8_76';'NNS';'valid/website_example#t8_76';'n/a';'n/a';'Plur';'valid/website_example#t8_76';'obj';'valid/website_example#t5_68';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'No';'valid/website_example#t8_76';'n/a';'n/a'
'.';'valid/website_example#t9_66';'.';'valid/website_example#t9_66';'PUNCT';'valid/website_example#t9_66';'.';'valid/website_example#t9_66';'n/a';'n/a';'n/a';'n/a';'punct';'valid/website_example#t5_68';'1';'valid/website_example#s1_1';'They buy and sell books.';'valid/website_example#s1_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
'I';'valid/website_example#t14_1';'I';'valid/website_example#t14_1';'PRON';'valid/website_example#t14_1';'PRP';'valid/website_example#t14_1';'Nom';'valid/website_example#t14_1';'Sing';'valid/website_example#t14_1';'nsubj';'valid/website_example#t14_57';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'1';'valid/website_example#t14_1';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
'have';'valid/website_example#t14_57';'have';'valid/website_example#t14_57';'VERB';'valid/website_example#t14_57';'VBP';'valid/website_example#t14_57';'n/a';'n/a';'Sing';'valid/website_example#t14_57';'n/a';'n/a';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'1';'valid/website_example#t14_57';'Pres';'valid/website_example#t14_57';'punct;obj;nsubj';'valid/website_example#t17_54;valid/website_example#t16_38;valid/website_example#t14_1';'n/a';'n/a';'n/a';'n/a'
'no';'valid/website_example#t15_64';'no';'valid/website_example#t15_64';'DET';'valid/website_example#t15_64';'DT';'valid/website_example#t15_64';'n/a';'n/a';'n/a';'n/a';'det';'valid/website_example#t16_38';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'Neg';'valid/website_example#t15_64'
'clue';'valid/website_example#t16_38';'clue';'valid/website_example#t16_38';'NOUN';'valid/website_example#t16_38';'NN';'valid/website_example#t16_38';'n/a';'n/a';'Sing';'valid/website_example#t16_38';'obj';'valid/website_example#t14_57';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'n/a';'n/a';'n/a';'n/a';'det';'valid/website_example#t15_64';'No';'valid/website_example#t16_38';'n/a';'n/a'
'.';'valid/website_example#t17_54';'.';'valid/website_example#t17_54';'PUNCT';'valid/website_example#t17_54';'.';'valid/website_example#t17_54';'n/a';'n/a';'n/a';'n/a';'punct';'valid/website_example#t14_57';'2';'valid/website_example#s10_34';'I have no clue.';'valid/website_example#s10_34';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a';'n/a'
25 changes: 12 additions & 13 deletions src/exporter/snapshots/annatto__exporter__table__tests__edge_features.snap
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,15 @@ source: src/exporter/table.rs
expression: export.unwrap()
---
---- website_example.csv:
lemma id_lemma upos id_upos xpos id_xpos Case id_Case Number id_Number in_Pointing__dep_deprel in_Pointing__dep sent_id id_sent_id text id_text Person id_Person Tense id_Tense SpaceAfter id_SpaceAfter PronType id_PronType
they valid/website_example#t5_1 PRON valid/website_example#t5_1 PRP valid/website_example#t5_1 Nom valid/website_example#t5_1 Plur valid/website_example#t5_1 nsubj valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1
buy valid/website_example#t5_68 VERB valid/website_example#t5_68 VBP valid/website_example#t5_68 Plur valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1 3 valid/website_example#t5_68 Pres valid/website_example#t5_68
and valid/website_example#t6_67 CONJ valid/website_example#t6_67 CC valid/website_example#t6_67 cc valid/website_example#t7_32 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1
sell valid/website_example#t7_32 VERB valid/website_example#t7_32 VBP valid/website_example#t7_32 Plur valid/website_example#t7_32 conj valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1 3 valid/website_example#t7_32 Pres valid/website_example#t7_32
book valid/website_example#t8_76 NOUN valid/website_example#t8_76 NNS valid/website_example#t8_76 Plur valid/website_example#t8_76 obj valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1 No valid/website_example#t8_76
. valid/website_example#t9_66 PUNCT valid/website_example#t9_66 . valid/website_example#t9_66 punct valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1
I valid/website_example#t14_1 PRON valid/website_example#t14_1 PRP valid/website_example#t14_1 Nom valid/website_example#t14_1 Sing valid/website_example#t14_1 nsubj valid/website_example#t14_57 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34 1 valid/website_example#t14_1
have valid/website_example#t14_57 VERB valid/website_example#t14_57 VBP valid/website_example#t14_57 Sing valid/website_example#t14_57 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34 1 valid/website_example#t14_57 Pres valid/website_example#t14_57
no valid/website_example#t15_64 DET valid/website_example#t15_64 DT valid/website_example#t15_64 det valid/website_example#t16_38 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34 Neg valid/website_example#t15_64
clue valid/website_example#t16_38 NOUN valid/website_example#t16_38 NN valid/website_example#t16_38 Sing valid/website_example#t16_38 obj valid/website_example#t14_57 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34 No valid/website_example#t16_38
. valid/website_example#t17_54 PUNCT valid/website_example#t17_54 . valid/website_example#t17_54 punct valid/website_example#t14_57 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34

annis::tok id_annis::tok lemma id_lemma upos id_upos xpos id_xpos Case id_Case Number id_Number in_Pointing__dep_deprel in_Pointing__dep sent_id id_sent_id text id_text Person id_Person Tense id_Tense SpaceAfter id_SpaceAfter PronType id_PronType
They valid/website_example#t5_1 they valid/website_example#t5_1 PRON valid/website_example#t5_1 PRP valid/website_example#t5_1 Nom valid/website_example#t5_1 Plur valid/website_example#t5_1 nsubj valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1
buy valid/website_example#t5_68 buy valid/website_example#t5_68 VERB valid/website_example#t5_68 VBP valid/website_example#t5_68 Plur valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1 3 valid/website_example#t5_68 Pres valid/website_example#t5_68
and valid/website_example#t6_67 and valid/website_example#t6_67 CONJ valid/website_example#t6_67 CC valid/website_example#t6_67 cc valid/website_example#t7_32 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1
sell valid/website_example#t7_32 sell valid/website_example#t7_32 VERB valid/website_example#t7_32 VBP valid/website_example#t7_32 Plur valid/website_example#t7_32 conj valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1 3 valid/website_example#t7_32 Pres valid/website_example#t7_32
books valid/website_example#t8_76 book valid/website_example#t8_76 NOUN valid/website_example#t8_76 NNS valid/website_example#t8_76 Plur valid/website_example#t8_76 obj valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1 No valid/website_example#t8_76
. valid/website_example#t9_66 . valid/website_example#t9_66 PUNCT valid/website_example#t9_66 . valid/website_example#t9_66 punct valid/website_example#t5_68 1 valid/website_example#s1_1 They buy and sell books. valid/website_example#s1_1
I valid/website_example#t14_1 I valid/website_example#t14_1 PRON valid/website_example#t14_1 PRP valid/website_example#t14_1 Nom valid/website_example#t14_1 Sing valid/website_example#t14_1 nsubj valid/website_example#t14_57 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34 1 valid/website_example#t14_1
have valid/website_example#t14_57 have valid/website_example#t14_57 VERB valid/website_example#t14_57 VBP valid/website_example#t14_57 Sing valid/website_example#t14_57 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34 1 valid/website_example#t14_57 Pres valid/website_example#t14_57
no valid/website_example#t15_64 no valid/website_example#t15_64 DET valid/website_example#t15_64 DT valid/website_example#t15_64 det valid/website_example#t16_38 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34 Neg valid/website_example#t15_64
clue valid/website_example#t16_38 clue valid/website_example#t16_38 NOUN valid/website_example#t16_38 NN valid/website_example#t16_38 Sing valid/website_example#t16_38 obj valid/website_example#t14_57 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34 No valid/website_example#t16_38
. valid/website_example#t17_54 . valid/website_example#t17_54 PUNCT valid/website_example#t17_54 . valid/website_example#t17_54 punct valid/website_example#t14_57 2 valid/website_example#s10_34 I have no clue. valid/website_example#s10_34
Loading

0 comments on commit a228c60

Please sign in to comment.