word segmentation in kant_aufklaerung_1784 GT PageXML

[bertsky]

Is it correct for assets/data/kant_aufklaerung_1784/page/kant_aufklaerung_1784_0020.xml to have punctuation characters as separate Word elements, even if they are written adjacent to other words (i.e. without whitespace)?

For example, in the first line tl_2 of region r_2_1, the word word_1478541900479_904 depicts a single semicolon token. But the line reads

gewiegelt worden; ſo ſchaͤdlich iſt es Vorurtheile zu

so it should be part of the second Word (i.e. worden;).

In principle, punctuation characters can occur

• at the start of a token (e.g. opening parenthesis or quotation marks etc.)
• at the end of a token (e.g. closing parenthesis or quotation marks, comma, colon, semicolon, sentence punctuation, hyphen etc.)
• as an isolated token (e.g. dash)

And of course, these might be combined, as in

„die Urſach jener intereſſanten Erſcheinung ſeyn ſollte?“ —

or

sind ganz entzückt über diesen glänzenden (!) Sieg der gerechten Sache

or

„Wir alten Republikaner“, sagt Guinard, „die seit 30 Jahren über die Republik wachten, wir werden doch einem Leon Faucher zur Seite noch ferner darüber wachen dürfen.“

So again (see #12) there is no way to reproduce the TextLine content from Word level annotation, except using coordinate heuristics.

Wouldn't it be better to have a strict rule to only segment at whitespace? (This is what segmentation using OCR-D/ocrd_tesserocr does now.)

@tboenig @kba @finkf

[wrznr]

@bertsky Yes, such a strict rule would be better. Since we do not have a designated space element as in ALTO and no wordStart attribute as in Abbyy XML, I would even say that this is the only possible way to unambiguously represent whitespace.

[wrznr]

@tboenig assets and our own GT has to be thoroughly checked wrt this issue.

[bertsky]

Excellent! Maybe having an automatic validation step (see #16) for consistency between PAGE element levels would quickly reveal such non-abiding annotations. (Or even allowed exceptions to it in corner cases like extra whitespace at paragraph column crossings, drop-capital regions, etc. So one might need a semi-automatic procedure to get there.)

[wrznr]

@tboenig Push.

[wrznr]

@tboenig Push.

[bertsky]

@tboenig Please reopen. I understand that this is allowed now by the spec under a "lax" consistency rule. In fact, all variants kant_aufklaerung_1784{,-jp2,-binarized,-page-block-line-word_glyph} fall under this case.

But can we have at least one representation here in the GT assets repo that does meet the strict rules for tokenisation? (We need this to be able to write automated module tests without having to download GT files each time ...)

[tboenig]

We made a new example this month.

[bertsky]

In the assets repo? Please publish!

[kba]

I think @tboenig meant we WILL provide a new example in the coming weeks.