Strange character converted as space

[Yichen-fqyd]

I have a sample pdf which has the special character very much looks like ti, but when extract data from json format, only space character is extracted at the corresponding position, I am wondering is there a way to distinguish the this special character? From my experiment, it is not drawing as well. Or any encoding information I could obtain to distinguish this pdf file from others?

Thanks a lot!

[JorjMcKie]

This is a so-called ligature. These are frequent character combinations which are encoded as one single glyph in fonts which support that.
As usual: not all font support them or all of them.
I know that MuPDF supports 6 ligatures: fi, fl, ffi, ffl, ff, st. I am afraid there is no support for this ligature "ti" in MuPDF.
But you can try to let MuPDF decompose ligatures: this is one of the option bits in the flags integer in text extractions: switch off "TEXT_PRESERVE_LIGATURES" in those flags, e.g. set flags=0 and see what happens.

[Yichen-fqyd]

Thank you for your reply. I tried with page.get_text(flags = 0), page.get_text(flags = 1), but seems both of them just returned space as the text.

[JorjMcKie]

If you want, you can send me the file - or even better just the page, and I will have a look.

[Yichen-fqyd]

I sent you via email due to privacy issues

[JorjMcKie]

@Yichen-fqyd - I saw you using JSON as output option. Any specific reason for this? You know that this actually is DICT plus conversion to json format using the json module?
There may a chance to better understand what was happenening when looking at what was there under DICT.
Also, if you use the new OCR feature of v1.19.0,, you may a chance to come over this problem.

[JorjMcKie]

@Yichen-fqyd - I had a look at the file: strange indeed!
Something weird has happened to what I knew as a very standard font: Calibri.
The only "solution" that I could figure out was using PyMuPDF's now inbuilt OCR feature since v1.19.0. Works like that:

# choose high enough dpi, e.g. dpi = 300
dpi = 300
zoom = 300 / 72  # zoom factor, 72 dpi is standard!
matrix =fitz.Matrix(zoom, zoom)
pix = page.get_pixmap(matrix=matrix)
pdfbytes = pix.pdfocr_tobytes()
ocr_pdf = fitz.open("pdf", pdfbytes)  # this is a 1-page PDF with OCR-ed text
ocr_page = ocr_pdf[0]  # read that page.
print(ocr_page.get_text())  # this contains the original text including all the "ti" correctly

[Yichen-fqyd]

Hi, I still have a further question, how to distinguish this pdf with this ti ligature from other pdfs, any other meta data I could obtain from this pdf?

Thank you so much,

[JorjMcKie]

No, unfortunately not. The problem is also not connected with (Py-) MuPDF: if you copy / paste text from it using Adobe Acrobat, you also get "leadership capabili es, highly organized". Same thing happens with PDF to text conversion using Nitro PDF read.
Also using XPDF's pdftotext program shows no success.
It must be some error buried in the creation of that subset of Calibri: the glyph looks good, but is generated by the space character.
In contrast to this:
The glyph that looks like the space character is generated by unicode 0xA0. So if you do: text = page.get_text().replace(" ", "ti"), everything looks fine, and even better if you execute instead text = page.get_text().replace(" ", "ti").replace(chr(0xa0), " ").

Why all this was done: no idea.