Pdf size tripled after applying redactions

[cconur]

Hi all,

Why the output pdf file size after removing specific text in the header of every page from a very large pdf file by using the add_redact_annotation method gets tripled (2,7GB) respect the original file size (900GB)?

This is the code I have used:

`import fitz
doc = fitz.open("input.pdf")

t = "Text to be deleted"

for page in doc:

rl = page.search_for(t, quads = True)
page.add_redact_annot(rl)
page.apply_redactions()

doc.save("OUTPUT.pdf")`

Is it possible to refine this to get an output file with the same file size than the original one?

Many thanks in advance.

Regards

[JorjMcKie]

This is caused primarily by the careless (😉) use of the .save method:
Without any garbage collection choice and no deflating option, the following undesired effects will result:

• Objects rendered unused by the redaction process will be still present in the file (as so-called "ghost" objects). This also is a data protection issue: presumably / possibly among your motivation to use redactions at all. Among those ghost objects are original images and original /Contents objects of the page, still containing the removed text. Malignant code may still be able to extract ghost objects, thus circumventing the redaction intent.
• Images overlapping any of the redaction rectangles will be replaced by versions that have holes punched into them. These new image versions are PNG format images which are not deflated by the redaction process.

So to be sure to get the maximum possible file size redaction and to ensure ghost object removal, use doc.save("output.pdf", garbage=4, deflate=True).

[cconur]

Hi Jorj! you are great! the new output file size is 500MB! half the original! Many thanks for you effective, quick and kind answer.
Taking advantage of your courtesy, in order to extract ALL the text from my pdf, I have coded it this way, willing to use the tesseract library:
`
import fitz
from PIL import Image
import pytesseract

input_file = 'output.pdf'
full_text = ""
zoom = 1.2

with fitz.open(input_file) as doc:
mat = fitz.Matrix(zoom, zoom)
for page in doc:
pix = page.get_pixmap(matrix=mat)
output = f'images/{page.number}.jpg'
pix.save(output)
res = str(pytesseract.image_to_string(Image.open(output)))
full_text += res

full_text = full_text.splitlines()
print(full_text) + str(pageNo) + '.jpg'`

Do you find this code ok? anything to improve? I'll test it as soon as I have the pytesseract library installed.
Many thanks in advance.

Regards!

[JorjMcKie]

Hi @cconur - I'm glad you took my comment the right way 😎.

To your new question:
Any specific reason why you don't use PyMuPDF's direct API to Tesseract? That would simplify your code significantly and also avoid intermediate pixmaps saved to somewhere.
In addition, the DPI parameter is much handier than a zoom matrix (although the DPI value is implemented via a matrix nternally).

full_text = ""
for page in doc:
    tp = get_textpage_ocr(flags=3, language='eng', dpi=150, full=True)
    full_text += page.get_text(textpage=tp)

[cconur]

Definitely looks much better your proposal, so, the complete code is just this? No libraries needed apart from fitz?

import fitz

doc = 'HI-WAY-clean.pdf'
full_text = ""
for page in doc:
    tp = get_textpage_ocr(flags=3, language='eng', dpi=150, full=True)
    full_text += page.get_text(textpage=tp)

Thanks again!

[JorjMcKie]

correct - but remember to look at the installation requirements again: you need to store an environment variable TESSDATA_PREFIX, which contains the path to the language path of your Tesseract installation.

[cconur]

Hi Jorj,
I have added the environment variable as you pointed:
export TESSDATA_PREFIX=/opt/homebrew/Cellar/tesseract/5.3.1/share/tessdata

And when running the script:

import fitz

doc = 'HI-WAY-clean.pdf'
full_text = ""
for page in doc:
    tp = get_textpage_ocr(flags=3, language='eng', dpi=150, full=True)
    full_text += page.get_text(textpage=tp)

I get this error:

Traceback (most recent call last):
File "/Users/user/python/test13.py", line 6, in
tp = get_textpage_ocr(flags=3, language='eng', dpi=150, full=True)
NameError: name 'get_textpage_ocr' is not defined

Any clue? sorry for my complete incompetency! but I have the best learning attitude ;)

[JorjMcKie]

my fault: should have been page.get_textpage_ocr 😒

[cconur]

Changed to:
tp = page.get_textpage_ocr(flags=3, language='eng', dpi=150, full=True)

Now I get this error:

AttributeError: 'str' object has no attribute 'get_textpage_ocr'

Thanks in advance!

[cconur]

Solved, the input definition was wrong...
I had: doc = 'HI-WAY-clean.pdf'
Instead of: doc = fitz.open("HI-WAY-clean.pdf")

Now I face problems with the language. I need it in Spanish, do I need to download it manually to the tessdata folder?

[JorjMcKie]

You can check which languages are currently supported by your Tesseract: tesseract --list-langs.
If Spanish is missing, then yes, you must add it.

[cconur]

Many thanks for all!!

[cconur]

Hi Jorg,
I have the code refined and works great:
My question now is, what parameter (flags 4?) should i set to extract both text and images?

tp = page.get_textpage_ocr(flags=3, language='spa', dpi=150, full=True)
This is an example of a kind of page I would like to OCR, keeping both text and images:

What should be the export file format to save the output? .pdf?

import fitz
from progress.bar import ShadyBar

doc = fitz.open("HI-WAY-clean.pdf")
full_text = ""
bar = ShadyBar('Processing', max=len(doc))

for page in doc:
    tp = page.get_textpage_ocr(flags=4, language='spa', dpi=150, full=True)
    full_text += page.get_text(textpage=tp)

    bar.next()

file = open('full_text.pdf', 'w')
print("   >>> Generating output file...")
file.write(full_text)
file.close()
bar.finish()

Thanks a lot in advance!

[JorjMcKie]

Storing images within the page makes no sense, because they are already visible.

[JorjMcKie]

of course you can store whatever you like somewhere else, as image file or whatever.

[cconur]

How can I combine these 2 methods?
get_textbox()
get_textpage_ocr()

I would like to extract the text with OCR only from a specific area of each page (rectangular area). The text in the original file is in image mode (it's a .PNG).
Does the library discriminate while performing the OCR what is text and what are graphics? If so, I would like to save the graphics individually.
Can you share with me an example code to do this? (something similar I can use to investigate). Thanks in advance, I'm stuck!

[JorjMcKie]

You cannot. tp = page.get_textpage_ocr(...) always makes a TextPage for the full page.
But you can extract text such that you get the text bboxes as well:

blocks = page.get_text("dict", textpage=tp)["blocks"]

def get_text_in_rect(rect):
    text = ""
    for b in blocks:
        for l in b["lines"]:
            for s in l["spans"]:
                bbox = fitz.Rect(s["bbox"])
                if bbox in rect:
                    text += s["text"]
    return text

Does the library discriminate while performing the OCR what is text and what are graphics? If so, I would like to save the graphics individually.

No, Tesseract only recognizes text and nothing else.

I would like to extract the text with OCR only from a specific area of each page (rectangular area).

That is no problem. Make a Pixmap from an image or from that area of the page and use pdfbytes = Pixmap.tobytes_ocr(...), which gives you a 1-page PDF as a bytes object. Open this as a document like doc = fitz.open("pdf", pdfbytes) and extract the text from the first (and only) page.

[cconur]

Hi Jorg,

I have tested your code but I'm not able to get the OCR extracted text:

import fitz

doc = fitz.open("page-test.pdf")
page = doc[0]
tp = page.get_textpage_ocr(flags=3, language='spa', dpi=150, full=True)
blocks = page.get_text("dict", textpage=tp)["blocks"]
def get_text_in_rect(rect):
    text = ""
    for b in blocks:
        for l in b["lines"]:
            for s in l["spans"]:
                bbox = fitz.Rect(s["bbox"])
                if bbox in rect:
                    text += s["text"]
    return text
    print(text)

This is the file I have used for testing:
page-test.pdf

Any clue? many thanks in advance.

[JorjMcKie]

What is happening - pls give me some clue!
My test works fine:

In [1]: import fitz
In [2]: doc=fitz.open("page-test.pdf")
In [3]: page=doc[0]
In [4]: tp=page.get_textpage_ocr(flags=3,dpi=150,full=True)
In [5]: print(page.get_text(textpage=tp))
Nuevo Stralis Hi-Way— Sistema Eléctrico/Electrénico
SISTEMA ELECTRICO/ELECTRONICO
CALEFACTOR DE AGUA HYDRONICD
10 [84000]
553210
CALEFACTOR DE AGUA HYDRONICD 10 [84000]
Funcionamiento del sistema
Activando el quemador, la centralita electrénica controla: la bomba de agua, la bomba dosificadora de combustible, el motor del
quemadory la bujla de encendido.
La bomba de agua pone en circulacién el liquido refrigerante del circuito de refrigeracién del motor a calentar.
El motor del quemador aspira e introduce la cantidad de aire exacta necesaria para la combustién.
La bujia de encendido calienta el aire y quema el combustible inyectado por la bomba dosificadora.
La bomba dosificadora tiene la funcién de tomar e inyectar el combustible en la buja.
Después de 90 segundos la centralita, mediante el sensor de llama, controla que el vapor de la temperatura de los gases de escape
esté comprendida entre 500 - 600 °C. (Encendido correcto).
Cuando se alcanzan aproximadamente 700° €, la centralita desactiva la bujia y el quemador trabaja en autocombustién.
Si el sensor de llama no detecta dicha temperatura y por lo tanto la caldera no esta encendida, la centralita repite el procedimiento
de encendido, Si no se pone en marcha de nuevo, la centralita apaga el quemador y seré necesario volver a encenderlo manual-
mente.
Si el calefactor no se pone en marcha al tercer intento consecutivo, se bloquea.La Centralita controla, mediante el sensor de temperatura de agua y el sensor de recalentamiento, que la combustién y el quema-
dor funcionen correctamente,
El sensor de temperatura del agua es utilizado por la centralita para regular la potencia de trabajo del quemador (maxima,
media, minima y stand-by),
En la versign manual, cuando se alcanzan los 55°C, el sistema activa los ventiladores dentro de la cabina.
La centralta utiliza el sensor de sobrecalentamiento para apagar el quemador en caso de excesiva temperatura del agua
aproximadamente:
Eltiempo de funcionamiento del quemador depende de la programacién que haya realizado el conductor,
Una vez apagado, el quemador realiza una fase de lavado que dura 3 minutos aproximadamente, para estar listo para la proximal
puesta en marcha,
El calefactor estd conectado, con su propia légica de mando, en el sistema de climatizacién y esta dotado de una centralita electré-
nica con el correspondiente conector.
La centralita esta instalada directamente en el calefactor.
El calefactor puede ser pilotado directamente desde el sistema de climatizacién automatico, desde la versién de mando manual 0
mediante la centralita Bed Module,
— printed 603,95.405= base 06/2013 - 2nd ED.

In [6]:

[JorjMcKie]

Did you ever confirm, that text was indeed recognized at all?
If yes, then obviously the problem is the size of your "rect"!

[cconur]

Hi Jorj. Yes, the text has been recognized finally. Thanks for the help!
Can you give me a tip for the previous example on how to exclude the header an footer of the page for OCR text extraction applied to several pages with the same template?

[JorjMcKie]

Hi Jorj. Yes, the text has been recognized finally. Thanks for the help! Can you give me a tip for the previous example on how to exclude the header an footer of the page for OCR text extraction applied to several pages with the same template?

There is no systematic way to do that, because header / footer also are just text. But if you know which sub-rectangle of the page contains the significant text, you can doc page.get_text(...) specifying parameter clip. This ensures, that the returned text data are contained in that rectangle.

[cconur]

I'm not able to get the OCR only on the rect (the clip), it runs the OCR for the whole page:

import fitz

doc=fitz.open("page-test.pdf")
page=doc[0]

rect = fitz.Rect(80, 140, 550, 740)

# Draw a red box to visualize the rect's area (text)
page.draw_rect(rect, width=1.5, color=(1, 0, 0))

text = page.get_textpage_ocr(flags=3,dpi=150,full=True)

txt = page.get_text(textpage=text, clip=rect)

doc.save("output-with-rect-drawn.pdf")

print(txt)

[JorjMcKie]

Oh sorry, that is true: OCR used in that way only works for the full page.
So you have to manually exclude text outside clip.

[cconur]

Hi Jorj,
I have tried this:

import fitz

doc=fitz.open("page-test.pdf")
page=doc[0]

rect = fitz.Rect(80, 140, 550, 740)

# Draw a red box to visualize the rect's area (text)
page.draw_rect(rect, width=1.5, color=(1, 0, 0))

text = page.get_textpage_ocr(flags=3,dpi=150,full=True)

def get_text_in_rect(rect):
    text = ""
    for b in blocks:
        for l in b["lines"]:
            for s in l["spans"]:
                bbox = fitz.Rect(s["bbox"])
                if bbox in rect:
                    text += s["text"]
    return text

doc.save("output-with-rect-drawn.pdf")
doc.close()

print(text)

But I obtain this error: <fitz.fitz.TextPage; proxy of <Swig Object of type 'TextPage *' at 0x104e9d8d0> >

[JorjMcKie]

You have defined "text" to be a textpage. Trying to print this makes no sense at all, and using it after close all its parents (page, document) are gone, will give you that error.

[cconur]

yes, makes sense what you point out. Is there any parameter to not OCR to whole textpage?