pdf/a validation check failed

[Daniel-WW]

Hey there,
I'm using ocrmypdf within a paperless ngx (docker) installation, and it seems, that it is not able to produce reliable pdf/a files. If I run a validation check (e.g. with verapdf) it always fails. I can not find any logs, that would show me, that the pdf/a conversation doesn't work correctly:

[2024-05-25` 12:52:33,964] [INFO] [paperless.parsing.tesseract] pdftotext exited 0

[2024-05-25 12:52:34,132] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': PosixPath('/tmp/paperless/paperless-ngxjjdvb9x7/20240525_124831_003022.pdf'), 'output_file': PosixPath('/tmp/paperless/paperless-c262wq3o/archive.pdf'), 'use_threads': True, 'jobs': 4, 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'color_conversion_strategy': 'RGB', 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': PosixPath('/tmp/paperless/paperless-c262wq3o/sidecar.txt'), 'continue_on_soft_render_error': True}

[2024-05-25 12:52:34,260] [INFO] [ocrmypdf._pipelines.ocr] Start processing 2 pages concurrently

[2024-05-25 12:52:35,226] [INFO] [ocrmypdf._pipeline] page is facing ⇧, confidence 7.99 - no change

[2024-05-25 12:52:35,951] [INFO] [ocrmypdf._pipeline] page is facing ⇧, confidence 8.10 - no change

[2024-05-25 12:52:38,057] [WARNING] [ocrmypdf._exec.tesseract] [tesseract] lots of diacritics - possibly poor OCR

[2024-05-25 12:52:48,314] [INFO] [ocrmypdf._pipelines.ocr] Postprocessing...

[2024-05-25 12:53:02,714] [INFO] [ocrmypdf._pipeline] Image optimization ratio: 1.00 savings: 0.0%

[2024-05-25 12:53:02,715] [INFO] [ocrmypdf._pipeline] Total file size ratio: 0.57 savings: -76.1%

[2024-05-25 12:53:02,717] [INFO] [ocrmypdf._pipelines._common] Output file is a PDF/A-2B (as expected)

[2024-05-25 12:53:02,726] [WARNING] [ocrmypdf._validation] The output file size is 1.76× larger than the input file.

Possible reasons for this include:

--deskew was issued, causing transcoding.

PDF/A conversion was enabled. (Try --output-type pdf.)

I already tried several specific pdf/a settings (e.g. pdfa-2) in the docker-compose file environment and in the webgui settings for testing, but without success.

Could anybody help me with this issue? What could be a reason for the not correct creating/archiving pdf/a files?

Thanks for help in advance - Daniel

[jbarlow83]

Investigating further will need:

• ocrmypdf/ghostcript version
• example input file that failed conversion
• output of verapdf

[Daniel-WW]

Thanks for the fast response. The version of ocrmypdf is 15.4.4 and of ghostscript 10.02.1.
Here is the input file:
TestPDF_01.pdf
This is the output file:
2024-05-26_Versandunterlagen_TestPDF_01.pdf
This is the verapdf report:
TestPDF_01_xmlReport.txt

And these are the logs from paperless ngx/ocrmypdf:

click here, to show logs

[2024-05-26 07:27:36,085] [DEBUG] [paperless.management.consumer] Waiting for file /usr/src/paperless/consume/TestPDF_01.pdf to remain unmodified

[2024-05-26 07:27:41,089] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/TestPDF_01.pdf to the task queue.

[2024-05-26 07:27:41,157] [DEBUG] [paperless.tasks] Skipping plugin CollatePlugin

[2024-05-26 07:27:41,157] [DEBUG] [paperless.tasks] Skipping plugin BarcodePlugin

[2024-05-26 07:27:41,157] [DEBUG] [paperless.tasks] Executing plugin WorkflowTriggerPlugin

[2024-05-26 07:27:41,159] [INFO] [paperless.tasks] WorkflowTriggerPlugin completed with:

[2024-05-26 07:27:41,159] [DEBUG] [paperless.tasks] Executing plugin ConsumeTaskPlugin

[2024-05-26 07:27:41,171] [INFO] [paperless.consumer] Consuming TestPDF_01.pdf

[2024-05-26 07:27:41,173] [DEBUG] [paperless.consumer] Detected mime type: application/pdf

[2024-05-26 07:27:41,179] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser

[2024-05-26 07:27:41,181] [DEBUG] [paperless.consumer] Parsing TestPDF_01.pdf...

[2024-05-26 07:27:41,195] [INFO] [paperless.parsing.tesseract] pdftotext exited 0

[2024-05-26 07:27:41,342] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': PosixPath('/tmp/paperless/paperless-ngxx7031_pb/TestPDF_01.pdf'), 'output_file': PosixPath('/tmp/paperless/paperless-q4zgi1dp/archive.pdf'), 'use_threads': True, 'jobs': 4, 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'color_conversion_strategy': 'RGB', 'skip_text': True, 'clean': True, 'deskew': True, 'rotate_pages': True, 'rotate_pages_threshold': 12.0, 'sidecar': PosixPath('/tmp/paperless/paperless-q4zgi1dp/sidecar.txt'), 'continue_on_soft_render_error': True}

[2024-05-26 07:27:41,509] [INFO] [ocrmypdf._pipelines.ocr] Start processing 2 pages concurrently

[2024-05-26 07:27:41,510] [INFO] [ocrmypdf._pipeline] skipping all processing on this page

[2024-05-26 07:27:41,511] [INFO] [ocrmypdf._pipeline] skipping all processing on this page

[2024-05-26 07:27:41,513] [INFO] [ocrmypdf._pipelines.ocr] Postprocessing...

[2024-05-26 07:27:41,662] [INFO] [ocrmypdf._pipeline] Image optimization ratio: 1.00 savings: 0.0%

[2024-05-26 07:27:41,662] [INFO] [ocrmypdf._pipeline] Total file size ratio: 1.02 savings: 2.1%

[2024-05-26 07:27:41,664] [INFO] [ocrmypdf._pipelines._common] Output file is a PDF/A-2B (as expected)

[2024-05-26 07:27:41,670] [DEBUG] [paperless.parsing.tesseract] Incomplete sidecar file: discarding.

[2024-05-26 07:27:41,716] [INFO] [paperless.parsing.tesseract] pdftotext exited 0

[2024-05-26 07:27:41,717] [DEBUG] [paperless.consumer] Generating thumbnail for TestPDF_01.pdf...

[2024-05-26 07:27:41,719] [DEBUG] [paperless.parsing] Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -auto-orient -define pdf:use-cropbox=true /tmp/paperless/paperless-q4zgi1dp/archive.pdf[0] /tmp/paperless/paperless-q4zgi1dp/convert.webp

[2024-05-26 07:27:42,667] [INFO] [paperless.parsing] convert exited 0

[2024-05-26 07:27:44,591] [DEBUG] [paperless.consumer] Saving record to database

[2024-05-26 07:27:44,593] [DEBUG] [paperless.consumer] Creation date from st_mtime: 2024-05-26 07:26:42.414793+02:00

[2024-05-26 07:27:45,068] [INFO] [paperless.handlers] Assigning correspondent Noctua RMA to 2024-05-26 TestPDF_01

[2024-05-26 07:27:45,090] [INFO] [paperless.handlers] Assigning document type Versandunterlagen to 2024-05-26 Noctua RMA TestPDF_01

[2024-05-26 07:27:45,111] [INFO] [paperless.handlers] Assigning storage path 3040_Reklamationen to 2024-05-26 Noctua RMA TestPDF_01

[2024-05-26 07:27:45,180] [DEBUG] [paperless.filehandling] Document has storage_path 66 (30_Konsum/3040_Reklamationen/{created_year}_{correspondent}/{created_year}-{created_month}-{created_day}_{document_type}_{title}) set

[2024-05-26 07:27:45,223] [DEBUG] [paperless.filehandling] Document has storage_path 66 (30_Konsum/3040_Reklamationen/{created_year}_{correspondent}/{created_year}-{created_month}-{created_day}_{document_type}_{title}) set

[2024-05-26 07:27:45,227] [DEBUG] [paperless.consumer] Deleting file /tmp/paperless/paperless-ngxx7031_pb/TestPDF_01.pdf

[2024-05-26 07:27:45,236] [DEBUG] [paperless.parsing.tesseract] Deleting directory /tmp/paperless/paperless-q4zgi1dp

[2024-05-26 07:27:45,237] [INFO] [paperless.consumer] Document 2024-05-26 Noctua RMA TestPDF_01 consumption finished

[2024-05-26 07:27:45,240] [INFO] [paperless.tasks] ConsumeTaskPlugin completed with: Success. New document id 728 created

If I can provide any further information, let me know. - Thanks for help!

[jbarlow83]

I couldn't reproduce the issue using your example file and

ocrmypdf 16.3.0
ghostscript 10.03.1
veraPDF 1.26.2

veraPDF validated the output as PDF/A-2B.

The command line I used was
ocrmypdf -l deu --color-conversion-strategy RGB --skip-text --clean --deskew --rotate-pages --rotate-pages-threshold 12 --continue-on-soft-render-error issuepdf/1320.pdf _.pdf

So it appears to me that the issue is due to old versions of ocrmypdf and/or ghostscript.

[stumpylog]

Unfortunately paperless is not able to update to a newer ocrmypdf, as support for Python 3.9 was dropped. And personally ghostscript 10.02.1 is already a very recent version, even more recent than Debian provides normally for bookworm.

Basically, I don't think this is just "old versions"

[jbarlow83]

Ghostscript 10 through 10.02.0 are blacklisted by ocrmypdf because they all have unacceptable errors, specifically corrupting OCR text and pre-existing text on certain documents, not to mention a few serious CVEs (FWIW). The only Ghostscript versions allowed are 9.54, 9.55, and 10.02.1+. I understand this forces people into newer Ghostscript versions but no one should be using the early 10.x releases.

I have limited resources for this project and drawing the line at the Ubuntu 22.04 / RHEL9 base system is something I needed to do keep on top of issues, which gives us Python 3.10 as the base system.

[Daniel-WW]

Ok, thanks for help and testing. - I think I have to look for a solution in the paperless ngx community, because everything runs in the Paperless Docker container and I can't do a manual update.

[jbarlow83]

@Daniel-WW I did check out ocrmypdf 15.4.4 with ghostscript 10.02.1 and that also does not reproduce the issue on my end. We have the same veraPDF version. I have no further ways to investigate.

I don't think that the provide output file is the actual output file. 2024-05-26_Versandunterlagen_TestPDF_01.pdf appears to be produced by LibreOffice Writer, and has never been passed through OCRmyPDF. OCRmyPDF changes certain metadata after it runs.

[stumpylog]

Thanks for checking again with those versions. I'll have to see about ghostscript 10.03.x, if that has reached the trixie repos yet for our users.