table lines and cells being extracted in a weird way

[AttentiveNader]

I was trying to detect and extract tables from the drawings of a page but some times fitz extracts the table lines (that make up the table cells) as many rectangles that have really small area (so not the cell as a rectangle) and it does that without an obvious patters also sometimes the rectangle extract isn't actually a line but just a point.

I created this notebook
to reproduce the behavior
and this is the pdf am using tableSample

here the coordinates of the rectangle indicate that this a line or a point I was trying to use lines or rectangles to extract cells
but they don't have a pattern and it detects too many drawings for one small table (100 lines/points )

pdf_path = "./tableSample.pdf"
doc_pdf = fitz.open(pdf_path)
len([ s["items"][0][1] for s in doc_pdf[0].get_drawings() if s["items"][0][0] =="re" and len(s["items"]) ==1 ] )

[JorjMcKie]

This depends on the PDF creator software:
E.g. Word exports lines always as thin rectangles, and so do other office products.
You must use some logic that treats rects thinner than x points like lines.
Also please read the docu again: since some PyMuPDF versions, multiple "re" items may occur within the same path.

You can also not rely on the false assumption that tables having grid lines encircle each (or any) of their cells by some rectangle. There may just be lines (or as mentioned, those thin rectangles). Or there may be a mixture of both. Whatever you can imagine will be found somewhere.
So to find the cell rectangles you must compute the crossing points of horizontal and vertical (pseudo-) lines.

[AttentiveNader]

yes this happened when I used word to export a pdf with tables, but do you think there is any pattern for that ? because there are 100 (lines or points) detected for a table that is formed with 12 lines

[JorjMcKie]

Well, who do you want to blame?
You never know how sloppy these things are programmed. You simply have to develop adapting logic, which is e.g. able to ignore lines that do not add information - i.e. only differ by 1 or 2 pixels, or are subsets of each other: say line 1 goes from point 1 to point 2, line 2 goes from point1 to point 2 and then to point 3, etc., etc.
Probably best would be to build (Python!) sets of all occurring x-coordinates (= column borders) and of all y-coordinates (row borders). In this process, also treat values as equal that do not differ by more than a threshold.
Then combine those x and y values to make your table cell rects.

[AttentiveNader]

I don't mean to blame anyone, sorry if it seemed that way😅 , I will try filtering the lines as you said, I just thought if there is a pattern to how these many lines formed the table, I would be able to optimize the process better, as detecting the table cells and their associated text became really slow when I included the lines exported as rectangles.

[JorjMcKie]

Was just kidding, no worries.
As per the effort / performance: it can't be that bad. Define x_values = set(), y_values = set(). Then for every line withs its points p1 and p2: x_values.add(round(p1.x)), x_values.add(round(p2.x)).
Analogously for the y values.
For pseudo-lines (width < 3 and height > 20 = vertical line, horizontal line similar), use the the middle points of the top and the bottom instead. So if width is small, use (tl + tr)/2 as p1, (bl + br)/2 as p2. Or simply tl / bl.
Similar for small heights.

Then convert back to lists the x_values and y_values: x_values = sorted(list(x_values)), y_values = sorted(list(y_values)).

cells = []
for i in range(len(y_values)-1):
    line = []
    for j in range(len(x_values)-1):
        cell = fitz.Rect(x_values[j], y_values[i], x_values[j+1], y_values[i+1])
        line.append(cell)
    cells.append(line)

If I didn't mess up something too badly, you then should be able to access each cell as cells[i][j] where i is row number, j is column number.
Cell content can be extracted via page.get_textbox(cells[i][j]).

But here is a little present, that handles the mess of your example table quite well 😎:
tabext.zip