URGENT!!! TEDS score different from the one reported in the paper.

[dimitri009]

Discussed in #14574

^{Originally posted by dimitri009 January 21, 2025}
I already asked this but I didnt received any response.

As a report for a work I wanted to test the TEDS of various models but the TEDS score of SLANet on PubTabNet is different from the one reported in the documentations.

I downloaded SLANet from here and tested with PubTabNet but I ended with a TEDS of 0.468942475598164

code:
python3 ppstructure/table/eval_table.py --det_model_dir=output2/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=output2/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=output2/en_ppstructure_mobile_v2.0_SLANet_infer --image_dir=/data/Datasets/pubtabnet/pubtabnet/val_/ --rec_char_dict_path=ppocr/utils/dict/table_dict.txt --table_char_dict_path=ppocr/utils/dict/table_structure_dict.txt --det_limit_side_len=736 --det_limit_type=min --rec_image_shape=3,32,320 --gt_path=output2/gt_.txt

I would like to know if someone has tested the TEDS score and how he did it please it is urgent.

I checked image per image to see if the problem was my dataset it seems my dataset is ok, it just that the performance does not match the one in the paper.

as example here for the first image in dataset

`pred1 = '

Weaning Week 15 Off-test	– – –	– 0.17 ± 0.08 0.80 ± 0.24	– 0.16 ± 0.03 0.19 ± 0.09

'
true1 = '

	Weaning	Week 15	Off-test
Weaning	–	–	–
Week 15	–	0.17 ± 0.08	0.16 ± 0.03
Off-test	–	0.80 ± 0.24	0.19 ± 0.09

'
from ppstructure.table.table_metric import TEDS

teds = TEDS()

score = teds.evaluate(pred1, true1)
print('TEDS score:', score)`

TEDS score: 0.3615999999999999

And I did the same for the firsts three images: [0.3615999999999999, 0.4781577297371583, 0.5670696970573337]

[GreatV]

Which version of PaddleOCR are you using? You can try switching to other versions such as 2.8 or 2.7.

[dimitri009]

I am on Python 3.11
with this version of paddle:

on CUDA 12.4

[dimitri009]

Then I tried with pip install paddleocr==2.8.0 on the first image and I ended with the same result ppocr INFO: teds: 0.3615999999999999
I tried also with pip install paddleocr==2.7.0.0 same result ppocr INFO: teds: 0.3615999999999999

Note: I canceled the ocr.pickle and structure.pickle before every run.

[dimitri009]

@GreatV , @jzhang533 , @wencan Could someone helps me to solve this issue ? I want to be sure that I am doing wrong and it is not a problem from Paddle's script ?
I also tried with paddleocr==2.6.0
Could someone share with how he gets the right result ?

[GreatV]

Delete the cached result, then switch to a different ppocr version.

[dimitri009]

As I mentioned in my note, three comments above, I canceled all the cached files before every run.
I tried on another computer but I get the same results, I tried also with TableRec Master but the results are worse.
I canceled all the cached file before every run.