Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use PGCN to postprocess the results? #32

Open
JosephineRabbit opened this issue Nov 10, 2020 · 17 comments
Open

How to use PGCN to postprocess the results? #32

JosephineRabbit opened this issue Nov 10, 2020 · 17 comments

Comments

@JosephineRabbit
Copy link

Hi, I use the 'detection.json' file in the output folder to generate a similar propfile with bsn_test_prop_file in PGCN. However, I got lower results than the original results of gtad itself. I don't know where I did wrong.

@RonanDou
Copy link

RonanDou commented Dec 1, 2020

I meet the same problem, did you solve it? thanks u!!

@Zoey-Yue
Copy link

Postprocessing without conducting NMS might be help.

@JosephineRabbit
Copy link
Author

Does it works for you?

Postprocessing without conducting NMS might be help.
Does it works for you?

@frostinassiky
Copy link
Owner

Guys sorry for the delay! Here is the update.
The G-TAD+PGCN experiment was implemented on another machine, but I have a chance to find the script that I used for the experiment. 💯

Some extra details:
1, G-TAD predictions that have low confidence scores (e.g. 0.1) are discarded.
2, Original P-GCN proposals are kept to maintain the distribution of the start/end times. This is important because the P-GCN model is already trained with its own proposal.
3, Some hyperparameters of P-GCN need to be adjusted accordingly, such as combination weights, nms threshold.

New code:
Please add the following script in the dataset class (pgcn_dataset.py) of P-GCN code here.

        ###################################      GTAD           ########################################################
        use_gtad = True
        gtad_thres = 0.1
        if use_gtad:
            '''Load GTAD result'''
            import pandas as pd
            data = pd.read_csv('/home/xum/PGCN/data/detection_results.txt', sep=" ", header=None)
            data.columns = ["vid", "s", "e", "cls", "score"]
            vids = [x for x in set(data.vid.values.tolist()) if type(x) != float]
            '''Load THUMOS infomations'''
            thumos_info = pd.read_csv('/home/xum/PGCN/data/thumos_info.csv', sep=",")
            thumos_cls = pd.read_csv('/home/xum/PGCN/data/detclasslist.txt', sep=" ", header=None)
            thumos_cls_idx = {thumos_cls.iloc[x, 0]: x for x in range(20)}
            '''Replace each proposal'''
            for per_prop in prop_info:
                vid = per_prop[0].split('/')[-1]
                if not vid in vids:
                    print('video {} has zero proposal from gtad'.format(vid))  # check it exists
                    continue
                gtad_vid = data[data.vid == vid]  # unit is second
                frate = thumos_info[thumos_info['video-name'] == vid]['frame-rate'].values[0]
                starts = (gtad_vid.s * frate).values
                ends = (gtad_vid.e * frate).values
                scores = gtad_vid.score.values
                clses = [thumos_cls_idx[x] for x in gtad_vid.cls.values]


                for idx, (c, sc1, sc2, s, e) in enumerate(zip(clses, scores, scores, starts, ends)):
                    if sc1 > gtad_thres:
                        per_prop[3].insert(0, [str(int(c)), str(sc1), str(sc2), str(int(s)), str(int(e))])

        ################################################################################################################

Please let me know if you have any questions.

@RonanDou
Copy link

RonanDou commented Jan 9, 2021

Guys sorry for the delay! Here is the update.
The G-TAD+PGCN experiment was implemented on another machine, but I have a chance to find the script that I used for the experiment. 💯

Some extra details:
1, G-TAD predictions that have low confidence scores (e.g. 0.1) are discarded.
2, Original P-GCN proposals are kept to maintain the distribution of the start/end times. This is important because the P-GCN model is already trained with its own proposal.
3, Some hyperparameters of P-GCN need to be adjusted accordingly, such as combination weights, nms threshold.

New code:
Please add the following script in the dataset class (pgcn_dataset.py) of P-GCN code here.

        ###################################      GTAD           ########################################################
        use_gtad = True
        gtad_thres = 0.1
        if use_gtad:
            '''Load GTAD result'''
            import pandas as pd
            data = pd.read_csv('/home/xum/PGCN/data/detection_results.txt', sep=" ", header=None)
            data.columns = ["vid", "s", "e", "cls", "score"]
            vids = [x for x in set(data.vid.values.tolist()) if type(x) != float]
            '''Load THUMOS infomations'''
            thumos_info = pd.read_csv('/home/xum/PGCN/data/thumos_info.csv', sep=",")
            thumos_cls = pd.read_csv('/home/xum/PGCN/data/detclasslist.txt', sep=" ", header=None)
            thumos_cls_idx = {thumos_cls.iloc[x, 0]: x for x in range(20)}
            '''Replace each proposal'''
            for per_prop in prop_info:
                vid = per_prop[0].split('/')[-1]
                if not vid in vids:
                    print('video {} has zero proposal from gtad'.format(vid))  # check it exists
                    continue
                gtad_vid = data[data.vid == vid]  # unit is second
                frate = thumos_info[thumos_info['video-name'] == vid]['frame-rate'].values[0]
                starts = (gtad_vid.s * frate).values
                ends = (gtad_vid.e * frate).values
                scores = gtad_vid.score.values
                clses = [thumos_cls_idx[x] for x in gtad_vid.cls.values]


                for idx, (c, sc1, sc2, s, e) in enumerate(zip(clses, scores, scores, starts, ends)):
                    if sc1 > gtad_thres:
                        per_prop[3].insert(0, [str(int(c)), str(sc1), str(sc2), str(int(s)), str(int(e))])

        ################################################################################################################

Please let me know if you have any questions.

Thanks for your reply, but how can I get the "/home/xum/PGCN/data/detection_results.txt"?

@frostinassiky
Copy link
Owner

Hi @RonanDou The "detection_results.txt" file is the txt format of the json file after post-processing.

You may need to change the line here.

gtad/gtad_postprocess.py

Lines 154 to 156 in 6deb5b1

with open(opt["output"] + '/detection_result.json', "w") as out:
json.dump(output_dict, out)

@RonanDou
Copy link

Hi @RonanDou The "detection_results.txt" file is the txt format of the json file after post-processing.

You may need to change the line here.

gtad/gtad_postprocess.py

Lines 154 to 156 in 6deb5b1

with open(opt["output"] + '/detection_result.json', "w") as out:
json.dump(output_dict, out)

I do the experiment according to what you describe, but I get lower results than the original results of gtad itself. Can you give a little more implementation detail? such as: hyperparameters. Thank you !!

@frostinassiky
Copy link
Owner

Hi @RonanDou As I remember, I changed the NMS threshold and weights to combine RBG and optical flow results.

Since you find the P-GCN result is worse than G-TAD, what if you decrease gtad_thres to improve more low-confident predictions?

@mrlihellohorld
Copy link

Hi @RonanDou The "detection_results.txt" file is the txt format of the json file after post-processing.
You may need to change the line here.

gtad/gtad_postprocess.py

Lines 154 to 156 in 6deb5b1

with open(opt["output"] + '/detection_result.json', "w") as out:
json.dump(output_dict, out)

I do the experiment according to what you describe, but I get lower results than the original results of gtad itself. Can you give a little more implementation detail? such as: hyperparameters. Thank you !!

Hi, would u tell me how to get this file 'thumos info.csv'? thanks a lot!

@mrlihellohorld
Copy link

Hi, I use the 'detection.json' file in the output folder to generate a similar propfile with bsn_test_prop_file in PGCN. However, I got lower results than the original results of gtad itself. I don't know where I did wrong.

Hi, Could you share your code for generating the propfile with bsn_test_prop_file?

@mrlihellohorld
Copy link

I meet the same problem, did you solve it? thanks u!!

Hi, Could you share your code for generating the propfile with bsn_test_prop_file?

@mrlihellohorld
Copy link

Postprocessing without conducting NMS might be help.

Hi, Could you share your code for generating the propfile with bsn_test_prop_file?

@JosephineRabbit
Copy link
Author

Hi @RonanDou The "detection_results.txt" file is the txt format of the json file after post-processing.
You may need to change the line here.

gtad/gtad_postprocess.py

Lines 154 to 156 in 6deb5b1

with open(opt["output"] + '/detection_result.json', "w") as out:
json.dump(output_dict, out)

I do the experiment according to what you describe, but I get lower results than the original results of gtad itself. Can you give a little more implementation detail? such as: hyperparameters. Thank you !!

Hi, could you share how to get the 'detection_results.txt'? I try to save the content of json file as txt. But it can not be used by the provided code.

@NooneOnlyOne
Copy link

Hi ,Could you help me solve a few problems

  1. I get the ”detection_result.txt“ The format is as follows:
vid,s,e,cls,score
video_test_0000004,19.5,22.3,CricketBowling,0.323464
video_test_0000004,19.5,22.3,CricketShot,0.271474
video_test_0000004,0.0,0.5,CricketBowling,0.177645
video_test_0000004,0.0,0.5,CricketShot,0.149093
video_test_0000004,21.3,22.5,CricketBowling,0.156935
video_test_0000004,21.3,22.5,CricketShot,0.131711
...........

I get the “thumos_info.csv” ,The format is as follows:

video-name,t-init,t-end,f-init,f-end,video-duration,frame-rate,video-frames,label-idx
video_test_0000278,0.0,1.4,0,42,215.166667,30.0,6455,-1
video_test_0000278,95.7,97.2,2871,2916,215.166667,30.0,6455,-1
video_test_0000293,50.6,54.6,1518,1638,233.466667,30.0,7004,-1
video_test_0000293,67.4,71.7,2022,2151,233.466667,30.0,7004,-1
video_test_0000293,99.7,106.4,2991,3192,233.466667,30.0,7004,-1
video_test_0000293,118.1,126.4,3543,3792,233.466667,30.0,7004,-1
..........

the "detclasslist.txt" ,The format is as follows:

BaseballPitch
BasketballDunk
Billiards
CleanAndJerk
CliffDiving
CricketBowling
CricketShot
Diving
FrisbeeCatch
........

I did not add the above Gtad code in PGCN and run test_two_stream.sh
the result:
average map :0.4266

I add the above Gtad code in PGCN and run test_two_stream.sh
the result:
average map :0.4266
Why are the above two results the same ?

Could you tell me where I went wrong?Thanks

@cecilia930426
Copy link

Hi ,Could you help me solve a few problems

  1. I get the ”detection_result.txt“ The format is as follows:
vid,s,e,cls,score
video_test_0000004,19.5,22.3,CricketBowling,0.323464
video_test_0000004,19.5,22.3,CricketShot,0.271474
video_test_0000004,0.0,0.5,CricketBowling,0.177645
video_test_0000004,0.0,0.5,CricketShot,0.149093
video_test_0000004,21.3,22.5,CricketBowling,0.156935
video_test_0000004,21.3,22.5,CricketShot,0.131711
...........

I get the “thumos_info.csv” ,The format is as follows:

video-name,t-init,t-end,f-init,f-end,video-duration,frame-rate,video-frames,label-idx
video_test_0000278,0.0,1.4,0,42,215.166667,30.0,6455,-1
video_test_0000278,95.7,97.2,2871,2916,215.166667,30.0,6455,-1
video_test_0000293,50.6,54.6,1518,1638,233.466667,30.0,7004,-1
video_test_0000293,67.4,71.7,2022,2151,233.466667,30.0,7004,-1
video_test_0000293,99.7,106.4,2991,3192,233.466667,30.0,7004,-1
video_test_0000293,118.1,126.4,3543,3792,233.466667,30.0,7004,-1
..........

the "detclasslist.txt" ,The format is as follows:

BaseballPitch
BasketballDunk
Billiards
CleanAndJerk
CliffDiving
CricketBowling
CricketShot
Diving
FrisbeeCatch
........

I did not add the above Gtad code in PGCN and run test_two_stream.sh
the result:
average map :0.4266

I add the above Gtad code in PGCN and run test_two_stream.sh
the result:
average map :0.4266
Why are the above two results the same ?

Could you tell me where I went wrong?Thanks

Could you please share the detclasslist.txt and thumos_info.csv to me? Thanks

@cecilia930426
Copy link

Hi ,Could you help me solve a few problems

  1. I get the ”detection_result.txt“ The format is as follows:
vid,s,e,cls,score
video_test_0000004,19.5,22.3,CricketBowling,0.323464
video_test_0000004,19.5,22.3,CricketShot,0.271474
video_test_0000004,0.0,0.5,CricketBowling,0.177645
video_test_0000004,0.0,0.5,CricketShot,0.149093
video_test_0000004,21.3,22.5,CricketBowling,0.156935
video_test_0000004,21.3,22.5,CricketShot,0.131711
...........

I get the “thumos_info.csv” ,The format is as follows:

video-name,t-init,t-end,f-init,f-end,video-duration,frame-rate,video-frames,label-idx
video_test_0000278,0.0,1.4,0,42,215.166667,30.0,6455,-1
video_test_0000278,95.7,97.2,2871,2916,215.166667,30.0,6455,-1
video_test_0000293,50.6,54.6,1518,1638,233.466667,30.0,7004,-1
video_test_0000293,67.4,71.7,2022,2151,233.466667,30.0,7004,-1
video_test_0000293,99.7,106.4,2991,3192,233.466667,30.0,7004,-1
video_test_0000293,118.1,126.4,3543,3792,233.466667,30.0,7004,-1
..........

the "detclasslist.txt" ,The format is as follows:

BaseballPitch
BasketballDunk
Billiards
CleanAndJerk
CliffDiving
CricketBowling
CricketShot
Diving
FrisbeeCatch
........

I did not add the above Gtad code in PGCN and run test_two_stream.sh
the result:
average map :0.4266

I add the above Gtad code in PGCN and run test_two_stream.sh
the result:
average map :0.4266
Why are the above two results the same ?

Could you tell me where I went wrong?Thanks

Because the test_two_stream.sh file reads the results from already generated results/Flow_result results/RGB_result files

@NooneOnlyOne
Copy link

Hi ,Could you help me solve a few problems

  1. I get the ”detection_result.txt“ The format is as follows:
vid,s,e,cls,score
video_test_0000004,19.5,22.3,CricketBowling,0.323464
video_test_0000004,19.5,22.3,CricketShot,0.271474
video_test_0000004,0.0,0.5,CricketBowling,0.177645
video_test_0000004,0.0,0.5,CricketShot,0.149093
video_test_0000004,21.3,22.5,CricketBowling,0.156935
video_test_0000004,21.3,22.5,CricketShot,0.131711
...........

I get the “thumos_info.csv” ,The format is as follows:

video-name,t-init,t-end,f-init,f-end,video-duration,frame-rate,video-frames,label-idx
video_test_0000278,0.0,1.4,0,42,215.166667,30.0,6455,-1
video_test_0000278,95.7,97.2,2871,2916,215.166667,30.0,6455,-1
video_test_0000293,50.6,54.6,1518,1638,233.466667,30.0,7004,-1
video_test_0000293,67.4,71.7,2022,2151,233.466667,30.0,7004,-1
video_test_0000293,99.7,106.4,2991,3192,233.466667,30.0,7004,-1
video_test_0000293,118.1,126.4,3543,3792,233.466667,30.0,7004,-1
..........

the "detclasslist.txt" ,The format is as follows:

BaseballPitch
BasketballDunk
Billiards
CleanAndJerk
CliffDiving
CricketBowling
CricketShot
Diving
FrisbeeCatch
........

I did not add the above Gtad code in PGCN and run test_two_stream.sh
the result:
average map :0.4266
I add the above Gtad code in PGCN and run test_two_stream.sh
the result:
average map :0.4266
Why are the above two results the same ?
Could you tell me where I went wrong?Thanks

Could you please share the detclasslist.txt and thumos_info.csv to me? Thanks
You can contact me via my email [email protected]. I don’t know if the two files I generated are correct. Maybe we can discuss it in detail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants