Exact custom inputs yet zero accuracy while training

[MohamadJaber1]

Firslty, I would like to thank you for this work. And also a big thanks for @juanmed for his amazing fork where it saved me a lot of time for labelling. And I saw that he has test.py file which I am looking forward to test my new images later in a robotic simulation environment and get the 6D pose. But with the results I am getting so far, I am wondering how can I proceed.

My .data file is meeting all the exact requirements. I don't know whats the problem with learning it with singleshotpose?
Before I raised this issue, I almost went through all the issues (opened and closed ones). I am using, PyTorch 0.4.1 with Python 3.7, CUDA 9.0, cuDNN 7. I forked @btekin work, as it is said to be compatible with Python 3 and PyTorch 0.4. My GPU is NVIDIA Quadro K2200 4 GB memory.

I am running the following line at cmd (.cfg file changed to batch size = 4 and subdivision = 4 as it was showing that CUDA is out of memory). I have changed the utils.py to match my camera intrinsic.
python train.py cfg/lego.data cfg/yolo-pose-pre.cfg cfg/darknet19_448.conv.23 cp backup/benchvise/init.weights

And what I can observe is that the model.weights file is just created once after first 10 epochs but never updated to the latest checkpoint. It means, the model is not improving at all. And after 177 epochs, the model got crashed with such an error.

I will go through the exact steps I have followed to generate my own dataset. I know many issues have been opened just for this question. I am sure that these steps will be of huge help to other users :).

1. Generated images from Nvidia Deep Learning Data Synthesizer NDDS (Unreal engine project). I had 1170 RGB images, 1170 masked images and their 3D bounding boxes. I had also the camera intrinsic parameters.
   

My object is a lego piece and below is the output of one image. This is 1 out 65 different oritentations of the lego piece..

2. For my CAD model which is .STEP file, I have imported it to the unreal engine project using datasmith plugin. Exported it as .FBX and then from Blender, I imported the .FBX file and exported it as .PLY. Initial weights when training for new objects (Suggestion) #68

3. From @juanmed fork, labelling was done with a python file called ndds_dataset_creator.py. I have gave my json and RGB images as input and got the corresponding label txt files. And the train.txt and test.txt. It generated 21 values as needed. It mapped points from the 3D bounding box to points in the 2D image. I have initially kept the training/testing ratio as 15%/85% as in the paper but with zero accuracy and then swapped this ratio to 85%/15% respectively.
   One picture of the labelling, which seems to be correct.
   

4. To give an accuracte value of the diameter which is in the .data file for my PLY file, I ran the following code to check my object diam of my own PLY file which is already provided at valid.py but left commented. I also compared it with ape.ply to see if my values are accuracte and indeed it was.
   

[btekin]

As discussed in #75, the issue seems to stem from providing wrong labels for the bounding box coordinates. Closing this as this is also being discussed in #75. Please feel free to continue discussing this in #75