Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openvino inference crachs after several successful inferences. my bug? #27054

Open
grofitGV4M opened this issue Oct 15, 2024 · 1 comment
Open
Assignees

Comments

@grofitGV4M
Copy link

grofitGV4M commented Oct 15, 2024

I convert torch model to scripted jit model, and than to openvino.
the model get 105 inputs and return 106 outputs.
the reasone for so many inputs and outputs is that this model is a stage on the way to stateful model.
however, I do:

        scripted_model = torch.jit.script(model.eval(), (dummy_inputs[0], prev_input, prev_ft, prev_mask, *buffers_list))
        scripted_model.eval()
        scripted_model_path = osp.join(osp.dirname(output_path), 'script_model.pt')
        scripted_model.save(scripted_model_path)
        torch._C._jit_pass_inline(scripted_model.graph)
        torch._C._jit_pass_dce(scripted_model.graph)  # Dead code elimination
        torch._C._jit_pass_constant_propagation(scripted_model.graph)
        torch._C._jit_pass_peephole(scripted_model.graph, True)
        scripted_model.eval()
        input_shapes = {'new_frame.1': ([1, 1, 160], ov.Type.f32), 'prev_input.1': ([1, 1, 160], ov.Type.f32), 'prev_frame_ft.1': ([1, 1, 321, 1, 2], ov.Type.f32),
                        'prev_mask.1': ([1, 1, 321, 1, 2], ov.Type.f32)}
        for i, buffer_shape in enumerate(buffer_manager.buffer_shapes):
            input_shapes[f'unet_buff_{i}.1'] = (buffer_shape, ov.Type.f32)
            print(f'input_shapes[unet_buff_{i}] = {buffer_shape}')
        example_input = [torch.randn([1, 1, 160]), torch.randn([1, 1, 160]), torch.randn([1, 1, 321, 1, 2]), torch.randn([1, 1, 321, 1, 2])]
        for shape in buffer_manager.buffer_shapes:
            example_input.append(torch.randn(shape))
        ov_model_32 = ov.convert_model(input_model=scripted_model, example_input=example_input, input=input_shapes, verbose=True, share_weights= False)
        ov.save_model(ov_model_32, osp.join(osp.dirname(output_path), 'ov_scripted_model.xml'), compress_to_fp16=False)
            ov_scripted_path = osp.join(osp.dirname(output_path), 'ov_scripted_model.xml')
            ovc_model = ov.runtime.compile_model(ov_scripted_path, config={'device': 'CPU'})
            infer_request1 = ovc_model.create_infer_request()
            ov_combined_buffer = combined_buffer.clone().zero_()
            manual_states = []
            prev_input = torch.zeros([1, 1, 160], dtype=torch.float32)
            prev_ft = torch.zeros([1, 1, 321, 1, 2], dtype=torch.float32)
            prev_mask = torch.zeros([1, 1, 321, 1, 2], dtype=torch.float32)
            manual_states.append(prev_input.numpy())
            manual_states.append(prev_ft.numpy())
            manual_states.append(prev_mask.numpy())
            ov_buffers = buffer_manager.get_buffers_list(ov_combined_buffer)
            for buff in ov_buffers:
                manual_states.append(buff.numpy())
            for i in range(num_frames):
                try:
                    inputs = {ovc_model.inputs[0].any_name: dummy_inputs[i].numpy()}
                    for k, state in enumerate(manual_states):
                        inputs[ovc_model.inputs[k+1].any_name] = state
                    infer_request1.infer(inputs=inputs)
                except Exception as e:
                    print(e)
                    print('Error at frame:', i)
                    break
                output_frame_ov = np.array(infer_request1.get_tensor(ovc_model.output(1)).data)
                frame_toll_rate = np.mean(np.abs(output_frame_ov - output_frame_th_list[i]) < 1e-4)
                print(f'frame {i} toll rate: {frame_toll_rate}')
                for k, state in enumerate(manual_states):
                    manual_states[k][:] = np.array(infer_request1.get_tensor(ovc_model.output(k+2)).data)

after some inferences that work correctly, with valid outputs:

Exception from src/inference/src/cpp/infer_request.cpp:223:
Check 'found_port.found()' failed at src/inference/src/dev/isync_infer_request.cpp:223:
Cannot find tensor for port opset1::Parameter new_frame.1[0]:f32[1,1,160]

Error at frame: 43

@Iffa-Intel
Copy link

@grofitGV4M it seems that the script that you share is incomplete for us to validate.

Could you help to clarify & provide:

  1. The model files or the complete script to generate them
  2. I believe this is a custom model?
  3. That error persist when trying to convert from original format into OpenVINO format or during inferencing?
  4. If during inferencing, could you share the complete inferencing code that you use? or are you referring to any of this OpenVINO sample?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants