Problem with request authentication when running job

[eduardpauliuc]

Hi! I am working on a federated non-HA project and I am trying to run it in Azure ML workspaces. (https://github.com/eduardpauliuc/fedpix/blob/main/project.yml, app dppix)

Both the server and the clients start alright, here is some of the client log.

2023-05-11 19:53:30,297 - Cell - INFO - FL-Europe-Site: created backbone external connector to grpc://dppix.northeurope.cloudapp.azure.com:8002
2023-05-11 19:53:30,297 - ConnectorManager - INFO - 34696: Try start_listener Listener resources: {'secure': False, 'host': 'localhost'}
2023-05-11 19:53:30,298 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:23016] is starting
2023-05-11 19:53:30,798 - Cell - INFO - FL-Europe-Site: created backbone internal listener for tcp://localhost:23016
2023-05-11 19:53:30,799 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE grpc://dppix.northeurope.cloudapp.azure.com:8002] is starting
2023-05-11 19:53:30,800 - FederatedClient - INFO - Wait for engine to be created.
2023-05-11 19:53:31,301 - FederatedClient - INFO - Successfully registered client:FL-Europe-Site for project dppix. Token:1693b062-9168-4a69-85e2-e824226b4f02 SSID:ebc6125d-0a56-4688-9b08-355fe9e4d61a
2023-05-11 19:53:31,308 - FederatedClient - INFO - Got engine after 0.507443904876709 seconds
2023-05-11 19:53:31,308 - FederatedClient - INFO - Got the new primary SP: grpc://dppix.northeurope.cloudapp.azure.com:8002

However, when submitting the job from admin console, the client crashes:

2023-05-11 19:54:49,919 - ClientRunner - INFO - [identity=FL-Europe-Site, run=ff2d3fd7-54e1-42b8-acb7-425087837361]: client runner started
2023-05-11 19:54:49,931 - Communicator - WARNING - get_task request authentication failed.
2023-05-11 19:54:54,934 - FederatedClient - INFO - pull_task completed. Task name:None Status:False 
2023-05-11 19:55:03,987 - ClientRunner - INFO - [identity=FL-Europe-Site, run=ff2d3fd7-54e1-42b8-acb7-425087837361]: ABORT (RUN) command received

Some of the server log:

2023-05-11 19:54:56,010 - ServerRunner - INFO - [identity=dppix, run=ff2d3fd7-54e1-42b8-acb7-425087837361, wf=scatter_and_gather, peer=FL-US-Site, peer_run=ff2d3fd7-54e1-42b8-acb7-425087837361, task_name=train, task_id=53c94ff0-7e43-4cca-9ffa-498bdd3b39f5]: assigned task to client FL-US-Site: name=train, id=53c94ff0-7e43-4cca-9ffa-498bdd3b39f5
2023-05-11 19:54:56,011 - ServerRunner - INFO - [identity=dppix, run=ff2d3fd7-54e1-42b8-acb7-425087837361, wf=scatter_and_gather, peer=FL-US-Site, peer_run=ff2d3fd7-54e1-42b8-acb7-425087837361, task_name=train, task_id=53c94ff0-7e43-4cca-9ffa-498bdd3b39f5]: sent task assignment to client. client_name:FL-US-Site task_id:53c94ff0-7e43-4cca-9ffa-498bdd3b39f5
2023-05-11 19:54:58,292 - GetTaskCommand - INFO - return task to client.  client_name: FL-Europe-Site  task_name: train   task_id: 557fa208-a328-449d-af27-e0f527d5f960  sharable_header_task_id: 557fa208-a328-449d-af27-e0f527d5f960
2023-05-11 19:54:59,089 - GetTaskCommand - INFO - return task to client.  client_name: FL-US-Site  task_name: train   task_id: 53c94ff0-7e43-4cca-9ffa-498bdd3b39f5  sharable_header_task_id: 53c94ff0-7e43-4cca-9ffa-498bdd3b39f5
2023-05-11 19:55:03,089 - ServerEngine - INFO - Job: ff2d3fd7-54e1-42b8-acb7-425087837361 child process exit with return code -9
2023-05-11 19:55:03,975 - JobRunner - INFO - [identity=dppix, run=?]: Try to abort run (ff2d3fd7-54e1-42b8-acb7-425087837361) on clients.

I opened ports 8002-8003 on the server (which is an azure VM), uploaded manually the provisioned site directories to each client. Is there something else to do regarding authentication that I am missing?

[IsaacYangSLA]

From the server log, both FL-Europe-Site and FL-US-Site were registered without issues and thus those certificates were good. Upon job submission, the get_task request from EU client got authentication failed. It seemed the cell communication couldn't authenticate peer. @yhwen , can you check if the log showed some cell issues?

[eduardpauliuc]

If it is of any help, here are the full logs from server and client. https://gist.github.com/eduardpauliuc/c9f9b223da3be68494e13eeaa704829c

In some runs, there are multiple get_task request

2023-05-11 19:54:49,807 - ClientRunner - INFO - [identity=FL-US-Site, run=ff2d3fd7-54e1-42b8-acb7-425087837361]: client runner started
2023-05-11 19:54:49,878 - Communicator - WARNING - Failed to get_task from dppix server. Will try it again.
2023-05-11 19:54:49,878 - FederatedClient - INFO - pull_task completed. Task name:None Status:False 
2023-05-11 19:54:50,451 - Communicator - WARNING - get_task request authentication failed.
2023-05-11 19:54:55,457 - FederatedClient - INFO - pull_task completed. Task name:None Status:False 
2023-05-11 19:55:03,269 - ClientRunner - INFO - [identity=FL-US-Site, run=ff2d3fd7-54e1-42b8-acb7-425087837361]: ABORT (RUN) command received

[yhwen]

Hi @eduardpauliuc, Your error experienced is not because of the client request authentication issue when running the job. Both "FL-Europe-Site" and "FL-US-Site" have been successfully registered to the server and were able to send "get_task" request to the server. However, when running the job the server, there's following error which caused the server job process crashed. This is due to the missing library of tensoflow. This error caused the server job process fail and not be able to authenticate the client get_task request.

2023-05-11 19:54:47.590299: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/compilers_and_libraries_2018.1.163/linux/tbb/lib/intel64_lin/gcc4.7:/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64_lin::/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64/
2023-05-11 19:54:47.590620: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/compilers_and_libraries_2018.1.163/linux/tbb/lib/intel64_lin/gcc4.7:/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64_lin::/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64/
2023-05-11 19:54:47.590712: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

[YuanTingHsieh]

@eduardpauliuc thanks for the logs, can you also attach job logs? (should be found in "client-site/job_id/log.txt")
And both US-site and EU-site logs?

Thanks

[eduardpauliuc]

Thank you for your answers! I blindly ignored those warnings thinking that I do not use TensorFlow in my trainer so it is not relevant..
I had TensorFlow installed but I think that the problem was with missing libnvinfer_plugin.so.7 and tensorRT. I followed the discussion from here (https://forums.developer.nvidia.com/t/could-not-load-dynamic-library-libnvinfer-so-7/231606/11) and after getting rid of all the warnings, the get_task request is properly received.

[YuanTingHsieh]

thanks for sharing your solution!