unable to handle command: simulator due to: 'run_status', please check syntax after run hello-tf2 example

[plaipmc]

The code that I use is

sudo apt update
sudo apt upgrade
git clone https://github.com/NVIDIA/NVFlare.git
cd NVFlare
git checkout 2.3
sudo apt-get install python3-venv
sudo apt install python3-pip
python3 -m pip install --user --upgrade pip
python3 -m pip install --user virtualenv

cd examples
. set_env.sh
python3 -m pip install -U pip
python3 -m pip install nvflare==2.3.8
cd hello-world/hello-tf2
python3 -m pip install -r requirements.txt
cd ..
nvflare simulator -w /tmp/nvflare/hello-tf2 -n 2 -t 2 hello-tf2/jobs/hello-tf2

And the response is

...
2023-10-30 01:24:45,444 - ServerCommandAgent - INFO - ServerCommandAgent call registre_request_cb: server.simulate_job
unable to handle command: simulator due to: 'run_status', please check syntax
usage: nvflare simulator [-h] -w WORKSPACE [-n N_CLIENTS] [-c CLIENTS] [-t THREADS] [-gpu GPU] job_folder

positional arguments:
job_folder

optional arguments:
-h, --help show this help message and exit
-w WORKSPACE, --workspace WORKSPACE
WORKSPACE folder
-n N_CLIENTS, --n_clients N_CLIENTS
number of clients
-c CLIENTS, --clients CLIENTS
client names list
-t THREADS, --threads THREADS
number of parallel running clients
-gpu GPU, --gpu GPU
list of GPU Device Ids, comma separated
-m MAX_CLIENTS, --max_clients MAX_CLIENTS
maximum number of clients

I run other example (sag, cross_val, pt) already before install tensorflow. It worked. How can I fix it?? Thank you so much

ps. It has a bug about "unable to handle command: simulator due to: 'run_status', please check syntax" on runing cifar10 simulation before but it's not the same simulation

[YuanTingHsieh]

@plaipmc Looks like an NVFlare version mismatch.

Let's remove all the previously installed NVFlare version using:

python3 -m pip uninstall nvflare
python3 -m pip uninstall nvflare-nightly

Make sure to run these two lines before and after your activate your virtual environment (./set_env.sh).
(that means you need to remove nvflare installation both on your system and inside the virtual environment)

Then after that, do the setup steps as you pasted:

cd examples
. set_env.sh
python3 -m pip install -U pip
python3 -m pip install nvflare==2.3.8
cd hello-world/hello-tf2
python3 -m pip install -r requirements.txt
cd ..
nvflare simulator -w /tmp/nvflare/hello-tf2 -n 2 -t 2 hello-tf2/jobs/hello-tf2

[plaipmc]

@YuanTingHsieh I do like you recommend already but it's still the same.

If I understand correctly, the system should not have nvflare and virtual environment should have version 2.3.8, right?

Spec :

VM in VirtualBox using Ubuntu 22.04
Python 3.10.12
Tensorflow 2.14.0

[YuanTingHsieh]

@plaipmc
Yes, you are right.

Are you running on a Mac Apple M2 Pro chip?
Is your situation similar to this discussion: #1843?

[plaipmc]

@YuanTingHsieh No, I’m running VirtualBox on OMEN HP Laptop Intel Core i7-8750H CPU. I uninstall tensorflow to run another example which gonna be fine but in this example can’t run if no tensorflow so…

[YuanTingHsieh]

@plaipmc
Thanks for the updates, so it might be a general issue related to Virtual Machine/Virtual Box instead of Hardware related issues.
Tensorflow installation is very tricky, I will suggest use install "docker" and use the official docker image in NVIDIA.
This container: nvcr.io/nvidia/tensorflow:23.10-tf2-py3
From within the docker container, you install NVFlare.
And you run all these examples within the docker environment.

[chesterxgchen]

not sure this is VirtualBox issue. I have experienced problem when install tensorflow after I install pytorch, after that the tensorflow example simply doesn't work. @plaipmc seems have installed pytorch first, so he ran several examples. The error occurs when he ran tensorflow. I suggest to run to tensorflow without pytorch pre-installed (i.e. uninstall pytorch etc.) and see if that works for you