[BUG] Homomorphic encryption with simulator

[Sahilnalawade]

Describe the bug
I was executing the Homomorphic encryption workflow on the Transformer Model. I was using the shareable_geenerator as HEModelShareableGenerator and aggregator as HEInTimeAccumulateWeightedAggregator in the server_config.json file. And teh task_result_filters used were HEModelEncryptor and HEModelDecryptor.

This workflow was executed on a Bert based model on NLP dataset, and I did encounted error message.
This entire run was simulated using the command, "nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 nlp_task_he", where 'nlp_task_he' was the name of the package.

I have attached the error message (screen capture), also I have uploaded the error logs for the server and both sites.

To Reproduce
Steps to reproduce the behavior:

1. nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 nlp_task_he
2. Error Message -

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Logs
server_log.txt
site1_log.txt
site2_log.txt

Versions - Dependencies

• Python Version [e.g. 3.9]
• NVFlare Version [e.g. 2.3.1]

[chesterxgchen]

@Sahilnalawade thanks for reporting this issue. We will take a look. @holgerroth is this related to : #1959 ?

[holgerroth]

Yes, this is a related issue as HE is not supported in simulator. @Sahilnalawade, please follow the example here to run an HE experiment with local provisioning: https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/cifar10/cifar10-real-world#31-secure-fl-workspace

[Sahilnalawade]

Hi Holgerroth,
Thank you for the feedback,
I tried running the above workflow locally.
I have few questions,

1. I am not sure what is the exact steps to follow to run the automated experiments.
2. Configuration, I am running on a non-GPU machine.
3. How do I monitor the runs, are there logs generated to check the runs for the experiment.
4. Where would the directory be located, '[DOWNLOAD_DIR]/[JOB_ID]/workspace/cross_site_val/cross_val_results.json'.
5. Normally, I would run the 'nvflare simulator' to run the experiment, is there a similar way to run the HE experiment.
6. Lastly, how do you know if the run is successfull, nor mally in the simulator mode, you can monitor the logs. How to do that with the above experiment for the cifar-10-real-world

[Sahilnalawade]

Just to add to the above point, I have updated the nvflare package to version 2.3.1, Do let me know, if you want me to test with a different version.

[holgerroth]

HE is not directly supported in the simulator at this point (see #1959). You'll need to use the admin console to monitor jobs and download results. The results will be available after download in your admin folder inside the secure workspace that you created.

[Sahilnalawade]

Hi Holgerroth,

I did run the the workflow as per the instructions,
There were few issues,

1. The current workflow of HE did not intitate even after the client run has been triggered.
2. The server and client are just waiting for for instructions and commands even after the run has been submitted successfully.
3. Please find the detailed logs for the Server and the client.
4. Also, I had a question on HE workflow, If I have to customise the number of layers to be encrypted, can you point out the function of parameters that needs to changed either in the server / client json file or in the script file.
5. Also, to stop the HE workflow, it is not currently supported in the nvflare current version (2.3.3). TheHE workflow doesnot stop using the shutdown_fl_run.sh script.

Attachments --
client_log.txt
server_log.txt

Regards
Sahil Nalawade

[holgerroth]

Seems the job is not being scheduled as you have less clients connected than the minimum clients specified in meta.json

2023-09-21 15:54:25,003 - DefaultJobScheduler - INFO - [identity=secure_project, run=?]: Try to schedule job aa98f64f-ba0f-4a7c-a913-cd028bf48253, get result: (connected sites (2) < min_sites (8)).

[Sahilnalawade]

Hi Holgerroth,

I tried running the nvflare example and it failed due to the error message,
ModuleNotFoundError: No module named 'nvflare.app_common.executors.model_learner_executor'

I am attaching the logs for server and client and also I am attaching the server and client json files.

To run few clients on the nvflare, I have selected only 2 clients instead of 8. I have modified the meta.json and config_fed_server.json to have only 2 clients.

The error message seems to be for a module, nvflare.app_common.executors.model_learner_executor, this module seems to be missing in the latest version of nvflare package.
Also, the shutdown command does not work to close the server and the client jobs.

Attachments ---
config_fed_client.txt
config_fed_server.txt
meta.txt
server_log_v1.txt
client_log_v1.txt

Regards
Sahil Nalawade

[holgerroth]

Thanks. This is a version issue. If you have the latest release installed (2.3.5), please use the cifar10 code from the tagged release branch https://github.com/NVIDIA/NVFlare/tree/2.3.5/examples/advanced/cifar10/cifar10-real-world.

If you'd like to use the code from the main branch, you can install nvflare from source using pip install -e .

[Sahilnalawade]

Hi Holgerroth,

I was able to to execute the Homomorphic encryption run after upgrading the nvflare package to the development version of nvflare (version – 2.4.0rc2).
I tried using the 2.3.5 version but looks like that was not compatible as some of the files were missing.
Specifically - 'nvflare.app_common.executors.model_learner_executor'

Reagrds
Sahil Nalawade

[YuanTingHsieh]

ModelLearner and ModelLearnerExecutor is in new 2.4.x not in 2.3 release.

[chesterxgchen]

Model learner is new 2.4.x not in 2.3 release

…

________________________________ From: Sahil Nalawade ***@***.***> Sent: Wednesday, October 11, 2023 10:37 AM To: NVIDIA/NVFlare ***@***.***> Cc: Chester Chen ***@***.***>; Comment ***@***.***> Subject: Re: [NVIDIA/NVFlare] [BUG] Homomorphic encryption with simulator (Discussion #1994) Hi Holgerroth, I was able to to execute the Homomorphic encryption run after upgrading the nvflare package to the development version of nvflare (version – 2.4.0rc2). I tried using the 2.3.5 version but looks like that was not compatible as some of the files were missing. Specifically - 'nvflare.app_common.executors.model_learner_executor' Reagrds Sahil Nalawade — Reply to this email directly, view it on GitHub<#1994 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAD5FQ4EPHNXVJMDPS3SRFTX63KPJANCNFSM6AAAAAA5DKGZMU>. You are receiving this because you commented.Message ID: ***@***.***>