EHR retrieval and query with GenAI models

[MMelQin]

This application presents a framework including multiple component services to integrate with FHIR service to retrieve patient EHR and a RAG pipeline to generate insights with the use of Embedding and LLM models.

• FHIR application. This is a client of FHIR service while at the same time a server to the RAG pipeline for on demand patient EHR retrieval. It authenticates with OAuth2 for bear token, which is used when getting and posting FHIR resources from/to a FHIR server identified by its URL
• RAG pipeline. It contains a RAG agent for building vector database with retrieved patient EHR resources, and a query agent answering end users' query. A Web UI is used and supports speech to text as well as text to speech.

The FHIR application can be run multiple ways:

• as a HoloHub app container, build and run with a single dev_container CLI command
• built and run in a HoloHub Dev container as console app
• built and run as console app in dev env without building container
• packaged and run as a Holoscan App Package (container), suitable for image distribution and deployment in "production" environment

The RAG pipeline application is only tested for running in HoloHub app container with the help of a start-up shell script, started manually. The whole process can potentially be automated with the dev_container build and run CLI command with the start-up script being the entrypoint of the app container.

[tbirdso]

To all involved: awesome work here! 🚀

Took an initial pass focused on FHIR, will take another pass soon to review LMM workflow.

[tbirdso]

Given the scope of these changes, it would be good to have some level of CTest unit test coverage to integrate with nightly dashboards. See Endoscopy Tool Tracking CMakeLists.txt for an example.

[tbirdso]

Took a best-effort pass through the lmm agent folder, looking good! Left some comments, mostly minor edits or fixes.

[tbirdso]

There are a few large JSON files in this PR. Would it be possible to reduce those contents to <1MB, provide generation instructions, or even distribute them ourselves rather than adding to commit history?

[jjomier]

There are a few large JSON files in this PR. Would it be possible to reduce those contents to <1MB, provide generation instructions, or even distribute them ourselves rather than adding to commit history?

We should put the data on NGC.

[cdinea]

There are a few large JSON files in this PR. Would it be possible to reduce those contents to <1MB, provide generation instructions, or even distribute them ourselves rather than adding to commit history?

We should put the data on NGC.
thank you for the suggestion @tbirdso @jjomier - do you think this NGC upload can be done in a subsequent PR or do you feel strongly about making the change in this PR?

[MMelQin]

There are a few large JSON files in this PR. Would it be possible to reduce those contents to <1MB, provide generation instructions, or even distribute them ourselves rather than adding to commit history?

We should put the data on NGC.
thank you for the suggestion @tbirdso @jjomier - do you think this NGC upload can be done in a subsequent PR or do you feel strongly about making the change in this PR?

It defeats the purpose if we let in the files on first merge, and we had made the decision.
The JSON files used for locally populating vector DB is not really needed (used as a safety net for the GTC demo), so it will be removed from the PR.
The FHIR resource bundle for testing resource sanitization will be downloaded from either NGC or the original source Synthea.
I am handling this.

[cdinea]

There are a few large JSON files in this PR. Would it be possible to reduce those contents to <1MB, provide generation instructions, or even distribute them ourselves rather than adding to commit history?

We should put the data on NGC.
thank you for the suggestion @tbirdso @jjomier - do you think this NGC upload can be done in a subsequent PR or do you feel strongly about making the change in this PR?

It defeats the purpose if we let in the files on first merge, and we had made the decision. The JSON files used for locally populating vector DB is not really needed (used as a safety net for the GTC demo), so it will be removed from the PR. The FHIR resource bundle for testing resource sanitization will be downloaded from either NGC or the original source Synthea. I am handling this.

thank you for the reply @MMelQin . I was not aware a decision has been made one way or the other as I saw no comment addressing this issue in the PR nor any other communication regarding this in Slack/e-mail.

RE: "The JSON files used for locally populating vector DB is not really needed (used as a safety net for the GTC demo),"

are you referring to ehr_data.json file?If so, that is not precisely a safety net, that is the result of running the script create_ehr_database_local and have it as part of the PR.

[MMelQin]

There are a few large JSON files in this PR. Would it be possible to reduce those contents to <1MB, provide generation instructions, or even distribute them ourselves rather than adding to commit history?

We should put the data on NGC.
thank you for the suggestion @tbirdso @jjomier - do you think this NGC upload can be done in a subsequent PR or do you feel strongly about making the change in this PR?

It defeats the purpose if we let in the files on first merge, and we had made the decision. The JSON files used for locally populating vector DB is not really needed (used as a safety net for the GTC demo), so it will be removed from the PR. The FHIR resource bundle for testing resource sanitization will be downloaded from either NGC or the original source Synthea. I am handling this.

thank you for the reply @MMelQin . I was not aware a decision has been made one way or the other as I saw no comment addressing this issue in the PR nor any other communication regarding this in Slack/e-mail.

RE: "The JSON files used for locally populating vector DB is not really needed (used as a safety net for the GTC demo),"

are you referring to ehr_data.json file?If so, that is not precisely a safety net, that is the result of running the script create_ehr_database_local and have it as part of the PR.

Hate to say it in this venue, but to prevent misconception, @cdinea you are wrong here. create_ehr_database_local consumes that pre-staged json file that contains resource that had been retrieved from the FHIR server (by the create_ehr_database at runtime or using client to drive the FHIR client). the local version was there in case there was no internet connection at GTC venue to the FHIR server hosted internally.
Keeping the json files out of the source repo (even though they could be made smaller) was discussed and decided on our weekly horizonal meeting; you might not have attended the meeting then.

Also, I'd strongly suggest use separate channel to discuss and sync up on the details.

[cdinea]

thanks @MMelQin . Indeed I reread my comment and would like to rectify, sent it too soon :) - get_ehr_data is needed in the create_ehr_db_local() to create the local database

`# Path to the EHR data as input
EHR_DATA_JSON = "/workspace/holohub/applications/ehr_query_llm/lmm/rag/ehr/ehr_data.json"

Path to the downloaded fine tuned model

EHR_FINETUNED_MODEL = "/workspace/volumes/models/bge-large-ehr-finetune"
CACHE_FOLDER = "/workspace/volumes/models"

Persistent storage folder for the Vector DB

PERSISTENT_FOLDER = "/workspace/holohub/applications/ehr_query_llm/lmm/rag/ehr/db"

def get_ehr_data():
with open(EHR_DATA_JSON) as f:
ehr_data = json.load(f)
return ehr_data`

[MMelQin]

Given the scope of these changes, it would be good to have some level of CTest unit test coverage to integrate with nightly dashboards. See Endoscopy Tool Tracking CMakeLists.txt for an example.

Will consider adding this in the next phase of this example app, at least for the operators.

[MMelQin]

There are a few large JSON files in this PR. Would it be possible to reduce those contents to <1MB, provide generation instructions, or even distribute them ourselves rather than adding to commit history?

We should put the data on NGC.

As explained and discussed above, the couple large JSON files were removed from this PR as they were indeed unnecessary. Large JSON file for FHIR Sanitizer module testing can be downloaded from public source (added instructions in code with this commit), and the other one used by the EHR Builder Agent was for a edge case for a demo, but the code was erroneously copied over; fixed with this commit.