Earth-Mission-Control

An immersive, multi-user VR/AR data visualization platform, aimed at enabling climate scientists to more effectively communicate their data stories to policy makers to drive more informed policy decisions.

Game Object Assets and Interactives

• Interactive Globe Module
  • Travel and visualize and city/region on the map table by adding and selecting locations on the Earth globe object. This object is particularly useful in visualizing areas that are most impacted by climate change.
  • Modifiable texture allowing for different global data visualizations.
• Map Table
  • Visualization of selected locations from the globe. Position and zoom levels are configurable using the palm-up hand console.
• Slider Mechanism
  • Allows for dynamic visualizations where the user may slide through specific values and see how a particular environment or dataset changes, dependent on that value. An example scene contains a slider mechanism displaying how Venice, Italy is affected by sea level rise.
• EarthBot
  • Live AI bot powered by ChatGPT to ask climate-related questions that a user may have when interacting with the platform.
  • Many controls and teleportation can also be made using EarthBot
• Hyperwalls
  • Display multiple Earth observation data visualizations on one data panel, allowing for the user to compare and contrast different visualizations at the same time.
  • Magnification capabilities of specific panel along the entire hyperwall also exist.
• Palm-Up Hand Console
  • Pinnable panel which contains all of the controls for hyperwall and global visualizations.
  • Movement in the map table is controlled by this panel as well.
• Multi-User Capability
  • Interact with up to 16 players within EMC at once no matter their location!
  • With hand tracking capabilities the users can create dynamic gestures and point at different items within the environment, along with a live voice chat.

Set-Up/Installation

This project was built with Unity and the final application was compiled into an Android .apk file since that is the format that the Quest headsets use. To test out the application, simple download the apk file from the releases page and install it on your headset using either Sidequest or the command line.

To test out the application in the Unity Editor, you will need to set up a few things:

• First, download Unity Hub and install Unity 2021.3.20f, making sure to install the Android Development tools. This process will also install Visual Studio to be able to to view and edit the code. You can follow these instructions for more details.
• Ensure your headset is properly set up for development, following these instructions.
• Next clone the repo or download it as a zip file and extract it. In Unity Hub, add the project and open it.
• Once the project is open, load the 'EMC-VR-layout1' Scene from the Assets/Scenes folder.
  • If you want to explore the desktop version, open the scene titled 'EMC-desktop-layout1-updated'
• Add the MapSessionConfig and ChatGPTSettings files you have into the Resources folder.
• Ensure that the build settings are set to Android and if the Oculus software is running locally and the headset is connected via Link, you can just press the play button in the Editor and the application will load on the headset.

Description of Files

The relevant files for the EMC environment are:

• Assets/Scripts/EnvironmentManager.cs
  • The manager script for controlling environment parameters, used by the control panel component.
• Assets/Scripts/GlobeManager.cs
  • The manager script for the globe module, responsible for placing the map pins in the correct locations and controlling the visuals on the globe.
• Assets/Scripts/MapManager.cs
  • The manager script for the map table and is responsible for controlling the zoom level, map location, water level visualization, particle simulation, and location info for the dashboard.
• Assets/Scripts/MapPinManager.cs
  • The manager script for the map pins that appear on the globe. The data is loaded from a JSON file and sets up the visuals of the pins.
• Assets/Scripts/HyperwallManager.cs
  • The manager script is used to change the content appearing in the hyperwall. Once the script is added as a component on an object the user can add which media assets they would like to have for selection.

The relevant files for EarthBot are:

• Assets/Scripts/ChatGPTAssistant.cs
  • The main interface for the ChatGPT game object that connects with the text-to-speech module from the voice SDK and sends text to it in chunks because the Voice SDK has a limit to how much text it can process at once.
  • Adapted from DilmerVR chatbot implementation.
• Assets/Scripts/VoiceIntentController.cs
  • Connects the speech understanding to the system and allows actions to be triggered when key words or intents are recognized.
  • It specifies which actions the system can take and connects them to event listeners
  • Adapted from DilmerVR chatbot implementation.
• Assets/Scripts/ChatGPT/ChatGPTSettings.cs
  • A helper file used to store ChatGPT settings
  • Adapted from DilmerVR chatbot implementation.
• Assets/Scripts/ChatGPT/ChatGPTClient.cs
  • The main interface with the ChatGPT REST API. Includes logic for message history storage and sending the ChatGPT response to the ChatGPTAssistant script.
  • Adapted from DilmerVR chatbot implementation.

Troubleshooting

• If the scene loads without the environment fully loaded, open the package manager and re-import the asset called Nature Starter Kit 2. If it's not listed under the package manager, you can get it from the asset store as a free asset.
• If the EarthBot doesn't respond to questions or if there is a console error that says there are ChatGPT settings missing even though they are in the Resources folder, select the ChatGPTClient object in the Hierarchy and set the ChatGPTSettings property to the file in the Resources folder.