Add README (#34)

* Create README

* Update appium instructions

* Add GH actions instructions

* Add instructions for triggering workflow runs

* Add instructions for creating a new benchmark

README.md

@@ -0,0 +1,226 @@
+# Benchmarking Test App
+
+## Overview
+
+Benchmarking test app provides a simple harness for running React Native benchmarks on a real mobile device. The repository currently supports iOS benchmarks via Appium, and can be easily extended to support Android or any other platform supported by Appium and React Native. In this repository you'll find a few packages:
+
+- `apps/benchmarking-test-app`: A React Native app with built-in benchmarks for bitECS, three.js, and ir-engine.
+- `packages/benchmarking`: An Appium harness for running benchmarks within AWS device farm.
+- `packages/benchmark-parser`: A utility library for parsing benchmark files generated for the app from AWS device farm.
+
+There are currently two classes of benchmarks generated by the benchmarking test app: Xcode profiles and wall clock time benchmarks. The wall clock time benchmarks serve as a regression test for the benchmarks, tracking the performance of the profiled libraries over time, with results posted to: https://rebeckerspecialties.github.io/benchmarking-test-app/dev/bench/. When regressions happen, the Xcode instruments profiles can be used to dig into the root cause of the performance issues.
+
+## Using this repository
+
+The benchmarks in this repository measure the performance of various Entity Component System (ECS) libraries and JavaScript graphics engines. However, the benchmarking test app can easily be used to profile any JavaScript code, and can even be extended to profile native modules. Anyone can use this repository as a template to create their own benchmark suite for their device test cloud setup.
+
+We do not currently have a contribution guide or staffing to accept contributions, but it would make sense to set up extensions for running Android, Windows, and Mac benchmarks on a variety of test clouds. The repository is currently set up to run benchmarks on AWS, but it could be used on Firebase test lab with some wrangling.
+
+## Running benchmarks on a local device
+
+In order to build and run the benchmarks on a local test device with Appium, there are a few prerequisistes:
+
+### General Development
+
+1. Install node and npm
+
+We recommend installing node with Node Version Manager (nvm): https://github.com/nvm-sh/nvm. This repository supports Node 20+
+
+2. Install react native app dependencies
+
+```
+npm install -w benchmarking-test-app
+```
+
+### iOS Development (Mac Required)
+
+1. Install/update Cocoapods
+
+CocoaPods is installed by default on all Macs, but it may be out of date. Follow the instructions here: https://guides.cocoapods.org/using/getting-started.html to update Cocoapods. Cocoapods is used to manage native libraries for Xcode builds.
+
+2. Install/update Xcode
+
+Xcode is required for building iOS apps from native code. Download Xcode and Xcode command line tools from https://developer.apple.com/xcode/resources/
+
+3. Install React Native pods for Hermes
+
+```
+cd apps/benchmarking-test-app
+npm run install:ios:hermes
+```
+
+4. Connect an iOS device (or use an Xcode simulator)
+
+5. Launch the benchmarking test app on the device
+
+```
+cd apps/benchmarking-test-app
+npm run ios
+```
+
+6. (Optional) Build the app for release
+
+Debug builds of the app may contain additional code that can affect performance. When collecting data, bundles should be generated for release.
+
+```
+cd apps/benchmarking-test-app
+npm run build:ios:release
+npm run ios -- --mode=Release
+```
+
+If you run into issues with "no development team provided", open the generated xcworkspace, click on ReactTestApp -> Signing, and add your development team.
+
+### Android Development
+
+**_ UNDER CONSTRUCTION _**
+
+### Appium setup
+
+1. Install Appium
+
+```
+npm i --location=global appium
+```
+
+2. Install the xcuitest driver for Appium
+
+This allows Appium to run tests on an iOS device
+
+```
+appium driver install xcuitest
+```
+
+3. Run appium driver doctor to ensure that all dependencies are installed
+
+```
+appium driver doctor xcuitest
+```
+
+4. (Optional) Install the uiautomator2 driver for Appium
+
+This allows Appium to run tests on an Android device. Android is not fully supported, but there is only iOS specific appium code in the xcode instruments profiling benchmarks.
+
+```
+appium driver install uiautomator2
+appium driver doctor uiautomator2
+```
+
+5. Run appium
+
+Run appium with the desired capabilities. the `udid` option can be omitted if using a simulator.
+
+```
+appium --relaxed-security --default-capabilities \
+  "{\"appium:deviceName\": \"<Your device name here>", \
+  \"platformName\": \"iOS", \
+  \"appium:app\": \"com.rbckr.TestApp", \
+  \"appium:udid\":\"<Your device's udid>", \
+  \"appium:automationName\": \"XCUITest\"}"
+```
+
+### Running automated benchmarks
+
+1. Install scripted benchmark dependencies
+
+```
+npm install -w benchmarking-scripts
+```
+
+2. Bundle the benchmark scripts
+
+```
+npm run bundle -w benchmarking-scripts
+```
+
+3. Run the benchmarks
+
+```
+node packages/benchmarking/dist/benchmark.js
+```
+
+## Running automated benchmarks in AWS Device Farm
+
+### Creating an AWS Device Farm project
+
+1. Navigate to https://us-west-2.console.aws.amazon.com/devicefarm
+2. Click `Create mobile project` and follow instructions
+3. Once the project is created, navigate to the `Project settings` tab and click `Create device pool` to select the devices you want your benchmarks to run on
+
+### Updating the GitHub Actions workflow
+
+1. Update the `devicePoolArn` to match the device pool you created in the previous step. This will be configured via an environment variable in the future.
+2. Add secrets for iOS provisioning. Follow the guide here for more information: https://www.andrewhoog.com/post/how-to-build-an-ios-app-with-github-actions-2023/
+3. If you do not have a paid iOS developer account, you can find your DEVELOPMENT_TEAM by inspecting the Info.plist of any xcarchive you've generated. The ID will be a 10 digit alphanumeric value.
+4. If you do have a paid iOS developer account, the development team ID can be found in the Mac Apple Developer application. You should also be able to remove the `allowProvisioningUpdates` option from the `xcodebuild` command in `benchmark.yaml`
+
+### Generating an IAM role for the DEVICE_FARM_IAM_ARN secret
+
+1. Navigate to https://console.aws.amazon.com/iam
+2. Click `Roles` in the side bar
+3. Create a new role by clicking `Create Role`
+4. Select `Web Identity`
+5. Choose `tokens.actions.githubusercontent.com` as the Identity provider and fill in all of the fields below. You may need to create a new OpenID Connect Identity provider with this URL if it does not exist yet.
+6. Once the role is created, paste it into the `DEVICE_FARM_IAM_ARN` secret in GitHub.
+
+### Running the benchmark GitHub action
+
+Benchmarks will automatically run when a commit is pushed to the main branch on GitHub. Benchmarks can also be manually run from a branch by navigating to `Actions` -> `Run Benchmarks` -> `Run Workflow`.
+
+## Adding a benchmark
+
+Benchmarks need to be registered in two places: within the React Native app and in the Appium test.
+
+1. Write the JavaScript benchmark
+
+```ts
+// apps/benchmarking-test-app/src/benchmarks/fibonacci.ts
+const ITERATIONS = 10000;
+
+export const fibonacciBenchmark = async () => {
+  let prev = 0,
+    curr = 1,
+    next = -1;
+
+  for (let i = 1; i <= ITERATIONS; i++) {
+    next = prev + curr;
+    prev = curr;
+    curr = next;
+  }
+};
+```
+
+2. Register the benchmark in `App.tsx`
+
+```ts
+// apps/benchmarking-test-app/App.tsx
+import { fibonacciBenchmark } from "./src/benchmarks/fibonacciBenchmark";
+
+const App = () => {
+  return (
+    <>
+      ...
+      <SafeAreaView ...>
+        ...
+        <Benchmark name="fibonnaciBenchmark" run={fibonacciBenchmark} /> // New benchmark
+      </SafeAreaView>
+    </>
+  );
+};
+```
+
+3. Register the benchmark in `benchmark.ts`
+
+```ts
+// packages/benchmarking/benchmark.ts
+const runBenchmarkSuite = async () => {
+  console.log("Running benchmarks with profiler");
+  ...
+  await benchmarkWithProfiler("fibonnaciBenchmark");
+
+  console.log("Running benchmarks with wall clock time");
+  ...
+  await benchmarkWithWallClockTime("fibonnaciBenchmark");
+};
+```
+
+4. Follow the directions above to rebuild the app and rebundle the appium benchmark