-
Notifications
You must be signed in to change notification settings - Fork 200
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Terry Feng - Google Summer of Code Documentation (#396)
* terry initial contributor doc * fix final documentation
- Loading branch information
1 parent
d64f4e7
commit 01080a0
Showing
1 changed file
with
125 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| # Terry Feng - Google Summer of Code 2024 | ||
|
|
||
| ## Overview | ||
| **Contributor**: Terry Feng ([@terryzfeng](https://github.com/terryzfeng)) <br> | ||
| **Mentors**: Hongchan Choi ([@hoch](https://github.com/hoch)), | ||
| Michael Wilson ([@mjwilson-google](https://github.com/mjwilson-google)) <br> | ||
| **Organization**: Chromium - Web Audio Team <br> | ||
| **Project(s)**: | ||
| 1. [Web Audio Test Suite](#web-audio-test-suite) | ||
| 2. [Rainfly: An AudioWorklet DSP Playground](#rainfly-an-audioworklet-dsp-playground) | ||
|
|
||
| ## Description | ||
|
|
||
| This summer, I had the pleasure of working with the Chromium Web Audio Team on | ||
| two projects: building a **Web Audio Test Suite** and creating **Rainfly: An AudioWorklet DSP Playground**. | ||
| The Web Audio Test Suite enables Web Audio developers to write performance and | ||
| benchmark tests for Web Audio, executing them locally in an interactive webpage | ||
| or via the Playwright test framework for CI. Rainfly is an online IDE for writing | ||
| and running `AudioWorklet` code in JavaScript and visualizing real-time | ||
| `AudioContext` DSP output for analysis. | ||
|
|
||
| ## What I Did | ||
|
|
||
| ### Web Audio Test Suite | ||
|
|
||
| GitHub: [Web Audio Test Suite](https://github.com/GoogleChromeLabs/web-audio-samples/tree/main/src/tests/playwright) | ||
|
|
||
| 1. The Web Audio Test Suite began with researching JavaScript testing frameworks | ||
| for a lightweight solution offering browser JS executing and verification. We | ||
| decided upon the [Playwright](https://playwright.dev) testing framework | ||
| developed by Microsoft. The main reasons I found were: | ||
| - Streamlined E2E testing and browser emulation (Chromium, Canary, Chrome) | ||
| via CLI | ||
| - The ability to write vanilla Web Audio JS and use Hooks and Promises | ||
| to verify test logic | ||
| - Minimal overhead for Web Audio performance benchmarking | ||
| - CI via GitHub Actions | ||
| 2. I helped create a custom Web Audio test library (`audit.js`) that sandboxed | ||
| test code for execution timing and evaluation code for logic verification. The | ||
| challenge was to make everything as vanilla as possible while being compatible | ||
| with the Playwright test framework. Challenges that we overcame include: | ||
| - Creating the API with custom global test logic needed for arbitrary | ||
| asynchronous JS execution and Web Audio verification | ||
| - Finding appropriate thresholds for verifying "correct" `float32` samples | ||
| generated by different machines, browsers, and architectures | ||
| - Designing the writing experience of tests to be natural and intuitive for | ||
| future developers | ||
| 3. After Playwright CLI testing, I worked with ([@kizjkre](https://github.com/kizjkre)) | ||
| to build an interactive Live Suite webpage allowing for running Web Audio tests | ||
| locally in the browser. | ||
| 4. I finally ported all of Web Audio's WPT tests into our Web Audio Test Suite | ||
|
|
||
| ### Rainfly: An AudioWorklet DSP Playground | ||
|
|
||
| GitHub: *In Progress* | ||
|
|
||
| 1. Rainfly is a modern version of Hongchan's [Canopy](https://hoch.github.io/canopy/) | ||
| project designed specifically for the `AudioWorklet` (AW) DSP paradigm. It aims | ||
| to streamline the development of `AudioWorkletProcessor` DSP code, offering helpful | ||
| visualization and analysis tools. [@kizjkre](https://github.com/kizjkre) and I | ||
| created a [Figma](https://www.figma.com/proto/NrpvA1wwQNoc4hnKTAORIg/Rainfly?node-id=1-2&t=yNQPbNlEnX24i2MJ-0) | ||
| prototype to capture and design ideas for the tools and features that an AW DSP | ||
| playground could offer. | ||
| 2. I implemented Rainfly's code editor using [Monaco](https://microsoft.github.io/monaco-editor) | ||
| and handled JS code preprocessing, execution, and sandboxing for | ||
| `AudioWorkletProcessor` and `AudioContext` creation. I additionally created | ||
| various [Svelte](https://svelte.dev) components to control audio host playback. | ||
| 3. Rainfly includes an internal AW audio recorder that captures PCM samples and | ||
| streams them for visualization. To ensure this process was efficient and | ||
| seamless for the user, I: | ||
| - JS injected Rainfly's recorder node into the user's code just before | ||
| `context.destination` | ||
| - Efficiently handled real-time writing of millions of `process()` samples at | ||
| audio rate while reading and rendering them using the Canvas API at graphics | ||
| rate. | ||
|
|
||
| ## Code and PRs | ||
|
|
||
| ### Merged: | ||
| - [Playwright MVP](https://github.com/GoogleChromeLabs/web-audio-samples/pull/372) | ||
| - [Web Audio Test Suite Documentation](https://github.com/GoogleChromeLabs/web-audio-samples/pull/381) | ||
| - [Port Web Audio Performance Tests from WPT](https://github.com/GoogleChromeLabs/web-audio-samples/pull/384) | ||
|
|
||
| ### Open PRs: | ||
| - [Rainfly MVP](https://github.com/GoogleChromeLabs/web-audio-samples/pull/395) | ||
|
|
||
| ## Future Work | ||
| - Web Audio Test Suite: Ensuring that developers are able to effectively write | ||
| tests and benchmark Web Audio performance and functionality. | ||
| - Rainfly: Merging and deploying the current PR and MVP of Rainfly to | ||
| Web Audio Samples repository. | ||
| - Later improvements to add: | ||
| - Improving the visualizer by adding a time axis and sample pop-up on | ||
| hover for numerical feedback | ||
| - Smooth zooming in and out and scrolling left and right in the visualizer | ||
| window | ||
|
|
||
| ## Reflections and Takeaways | ||
|
|
||
| Working with my mentors Hongchan and Michael, I've learned a lot of best | ||
| practices for writing easy-to-read code, communicating effectively during code | ||
| review, and how to build projects effectively while navigating engineering, | ||
| design, and architectural decisions. It was a great experience to practice these | ||
| skills more, and my mentors were really supportive showing me the ropes. | ||
|
|
||
| In laying the groundwork for the Web Audio Test Suite, it was challenging and | ||
| quite tedious to spend time considering _all_ the current testing needs of the | ||
| team, finding good solutions to benchmark and analyze the performance of real-time | ||
| audio (this was harder than I thought), and to build a test framework | ||
| flexible enough for future use cases. This required extensive research to implement | ||
| an all-in-one API balancing tools, technologies, and code. However, this project | ||
| was extremely successful and will be beneficial for future Web Audio | ||
| testing | ||
|
|
||
| In working on Rainfly, I got the chance to learn and use a JS framework for the | ||
| first time (Svelte), make stylistic and creative UI/UX decisions for a "cute" IDE, | ||
| and implement important AW tooling for writing custom DSP code. This leveraged | ||
| a lot of my skills beyond pure software engineering which was fun, rigorous, | ||
| imaginative, and rewarding. | ||
|
|
||
| Overall I'd like to thank my mentors Hongchan Choi and Michael Wilson, my | ||
| fellow summer partner in crime and Web Audio GSoC contributor [@kizjkre](https://github.com/kizjkre), | ||
| and look forward to seeing my project work this summer being used internally | ||
| and externally for Web Audio developers. I hope to continue contributing to the | ||
| Chromium organization and the Web Audio project in the near future! |