feat(ast_tools): pre-compile #[ast] macro output

[overlookmotel]

Work in progress.

Generate code for #[ast] macro with an oxc_ast_tools generator.

oxc_ast_macros crate now has no dependencies (no syn etc).

The guts of it is the code! macro. This is the same as quote!, except it evaluates to a TokenStream of code to create the TokenStream which quote! evaluates to.

Unfortunately I'm not seeing an improvement in compile times, because other dependencies of oxc_ast also use syn, so it doesn't drop from the dependency graph. The generated code is 3000 lines, and could likely be shrunk down, which also could help. The generated code is now 600 lines, and now there is a 5% reduction in debug build time for oxc_ast crate - but that's still a disappointingly small improvement.

Am opening this PR not necessarily with the intent of merging it, but to discuss whether this is maybe a useful approach. @rzvxa would appreciate your feedback.

Note: This technique would allow #[generate_derive] to inject the derived impl next to the type def, same as #[derive] does, which would solve the problem of types/fields needing to be pub(crate).

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “0-merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[overlookmotel]

• feat(ast_tools): pre-compile #[ast] macro output #5796 👈
• refactor(ast): move functions to top level in ast macro #5793
• refactor(ast): import syn types in ast macro #5792
• refactor(ast): split ast macro into multiple files #5791
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on Graphite

[codspeed-hq]

CodSpeed Performance Report

Merging #5796 will not alter performance

_{Comparing 09-14-feat_ast_tools_pre-compile_ast_macro_output (90746ab) with main (bb95306)}

Summary

✅ 29 untouched benchmarks

[rzvxa]

This is super smart! I like what I'm seeing👍

[overlookmotel]

Thanks. But unfortunately it's not moving the needle on performance.

I suspect (but don't know) that creating TokenTrees and TokenStreams is inherently expensive due to how proc macros interface with the compiler. There may just be no way to make proc macros cheap.

Or maybe the #[ast] macro just isn't that costly in the first place, so optimizing it is never going to pay dividends.

Thanks for taking a look. I'm going to pause on this though for now as am planning to spend this week focused on transformer. Must... not... get... distracted...

By the way, these 2 articles were the inspiration. They do a very good job of explaining how proc macros work internally. I always thought it involved serialization, but it turns out not:

• https://blog.jetbrains.com/rust/2022/03/18/procedural-macros-under-the-hood-part-i/
• https://blog.jetbrains.com/rust/2022/07/07/procedural-macros-under-the-hood-part-ii/

[rzvxa]

Or maybe the #[ast] macro just isn't that costly in the first place, so optimizing it is never going to pay dividends.

I think this is more likely; No matter how much proc macro overhead costs us, If we do enough work in a macro eventually caching it would be faster. I suspect we have a better chance of getting some juice out of this approach in #4112 However it might be a wild goose chase.

And thanks for the links, They seem really interesting especially the second part(gave them a cursory look), I'll give them a read.