lang: Anchor IDL should be uploaded to program binary instead of separate PDA

[ngundotra]

To date, only 25% of Anchor programs have uploaded their IDL 🤯 (1,324 programs out of 5,335 anchor programs currently on mainnet).

If anchor IDLs were uploaded with program binaries, then we would have 100% IDL adoption, leading to increased transparency on Solana.

Instead of requiring developers to write the compressed IDL bytes to the IDL address, anchor build should write the zipped bytes to an ELF section called .anchor.idl. Example: https://github.com/nifty-oss/asset/blob/main/include-idl/src/lib.rs

Developers should still be able to opt-out of this default behavior and deploy the IDLs the old way.

[acheroncrypto]

Overall, this would be great.

If anchor IDLs were uploaded with program binaries, then we would have 100% IDL adoption, leading to increased transparency on Solana.

It would increase the percentage, but I don't think we'd ever see 100% IDL adoption, especially if we give people the chance to opt out of this behavior.

Developers should still be able to opt-out of this default behavior and deploy the IDLs the old way.

Is this for backwards compatibility? We have to get rid of the IDL instructions when the IDL exists in the binary, otherwise the binary size will grow significantly.

[ngundotra]

It would increase the percentage, but I don't think we'd ever see 100% IDL adoption, especially if we give people the chance to opt out of this behavior.

Yes this is true. I think aiming for 50% adoption or higher over next year would be attainable, with this change alone.

On backwards compatibility, I am agnostic.

On removing IDL instructions from the binary, this is super exciting because then deploying IDLs with your binary is essentially "free". In other words - this shouldn't change deployment costs 😄 ! That should make adopting this easier🤞

[ngundotra]

Core does not recommend putting IDLs in program binary. They are trying to shave all bits of program binaries down as much as possible.

Accordingly, working on a different approach. Foundation is working on getting a program to upload program metadata on-chain like IDL, solana-verify args, and more. It will handle buffer history versioning too. It will be immutable ASAP.

[acheroncrypto]

Core does not recommend putting IDLs in program binary. They are trying to shave all bits of program binaries down as much as possible.

Makes sense. Similarly, I was going to say this is going to be very inefficient for clients as well, since they'd have to fetch the whole programdata account just to read a tiny portion of it. Adding a new RPC method probably isn't desired either.

Accordingly, working on a different approach. Foundation is working on getting a program to upload program metadata on-chain like IDL, solana-verify args, and more. It will handle buffer history versioning too. It will be immutable ASAP.

That sounds great! I always thought having an IDL program would be better than including all IDL instructions inside all programs (like Anchor does now) anyway. What's planned sounds similar, but more general.

[ngundotra]

Here is roughly the program spec we are thinking about: https://solanafoundation.notion.site/Program-Metadata-128d36dad52d801e974fcc3a30736506

[ngundotra]

Okay, after further discussion with Anza VM devs here, it seems that this was a miscommunication. Core seems to be fine with having arbitrary metadata in the ELF section headers.

Given how much friction is caused by uploading IDLs via IdlInstructions, I think uploading IDLs in section headers is the better way to go.

@febo and I have published solana-include-idl and solana-include-idl-cli https://github.com/nifty-oss/asset/blob/main/include-idl/README.md under solanafoundation are testing it out with OreV2 & some other programs. We will be working to support this new place for IDLs in the Explorer over the next 2-3 weeks.

An additional major benefit of uploading IDLs into binaries is that we can avoid the social consensus issue of getting program developers to use an IDL program. It's not impossible, but would suck to manage.