Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

target-independent bindings #3

Open
Irate-Walrus opened this issue Dec 24, 2024 · 13 comments
Open

target-independent bindings #3

Irate-Walrus opened this issue Dec 24, 2024 · 13 comments
Assignees
Labels
enhancement New feature or request

Comments

@Irate-Walrus
Copy link
Contributor

Irate-Walrus commented Dec 24, 2024

Currently bindings are restricted to windows and x86_64/x86. This is reflected in the src/ffi/generated.rs:

compile_error!("These bindings can only be used on `x86_64` architectures. To generate bindings for your target architecture, consider using the `regenerate` feature.");

There are two reasons I found for this:

  • The cty crate depends on the target to create ctype definitions
  • extern blocks rely on linked symbols and ABIs such as stdcall may not be possible on other targets.

My use case is such that I wish to use structures such as PPEB on non-native targets such as i686-unknown-linux-gnu.

This being the case I forked this repository, which can be found here: https://github.com/Irate-Walrus/phnt-rs

In this fork I introduced a pre-generated x86_64_bindgen.rs and x86_bindgen.rs as well as two feature flags externs and fn_types.

externs allows externally linked symbols to be enabled/disabled:

#[cfg(feature="externs")]
extern "C" {
   pub static GUID_NULL: GUID;
   pub static mut NlsAnsiCodePage: USHORT;
   ...

While fn_types defines the externally linked functions as types:

#[cfg(feature="fn_types")]
mod fn_types {
    pub type NtCallbackReturnFn = unsafe extern "C" fn(OutputBuffer: PVOID, OutputLength: ULONG, Status: NTSTATUS) -> NTSTATUS;
    pub type NtFlushProcessWriteBuffersFn = unsafe extern "C" fn() -> NTSTATUS;
    pub type NtQueryDebugFilterStateFn = unsafe extern "C" fn(ComponentId: ULONG, Level: ULONG) -> NTSTATUS;
    ...

This is helpful if you are trying to call these functions directly in memory:

let rtl_allocate_heap: RtlAllocateHeapFn = core::mem::transmute(rtl_allocate_heap_addr);
let ptr = rtl_allocate_heap(self.handle(), HEAP_ZERO_MEMORY, layout.size() as _)

I've also added to the GitHub workflows to auto-regenerate both x86 and x86_64 bindings and commit them to the repo.

There are a few issues that I've seen so far:

  • Bad regexes in my build.rs lead to failure with the latest bindgen version 0.71.1
  • Dropping the use of cty for internally defined ctypes may break compatibility with pre-existing users.
  • Custom rustfmt to allow for the use of regex when feature-gating externs and function types.

This issue is to gauge your interest in these changes, although it's likely I may continue in the fork to add further definitions from other header files. Thanks for you hard work, it made these changes relatively straightforward.

@oberrich oberrich added the enhancement New feature or request label Dec 24, 2024
@oberrich oberrich self-assigned this Dec 24, 2024
@oberrich
Copy link
Collaborator

oberrich commented Dec 24, 2024

First of all thanks for the write-up!

The cty crate depends on the target to create ctype definitions

I'm happy to remove cty in the future for custom rolled type definitions.

extern blocks rely on linked symbols and ABIs such as stdcall may not be possible on other targets.

Does your fork address this or is that what the feature guards are for?

I'm not entirely sure what your use case is for using a different target like i686-unknown-linux-gnu, would you like to regenerate the bindings from linux (thinking of Windows headers required on the clang side here) or just use them and be able to disable the function definitions so you can operate on the structures?

In this fork I introduced a pre-generated x86_64_bindgen.rs and x86_bindgen.rs

Good idea! I see you've added the regenerate step to the CI and upload the artifacts from there.
Maybe we should add prebuilt ARM64 bindings as well to support all of the major architectures without having to install C tools and rebuild it.

Thanks again and merry Christmas!

@Irate-Walrus
Copy link
Contributor Author

Merry Christmas! 🎄

Does your fork address this or is that what the feature guards are for?

I use the extern feature guard to prevent linked symbols or incompatible ABIs such as stdcall.

I'm not entirely sure what your use case is for using a different target

See https://github.com/Irate-Walrus/stardust-rs for an example use case. However, I have another use case which involves using some of the defined structs in riscv64 (this resulted in the target_arch checks failing), see https://secret.club/2023/12/24/riscy-business.html .

would you like to regenerate the bindings from linux

I didn't have much luck regenerating on linux and if the bindings are pre-built then I don't need to. In stardust-rsI actually need the function type definitions not the function definitions themselves, due to the regex this meant that function bindings had to be generated.

Maybe we should add prebuilt ARM64 bindings as well.

I'd be keen, I was making the edits on my fork with this in mind. However, there is no Windows ARM64 github runner so we'd probably have to pursue cross-compilation first.

@Irate-Walrus
Copy link
Contributor Author

Just realized I'd made a silly mistake regarding imports when using the fn_types feature, resolved now. Also this is the error expected when you compile with externs and have an incompatible ABI:

image

@oberrich
Copy link
Collaborator

oberrich commented Dec 29, 2024

I'd be keen, I was making the edits on my fork with this in mind. However, there is no Windows ARM64 github runner so we'd probably have to pursue cross-compilation first.

I've managed to generate bindings for i686 (x86), x86_64 and aarch64 (ARM64) using windows-2025 GitHub runner. 🥳🎉

Update: Got a bit too excited, while I am able to generate the bindings the free runner is in fact not aarch64.
I've moved the project into an organization to get larger runners, unfortunately the Windows runner for ARM barely has any software yet and not even git and bash are working as expected.

The weirdest part about all of this is that I have "Git for Windows" installed which is on PATH but I don't have bash.

@Irate-Walrus
Copy link
Contributor Author

RIP, I do that all the time. So if that doesn't decide to work it's a hosted runner (windows on a rpi always feels so wrong to me) or cross-compilation.

@oberrich
Copy link
Collaborator

oberrich commented Jan 4, 2025

RIP, I do that all the time. So if that doesn't decide to work it's a hosted runner (windows on a rpi always feels so wrong to me) or cross-compilation.

After spending almost 60 USD and well over 200 runs on the ARM64 machine, I've finally managed to install

  • Git for Windows (Bash)
  • LLVM 19 (Clang, required for bindgen)
  • Visual Studio 2022 Build Tools
  • Rust (nightly)

It takes around 10 minutes to set everything up, under normal circumstances the CI will not be triggered very often, however.
CI cost should go down significantly once they've pre-installed build tools on the image and/or release the runner into free tier.

@Irate-Walrus
Copy link
Contributor Author

Far out man you're throwing cash around 💸. Great job getting it working and I hope you don't have to run those builds too often 0.0

@oberrich
Copy link
Collaborator

oberrich commented Jan 9, 2025

@Irate-Walrus I've replaced cty with ::core::ffi:: and have upgraded Rust edition to 2024. Does this solve the cty issue for you?

@Irate-Walrus
Copy link
Contributor Author

Apologies for the slow reply, I'm just back from some time I took to disconnect.

I've replaced cty with ::core::ffi:: and have upgraded Rust edition to 2024. Does this solve the cty issue for you?

Regarding this replacement I believe that this still suffers from the issue that ::core::ffi is architecture + os dependent. This means that I cannot compile/use the windows bindings for x86_64 when targeting arm as the ctype definitions may be invalid with the generated bindings.

@oberrich
Copy link
Collaborator

Alright, I've shipped a custom cty implementation that always assumes cfg(windows) instead.

@Irate-Walrus
Copy link
Contributor Author

Alright, I've shipped a custom cty implementation that always assumes cfg(windows) instead.

Yeah easy that appears to work, although the use case of using x86_64 bindings while target_arch is something other than x86_64 does not appear to be met.

@oberrich
Copy link
Collaborator

oberrich commented Jan 13, 2025

Yeah easy that appears to work, although the use case of using x86_64 bindings while target_arch is something other than x86_64 does not appear to be met.

I think you're referring to the conditional compilation, I'm not comfortable regexing over the bindings.
Instead I want to get rust-lang/rust-bindgen#3088 merged, so we can implement this in a cleaner way.

Unfortunately there hasn't been any feedback from the maintainers yet.

@Irate-Walrus
Copy link
Contributor Author

Yeah awesome, the regex is just a quick hack. Just checked out that PR, I hope it gets merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants