Implement blocking eventfd #3939

tiif · 2024-10-04T10:48:47Z

This PR

Implemented blocking for both read and write of eventfd
Added test for eventfd blocking read and write
Removed eventfd blocking tests from fail-dep
Added a new BlockReason::Eventfd

cc #3665

tiif · 2024-10-04T10:49:24Z

@rustbot author

tiif · 2024-10-05T10:16:30Z

Hmm...why is rustfmt complaining? Let me try rebase it instead.

src/shims/unix/linux/eventfd.rs

bors · 2024-10-08T13:47:06Z

☔ The latest upstream changes (presumably #3951) made this pull request unmergeable. Please resolve the merge conflicts.

tests/fail-dep/libc/eventfd_block_twice.stderr

src/shims/unix/linux/eventfd.rs

It still panic on blocked_write_tid borrow

This reverts commit 42890a9.

tiif · 2024-10-17T15:36:00Z

src/shims/unix/linux/eventfd.rs

+    let mut waiter = Vec::new();
+    let mut blocked_write_tid = eventfd.blocked_write_tid.borrow_mut();
+    while let Some(tid) = blocked_write_tid.pop() {
+        waiter.push(tid);
+    }
+    drop(blocked_write_tid);
+
+    waiter.sort();
+    waiter.dedup();
+    for thread_id in waiter {
+        ecx.unblock_thread(thread_id, BlockReason::Eventfd)?;
+    }


I couldn't think of a testcase that can actually produce thread id duplication in blocked_write/read_id, but I will just keep the dedup here first.

we'll get a panic if that happens, so maybe just don't sort and dedup and wait until we get a test case. I don't think it can happen though, as a thread would need to get blocked again without having been removed and unblocked.

Also: the unblocking order is user-visible, and tho I assume that eventfd does not specify which thread gets woken first, we'd probably want a random (deterministic, but depending on the seed) order of unblocking to happen

cc @RalfJung I think we need a new system for unblocking multiple threads at the same time. So far we only ever unblocked one thread, and "unblock N threads, then randomly run them with the normal rules for picking the next thread" seems like a recurring thing.

Also I wonder if the normal thread unblocking is subtly wrong, too. Or just wrong in this PR:

When a thread gets unblocked and immediately performs some operation that is visible from other threads, then that means when the current thread continues after unblocking the other thread, it may behave as if the unblocked thread already had a few CPU cycles to do something. While that may be expected behaviour depending on preemption, the behaviour here does not depend on preemption. So, to avoid such a footgun, thread unblocking should not immediately execute the unblocking operation, but just unblock the thread and make the thread execute the unblocking operation when it gets scheduled next.

tiif · 2024-10-18T05:10:07Z

@rustbot ready

oli-obk

not finished with my review, but we need to figure out a few thread unblocking things first

src/shims/unix/linux/eventfd.rs

oli-obk · 2024-10-26T11:10:43Z

src/shims/unix/linux/eventfd.rs

+    let mut waiter = Vec::new();
+    let mut blocked_write_tid = eventfd.blocked_write_tid.borrow_mut();
+    while let Some(tid) = blocked_write_tid.pop() {
+        waiter.push(tid);
+    }
+    drop(blocked_write_tid);
+
+    waiter.sort();
+    waiter.dedup();
+    for thread_id in waiter {
+        ecx.unblock_thread(thread_id, BlockReason::Eventfd)?;
+    }


we'll get a panic if that happens, so maybe just don't sort and dedup and wait until we get a test case. I don't think it can happen though, as a thread would need to get blocked again without having been removed and unblocked.

src/shims/unix/linux/eventfd.rs

oli-obk · 2024-10-26T11:19:31Z

src/shims/unix/linux/eventfd.rs

+    let mut waiter = Vec::new();
+    let mut blocked_write_tid = eventfd.blocked_write_tid.borrow_mut();
+    while let Some(tid) = blocked_write_tid.pop() {
+        waiter.push(tid);
+    }
+    drop(blocked_write_tid);
+
+    waiter.sort();
+    waiter.dedup();
+    for thread_id in waiter {
+        ecx.unblock_thread(thread_id, BlockReason::Eventfd)?;
+    }


Also: the unblocking order is user-visible, and tho I assume that eventfd does not specify which thread gets woken first, we'd probably want a random (deterministic, but depending on the seed) order of unblocking to happen

cc @RalfJung I think we need a new system for unblocking multiple threads at the same time. So far we only ever unblocked one thread, and "unblock N threads, then randomly run them with the normal rules for picking the next thread" seems like a recurring thing.

Also I wonder if the normal thread unblocking is subtly wrong, too. Or just wrong in this PR:

When a thread gets unblocked and immediately performs some operation that is visible from other threads, then that means when the current thread continues after unblocking the other thread, it may behave as if the unblocked thread already had a few CPU cycles to do something. While that may be expected behaviour depending on preemption, the behaviour here does not depend on preemption. So, to avoid such a footgun, thread unblocking should not immediately execute the unblocking operation, but just unblock the thread and make the thread execute the unblocking operation when it gets scheduled next.

src/shims/unix/linux/eventfd.rs

oli-obk · 2024-10-26T15:57:17Z

@RalfJung we raced on the review, do you have any thoughts on #3939 (comment)

Co-authored-by: Oli Scherer <[email protected]>

…rite

RalfJung · 2024-10-26T19:42:36Z

I think for now making wakeup deterministic with a queue (via VecDeque) is fine. We can leave an issue to randomize this. Regarding the unblocking callback, it is required for correctness right now to run immediately. As long as this does not perform atomic ops (that can be observed from other threads), I think that is fine? What exact issue are you concerned about?

oli-obk · 2024-10-27T07:18:07Z

What exact issue are you concerned about?

well, in this case, that we perform the reads and writes of the unblocked thread immediately, but then continue on the current thread. So unblocking a thread affects the current thread's next operations (there can now be more space to write to, even tho no thread switch happened). This makes no-preemption tests harder to write, but is not a fundamental issue I guess, just weird and not behavior a real system would have

tiif · 2024-10-27T09:12:20Z

that we perform the reads and writes of the unblocked thread immediately, but then continue on the current thread.

Oh that's surprising. I always assumed the current thread should finish first before executing other newly unblocked threads.

But I think I observed what is mentioned here before, and I was pretty confused by the execution sequence.

rustbot added the S-waiting-on-author Status: Waiting for the PR author to address review comments label Oct 4, 2024

tiif force-pushed the blockeventfd branch from 7bbf6fb to fee11aa Compare October 5, 2024 10:17

oli-obk requested changes Oct 5, 2024

View reviewed changes

src/shims/unix/linux/eventfd.rs Outdated Show resolved Hide resolved

src/shims/unix/linux/eventfd.rs Outdated Show resolved Hide resolved

tiif added 10 commits October 13, 2024 23:43

Add test for blocking eventfd read

03fb922

Add eventfd blocking test

0802154

Remove fail-dep test

a5cb910

Remove previously fail-dep comment

e7ffec4

Fix test error

3fef8b5

Add blocking support for eventfd

079cd35

Add and delete comments

e0485e4

Fix rustfmt?

70356b2

Add test for two threads blocked on eventfd, currently deadlocked

88d2ecd

Fix rebase error

37328f8

tiif force-pushed the blockeventfd branch from fee11aa to 37328f8 Compare October 13, 2024 15:51

tiif added 5 commits October 14, 2024 00:11

Split check counter then block out of eventfd::write

4574a11

Split check value and block thread operation out of eventfd::read

cffaf49

Make blocking on several thread works

73f5e46

Apply suggestion: do refcell borrow and push together

128b68c

Add currently failed test

4bcc55c

tiif commented Oct 13, 2024

View reviewed changes

tests/fail-dep/libc/eventfd_block_twice.stderr Outdated Show resolved Hide resolved

tiif added 3 commits October 14, 2024 14:39

Make block then unblock then block again works

9f9813e

Add and remove comments

00aea80

Bless the test

69543c5

This comment was marked as resolved.

Sign in to view

tiif commented Oct 14, 2024

View reviewed changes

src/shims/unix/linux/eventfd.rs Outdated Show resolved Hide resolved

tiif added 2 commits October 14, 2024 15:06

Change name and comment

213885b

Add one more error annotation

18f9e84

tiif force-pushed the blockeventfd branch from bc3f54d to 18f9e84 Compare October 14, 2024 07:18

tiif added 10 commits October 14, 2024 22:45

Try to make the test execution clearer by using thread::park

e5a7771

Add test for read block unblock then block behaviour

0d76514

Improve tests comment

a9a244e

Failed attempt to get rid of reblock

42890a9

It still panic on blocked_write_tid borrow

Revert "Failed attempt to get rid of reblock"

d9dc134

This reverts commit 42890a9.

rustfmt

38707e4

Remove the reblock guard as it is incorrect

733cebc

Fix the BorrowMutError ICE

c4abda9

Bless the tests

41ba16e

Remove todo

64d9cfc

tiif commented Oct 17, 2024

View reviewed changes

Update comments

e14d77e

tiif marked this pull request as ready for review October 18, 2024 05:09

rustbot added S-waiting-on-review Status: Waiting for a review to complete and removed S-waiting-on-author Status: Waiting for the PR author to address review comments labels Oct 18, 2024

oli-obk requested changes Oct 26, 2024

View reviewed changes

RalfJung reviewed Oct 26, 2024

View reviewed changes

src/shims/unix/linux/eventfd.rs Outdated Show resolved Hide resolved

RalfJung reviewed Oct 26, 2024

View reviewed changes

src/shims/unix/linux/eventfd.rs Outdated Show resolved Hide resolved

RalfJung reviewed Oct 26, 2024

View reviewed changes

src/shims/unix/linux/eventfd.rs Outdated Show resolved Hide resolved

tiif and others added 4 commits October 27, 2024 00:50

Improve documentation for mem::take

cb2c1b2

Co-authored-by: Oli Scherer <[email protected]>

Use mem::take

bc13ef1

Co-authored-by: Oli Scherer <[email protected]>

Consistently use mem::take and same documentation for both read and w…

8a25bc2

…rite

Inline previous eventfd_read and eventfd_write

bab1ffe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement blocking eventfd #3939

Implement blocking eventfd #3939

tiif commented Oct 4, 2024 •

edited

Loading

tiif commented Oct 4, 2024

tiif commented Oct 5, 2024

bors commented Oct 8, 2024

This comment was marked as resolved.

tiif Oct 17, 2024 •

edited

Loading

oli-obk Oct 26, 2024

oli-obk Oct 26, 2024

tiif commented Oct 18, 2024

oli-obk left a comment

oli-obk Oct 26, 2024

oli-obk Oct 26, 2024

oli-obk commented Oct 26, 2024

RalfJung commented Oct 26, 2024 via email

oli-obk commented Oct 27, 2024

tiif commented Oct 27, 2024

Implement blocking eventfd #3939

Are you sure you want to change the base?

Implement blocking eventfd #3939

Conversation

tiif commented Oct 4, 2024 • edited Loading

tiif commented Oct 4, 2024

tiif commented Oct 5, 2024

bors commented Oct 8, 2024

This comment was marked as resolved.

tiif Oct 17, 2024 • edited Loading

Choose a reason for hiding this comment

oli-obk Oct 26, 2024

Choose a reason for hiding this comment

oli-obk Oct 26, 2024

Choose a reason for hiding this comment

tiif commented Oct 18, 2024

oli-obk left a comment

Choose a reason for hiding this comment

oli-obk Oct 26, 2024

Choose a reason for hiding this comment

oli-obk Oct 26, 2024

Choose a reason for hiding this comment

oli-obk commented Oct 26, 2024

RalfJung commented Oct 26, 2024 via email

oli-obk commented Oct 27, 2024

tiif commented Oct 27, 2024

tiif commented Oct 4, 2024 •

edited

Loading

tiif Oct 17, 2024 •

edited

Loading