Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share memory by copying. #180

Merged
merged 6 commits into from
Oct 31, 2024
Merged

Share memory by copying. #180

merged 6 commits into from
Oct 31, 2024

Conversation

ncruces
Copy link
Owner

@ncruces ncruces commented Oct 30, 2024

"Don't communicate by sharing memory, share memory by communicating."

I'm not entirely convinced this is a good idea.
Still, it repeatedly passes all tests locally and has zero dependencies.
Could probably address #75 (Windows) as well.

@ncruces
Copy link
Owner Author

ncruces commented Oct 30, 2024

Checks for write-write conflicts indicate that there are none, which maybe indicates that this might be a viable, if slow, way of doing this.

I'll add further runtime checks to better understand behavior (what writes happen, and when).

@ncruces
Copy link
Owner Author

ncruces commented Oct 31, 2024

I'm beginning to warm up to this. It looks like a safe, if inefficient, way of keeping memory in sync.

The WAL-index file starts with a header.
This header starts with two 48 byte, checksummed, copies of the same information, which are accessed independently between memory barriers.
The checkpoint information that follows uses 4 byte aligned words.

Finally, we have the WAL-index hash tables, which are only modified holding the exclusive WAL_WRITE_LOCK.

Since all the data is either redundant+checksummed, 4 byte aligned, or modified under an exclusive lock, the copies being done should correctly keep memory in sync.

This looks good enough to submit as is.

Windows can likely reuse significant portions of this. It could trade the in memory locks for file locks, map the file in main memory, and support multiple processes sharing the WAL. If so, care needs to be taken that copies will no longer be made under a lock (they may need to be made on the right side of a file lock).

@ncruces ncruces marked this pull request as ready for review October 31, 2024 11:06
@ncruces ncruces merged commit 17f7840 into main Oct 31, 2024
11 checks passed
@ncruces ncruces deleted the copy branch November 2, 2024 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant