Robinhood Sync Acceleration for PanFS #108
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change-Id: I88c3deb32f977452cbcdb9b0cc3521f1cbc58eaa
Panasas pan_snap_delta update functionality 6 See merge request panasas/robinhood!1
tl-cea
reviewed
Aug 30, 2024
| @@ -2174,11 +2183,42 @@ int path2id(const char *path, entry_id_t *id, const struct stat *st) | |||
| path, strerror(-rc)); | |||
| return rc; | |||
| } | |||
| #ifdef _PANFS | |||
| st = &stn; | |||
There was a problem hiding this comment.
I'd prefer you include this large piece of code in a subfonction.
tl-cea
reviewed
Aug 30, 2024
| @@ -464,10 +468,12 @@ static int TerminateScan(int scan_complete, time_t end) | |||
| g_strfreev(cmd); | |||
| } | |||
| } | |||
|
|
|||
| #ifdef _PANFS | |||
| signal_scan_finished(); | |||
There was a problem hiding this comment.
I don't understand why a scan with PanFS would require a different behavior here.
tl-cea
reviewed
Aug 30, 2024
| @@ -795,6 +801,35 @@ static int stat_entry(const char *path, const char *name, int parentfd, | |||
| if (parentfd != -1) { | |||
| if (fstatat(parentfd, name, inode, AT_SYMLINK_NOFOLLOW) == -1) | |||
| return -errno; | |||
| #ifdef _PANFS | |||
| ssize_t vallentmp; | |||
There was a problem hiding this comment.
I'd prefer this code is included to a subfunction here, to avoid overloading the code.
tl-cea
reviewed
Aug 30, 2024
| @@ -810,6 +845,37 @@ static int stat_entry(const char *path, const char *name, int parentfd, | |||
| if (lstat(path, inode) == -1) | |||
| return -errno; | |||
|
|
|||
| #ifdef _PANFS | |||
| else{ | |||
tl-cea
reviewed
Aug 30, 2024
| @@ -1207,6 +1273,36 @@ static int push_dir_list(robinhood_task_t *parent_task) | |||
| return rc; | |||
| } | |||
|
|
|||
| #ifdef _PANFS | |||
| size_t vallentmp; | |||
tl-cea
reviewed
Aug 30, 2024
| @@ -1246,6 +1342,36 @@ static int process_one_task(robinhood_task_t *p_task, | |||
| "Error accessing filesystem: exiting"); | |||
| Exit(1); | |||
| } | |||
| #ifdef _PANFS | |||
| ssize_t vallentmp; | |||
tl-cea
reviewed
Aug 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request contains changes made this summer by employees at Panasas to provide support akin to what already exists for Lustre to accelerate updates to the Robinhood DB for the Panasas filesystem. Unlike in Lustre, which leverages it's changelog to keep the Robinhood DB up-to-date, in this change we use the PanFS snapshot delta utility to compute differences between successive snapshots, track which was the last snapshot we successfully updated from, and update when a newer snapshot is available using just the computed differences.
In the interest of de-risking existing users of Lustre or just the POSIX version of Robinhood that does a full tree walk, we compile out almost all of our code unless --enable-panfs is passed to configure.
Detailed changes include the following:
Please let me know if there are any issues with the code in this request and I'll be glad to make modifications to alleviate such concerns. Thanks!