Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for Retry in WP CLI Core Download Command #258

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add Support for Retry in WP CLI Core Download Command #258

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
161eb20
Add retry and retry-delay in core download cli
abhi3315 Apr 26, 2024
742714d
Exit early from the retry loop if the status is ok or 404
abhi3315 Apr 26, 2024
76c4f61
Fix phpcs code sniffing issue
abhi3315 Apr 26, 2024
dd81ca8
Add max retry to 5 and remove control of retry-delay
abhi3315 Apr 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 57 additions & 15 deletions src/Core_Command.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,12 @@ public function check_update( $_, $assoc_args ) {
* [--version=<version>]
* : Select which version you want to download. Accepts a version number, 'latest' or 'nightly'.
*
* [--retry=<retry>]
* : Number of retries in case of failure. Default is 0.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think by default it can be 3. Since this retry functionality is in place, a default would be better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this part though. If it's just user opt-in, then makes sense to keep it 0?

*
* [--retry-delay=<retry-delay>]
* : Delay between retries in seconds. Default is 5 seconds.
*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given our conversation on Zoom today, I think we should change this slightly.

  • Within wp core download, I think we should only have --retry=<count> as an argument. This will let us be opinionated about exponential backoff.
  • We can update Utils\http_request() to support 'retry' and 'retry_delay' as arguments.
  • The --retry=<count> argument should be applied to all HTTP requests within wp core download.

If WordPress.org is down in some manner, it's not helpful for WP-CLI users to keep hammering it with requests. Their systems should be tolerant to this fault. However, --retry=<count> with exponential (or linear) backoff (5 to ~30 seconds) and a maximum count seems like a good compromise.

We should have some tests!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danielbachhuber As discussed in the call, we decided to wrap the whole try-catch block inside the do-while loop to handle the retry feature. But after taking a deeper look into it there are many early and exception throwing that may break the retry loop in between.

We should keep the retry loop in the download command only because updating the http_request util will affect other commands as well.

* [--skip-content]
* : Download WP without the default themes and plugins.
*
Expand Down Expand Up @@ -290,28 +296,64 @@ function () use ( $temp ) {
'insecure' => $insecure,
];

$response = Utils\http_request( 'GET', $download_url, null, $headers, $options );
$retry = (int) Utils\get_flag_value( $assoc_args, 'retry', 0 );
$retry_delay = (int) Utils\get_flag_value( $assoc_args, 'retry-delay', 5 );

if ( 404 === (int) $response->status_code ) {
WP_CLI::error( 'Release not found. Double-check locale or version.' );
} elseif ( 20 !== (int) substr( $response->status_code, 0, 2 ) ) {
WP_CLI::error( "Couldn't access download URL (HTTP code {$response->status_code})." );
}
do {
$response = Utils\http_request( 'GET', $download_url, null, $headers, $options );
$status_code = (int) $response->status_code;
$is_response_ok = $status_code >= 200 && $status_code < 300;

// Exit the loop if the response is OK or the status code is 404.
if ( $is_response_ok || 404 === $status_code ) {
$retry = 0;
}

if ( 404 === $status_code ) {
WP_CLI::error( 'Release not found. Double-check locale or version.' );
} elseif ( ! $is_response_ok ) {
WP_CLI::warning( "Couldn't access download URL (HTTP code {$response->status_code})." );
if ( 0 < $retry ) {
WP_CLI::log( "Retrying in {$retry_delay} seconds..." );
sleep( $retry_delay );
}
}
} while ( 0 < $retry-- && ! $is_response_ok );

if ( 'nightly' !== $version ) {
unset( $options['filename'] );
$md5_response = Utils\http_request( 'GET', $download_url . '.md5', null, [], $options );
if ( $md5_response->status_code >= 200 && $md5_response->status_code < 300 ) {
$md5_file = md5_file( $temp );
$retry = (int) Utils\get_flag_value( $assoc_args, 'retry', 0 );

if ( $md5_file === $md5_response->body ) {
WP_CLI::log( 'md5 hash verified: ' . $md5_file );
do {
$md5_response = Utils\http_request( 'GET', $download_url . '.md5', null, [], $options );
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just discussed the retries with @schlessera and the Requests library does provide a way to hook options in curl - https://requests.ryanmccue.info/docs/hooks.html#:~:text=Requests%5Crequest_multiple().-,curl.before_request,-Set%20cURL%20options
#259

Not sure though if we should add it by default in Utils\http_request(). Thoughts @schlessera?

$status_code = (int) $md5_response->status_code;
$is_response_ok = $status_code >= 200 && $status_code < 300;

// Exit the loop if the response is OK or the status code is 404.
if ( $is_response_ok || 404 === $status_code ) {
$retry = 0;
}

if ( $is_response_ok ) {
$md5_file = md5_file( $temp );

if ( $md5_file === $md5_response->body ) {
WP_CLI::log( 'md5 hash verified: ' . $md5_file );
} else {
WP_CLI::error( "md5 hash for download ({$md5_file}) is different than the release hash ({$md5_response->body})." );
}
} elseif ( 404 === $status_code ) {
WP_CLI::warning( 'md5 hash not found for release.' );
} else {
WP_CLI::error( "md5 hash for download ({$md5_file}) is different than the release hash ({$md5_response->body})." );
WP_CLI::warning( "Couldn't access md5 hash for release ({$download_url}.md5, HTTP code {$md5_response->status_code})." );

if ( 0 < $retry ) {
WP_CLI::log( "Retrying in {$retry_delay} seconds..." );
sleep( $retry_delay );
}
}
} else {
WP_CLI::warning( "Couldn't access md5 hash for release ({$download_url}.md5, HTTP code {$md5_response->status_code})." );
}
} while ( 0 < $retry-- && ! $is_response_ok );

} else {
WP_CLI::warning( 'md5 hash checks are not available for nightly downloads.' );
}
Expand Down