Shopify API Limit Throttling

[jonathan-bird]

Shopify API uses a bucket for api limiting and it'd be super super useful to add it automatically to your calls (get, post, put etc) that it automatically sleeps if it gets close. You can use something like this (except this was used with Guzzle pool, so needs change):

// Respect rate limit
$rateLimit = explode('/', $response->getHeader('x-shopify-shop-api-call-limit')[0]);
$volume = intval($rateLimit[0], 10);
$limit = intval($rateLimit[1], 10);
$limitPercentage = ($volume / $limit) * 100;

// Let user know about success
$this->info(sprintf(
    'Shopify API use at %0.2f%%',
    number_format($limitPercentage)
));

// Throttle sleep
if ($limitPercentage > 70) {
    sleep(1);
}

[oseintow]

Just came back from the Easter break. I know about the request throttling / rate limit. I intentionally left it out for users/developers of this package to handle it themselves. I second what you said. I think its will be useful (never faced throttling issues before cos of the way i consume the api) for those who face these challenges. Will work on it and push.

[jonathan-bird]

Yeah it's definitely an issue when doing bulk. We do tens of thousands of products so any use of the API goes through it.

There's a guzzle rate limit library I'd recommend using or building off: https://github.com/rtheunissen/guzzle-rate-limiter

[Jore]

This would be fantastic. I will need this very soon. :-)

[jonathan-bird]

@oseintow do you need any help?

[oseintow]

@jonathan-bird Can you please work on it for me, am stuck with something so unable to pay full attention to it. Thank you.

[jonathan-bird]

@oseintow Can you please add me as an co-owner? Happy to work on this with you long-term. Would make it much easier.

Also, you forgot to tag the last pull request as 1.0.6 - can you pls do this?

[corykeane]

Any updates here? I can help too if needed.

[oseintow]

@corykeane sure you can work on it.

[hsleewis]

Just curious, is this something that really belongs in the library?

Not every app (or every call) needs the same approach. In a couple apps I solved it with queues because due to the amount of calls it needs to do. And in other I've just built a sleep function whenever the rate limit is close or hit.

[awebartisan]

@hsleewis can you share your approach of sleep function? I love queues but I can't have shell access to client's server, so sleep functionality is the only go, can you share it with us?
Thanks :)

[hsleewis]

@awebartisan Almost all my calls to Shopify are queued. But what I basically do: After every call I save the current call limits reported back by Shopify (per shop and per app) in a Redis database. This key/value expires after 20 seconds. For every new call I do I first do a check with the Redis database and if the number is too low for the store I'm about to do a call for I let the process sleep for a number of seconds and retry.

In some cases this isn't enough as if a process will run too long the webserver can give you a timeout error so for some processes I just put them back in the queue so they will be retried through the queue processor.

[awebartisan]

@hsleewis can you share your code ? gist? it would be very helpful.
I am confused by following :
I have helper classes like ShopifyWebhook , ShopifyProduct, ShopifyPage to make API calls and I have small functions in them which make the relevant API call,

<?php

namespace App\Objects\Shopify;

use Oseintow\Shopify\Facades\Shopify;

class ShopifyProduct {

public function search($query)
{
        return Shopify::setShopUrl(session('domain'))
           ->setAccessToken(session('access_token'))
           ->get('admin/products.json' , ['title' => $query , 'fields' => 'id,title,handle,image']);
}

}

Like above, now , following your approach should I just store the rate limit in after every API call? like, in every function?

[unlockr-ca]

I'm looking for a example of any implementations. Can you guys give us a demo how you manage this issue?