Rate-limiting AggLayer transaction settlement

[iljakuklic]

UPDATE: This document has been updated a couple of times. The initial version remains below for historical reference. Updates are posted in subsequent posts. Please

Version history

• Version 1 (this post below)
  • this is the initial version
• Version 2
  • updated the structure of the config file
  • other smaller changes based on the discussion
• Version 3
  • documents what has been implemented in the initial PR

Rate-limiting AggLayer transaction settlement

Version 1.

This document proposes a mechanism and an interface specification of limiting the rate of L1 transaction settlement AggLayer performs on behalf of participating chains. This is needed to avoid excessive L1 fees.

Requirements

• Networks have to be prevented from settling more often than the configured rate.
• The limit is applied to the two RPC calls that eventually result in posting a transaction on L1:
  • sendTx - limit the number of transactions submitted within given time interval.
  • sendCertificate - limit the number of transactions submitted within a specified number of epochs.
• The settlement rate limits are configurable using a configuration file.
  • There is a configurable default settlement rate.
  • The default rate can be overridden on per-network basis.
• Participating chains are notified in case they have been rate-limited.
  • The response is informative and easily differentiated from other error states.
  • The error information is machine readable to allow for automated handling.

Non-goals:

• It's not supposed to be a general DoS prevention mechanism.
  • Other mechanisms are responsible for this.
• We do not consider the option for participating chains to pay for a more frequent settlement at this stage.
  • This is a longer-term goal for the decentralized version of AggLayer.

Configuration

The rate limit is specified as the maximum number of settlements N over given period T.

The period is specified as:

• wall clock time duration for transaction submissions
• number of epochs for certificate submissions

An example configuration:

# Default rate limiting

[rate-limiting]

# Three per epoch
epoch-limit = 3
epoch-interval = 1

# One per hour
time-limit = 1
time-interval = "1h"

# Configuration overrides for network #1

[networks.1.rate-limiting]

# Max 10 submissions over every 2-epoch window
epoch-limit = 10
epoch-interval = 2

# Max 3 submissions every 45 minute window
time-limit = 3
time-interval = "45min"

The time-limit and time-interval keys apply limits to submit_transaction calls. Similarly, the epoch-limit and epoch-interval keys apply limits to submit_certificate calls.

• The epoch-limit and time-limit keys are natural numbers (including 0) specifying the number of submissions.
• The epoch-interval is a non-zero natural number specifying the epoch count.
• The time-interval is a string giving a time duration within which the limit is enforced. Units such as 3h or 15min are allowed.

Note that setting time-limit or epoch-limit to 0 effectively disables the transaction-based settlement or certificate-based settlement, respectively.

RPC responses

When a network is rate-limited, a response is issued indicating the reason. The jsonrpc 2.0 Error object is used to communicate the request has been rate limited.

Rate-limited sendTx response

{
    "jsonrpc": "2.0",
    "id": CALL_ID,
    "error": {
        "code": -10001,
        "message": "Transaction settlement rate limit exceeded",
        "data": {
            "time-limit": LIMIT,
            "time-interval": INTERVAL,
            "earliest": EARLIEST_FREE,
        }
    }
}

Where:

• CALL_ID is the ID of the call we're responding to
• LIMIT and INTERVAL are the rate limiting parameters configured for this network
  • INTERVAL is an integer specifying the number of seconds
• EARLIEST_FREE is the timestamp at which the current rate limit expires and re-submitting the request is expected to pass the rate limiter

Rate-limited sendCertificate response

{
    "jsonrpc": "2.0",
    "id": CALL_ID,
    "error": {
        "code": -10002,
        "message": "Certificate settlement rate limit exceeded",
        "data": {
            "epoch-limit": LIMIT,
            "epoch-interval": INTERVAL,
            "earliest": EARLIEST_FREE,
        }
    }
}

Where:

• CALL_ID is the ID of the call we're responding to
• LIMIT and INTERVAL are the rate limiting parameters configured for this network
  • INTERVAL is an integer specifying the number of epochs
• EARLIEST_FREE is the epoch number at which the current rate limit expires and re-submitting the request is expected to pass the rate limiter

Open questions

Some discussion points and design alternatives.

Handling of interactions between epoch-based and time-based rate limiting

We have both time limit (for transactions) and epoch limit (for certificates). These potentially interact in subtle ways. In principle, a network should not use both tx submission and certificate submission, but the possibility still has to be handled.

Some options:

• Have the network somehow indicate which mode it is using and disable the other endpoint completely.
• Always have both the time limit and the epoch limit active. Apply time limit to tx submissions and epoch limit to certificate submission. This allows the network to exceed the intended limit if it uses both tx submission and certificate submission. Should the limit be shared?

Option to disable rate limiting

In principle, {time,epoch}-limit = false configuration option could be introduced to disable rate limiting altogether, whether by default or on per-network basis. Should this option be implemented?

Rate limiting configuration flexibility

The configuration section above proposes the rate limit is specified using two parameters, the limit N and interval T.

Note that limiting submissions to 1 in 10min (N=1, T=10min) is not the same as 3 submissions in 30min. Although in the long run, these are expected to have about the same gas consumption.

The design space here is quite rich:

• Simplify it by dropping the N parameter
  • Implicitly always N=1, i.e. spacing of submissions at least T
  • This may be a bit too cumbersome for epoch-based limiting since the resolution of an epoch may end up being too coarse.
• Going in the opposite direction, it can be conceived to have multiple limits
  • E.g. have at most 3 submissions in an hour AND at most 1 every 10 minutes

The answer to this for time-based rate limiting may be different from the answer for the epoch-based limiting.

Changing rate limits online

Should it be possible to change the rate limits without restarting the node? A privileged authenticated admin-level RPC endpoint would be introduced to make it possible.

Implementation notes

Potentially useful libraries and tools:

• jsonrpsee middleware + tower
• humantime crate for time interval parsing

Care must be taken to properly authenticate the network ID before rate limiting is applied to prevent malicious actors from exhausting somebody else's quota.

Conclusion

This is to start a discussion about L1 submission rate limiting. Any comment and/or something missing, please discuss below.

Also please tag anyone who might be interested in this.

[Freyskeyd]

Thanks for the proposal !

Some adjustments that we may need:

epochs

For epoch rate limiting, we do not need to have an epoch window. We want to rate limit per epoch. Currently, CDKs will push a certificate for the next settlement of an epoch and if a certificate is pushed for epoch N, but isn't settled in N, it's going to be settled in N+1.

configs

For configuration, I think that the config example has a little issue on how to handle custom configuration. The general idea is to have the global configuration, that also offer customization per network. Also, tagging that it's for RPCs can be better for the future iteration:

# Default rate limiting

[rate-limiting.rpc.send-certificate]
limit-per-epoch = 1

[rate-limiting.rpc.send-tx]
time-limit = 1
time-interval = "1h"

# Configuration overrides for network #1

[rate-limiting.rpc.network.1.send-tx]
time-limit = 1
time-interval = "15min"

[rate-limiting.rpc.network.1.send-certificate]
limit-per-epoch = 2

---

Note that setting time-limit or epoch-limit to 0 effectively disables the transaction-based settlement or certificate-based settlement, respectively.
In principle, {time,epoch}-limit = false configuration option could be introduced to disable rate limiting altogether, whether by default or on per-network basis. Should this option be implemented?

For the disabling the rate limiting, I think there are two propositions in the post:

• epoch-limit = 0
• epoch-limit = false

I think that the epoch-limit = 0 means that there is no limit applied to the RPC, is that what you meant?

Should it be possible to change the rate limits without restarting the node? A privileged authenticated admin-level RPC endpoint would be introduced to make it possible.

It could be, but not for now, if we need to change the rate limiting we need to change the config and reboot.

---

RPCs error code needs to be documented somewhere too :)

[Freyskeyd]

I'm mixed between the two as for some tools having limit = 0 means you have zero limit, go ahead.
But I agree that's not really clear. Maybe we can do something like:

epoch-limit = 0 # define that the maximum number of certificates is 0 per epoch
enabled = false # define that the rate limiting is disabled (true by default)

Maybe renaming the limit at some places to max could be helpful:

# Default rate limiting

[rate-limiting.rpc.send-certificate]
max-per-epoch = 1

[rate-limiting.rpc.send-tx]
max-per-interval = 1
interval-duration = "1h"

# Configuration overrides for network #1

[rate-limiting.rpc.network.1.send-tx]
max-per-interval = 1
interval-duration = "15min"

[rate-limiting.rpc.network.1.send-certificate]
max-per-epoch = 2

[rate-limiting.rpc.network.2]
enabled = false # No rate limit applied for network 2

[rate-limiting.rpc.network.3.send-certificate]
enabled = false # No rate limit for sendCertificate RPC applied for network 3

[iljakuklic]

Yea I think including max in the key is a good idea.

How about something like this (for epochs)?

Limit the number of certificate settlements per epoch to 7:

[rate-limiting.rpc.send-certificate]
max-per-epoch = 7

Don't limit the number of cert submissions:

[rate-limiting.rpc.send-certificate]
max-per-epoch = "unlimited"

That should be fairly self-explanatory...

[Freyskeyd]

I don't have a strong opinion on the two options. Having two fields is easier to parse (you don't need a custom parser) and it allows quick switch (toggle the feature without losing the numbers), but It is more verbose. I let you decide on that.

[iljakuklic]

Having two fields is easier to parse (you don't need a custom parser)

While I have not tried, I believe something like this will work with the single-field approach without a custom parser:

#[derive(Serialize, Deserialize)]
enum PerEpochLimit {
    Unlimited,
    #[serde(untagged)] Limited(u32),
}

and it allows quick switch (toggle the feature without losing the numbers), but It is more verbose. I let you decide on that.

This is a good point, will think about it.

[iljakuklic]

and it allows quick switch (toggle the feature without losing the numbers), but It is more verbose. I let you decide on that.

This is a good point, will think about it.

Probably still leaning towards the single-key format. It may be slightly confusing to have irrelevant option values in the config file.

Since the rate limiting configuration is light, I think it's fine to just rely on comments if the preservation of previous values is desired, like so:

[rate-limiting.rpc.send-certificate]
# Backing up the previous value and turning off the limit temporarily
#max-per-epoch = 7
max-per-epoch = "unlimited"

That makes it clear that the value of 7 is not used by the software.

[iljakuklic]

Thanks to @Freyskeyd for the comments. There is one point he made that's a bit more general and deserves a separate discussion. It revolves around the structure of the configuration file with respect to per-network overrides.

Basically we have two competing options here. In the following, I assume there is a possibility that there might be other options that are configurable on a per-network basis in the future.

Option A) Grouping the per-network overrides with the option in question

# General options from category A
[categoryA]
option1 = "..."
option2 = "..."

# Category A options overrides for network 1
[categoryA.networks.1]
option1 = "..."
option2 = "..."

# Category A options overrides for network 2
[categoryA.networks.2]
option2 = "..."

# General category B options
[categoryB]
option3 = "..."

# Category B options overrides for network 1
[categoryB.networks.1]
option3 = "..."

Option B) Grouping all the network-specific options into its own section

This hoists the per-network configuration to the top level and groups all the network-specific options together.

This is how the above example could look like. (Please note that the file format is ambiguous since the user can decide how to break the file down into sections. The format below would make sense to me.)

# General options

[categoryA]
option1 = "..."
option2 = "..."

[categoryB]
option3 = "..."

# Overrides for network 1
[networks.1]
categoryA.option1 = "..."
categoryA.option2 = "..."
categoryB.option3 = "..."

# Overrides for network 2
[networks.2]
categoryA.option2 = "..."

Trade-offs

• Option A makes it easier to see network-specific overrides for any given option
• Option B makes it easier to see all the option overrides for any given network
• Option B makes it more natural to introduce network-specific option that are not mere overrides of more general options.

I am in favour of Option B, where the network sub-sections mimick the general part of the config file. It also seems to me more natural fit when adding new network specific configuration.

Thoughts / comments?

[iljakuklic]

How do you represent option B in terms of configuration struct? I mean, how do you make it easily parsable but also usable for internal components?

As observed, the representation of the user-facing config file does not necessarily coincide with the representation suitable for the use by internal components. When that is the case, I'd probably separate the concerns by introducing a separate type for each.

// This is used internally by the rate limiting component.
// It is not Serialize/Deserialize.
struct RateLimitingConfig { ... }

// This is a struct reflecting the structure of the user-facing config file.
#[derive(Serialize, Deserialize)]
struct UserConfig { ... }

impl UserConfig {
    // Get the configuration of the rate limiting component
    fn rate_limiting_config(&self) -> RateLimitingConfig { ... }
}

As a general point, the implementation concerns should not, to the extent possible, leak into the user-facing configuration file format. That's the approach I took (or at least attempted) when thinking about the config layout.

[Freyskeyd]

Sorry if it's not clear, but the agglayer-config represents the configuration file used by the user to define how the node will behave.
If, for any reason, a component needs to have a configuration with extra or internal value, it will use a From<agglayer_config::Config> or a method on its own config struct. The agglayer_config::Config doesn't need to know anything more about the internal of the components than what we want the user to be able to configure.

What I want to point is that we do not aim at having a network centric config but a behavior centric config. As networks are units that are not part of the AggLayer, but clients, they shouldn't be defined as a top level thing. And based on my experience, we do not want to make them more present than that. If we take a step back, the goal of the config is to define the configuration that a node will use. I personally prefer to have everything related to a feature in one place than having to check the configuration of many entities at many places. If we take a look at how it could be defined in YAML instead of TOML:

---
full_node_RPCs:
  "1": http://zkevm-node:8123

rpcs:
  host: 0.0.0.0
  port: 9090

log:
  level: info
  outputs:
    - stderr

L1:
  chain_id: 1337
  node_url: https://rpc-cdk-validium-cardona-03-zkevm.polygondev.tools
  rollup_manager_contract: "0x000..000"

rate_limiting:
  jsonrpc:
    time_interval: 1h
    max_per_interval: 1

    rpcs:
      - send_tx:
         max_per_interval: 1
         time_interval: 1h
         overrides:
           - network_id: 1
             max_per_interval: 10
           - network_id: 2
             time_interval: 2h

      - send_certificate:
          max_per_epoch: 1
          overrides:
            - network_id: 1
              max_per_epoch: unlimited

auth:
  local:
    private_keys:
      - path: /pk/agglayer.keystore
        password: testonly

telemetry: null

outbound:
  rpc:
    settle:
      max_retries: 10
      retry_interval: 1
      confirmations: 3

shutdown:
  runtime_timeout: 5

Either we could group all rate-limiting stuff or group every related jsonrpc config.

I also think that we should stick to the current behavior until we have users reporting issues about the configuration options.
Switching to yaml seems also to be something possible if the configuration becomes complex.

[Freyskeyd]

If this becomes too long, let's discuss this into another discussion, as it is not fully related to this one.

[iljakuklic]

What I want to point is that we do not aim at having a network centric config but a behavior centric config. As networks are units that are not part of the AggLayer, but clients, they shouldn't be defined as a top level thing. And based on my experience, we do not want to make them more present than that. If we take a step back, the goal of the config is to define the configuration that a node will use. I personally prefer to have everything related to a feature in one place than having to check the configuration of many entities at many places.

If grouping by components rather by networks makes more sense here, that's a fair point. I found having the network-based exceptions together with the main configuration a bit cluttered but you have a better idea about the high level requirements and how is the thing going to be used. Ultimately this is a matter of tradeoffs and there is no perfect solution.

[iljakuklic]

Also, tagging that it's for RPCs can be better for the future iteration

Could you elaborate on this? Is there a vision for the rate limiting component to be used for other stuff too?

Also, I am not sure it has all that much to do with RPCs fundamentally. What we are limiting here is the L1 settlement. RPC is the only way currently to initiate a process that may result in L1 settlement. However, if there are potentially other ways to do it, the limit should apply too. As a silly example, let's say we allow the proof submission both over RPC and by e-mail. Both should be subject to the rate limit. Am I missing something here?

[iljakuklic]

Posting here the updated document. It's been modified based on the feedback so far.

Version 2.

This document proposes a mechanism and an interface specification of limiting the rate of L1 transaction settlement AggLayer performs on behalf of participating chains. This is needed to avoid excessive L1 fees.

Requirements

• Networks have to be prevented from settling more often than the configured rate.
• The limit is applied to the two RPC calls that eventually result in posting a transaction on L1:
  • sendTx - limit the number of transactions submitted within given time interval.
  • sendCertificate - limit the number of transactions submitted within a specified number of epochs.
• The settlement rate limits are configurable using a configuration file.
  • There is a configurable default settlement rate.
  • The default rate can be overridden on per-network basis.
• Participating chains are notified in case they have been rate-limited.
  • The response is informative and easily differentiated from other error states.
  • The error information is machine readable to allow for automated handling.

Non-goals:

• It's not supposed to be a general DoS prevention mechanism.
  • Other mechanisms are responsible for this.
• We do not consider the option for participating chains to pay for a more frequent settlement at this stage.
  • This is a longer-term goal for the decentralized version of AggLayer.
• The ability to change the rate limits without restarting the AggLayer node.
  • May be added in the future.

Configuration

The rate limit is specified as the maximum number of settlements N over given period T.

The period is specified as:

• a configurable wall clock time duration for transaction submissions
• one epoch for certificate submissions

An example configuration:

# Default rate limiting

[rate-limiting.rpc.send-certificate]
max-per-epoch = 1

[rate-limiting.rpc.send-tx]
max-per-interval = 1
interval-duration = "1h"

# Configuration overrides for network #1

[rate-limiting.rpc.network.1.send-tx]
max-per-interval = 1
interval-duration = "15min"

[rate-limiting.rpc.network.1.send-certificate]
max-per-epoch = 2

[rate-limiting.rpc.network.2]
# No rate limit applied for network 2
send-tx = "unlimited"
send-certificate = "unlimited"

[rate-limiting.rpc.network.3]
send-certificate = "unlimited" # No rate limit for sendCertificate RPC applied for network 3

• The max-per-interval and max-per-epoch keys are natural numbers (including 0) specifying the number of submissions.
  • Note that setting these keys to 0 effectively disables the transaction-based settlement or certificate-based settlement, respectively.
• The time-interval is a string giving a time duration within which the limit is enforced. Units such as 3h or 15min are allowed.
• The rate-limiting.rpc.send-tx and rate-limiting.rpc.send-certificate keys and their network-specific equivalents can have a special value of "unlimited". It switches the rate limiting off completely for given call.

RPC responses

When a network is rate-limited, a response is issued indicating the reason. The jsonrpc 2.0 Error object is used to communicate the request has been rate limited.

Rate-limited sendTx response

{
    "jsonrpc": "2.0",
    "id": CALL_ID,
    "error": {
        "code": -10501,
        "message": "Transaction settlement rate limit exceeded",
        "data": {
            "max-per-interval": LIMIT,
            "interval-duration": INTERVAL,
            "earliest": EARLIEST_FREE,
        }
    }
}

Where:

• CALL_ID is the ID of the call we're responding to
• LIMIT and INTERVAL are the rate limiting parameters configured for this network
  • INTERVAL is an integer specifying the number of seconds
• EARLIEST_FREE is the timestamp at which the current rate limit expires and re-submitting the request is expected to pass the rate limiter

Rate-limited sendCertificate response

{
    "jsonrpc": "2.0",
    "id": CALL_ID,
    "error": {
        "code": -10502,
        "message": "Certificate settlement rate limit exceeded",
        "data": {
            "max-per-epoch": LIMIT,
        }
    }
}

Where:

• CALL_ID is the ID of the call we're responding to
• LIMIT and INTERVAL are the rate limiting parameters configured for this network
  • INTERVAL is an integer specifying the number of epochs
• EARLIEST_FREE is the epoch number at which the current rate limit expires and re-submitting the request is expected to pass the rate limiter

RPC error codes

The RPC error codes relevant to rate limiting are:

• -10501: transaction settlement limit has been exceeded
• -10502: certificate settlement limit has been exceeded

Open questions

Some discussion points and design alternatives.

Handling of interactions between epoch-based and time-based rate limiting

We have both time limit (for transactions) and epoch limit (for certificates). These potentially interact in subtle ways. In principle, a network should not use both tx submission and certificate submission, but the possibility still has to be handled.

Some options:

• Have the network somehow indicate which mode it is using and disable the other endpoint completely.
• Always have both the time limit and the epoch limit active. Apply time limit to tx submissions and epoch limit to certificate submission. This allows the network to exceed the intended limit if it uses both tx submission and certificate submission. Should the limit be shared?

Implementation notes

Potentially useful libraries and tools:

• jsonrpsee middleware + tower
• humantime crate for time interval parsing

Care must be taken to properly authenticate the network ID before rate limiting is applied to prevent malicious actors from exhausting somebody else's quota.

Conclusion

This is to start a discussion about L1 submission rate limiting. Any comment and/or something missing, please discuss below.

Also please tag anyone who might be interested in this.

[iljakuklic]

Considering a recent discussion, it is probably better to have a single error code for rate limiting, without the distinction between sendTx and sendCertificate. Exact numerical value TBD.

[iljakuklic]

Updated based on what has actually been implemented the pull request so far. Only sendTx limiting is implemented at the time of writing but quite a few updates apply to both sendTx and sendCertificate.

• PR feat: rate limiting for sendTx #303
• PR feat: Rate limiting for sendTx with slot reservation #316
• Issue Implement rate limiting for sendTx #291

Rate-limiting AggLayer transaction settlement

Version 3.

This document proposes a mechanism and an interface specification of limiting the rate of L1 transaction settlement AggLayer performs on behalf of participating chains. This is needed to avoid excessive L1 fees.

Requirements

• Networks have to be prevented from settling more often than the configured rate.
• The limit is applied to the two RPC calls that eventually result in posting a transaction on L1:
  • sendTx - limit the number of transactions submitted within given time interval.
  • sendCertificate - limit the number of transactions submitted within a specified number of epochs.
• The settlement rate limits are configurable using a configuration file.
  • There is a configurable default settlement rate.
  • The default rate can be overridden on per-network basis.
• Participating chains are notified in case they have been rate-limited.
  • The response is informative and easily differentiated from other error states.
  • The error information is machine readable to allow for automated handling.

Non-goals:

• It's not supposed to be a general DoS prevention mechanism.
  • Other mechanisms are responsible for this.
• We do not consider the option for participating chains to pay for a more frequent settlement at this stage.
  • This is a longer-term goal for the decentralized version of AggLayer.
• The ability to change the rate limits without restarting the AggLayer node.
  • May be added in the future.

Configuration

The rate limit is specified as the maximum number of settlements N over given period T.

The period is specified as:

• a configurable wall clock time duration for transaction submissions
• one epoch for certificate submissions

An example configuration:

# Default rate limiting

[rate-limiting.send-certificate]
max-per-epoch = 1

[rate-limiting.send-tx]
max-per-interval = 1
interval-duration = "1h"

# Configuration overrides for network #1

[rate-limiting.network.1.send-tx]
max-per-interval = 1
interval-duration = "15min"

[rate-limiting.network.1.send-certificate]
max-per-epoch = 2

[rate-limiting.network.2]
# No rate limit applied for network 2
send-tx = "unlimited"
send-certificate = "unlimited"

[rate-limiting.network.3]
send-certificate = "unlimited" # No rate limit for sendCertificate RPC applied for network 3

• The max-per-interval and max-per-epoch keys are natural numbers (including 0) specifying the number of submissions.
  • Note that setting these keys to 0 effectively disables the transaction-based settlement or certificate-based settlement, respectively.
• The time-interval is a string giving a time duration within which the limit is enforced. Units such as 3h or 15min are allowed.
• The rate-limiting.send-tx and rate-limiting.send-certificate keys and their network-specific equivalents can have a special value of "unlimited". It switches the rate limiting off completely for given call.

RPC responses

When a network is rate-limited, a response is issued indicating the reason. The jsonrpc 2.0 Error object is used to communicate the request has been rate limited.

Rate-limited sendTx response

If sendTx is blocked by the rate limiter completely:

{
  "code": -10007,
  "data": {
    "rate-limited": {
      "detail": MESSAGE,
      "error": {
        "send-tx-diabled": {}
      }
    }
  },
  "message": "Rate limited"
}

If the limit defined by the rate limiter has been reached and the node has to wait:

{
  "code": -10007,
  "data": {
    "rate-limited": {
      "detail": MESSAGE,
      "error": {
        "send-tx-rate-limited": {
          "max-per-interval": LIMIT,
          "time-interval": INTERVAL,
          "until-next": WAIT_TIME
        }
      }
    }
  },
  "message": "Rate limited"
}

Where:

• CALL_ID is the ID of the call we're responding to
• MESSAGE is a detailed human-readable error message (string)
• LIMIT and INTERVAL are the rate limiting parameters configured for this network
  • INTERVAL is an integer specifying the number of seconds
• WAIT_TIME is the amount of time the client has to wait until submission is expected to pass the rate limiter
  • integer in number of seconds, or null if it cannot be determined

Rate-limited sendCertificate response

{
  "code": -10007,
  "data": {
    "rate-limited": {
      "detail": MESSAGE,
      "error": {
        "send-certificate-rate-limited": {
          "max-per-epoch": LIMIT
        }
      }
    }
  },
  "message": "Rate limited"
}

Where:

• CALL_ID is the ID of the call we're responding to
• MESSAGE is a detailed human-readable error message (string)
• LIMIT is the rate limiting parameter configured for this network specifying max number of submissions per epoch

RPC error codes

The RPC error codes relevant to rate limiting are:

• -10007: settlement limit has been exceeded

Open questions

Some discussion points and design alternatives.

Handling of interactions between epoch-based and time-based rate limiting

We have both time limit (for transactions) and epoch limit (for certificates). These potentially interact in subtle ways. In principle, a network should not use both tx submission and certificate submission, but the possibility still has to be handled.

Some options:

1. Have the network somehow indicate which mode it is using and disable the other endpoint completely.
2. Always have both the time limit and the epoch limit active. Apply time limit to tx submissions and epoch limit to certificate submission. This allows the network to exceed the intended limit if it uses both tx submission and certificate submission. Should the limit be shared?
3. A simple solution might be to only allow appears to be to only allow submitting sendCertificate after the full sendTx rate limiting period has expired, regardless of the limit within the period. Similarly, only allow sendTx in a new sendCertificate epoch, regardles of the limit within the epoch. This is expected to be fairly simple to implement and does not allow the networks to game the rate limiter by switching between sendTx and sendCertificate. It may be a bit too conservative, in the sense that networks might be rate limited more than necessary when switching between sendTx and sendCertificate. However, it is probably good enough for something that is supposed to be an exceedingly rare occurrence.

At this point, (3) seems a good compromise between robustness, accuracy and implementation simplicity.

Implementation notes

Potentially useful libraries and tools:

• jsonrpsee middleware + tower
• humantime crate for time interval parsing

Care must be taken to properly authenticate the network ID before rate limiting is applied to prevent malicious actors from exhausting somebody else's quota.