Reduce storage staking price by 10x

[bowenwang1996]

Storage staking mechanism has been a topic of discussion for a while. There is general agreement that it did not serve the original purpose well as no one seemed incentivized to delete state from smart contracts. However, there is no agreement on what we should do instead. For more details, please refer to #505

A more pressing issue, nevertheless, is starting to plague the NEAR ecosystem. Compared to six months ago, $NEAR price rose more than 7x and since storage staking price is denominated in $NEAR, it has become quite expensive for users and developers. For example, deploying a multisig contract costs ~3N and at today's price, it is more than $20. It not only makes it difficult to attract new developers, but also makes certain patterns (contract factories) prohibitively expensive. While we are taking our time to discuss what to do with the storage staking mechanism more generally, I suggest reducing the storage staking price by 10x as a stop gap solution. It helps neutralize the impact caused by $NEAR price movement in the past six months and allows us more time to come up with a comprehensive solution to revamp storage pricing.

[ilblackdragon]

There are two main alternatives to the current state staking model:

• compression: removing requirement of validators to store the state if was not touched within X days. The compressed state can be rehydrated by user within their transaction (ignored if state already present) or user can request to rehydrate by sending a request with attached cost.
  • Problems to consider: spam of indexers to are still tracking this data as it becomes too cheap to store data if it’s not hydrated.
• multi dimensional fee structure: have a separate fee structure for paying for storage. Track expected state growth per year at XX Gb and adjust price of storing based on how much already stored compared to target. At storage time the price is taken out of attached gas effectively burning the funds.
  • Problems to consider: it’s hard to refund when space is freed up because it would be exploited to fill the space to increase the price and free up or overwrite later (eg gas token on Ethereum)

[akhi3030]

Reducing the staking fee 10x is a reasonable stop gap solution while the NEAR price is high but can make the existing situation worse when the next bear market starts. If we do adopt it, we should also be prepared to prioritise the next set of feature developments to make the overall situation better.

A medium term solution can be to denominate the staking price in XDR essentially a basket of fiat currencies instead of NEAR. I haven't thought through all the edge cases here (and there will be plenty of them) but at a high level, the goal will be that the basic fee for staking does not change regardless of NEAR price movements. If there is interest in pursuing such a solution, I can spend some time writing down how it might work.

I still think that the best long term solution to solving the storage problem is to move to a model where storage is effectively rented. This is also discussed at some high level in #505.

[mfornet]

Reducing the staking fee 10x is a reasonable stop gap solution while the NEAR price is high but can make the existing situation worse when the next bear market starts. If we do adopt it, we should also be prepared to prioritise the next set of feature developments to make the overall situation better.

Agree with @akhi3030. If the price goes 7x again, we will be in a similar situation. If the price goes x/7, it will be 10x cheaper to buy bytes with fiat and increase the state with garbage.

multi dimensional fee structure: have a separate fee structure for paying for storage. Track expected state growth per year at XX Gb and adjust price of storing based on how much already stored compared to target. At storage time the price is taken out of attached gas effectively burning the funds.

Increasing the price could break some contracts that hardcode the current state price. Most factories are in this category.

---

A combination of compression and state rent would be ideal. Individually, I see some fundamental disadvantages of both approaches.

1. state rent: If some state is discarded due to expiration (rented for so long), some invariants in other contracts (including itself) may be broken.
2. compression: It creates some UX friction when hydration is required. Where will the user get the data from? What is the cost of hydration?

Having both compression and state rent removes problem 1): Invariants won't be broken since no state will be discarded completely (due to compression). State rent will provide a way for the maintainer of a contract to keep the state hydrated for as long as they are willing to pay.

[birchmd]

makes certain patterns (contract factories) prohibitively expensive

This specific issue can be addressed if there were a way to store code once but use it for multiple accounts (each account keeping its own context where that code is executed). This "code sharing" idea is something we have talked about a few times before, and maybe it is worth visiting again.

[akhi3030]

I like the idea of compression. I think there two interesting things to investigate here though.

• First, we should check how much does compression help in practice. Given that we are storing a merkle tree with a lot of nodes (which are hashes) I imagine that the compression factor of such data will be low.
• Second, in the ideal world, we would have created the right incentives for the smart contract developers and the node operators that they would explore compressing the data without it needing to be specified in the protocol. I would've imagined just the price of hosting the data and paying the storage staking fee might've been sufficient incentives but it appears that that is not the case.

state rent: If some state is discarded due to expiration (rented for so long), some invariants in other contracts (including itself) may be broken.

I think the impact of this can be reduced by allowing data to go into hibernation before it is actually deleted. The idea is that when a piece of data is about to expire, instead of immediate deletion, the data is still stored on chain for some time (e.g. 1 month) but attempts to access it are denied. Additionally, it should be possible for anyone, not just the contract owner to deposit funds on the account to resume paying the rent.

Essentially the idea is that even if the contract owner goes on vacation or stops paying rent for whatever reason, anybody who still depends on the data is still given a chance to pay for it before the data is actually deleted.

[ilblackdragon]

I suggest we never fully delete data - that is way to complex to model and deal for developers and users. Everything can just be compressed into state root.

There are also two consideration here:

• auto compression, where we just remove things that haven't been touched in X blocks by just keeping the state root of the account
• manual compressed accounts - just specify that account's state will always be compressed, allowing to have massive storage if needed that can be indexed by app developers.

Increasing the price could break some contracts that hardcode the current state price. Most factories are in this category.

Multi dimensional fee structure idea would be about going away from storage staking and replacing with paying for storage during the writes.

Ideally we just remove storage deposit in this case and only charge storage fee as part of gas fee charge. Just have separate cost that accounts for block limit compared to limit for storage writes.

[mfornet]

First, we should check how much does compression help in practice. Given that we are storing a merkle tree with a lot of nodes (which are hashes) I imagine that the compression factor of such data will be low.

compression is a bit of a misnomer here. Nothing is directly compressed. Instead, unused data gets deleted, which is how you "compress" the data on the validators.

The idea is to delete unused data from the validator database but not from the Merkle Tree. Validators will keep only a view of the Merkle Tree. If some subtree has not been used in X months, validators drops the data without modifying the Merkle tree (the root remains the same). You can operate on branches that are in the database. If some operation requires access to a dropped subtree, someone needs to re-send the data again first (what @ilblackdragon mentioned as rehydration).

To measure how practical this approach will be, we should check how much data has been touched (writes/reads) in the last X months (for several values of X). The data on the validators will be proportional to that value. Given the gas limits, there is a hard cap on the state you can be written.

The rehydration process could be expensive (gas-wise) for contracts with large state, since potentially all the data needs to be resubmitted for it to become useful again.

[akhi3030]

thanks for the explanation! I agree then that in my idea of renting storage, we do not need to actually delete any data but instead could follow the above idea and allow users to rehydrate it later.

[lachlanglen]

Is a replacement to the storage staking mechanism necessary right now? It seems like two issues are conflating here, one (the title of this issue) being the significant fiat-equivalent cost of storage on NEAR, in particular when the token price rises, and the other being whether the storage staking mechanism achieves its original intent of aligning incentives and encouraging developers to maintain/reduce their storage footprint.

I would propose separating these discussions, as they really are two independent issues (though of course there is some effective overlap).

With regard to the fiat-equivalent storage cost issue, which I'm not sure most of the proposed solutions actually address, a goal might be to keep the fiat-equivalent cost of storage as consistent as possible even during token price fluctuations, perhaps increasing on some schedule to keep pace with inflation but generally dynamically calculated according to the current/recent market price of $NEAR. Surely this would be more easily implemented than a full rework of the storage staking mechanism, and therefore a more viable solution since this is a pressing, time-sensitive issue that needs to be addressed. Solving this would provide a huge benefit to both developers and users; developers wouldn't have to spend countless hours trying to reduce storage costs byte-by-byte at the expense of usability (the general lack of robust indexing infrastructure for NEAR in many cases requires that data can be efficiently queried directly from contracts as a backup), and users wouldn't have to spend a lot of fiat in order to process basic transactions that store some data on-chain.

env::storage_byte_cost() being a function available at execution time suggests that the rate will be dynamic, and it doesn't necessarily make sense for it to be static given token price fluctuations. It also doesn't make sense to just cut it by 10x, because as others have pointed out this will create issues when the token price falls. Any static solution doesn't make sense for what is a dynamic problem.

Might there be an option to use an oracle system to monitor the $NEAR market price and adjust the current cost per byte of storage accordingly to remove both excessive volatility and expense for users and developers?

[birchmd]

As a working group member, I lean towards rejecting this proposal.

While it is fair to say that the storage staking model has not encouraged developers to delete unused state, I think it is less clear whether the model is successfully discouraging the creation of garbage state in the first place. My personal opinion is that it has been successful in doing this and therefore it is risky to start intentionally undercharging for storage on Near. I understand that the point of this NEP is not to undercharge from the point of view of the fiat cost, however there is no guarantee that the $NEAR price will remain high and if the price returns to its previous value then the storage will be undercharged.

For this reason I think this proposal creates risk that the Near state will grow too quickly for validators to handle. And without a clear timeline to replace the current storage cost model with a new one, I do not think we can tolerate this risk. If there was a clear maximum time (undercharged) storage staking would be live before being replaced then it could be worth reconsidering this interim solution as a way to incentivize development on Near in the short term.

[victorchimakanu]

NEP Status (Updated by NEP Moderators)

Status: Rejected

Protocol Work Group voting indications (❔ | 👍 | 👎 ):

• 👎 @birchmd
• 👎 @mfornet
• 👎 @bowenwang1996
• 👎 @mm-near