Why NEAR Data Availability?

NEAR Protocol is built to onboard 1B+ users to Web3. Since its mainnet launch in October 2020, NEAR Protocol has achieved 100% uptime with 4 shards and has onboarded 35M accounts, of which 16M are monthly active accounts.

Supporting the most users of any network in Web3 requires a scalable blockchain that is capable of having both high data availability and low-cost data posting in order for applications to scale to the level of mainstream global adoption. This post will explain NEAR’s technical approach to data availability and how NEAR DA, a solution designed for Ethereum rollups, will scale the modular ecosystem and defragment all of Web3.

The modular Ethereum ecosystem seeks to scale the network by separating the functions of a blockchain––settlement, execution, and data availability––into separate layers. The Ethereum layer-one simply could not scale to accommodate the level of demand it reached, particularly during bull markets, without prohibitively high transaction fees. This demand becomes even more problematic when users want to do swaps on Ethereum or post transaction data for layer-twos to the layer-one network. 

NEAR DA offers a solution to this scalability roadblock. With a simple architecture that is easy to implement and constantly improving efficiency, and thanks to NEAR having the cheapest transaction fees of any network in Web3, NEAR DA is an incredibly fast and cost-effective data availability option. Depending on the gas fees, NEAR DA is up to 85,000x cheaper than posting blob submissions on Ethereum and 30x cheaper than doing the same on Celestia. 

NEAR DA and the Chain Abstraction Vision

The NEAR ecosystem’s focus for 2024 is expanding chain abstraction across more chains & ecosystems, with the goal of abstracting the blockchain away from the user & developer experience. While the modular Ethereum ecosystem has made Ethereum more scalable in some ways, it’s also resulted in a lot of incompatible infrastructure and fractured liquidity that places a huge burden on app builders and end users to manage. The chain abstraction vision is that apps should just work with all chains, onboarding should be seamless, and things like gas fees and bridging should become invisible to end users. 

Data availability and NEAR DA play an important role in chain abstraction. As zero-knowledge technology enables unification of security across chains via state proofs, as described in the chain abstraction deep dive post linked above, settlement data will be easier to fetch from different networks. As dapps become truly multichain thanks to account aggregation and decentralized frontends, they can query data from multiple chains more easily when data is all posted in one place. Doing so on NEAR will be much cheaper for rollups.

How is NEAR DA Implemented?

NEAR DA leverages an important part of NEAR’s consensus mechanism, known as Nightshade, which parallelizes the network into multiple shards (essentially, multiple parallel blockchains). Each shard on NEAR produces a small portion of a block, which is called a chunk. These chunks are aggregated to produce blocks. All of this happens entirely at the protocol level and so is invisible to users and developers.

NEAR DA uses this infrastructure to an ETH rollup’s benefit. When a chunk producer processes a receipt, there is consensus around the receipt. However, once the chunk has been processed and included in the block, the receipt is no longer required for consensus and can be pruned from the blockchain’s state. The pruning time is at least 3 NEAR epochs, where each epoch is 12 hours. In practice, this is usually around 5 NEAR epochs, so data is available in the network for around 60 hours. Once the receipt is pruned, it is the responsibility of archival nodes to retain the transaction data. 

This means that NEAR doesn’t slow down its consensus with more data than it requires, yet any user of NEAR DA would have ample time to query transaction data. The advantage this architecture provides to rollups is cost-effective data availability, especially to those with high transaction volume, such as gaming chains. 

NEAR DA provides an out-of-the-box solution for developers, making it easy for rollup builders to make the most of NEAR’s infrastructure––which has 3+ years of proven reliability behind it. The Pagoda engineering team has already built three important components that are open-source and ready to be integrated into any OP Stack, Polygon CDK, or Arbitrum Nitro rollups:

• The Blob Store Contract: A contract that stores arbitrary data.
• Light Client: A trustless off-chain client that provides easy access to transaction and receipt data.
• RPC Client: A client to publish data blobs to the NEAR Protocol.

Rollup builders looking to work with a cloud-like service could also leverage NEAR DA’s rollup-as-a-service (RaaS) providers (list provided here), in order to utilize these three components all in one place in a more abstracted manner. 

What’s next? The Future of NEAR DA and Data Availability Sharding

Another advantage NEAR’s sharding approach offers to users of NEAR DA is the future scaling roadmap. The NEAR Protocol Engineering team recently announced the move towards stateless validation, the next phase of sharding. This will further decrease the hardware requirements of certain types of validators (chunk validators) and move the state into memory, which will allow for more shards and increase the decentralization in the system by lowering the requirements to become a validator in the system. 

This means that the number of shards will increase, greatly increasing the overall throughput of the NEAR Protocol. While NEAR is already fast, with 4 MB/s per shard with 4 shards, scaling towards n shards means that rollups, applications, and any other project or developer using NEAR will not have to compete for blockspace. This is a major contrast to the scaling limits of both monolithic systems as well as the modular ETH ecosystem and offers a substantial future-proof for anyone building on NEAR, including users of NEAR DA.

As the NEAR Protocol starts increasing the number of shards to n, the amount of data that has to be stored for a single shard decreases. Eventually, in theory every account on NEAR Protocol could potentially become its own shard. This allows running very light RPC nodes that would only track the shard that a given user is interested in. 

For L2s that use NEAR DA, this would mean running an in-memory RPC client that would only track the shard that your data blob contract is in for your retention period (for fraud proofs), or for a given interval of time (retention is configurable on the RPC nodes). Instead of relying on data availability sampling (DAS), a rollup can rely on its own shard by running a lightweight RPC through Data Availability with Sharding. The additional benefit here is that the RPC is also applying the chunks they are tracking, essentially acting as a validator without participating in the consensus, as opposed to statistical confidence with DAS. 

While Data Availability Sharding remains in the research phase of development, it shows the major advantages the NEAR Protocol design offers to many kinds of builders and ecosystems. As the Web3 space pursues mainstream adoption, the scalability hurdles of the first ten years of Ethereum will seem minor, and it is not clear that modularity alone will solve the problem. The NEAR ecosystem’s goal of mainstream adoption of Web3 means the network is available for all builders to use for creating great applications and delivering the next generation of the internet.

Fast, Cost-Effective DA is NEAR

As Web3 adoption grows and as the number of rollups continues to climb, the need to scale the modular scalability solutions becomes greater. NEAR offers its innovative, reliable technology as a data availability solution for rollups that can save time and money while future-proofing projects for an increasingly multichain (and cross-chain) future. The bleeding-edge research driving the future development of NEAR DA will provide rollups with even more advantages over time. Interconnecting and defragmenting Web3 via chain abstraction will be a major theme of 2024 and utilizing NEAR DA puts rollups at the forefront of this evolution.