The evaluation of blockchain performance metrics has become increasingly misleading in the face of a growing number of blockchain networks. As the landscape expands, there is a pressing need for metrics that focus on efficiency and provide clarity to the public, rather than relying on exaggerated claims. In a conversation with BeInCrypto, Steven Pu, the Co-Founder of Taraxa, highlighted the challenges of accurately comparing blockchain performance. He pointed out that many of the metrics currently in use are based on overly optimistic assumptions rather than concrete, evidence-based results. To address this issue, Pu has proposed a new metric he terms TPS/$.
Why Are Reliable Benchmarks Lacking in the Industry?
With the rise of numerous Layer-1 blockchain networks, the demand for clear differentiation among them has intensified. As developers tout the speed and efficiency of their platforms, it becomes crucial to rely on metrics that effectively highlight their performance. However, the industry still faces a significant shortfall in dependable benchmarks for assessing real-world efficiency, often resorting to fluctuating waves of hype-driven popularity instead. Pu argues that the market is currently inundated with misleading performance metrics, which obscure the true capabilities of various projects. He remarked, “Opportunists can easily exploit the situation by promoting overly simplistic and exaggerated narratives for personal gain. Every conceivable technical concept and metric has been used to hype projects that may not deserve such attention: TPS, finality latency, modularity, and more.”
Pu emphasized that some blockchain projects manipulate TPS figures, leveraging them as marketing tools to present their performance in a more favorable light than reality would suggest.
Understanding the Misleading Aspects of TPS
Transactions per second (TPS) is a commonly used metric that indicates the average number of transactions a blockchain can process and finalize each second under normal conditions. Unfortunately, this metric is often misrepresented to create a distorted perception of a project’s overall performance. Pu noted, “Decentralized networks are complex systems that must be understood in their entirety and within the context of their specific use cases. However, the market frequently oversimplifies and overhypes a single metric while ignoring the broader picture. While a highly centralized network with high TPS might serve certain scenarios with specific trust models, the market lacks an appetite for such nuanced evaluations.”
He pointed out that blockchain projects boasting extreme claims regarding TPS may have compromised other critical factors such as decentralization, security, and accuracy. “For instance, consider TPS. This single metric hides numerous other elements of the network, including how that TPS was achieved and what compromises were made. If I operate one node using a WASM JIT VM, I could easily claim several hundred thousand TPS. By replicating this setup across a thousand machines and calling it sharding, I could then assert hundreds of millions of TPS. By introducing unrealistic assumptions about non-conflicting transactions, one might even argue for billions of TPS. TPS itself isn’t inherently bad, but it should never be viewed in isolation, as there is often a wealth of hidden information behind the numbers,” he elaborated.
The Glaring Gap Between Theoretical and Real-World TPS
To substantiate his arguments, Pu sought to quantify the disparity between the maximum TPS recorded on a blockchain’s mainnet and its theoretical maximum. Among the 22 permissionless and single-shard networks he examined, he discovered an average discrepancy of 20 times between theoretical TPS and actual mainnet TPS. In simpler terms, the theoretical figures were often 20 times higher than what was observed in practice. “The overestimation of metrics like TPS is a response to the speculative and narrative-driven nature of the crypto market. Everyone aims to present their projects in the best possible light, often resorting to theoretical estimates or tests based on unrealistic assumptions to produce inflated metrics. It amounts to dishonest advertising,” Pu stated.
Introducing TPS/$: A More Balanced Performance Metric?
In response to the proliferation of exaggerated metrics, Pu and his team devised a new measure: TPS realized on the mainnet divided by the monthly cost of a single validator node, referred to as TPS/$. This metric evaluates performance based on verifiable TPS achieved in a live environment while accounting for hardware efficiency. The stark 20-fold difference between theoretical and actual throughput prompted Pu to disregard metrics based solely on hypothetical scenarios or lab conditions. He aims to demonstrate how some blockchain projects inflate their performance metrics by depending on costly infrastructure. “Often, performance claims are exaggerated by the use of exceptionally expensive hardware, particularly for networks with centralized consensus mechanisms, where the bottleneck shifts from networking latency to hardware performance. Such reliance on high-cost hardware not only contradicts the principles of decentralized consensus algorithms but also excludes vast segments of the population from participating in the consensus process,” Pu explained.
Pu’s team assessed the minimum hardware requirements for validators across various networks to calculate the cost for each validator node. They then estimated their monthly expenses, focusing on the relative sizing necessary for calculating TPS per dollar ratios. “The TPS/$ metric seeks to address two significant areas of misinformation: it ensures that TPS performance is based on actual mainnet data and highlights the inherent tradeoffs associated with using expensive hardware,” he added.
Permissioned vs. Permissionless Networks: Implications for Decentralization
A blockchain’s security level can be largely determined by whether it functions as a permissioned or permissionless network. Permissioned blockchains restrict access and participation to a specific group of users, requiring approval from a central authority. In contrast, permissionless blockchains allow anyone to join. Pu argues that the permissioned model contradicts the core tenets of decentralization. “A permissioned network, where validation is controlled by a single entity, or if there is just one operator, as seen in many Layer-2 solutions, offers insight into the network’s degree of decentralization. A defining characteristic of decentralization is its capacity to bridge trust gaps. When you remove decentralization, the network essentially becomes a cloud service,” he stated.
Paying attention to these metrics is crucial over time, as networks with centralized governance may be more susceptible to certain vulnerabilities. “In the long run, we require a standardized set of attack vectors for Layer-1 infrastructure that can expose weaknesses and tradeoffs inherent in any architectural design. Many of today’s mainstream Layer-1 networks make unreasonable sacrifices regarding security and decentralization. These traits are often invisible and difficult to detect until a catastrophe occurs. I hope that as the industry matures, we will see the emergence of a standardized battery of tests,” Pu added.
State-Sharding vs. Single-State: Insights into Data Management
In the context of blockchain performance, latency refers to the delay between transaction submission, confirmation, and its incorporation into a block on the blockchain. It indicates how quickly a transaction is processed and becomes a permanent entry in the distributed ledger. Analyzing whether a network uses state-sharding or maintains a single state can provide insight into its latency efficiency. State-sharded networks segregate the blockchain’s data into multiple independent segments called shards, with each shard operating semi-independently without immediate access to the complete state of the entire network. Conversely, a non-state-sharded network shares a single state across the network, allowing all nodes to access and process the complete data set.
Pu acknowledged that while state-sharded networks aim to enhance storage and transaction capacity, they often experience longer finality latencies due to the need to process transactions across multiple independent shards. He also pointed out that many projects that adopt a sharding approach inflate their throughput claims by merely replicating their networks rather than developing an integrated and scalable architecture. “A state-sharded network that lacks a shared state is simply creating unconnected copies of itself. If I replicate a Layer-1 network across a thousand independent instances, it is misleading to sum the throughput across these copies and present it as a single network’s performance. Some architectures do synchronize states and shuffle validators across shards, but many projects that make extravagant throughput claims are merely duplicating their networks,” Pu explained.
What Fundamental Changes Are Needed in Blockchain Evaluation?
Pu’s observations highlight a pressing need for substantial changes in how blockchain projects are assessed, financed, and ultimately succeed within the Layer-1 space, where misleading performance metrics vie for attention. Establishing reliable and effective benchmarks is critical for countering these false representations. “You can only measure what you can observe, and right now in crypto, the figures often resemble hype narratives rather than objective measurements. Standardized, transparent metrics facilitate straightforward comparisons across different products, allowing developers and users to understand what they are utilizing and the tradeoffs involved. This is a hallmark of any mature industry, and we still have significant progress to make in crypto,” Pu concluded.
Implementing standardized and transparent benchmarks will enable informed decision-making and promote genuine advancement beyond mere promotional claims as the industry evolves.
Disclaimer: This article presents opinions and insights from industry experts. BeInCrypto aims for transparent reporting, but the views expressed do not necessarily represent those of BeInCrypto or its staff. Readers are encouraged to verify information independently and consult with professionals before making decisions based on this content. Additionally, our Terms and Conditions, Privacy Policy, and Disclaimers have been updated.
