How Hedera Ensures Quality in Releases of the Hiero Consensus Node Software

3d ago•

bullish:

bearish:

By: Ed Marquez

https://hedera.com/blog/how-hedera-ensures-quality-in-releases-of-the-hiero-consensus-node-software

Why Testing Matters

The Hedera mainnet is a public network that is mission-critical infrastructure. Applications, enterprises, and users worldwide rely on it for stable, secure, and high-performance operations. New features or updates to the Hiero Consensus Node software undergo thorough testing to maintain the enterprise-grade reliability that Hedera provides. The network and its contributors are committed to quality assurance. Thus, rigorous testing is a fundamental discipline that safeguards every update before it ever reaches mainnet. This article provides a deep dive into the testing strategy that Hedera follows and how it supports the release cycle.

Testing Strategy and Release Cycle

Hedera’s structured release process and multi-layered testing approach ensure that each code change is thoroughly vetted. New features to the Hiero Consensus Node codebase pass through successive stages of validation before they are considered “production-ready.”

Engineers perform continuous integration (frequent integration of changes) and run extensive automated test suites to catch issues early. These include unit tests, integration tests, performance benchmarks, longevity runs, and even chaos simulations, all described in detail in this article. Only once the code passes QA gates is it rolled out to the Hedera public networks in phases: first to previewnet, then to testnet, and finally to mainnet. This phased progression allows the community to experiment with and validate new functionality on previewnet and testnet, providing crucial feedback or catching any remaining edge cases in a safe setting. By the time a release reaches mainnet, it has been tested through multiple layers of automated QA and real-world trials, giving developers and users confidence in each upgrade.

For a detailed look at the release process, environments, and timeline, see the Hedera Release Cycle Overview article.

Longevity Testing

This testing involves running the node software under continuous heavy load for an extended period (often a full week or more) to prove its stability and resilience over time. Bugs like memory leaks, gradual performance degradation, or rare timing issues might not appear in short test runs, so the engineers literally stress-test the network for days. This stress-testing is done on mainnet-like nodes distributed geographically to better reflect real-world conditions. During a longevity test, this test network of nodes is loaded with transactions at high throughput 24/7, simulating worst-case production conditions non-stop. Often, longevity tests are run for seven days straight without taking a break.

This intense soak test helps ensure that the consensus nodes can run reliably for long periods without crashing, slowing down, or accumulating errors. Engineers treat any issue uncovered in longevity testing with high urgency. If a node crashes or computes an incorrect result during the run, the issue is flagged with critical priority to fix it immediately before any release proceeds.

In short, longevity testing provides the confidence that when the code is deployed to mainnet, it will handle continuous operation and real-world traffic volumes over weeks and months.

Performance Testing (Throttled & Unthrottled)

Another vital aspect of Hedera’s testing strategy is performance testing, where engineers measure how the network performs under various throughput levels and transaction mixes. Hedera is known for its high throughput. In testing, the engineers push those limits to understand the true capacity and to ensure plenty of headroom. Throttled performance tests run the system at or near the mainnet’s configured TPS limits to verify that new features don’t introduce latency or slowdowns under normal expected load. Equally importantly, unthrottled performance tests intentionally disable or raise these limits in a controlled environment to push the software to its maximum throughput. Hedera’s internal performance runs have achieved very high figures. For instance, testing scenarios have resulted in the order of many tens of thousands of operations per second, far above the current production limit of 10,000 TPS.

By testing “way above” the safe production levels, the team ensures that mainnet’s configured limits are well within the system’s true capabilities, and that any performance bottlenecks are discovered and addressed early.

These tests monitor metrics like transaction latency, consensus finalization time, CPU/memory utilization, and throughput sustainability. If a code change causes a regression, say a certain transaction now takes 10% longer to process, the team can catch it before release. The result of this exhaustive performance testing is a high-performance network where the mainnet operates with a comfortable safety margin, and any future scale requirements (higher TPS or bursty workloads) are already anticipated by prior unthrottled tests. In summary, Hedera’s performance testing guarantees that changes and new features maintain the network’s speed and scalability, and that the mainnet’s high throughput remains rock-solid.

Regression Testing

Every new release of the Hiero Consensus Node must continue to support the rich feature set and APIs that developers rely on. Regression testing is how Hedera ensures that existing functionality remains correct and unaffected by new changes. The development team maintains an extensive automated regression test suite encompassing everything from low-level unit tests to full end-to-end scenarios. In fact, during the v0.49 upgrade that introduced Modularization, the team wrote and ported thousands of unit tests to cover the refactored modules. These unit tests exercise individual functions and classes in isolation, verifying edge cases and making sure that bug fixes from the past stay fixed. Thousands of end-to-end tests of the Hedera API (HAPI) were executed in that single release to verify backwards compatibility. This means that for each change, the team checks that all previous behaviors (and error conditions) still work exactly as expected. Any deviation flags a regression that must be corrected or explicitly accepted. Regression testing covers functional correctness, but also compatibility and performance baselines (e.g., ensuring a new release doesn’t use more memory or produce different fee results than prior versions, unless intended). All of this testing is automated and integrated into the CI pipeline (running on every pull request or build). The vast majority of tests are designed to be deterministic and fast, giving quick feedback. For example, hundreds of thousands of unit tests run in a few minutes in CI. By requiring all these tests to pass, Hedera prevents regressions from slipping into a release. In short, the regression test suite provides a safety net so that innovation never comes at the expense of stability or compatibility for existing features.

Integration Tests

In addition to low-level tests, Hedera employs automated integration tests (sometimes called end-to-end tests or acceptance tests) to validate how the system’s components work together in real scenarios. Whereas a unit test might verify one module in isolation, an integration test can spin up multiple parts of the system (or even a full local network) and simulate a sequence of operations as a client would. Hedera’s open-source codebase actually includes a suite of Behavior-Driven Development (BDD) style test clients that orchestrate complex scenarios, like creating a token, associating accounts, performing transfers, querying mirror nodes for confirmation, etc. These scenarios cover various Services (accounts & HBAR, consensus, token, smart contracts) in an integrated fashion.

Automated integration tests often run against ephemeral networks (for instance, a set of Dockerized nodes or an in-memory network) or against testnet-like environments. They verify not only correctness (e.g., a token transfer succeeds and final balances match expected values) but also that different services interoperate seamlessly. Integration tests are particularly useful for catching issues that only appear when multiple features interact. For example, a test might create a smart contract that calls the Hedera Token Service, ensuring that smart contracts and tokens work together properly. By automating end-to-end use cases, integration tests provide confidence that a new software version will function correctly in the real world. They serve as a dry run for how developers and applications will experience the network after the update.

Chaos Testing

It’s not enough for Hedera’s consensus nodes to be fast; they also must be resilient in the face of failures or adverse conditions. That’s where chaos testing comes in. Hedera engineers deliberately introduce failure scenarios and network chaos to ensure the network can recover gracefully. For example, during high-throughput performance tests, the team will start randomly killing nodes or severing network connections, verifying that the system reacts appropriately.

The expectation (and result) is that even with some nodes going offline unexpectedly, the remaining nodes continue processing transactions, and the offline nodes automatically catch up via the “reconnect” protocol when they come back. Chaos tests validate this by simulating worst-case scenarios: imagine a node drops out for a few minutes while the network is handling 10,000 TPS. That node could miss hundreds of thousands of transactions. Hedera’s software includes a robust reconnect mechanism to sync the node with missed events, and chaos testing verifies that this works under real load.

Chaos testing can involve injecting other faults: high latency, out-of-order messages, or even malicious-like behavior. Remember that Hedera’s consensus algorithm, Hashgraph, is asynchronous Byzantine Fault-Tolerant (aBFT) by design. By conducting these trials in a controlled environment, the engineers ensure that any weakness in fault recovery is identified and fixed long before the code reaches mainnet.

The takeaway is that the engineers literally try to break the network in the lab so that it won’t break in production. This chaos engineering discipline gives Hedera its enterprise-grade robustness. Engineers and users can trust that the network has been proven to withstand node outages, network partitions, and other chaos with zero data loss and minimal impact on performance.

Continuous Integration and the CI/CD Pipeline

A Continuous Integration/Continuous Deployment (CI/CD) pipeline that automates QA and ensures safe rollouts is foundational to all of the test practices mentioned so far. Every code change in the Hiero Consensus Node repository triggers the CI pipeline to run a battery of tests and checks. For example, when a developer opens a pull request, the pipeline automatically compiles the code, runs the full suite of unit tests, followed by some integration tests, and other static analysis, and so on. Only if these CI tests all pass can the change be merged. This prevents unstable code from ever entering the main branch. The pipeline provides rapid feedback to developers; if a test fails, developers are immediately alerted to investigate, driving a “fail fast, fix fast” culture.

The CI/CD pipeline uses infrastructure-as-code and containerization techniques to ensure that the deployment process is consistent and reproducible across previewnet, testnet, and mainnet. Critical steps (like promoting a build to testnet or mainnet) are, of course, closely managed and require approval by release teams and the Governing Council, but the groundwork of compiling artifacts, running final regression suites, packaging the release, etc., is largely automated for reliability. This automation minimizes human error and allows the team to push out fixes or patches quickly when needed, with high confidence.

In summary, Hedera’s CI/CD pipeline is the engine that facilitates quality and agility. It automates testing and deployment, provides quick feedback on code health, and orchestrates the safe and smooth promotion of code through various environments. This enables the engineering teams to iterate fast on new features while maintaining the assurance that each deployment is safe.

Release Progression: Previewnet to Mainnet

As explained in the Hedera Release Cycle Overview article, new features don’t just go straight to the mainnet after passing all internal QA gates. They first must go through real-world test environments. Hedera provides three public networks (previewnet, testnet, mainnet) that align with stages of the release lifecycle. Together, these network environments enable a phased rollout of new Hedera features to the live production environment (mainnet). New features are often in development for many weeks and months before becoming available on previewnet, and then testnet. The goal is for major new features to be available on testnet for a long enough period of time to allow developers to adjust to these new capabilities before seeing them on mainnet.

Hedera provides public release notes, maintenance schedules via the Status Page, and tools like the Hedera Status API so developers can prepare for upgrades. The net result is a disciplined release cycle where quality assurance is baked in at every step.

Summary

The Hedera release cycle is designed to deliver innovation steadily and safely. As a developer, you can align your development cycle with that of Hedera: keep an eye on upcoming releases, test early on previewnet and verify on testnet, and confidently deploy to mainnet. Be sure to follow the release notes and network announcements to stay up-to-date with the latest upgrades. New features and improvements are constantly in the works, and now you know exactly where to find them first, and when they’ll arrive on mainnet. Stay tuned, stay involved, and happy building on Hedera.

Links

GitHub Repositories

Hedera Resources

Network Monitoring & Status

Community & Social

Linux Foundation / Hiero Participation

Linux Foundation Zoom Meetings for Hiero

How Hedera Ensures Quality in Releases of the Hiero Consensus Node Software was originally published in Hedera Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

3d ago•

HBAR

bullish:

bearish:

Hedera

-0.46%

$0.2021

Manage all your crypto, NFT and DeFi from one place

Securely connect the portfolio you’re using to start.

Hedera taps crypto custodian Copper to expand institutional access to HBAR tokens

bullish:

bearish: