Skip to content

Google Consensus Test

2024-06-19 - 1,572,868 validators

We thoroughly tested our consensus mechanism with a large number of validators to analyze how the network scales as their count increases.

We conducted two key tests:

  • A preliminary test in a single Google data center to assess initial performance.
  • A main test across multiple global regions to evaluate decentralization and network resilience.

Preliminary Test

The primary objective of this test was to determine the minimum hardware requirements for the second, larger-scale test. This was necessary due to Google’s logical limitations on the number of servers that can be deployed simultaneously.

Test Setup

The following server configurations were used:

  • 128 servers hosting a total of 12,288 validators each:

    • Machine type: e2-highcpu-32
    • vCPUs: 32
    • Memory: 32 GB

  • 4 servers hosting 256 validators each:

    • Machine type: e2-highmem-4
    • vCPUs: 4
    • Memory: 32 GB

  • 4 servers hosting 1 validator each:

    • Machine type: e2-standard-4
    • vCPUs: 4
    • Memory: 16 GB

Test Execution & Results

The network operated with the consensus mechanism for 2 hours before being shut down.

In total, the network successfully ran 1,573,892 validators, calculated as follows: (128 × 12,288) + (4 × 256) + (4 × 1) = 1,573,892

Main Test

After completing the preliminary test, we proceeded with the main test.
To run 256 validators per node, a minimum of 32 GB RAM and 8 vCPUs was required.
However, since our goal was to deploy over 6,000 nodes, we encountered Google’s resource limitations, making it impossible to allocate the necessary infrastructure.

To overcome this challenge, we decided to take a risk by launching nodes with 4 vCPUs and 32 GB RAM instead.

Test Setup

The following server configurations were deployed:

  • 6,144 servers hosting 256 validators each:

    • Machine type: e2-highmem-4
    • vCPUs: 4
    • Memory: 32 GB

  • 4 servers hosting 1 validator each:

    • Machine type: e2-standard-4
    • vCPUs: 4
    • Memory: 16 GB

The servers were evenly distributed across the following regions:

Region list 
europe-west1-c - Belgium, St. Ghislain
europe-west3-c - Germany, Frankfurt
europe-west4-c - Netherlands, Eemshaven
europe-west6-a - Switzerland, Zurich
europe-west2-c - UK, London
europe-north1-a - Finland, Hamina
us-east1-c - USA, South Carolina, Moncks Corner
us-east4-c - USA, Northern Virginia, Ashburn
us-west1-c - USA, Oregon, The Dalles
us-west2-c - USA, California, Los Angeles
us-west3-c - USA, Utah, Salt Lake City
us-west4-c - USA, Nevada, Las Vegas
europe-west10-c - Germany, Berlin
southamerica-east1-c - Brazil, São Paulo
asia-east1-c - Taiwan, Changhua County
asia-northeast1-c - Japan, Tokyo
asia-south1-c - India, Mumbai
northamerica-northeast1-c - Canada, Montreal
northamerica-northeast2-a - Canada, Toronto
southamerica-west1-b - Chile, Santiago
me-central1-c - Qatar, Doha
me-central2-c - UAE, Abu Dhabi
me-west1-c - Israel, Tel Aviv
africa-south1-c - South Africa, Johannesburg
europe-southwest1-c - Spain, Madrid

Test Execution and Results

The network operated with the consensus mechanism for 2 hours before being shut down.

A total of 1,572,868 validators were successfully deployed, calculated as follows: (6,144 × 256) + (4 × 1) = 1,572,868

Additionally, up to 1 million transactions were generated and sent during the test.

Resource Consumption Measurements

We measured resource consumption in three phases:

  1. Idle state (before network activation)
  2. During network operation

Nodes with 256 Validators

Before network start:
Before - Node 1
Before - Node 5121

After network start:
After - Node 1
After - Node 5121

Nodes with 1 Validator

Before network start:
Before - Node 6144
Before - Node 6145

After network start:
After - Node 6144
After - Node 6145

Memory Usage Observations

Unfortunately, memory consumption was not captured in the graphs, but we conducted real-time monitoring and can share our findings:

  1. During idle state (before network activation):

    • Average memory consumption: 5 GB

  2. During network operation:

    • Nodes with 256 validators: 20–28 GB RAM usage
    • Nodes with 1 validator: Up to 16 GB RAM usage

Conclusions

  • A high number of validators significantly impacts memory consumption, as a large amount of validator data must be stored in RAM.
  • Network traffic consumption increases with the number of validators per node:
    • 256-validator nodes consumed 4,000 KiB/s
    • 1-validator nodes consumed 2,000 KiB/s
  • CPU utilization remained similar across all nodes (60–80%), but nodes with 1 validator exhibited more stable behavior.
  • We gathered a wealth of valuable data, which we will leverage to optimize node resource consumption in future deployments.