Verify a decision

{"actor":"system:backfill","investigation_id":"d20601ba-e132-4ef8-93ac-d9a532454b0f","kind":"publish","page_slug":"teraswitch-solana-validator-hosting-concentration-incident","published_at":"2026-09-09T01:43:14.420Z","sequence_num":1,"snapshot":{"content_type":"investigation","entity_name":"Teraswitch (Solana validator hosting concentration incident)","sections":[{"content":"On the morning of August 12, 2026, Teraswitch, a hosting and network provider used by dozens of Solana validators, experienced a routing failure that disconnected its sites in London, Amsterdam, Dublin, Frankfurt, Singapore and Tokyo from the internet and from Teraswitch's internal backbone, while its North American sites were unaffected. According to Teraswitch's own status reporting and subsequent reporting, a malformed default route originating at Teraswitch's MIA1 facility in Miami was propagated, with its metric and communities stripped, through a route reflector at the company's AMS2 site in Amsterdam into its European and Asia-Pacific markets; edge routers in those regions incorrectly preferred the invalid route over valid paths, cutting reachability to twelve sites. Teraswitch engineers reportedly identified the fault and removed the Miami site from the backbone, restoring service within roughly ten minutes of diagnosis, with full recovery across the affected sites taking approximately 33 minutes in total. This is described in reporting as an operational and configuration failure at Teraswitch, not as any form of intentional misconduct, and Teraswitch's status page and public statements characterized the event as a technical routing bug that was resolved same-day.","heading":"The August 12, 2026 Routing Failure","severity":"medium","sources":[{"credibility":2,"name":"Solana Nears Transaction Finalization Halt Due to TeraSwitch Outage — ForkLog","type":"news_article","url":"https://forklog.com/en/solana-nears-transaction-finalization-halt-due-to-teraswitch-outage/"},{"credibility":2,"name":"Solana Hit 86% of Its Halt Threshold After a Teraswitch Routing Failure — SolanaFloor","type":"news_article","url":"https://solanafloor.com/news/solana-hit-86-of-its-halt-threshold"},{"credibility":2,"name":"A Routing Bug Took Solana 86% of the Way to Losing Finality — Decrypt","type":"news_article","url":"https://decrypt.co/375404/a-routing-bug-took-solana-86-of-the-way-to-losing-finality"}]},{"content":"Solana's consensus mechanism requires more than 66.67% of staked SOL to actively participate for transactions to reach finality; if delinquent (offline) stake reaches 33.34%, the network loses the ability to finalize new blocks. Marinade Finance, a Solana liquid-staking protocol, reported that the outage sent 28.83% of all staked SOL delinquent simultaneously, placing the network approximately 4.5 percentage points — described elsewhere as roughly 86% of the way to the threshold — from a finality halt. CoinDesk reported Marinade Finance's statement that the blockchain \"neared a freeze threshold\" during the incident. The Solana network did not halt: block production and transaction settlement continued throughout, and reporting indicates finality was never actually interrupted, though the margin of safety was substantially narrower than under normal operating conditions.","heading":"Impact on Solana Network Finality","severity":"high","sources":[{"credibility":1,"name":"Smart contract blockchain Solana neared a freeze threshold Wednesday, Marinade Finance says — CoinDesk","type":"news_article","url":"https://www.coindesk.com/tech/2026/08/12/smart-contract-blockchain-solana-nearly-froze-wednesday-marinade-finance-says"},{"credibility":2,"name":"Solana Hit 86% of Its Halt Threshold After a Teraswitch Routing Failure — SolanaFloor","type":"news_article","url":"https://solanafloor.com/news/solana-hit-86-of-its-halt-threshold"},{"credibility":2,"name":"Teraswitch Routing Failure: 28.83% of Staked SOL Goes Delinquent on Solana — Solana Compass","type":"news_article","url":"https://solanacompass.com/news/teraswitch-routing-failure-brought-solana-within-45-points-of-a-finality-halt"}]},{"content":"Reporting indicates Teraswitch's autonomous system, AS20326, hosted approximately 118.9 million SOL, or 27.34% of all staked SOL on the network, and that roughly 94% of that stake went offline simultaneously during the outage. This concentration reportedly exceeded a rule adopted by the Solana Foundation Delegation Program (SFDP), effective May 1, 2026, requiring program participants to operate on an ASN and hosting provider holding less than 25% of overall network stake — a rule intended to reduce systemic risk from any single infrastructure provider. Reporting attributed to Marinade Finance additionally disclosed that Marinade's own delegation model concentrated roughly two-thirds of its delegated stake across only four autonomous systems, with a single ASN (AS395201) accounting for 36.94% of Marinade's delegated stake, and that Marinade stated \"Nobody should be comfortable with that, us included.\" Community discussion following the incident reportedly converged on guidance that geographic or ASN concentration should be kept below 25% of cluster stake, ideally closer to 20%, and that validator failover nodes should sit in a different physical location and on a different ASN from the primary validator.","heading":"Stake Concentration Behind a Single Autonomous System","severity":"high","sources":[{"credibility":2,"name":"Teraswitch Routing Failure: 28.83% of Staked SOL Goes Delinquent on Solana — Solana Compass","type":"news_article","url":"https://solanacompass.com/news/teraswitch-routing-failure-brought-solana-within-45-points-of-a-finality-halt"},{"credibility":2,"name":"Solana Hit 86% of Its Halt Threshold After a Teraswitch Routing Failure — SolanaFloor","type":"news_article","url":"https://solanafloor.com/news/solana-hit-86-of-its-halt-threshold"},{"credibility":2,"name":"A Routing Bug Took Solana 86% of the Way to Losing Finality — Decrypt","type":"news_article","url":"https://decrypt.co/375404/a-routing-bug-took-solana-86-of-the-way-to-losing-finality"},{"credibility":1,"name":"Delegation Criteria — Solana Foundation","type":"official","url":"https://solana.org/delegation-criteria"}]},{"content":"Approximately 90 validators across the affected regions went delinquent during the outage, according to multiple reports. Helius, described in reporting as Solana's second-largest validator, was reportedly offline for the entire approximately 33-minute outage window; Marinade Finance reportedly noted that Helius's backup or failover systems did not activate during the event. Separately, reporting indicates 59 validators representing about 80.2 million SOL returned online within the same recovery window across Amsterdam, Frankfurt and Tokyo without automated failover having triggered, and that of 74 measurable operators, only three — Laine, Cogent Crypto (both operated by Sol Strategies) and Lion3d — were reported to have recovered cleanly via redundant infrastructure. Affected validators collectively missed approximately 333 SOL in staking rewards during the downtime, an amount reported to be covered by validator bonds. These findings describe an operational availability gap in failover tooling and configuration across multiple validator operators, not misconduct by any individual operator.","heading":"Validator-Level Impact, Including Helius","severity":"medium","sources":[{"credibility":2,"name":"A Routing Bug Took Solana 86% of the Way to Losing Finality — Decrypt","type":"news_article","url":"https://decrypt.co/375404/a-routing-bug-took-solana-86-of-the-way-to-losing-finality"},{"credibility":2,"name":"Solana Hit 86% of Its Halt Threshold After a Teraswitch Routing Failure — SolanaFloor","type":"news_article","url":"https://solanafloor.com/news/solana-hit-86-of-its-halt-threshold"}]},{"content":"In the weeks following the outage, validator-community discussion (summarized in a Solana-focused newsletter covering discussions from August 13–21, 2026) reportedly converged around treating validator failover design as an operational priority rather than a background concern, partly because the incident produced evidence — described as instances where \"the same validator identity apparently produced two distinct blocks for the same slot\" — that split-brain risk from misconfigured failover is a concrete rather than purely theoretical concern. Marinade Finance stated it would review its own concentration limits by autonomous system and data center and publish guidance on validator hot-swap and automatic-failover capabilities. Separately, reporting on Solana's roadmap indicates a consensus-layer change (referred to as the \"4.3\" release) had completed external audits and been cut for testnet as of early September 2026, with mainnet activation targeted for September 21, 2026, though this change is described as a broader consensus overhaul rather than a direct, purpose-built response to the Teraswitch incident specifically. No public evidence was found indicating that Teraswitch's or any other single provider's share of staked SOL has been reduced below the Solana Foundation Delegation Program's 25% ASN/hosting-provider cap since the incident; the underlying concentration risk that made the August 12 event possible has not been confirmed as resolved.","heading":"Community and Foundation Response Since the Incident","severity":"medium","sources":[{"credibility":3,"name":"Summary of Solana Validator Discussions: August 13 - 21 — Chainflow (Substack)","type":"community_report","url":"https://chainflowsol.substack.com/p/summary-of-solana-validator-discussions-9d1"},{"credibility":2,"name":"Solana Hit 86% of Its Halt Threshold After a Teraswitch Routing Failure — SolanaFloor","type":"news_article","url":"https://solanafloor.com/news/solana-hit-86-of-its-halt-threshold"}]}],"sources_used":[{"credibility":1,"name":"Smart contract blockchain Solana neared a freeze threshold Wednesday, Marinade Finance says — CoinDesk","type":"news_article","url":"https://www.coindesk.com/tech/2026/08/12/smart-contract-blockchain-solana-nearly-froze-wednesday-marinade-finance-says"},{"credibility":2,"name":"Solana Nears Transaction Finalization Halt Due to TeraSwitch Outage — ForkLog","type":"news_article","url":"https://forklog.com/en/solana-nears-transaction-finalization-halt-due-to-teraswitch-outage/"},{"credibility":2,"name":"Solana Hit 86% of Its Halt Threshold After a Teraswitch Routing Failure — SolanaFloor","type":"news_article","url":"https://solanafloor.com/news/solana-hit-86-of-its-halt-threshold"},{"credibility":2,"name":"Teraswitch Routing Failure: 28.83% of Staked SOL Goes Delinquent on Solana — Solana Compass","type":"news_article","url":"https://solanacompass.com/news/teraswitch-routing-failure-brought-solana-within-45-points-of-a-finality-halt"},{"credibility":2,"name":"A Routing Bug Took Solana 86% of the Way to Losing Finality — Decrypt","type":"news_article","url":"https://decrypt.co/375404/a-routing-bug-took-solana-86-of-the-way-to-losing-finality"},{"credibility":2,"name":"Solana Narrowly Avoids Finality Halt After Major Routing Fault — CryptoTimes","type":"news_article","url":"https://www.cryptotimes.io/2026/08/12/solana-narrowly-avoids-finality-halt-after-major-routing-fault/"},{"credibility":1,"name":"Delegation Criteria — Solana Foundation","type":"official","url":"https://solana.org/delegation-criteria"},{"credibility":3,"name":"Summary of Solana Validator Discussions: August 13 - 21 — Chainflow (Substack)","type":"community_report","url":"https://chainflowsol.substack.com/p/summary-of-solana-validator-discussions-9d1"}],"summary":"Teraswitch is a data-center and network hosting provider whose autonomous system (AS20326) hosted roughly 27% of all staked SOL as of August 2026. A routing misconfiguration originating at Teraswitch's Miami facility on August 12, 2026 knocked out connectivity to its European and Asia-Pacific sites for about 33 minutes, simultaneously disconnecting approximately 90 Solana validators representing up to 28.83% of staked SOL and pushing the network to roughly 86% of the 33.34% delinquent-stake threshold at which Solana would lose transaction finality. This is documented as an infrastructure-concentration and availability incident, not an allegation of fraud or misconduct by Teraswitch; the network recovered fully and finality was never actually lost, but the event is treated by validators, delegators and the Solana Foundation as evidence of a systemic single-point-of-failure risk in the validator hosting landscape.","timeline":[{"date":"2026-05","event":"Solana Foundation Delegation Program rule takes effect requiring participants to operate on an ASN and hosting provider holding less than 25% of overall network stake.","source":"Solana Foundation Delegation Criteria","source_url":"https://solana.org/delegation-criteria"},{"date":"2026-08","date_evidence":"A routing failure at Teraswitch, a hosting provider used by Solana validators across Europe and Asia-Pacific, knocked roughly 90 validators offline for approximately 33 minutes on August 12","date_original":"2026-08-12","event":"A malformed routing announcement originating at Teraswitch's Miami facility propagates to its European and Asia-Pacific sites, disconnecting roughly 90 Solana validators representing up to 28.83% of staked SOL for about 33 minutes and pushing the network to roughly 86% of its finality-halt threshold.","source":"Solana Compass","source_url":"https://solanacompass.com/news/teraswitch-routing-failure-brought-solana-within-45-points-of-a-finality-halt"},{"date":"2026-08-12","date_evidence":"Solana came close to a network finality halt on the morning of August 12, 2026, after 28.83% of staked SOL went delinquent following a routing failure at infrastructure provider Teraswitch.","event":"Marinade Finance publicly states that Solana \"neared a freeze threshold\" during the outage; CoinDesk reports the statement.","source":"CoinDesk","source_url":"https://www.coindesk.com/tech/2026/08/12/smart-contract-blockchain-solana-nearly-froze-wednesday-marinade-finance-says"},{"date":"2026-08","event":"Validator-community discussion converges on recommended thresholds (ASN/geographic concentration below 25%, ideally closer to 20%) and failover-design best practices in the wake of the outage.","source":"Chainflow validator discussion summary","source_url":"https://chainflowsol.substack.com/p/summary-of-solana-validator-discussions-9d1"},{"date":"2026-09","event":"Reporting indicates Solana's next consensus-layer release (\"4.3\") has completed external audits and been cut for testnet, with mainnet activation targeted for September 21, 2026.","source":"Chainflow validator discussion summary","source_url":"https://chainflowsol.substack.com/p/summary-of-solana-validator-discussions-9d1"}]},"v":1}