This weekend, Solana experienced a perfect storm of technical issues and increased activity (including heavy bot spamming). The resultant network slowdown caused multiple transaction failures and coincided with a sharp decrease in SOL price.
Thanks to hard work from the Solana community members, the Solana network performance has been restored.
In this article, we’re going to look at what happened over this 48 hour period, including:
- The Situation Leading up to the Solana Slowdown
- Why the Solana Slowdown Happened
- How We Helped Restore Solana’s High-Speed Network
- How We Are Helping Solana Build for the Future
- How To Manage Your Risk with Blockdaemon
Before that though, it’s worth mentioning the Solana community spirit and joint effort.
While everyone experienced difficulties running Solana this past week, what is important is how we came together to fix these issues. As a key member of many communities, Blockdaemon is always on hand when our partners need us.
We are proud to have played an active and crucial role in fixing the Solana issues.
We didn’t do it for the praise, we did it because we’re committed to our partners and a decentralized future. However, it is always nice to see Anatoly Yakovenko acknowledge the community and share his appreciation.
Huge thanks to all the folks that debugged these issues over the last 48 hours. @leolukde @CertusOne @buffalu__ @BlockdaemonHQ @triton_one @ShinobiSystems @Neodyme and the folks at solana labs who didn’t sleep for 48 hours. https://t.co/bq1GZ78rnt
— anat◎ly 🦀🤿🏒 🤙 (@aeyakovenko) January 23, 2022
Now, back to the Solana slowdown. As a key player in the recovery effort, let us share our opinion of the events…
The Situation Leading up to the Solana Slowdown
Recent market movements have resulted in an increase in Solana’s on-chain activity. This has been strongly linked to the US Federal Reserve’s planned interest rate hike. As rates increase, credit becomes more expensive and traditional liquid assets become more appealing.
SOL holders started to sell which caused subsequent downward pressure on the token. As the token value fell, an increasing number of trading bots started to liquidate their positions. So far this is all normal cryptocurrency activity.
Why the Solana Slowdown Happened
Solana 1.8.12 had a bug that resulted in inefficient deduplication. Unfortunately, bots were increasingly spamming duplicate transactions to increase their chances of getting a successful transaction.
The bots were effectively carpet-bombing the network with around 300k transactions/second. All to get just a few liquidations going.
The increase in duplicate transactions overloaded many nodes and put pressure on the network. Eventually, this pressure became too much for the network to handle at adequate performance levels.
As performance degraded, the network and price feeds lagged. This led to trading bots sending yet more transactions and a vicious circle developed.
Some validators with low bandwidth started to drop packets. As more and more packets were dropped, the network became increasingly unstable and started forking.
How We Helped Restore Solana’s High-Speed Network
Solana needed to address these issues as quickly as possible. Within hours, Solana 1.8.14 was released with performance fixes included, but remedial action was needed. That being the case:
- Blockdaemon and Triton One implemented network rate limits and abuse detection to prevent block spammers.
- We discovered that 3rd party nodes with open RPC ports were being used to spam the network.
- We gave our tooling to other validators to help them apply the fixes
- The Solana foundation informed node operators, who then started to apply fixes.
This combined effort restored normal performance to the Solana network.
How We Are Helping Solana Build for the Future
Following the slowdown, Blockdaemon engineers have been working alongside Jito Labs to help avoid any recurrence.
Together, we have submitted this architecture improvement proposal to allow better scaling for Solana validators.
How To Manage Your Risk with Blockdaemon
Everyone experienced difficulties running Solana this past week. What is important is how we come together as a community to help this game-changing technology persevere.
The unfortunate truth is that all software has bugs. It is therefore vital that we all risk-manage the elements within our own control.
Of course, this is often difficult considering the blockchain trilemma and maintaining decentralization and security, at scale. Because of this, Blockdaemon employs a 4 layered approach to risk management:
SRE practices and Bare-Metal Nodes Protected Our Infrastructure
During the Solana slowdown, our SRE principles and high spec bare-metal nodes ensured that our infrastructure didn’t go down. Unlike some providers, we will only ever run 100% premium enterprise-grade Solana boxes.
With 200gbps local DDoS protection and multi-terabit remote protection on a crypto-optimized network blend of infrastructure.
If you want to manage your blockchain risk, get in touch with Blockdaemon today.