Inadvertent Routing Error Causing Major Outage
On Tuesday, September 29th 2020, some Radware customers experienced cloud service interruptions. The outage was caused by a so-called BGP Hijacking incident as a consequence of an unintended and unfortunate technical error in one of the world’s largest ISPs, Telstra. While Radware immediately detected and cooperated closely with Telstra to mitigate the problem, the interruptions and suffered outages were out of Radware’s direct control.
BGP Hijacking
BGP hijacking is a malicious or inadvertent rerouting of internet traffic through erroneous paths, paths that are different than the intended paths and can result in global outages. While malicious intent cannot be excluded, it is more common for BGP hijacking to be the result of technical or routing configuration mistakes.
Internet BGP routers have trusted relationships with each other, allowing them to exchange and announce routing information for network pre-fixes, which enables every internet device to find a path to organizations’ services and networks. An incorrect path in a trusted peer can quickly spread and create incorrect routing of traffic such that packets destined for a specific service get lost along the way and eventually discarded.
Internet or BGP Hijacking is unfortunately a common, but mostly short-lived, incident caused by an unintended configuration error. While there have been instances of malicious intent through hacked internet routers and by spoofing trusted BGP peers, those are less common. Rerouting traffic could allow attackers to conduct man-in-the middle attacks or create outages leveraged for extortion.
What Happened?
In this particular incident, the BGP Hijacking was unintended and an unfortunate incident that originated at Telstra, one of the world’s trusted ISPs.
Last Tuesday, at approximately 17:49 UTC, a major global incident at Telstra caused severe problems to hundreds of networking providers worldwide. A portion of the Radware Cloud DDoS Protection Service was impacted by this incident, as well as some Radware customers that saw their routes hijacked.
According to Down Detector, several locations in Australia reported severe outages caused by this incident:
BGP MON provided more insights and details on some of the impacted networks:
Radware’s Mitigation & Actions
Radware’s monitoring system identified the outage in near real time and our cloud operations team immediately started working with Telstra on a resolution as well as a transparent communication to our customers through our Radware ERT team.
Telstra’s Prompt Cooperation and Support
During the incident, Telstra received an enormous number of support requests and continued updating Radware through direct communication channels on their progress.
The widespread incident did not go without notice and created ripples in the media:
A Telstra spokesman confirmed that the BGP Hijack was caused by a technical error and not the cause of an attack or breach:
