V3.co.uk

Have Fortnite's outages put it on its last life?

The most popular game in the world is suffering increasingly high-profile outages

Have Fortnite's outages put it on its last life?
Have Fortnite's outages put it on its last life?
  • Angelique Medina
0 Comments

You know that a game has 'made it' when its title becomes synonymous with its fanbase. Another clear sign is that your competitors see a distinct drop-off in their user numbers. After recruiting millions of followers, Fortnite has very much usurped the competition to become the leading online massively-multiplayer title; with more than 92 million events processed every minute, it's become something of a phenomenon.

Many have supposed that Fortnite's free-to-play model, generating much of its revenue from in-game purchases such as custom skins and personal modifications, has been a huge contributor towards its success. But, with access to the game being critical to its revenue generation, outages pose more than just a threat to user experience and player adoption.

As player figures have grown at an incredible rate, Fortnite's creator, Epic Games, has had its fair share of outages due to everything from unscheduled maintenance windows to application scalability, all of which have become increasingly high profile as the game has risen in popularity, and the rate of outages seems to be increasing regularlarly. Looking at everything from the anatomy of the network, to the processes performed across it, here's why we think the players are being denied their kill shots.

Gaining critical mass

The most understandable reason is immediately apparent from the gameplay. For those of us who haven't given it a test-run, Fortnite is a first-person shooter game that revolves around a ‘Battle Royale' format. Simply, a hundred players enter and the last one standing wins. Players are dropped on an island and are tasked with having to battle it out with all their competitors, as the available battleground slowly shrinks.

Having a large number of users is very much critical to the enjoyment of the game. But as application providers know all too well, application infrastructure becoming overwhelmed can lead to perceived crashes in their thousands.

The issue with this pitfall is that user figures aren't something that can be easily curbed without Epic Games undercutting its own revenue. The way that the company has sought to deal with this undoubtedly regular occurrence is by transparently communicating the health of its network in both an in-depth and immediate fashion. Whilst this level of clarity has gone some way to assuage the rage on its millions of players, there are just times when you can't simply blame popularity for poor performance.

Ultimately disruptive gameplay

A few months back, some users connecting to Fortnite: Battle Royale would have experienced first a total outage, then significant performance degradation, lasting about thirty minutes. The cause of the outage wasn't due to an application issue, or even network performance. In this case, it was caused by instability of advertised Internet routes - also known as BGP route flapping. It's as if the entire last mile of highway (from every direction) to a football stadium simply vanished - leaving fans completely cut off.

The March outage, which was captured using the ThousandEyes application, wasn't application-related or lengthy, but it does illustrate the many dependencies of delivering gaming services via the Internet.

When the outage began, at around 2am BST, users connecting to the Fortnite instance qos1.ol.epicgames.com via hosting provider Seflow would have experienced a total disruption in gameplay. The Path Visualisation view below shows massive packet loss occurring at every edge node connected to this hosting provider. This massive loss would have completely prevented some users from connecting to the game.

The cause of this outage wasn't due to a collective failure of peering ISPs. Instead, routing updates originating from AS49367, the host provider, were disrupting the Internet's ability to reach it. The continuous withdrawal and announcement of BGP route advertisements - also known as flapping - would have prevented users from connecting to game instances hosted out of that particular data centre.

What's interesting about this particular outage is that it's unrelated to any architectural decisions Epic Games could have made. Epic Games has a multi-cloud strategy, with instances hosted out of AWS, Google Cloud and potentially other providers. The company has also recently taken steps to improve its ability to scale.

However, in this case, it wasn't as if there was actually anything wrong with Fortnite's servers. There they were, sitting ready in the data centre, waiting for users to invoke processes for gameplay. Instead, this was about the 'Internet map' that users relied upon to get to them.

Updates versus outages

Internet route advertising is notoriously tenuous - effectively built on a chain of trust that spans every ISP on the planet, regardless of reputation. For instance, in the last year alone, there were nearly 14,000 BGP routing incidents.

Not all of these incidents were malicious; many were due to configuration errors or outages, and even more are likely to be down to the game's aggressive update schedule pushing patches to users around the world.

Regardless of the cause, routing issues have led to some high-visibility events, including a route leak by Google that effectively shut down the Internet for Japan in August of last year. Any disruptions in routing, whether due to a hijack, leak or flapping, can have considerable disruption.

The complexity of application delivery today means that many dependencies must all work flawlessly together to ensure a good user experience. When a video game's ability to deliver revenue is directly correlated to its up-time, the network ceases to be a utility factor and takes centre stage as an essential profit component.

Whether a game is having to combat an influx of user-traffic or is essentially creating its own problems by pushing multiple application updates on a daily basis, it still has the complex realm of BGP routing to navigate. As Fortnite continues to grow in popularity, Epic Games can only hope that the loyalty of its fans will continue to grow and not wane as plays its own delicate game of Battle Royale on the network.

Angelique Medina is a senior product marketing manager at ThousandEyes

Further reading

V3 Latest