Bandwidth costs in Australia "over a hundred times more expensive" than other countries.
By Tim Colwill on May 21, 2013 at 3:53 pm
With the news that Path of Exile were launching a gateway to bring down the lag for players in Australia and New Zealand, we called Grinding Gear Games’ Chris Wilson to get the low-down on where the F2P-ARPG is at, why now is the right time for a low-ping gateway, and their plans for the future.
If you’d like to play Path of Exile, you can download the beta client free from our file library here (quota-free for iiNet group customers).
GON: Chris, you’ve been in open beta for a little while now, since January in fact — can you tell us a little bit about how everything’s progressing?
GON: So what is your peak concurrent player count?
Chris: We’ve hit 70,000 in the first week of release. We’re very pleased with that. There was an awful lot of stuff combining at the same time with the game being very new and a lot of people talking about it which caused us to really strain the servers, but thankfully it’s dropped a bit to a more maintainable rate at the moment.
GON: Have you identified any issues in the beta that you’ll need to address going forward before launch?
Chris: We have found a lot of things based on player feedback and our own analytics that we can improve. There are various systems that we’ve been working on with the players to find better solutions to. For example loot allocation in the game, a lot of players would like it to be a bit more forgiving and less free-for-all, so we’ve been looking into a variety of options which will hopefully launch by the end of the month.
GON: When we last spoke you anticipated the third act being the final before launch. How do you feel about that now?
Chris: We added most of the third act in beta release, missing off about the last third of that act. And we are expecting that we will have that as the major content added at launch. It basically finishes off the story, adds a much more climactic boss encounter, and a lot of new areas. And we’re also working on the fourth act of the game which isn’t going to be ready for launch at the end of this year, but it will come maybe 9-12 months after that.
GON: So you’re still anticipating a launch in October-December?
Chris: Yeah, at the moment we’re targeting very late September or early October.
GON: And 9-12 months after that, the fourth act?
Chris: Yes, although that will depend a lot on support from the players. If they continue being generous with their support then we’ll be able to do it quickly, and if they’re not then I guess it will take longer. We have to use their money to develop it, and the rate of support dictates how many staff we have working on it.
GON: How many staff do you have now? Have you scaled up since beta?
Chris: We’re up to 39 now, including our customer support department, and that’s the department that’s grown the most.
GON: That’s really good! That must make you… the third biggest developer in New Zealand now?
Chris: Something around there. I’m not sure of the exact studio numbers. After we last spoke, everyone came down on me for misquoting the relative sizes of the NZ game development industry. So I’ll check my figures before I actually comment!
GON: Fair enough! So you’ve activated the in-game cash shop?
Chris: Yeah, when open beta started we stopped selling access to it and so it became public, and we allowed people to use the credit that they’d purchased during closed beta and also to purchase new credit that they can use to buy microtransactions. That’s been very popular. We’re constantly adding new ones, trying to get a few in every week, and people have been letting us know what they’re interested in seeing next.
GON: Have you been seeing a lot of people willing to pay out for new stuff?
Chris: It’s been very good, looking at the rates of support from the users and we’ve been finding it helps to run specials so that people can get some of the more premium things at a discount occasionally if they’re patient.
GON: So you’ve just activated the AU/NZ gateway for a week or so now. Can you tell us about how the ping has gone down since then and what feedback you’ve been seeing from affected players?
Chris: Sure. So we added it to the realm toward the end of last week but in a way that the public couldn’t see, but as of yesterday we’ve made it so the public can log in to it, so we’ve got one day of feedback from the public plus one more of testing from ourselves. For people who are used to playing on a 200-250ms connection to America, they say it’s like a different game because they’re playing in Australia with sometimes less than 30 ping. From NZ we get about 50 ping. There are some issues with the server that we’re working on resolving with the host, some initial configuration issues, and we suspect it’s just a matter of replacing that physical machine with one with different hardware. But from a networking point of view as far as we’re concerned it’s working extremely well, and it validates a lot of what we were hoping we would get. It’s a strange case because we’re running the realm internationally through different server hosts.
The thing about Path of Exile is that players can play together regardless of which server they’re playing on, so if you have a friend in Europe you can play with them and trade items, even though their server is located in another continent. So normally we do this because the data centers we use are linked together, they have really good connections, but in the case of the Australian one it’s with a different company so it’s our first case of actually going through ‘the wild internet’ as it were, to connect between them.
GON: So where is your gateway located physically? And what did you have to do to get it set up?
Chris: I have no idea what city in Australia it’s located in. Which in a way shows that this kind of stuff is handled transparently (laughs). In order to set it up, the first challenge was finding a server host that was able to do it. And the difficulty there is the cost of international bandwidth. It’s prohibitively expensive, but we have a good deal on that with this server, and it was just a matter of soliciting a test server to see how well it performs and then once we were happy with that we can order a lot more and scale it up to support a larger Australian population on there.
GON: Do you have a lot of AU/NZ players? Can you give us a rough estimate of how many there are?
Chris: It’s difficult to know exactly, but my estimate is somewhere in the region of about 4%.
GON: So 4% of your 2.4 million?
Chris: Yeah I would say it’s somewhere in the region of about 100,000 game accounts signed up through Australia and New Zealand.
GON: You said that the cost of international bandwidth is prohibitively expensive. Is this something you’re finding worse in Australia than any other country?
Chris: In New Zealand and Australia it’s SUPER expensive compared to the rest of the world. In some cases, one hundred times as expensive. So that is prohibitive. Except now that we have a good deal arranged with the server, it should be sustainable.
GON: Now one of the most common reactions to the gateway news was “But these are NZ devs, why did they take so long to make a gateway for themselves”. Can you explain to our readers the realities of game development and why it took this long to make this decision?
Chris: Well I guess there are a few things in play here, and the first one is that although we’re based in New Zealand, our customers generally aren’t. We’d love to be able to support them, the New Zealanders, especially because we’re based there, but the priority has to be the other 98% of players who we have to make sure come first. Because of that, launching an international realm was the first goal in getting that sorted out. And now we’re in that stage of open beta where we’re not encountering stability issues of the core game, you know, everyone else is playing happily overseas, that’s about the time we can look into supporting other territories. In addition of course New Zealand and Australia are very hard to target because of the cost of bandwidth.
We’ve received quotes in the order of multiple dollars per gigabyte of data, you know, like domestic internet rates or worse in some cases because it’s reliable co-located rates. So we’ve received quotes that… there is no way our company could afford the quantity of bandwidth that would be required to run servers there if it was that bad per gigabyte value. Thankfully you can get better rates if you organise larger bulk deals and tie them into server hosting. So we have eventually been able to get one sorted out but it’s not an exaggeration to say that this is literally the sixteenth or seventeenth server hosting company we’ve spoken to.
GON: How do you feel when you see New Zealanders or Australians complaining about this? It’s obviously very difficult for you to be in this position, since you obviously have to deal with the business realities but also you want to support your home team at the same time?
Chris: I would love to give the New Zealanders and Australians a place to play, I mean, if they were playing locally, it’d be easier for us to be greedy and to go and do that at the cost of everyone else, but we decided it’d be best to put most of the players first despite that fact it actually inconveniences us. Another thing is: having the game from New Zealand run at 200-250 milliseconds of ping has encouraged us to make sure the game works acceptably at that latency.
There are a lot of games like Vindictus for example, which is an Asian action-MMO, that game limits it so you can’t connect if your latency is over a certain amount, and they did this because they have quite a fast visceral experience but it requires the latency to be low. Whereas in our case we know that some people are connecting from… like we had connections from the middle of Africa. I don’t know how they’re getting internet, but they’re playing our game and it’s important that the game is playable with a lot of ping. We do an awful lot of action-prediction and stuff like that. When you click to attack it starts the animation going straight away even if it’s going to be a full second before the data gets back to you with the result. And while you certainly suffer playing under conditions like that, it still feels relatively immediate to do so.
GON: I guess this must have made you a stronger developer to have to work under these conditions, but did it provide any difficulties in testing? Were you able to really test the game properly?
Chris: We have our internal servers in our office and we also had the servers in the states so to some extent the experience that we were missing was the experience between these two. The ones in the office have less than a millisecond of latency, and the ones in the states maybe 200 ms away. So we were missing the experience of trying it at any less than 200 ms but thankfully, that’s what our giant army of beta testers was for and they gave us plenty of feedback on that.
GON: When you look to the future you’ve got the fourth act coming and before that, the public launch, but what’s the next step for Path of Exile beyond more acts and more content. Do you have plans for maybe a foundry-style mod toolkit, or any sort of player creation tools? Just spitballing — what have you got in the works?
Chris: We’ve looked into the idea of community content and there are definitely some opportunities there. One of the things to note is that because it’s an online game with a secure economy, you can’t let the players literally decide “this is the exact layout of the dungeon and we’ll put the monsters where we want” because they could make it too easy. So there has to be a degree of challenge that’s maintained. We have quite an interesting plan for the future of Path of Exile which involves both continuing forward with the core game, new acts and so on, but also taking advantage of the league system where we can run — think of it as new servers of the game that have a fresh economy and new rulesets.
We’ve been running race events that last between say, one and eight hours, using the system for quite a while now, I mean there’s several running a day now, six today for example. It’s like a fresh start in the game and you get the chance to race other players. It’s unique in action RPGs because it lets you see who’s better at the game rather than who has spent the most time accumulating items. We have some interesting twists on that in the future that we believe will be very addictive, and our goal is that alongside having a strong normal action-RPG we’d like there to be a lot of repeatable content in these interesting events.
GON: Do you have any plans to introduce a real cash economy into these events? Betting on peoples’ performance for example?
Chris: We are very interested in catering towards the more meta e-sports style stuff. With regards to wagering, I mean we’ve discussed wagering from in-game items all the way up to real money and… of course there a lot of legal implications of real-money wagering, and similar kind of abuse issues that can happen with in-game wagering. I mean people will happily throw matches and stuff to get a better sword, if needed. It does have some interesting points but we really value our streaming community and the people who take the events seriously as a pseudo e-sport, and we have better streaming integration coming in the future for example.
GON: I’ve been playing Neverwinter, and a lot of people are using their Foundry tool to create maps that are basically just easy mob grinds and experience farms. Is this the sort of thing you’d have to worry about with player-authoring tools as well?
Chris: Right. One of the things we are planning on selling is the ability for people to customise the parameters of one of these league events, but in a way that can only make it harder. That way you get bragging rights for completing a very difficult event, rather than the ability to just set it to be too easy and make a lot of progress.
GON: Making it too easy does sound pointless, but… I guess if they’re willing to pay you money?
Chris: It should be interesting. Though it really depends. As long as we can keep it safe for the other users. You’ve probably seen with the design of our microtransactions the goal is that people with money can’t get any kind of advantage over other people in terms of actual progress.
GON: And that seems to be working out quite well for you — there’s a lot of MMOs which are struggling with microtransactions because people know it’s pay-to-win, and so they avoid the whole game like the plague.
Chris: Yeah. It’s difficult as a player to commit yourself to playing something where you know that someone can just trump all of your work by spending twenty bucks.
GON: Chris, thanks a lot for your chat and it’s been really good to talk to you!
Chris: Thank you for your time!