Following major revelations on Twitter in October 2019, top VPN provider NordVPN confirmed it was hit by a security breach in March 2018.
Exactly what happened is a complicated story, but we'll begin with NordVPN's version of events, as detailed in a blog post titled 'Why the NordVPN network is safe after a third-party provider breach.'
The hack affected a single VPN server in Finland, NordVPN explained. Its own servers were not compromised.
NordVPN claimed the breach 'was made possible by poor configuration on a third-party datacenter’s part that we were never notified of.'
The hacker would not have been able to obtain user credentials, the company told us. Server access might have enabled the hacker to monitor the traffic of anyone using that server, at that time, but NordVPN says there's no evidence that happened. Even if it did, encrypted traffic - HTTPS, secure email connections and so on - would be protected.
Evidence indicates the attack most likely happened some time between January 31st, 2018, when the server came online, and March 5th, 2018.
The attack was made via a compromised data center account, not an account managed by NordVPN.
The data center deleted this account on March 20th, 2018, blocking any further access to the server.
NordVPN claims not to have been notified about the breach until April 13th, 2019, more than a year after it happened. It took down the server the same day, and began an immediate audit of its 5,000 servers.
The company wouldn't go public until evidence of the hack emerged some six months later. Why? The blog post stated: 'thoroughly reviewing the providers and configurations for over 5,000 servers around the world takes time. As a result, we decided we should not notify the public until we could be sure that such an attack could not be replicated anywhere else on our infrastructure.'
So apparently NordVPN was compromised at some point. Their (expired) private keys have been leaked, meaning anyone can just set up a server with those keys... pic.twitter.com/TOap6NyvNyOctober 20, 2019
What's the background?
On May 3rd, 2018, a user on the 8chan message board started a discussion asking for VPN recommendations, and other users began adding their favorites: NordVPN, Mullvad, TorGuard, VikingVPN, cryptostorm and more.
At 20:46, another user made a post commenting on these suggestions. Mullvad and cryptostorm got an approving 'good choice!', but NordVPN, TorGuard and VikingVPN got a 'lol, no', with links to evidence showing hacked server details from each provider: configuration files, private keys, basic session details and more.
Scanning the text, we noticed the VikingVPN and TorGuard links appeared to show session connection times and some file information from Thursday May 3rd, the day the 8chan discussion began.
That suggests the user hadn't just found these somewhere, or got them from someone else; he saw the thread and grabbed live server information almost immediately. That's either a very speedy hack, or the user already knew the vulnerability for each provider.
NordVPN's details didn't include any dating information. When did the hack occur, then? That's where the picture gets murky.
Date confusion
NordVPN initially told us: 'We believe that the discussion on 8chan was the cause for someone to start looking for vulnerabilities of different VPN service providers, and that discussion started on March 5th.'
The exposed configuration file indicated the attack happened on the same day, the company went on: 'March 5th was the last day when such configuration file existed. Later our configuration was changed, so the config file would have looked differently.'
That's interesting, but there's some confusion over times. The 8chan discussion started on 5-3-2018, yes, but that's the US month-day-year format (3rd May), not European day-month-year (5th March), as NordVPN initially believed. (We know this for sure because the 8chan site indicated 5-3 was a Thursday; 3rd of May 2018 was a Thursday, but 5th of March was a Monday.)
When we pointed this out, NordVPN conceded the error, but claimed this didn't make any significant difference to its account of the attack: 'It seems that we really made a mistake interpreting the date when the discussion started on 8chan. However, the actual timeline, except this one date, remains the same.'
While that may be true, we would argue that it changes what we know about the motivation for the attack.
NordVPN had suggested this all happened on a single day, as a response to an 8chan thread. The attacker read the messages, went off to look for a few vulnerabilities, reported the results and moved on, all more or less in a day. Nothing much to worry about, nothing to see here.
Now we can see the hacker compromised NordVPN sometime between January 31st (when the server came online) and March 5th, 2018, and separately attacked TorGuard and VikingVPN, maybe months later. The intruder may still have done nothing beyond spending a couple of minutes browsing the server, but it shows this wasn't someone casually playing around one evening, either. They knew what they were doing, and they'd been doing it for months, if not longer.
That's interesting, but it's not the end of our date concerns. It turns out that NordVPN isn't the only company which has questions to answer about the nature and timing of the attack.
TorGuard response
TorGuard's immediate response to the breach claimed it happened in September 2017, that it wasn't compromised externally, that users were never at risk, and that it reported the breach earlier, anyway.
But TorGuard's leaked information appears to include sessions dated Thursday May 3rd, the day of the 8chan thread, dating the attack to 2018, not 2017. It indicates the hacker had root access to the server, making it potentially as serious as NordVPN's breach. And if you read TorGuard's report of the earlier security issue, it looks like a separate event. The post talks about ‘2017 IPsec streaming server install scripts [that] had recently became open in error', for instance, which doesn’t seem to relate to what we see in the breach.
We put these apparent contradictions to TorGuard, and a company spokesman told us that 'Due to an ongoing lawsuit this issue relates to I cannot provide direct answers to your questions at this time.' But more information will be released eventually, he claimed, saying 'Things will make a lot more sense after the evidence is shown in court.'
In the meantime, TorGuard told us it stands by its main claims regarding the breach: '[its] server was not compromised externally and there was never a threat to other TorGuard servers or users', and 'TorGuard VPN or proxy traffic was not compromised during this isolated breach of a single VPN server and no sensitive information was compromised during this incident.'
We don't have enough information to come to any conclusions here, so we'll exclude these TorGuard concerns from the remainder of this article. But the general point to keep in mind is that there are some murky areas to this attack which extend beyond NordVPN, and it may be a while before we know more.
How bad was the breach?
The attack was against one of NordVPN's 5,000+ VPN servers, not its central infrastructure. As a result, the intruder had no access to user credentials, billing details or any other profile-related information.
The hacker did obtain Transport Layer Security keys which NordVPN uses to verify its website. In theory, that could be used to create a fake website that appeared to be the real NordVPN.com. The company played down the danger, though, saying 'an attack could only be performed on the web against a specific target and would require extraordinary access to the victim’s device or network (like an already-compromised device, a malicious network administrator, or a compromised network).' The keys also expired in October 2018.
The most significant risk is that the attacker could, in theory, have monitored the unencrypted traffic of users connected to that server. There was no way to link that traffic to a specific individual, but it would have been at least theoretically possible to log personal information shared in plain text (logging in to a website via an insecure HTTP page, say.)
NordVPN minimized this possibility in its initial blog post, saying: 'There are no signs that the intruder attempted to monitor user traffic in any way.'
We asked the company how it could be sure, and it told us: 'There were no changes made to our configuration, no additional processes running, no additional files left on the server. Such configurational changes were necessary to inspect the traffic.'
That's a reasonable interpretation. It's theoretically possible that intruder set up a monitoring scheme earlier, then removed it, perfectly, before his access was blocked, but that doesn't make a lot of sense. After all, this isn't someone who seems to be very interested in stealth; he admitted to the hack on 8chan (or at least, shared the details with someone who did.)
How many users were affected?
NordVPN has confirmed that the affected server was brought online on January 31st, 2018.
The company suggests the attack took place at around March 5th, 2018. As we've discussed, though, we know this is based on a misunderstanding of the 8chan discussion date. There's nothing to say it couldn't have happened as soon as the server came online.
Whatever the truth, NordVPN reports that the 'undisclosed insecure management account' used to carry out the hack was deleted by the data center on March 20th, 2018, blocking further access.
These dates suggest around a two-week window when the server was exposed. NordVPN was originally reported as suggesting 50-200 users may have been affected, but when we asked, it told us: 'We don’t know the exact time of the event, and we can’t tell how many people were connected to that server as we don’t keep any logs. We can only guess: our raw estimate is something around 20 to 70 active sessions.'
That's a wide range, maybe less than 20 to more than 200 users. As NordVPN's original dates were incorrect, we would opt for the higher estimate. That is, up to 200 users who made connections to a specific Finland server between March 5th and March 20th, 2018 were at theoretical risk of having their unencrypted traffic monitored. There's no evidence this happened, but it can't be ruled out.
The immediate impact of this attack seems to be limited, then. But that's not an excuse - NordVPN's own blog post described it as 'egregious mistake that never should have been made' - and the hack is only a part of this story.
NordVPN issues and concerns
Beyond the server traffic issue, NordVPN's statement conceded that the intruder acquired TLS keys which could have enabled a further attack. The company suggested the chances of this happening were slim - NordVPN said they required 'extraordinary circumstances' - but why were these keys compromised at all?
For comparison, TorGuard was also compromised by the same attacker, but an October 2019 blog post claims that because it was using secure PKI management, its main key was not on the affected VPN server.
A more fundamental concern is how the attack was able to happen. NordVPN's original statement blamed the company managing the server, saying: 'the breach was made possible by poor configuration on a third-party datacenter’s part that we were never notified of.'
In a later response to us, the company explained: 'There was an undisclosed IPMI (Intelligent Platform Management Interface) account [a powerful remote access system] left to access the server. That account was breached, and therefore the server was accessed.'
The Finnish company involved has suggested NordVPN has some responsibility for the issue. It claims that it has other VPN providers as customers, and they made more efforts than NordVPN to limit access to these remote access features, for example keeping the ports closed most of the time, and only bringing them up when they're needed.
Whatever the truth, NordVPN's 'it's their fault' line doesn't sit well with the website statement that 'We, NordVPN, confirm that we take full control of our infrastructure.' You can't claim you're in 'full control', then pass the buck if problems appear.
The real issue here is probably transparency. NordVPN admits that it learned of the attack in April 2019, but the company only went public some six months later, after details were exposed on Twitter.
NordVPN claims the delay was required to launch a 'thorough internal audit' of the providers and configurations for over 5,000 servers, and it decided to 'not notify the public until we could be sure that such an attack could not be replicated anywhere else on our infrastructure.'
Well, okay. It would surely have been possible to make a general statement about the attack without discussing the low-level technical details, though: a single VPN server has been compromised, no credentials or user details were exposed, we understand the attack vector, are auditing our systems to confirm they're safe, and will issue another statement soon. Not great for NordVPN, but an open and prompt response might have helped to build trust, and it's certainly better than seeming to have the truth dragged out of you at some later date.
What NordVPN did next
NordVPN deserves plenty of criticism for the breach and how it was revealed, but that's not the whole story. It's also worth looking at what the company has learned from the hack, and what it plans to do next.
After learning of the attack, NordVPN says it immediately launched a 'thorough internal audit' of its entire infrastructure. The company told us this revealed 'a few servers that could potentially be at risk' via a similar remote access system, but these have either been patched or removed.
Server security has been hardened with encrypted storage, making it much more difficult to access data via a remote management system.
In a particularly important move, NordVPN has partnered with security consultancy VerSprite to work on penetration testing, intrusion handling and source code analysis.
A bug bounty program, due for release any day now, should encourage others to catch bugs early and get them fixed before they can do any harm.
The company is promising a 'full-scale third-party independent security audit' of its entire infrastructure in 2020: hardware, software, backend architecture and source code, and internal procedures. That sounds like it'll easily outdo every VPN security audit we've seen so far, although we'll need to see the details to be sure.
Longer-term plans include building a network of colocated servers (owned exclusively by NordVPN) which run entirely in RAM. They'll have no locally stored data or configuration files, nothing that can be exposed in a hack. That's also good news, although not quite so leading-edge, as ExpressVPN introduced its similar-sounding RAM-based TrustedServer technology more than six months ago.
If you're the sceptical type - and in this business, you really should be - you might wonder if NordVPN has just thrown a few airy promises into a blog post to try and win back a little trust. But there's evidence to say otherwise. The company has already made advances in some areas, and the benefits began appearing even before the hack was made public.
On October 9th, for instance, NordVPN announced the results of a VerSprite audit of its apps, with 17 bugs found and fixed.
That's a big deal, all on its own. We've seen enough terrible VPN apps to know that many providers will most likely never, ever, ever open themselves up to that level of scrutiny.
The VerSprite success doesn't make up for the hack, of course, or absolutely guarantee NordVPN will deliver on its other promises. It's an excellent first step, though. And with the VPN world paying very close attention, NordVPN must know that even the smallest failure isn't an option.
Review scores
Deciding how an incident like this should affect a review score is difficult, as there are several important issues to consider, and no clear rules over which should get priority. All we can do is explain our thinking, and make a judgement call.
Our major privacy concern here is the possibility of traffic being monitored, however small. That simply should not happen with any VPN.
On the other hand, NordVPN says, and we tend to agree, that the scope of the attack was limited: just one affected server, no exposure of user credentials, no way to link traffic to a specific user, no risk to traffic already encrypted. This was not some system-wide vulnerability.
Further, NordVPN says the breach was due to a fault by the data center, and although the data center has criticized the company in turn, we've not seen it answer the specific allegations (a particular account, unknown to NordVPN was compromised, then deleted by the data center, also within informing NordVPN.)
Keep in mind that these aren't just NordVPN issues. Every VPN relies to some degree on the management abilities of its data centers. Providers can act to minimize this - as NordVPN is doing, for example with its disk encryption - but there's still a potential vulnerability.
We can see evidence of this in the breach, which included leaked information from other VPNs. The hack doesn't represent some NordVPN-specific problem, and any adjustment in our review score should take that into account.
While these are points in NordVPN's defense, they don't absolve it of all responsibility. The data centers are a vital element of NordVPN's infrastructure, which the company proudly claims to control; any issue also represents a NordVPN failure, at some level.
There's a related point with the loss of NordVPN's TLS keys. It only happened because of this specific vulnerability, and the impact appears to be limited, but that's not an excuse: it's still a security failure.
NordVPN's reluctant disclosure of events has to be a black mark. VPNs depend on trust, and you don't build that by creating the impression that you're concealing problems.
But whatever we think of its lengthy silence, NordVPN has clearly been using this time to address potential vulnerabilities.
As we mentioned above, hiring VerSprite to test security isn't some blue sky 'we'll do that one day' idea that the company has dropped into a press release to make itself look good; it began some time ago, and the first results appeared before the hack was exposed. NordVPN hasn't been shamed into improving its systems; it was doing that already.
Put this all together, and although we believe NordVPN is at fault in some areas, we think the limited nature of the breach, and the corrective actions taken to date, justify dropping NordVPN's by only 0.5 to 4. But that isn't necessarily the end of the story. We're not entirely clear about every aspect of the attack, but we'll keep an eye on any developments, and if NordVPN turns out to be more culpable than we believe right now, we'll adjust our rating accordingly.
