T-Mobile Austria's Customer Service Car Crash
It all started on April 4th when a well—meaning tweet was sent to T-Mobile Austria:
Does T-Mobile Austria in fact store customers’ passwords in clear text @tmobileat? @PWTooStrong @Telekom_hilft https://t.co/ydFdVRWgE4— Claudia Pellegrino (@c_pellegrino) April 4, 2018
T-Mobile sent the following alarming response:
Hello Claudia! The customer service agents see the first four characters of your password. We store the whole password, because you need it for the login for https://t.co/vJapgJ50qc ^andrea— T-Mobile Austria (@tmobileat) April 4, 2018
To understand why this is alarming, I need to take a little trip down the hallways of cybersecurity.
How companies store passwords
It's important to understand that when you log into a secure website, the company that you're sending your password to (should) never actually store that password. That sounds counter-intuitive on the first glance, after all, how can you validate the password that the user sent if you don't store it?
Let's start with the most simple version, we simply have a secret code to 'decode' your password, like a Caesar cipher, where I add one to every letter. Your password is 'hunter2', so when I save it in my database, I save
ivoufs3. Next time you try to log in, you'll send me your password, and I'll add one to every letter, then compare it to what's in my database. If you type
hunter2, then I'll always end up with
ivoufs3, and I'll know you typed in the password. However, a hacker who breaks into the database only knows the encoded version of the password, not your original password.
This allows us to validate your password without ever actually saving your password. There are some really clever encryption methods which are much stronger than a Caesar cipher, but the principle is still the same.
A very clever encryption method might even be one-way. I won't go into the details of these here, but basically, what this means is that if you have the output, you can't 'reverse' it easily to get the input. A very simple example of a one-way encryption method would be to do something like:
- Convert using the Caesar cipher (
- Create a sum using the differences between letters (so, i → v = 13, v → o = -7, giving '13 - 7 + 6 - 15 + 13 + 10', assuming the numbers are listed after the letters)
- Complete the sum, giving
- Store this number in your database.
Even if a hacker gets the number 20 somehow, they will never be able to tell that your original password was
This is a very simple example for illustration, and has serious problems because it produces a lot of 'collisions' (if I typed in
au, as a password, then it would also give me
20, so I'd be able to log in using this 'wrong' password). There are some very well known functions which don't have this problem, and you can spend all day giving them different passwords, and they'll produce a different so-called 'hash' each time, and they're completely one-way.
But hackers are a clever bunch, and they soon realised that if these functions are reliable and quick in converting inputs to outputs, they can just try to create lists of every possible password. That way, they don't need to 'reverse' the maths, they just need to look it up. For example, one of the most popular algorithms is known as SHA-256, which gives us
hunter2. Unfortunately, this is such a commonly used password, that
F52FBD32B2B3B86FF88EF6C490628285F482AF15DDCB29541F94BCF526A3F6C7 is already listed in what are called 'Rainbow tables'—huge lists of passwords and their hashes, making it very simple for a hacker to look them up.
Determined hackers don't even need rainbow tables any more. For retail computer prices, anyone can get a computer built out that can run through every single possible 8 character password in less than 4 days.
To make it difficult to crack a large number of passwords at once, security experts recommend using a 'salt'. This is a random string added on to each password before hashing it.
1EF9888BCA gives us
This second part (
1EF9888BCA) is stored in the database next to my password's hash. Because it's different for every user, it makes it impractical to use a rainbow table to get a massive list of passwords all at once. It doesn't protect single users, so if a hacker gets access to this database, they can use the method described in the link above to get a single user's password in less than four days. It does mean though that it would take more than a year to get 100 users' passwords, and so on.
So, to protect individual users, there's often a secret 'pepper' used, which is stored somewhere away from the database, in a secure location. Then, when you 'hash' the password, you use the secret pepper as well:
Even if the database is compromised, individual users passwords are still protected by the 'pepper', so the hacker would also need to find where this is stored, and compromise this too.
So what did T-Mobile get wrong?
Because these 'hashes' are not reversible, and it's unlikely the T-Mobile customer service representatives are cracking the passwords every time to get the first four characters, we know that one of three things is happening (from least likely to most likely):
- T-Mobile are not salting their passwords, and are maintaining an internal rainbow table (maybe they are using a 'pepper')
- T-Mobile are not hashing their passwords at all
- T-Mobile are storing the first four characters of a user's password separately in plain text.
Every single one of these is a bad idea, security speaking.
'But Joe', you say, 'I thought this article was about customer service?!'. OK, OK, here we go:
Mistake № 1: Handwaving
Once Andrea's response was sent out, the original tweeter replied (politely), asking how it could be fixed.
Thanks for your reply Andrea! Storing cleartext passwords in a database is a naughty thing to do. https://t.co/pbTxmSJrOP What can we do to get your devs to fix that?— Claudia Pellegrino (@c_pellegrino) April 4, 2018
What followed from T-Mobile was an absolute disaster of a tweet from a different social media manager:
Hi @c_pellegrino, I really do not get why this is a problem. You have so many passwords for evey app, for every mail-account and so on. We secure all data very carefully, so there is not a thing to fear. ^Käthe— T-Mobile Austria (@tmobileat) April 5, 2018
Lesson: don't patronise your customers, or dismiss their concerns.
Mistake № 2: Doubling down
By this point, the story was starting to pick up a bit of momentum, and another tweeter weighed in asking:
Well, what if your infrastructure gets breached and everyone’s password is published in plaintext to the whole wide world?— Eric™ (@Korni22) April 6, 2018
At this point, Käthe should probably have checked with her boss before replying, but didn't. Her response was a hubris-filled surprise:
@Korni22 What if this doesn't happen because our security is amazingly good? ^Käthe— T-Mobile Austria (@tmobileat) April 6, 2018
Now, I'm sure T-Mobile take security seriously, but this is Donald Trump-level bluster.
Lesson: rather than making a broad sweeping statement on a topic you clearly don't understand, check with an expert.
Mistake № 3: Making it personal
The following few tweets are some of the most bizarre tweets I've ever seen from a corporate account:
@Korni22 Excuse me? Do you have any idea how telecommunication companies work? Do you know anything about our systems? But I'm glad you have the time to share your view with us. ^Käthe— T-Mobile Austria (@tmobileat) April 6, 2018
I'm surprised that no-one had relieved Käthe by this point, but she released a passive-aggressive tirade against @Korni22 until her shift ended.
Lesson: when you're in a hole, stop digging, and certainly don't start insulting people
Mistake № 4: Not owning it
Eventually, it seems Käthe disappeared from the picture, and T-Mobile's 'company spokesperson' Helmut weighed in with what is presumably an official opinion and statement which has been the subject of hasty conference calls.
Customer service agents see only parts of customers‘ passwords which are safely stored in encrypted databases via industry standard encryption algorithm. We are also using one-time-PINs for customer authentication and are evaluating voice biometrics. ^Helmut @ojour— T-Mobile Austria (@tmobileat) April 6, 2018
No apology, no acknowledgement of concerns, and no explanation that passes any muster. I'm not a PR wizard, but at this point, surely a better approach would have been:
"We understand customers are concerned about security processes—customers' passwords are stored in encrypted databases, and we use one-time-PINs. Our responses yesterday were overconfident this was adequate. We're reviewing urgently and we'll let everyone know the outcome."
Lesson: own your mistakes, don't be afraid to apologise when needed
All in all, a bad day at the office for T-Mobile Austria. Customer service is rarely easy, but it's also pretty hard to get it this wrong.