Collection #1 is a set of email addresses and passwords totalling 2,692,818,238 rows. It’s made up of many different individual data breaches from literally thousands of different sources. (And yes, fellow techies, that’s a sizeable amount more than a 32-bit integer can hold.)
In total, there are 1,160,253,228 unique combinations of email addresses and passwords. This is when treating the password as case sensitive but the email address as not case sensitive. This also includes some junk because hackers being hackers, they don’t always neatly format their data dumps into an easily consumable fashion. (I found a combination of different delimiter types including colons, semicolons, spaces and indeed a combination of different file types such as delimited text files, files containing SQL statements and other compressed archives.)
The unique email addresses totalled 772,904,991. This is the headline you’re seeing as this is the volume of data that has now been loaded into Have I Been Pwned (HIBP). It’s after as much clean-up as I could reasonably do and per the previous paragraph, the source data was presented in a variety of different formats and levels of “cleanliness”. This number makes it the single largest breach ever to be loaded into HIBP.
There are 21,222,975 unique passwords. As with the email addresses, this was after implementing a bunch of rules to do as much clean-up as I could including stripping out passwords that were still in hashed form, ignoring strings that contained control characters and those that were obviously fragments of SQL statements. Regardless of best efforts, the end result is not perfect nor does it need to be. It’ll be 99.x% perfect though and that x% has very little bearing on the practical use of this data. And yes, they’re all now in Pwned Passwords, more on that soon.
That’s the numbers, let’s move onto where the data has actually come from.
Data Origins
Last week, multiple people reached out and directed me to a large collection of files on the popular cloud service, MEGA (the data has since been removed from the service). The collection totalled over 12,000 separate files and more than 87GB of data. One of my contacts pointed me to a popular hacking forum where the data was being socialised, complete with the following image:
As you can see at the top left of the image, the root folder is called “Collection #1” hence the name I’ve given this breach. The expanded folders and file listing give you a bit of a sense of the nature of the data (I’ll come back to the word “combo” later), and as you can see, it’s (allegedly) from many different sources. The post on the forum referenced “a collection of 2000+ dehashed databases and Combos stored by topic” and provided a directory listing of 2,890 of the files which I’ve reproduced here. This gives you a sense of the origins of the data but again, I need to stress “allegedly”. I’ve written before about what’s involved in verifying data breaches and it’s often a non-trivial exercise. Whilst there are many legitimate breaches that I recognise in that list, that’s the extent of my verification efforts and it’s entirely possible that some of them refer to services that haven’t actually been involved in a data breach at all.
However, what I can say is that my own personal data is in there and it’s accurate; right email address and a password I used many years ago. Like many of you reading this, I’ve been in multiple data breaches before which have resulted in my email addresses and yes, my passwords, circulating in public. Fortunately, only passwords that are no longer in use, but I still feel the same sense of dismay that many people reading this will when I see them pop up again. They’re also ones that were stored as cryptographic hashes in the source data breaches (at least the ones that I’ve personally seen and verified), but per the quoted sentence above, the data contains “dehashed” passwords which have been cracked and converted back to plain text. (There’s an entirely different technical discussion about what makes a good hashing algorithm and why the likes of salted SHA1 is as good as useless.) In short, if you’re in this breach, one or more passwords you’ve previously used are floating around for others to see.
So that’s where the data has come from, let me talk about how to assess your own personal exposure.
Robin Edgar
Organisational Structures | Technology and Science | Military, IT and Lifestyle consultancy | Social, Broadcast & Cross Media | Flying aircraft