In a Nutshell

With good faith in most of developers, it is unlikely anyone would be vicious enough to deliberately check in passwords or keys to GitHub and make it public. But developers are only humans, and humans do make mistakes. It is not hard to find a lot of AWS keys are checked in, and publicly available, even if you just do a quite simple search on GitHub.

While some of which we can get away with, some of which are just gonna cost you real money, like this dude’s 500 dollar lesson. And obviously he’s not the only one, my team has recently received an email from Amazon:

Amazon Security Alert Email

Ouch…

How Did this Happened?

It starts from a good will from an ex-teammember we’ve never met and his dedication to open source a project he’s created. It all sounds good until shit hits the fan.

Initial Check-in with AWS Credentials

Just like any sensible developer, he soon spotted his mistake and corrected it by removing the sensitive information. But because of the unforgiving nature of version control, whatever is checked-in remains in the Git history. Hence why we received this Amazon email out from blue.

Fire Fighting

As soon as the Amazon security email is received, it is a code red for us. As the security guy in the team, I soon draw out the actions:

• Revoke the compromised key in AWS
• Make the GitHub repo private for further investigation
• Rotate the key to resume the system
• Investigate policies associated with the user
• Evaluate the impact and check for unauthorised usages

Aftermath

Luck enough, the user which assigns the credential is only used for integration testing on a test account with limited permissions attached.

While the management won’t be pleased with the implications to come, because this is not only a security risk to our own account which could lead to unexpected bills on that account, but also violates the AWS Customer Agreement, and that could mean anything from a warning to a serious fine.

I suppose it is too late to cry over spilled milk. But there are preventions and actions we could take on moving forward:

• First and foremost, never check passwords or keys into source code!
• If unfortunately sensitive information is committed, use git filter-branch to remove it before pushing the repo.
• Lastly, use AWS Temporary Credentials for running ad-hoc tasks.
• More resources from Amazon