Hopefully you haven’t had reason to notice yet, but there’s a rising problem with AI services on Google Cloud, AWS, and other platforms sticking their customers with bills in the tens of thousands of dollars.
This week’s episode of the Kettle focuses on two such stories that The Register published this week, one concerning Google and another involving AWS. In both cases, cloud customers using AI incurred massive bills without any prior notification from their provider and not a lot of help to resolve the matter with any sense of urgency.
[…]
So if you’re a developer and you’ve created an API key for your projects, if your project uses Maps, you’ll create an API key. And for years, the advice from Google was put that API key on the front end of that, make it public so that when users are using your site, it links back to your project.
The problem was a couple years ago, they allowed those API keys, if they were configured correctly, to also access Gemini. And a lot of folks who were early adopters of AI went in and said, okay, I want to use Gemini with my project. And not really connecting the dots that their API key on the front end that was publicly available would now also allow anybody to inference Google’s Gemini platform.
And it wasn’t a big deal, I think, for a lot of years because I don’t think the platform was really that amazing.
Brandon (02:01)
Yeah, because you said this is a three year old change, right?
O’Ryan Johnson (02:22)
But recently…Nano Banana and the Veo 3 models came out. And that’s when I think we started to see a lot of this. This great security company named Truffle wrote something about this in February saying, look, be careful because if you’ve put your API key out according to Google’s instructions, and if you’ve also been working with Gemini models, there’s a chance that you may have inadvertently opened up your API key to anybody to be able to inference [Veo] and NanoBanana to their heart’s content.
Brandon (02:40)
And specifically a Maps API key, right? Okay,
O’Ryan Johnson (02:51)
Correct. Which again was, was Google had told everybody for quite a while was safe. And so, what happened kind of inevitably is folks were bad actors were in fact using that for for those purposes So you’d have these you know sort of like horror stories of waking up in the morning and seeing your Google account Which you maybe you never spent more than fifty dollars a month, all of a sudden you have a $3,000 bill, $5,000 bill.
[…]
how you figure this out, is kind of buried, right? It’s hard to find, right? So as he’s looking, trying to frantically figure out what’s happening, more charges are being added.
[…]
that’s the first part. The second part is that, you know, this happened to people who had spending caps in place. And Google has only recently put spending caps in place, but they’re really loose caps. I talked to a developer in Australia who said, “Look, I put a $250 spending cap in place. And when I woke up, I had a $10,000 bill.” …And he said, “When I was going to going through afterwards, I looked and I said my spending tier was at the $100,000 limit. And I said, how does it happen?”
Well, if you look like Google was actually very upfront about this. In March, they put out a blog and said, “Hey, we’re going to help you out. If you’ve only got a $250 spending cap, if you spent $1,000 in the lifetime of your account and you’ve been a Google member for 30 days or more, a Google Cloud developer for 30 days or more, you can spend $100,000.”
Brandon (04:47)
And there’s no notification to the user accounts that this is being done?
O’Ryan Johnson (04:50)
Except for the emails that say this is how much you owe us, which is all after the fact.
Brandon (04:57)
And if you’re less than 30 days, right, it’s moving to tier two is, I think, what’s the cap on that?
O’Ryan Johnson (05:01)
Two thousand dollars.
Brandon (05:04)
But even then, it’s spend a hundred bucks in the lifetime of your account?
O’Ryan Johnson (05:06)
A hundred dollars and be three days old, and Google will give you a $2,000 cap to spend. Those are the most generous terms – you guys have been around IT for years, what distributor would ever give you terms like that like if you went to TD Synnex or if you went to Ingram Micro and said, “Hey, I’m 30 days old and I’ve spent $1,000. I would like $100,000 in credit with you.” They would laugh .
[…]
And then of course the problem is trying to get that money … trying to get your account restored. like in two of the cases, the money had already been spent. So the credit cards, one was $17,000, one was $10,000. The money was already out of their account and they have this project. If they charge it back, they’re afraid that Google’s going to shut down their project and delete it. If they stick with the bill, then they’re stuck with this debt that is obviously outside the bounds of any budget that they had set for their Google Cloud project.
[…]
The automatic tier upgrades are obviously a problem, but are all these cases that you’re seeing, are they tied back to the Truffle notice? I mean, these are all Maps API keys?
O’Ryan Johnson (07:02)
Not all of them.
Some people say like, “Look, I never put my API key out publicly.” And I talked to a guy yesterday who said, “Look, my API key has been hidden from everybody. I think I got brute forced.” ….I don’t possibility or the probability of being able to brute force an API key, they’re huge, long chains of numbers and texts. Probably not impossible…But this guy, his bill was $127,000, which is just a huge, huge amount.
[…]
You wrote a story this week about an AWS customer who was billed $30,000 despite supposedly having a setting enabled to prevent this. So what’s this all about?
[…]
a user who was using AWS Bedrock. He wanted to take Claude Opus out for a spin, try it out. He had some startup credits fired by Activate. All great. Now he was using a tool called the AWS Cost Anomaly Detection Tool. What that does, that actually sends you alerts if you’re doing some odd things and your account is incurring additional costs, and as well as using AI machine learning, you can also set some custom thresholds… “If I spend more than this then stop or shout at me or whatever
Brandon (09:39)
Yeah, cut me off. Yeah.
Richard Speed (09:45)
So he thought, “Great, I’ve got that, what could possibly go wrong?”
And so he began to use his AWS Bedrock and no alerts were fired, all was good until about a month after he began using it he got a bill for $30,000 or $38,000 through where he was expecting hundreds. And the reason being was that AWS Bedrock apparently bills through AWS Marketplace, and that is not compatible with the cost anomaly detection.
Brandon (10:06)
So Marketplace is where you can pick up third party integrations for AWS, right?
Richard Speed (10:17)
Right, and that’s where AWS Bedrock was being billed, was basically invoiced through. And to be completely fair to AWS, that is documented. It is in the documentation, “This will happen.” So, hence the cautionary tale aspect. But again, I’ve had a few people say, actually it’s pretty unintuitive, this. You kind of would assume it’s being caught and it wasn’t caught.
[…]
So does Cost Anomaly Detection function only with first-party Amazon products then basically? Anything in the Marketplace that you’re pulling from a third-party provider doesn’t get included in this?
Richard Speed (11:59)
Yeah, I believe so. Yeah, it’s just through AWS services except for Marketplace stuff. But there are other checks and things in place in AWS. It’s just in this instance, the expectation was if I’m using Cost Anomaly Detection, it should stop me running up a massive invoice or running up a massive bill using AWS Bedrock. In this case, it didn’t. It was completely silent as the thousands and thousands and thousands began to rack up on the account.
Brandon (12:05)
And even, I think you wrote, even when his credits ran out. Like, he ran out of credits and switched to cash billing and there was no notice.
Richard Speed (12:29)
Exactly. It suddenly went from from credits to cash billing again with no notification or warning.
[…]
One thing that is kind of universal across this that one of these users pointed out, is that the most frustrating part is that they have the information. They can see what you’re doing in your account and they don’t stop it. All this information that we’re talking about, whether it’s your usage, whether it’s your billing, all that stuff is within the four walls of, whether it’s Google or AWS
[…]
Like if a user could shut off – if there was a notification that came in and said, “Hey, did you know that you’re on Veo right now and you’re generating videos? Would you like to shut that off?” Think about your credit card company. If I go one county over and I spend $10 at a Target, I’ll get an alert from my card company. “Hey, are you sure?” Are you telling me, Google and AWS, that you can’t do that?
[…]
There’s no way, there’s no way that ⁓ Google and AWS don’t see this usage or can’t monitor it. Can’t pop a large language model on there to keep an eye out for ⁓ unusual billing and notify people. Like you said, if you never use [Veo] or never use NanoBanana and all of a sudden your account’s racking up thousands of dollars of charges on it, Google should probably say, “Hey, is this you?”
[…]
Richard Speed (18:11)
There have been some issues on Azure. I read a piece, oh crikey, several weeks, maybe even months ago now, regarding a similar thing to what’s happened with AWS with a user who had, he hadn’t realized that his startup credits didn’t count towards AI usage. And then he found himself hit with a massive invoice because again, Microsoft just quietly said, “Yeah, sure. You want that service? No problem. Here you go. Use it.” And so he used it and then the huge invoice came through. I think… I think it’s important to point out that these companies, they’re not doing anything wrong legally. Ethically, I’m with O’Ryan, they should be warning you to say, “Hey, you know, you’re spending way more now than you ever used to before. These services that you’ve never used before, are you sure you want to be doing that? Are you sure about that?”
[…]
it looks like the AWS user might be a little bit hosed on getting a refund. Do you know is Amazon – did you talk to Amazon for the story? Do they have any intention to change the marketplace versus non-marketplace CAD policy?
Richard Speed (xx:xx)
They did respond, and at the moment there’s no plans to change it.
O’Ryan Johnson (xx:xx)
Google is also, they’re sticking by their automatic tier upgrades. They like the flexibility that it gives to developers. Flexibility, of course, meaning that developers can spend a lot more than they initially wanted to, or agreed to.
Brandon (xx:xx)
It’s a very one-sided flexibility, really, when you think about it.
[…]