The AI Poop Ouroboros

Today one of my IT customers pinged me asking if I could help with a quality-of-life issue in one of our apps. They had a few ideas about tools we could adopt, but if I can figure out how to solve the problem with what's already in the stack, I want to go that way. 

So I do a quick search. Since I'm in my work profile in Chrome, my default search engine is still Google. Probably that's fine, because it's a Google product I'm trying to get more detail on. 

Google's handy new AI search result widget kicks up a very confident answer that I just need to do three easy steps to use a particular feature to solve this problem.  I get a little excited, because maybe I can go back to the user with "sure, just do this, problem solved." It seems helpful enough that I'm already writing the quick little Slack/email campaign to inform users of a cure for this widespread papercut. 

I flip over to the product and ... no ... none of the things I'm supposed to click are there in the UI. 

Huh. Well, Google will push stuff out to its consumer accounts that aren't considered ready for its enterprise ones. And it also lets enterprise admins exercise some latitude on when they enable that stuff. So I go check the relevant part of the admin panel hoping there's a checkbox. 

No. 

Well, the great thing is that I've got that AI answer that specifically names the feature I'm after, so this ought to go down quick ...

Oh, huh. First result is a content marketing blog from a well known and very online marketing automation company, and the snippet is eerily similar to the answer I got from the AI search snippet. 

Click through, and it is plainly the source of the AI snippet. Word for word. It doesn't have any detail about turning the feature on, but that's fine. Why would it. It's pretty end user oriented. 

So, okay, so back out and look at the other results with a few qualifiers to suggest I want to enable this feature, not just use it. Eventually, after clicking through several, it dawns on me: The steps described in the blog post would work perfectly well for a product by one of Google's competitors. They describe exactly how to use that feature as it appears in the competitor. 

Go back, check the dates, and the original misleading content marketing post is from this year, hyper SEO-optimized, and unattributed. So I'm guessing a bot wrote it. Then Google indexed it. Then Google's AI decided that seemed like pretty authoritative stuff and suggested it to me. About its own product. 

Hallucinations all the way down.