I kept wondering why Grok never showed up in aeo.press server logs. So I went looking.
The way Grok’s bots behave feels very on-brand for Elon Musk…
I ran a simple test in Grok to better understand. I wrote:
“What does the aeo.press llms.txt file say about itself?”
And then Grok happened.
Grok immediately sent 70+ AI bots to pull content. And they were hungry for everything, including .txt files. Compare that to ChatGPT, which sent 10 bots and mostly stuck with HTML.
Another big difference: every major LLM identifies itself clearly. This is ChatGPT’s user agent (notice the tail end):Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +[https://openai.com/bot](https://openai.com/bot%60)
This is Grok’s:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[142.0.0.0](http://142.0.0.0/) Safari/537.36
Grok doesn’t include anything to help you identify it. All it really tells us is that the site was accessed by an older Mac running Chrome.
So Grok plays a very different game:
- No self-identification
- Aggressive parallel crawling
- Happy with any file type to answer a question
Which makes it fascinating… and hard to measure. We just don’t know how big a role Grok plays in the AI space.
