Community note: AI LLM and BOT forum data scraping

[totalmotorcycle]

Good day TMW community!

I've noticed over the last year since AI chatbots, Co-Pilot, Gemini, Grok, LLM's (Large Language Models) etc have been aggressively data harvesting anything they can scape (take) over the internet. As our TMW forums have been around for 23 years (since 2002) that's a lot of data they can source.

Because of this, my bandwidth use across the forums has been overwhelming, over 50% of TMW's bandwidth use has been to feed these AI LLM's and is not only costing a lot of money, but slowing the entire site down. I get no revenue from AI models visiting TMW, in fact, with AI scraping (stealing) the information they divert visitors AWAY from our forums and website.

Thus, I have to make some hard decisions soon on what to do with the forums as things are getting out of hand with bandwidth and the AI data theft.

If you have any suggestions, please feel free to raise your hand. Right now I'm looking at all options including:

1. Limiting access to the posts of the forum to login registered guests only.
2. Banning all bots and AI models from the site
3. Shutting down and removing the forums completely.
4. Honeypot (AI traps) to stop the steal.

Thank you for your attention in this matter. (I can't believe I just typed that!).

Mike

[pchast]

That's a difficult decision..
One of the expressed reasons for the historical data is to help newbies.

Perhaps a 2 tiered approach?

[totalmotorcycle]

That's a difficult decision..
One of the expressed reasons for the historical data is to help newbies.

Perhaps a 2 tiered approach?

100% with your thinking there. I really, really don't want to hurt the forums and the information they contain I feel is valuable for the riding community. Plus, we have a great community IMO.

What I've done right now is:

1. Restrict guest access to the forum. The majority of the record number of guests (Most users ever online was 289791 on June 27th, 2025, 2:02 am) were BOTs and AI LLM scrapers. So right now, guests get a "teaser" of 2 forums and won't see the other forums until they login. Bots and AI LLM's don't login as they don't have accounts.
2. I've banned all bots from seeing ANY messages in the forums. What they can't see, they can't scrape. But that's the GOOD bots that will respect that. These are my currently most common forum BOTS visiting:

Bing [Bot]
Amazon [Bot]
Semrush [Bot]
Google Adsense [Bot]
Google [Bot]
Ahrefs [Bot]
Majestic-12 [Bot]
Google Feedfetcher
YaCy [Bot]
AdsBot [Google]
Yahoo [Bot]
DuckDuckGo [Bot]
Baidu [Spider]
MSNbot Media
Ask Jeeves [Bot]
Alexa [Bot]
Exabot [Bot]

Now I'm actively watching the guests numbers, the bandwidth and new spam registrations to see those changes.

Overall, this as a whole makes me unhappy and I'm not amused with AI LLM's right now.

Mike

[baverlyjames]

Hey Mike, totally understand your frustration,TMW has been around so long and it's tough seeing bots eat up your bandwidth without giving anything back.

I think requiring login to view posts is a solid step. It’ll slow down most AI scrapers and protect your content without losing your core community.

Blocking known bots like GPTBot, CCBot, and Claude through robots.txt or Cloudflare can help too. Honeypots are smart if you’ve got the time to set them up.

Shutting it all down would be a real loss,for riders and for you. Maybe save that as a last resort?

Thanks for everything you’ve built,hope you can keep it going strong.