GossipLankaBot

GossipLankaBot is the crawler behind the Gossip Lanka news aggregator. It reads public RSS feeds, sitemaps and article pages, respects robots.txt and links back to every original story.

How it identifies itself

Every request sends this User-Agent:

GossipLankaBot/1.0 (+https://gossiplanka.news/bot)

What it does

• Reads RSS feeds and sitemaps first, and only fetches article pages that they list.
• Obeys robots.txt, including Disallow rules.
• Limits itself to a few requests per minute for each site.
• Stores a headline, a short summary and the lead image, and always shows the source name with a link to the original article.
• Never signs in, submits forms or bypasses paywalls.

Opting out

To stop GossipLankaBot, add this to your robots.txt:

User-agent: GossipLankaBot
Disallow: /

Publishers can also email news@gossiplanka.news to have content removed or corrected.