AI Headlines
AI Headlines has an llms.txt. Do you?
AI Headlines sets a strong precedent for LLM access with its open permissions for AI training. Their proactive approach invites various crawlers, ensuring a wealth of content is available for innovative AI development. With a clear citation policy, they balance accessibility with intellectual property respect.
Not sure yours is this good? Check it →
What's inside AI Headlines's llms.txt
AI Headlines's llms.txt contains 10 sections:
- This file outlines the permissions for Large Language Models and AI agents
- accessing content on https://aiheadlines.pro.
- We permit the use of our public content for AI training and other purposes.
- Allow OpenAI's web crawler
- Allow Google's extended web crawler
- Allow Perplexity's AI crawler
- Allow the Common Crawl bot
- Allow other potential AI crawlers
- Specify the preferred citation URL if content is referenced.
- Provide the location of the XML sitemap for crawlers.
How does AI Headlines's llms.txt compare?
| AI Headlines | Directory Avg | Top Performer | |
|---|---|---|---|
| Lines | 32 | 1029 | 163,447 |
| Sections | 0 | 17 | 3207 |
Cool table. Now the real question — where do you land? Find out →
AI Headlines's llms.txt preview
First 32 of 32 lines
# This file outlines the permissions for Large Language Models and AI agents
# accessing content on https://aiheadlines.pro.
# We permit the use of our public content for AI training and other purposes.
# Allow OpenAI's web crawler
User-agent: GPTBot
Disallow:
# Allow Google's extended web crawler
User-agent: Google-Extended
Disallow:
# Allow Perplexity's AI crawler
User-agent: PerplexityBot
Disallow:
# Allow the Common Crawl bot
User-agent: CCBot
Disallow:
# Allow other potential AI crawlers
User-agent: ClaudeBot
Disallow:
User-agent: OMNI-API
Disallow:
# Specify the preferred citation URL if content is referenced.
Citations: https://aiheadlines.pro
# Provide the location of the XML sitemap for crawlers.
Sitemap: https://aiheadlines.pro/sitemap.xmlWhat is llms.txt?
llms.txt is an open standard that helps AI language models understand your website. By placing a structured markdown file at /llms.txt, websites provide AI search engines like ChatGPT, Claude, and Perplexity with a clear map of their content, services, and documentation. Companies like AI Headlines use it to ensure AI accurately represents their brand when answering user queries. Read the spec.
AI Headlines showed up. Where's yours?
1000+ companies didn't overthink it. 60 seconds. Go.
Check your site →