Two Years Partnering with Profound to Win in AI Search

How Brave Search discovers new pages: A deep dive into the Web Discovery Project

Claude's web search runs on Brave, and 79% of its citations come from Brave's top 10 results. By reading Brave's open-source Web Discovery Project client, we traced exactly how pages enter that index: through a hard door requiring approximately 20 distributed users, or an easy door triggered by a single search on Google or Bing. The findings reveal specific, testable rules for making your site discoverable to Brave and, downstream, to Claude.

The agent-readable web: serving the right representation to the right crawler

Most web content is still built for browsers, but agents increasingly need a cleaner representation of the same underlying content. Rather than creating a parallel web of separate Markdown URLs, the better model is HTTP content negotiation: one canonical URL that returns HTML for browsers and Markdown for agents that ask for it. This piece lays out a practical rollout covering .md endpoints, caching, and the fidelity checks that determine whether the representation actually improves retrieval.

Your Accept-Language Redirects Could Be Blocking Search Engines and AI Crawlers

When Search Stops Playing By Old Rules

Rendering, Style, and Layout: When Things Go Wrong

Don’t Block What You Want: DuckDuckGo and Common Crawl to Provide IP Address API Endpoints