Managing Director
Profound, Head of Special Projects
14 min read
When Claude searches the web, Brave Search does the searching, according to Google Cloud’s documentation for Anthropic’s models. Across roughly 35,000 Claude citations analysed by Profound, 79.2% of Claude’s cited URLs appear in Brave’s top 10 for the matching query, compared with 34% for Google’s top 10. Whether Claude cites a page therefore depends largely on whether Brave has indexed and ranked it. How pages enter Brave's index in the first place is barely documented.
To close that gap, we read the source code. Brave states that its crawler is “partially powered” by the Web Discovery Project (WDP), an opt-in data-collection client inside the Brave browser and the one part of the system that is open source. We traced its mechanics at commit f25eb3d. Every mechanical claim below links to the exact lines that prove it; where we describe Brave’s closed, server-side systems instead, we say so in the sentence.
The Web Discovery Project sits inside opted-in Brave browsers and reports what its users see, which indirectly includes what other search engines rank.
What we found, in brief:
- There is no front door. Brave has no search console, no sitemap submission and no IndexNow. The
submit-urlform’s only stated function is re-fetching a page. - The Web Discovery Project has two doors, with wildly different bars. A page discovered through user visits needs around 20 different opted-in users on different networks. A page discovered through another engine’s results page needs one user running one search.
- Brave’s browser reads other engines’ results. Opted-in browsers recognise Google, Bing, Yahoo and DuckDuckGo results pages, then anonymously re-fetch them after a random 1–20 minute delay and report the listed URLs.
- Eligibility has sharp, testable edges. The verification fetch never executes JavaScript, only honours
noindexand canonical in the HTML (never HTTP headers), rejects any redirect, and gives up beyond 10 seconds or 2 MB. - Canonical tags decide what Brave gets told about. When a visited page carries a clean canonical, the page channel reports the canonical, not the URL in the address bar. Without a canonical that differs from the URL, the URL usually has to be almost spotless: a second query parameter or a query string over 22 characters and the page is never reported.
- A signed pattern file is the live kill switch. What the client actually collects at any moment is set by a server-controlled, cryptographically signed file the browser polls hourly. The code ships the machinery; Brave’s servers decide what runs.
Three routes into Brave’s index
Two different machines are involved, and it matters which one a claim is about. The Web Discovery Project (WDP) is the open-source client we examined: it runs in the browser of every user who opts in, and its only output is anonymised reports; it contains no crawler and no ranking logic. Brave’s crawler and index are closed and server-side: for those we have only Brave’s own documentation and, where flagged, our inference.
3 routes
How a URL gets into Brave's discovery pipeline
Any one of these doors is enough. Discovery means getting one URL through a door and letting the crawler do the rest.
- Verified in code
WDP page channel
A page that ~20 different opted-in users visit. High bar, strong quality signal.
- Verified in code
WDP query channel
A results page that one opted-in user views, on an engine where the page ranks. The lowest-friction path.
- Documented
Brave's crawler
Closed-source; Brave documents that it exists and that it is gated on Googlebot-crawlability.
Brave's crawler fetches the seed
InferredIt presumably expands across the site from there, as any crawler would. Brave does not document its expansion behaviour.
The practical consequence: discovery means getting one URL through a door and letting the crawler do the rest, not pushing every page individually. The rest of this article walks through the two doors we can actually see, hardest first.
Route 1, the hard door: pages users visit
When an opted-in user visits a page in a normal tab (private tabs are excluded), the client builds a page message about it. What goes in that message is strictly structural: the URL, title, canonical URL, HTML and text lengths, counts of links, forms, inputs and password fields, page language and country, but never the body text. Engagement is reported only as heavily capped counters (at most 100 mouse moves, 100 scrolls, 10 copy events, and active time that only accrues while the user is actually interacting). The links the user clicked on the page are recorded locally, then nulled just before sending; the click graph never leaves the browser.
One scoping caveat for completeness: Brave’s support article separately documents “fetch jobs”, where an opted-in browser “might be instructed to fetch a webpage and send the HTML back to Brave” for “a small set of innocuous and reputable domains”. That machinery is not part of the client codebase we examined, so the never-the-content guarantee above is a property of the page and query reports specifically.
Before the message can be sent, the page must survive the double-fetch, an anonymous re-fetch that checks the page looks the same to a logged-out stranger as it did to the visitor (the full rule set gets its own section below). Even after it passes, Brave still cannot read the report. The client encrypts each page message with STAR, a cryptographic secret-sharing scheme, attaching one “share” per report. The threshold is a constant in the client: 20. Below 20 shares from different users, Brave’s own design doc reads: “it is impossible for the server to retrieve the content of the page message.”
Quorums can be gamed from a single network, so each report also carries oc, the last octet of the client’s IPv4 address, which the browser cannot know behind NAT and therefore fetches from Brave’s quorum config server. The design doc states the server “will only decrypt page messages which have been sent by enough people with different values of oc”: an office or household sharing one public IP therefore counts as one voice, however many machines it runs. All 20 must also have opted in to WDP, which is off by default. By design, this is not a door that can be pushed open on demand.
Two exemptions are worth knowing, both stated in the design doc (the enforcement is server-side, so we cannot observe it): the quorum applies only to URLs with a real path or query string (https://merj.com/ is exempt as a homepage where https://merj.com/blog is not), and it does not apply to pages carrying a qr field. That second one deserves unpacking, because the qr mechanism is verified in the client: when a user searches on Brave Search itself and clicks through, the pages they land on carry a record of the originating Brave query in their page message, up to two clicks deep. In other words, a click-through from a Brave search both tells Brave which query led to your page and, per the design doc, skips the 20-user wall.
Route 2, the easy door: someone else’s results page
The client ships dedicated recognisers for the results pages of Google (web, image and video search), Bing (web and image), Yahoo and DuckDuckGo, plus Brave’s own results pages. The same pattern list also recognises Amazon search and product URLs and LinkedIn people-directory (/pub/dir) URLs, an unexpected pair, but both are flagged isSearchEngine: false, which keeps them out of the results-harvesting flow described next; what, if anything, is collected for them is decided elsewhere.
Much of the client’s behaviour is, in fact, decided remotely. Each recogniser only activates if matching rules exist in a pattern file that the browser fetches from Brave’s servers, verifies against a pinned RSA-PSS key, and re-polls hourly. If no patterns arrive, nothing is collected. The precise statement is therefore that the browser ships the machinery to harvest those engines, while Brave’s servers hold the on/off switch; which engines are live at any moment is invisible from the client side.
When a recognised results page is viewed, the flow is deliberately indirect:
7 steps
How a URL reaches Brave via the query channel
A single opted-in Brave user searches another engine where your page already ranks. The WDP client recognises the results page, screens the query, builds a clean generic version of the results URL, anonymously re-fetches it after a random delay, extracts the listed URLs and reports them if enough clean results survive, with no 20-user STAR quorum required.
- Verified in code
A user searches Google, Bing, Yahoo or DuckDuckGo
The WDP client ships recognisers for these engines' results pages, activated by Brave's server-side pattern file.
- Verified in code
The query is screened
Personal or suspicious queries are dropped before anything is reported.
- Verified in code
A clean, generic results-page URL is built
The client rebuilds the URL as engine hostname plus encoded query, with nothing from the user's session.
- Verified in code
The results page is anonymously re-fetched
The URL is queued with a random 1-20 minute delay, then fetched with no cookies, no identity and no temporal link to the user's actual search.
Why this matters
The random delay and stripped identity make it impractical to correlate the re-fetch with the original searcher.
- Verified in code
Result URLs and titles are extracted
The pattern-driven extraction engine pulls the result URLs and titles out of the fetched page. The result URLs are listed, never individually fetched.
- Verified in code
The message is sent only if enough results survive
At least four results must survive URL sanitisation; fewer suggests a query too specific to share safely.
- Verified in code
The URL reaches Brave
Unlike the page channel, this query-channel message carries no 20-user STAR quorum; one opted-in user's search on an engine where you rank is enough.
That flow is the entire gate. Query-channel messages carry no STAR quorum: STAR is injected for page messages only. One opted-in user, one search on an engine where you rank, and your URL reaches Brave. (The client is even careful not to report the results page itself as a visited page.)
Side by side, the asymmetry is the story:
|
| |
|---|---|---|
Trigger | A WDP user visits the URL | A WDP user searches an engine where the URL ranks |
Client verifies the URL? | Yes: anonymous double-fetch plus structural comparison | No: result URLs are listed, never fetched |
Quorum before Brave can read it | 20 reports, on different networks (STAR) | None: a single search suffices |
Other gates | The full eligibility check below | Clean query; ≥4 clean results on the page |
The takeaway: ranking on Google or Bing is not just a path into Brave’s index; it is the cheapest and most controllable one, because a single searcher triggers it. Brave bootstraps its index partly by reading what the incumbents already rank, which is why classic search visibility underpins Brave visibility, and downstream, Claude’s.
The eligibility gauntlet: what the double-fetch filters out
The hard door’s verification step deserves a closer look, because it is where technical SEO detail decides outcomes. The double-fetch re-requests the visited page with no cookies and no credentials, then compares the anonymous version against what the user saw. It is a public-page filter with sharp edges:
- Any redirect means private. If the anonymous fetch does not land on exactly the requested URL with HTTP 200, the page is rejected as private; the code comment literally begins
DANGER. - Login walls are inferred structurally. If password inputs or forms appear in the anonymous version but not the user’s, or titles vanish, or the HTML-length and link-count ratios between the two versions are badly imbalanced, the page is dropped.
- JavaScript is never executed. The double-fetch reads raw HTML off the wire. Client-side-rendered content simply does not exist at this layer. Anything that must be discoverable needs to be server-rendered or pre-rendered.
- There is a performance budget. The fetch aborts after 10 seconds and refuses responses over 2 MB (checked via the
content-lengthheader). Slow or bloated pages fall out silently, after at most three attempts. - Messy URLs get dropped, and canonical is the escape hatch. The sanitiser rejects the extremes outright: embedded credentials, uncommon ports, IP hostnames, hostnames over 50 characters, URLs over 800 characters, and a specific word-list of path segments (
login,token,checkout,account,adminandwp-admin, among others). After that, what matters is the canonical. A page whose canonical points somewhere shorter on the same host, as a parameterised URL canonicalising to its clean form does, is judged gently, allowing up to four query parameters and 30 characters of query string, and the canonical is what gets reported in any case. A canonical that merely repeats the page’s own URL earns no such credit. Without that credit, or a link from a publicly verifiable referring page, the page is judged strictly: a second query parameter or a query string over 22 characters marks the page private, as do number and hash tokens that look like identifiers, which catches most click IDs.
Two findings surprised us most, because they diverge from how Googlebot behaves:
Signal | Honoured in HTML? | Honoured as HTTP header? |
|---|---|---|
| ✅ Yes: any occurrence of the string “noindex” anywhere in the | ❌ |
Canonical | ❌ the |
The header blindness is structural: across the whole client, the only response headers ever inspected are content-length and location. A page noindexed only via X-Robots-Tag, or a PDF whose canonical lives in a Link: header, is invisible to these checks. One boundary note: these are the client’s eligibility rules. Brave’s crawler may well honour those headers when it fetches; we cannot see it, so treat this as “WDP will not filter on them” rather than “Brave ignores them”.
After discovery: the part we cannot see
Everything downstream of the seed report is closed, so this section is short and sourced from Brave’s documentation. The crawler help page states the one hard rule: “if a domain or page is not crawlable by Googlebot, then Brave Search’s bot will not crawl it either.” The same page explains why it will not appear in your logs: “The Brave Search crawler does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them.” (Third-party bot directories list a Bravebot/1.0 string, but Brave’s own documentation contradicts it; we side with Brave’s page.) A blanket “block everything except Googlebot” rule therefore cuts Brave off silently; robots.txt rules that block Googlebot block Brave by extension.
Brave does not document how its crawler expands from a seed. Sitemaps and internal links presumably work the way they do for every crawler, but that is our inference rather than Brave’s statement, and there is no way to submit a sitemap regardless. Separately, Brave documents Google fallback mixing: for thin result sets, users can opt to let the browser “anonymously check Google for the same query”, which “sends the query results back to Brave Search so we can improve responses next time”. That is one more documented channel through which incumbent results reach Brave.
How often the crawler visits, and what it does with WDP’s engagement signals in ranking, cannot be answered from outside. Those are exactly the questions our follow-up article addresses empirically, by taking pages Brave has not indexed and testing each lever (the submit-url form, the query channel, distributed visits) one at a time.
What this means for your site
- Earn classic visibility first. The query channel converts your Google and Bing rankings into Brave seeds automatically. If you rank for queries real Brave users run, discovery follows; per the design doc, click-throughs from Brave’s own results even bypass the page-channel quorum.
- Server-render anything that must be found. The discovery layer never executes JavaScript.
- Put
noindexand canonicals in the HTML, not only in HTTP headers: at this layer, headers are not read. (Keep header and HTML directives consistent anyway; the crawler may read both.) - Keep public pages identical logged-out. Redirects to consent walls, login prompts or geo-gates on the canonical URL read as “private” and end discovery for that page. Serve the same canonical to both states as well, because the comparison declines pages whose canonicals differ between the two fetches.
- Keep URLs short, clean and fast. A canonical that differs from the page’s own URL is your safety net: without one, a second query parameter or a query string over 22 characters loses the report. Risky path words such as
token, odd ports and slow (>10 s) or heavy (>2 MB) responses drop out. - Do not block unknown crawlers. Brave’s bot is deliberately indistinct, and Googlebot-crawlability is its stated prerequisite.
- Volume alone does not accumulate. The 20-user quorum counts distinct opted-in users on distinct networks; reports sharing the same IP last-octet count once, however many machines or visits they represent. Activity concentrated on a single network therefore does not add up towards the quorum. The architecture ties discovery to distribution across networks rather than to raw visit volume.
What’s proven, what’s inferred
Everything in the left column is verifiable, either in the pinned client source or in Brave’s own pages. The right column is reasoned inference about systems we cannot see; treat it as hypothesis, not fact.
Hard facts (code / Brave docs) | Working hypotheses (inference) |
|---|---|
The WDP client only reports URLs and metadata; it contains no crawler and no ranking logic | Brave’s crawler expands from seeds via internal links and sitemaps, as crawlers conventionally do (likely) |
Two channels, two bars: | Incumbent rank position feeds Brave’s own ranking; the client collects ordered results, but server-side use is unprovable from outside (medium) |
The harvest flow covers Google, Bing, Yahoo and DuckDuckGo; Amazon/LinkedIn recognisers exist but sit outside it; live scope is set server-side | WDP engagement counters (active time, scrolls, copies) are live ranking inputs (medium) |
| Optimising for Brave is, in practice, optimising for Claude’s citations (strong on the link at 79.2%; weaker on durability) |
Page reports carry no body text, no click graph, no persistent ID; WDP is opt-in and off by default | Brave’s crawler honours |
Brave’s crawler is Googlebot-gated and deliberately advertises no distinctive user-agent | |
Google Cloud’s documentation lists Brave Search as the sole third-party service behind Claude’s web search; 79.2% of Claude’s citations sat in Brave’s top 10 in June 2026 (Profound) | The Brave dependency persists in its current form; Anthropic’s own Claude-SearchBot crawler shows in-house search capability under construction (holds as of July 2026, not guaranteed) |
On independence, since it always comes up: Brave is not a Google or Bing wrapper. At launch in June 2021 about 13% of queries needed third-party help; Brave dropped Bing’s index entirely in April 2023 and serves 100% of default results from its own index. It bootstraps discovery from the incumbents through the query channel above, while engineering the dependency down.
FAQ
Anthropic has never announced it, but the trail is consistent: Anthropic added Brave Search to its subprocessor list on 19 March 2025; Simon Willison then found a BraveSearchParams object inside Claude’s web-search tool schema, and Claude’s ten citations for his test query matched Brave’s results exactly; and Google Cloud’s documentation for Anthropic’s partner models lists Brave Search as the only third-party search service the feature calls. Two caveats belong alongside that trail. Anthropic’s own developer documentation names no search provider, and Anthropic operates a crawler of its own, Claude-SearchBot, documented as navigating the web to “improve search result quality for users”, which points to in-house search infrastructure. The relationship is well evidenced as of July 2026, but it is neither confirmed by Anthropic nor guaranteed to persist.
Not for new pages. There is no search console and no sitemap intake; the submit-url form’s only stated function is re-fetching a page. Brave is also not an IndexNow participant (participants include Bing, Yandex, Seznam.cz, Naver and Yep).
The page channel requires reports from about 20 different opted-in users on different networks before Brave can even decrypt the URL. The query channel requires a single opted-in user’s search on an engine where the page ranks.
No. Reports sharing your network’s IP last-octet do not accumulate towards the quorum, and only opted-in users count.
The client ships recognisers for Google, Bing, Yahoo and DuckDuckGo results pages and harvests them via anonymous, delayed re-fetches, but which engines are live at any moment is controlled by Brave’s signed server-side pattern file.
Yes. Brave’s crawler requires Googlebot-crawlability and deliberately arrives without a distinctive user-agent, so “block all unknown bots” tends to block Brave.
The page and query reports carry structure and capped engagement counters, never body text, and clicked links are nulled before sending. (Brave’s docs separately describe opt-in “fetch jobs” for a small set of reputable domains, which do return HTML.)
The foundations overlap (that is the point of the query channel), but the JS-blind double-fetch, the header blindness and the quorum are a distinct technical layer with their own failure modes.
Appendix: method and citations
Method. All code claims were verified by inspecting Brave’s open-source web-discovery-project client at commit f25eb3d (June 2026). We traced the specific mechanisms described here rather than auditing the whole codebase, and every link opens the exact lines at that commit, so references stay stable if Brave changes the code. Brave can change both the code and, faster still, the signed pattern file at any time; treat this as a snapshot. Claims about Brave’s servers come from the linked Brave documentation; explicit inferences are marked in the text.
Key code references:
What it shows | Where |
|---|---|
Engine recognisers: Google (web/image/video), Bing (web/image), Yahoo, DuckDuckGo; Amazon and LinkedIn URL types flagged | |
Brave’s own SERP recogniser | |
Recognisers inert without server-side pattern rules | |
Signed pattern file: pinned RSA-PSS keys, hourly poll | |
Cleaned query URL built for the anonymous re-fetch | |
SERP re-fetch queued with random 1–20 minute delay | |
Pattern-driven extraction engine (selectors live in the pattern file) | |
Four-clean-results gate and per-result sanitisation | |
STAR applied to | |
Quorum threshold of 20 | |
|
|
Quorum mechanics and exemptions in the design doc (bare homepages; | |
Brave-search click-throughs carry the originating query ( | |
Double-fetch: no cookies, 10 s timeout, non-200 and redirects rejected | |
2 MB cap; only | |
Structural comparison of user vs anonymous fetch (login-wall detection) | |
| |
Canonical read from the HTML | |
URL sanitiser: risky path words, truncation, drops | |
Gentle vs strict URL limits: four params/30-char query string with a same-host canonical shorter than the URL, one param/22 chars without | |
Strictness set by canonical, allowlist and verified public referrer | |
Canonical substituted as the reported URL; messy URLs rescued by canonical or dropped | |
Page message structure (no body text) | |
Clicked links nulled before sending; engagement counters capped; active time gated on real interaction |
|
Private tabs excluded; visited SERPs excluded from the page channel |
Brave documentation and external sources:
- Brave Search crawler help (Googlebot gate; no distinctive user-agent): search.brave.com/help/brave-search-crawler
- What is the Web Discovery Project (opt-in default; 20-person quorum; metadata not content; fetch jobs): support.brave.app/hc/en-us/articles/4409406835469
- WDP design doc (quorum mechanics,
oc, exemptions):modules/web-discovery-project/sources/README.mdatf25eb3d - STAR protocol paper: arxiv.org/abs/2109.10074
- Brave
submit-urltool (“Insert the URL to be re-fetched”): search.brave.com/submit-url - Google fallback mixing: search.brave.com/help/google-fallback
- Brave Search independence (13% third-party at launch; 100% since April 2023): brave.com/search-independence/, and TechCrunch’s coverage: Brave Search doesn’t use Bing’s index anymore
- Brave Search behind Claude’s web search: Google Cloud, Claude partner-model web search docs; Simon Willison on the subprocessor listing and
BraveSearchParams; TechCrunch coverage - 79.2% of Claude citations in Brave’s top 10 (~35,000 citations, 400 queries; 34% for Google’s top 10): Josh Blyskal, Profound, Zero Click NY
- Anthropic’s web crawlers, including the Claude-SearchBot search crawler: Does Anthropic crawl data from the web?
- IndexNow participants (no Brave): indexnow.org/faq
- Brave Search API: brave.com/search/api
Merj and Profound work together to help brands win in AI search. Profound shows you how AI answer engines talk about your brand across millions of conversations. Merj supports on the infrastructure layer: testing, building and shipping the changes that get the right content to the right bots for the right reasons. See where you stand with Profound, and talk to us about what to do next.
Next in this series: we take pages Brave has not indexed and test each discovery lever in isolation.
