Research & insights

How AI Agents Use Websites, Where They Fail, and What to Fix

As browser-based AI agents begin navigating real websites, many failures go unnoticed and can silently distort analytics or break critical journeys. This research argues that accessibility engineering provides much of the practical foundation for agent-compatible interfaces, but readiness scores and benchmarks miss what matters - task-level outcomes on your own site. It outlines a failure pipeline and a monitoring approach to measure where agents go wrong.

Two Years Partnering with Profound to Win in AI Search

Celebrating a two-year milestone between Merj and Profound to accelerate the AI search industry

Watch the Page 2 Podcast, Then Join Us for After Hours

How Brave Search discovers new pages: A deep dive into the Web Discovery Project

Claude's web search runs on Brave, and 79% of its citations come from Brave's top 10 results. By reading Brave's open-source Web Discovery Project client, we traced exactly how pages enter that index: through a hard door requiring approximately 20 distributed users, or an easy door triggered by a single search on Google or Bing. The findings reveal specific, testable rules for making your site discoverable to Brave and, downstream, to Claude.

The agent-readable web: serving the right representation to the right crawler

Most web content is still built for browsers, but agents increasingly need a cleaner representation of the same underlying content. Rather than creating a parallel web of separate Markdown URLs, the better model is HTTP content negotiation: one canonical URL that returns HTML for browsers and Markdown for agents that ask for it. This piece lays out a practical rollout covering .md endpoints, caching, and the fidelity checks that determine whether the representation actually improves retrieval.

Update Your Google Crawler IP Range Endpoints

Your Accept-Language Redirects Could Be Blocking Search Engines and AI Crawlers

When Search Stops Playing By Old Rules

Rendering, Style, and Layout: When Things Go Wrong

URL Encoding Done Right: Best Practices for Web Developers and SEOs

Don’t Block What You Want: DuckDuckGo and Common Crawl to Provide IP Address API Endpoints

Introducing our Bing Webmaster Tools API Python Client

How Extra HTML Attributes in Canonical Tags Impact Search Engines

Google's JavaScript Rendering Capabilities

Investigating Reddit's robots.txt Cloaking Strategy

Discovering and Diagnosing a Google AdSense Rendering Bug

Optimising Next.js Image Component for Image Search

Managing Webpages Resources for Efficient Crawling and Rendering

Strategic Use of Typo Domains: UX, Traffic and Brand Protection

The Privacy-First Ad Technologies to Replace Third-Party Cookies

Persistent Kubernetes Storage and Edge Compute – Akamai acquires Ondat

Capturing data from Bing Webmaster Tools API

Validating Session Isolation for Web Crawling to Provide Data Integrity

Bing & Yandex initiative IndexNow highlights the value of treating ‘time-to-index’ as a KPI