Research &
Insights

The agent-readable web: serving the right representation to the right crawler

Most web content is still built for browsers, but agents increasingly need a cleaner representation of the same underlying content. Rather than creating a parallel web of separate Markdown URLs, the better model is HTTP content negotiation: one canonical URL that returns HTML for browsers and Markdown for agents that ask for it. This piece lays out a practical rollout covering .md endpoints, caching, and the fidelity checks that determine whether the representation actually improves retrieval.

Update Your Google Crawler IP Range Endpoints

Your Accept-Language Redirects Could Be Blocking Search Engines and AI Crawlers

When Search Stops Playing By Old Rules

Rendering, Style, and Layout: When Things Go Wrong

URL Encoding Done Right: Best Practices for Web Developers and SEOs

Don’t Block What You Want: DuckDuckGo and Common Crawl to Provide IP Address API Endpoints

Introducing our Bing Webmaster Tools API Python Client

How Extra HTML Attributes in Canonical Tags Impact Search Engines

Google's JavaScript Rendering Capabilities

Investigating Reddit's robots.txt Cloaking Strategy

Discovering and Diagnosing a Google AdSense Rendering Bug

Optimising Next.js Image Component for Image Search

Managing Webpages Resources for Efficient Crawling and Rendering

Strategic Use of Typo Domains: UX, Traffic and Brand Protection

The Privacy-First Ad Technologies to Replace Third-Party Cookies

Persistent Kubernetes Storage and Edge Compute – Akamai acquires Ondat

Capturing data from Bing Webmaster Tools API

Validating Session Isolation for Web Crawling to Provide Data Integrity

Bing & Yandex initiative IndexNow highlights the value of treating ‘time-to-index’ as a KPI

12 Experiments for Tabbed Content SEO

How to Attack & Prevent Subdomain Hijacking with DNS Governance

A Study of HTTP/2 & Push: Coverage of the top 100,000 websites

Technical SEO Roundup March 2020