Bing & Yandex initiative IndexNow highlights the value of treating ‘time-to-index’ as a KPI

Finding ways to index content in the quickest, most efficient way possible has long been one of the pillars of organic search and technical SEO.

It’s a complex landscape though, and one that’s easy to get lost in. Let’s take the example of a large-scale eCommerce business. With thousands of new product pages added and removed daily, slow or inaccurate indexing can translate directly into loss of traffic, and potentially sales.

It’s not just the indexing of new pages that can be problematic, either. Think of the implications slow indexing might have for a company that’s been asked to remove a page for legal reasons. This is a rarer problem to be sure, but one that can have weighty consequences, not least around reputational damage.

It’s welcome news to many, then, that an addition to the current indexing toolkit has been introduced by Microsoft Bing and Yandex. The initiative, called IndexNow, means any search engine that wants to can enable site owners to easily notify search engines of new, updated, or deleted content.

“Once search engines are notified of updates they quickly crawl and reflect website changes in their index and search results,” Microsoft writes on its blog.

Websites will be allowed to submit up to 10,000 URLs a day. Of course, this means those with more than this number will have to prioritise which URLs they want to flag, and in turn to understand what pages matter to them most at different points in the year. (This is particularly relevant to those in seasonal industries, like fashion and sports.)

It’s worth noting as well that the development comes just in time for Black Friday and Cyber Monday. It shouldn’t be overlooked that IndexNow provides another valuable indexing tool for eCommerce businesses, who naturally have their work cut out for them during this major annual sales period.

We’re heading towards a more efficient Internet

One of IndexNow’s most powerful features is that URLs submitted to an IndexNow-enabled search engine automatically get submitted to all other IndexNow-enabled search engines. This is the first indexing initiative to offer anything like this. Similar protocols up till now have only applied to the search engine at hand:

• Bing already has an API plug-in that allows users to flag updated content, but URLs submitted are only flagged to Bing.
• Google’s indexing API only applies to job postings and livestream content at the moment.

All this feeds into a wider vision that’s only going to grow more prevalent in coming months and years. Microsoft calls this vision out in its press around IndexNow, saying that the aim of the initiative is to “make the Internet more efficient“. IndexNow helps business get the most timely and relevant information to their customers while creating a better overall experience for everyone” Bing says.

A more efficient Internet, of course, is not only good news for businesses and consumers, but for the environment too. Any innovations around indexing speed have a positive knock-on effect on the Internet’s carbon footprint. In other words, the less computational energy that’s required to index pages, the less of an impact the Internet has environmentally.

Right now, Bing and Yandex are the two search engines that have adopted the protocol, but any are welcome to take part. So, who will be to follow?

We think it’s likely that more of the smaller local search engines may adopt IndexNow, like the citizens of the Czech Republic, for example, who often use local search engines. Google hasn’t disclosed any plans to follow suit yet, but we don’t think it’s too far-fetched to expect it to in the near future. Likewise, we think there’s potential for Naver to adopt the protocol too.

What’s perhaps most indicative of IndexNow’s potential adoption rate is the fact that Akamai and Cloudflare – two of the biggest content network delivery providers – have already integrated it.

Content Management Systems that have plans to support it in the coming weeks (according to this Bing blog) include:

• Wix
• Duda

Some search industry players are more tentative, though. Joost de Valk, founder of WordPress’s SEO plugin Yoast SEO, Tweeted recently: “We haven’t added support for @yoast SEO for this, the reason is simple, we haven’t seen its value yet.”

IndexNow should be used in line with a full range of indexing methods

IndexNow is definitely an interesting addition to the indexing inventory at businesses’ disposal. But it’s crucial to remember that the process of promoting faster indexing is multi-faceted. There are lots of techniques a company can use to reduce their time-to-index metric, and they should use all methods that they can.

For example, techniques like using XML sitemaps are still highly valuable. (This is the traditional way of listing out all URLs in XML for search engines to come and check.) Likewise, showcasing your most important links closest to your homepage or other key pages is a crucial part of helping search engines find your new or updated content faster. RSS feeds and the WebSub protocol are also useful indexing tools that shouldn’t be overlooked (more on these techniques below).

With all this in mind, the value of working with a partner to help tackle the technical landscape of indexing really can’t be underestimated.

All businesses should treat time-to-index as a KPI

Sure, IndexNow hasn’t come from Google. But the fact it’s been brought from concept to execution by two of the world’s major search engines feels symbolic of something – mainly that the topic of faster indexing should be consistently given the respect it’s due. Businesses should, in reality, be treating time-to-index as a KPI. And they should be handling this metric with the same level of diligence and care as any other marketing or operational indicator. It feels fitting to take a moment, here, to give credit to Bing and Yandex for driving this initiative. It’s always exciting to see a positive and proactive move coming from outside of Google’s arena.

At Merj, we help our clients improve their time-to-index metric by taking a fully holistic and granular approach to indexing. As mentioned above, this really isn’t a pick-and-choose situation. Businesses should use all indexing tactics available to them, although we understand this may be easier said than done due to technical platform constraints and limitations on the data available to companies. We help our clients overcome these barriers by researching, prototyping and integrating solutions that provide the maximum coverage.

Here’s a quick breakdown of some of the techniques we use:

Dynamic XML Sitemaps

One solution we have developed is the use of temporary dynamic XML sitemaps. This involves generating dynamic XML sitemaps with new web pages, which are then controlled by the server log monitoring. This allows us to keep track of when search engines visit those new webpages. Once a visit has been triggered, we can then remove it from the appropriate dynamic XML sitemap so search engines can easily access other URLs without any noise.

RSS & ATOM Feeds

These feeds primarily allow for third-party aggregators like Feedly and Slack to show new content to subscribed users. However, Google allows URLs to be submitted through RSS feeds to discover new content too.

Websub

Websub (formerly PubSubHubbub) is an extension to the RSS feed whereby a producer and consumer stream is enabled. Similarly to IndexNow, Websub allows search engines to listen for updates. Although it is supported by Google, this method has had less adoption in the past, potentially due to lack of examples and documentation.

Timestamping

We create a time-series event notification audit that timestamps any new web pages that are added to a website. Web pages are classified into page types allowing us to view individual page changes or at an aggregated view. The new webpages are pushed out to the various indexing services, recording the time they were submitted.

Internal linking

Linking to the most important new web pages from existing important web pages can help search engines discover those new web pages faster. This is because existing web pages will have a higher crawl frequency. Important web pages may include the homepage, level one product listing web pages or top-level line of business web pages.

Ultimately, IndexNow pulls our attention back around to a theme that’s only going to grow more prevalent as the Internet expands. It signifies the need for constant ideation around how we handle such a gargantuan, complex landscape – and we, for one, are here for it.