A Study of HTTP/2 & Push: Coverage of the top 100,000 websites

Development News

01 Sep 2020
13 minutes
By Roman Borissov
Author Avatar

As search engines prioritise methods for providing the best possible experience for a user base that is increasingly mobile, and websites seek a technical edge over their competitors, website performance and speed have become an even more crucial aspect of SEO.

One way that webmasters may look to improve their website performance is by enabling HTTP/2, which boasts the benefits of increased security, usability, and most importantly, speed.

In this article, we have conducted a series of tests and examinations to evaluate:

  1. The benefits that HTTP/2 offers
  2. How HTTP/2 compares to HTTP/1.1
  3. How many websites are utilising and benefiting from HTTP/2’s "push" feature.

First, however, here is a recap of what HTTP is and how it works.

A Short History of HTTP

The Hypertext Transfer Protocol (HTTP) is a type of protocol that facilitates the fetching of resources and serves as the foundation of exchange on the internet.

When a client (usually a web browser), initiates a request, a document is constructed from a series of sub-documents and delivered. Usually, these sub-documents consist of texts, images, videos, scripts, and other vital components.

Communication on the web is conducted through requests and responses between clients and servers. A standard and simplified sequence of events are as follows:

  1. A client makes a HTTP request to the web.
  2. A web server receives the request.
  3. The server processes the request.
  4. A HTTP response is returned to the client.
  5. The client receives the request.

Development of HTTP began in 1989 at CERN and was co-ordinated by Sir Tim Berners-Lee. Since its introduction in the early 1990s, there have been a series of evolutions in the protocol, most notably:

  1. HTTP/0.9
  2. HTTP/1.0
  3. HTTP/1.1
  4. HTTP/2

Released in 1997, HTTP/1.1 is the most common protocol today, although, since its release in 2015, HTTP/2 is becoming increasingly popular. Being that HTTP/1.1. and HTTP/2 are the subjects of our study, it is these protocols that we shall now discuss.

What is HTTP/1.1?

Officially released in January 1997, HTTP/1.1 served as an enhancement of HTTP/1.0 and introduced several enhancements, including:

  • Pipelined connections
  • Chunked transfers
  • Compression and decompression
  • Content negotiations
  • Virtual hosting
  • Faster responses
  • Bandwidth savings

While HTTP/1.0 supported GET, HEAD, and POST calling methods, HTTP/1.1 also supports PUT, DELETE, TRACE, and OPTIONS.

Previously, connections were both open and closed for each application request, but HTTP/1.1 introduced a persistent connection that allows for multiple requests to be either batched or pipelined.

The pipelining of requests means that the sequences of requests are reduced, so fewer information packets are sent. As a result of the pipelining, Transmission Control Protocol layers (TCPs) are more efficient, all of which results in less traffic and faster performance.

HTTP/1.1’s virtual hosting also meant that multiple domain names could share the same IP address, which simplified processing for web servers.

What is HTTP/2?

Calls for HTTP/2 proposals originated early in 2012, while the final specification was published in May 2015. The goal of the new protocol was to reduce web page load latency by focussing on prioritisation, multiplexing, and compression.

HTTP/2 introduced a range of new features, including the binary framing layer. Whereas HTTP/1.1 maintains formats in plain text format, HTTP/2 uses the binary framing layer to enclose all messages in binary.

While HTTP/1.1 also optimises its messages, the HTTP/2 binary framing layer encodes requests and responses, cutting them up into smaller packets of information. This increases the flexibility of the transfer.

Let’s take a look at the major differences between HTTP/1.1 and HTTP/2 that affect website performance.

Server Push

HTTP/2 allows for servers to send assets to clients before a request is made. This is more efficient, as modern websites often require assets from several resources.

The server can also send information about what it should expect from the server in the future. This allows for increased rendering speeds.

Multiplexing

A limitation of HTTP/1.1 is that it loads resources sequentially, which may cause head-of-line blocking. This is where the packet at the front of the line blocks proceeding packets from being transmitted.

HTTP/2, however, uses a single TCP connection to send multiple packets of data asynchronously. If one resource becomes blocked, therefore, it does not affect the loading of any other. This is considered one of the most important benefits of HTTP/2.

Header Compression

Both HTTP/1.1. and HTTP/2 compress HTTP messages, but HTTP/2 uses a method known as HPACK, which cleans redundant information within HTTP header packets.

Due to the large number of packets involved in the loading of every page, this can have a positive impact on page load times.

Disadvantages of HTTP/2

Although there are many advantages to HTTP2, there is one primary caveat that is worth bearing in mind. Visitors using 3g connections could experience difficulties if they lose connection to a site. This is because a broken connection will stop sites on HTTP/2 from loading.

This is in contrast to HTTP/1.1, which uses multiple connections, so if one breaks, the others will continue to load.

For example, if a website relied on a local audience that lived in an environment with poor 3G coverage, it would be advantageous for the site to remain on HTTP/1.1.

How to use HTTP/2 Push

Setting up HTTP/2 Push has a low barrier to entry, requiring only that the Link HTTP header be set. The three available preload types are style, script and image.

Link: </assets/application.css>; rel=preload; as=style
Link: </assets/init.js>; rel=preload; as=script
Link: </assets/logo.png>; rel=preload; as=image

The header can be set directly at server level (Apache, Nginx, IIS) or at programming language level (PHP, Ruby, Java, Golang, etc).

A PHP example would use the header function as follows:

header("Link: </css/styles.css>; rel=preload; as=style");

A Quick Note on HTTP/3

Although HTTP/2 was released just five years ago, HTTP/3 is the upcoming protocol and is currently internet-drafted.

Like its predecessors, HTTP/3 promises faster and more reliable and secure connections. Major browsers, including Chrome, Firefox, and Safari have already implemented the protocol.

It is also supported by Cloudflare’s edge network.

Originally named Hypertext Transfer Protocol (HTTP) over QUIC, the protocol has no official release date and refinement by the Internet Engineering Task Force (IETF) is still taking place.

Why Test HTTP/2's Uptake?

With HTTP/3 on the horizon, and being that HTTP/2 is now just over five years old, we sought to understand how many websites were using the protocol, and how the implementation affected performance.

We wanted to know if users interacting with some of the most popular websites in the world were benefiting from HTTP/2.

Our Research and Initial Findings

To do this, we tested the most popular 100,000 websites as identified by the SEO tool suite Ahrefs.

Our foundation research and analysis revealed that out of the top 100,000 domains, only 46,977 (47%) accept the HTTP/2 protocol. Additionally, only 702 (0.7%), are utilising HTTP/2’s server push feature.

Interestingly, out of the 100,000, only 77 websites (0.07%) were found to utilise HTTP/2s push feature potential to at least 50%, meaning that in the vast majority of cases, less than half the number of total assets loaded were utilising server push.

Tools and Methodology

To carry out our tests and research, we used both Curl 7.69.1 and PHP 7.

Metrics used

When carrying out our research, we attained data for and analysed two metrics for each domain:

  • Pretransfer_time
  • Starttransfer_time

Comparing HTTP/1.1 vs HTTP/2

Once we knew how many of the top 100,000 identified websites were using HTTP/2, we investigated how much of a difference in performance HTTP/2 made to websites.

Tests were carried out on all domains using HTTP/2, both with HTTP/2 enabled and disabled.

To do this, we used CURLOPT_HTTP_VERSION => 3 for HTTP/2 and CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1 for HTTP/1.1 requests.

To ensure that the tests were fair, and to eliminate possible discrepancies, each domain was tested a total of six times.

  • Three with HTTP/2
  • Three with HTTP/1.1

Tests Using Pretransfer_Time

Pretransfer_Time represents the amount of time in seconds it takes from the start of the process until the moment the file transfer is about to begin. It includes pre-transfer commands and negotiations specific to the protocols involved.

We found that on average, the pretransfer_time on HTTP/1.1 sites took 0.29 seconds, while using HTTP/2, this took 0.21, representing an average saving of 28%.

Interestingly, we discovered that the site with the slowest HTTP/1.1 pretransfer_time, at 7.25 seconds, enjoyed savings of 6.90% with a HTTP/2 pretransfer_time of 6.78 - representing a real-time difference of 0.47 seconds.

The real differences, however, are those sites with the quickest pretransfer_time. A site with a HTTP/1.1 pretransfer_time of 0.0033 enjoyed a 67.9% saving on HTTP/2, reducing it to 0.0012 seconds. This represents a real-time difference of 0.0021 seconds.

Pretransfer_timeHTTP/1.1HTTP/2% savings in time
MIN0.0033056670.00196967.88555951
MAX7.2478413336.7795276.907772966
AVG0.2851711330.20611227638.3571804

Tests Using Starttransfer_time

Starttransfer_time represents the amount of time, in seconds, it took from the start until the first byte is just about to be transferred. This includes CURLINFO_PRETRANSFER_TIME and also the time the server needs to calculate the result.

Using starttransfer_time, we found that on average, sites saved 19.06% of transfer time, between HTTP/1.1 and HTTP/2. On HTTP/1.1. The average starttransfer_time stood at 0.73 seconds, while HTTP/2 came back at an average of 0.61 seconds.

Again, the sites with the quickest starttransfer_times enjoyed the greatest savings, with the maximum saving being 34.09% with a HTTP/1.1 starttransfer_time of 0.004 seconds and a HTTP/2 starttransfer_time of 0.003.

The site with the maximum starttransfer_time from our tests received an average saving of 2.77%, with a HTTP/1.1 starttransfer_time of 9.887047 seconds and a HTTP/2 starttransfer_time of 9.619694667 seconds.

Starttransfer_timeHTTP/1.1HTTP/2% savings in time
MIN0.0040706670.00303566734.09465247
MAX9.8870479.6196946672.779218495
AVG0.7259504280.60968664419.06943269

Comparing HTTP/1.1 vs HTTP/2 + Push

As discussed earlier in the article, one of the primary advantages of HTTP/2 is that it utilises "push" technology. In this experiment, we compared the loading times between HTTP/1.1 and HTTP/2 + push.

Tools and Methodology

For this comparison, we used different technologies for our testing, and chose to use Sitespeed.io software and Google Chrome Version 81.0.4044.113.

For HTTP/1.1 checks we used the following command line:

browsertime https://domain.com --chrome.args='--disable-quic' --chrome.args='--disable-http2' -n=3 --plugins.add=analysisstorer --cacheClearRaw=true --timeouts.pageLoad=15000 --timeouts.script=15000 --timeouts.pageCompleteCheck=15000 --chrome.blockDomainsExcept=*domain.com

For HTTP/2 checks we used:

browsertime https://domain.com -n=3 --plugins.add=analysisstorer --cacheClearRaw=true --timeouts.pageLoad=15000 --timeouts.script=15000 --timeouts.pageCompleteCheck=15000 --chrome.blockDomainsExcept=*domain.com

Metrics used

We used different metrics for comparing HTTP/1.1 and HTTP/2 + push. For this research, we selected the “Resource Duration” time required to download all assets from a site.

To ensure fairness, all third-party assets were blocked by using --chrome.blockDomainsExcept=*domain.com to minimise the influence of any third-party servers.

By using *domain.com specifically, we were able to allow assets from the same domains, subdomains, and CDNs used by site owners.

Our Research and Findings

Although 702 of the 100,000 sites from Ahrefs utilised HTTP/2 + push, due to the methodology of our testing we were only able to analyse 572 domains.

This is because each domain underwent the test process three times with HTTP/2 and three times with HTTP/2 disabled. Any domain that did not return data in any one of those tests was dropped from our analysis.

On average, we found a resource duration difference of 95.39 and a fully loaded difference of 6.65% between HTTP/1.1 and HTTP2.

Difference H1/H2Resource Duration (%)Fully Loaded (%)
MIN-71.98695987-68.03140217
MAX4800177.1149144
AVG20.46486166.063676215

From the table above, we can see that some domains are losing up to 70% of their speed while using HTTP/2. Others, however, benefit by up to 4800%.

When we analysed the sites, they returned a HTTP code 200 and used HTTP header “link” to utilise the HTTP/2 push feature.

As expected, this was faster than HTTP/2 without push, and our findings confirmed that HTTP/2 is 19% faster, while HTTP/2 + push is faster by 20.4%.

However, we believe that there are some issues with the way some sites utilise push technology.

Therefore we analysed the sites using “browsertime” and recorded the number of loaded assets. The domain headers were then parsed to investigate how much of the resources are pushed.

Assets loadedAssets pushedDifference in %
MIN100
MAX25085250
AVG40.982486876.90893169924.23103432

As we can see, on average, only 24% of assets were pushed by sites. This is not an efficient use of server push and explains why the difference between HTTP/2 and HTTP/2 + Push sites provides only a slim margin.

In order to get a more comprehensive view, we then tested sites pushing at least 50% of assets; thinning the number of domains down to 77.

Difference H1/H2Resource Duration (%)Fully Loaded (%)
MIN-66.66666667-55.1006933
MAX4800115.8988257
AVG95.397177846.656787423

As we can see, some sites experienced savings of as high as 115% with fully loaded metrics, although the average sits at around 6.65%.

Testing Header Compression

To further our understanding, we wanted to understand what savings were enjoyed with header compression - the mechanism that compresses IP headers in a packet before the packet is transmitted.

Header sizeHTTP/1.1HTTP/2% Savings
MIN886535.38461538
MAX65248651370.170410059
AVG1093.8200681044.6925544.702581004

As we can see, switching protocol provides, on average, just 4.70% of data transfer savings.

Conclusion

With fully loaded metrics, we can conclude from our data that on average, HTTP/2 pages load 6.1% faster.

When we ask the question of whether those benefits are passed directly to the user, however, it is important to consider that there are other factors that affect how quickly the page loads, such as Internet speed and computer power.

From our data, we can conclude that enabling HTTP/2 on websites could decrease load time by up to 35% by compressing headers. With some modifications of the web server to enable HTTP/2 push features, we could see page speeds increased by up to 100%, but this is not without caveats.

For instance, HTTP/2 creates only one connection to a server, and all resources must be pushed through this single connection. Although this provides benefit in terms of speed, if the connection drops, loading will cease until a connection is re-established.

Websites in countries with poor Internet coverage must, therefore, consider whether enabling HTTP/2 is a worthwhile endeavour.

Get the Newsletter

Stay in the loop with our monthly Newsletter to get the complete round up of market shifts and how to stay ahead.

Let us help you solve your digital problems

We help leading organisations to optimise their digital presence the right way, by tailoring software to integrate business and digital processes, so the humans can focus on strategy, while the machines do the heavy lifting.