The Netflix test is an application-specific test, supporting the streaming of binary data from Netflix's servers using the same CDN selection logic as their real client uses. The test has been developed with direct cooperation with Netflix.

The test begins by calling a Netflix hosted web API. This API examines the client's source IP address and uses the existing proprietary internal Netflix logic to determine which Netflix server this user's IP address would normally be served content from. This logic will consider the ISP and geographic location of the requesting IP address. Where the ISP participates in Netflix's Open Connect programme, it is likely that one of these servers will be used. The API will return to the client a HTTP 302 redirect to a 25MB binary file hosted on the applicable content server.

The test will then establish an HTTP connection to the returned server and attempt to fetch the 25MB binary file. This runs for a fixed 20 seconds of real-time. HTTP pipelining is used to request multiple copies of the 25MB binary, ensuring that if the payload is exhausted before the 20 seconds are complete, we can continue receiving more data. The client downloads data at full rate throughout; there is no client-side throttling taking place.

It's important to note that this 25MB binary content does not contain video or audio; it is just random binary data. However, with knowledge of the bitrates that Netflix streams content at, we can treat the binary as if it were video/audio content operating at a fixed rate. This allows us to determine the amount of data consumed for each frame of video (at a set bitrate) and the duration that it represents. Using this, we then can infer when a stall occurred (by examining when our simulated video stream has fallen behind real-time). The test currently simulates videos at bitrates of 235Kbps, 375Kbps, 560Kbps, 750Kbps, 1050Kbps, 1750Kbps, 2350Kbps, 3000Kbps, 4500Kbps, 6000Kbps and 15600Kbps.

This approach also allows us to derive the 'bitrate reliably streamed', using the same methodology as the YouTube test. A small difference here is that we do not need to restart the download at a lower bitrate if a stall is encountered; because the incoming stream of binary data is decoded at a simulated bitrate, we can simply recompute the playback characteristics of the same network stream at a different bitrate entirely on the client side. This simply means that the test uses a predictable amount of bandwidth, even in cases where stalls occur.

The key outputs from this metric are:

• The bitrate reliably streamed
• The startup delay (the time taken to download two seconds of video)
• The TCP connection time
• The number of stalls and their duration
• The downstream throughput achieved

The BBC iPlayer test is an application-specific test, supporting the streaming of video and audio content from iPlayer using their protocols and codecs.

The test begins by fetching a list of the most popular videos from an iPlayer XML API. The most popular video is chosen for playback, on the basis that this is most representative of what users will be watching at the time. Moreover, if there are iPlayer caches present in the ISP's network then this content is more likely to be cached there.

The XML manifest for this video is then fetched from the BBC's web servers. This contains paths to all of the different bitrates that this video is encoded at, and the different CDNs that serve them. At the time of writing BBC iPlayer uses multiple CDNs to serve content. By having the probe directly fetch the XML manifest we can ensure that any decisions the BBC make (e.g. “ISP X should always be served by CDN Y”) are followed by our test. The test parses the XML manifest file, honouring the priority assigned to each CDN and building an ordered list of available bitrates.

At this point the test can begin to fetch video content from the content server. This is currently achieved over RTMP, mimicking the iPlayer web browser client. The test parses video frames as it goes, capturing the timestamp contained within each frame. After each frame we sample how much real-time has elapsed versus video time. If the video timestamp is greater than the amount of real-time elapsed at a sample period, then an underrun has not occurred. Otherwise, one has occurred.

As with the YouTube and Netflix tests, the client is configured to start testing at the highest supportable bitrate and then step down if and when stalls occur. This allows us to identify the 'bitrate reliably streamed'.

The key outputs from this metric are:

• The bitrate reliably streamed
• The startup delay (the time taken to download two seconds of video)
• The TCP connection time
• The number of stalls and their duration (this is only applicable if the test is not running in the 'bitrate reliably streamed' mode)
• The downstream throughput achieved

The Hulu test is another application specific test that makes use of a third party's live infrastructure, ensuring that the test is mimicking the real behaviour of applications. The test streams a reference video from Hulu's CDN and measures key metrics related to its performance. SamKnows has partnered with Hulu to facilitate these measurements; Hulu has placed a reference video on its infrastructure and provided SamKnows with the ability to stream this video from any source.

At the time of writing, Hulu makes use of two CDNs – Level3 and Edgecast. The test uses these same CDNs and will randomly choose between one of these two CDNs at startup. The test makes use of the same DNS names that the live Hulu service uses, ensuring that any CDN redirection logic put in place by the CDN operator, Hulu, or the ISP, is followed as it would be by a real client.

Hulu has placed a reference video on their two CDNs, transcoded at the same bitrates that Hulu clients typically use. This video is encoded using H264 at the following bitrates: 64kbps, 200kbps, 400kbps, 650kbps, 1Mbps, 1.5Mbps, 2Mbps and 3.2Mbps. The URL of the video encoded at each bitrate, on each CDN, is hard-coded into the Hulu test client.

At startup, the test randomly chooses the CDN to use, and begins to fetch video content using the list of URLs stored in the client. The test parses video frames as it goes, capturing the timestamp contained within each frame. After each frame we sample how much real-time has elapsed versus video time. If video time > real-time at a sample period, then an underrun has not occurred. Otherwise, one has occurred.

As with the other video tests, the client is configured to start testing at the highest supportable bitrate and then step down if and when stalls occur. This allows us to identify the 'bitrate reliably streamed'. Optionally the test may be instructed to continue at a fixed bitrate and simply record the number and duration of underrun events.

The key outputs from this metric are:

• The bitrate reliably streamed
• The startup delay (the time taken to download two seconds of video)
• The TCP connection time
• The number of stalls and their duration (this is only applicable if the test is not running in the 'bitrate reliably streamed' mode)
• The downstream throughput achieved

Shahid is a video on demand service with the biggest streaming library of Arabic content. In January 2020 it also partnered with Disney and Fox to provide more than 3000 hours of content, including Star Wars, Marvel and Disney productions.

This test measures the streaming performance of this service. At the time of writing, Shahid uses Akamai's CDN to deliver its content, and 1080p is the highest available video resolution.

As with our other video streaming measurements, this test finds a popular piece of publicly available content on Shahid's service, and then begins to stream it at the highest available bitrate. The test parses video frames as it goes, capturing the timestamp contained within each frame. After each frame we sample how much real-time has elapsed versus video time. If video time > real-time at a sample period, then an underrun has not occurred. Otherwise, one has occurred.

The client is configured to start testing at the highest supportable bitrate and then step down if and when stalls occur. This allows us to identify the 'bitrate reliably streamed'. Optionally the test may be instructed to continue at a fixed bitrate and simply record the number and duration of underrun events.

The key outputs from this metric are:

• The bitrate reliably streamed
• The startup delay (the time taken to download two seconds of video)
• The TCP connection time
• The number of stalls and their duration (this is only applicable if the test is not running in the 'bitrate reliably streamed' mode)
• The downstream throughput achieved

Video streaming services have largely standardised around HLS and DASH as the two most common streaming methods. These are used by major streaming providers, and also by ISPs themselves offering their own video streaming services.

Our HLS/DASH test supports measurement of any standards-compliant HLS/DASH stream. This includes both on-demand and live streaming content. The only input to the test is the path to a manifest file that defines the video stream and its available encodings.

As with our other video streaming tests, the on-demand side of the measurement can operate in a fixed-bitrate mode (whereby it identifies how frequently stalls occur and for how long), or an adaptive mode (whereby it seeks to find the bitrate that can be reliably streamed without stalls).

The HLS/DASH test also supports reporting VMAF scores. VMAF is a perceptual scoring mechanism developed by Netflix. It works by calculating reference scores against the different encodings of the source video, under optimal conditions, and then comparing the received video stream against these.

This test reports the following key metrics:

• The bitrate reliably streamed
• The startup delay (the time taken to download two seconds of video)
• The TCP connection time
• The number of stalls and their duration (this is only applicable if the test is not running in the 'bitrate reliably streamed' mode)
• The downstream throughput achieved
• The VMAF score