Spotlight: The Super Agent That Sam Built

Inside your broadband router, there’s every chance that there’s a little Agent beavering away. It isn’t doing anything malicious or sneaky – in fact, when any of the devices in your home are actively using the internet, it isn’t doing anything at all that might get in the way.

Instead, that software Agent waits patiently until the connection goes quiet. When the coast is clear, it gets to work on its tests. It might be testing the actual speed of the connection, the latency, or the kind of performance you’re getting from popular services such as Netflix, WhatsApp or Skype; it might be testing in-home performance; or it might even be testing the health of the router itself. It runs hundreds of these tests every day, around the clock, without you having to lift a finger – without you even having to use the services it’s testing.

The super Agent was built by SamKnows’ founder Sam Crawford and his team, and it’s sitting in tens of millions of routers around the world. Here, we tell the story of Sam’s Agent: how it started, what it does, and how it’s helping to improve broadband performance around the world.

Thinking outside the Whitebox

SamKnows is best known for helping governments identify ways to improve the internet for countries around the world. It does this by installing Whiteboxes, small bits of hardware that plug into home routers, monitoring the performance of the broadband connection. These Whiteboxes are unobtrusive, but a few years ago, one of the broadband providers that SamKnows works with came up with an idea to make them disappear altogether. “Why don’t we just merge the two together and do a proof of concept of putting your software inside our routers?”

Sam Crawford remembers the conversation clearly: “One of the ISPs we’re working with here in the UK came to us and said: ‘we’re distributing tens of thousands of our existing routers to customers every week and these devices are fundamentally small, embedded Linux devices. Your Whiteboxes are small, embedded Linux devices. Why don’t we just merge the two together and do a proof of concept of putting your software inside our routers?’ And that’s exactly what we did.”

The SamKnows software Agent runs the same set of tests as the Whiteboxes, collecting an almost identical set of data. It does this without any additional hardware, without requiring any user intervention, and without impacting router performance. The Agent can even detect when things are going wrong with the router itself.

Measuring up

The Agent that was first installed in that test deployment in 2016 is a thin shadow of what exists today. Even Sam admits that first Agent was a “disparate collection of bits of software held together with some sticky tape”.

At that time it was focused on collecting speed, latency and packet-loss data, but not a lot else. In addition to the critical performance data mentioned above, the Agent is now also taking a barrage of active measurements from many of the most popular internet services. Not by tracking customers’ usage of those services – your Netflix viewing habits or the photos you upload to Facebook aren’t logged by the SamKnows software. Instead, it generates its own test traffic.

According to Sam, this has several benefits. For one, it isn’t reliant on the customer using or subscribing to those services. “Even if the Agent is installed in my router and I don’t have a Netflix subscription, then our Agent can stream some content from Netflix and measure the performance, even while I’m asleep, not watching Netflix, and all of my computers are switched off,” he explains.

Perhaps more importantly, it doesn’t require any monitoring of service for which the consumer is using their broadband connection. The SamKnows Agent checks that the router isn’t busy at the time it performs its tests, but that’s it. It simply doesn’t care what the user is doing with their connection, because it’s irrelevant; it tests the well-known applications with its own data, not yours.

Other testing services may use deep-packet inspection to see what the customer is using their connection for and then attempt to measure the performance of those services. However, this approach has flaws that extend beyond compromising privacy. The biggest problem with packet inspection is that most traffic to major online services, such as Netflix, is encrypted. They might apply signature matching to work out that a particular traffic flow is a Netflix stream, for example, but they can’t tell what resolution you’re streaming the video at, nor if the video stalled during playback. Privacy is compromised, as is the data collected.

What’s wrong with regular speed tests?

Installing software in routers to test internet performance might seem like overkill. After all, there are plenty of speed tests available on the internet. Why go to the bother of integrating an Agent into the router?

The advantages of the Agent are simply too numerous to list here, but here are a few of them. For starters, an internet speed test won’t collect application-specific data, such as testing the performance of Google Meet, Twitter or WhatsApp. It simply tests how fast a small packet of data can travel from A to B.

When you’re relying on user-initiated speed tests, ‘A’ also becomes highly variable. They might take that speed test using a laptop in the same room as the router, or from their smartphone from a home ofice in the garden, with all the variation when someone else in the house is streaming a 4K movie on the living room TV, completely disrupting the results.

In addition, while the speed tests will only give you a snapshot of performance at the time they decide to take them. “A web-based speed test is a very self-selecting sample. People are only using it when they think they have a problem.” explains Sam.

The SamKnows Agent tests real-world app performance

“People are only using it when they think they have a problem. So, you’ll probably be biased towards slightly worse results.” On the flip side, people often run speed tests when they’ve just had their connection upgraded, which is likely to skew results the other way.

In short, web-based speed tests serve a very specific purpose – but the long-term performance testing of a broadband connection isn’t one of them.

Keeping an eye on the router

There’s another big advantage of SamKnows having its software router Agent inside the router – it can keep an eye on the router itself. This means the Agent can monitor critical performance metrics collected by the router, such as the DSL sync speed or when the internet connection last dropped.

It can also spot problems that might otherwise go undetected. Sam Crawford reveals how SamKnows recently detected a huge dip in the number of measurements the Agent was taking with a particular model of router. As we discussed earlier, the Agent is programmed not to run when the user is making significant use of the internet, but the tests are also disabled when the router’s CPU is under heavy load, so as to avoid any interference with internet traffic.

The SamKnows analysts spotted this drop-off was only happening with this select model of router and reported it to the manufacturer. It investigated and found that a firmware update on the router had contained a bug that put significant extra load on the router’s CPU. Not only would this bug have affected internet performance, it would have seen millions of routers worldwide using 50% more power than they actually needed – which would “add up significantly in terms of power consumption as well,” Sam Crawford says.

Thanks to the tip-off from the Agent, however, the manufacturer was able to issue a patch before any serious broadband performance issues or significant energy costs were generated.

Still, the issue of CPU consumption raises an interesting point. Routers have a long shelf life and don’t tend to be at the cutting-edge of chip technology in the first place. Doesn’t the SamKnows Agent and the barrage of tests it runs place unnecessary strain on the router itself?

The Agent detects problems before they hit consumers

SamKnows doesn’t want its Agent to interfere with people’s internet connections – it’s designed specifically not to do that. So Sam has been working with the router manufacturers, the relatively unknown companies that make the equipment rebadged by the big broadband providers, to ensure the Agent never gets in the way.

“Lots of these routers have dedicated packet processors alongside that main CPU, so you can hit, let’s say, one gigabit per second on your five-year-old or even older router, without going through the main processor,” explains Sam Crawford.

All of the router integration work is performed by SamKnows, so the router manufacturers and broadband providers don’t incur any additional overheads for putting the Agent inside the router. It’s all upside.

Broadband healing

So what future innovations does Sam have up his sleeve for the Agent inside our routers? One thing Sam is very keen to do is to help people identify and address their own connectivity problems.

SamKnows already has a product called RealSpeed, which is a two-step test that measures the performance from your router to the internet, and the performance from your device (laptop, smartphone, and so on) to the internet. Armed with this information, consumers can then see if the problem lies with the internet connection itself or with the device. For example, the laptop might be located too far from the router to achieve maximum throughput.

Sam wants to build on this kind of diagnostic data to help consumers make more informed decisions about their home network. “If you’re seeing poor performance from your device, but good performance from your router, then we might be able to say ‘this is actually the cause, and here’s how you could correct it,’ or maybe we will be able to correct it for you in the future,” says Sam. “In-home diagnostics is something we’ve already done with a couple of devices and we want to broaden that.”

What’s more, Sam is seeing an increasing number of broadband providers become willing to share the data collected by the Agent with their customers, once again helping them to be better informed about what’s going on with their connection. “I don’t think they’re yet showing the customers historical measurement results, but it’s easy to see that this is the direction in which things are heading,” he said.

You might not have realised the SamKnows Agent was inside your router up to this point, but you might get to know it an awful lot better in the years to come.