Wireless Testing: A Reality Check for Wifi Solutions

Are you going to test drive a Wi-Fi solution? What do you want to measure? You need more than a single speed test to find out whether a router, repeater, or mesh network is up to its task.

If you really want to find out if a router, repeater, or mesh network meets today’s expectations and is ready for tomorrow’s internet, you can’t settle for yesterday’s test strategies.

This applies whether you are going to test drive a Wi-Fi solution for yourself or do a review for others.

In the Beginning Was Maximum Speed

When the first wireless products came out, they did not have a lot of functionality or sophisticated technology, and the diversity of clients and usage patterns was quite limited. To compare the different manufacturers and products against each other, looking at the maximum speed was the usual way.

Since then, both the technology, the usage patterns and the expectations have changed radically, but old test strategies often remain–too often, in our opinion.

Below, we describe:

• A typical speed test – quick, but inadequate for most purposes
• A more thorough combination test – an OK approach to checking coverage or diagnosing a network
• A professional test strategy that includes complex usage scenarios.

If you are not planning on doing any testing of your own, keeping the advice here in mind when reading other people’s reviews can be helpful – to ask yourself what they have actually tested and how thorough they have been.

Smoke Test: Do You Have Wi-Fi at All? Is it fast here and now?

This is still the most common way consumers test their own wireless network, and it shows whether Wi-Fi is working here and now for one client:

1. Get close to the access point and run a speed test. Follow the instructions from the test application – make sure that other programs are not running simultaneously, and so on.
2. Do some more spot checks of speed here and there to see how far away you can get signals.

Speed tests are by far the most popular internet connection test tools, and for wired connections, they are excellent and reliable test tools. For a wireless connection, the speed test is a spot check — it shows how much speed you achieved exactly where you were with the client, exactly when you stood there. Two steps away or three seconds before, the result could have been completely different. Another client would probably also get a different result.

More on speed tests: Speed test: Take wireless speed with a grain of salt.

If you perform more such spot checks, they can be a useful indication, but they should be used along with other tools and tests:

A little more thorough: what does the coverage look like on a heat map?

A more thorough test will also measure coverage, which many people experience problems with:

1. Do everything described in the previous test.
2. In addition, measure coverage by creating a heat map on a client that supports it. We recommend the tool from Ekahau. See also Map your wireless network with a heatmapper.

Remember again that the test also reflects the Wi-Fi capacity of the client you use to run the test. For example, if it’s a new iPad, the coverage will look far more promising than will be the case for older clients on the same network.

These tests provide a good indication of the coverage for the clients it is tested with, but still say little or nothing about how the solution handles more realistic and complex usage scenarios.

If you want to know how the solution works in practice, you need to test even more thoroughly:

Professional Level: Does the Wi-Fi Solution Provide Full Coverage in Practice? Is it ready for the Internet of Things?

Today, most of us have the expectation that wireless should work everywhere, all the time, and it should be stable. It should work just as well in the living room on the ground floor as in the teenager’s room on the second floor, regardless of what other wireless users on the network are doing. The Internet of Things brings new devices that need coverage into homes and garages and offices.

Speed and coverage spot checks are not enough to test whether a Wi-Fi solution meets these expectations, although they can provide reasonable snapshots of wireless coverage. For example, if you’re evaluating wireless equipment for your business or customers, or writing a review of Wi-Fi solutions for a publication, you’re obviously going to go deeper.

When testing Wi-Fi solutions, it is hard to get results that are representative of more than one user on a single client in one place at a time. The result is always a combination of access points, client and physical conditions, including interference.

Below are three advice and three test scenarios that:

• Should provide results that can be compared from test to test.
• Includes the management of complex scenarios that are traditionally difficult for wireless networks, while still being quite common in everyday use.

Advice #1: Set Clear Objectives for Your Wireless Test

Our starting point in this test strategy is that our goal is to have total coverage with stable performance – not the highest possible speed for each individual client, but a network that is ready to deal with the Internet of Things. This means handling a large amount of clients with different standards that require uptime and reliable coverage.

Our criteria for full coverage are:

• Minimal dead zones.
• Seamless roaming within the network – regardless of the number of access points:
  • Can you move around the home/office with an active client with minimal disruption?
  • Do you have a single SSID (network name) and one password?
• Single clients with poor coverage (bad apples) are minimized and managed.
• Multiple clients can stream video or perform other capacity-intensive operations simultaneously without problems.

Advice #2: Choose a Representative Test Environment

Who are you testing for? If it’s just for yourself, you will of course perform the tests in your own home. If you are testing for customers, or for your business, or for readers of a publication — it’s crucial that the test environment is relevant:

• Home environment versus office environment will be different.
• For boligbruk, med mindre du eksplisitt skal teste løsninger for små boenheter, pass på at testarealet er stort nok til at det gir mening å teste funksjonalitet som roaming (forflytting av klient mellom ulike aksesspunkter).
  • A small apartment is a bad starting point for testing sophisticated solutions for full coverage, since the vast majority of access points will cover the majority of the home on their own anyway.

Advice #3: Compare Apples to Apples as Far as Possible

To assess multiple solutions, the conditions should be as similar as possible:

• Create a fixed test scenario – suggestions for scenarios and how they can be tested follow below.
• Have test clients of different types that your reuse–including older clients that don’t support all the latest wireless technology-unless you explicitly test for a setting in which all users have brand new equipment.
• Use the same tool every time-–if you run speed tests, make sure to use the same test, server, and client to be able to compare For heat maps, use the same heatmapper application and client.
• Use the same locations in your home/office for running speed tests and creating heat maps every time. Remember that building materials, physical obstructions and interference affect your Wi-Fi signals. Even the weather and temperature, and whether windows are open or closed or larger furniture being moved can make a difference. Minimize the differences as much as possible.

Read more about what materials and other conditions cause problems for Wi-Fi signals:

• 10 things that interfere with and block Wi-Fi signals
• Which Building Materials Can Block Wi-Fi Signals?

Scenario A: Multiple demanding users at the same time

Whether you are testing for home or office use, a realistic scenario would be that the wireless network must handle multiple wireless clients in different locations at the same time. Families with older children and teenagers are among those who report the most problems with Wi-Fi.

Here’s how to test it:

1. Place clients (notebooks, tablets, mobiles…) in different locations in your home or office. In the home it can be natural to think of bedrooms, basement, or attic, as well as the living room and any home office.
2. Start Nrk web TV or another demanding streaming service on all clients.
3. Tests:
   1. Check that all clients get satisfactory performance; video runs without pausing and buffering.
   2. Run a speed test from the client that is farthest from the closest access point while the others continue streaming video.

Scenario B: A rotten apple – dealing with clients with failing coverage

If you have already taken a heat map, you know where in your home or office you have the worst the coverage. This lets you test how the Wi-Fi solution handles a scenario where one (or more) clients have poor coverage, also known as “Bad Apple”, and whether this affects the performance of the network as a whole. For a more thorough introduction to the problem, see: Bad apple: How a single device with poor coverage can ruin your wireless network.

Here’s how to test it:

1. Place a client at the edge of the coverage area – preferably an older smartphone or other client that doesn’t necessarily support all the latest technology.
2. Repeat the test from scenario A and include the client with poor coverage.
3. Check:
   1. Is video streaming without pausing and buffering on the other clients?
   2. Is video streaming working on the client with poor coverage?
   3. If you have a solution with multiple access points, is the client with the worst coverage connected to the same access point as the others?

Scenario C: Roaming–Client on Walkabout

This test scenario is primarily relevant if you are testing a solution with more than one wireless access point, that is a mesh network (such as eero or Zyxel MPro Mesh) or a combination of routers and repeaters or similar: Go for a walk around the home/office with a client connected to a streaming service.

What you can test then is how the solution handles what is often called a “sticky client”. This means that a wireless client, once it has been connected to an access point, will often maintain this connection for as long as it possibly can, even when the signals are too weak to provide a functional connection. In an effective setup with multiple access points, the solution should ensure that the client is steered towards another access point that provides better signal strength. See also Sticky Clients: When Devices Cling to Bad Wi-Fi Connections.

Here’s how to test it:

1. Launch a web TV or other video streaming service on your tablet, mobile, or laptop.
2. Move between the access points and the outer edges of the home.
3. Check whether:
   1. You have and maintain coverage.
   2. The client connects to other access points as you move. The closest access point will normally provide the best connection, but a smart solution will also consider other factors, such as the number of devices already connected and the performance the client can expect to have if it changes. Therefore, if the performance you experience continues to be good, it is not necessarily wrong when a client does not connect to the closest access point.
   3. Delays/disruptions when the client disconnects are minimal.

Article by Jan Pedro Tumusok, Geir Arne Rimala, and Jorunn Danielsen