The Internet…Through Phone Power colored glasses

Most of you are familiar with our speedtest server, and for those of you that arent, go have a look. For those who have only just discovered this resource we perform a speed test which is in many ways similar to many other speedtests, but with some QOS measurements as well, however much more important than that, we also perform a VoIP test to measure your connections ability to sustain voice connections. When we originally put this resource up, it was intended to be used as a diagnostics tool for our own customers, and never imagined exactly how much use this tool would receive. It turns out, that there just aren’t many tools of this type floating around so thousands and thousands of users, beyond our own customer base have been making regular use of this resource to diagnose connection issues or just become more informed about their own available connectivity. This has actually resulted in a tremendous amount of very interesting data.

First things first, lets quickly go over some of the statistics we are looking at here.

MOS: This means Mean Opinion Score. This is an age-old telephony metric used to measure measure how a human will perceive the quality of the network. The scale runs from 1 to 5, with 1 being the lowest and 5 being the highest. To help you relate here are some common reference points:

• 5: In the same room as the speaker
• 4.2: POTS – The traditional AT&T “You can hear a pin-drop” PSTN service. This is voice encoded into G711 uLaw and transmitted across a problem-free TDM network
• 3.7: G729 – This is a phone call encoded into G729 for low bandwidth and transmitted across a problem-free network
• 3.5: Cell phone – This would be an average cellular phone call
• 2.5-3: Poor cell phone call.

Jitter: This is the variation in timing between arriving audio packets. If an audio packet contains 20ms of audio, the next packet MUST arrive and be ready to be played before the last packet is done being played. Jitter is corrected with a jitter buffer, typically a 100-300ms buffer used to  smooth out the arriving audio packets. Jitter in VoIP is most commonly experienced as audible clicks and pops, which is what happens when there is no audio to play for a few milliseconds while we wait for the next late packet to arrive. Something to note about jitter, it is perfectly OK for ping times to be high as long as packet delivery rates are consistent.

Packet Loss: This is exactly what it sounds like, audio packets that never make it to the other party, or arrive late. Unlike in a TCP stream where late packets are re-assembled into the final product, and missing packets are re-sent, once the next sound byte has played to a listening party, you cannot go back and re-play late arriving audio packets, you just discard them. Packet loss is caused by the traffic taking different network paths to the destination or by congestion, and most commonly manifests itself as one party sounding garbled or robotic as the jitter buffer slams mis-matching audio packets on top of one another.

Mbps: This is exactly what it sounds like. The raw measured speed of a connection in megabits per second. This metric is what most ISP’s will advertize in their literature, and really only pertains to TCP sessions. It is worth noting that its still a decent metric of the overall connection capacity.

Data Flow QoS is a measure of how smoothly data packets are moving. If a connection is un congested or unregulated then every packet should flow at a rate that matches the maximum capacity of the slowest part of the connection’s route. If regulation of congestion causes this pattern to change then the data flow QoS should drop. This metric is often linked to Jitter and Packet loss.

So now that we are all on the same page, lets take a look at what we have found.Keep in mind that ISP’s have been grouped by the names their network blocks were registered with ARIN, so this explains why there may be some unusual names here, and if you want to know where you land on this chart, you can lookup your IP at ARIN or mosey over to this easy whois tool that does most of the work for you, and find the OrgName that owns your IP address.

First lets take a look at how the top ISP’s we have sample data from line up with regard to MOS scoring.

There are 3 MOS scores plotted here:

• The Average MOS (the average we have seen across the entire sample range)
• The Mean MOS score (the score that the ISP produces most often)
• The Maximum MOS score (the best value we have seen from the ISP)

Now I know you statisticians out there will immediately mention I am missing the minimum and standard deviance, however the minimum is always going to be 1 since its guaranteed that at least one person ran a test on a LAN with issues so it was statistically insignificant, and the standard deviance value while it was easily calculable turned out to be pretty useless for producing a tangible feel for how VoIP traffic fared on that network, so to avoid confusion digesting some of this data we just omitted it.

What does this tell us? Well we can clearly see several things:

1. Nearly every ISP shows MOS scores that will produce a VoIP call quality that is virtually indistinguishable from POTS.
2. There is a clear divide with the wireline networks’s on the left of the graph, and the satellite providers on the far right. This reinforces what most of us knew which is satellite and indirect wireless connections are less capable of producing usable VoIP quality.
3. Geography seems to make much less difference in the MOS score than the network access method. DSL carriers in Europe, Australia and Asia scored consistently better than domestic wireless carriers.

Next lets see how all these carrier stack up with Jitter and Packet Loss.

So what does this tell us? Well first of all you can see that it again is perfectly clear that geography makes much less difference than the last mile technology in use by the network as nearly every network charted clearly illustrates the capacity to produce a jitter-free communication channel. We have also included the average TCP Pause which is indicative of round trip time in this graph to see where it falls in relation to the upstream and downstream measurements taken over a UDP stream.

When looking at the packet loss graph we again see that most show an average of under 1% packet loss both up and down. This is completely imperceptible, again showing absolutely no preference for geography.

Now lets take a quick look at the bandwidth we have seen from these carriers. First things first this is bandwidth as expressed in application speed, which means TCP sessions, i.e. horsepower to the wheels, as you will use it.

So again lets take a serious look at this and see what this tells us.

1. Bandwidth definitely doesn’t equal quality. Some of the carriers that showed the absolute highest MOS scoring and best jitter and packet loss stats land in the mid-low range from a bandwidth perspective,
2. BUT that doesn’t matter so much for voice. Remember, 1 uncompressed VoIP call (G711uLaw) only uses 84kbps of bandwidth (both up AND down). This means that even the most meager carrier here can support 2-3 uncompressed calls and if you add in compression with G729, you can do 4 times as many concurrent calls. Obviously this is assuming there is absolutely no other network traffic. Without getting too technical about buffer bloat and network saturation issues a safe bet when scoping out how many calls you can support across your link is 50% of your measured capacity can be available for voice consistently without causing quality issues. Going over 50% on a non-business grade line is a recipie for issues.
3. The state of broadband in North America is still nothing to be terribly proud about, given that the majority of the carriers still provide an upload under 1mbps and an download/upload speed ratio of often 4/1 which, while it certainly doesn’t hurt anything for your standard web/youtube/netflix/facebook internet experience, will certainly be a barrier as more and more real-time applications such as HD voice and video chat services start gaining popularity.
4. The major mobile providers now supply broadband connectivity that is easily equal to or better than most DSL or lower-end cable from a bandwidth perspective.

Speaking of mobile providers. Lets isolate them for a second since I know Voice over 3G/WiFi is on the forefront of everyones mind. In the following graphs the major US wireless data players are outlined, though their network names might confuse you:

• T-Mobile USA is self explanatory.
• Service Provider Corporation represents AT&T Wireless data (this is how their ARIN entries are presented)
• Clearwire US LLC represents obviously Clearwire as well as the Sprint 4G network
• Cellco Partnership DBA Verizon LLC is Verizon Wireless
• Sprint Nextel Corporation represnets Sprints 3G data network

Now this data was collected not off of mobile phones, since our speedtest presently doesnt work on mobile platforms but instead on laptops and PC’s using mobile hotspots (or cellphones in hotspot mode) or USB dongles, so if it doesn’t represent your carriers new flashy LTE network, this may be why.

This shows clearly that AT&T wireless, T-Mobile and Clearwire all produce average results that can produce VoIP call quality that is as good as a landline, while nealry all of them can produce VoIP call quality that is as good as a cell phone, and very usable, with the age of the respective carrier networks definitely being evident.

Now with Jitter the story shifts a little bit, with clearwire (the only player not running a voice network) showing a subtle performance boost, however notice that average jitter across the board never goes over 125ms which is still well within what most jitter buffers can accommodate for.

Now we take a look at the other side of the jitter equation, packet loss. The scale is in percentage of total packets lost, so AT&T clearly runs away here scoring an average of less than 0.2% lost, with Clearwire showing the next best performance, then T-Mobile close behind with Sprint and then Verizon. What is interesting here is the packet loss numbers from Verizon are nearly the inverse of their jitter numbers, which again levels the playing field pretty well.

Finally the good ol bandwidth test. While the download numbers definitely bounce around (and some are pretty high), what is most interesting here is nearly everyone is showing around or over 1mbps of upload speed; well more than enough for a video call over 3G or a voip conference call. Of course that is without getting into the topic of data caps, roaming, etc but ill save that for another post.

So what have we learned from all this? Well firstly geography doesn’t matter nearly as much as the quality of the network. Second bandwidth does not equal quality. Clearly some of the providers with the highest consistent quality scores were middle to low end in the speed department.

This is just the tip of the iceberg here, since we are still only looking at a few months of data, and more interestingly a few months of data, sampled when there isn’t a significant LTE play yet. I can assure you we will re-visit this topic in the future as the landscape is still changing.

Let us know how your ISP stacks up.

P.S.: For any of you out there that are just gaga for statistics, you can download the data in excel  below

Internet_According_To_PhonePower