Maxwell Network Emulator 
White Papers
White Papers
| End-to-End VoIP Product Comparison Testing |
|
Table 1: Voice Source, Cisco-7905-to-Cisco-7905 OverviewThis report summarizes the results of comparison testing between six Voice-over-IP products:
All these phones are real products being marketed today. All were tested under a variety of network conditions which are nearly identical to what they might encounter when operated in the open Internet. Although it is possible to upgrade network infrastructure to provide QoS guarantees, this will not be helpful in most cases. In the general case, "anyone-talking-to-anyone" communications, product performance could be affected by network disturbances. These disturbances include jitter, packet drop, reorder, duplication, and others. The report includes sound clips so you can hear and judge for yourself the voice quality results of representative conditions. We used our Maxwell[tm] (see sidebar) to impose these real-world network conditions. The test methodology is also described. SummaryMaxwell makes it very easy to perform side-by-side product, product-version, regression and -interoperability comparisons[1] under controlled and realistic network conditions. Some VoIP phone manufacturer's documentation recommends that the user's network be designed within certain quality-of-service conditions, such as jitter not to exceed 30ms, so we used those as a starting point in these measurements. We picked 25ms and 30ms. The Pingtels performed well at jitter levels far in excess of these, as you can hear from the sound clips. The sound clips also demonstrate how combinations of network disturbances or impairments affect the phones. Individual impairments may not affect voice quality, however, in combination with other impairments, voice quality is degraded. For example, at an average jitter of 25ms, we found no audible distortion in the Cisco 7960 phones unless we also added reordering. The sound clips are in WAV format, which most desktop computers can play. All are digitized at 8KHz, 16-bit resolution, monophonic. For reference, CD-quality sound is 44.1KHz, 16-bit resolution, stereo. You can listen to the recordings and judge for yourself. Reference recordings are also included for `best case' network conditions. You will need a PC with a sound card. It is best to listen with good headphones, rather than typical desktop PC speakers. Actual sound quality effects are more accurate iif the sound comes from a source near your ears, just as it does with a regular phone. Good headphones also block ambient noise, allowing you to hear just the recording. The following tables show network conditions at which the phones were tested. Only three of the many kinds of possible network impairments were tested. These three are: Jitter: uniformly-distributed random amounts of delay is added to voice data packets. Maxwell keeps track of the arrival time and the exit time of each packet, automatically calculating an average delay which is displayed and updated in real-time by the graphical user interface and also shown in the tables below. Drops: voice data packets are randomly selected to be dropped. Distribution function is uniform. The mean is given; e.g., at 1% drop, 1 out of a hundred packets is dropped. This number applies to both directions, which means that the effective packet loss in each direction is about half that number (e.g., when the drop rate is set to 3%, each direction was showing a drop rate of 1.5%. The table column for drops has been adjusted for this fact Reorder: the order in which packets arrive can be changed. The higher the number, the more reordering takes place. In real networks, packet-reordering can take place occasionally when routes are adjusted, and consistently over tandem links (a commonplace solution when a quick bandwidth-upgrade is needed). You can think of the reorder-number as being the number of extra data links: e.g., reorder 0 -> one data link, reorder 1-> two tandem data links, reorder 2 -> three tandem data links, etc. We did not duplicate, modify or corrupt packets, though Maxwell can do those things too.
The table below provides a sound clip for each phone under each
condition, along with a text notation of the voice quality. Click on the sound
clip to hear for yourself exactly what the indicated network conditions do to
the tested equipment (i.e., "what that sounds like"). These clips were digitized
at 8KHz, 16-bit monophonic. How We TestedOther than the effects introduced by the Maxwell, the network was a quiet internal 10/100 switched LAN, i.e., almost perfect. Two kinds of audio source material were used: a snippet from a local radio station's news reporting[2], and a 1000 Hz test tone. Both were recorded onto CD-R media and played using an RCA portable CD player. The headphone jack was connected via adapter to RJ11 connector on the phone. The CD player's volume control was adjusted so that with no impairments from the Maxwell, the signal was loud, clear and undistorted. For all measurements except the ones to the Cisco 7935 ConfPhone (which has no handset), the receiving phone's handset cord was connected thru an RJ11 adaptor to ministereo plug, and fed directly into a PC sound card. The purpose in doing so was to avoid speaker-to-microphone distortion and background noise pickup. The recording volume control was adjusted for maximum clarity and volume without distortion when the Maxwell was set to no impairments. Since the Cisco 7935 ConfPhone has no handset and no way to directly record the output signal, for these measurements, an AudioTechnica ATR20 cardioid low-impedance microphone was suspended one inch above the Cisco 7935 ConfPhone speaker. These measurements were taken in a separate and quiet (though not anechoic) room, away from our lab's equipment, RF emissions, and fan noises. For each set of tests, a reference recording was made. Listen [3] to the reference recording to hear what "best case" sounds like. For reference recordings, Maxwell was set to 0 ms jitter, 0% drop, no reordering. In other words, Maxwell did not impair any of the traffic. These reference files contain some noise picked up by the sound card and cabling, not introduced by either the Maxwell or its effect on VoIP traffic. It is recognizable as 60-Hz "hum" and also hiss. You hear it in all samples. Network effects on VoIP tends by be heard as gaps and dropouts, or in some cases like the person is gargling or talking underwater. For the test-tone measurements, instead of a steady tone, it sounds more like you're listening to Morse Code.
Table 1: Voice Source, Cisco-7905-to-Cisco-7905
Table 2: Voice Source, Cisco-7905-to-Cisco-7960
Table 3: Voice Source, Cisco-7960-to-Cisco-7935-ConfPhone
Table 4: Voice Source, Cisco-7960-to-Cisco-ATA186
Table 5: 1KHz Test Tone Source, Cisco 7960 to Cisco 7905
Table 6: 1KHz Test Tone Source, Cisco 7960-to-Cisco 7935 -ConfPhone
Tests started from no impairments, then increased drop percentage at one-
percentage-point intervals. At each interval, jitter started at 0 ms (the "set-point")
then increased, and the same with reordering. This order may be meaningful
depending upon how the receiving units compensated for these impairments. Table 7 - Cisco ATA186 POTS-Adapter
Table 8 - Pingtel xpressa PX-1
Table 9 - Pingtel xpressa Software Phones
Appendix A - Notes
|
Terms of Use - Privacy Policy - Trademarks
©2001 - 2009 InterWorking Labs, Inc. ALL RIGHTS RESERVED.
©2001 - 2009 InterWorking Labs, Inc. ALL RIGHTS RESERVED.
For more information, please contact InterWorking Labs.