Technical note on the objectives and operational use of the TCP/IP Test Suite.

The TCP/IP Test Suite is a set of impairments designed to exercise or otherwise stress the code of Internet Protocol stacks. A common problem in all product development is the failure of the developers and testers to systematically try out the paths through their code. Often only the primary path is tested. Yet users, applications, hosts, and other entities will often utilize the product in many new and unexpected ways. Hackers, of course, devote themselves to finding those untested paths through the code that will not only cause a product to crash, hang, reboot, but also exploit security vulnerabilities.

The Maxwell TCP/IP Test Environment provides deep-path testing, systematically exercising paths through the code during the protocol conversation by means of its stateful protocol session technology (U.S. Patent 7,310,316). Maxwell tracks and modifies sessions – protocol discussions between network devices – very precisely, in a controlled, stateful way, in real-time, while the protocol discussions continue. With TCP, for example, Maxwell can introduce random delays on every other acknowledgment to force two devices to invoke their congestion avoidance algorithms and re-compute their round trip time. This takes a session – an ongoing exchange of messages between two devices -- not just a one-way command. Applying impaired network conditions based on criteria, allows you to go more deeply through the code in the test device.

Tests can be initiated from the Graphical User Interface (GUI), shell commands, or scripts written in one's favorite language (support for shell scripts, Perl, Python, and Java are available right out of the box.)

Faults may occur in applications, in the behavior of socket API calls, in the operating system’s stack, and in network device drivers. But because Maxwell does not know anything about the specific implementation of any of these areas, it is impossible for it to make a “pass/fail” judgment on them. For example, if an IP fragmentation impairment causes a TCP/IP stack to leak memory, Maxwell would be unable to detect that since it is unable to see inside the stack (it isn’t even running on the device under test – never mind that resource usage is implementation specific). The best that Maxwell could hope to do is verify protocol correctness as it appears on the wire.

In order to observe the internal behavior and faults of the application, stack, or socket API, the code for that entity must be “instrumented”. That is, just as an electronics engineer would attach an instrument such as an oscilloscope or voltmeter to a circuit to observe its behavior when a test signal is injected, a software engineer must insert code to display or log the execution paths and memory values as the code executes. There is, unfortunately, no way to develop a generic set of utilities for all possible platforms and implementations, which is why Maxwell (or any other product) is not able to render a pass/fail judgment on the internal correctness of the code on the device under test for most of the tests.

When checking the wire-line behavior of the DUT, we recommend using Wireshark (or the older Ethereal product) or another packet analyzer to observe whether the receiving device received the packets and responded correctly. If Wireshark can be made to build and run on your DUT then it can also prove useful to run it there.