CD2H gitForager

Juniper/warp17

Name: warp17

Owner: Juniper Networks

Description: The Stateful Traffic Generator for Layer 1 to Layer 7

Created: 2016-05-26 08:47:40.0

Updated: 2018-04-02 18:42:52.0

Pushed: 2018-02-28 17:13:43.0

Homepage: null

Size: 2405

Language: C

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

WARP17, The Stateful Traffic Generator for L1-L7 is a lightweight solution for generating high volumes of session based traffic with very high setup rates. WARP17 currently focuses on L5-L7 application traffic (e.g., HTTP) running on top of TCP as this kind of traffic requires a complete TCP implementation. Nevertheless, WARP17 also supports application traffic running on top of UDP.

Developing network components or services usually requires expensive proprietary solutions for validating the implemented functionalities and scalability or performance requirements. WARP17 is a platform agnostic tool based on DPDK which:

allows extremely fast generation and sustaining of stateful sessions
offers configurable TCP/UDP infrastructure which can be used for generating high connection setup and data rates for application traffic
is Linux based so all the openly available tools can be integrated by the users of WARP17.

The WARP17 TCP/IP implementation runs completely in user-space thus avoiding the additional latency in the kernel stack. From a hardware perspective, WARP17 will be able to run on all the platforms that are supported by DPDK.

Performance benchmarks

Reference platform HW configuration

The configuration of the server on which the WARP17 benchmarks were run is:

Super X10DRX dual socket motherboard
Two Intel® Xeon® Processor E5-2660 v3
128GB RAM, using 16x 8G DDR4 2133Mhz to fill all the memory slots
2 40G Intel® Ethernet Converged Network Adapter XL710-QDA1

NOTE: In order for the best performance to be achieved when running only one instance of WARP17, NICs were installed on different CPU sockets. In our case the two XL710 40G adapters were installed on socket 0 and socket 1.

For all tests we used the following WARP17 configuration (detailed descriptions of the command line arguments can be found in the WARP17 command-line arguments section):

The 40G adapters were connected back to back
34 lcores (hardware threads): -c 0xFF3FCFF3FF
- 16 lcores were reserved for generating/receiving traffic on port 0
- 16 lcores were reserved for generating/receiving traffic on port 1
- 2 lcores are used for management of the test
32GB RAM memory allocated from hugepages: -m 32768

Three types of session setup benchmarks were run, while emulating both the servers and the clients when using 16 lcores for each ethernet port:

TCP sessions with raw (random) application payload
TCP sessions with real HTTP payload
UDP traffic with raw (random) payload

For each type of traffic 3 charts are presented with results collected when running the test with different request/response message sizes. These charts show the client port:

Session setup rate
Packets per second transmitted and received
Ethernet link utilization (percentage of 40Gbps)

It is interesting to see that when emulating real HTTP traffic on top of a few million TCP sessions, WARP17 can easily exhaust the 40Gbps throughput of the link.

NOTE: the script used to obtain the benchmark results is available in the codebase at examples/python/test_2_perf_benchmark.py. The script spawns WARP17 for each of the test configurations we were interested in.

TCP setup and data rates for RAW application traffic

NOTE: In the case when we only want to test the TCP control implementation (i.e., the TCP 3-way handshake and TCP CLOSE sequence), WARP17 achieved the maximum setup rate of 8.5M clients/s and 8.5M servers/s, so a total of 17M TCP sessions are handled every second.

The tests set up 20 million TCP sessions (i.e., 10 million TCP clients and 10 million TCP servers) on which clients continuously send fixed size requests (with random payload) and wait for fixed size responses from the servers.

TCP raw traffic link utilization reaches line rate (40Gbps) as we increase the size of the requests and responses. When line rate is achieved the number of packets that actually make it on the wire decreases (due to the link bandwidth):

TCP raw traffic setup rate is stable at approximately 7M sessions per second (3.5M TCP clients and 3.5M TCP servers per second)

TCP setup and data rates for HTTP application traffic

The tests set up 20 million TCP sessions (i.e., 10 million TCP clients and 10 million TCP servers) on which the clients continuously send HTTP GET requests and wait for the HTTP responses from the servers.

HTTP traffic link utilization reaches line rate (40Gbps) as we increase the size of the requests and responses. When line rate is achieved the number of packets that actually make it on the wire decreases (due to the link bandwidth):

HTTP traffic setup rate is stable at approximately 7M sessions per second (3.5M HTTP clients and 3.5M HTTP servers per second)

UDP setup and data rates for RAW application traffic

The tests continuously send UDP fixed size packets (with random payload) from 10 million clients which are processed on the receing side by 10 million UDP listeners.

UDP packets are generated at approximately 22 Mpps (for small packets) and as we reach the link bandwidth the rate decreases.

Installing and configuring

Prerequisites

Any 64 bit Linux distribution will do, however we have been testing this with Ubuntu Server 14.04 LTS. In addition we have made an OVF virtual machine image available, details can be found in the respective documentation.

Install build essential, python and ncurses

 apt-get install build-essential python ncurses-dev

Install DPDK 16.11.2

Download DPDK 16.11.2

xf dpdk-16.11.2.tar.xz
pdk-stable-16.11.2

Install DPDK:

 config T=x86_64-native-linuxapp-gcc

 make install

Load the igb_uio DPDK module, either as shown below or by running the $RTE_SDK/tools/dpdk-setup.sh script and selecting option [2] Insert IGB UIO module:
- add the following modules to /etc/modules:
```
DK additions


uio
kni
```
- reload all modules:
```
 depmod -a
 modprobe uio
 modprobe igb_uio
 modprobe rte_kni
```
Enable at least 32 1G hugepages and configure them (see section 2.3.2.1 from the DPDK Guide):
- add the following line to /etc/default/grub:
```
_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=32"
```
update grub:
```
```
sudo update-grub
```
```
reboot the machine
```
```
sudo reboot
```
```
Mount hugepages (see section 2.3.2.2 from the DPDK Guide):
- add the mountpoint:
```
 mkdir /mnt/huge_1GB
```
- make the mountpoint permanent by adding to /etc/fstab:
```
v           /mnt/huge_1GB   hugetlbfs pagesize=1GB  0       0
```
  - remount:
```
 mount /mnt/huge_1GB
```
Export the path to the DPDK SDK (where DPDK was installed) into the variable RTE_SDK. For example:
```
rt RTE_SDK=/usr/local/share/dpdk
```
Export the target of the DPDK SDK into the variable RTE_TARGET. For example:
```
rt RTE_TARGET=x86_64-native-linuxapp-gcc
```

Install Google Protocol Buffers

First install the protobuf compilers and python libraries.

 apt-get install protobuf-compiler libprotobuf-dev python-protobuf

If using Ubuntu Server 14.04 LTS then just install:

 apt-get install libprotobuf-c0 libprotobuf-c0-dev libprotobuf8 libprotoc8 protobuf-c-compiler

Otherwise (Ubuntu version >= 15.10):

Install libprotobuf-c, libprotobuf-c-dev from Ubuntu 14.04LTS:

 dpkg -i libprotobuf-c0_0.15-1build1_amd64.deb
 dpkg -i libprotobuf-c0-dev_0.15-1build1_amd64.deb

Install libprotobuf8 from Ubuntu 14.04LTS:

 dpkg -i libprotobuf8_2.5.0-9ubuntu1_amd64.deb

Install libprotoc8 from Ubuntu 14.04LTS:

 dpkg -i libprotoc8_2.5.0-9ubuntu1_amd64.deb

Install protobuf-c-compiler from ubuntu 14.04LTS:

 dpkg -i protobuf-c-compiler_0.15-1build1_amd64.deb

Get WARP17

Get the warp17-<ver>.tgz archive or clone the desired release.

Compile WARP17

xfz warp17-<ver>.tgz
arp17-<ver>

Configure Python virtualenv

 apt-get install python-pip
 pip install virtualenv
ualenv warp17-venv
ce warp17-venv/bin/activate
install -r python/requirements.txt

Once installed, whenever python tests need to run the virtual environment must be activated:

ce warp17-venv/bin/activate

To exit the virtual environment and return to the default python interpretor and libraries:

tivate

Configure DPDK ports

Use the $RTE_SDK/tools/dpdk-setup.sh script (as described in the DPDK Guide). Select which ports to be controlled by the IGB UIO module: option [8] Bind Ethernet/Crypto device to IGB UIO module.

How to run

From the top directory of WARP17:

ild/warp17 <dpdk-command-line-args> -- <warp17-command-line-args>

Running as non-root

After compiling WARP17 change the owner of the binary to root (in order to allow access to /proc/self/pagemap:):

 chown root build/warp17

Set the suid bit on the binary in order to allow the user to keep permissions:

 chmod u+s build/warp17

Command-line arguments

DPDK command-line arguments

-c <hex_mask>: bitmask specifying which physical cores the application will use. Each bit corresponds to a physical core (0-<max_cores>).
-n <chan_no> : number of memory channels to be used.
-m <mem_in_MB>: total memory available to the application (in MB).

Please check section 3.1 of the DPDK App Guide for more info about DPDK command-line arguments.

NOTE: For now WARP17 supports at most 64 cores.

WARP17 command-line arguments

--version: prints version and exit.
--help: prints the help and exit.
--qmap <port>.<hex_mask>: bitmask specifying which physical cores will control the physical port .
--qmap-default max-c: maximize the number of independent cores handling each physical port.
--qmap-default max-q: maximize the number of transmit queues per physical port.
--tcb-pool-sz: configure the size of the TCP control block pool (one TCB is used per TCP connection endpoint). The size of the pool will be given by the argument of this option multiplied by 1024. By default 10M TCBs are allocated.
--ucb-pool-sz: configure the size of the UDP control block pool (one UCB is used per UDP connection endpoint). The size of the pool will be given by the argument of this option multiplied by 1024. By default 10M UCBs are allocated.
--mbuf-pool-sz: configure the size of the packet pool. The size of the pool will be given by the argument of this option multiplied by 1024. By default 768K packets are allocated.
--mbuf-sz: configure the size of a packet fragment (mbuf) in bytes. By default fragments are 2048 bytes.
--mbuf-hdr-pool-sz: configure the size of the packet headers pool. The size of the pool will be given by the argument of this option multiplied by
By default 512K packet headers are allocated.
--mpool-any-sock: configure if memory pools should be created from any available memory if the local socket memory is exhausted. By default this feature is disabled as it might affect performance.
--ring-if-pairs: configure the number of in-memory-ring-based interfaces. NOTE: please check section Using In-Memory-Ring-Based Interfaces for more information.
--kni-ifs: configure the number of kni interfaces. NOTE: please check section Using Kernel Network Interface (KNI) Interfaces for more information.
--cmd-file=<file>: CLI command file to be executed when the application starts

NOTE: Options qmap, qmap-default max-c/max-q, cannot be combined. Only one can be passed at a given time.

NOTE: Users are encouraged to use the “qmap-default max-q” option whenever ethernet ports are on the same socket as the PKT cores as this usually gives the best performance!

NOTE: The lowest two cores will be dedicated to CLI and management processing, and can not be assigned to a physical port for packet processing using the --qmap option!

Example (on a x86 server with 32G RAM for WARP17 and 4 memory channels):

Determine the number of physical cores:
```
cpu | grep "CPU(s)"
s):                12
```
Decide how many cores WARP17 should use. In this example we consider WARP17 uses 8 cores:
- cores 6, 7, 8, 9, 10, 11 for packet processing
- cores 0, 1 for CLI and management
Based on that we determine the bitmask corresponding to the ids of the cores we would like to use. The bit index in the bit mask corresponds to the core id:
```
ask:  0  0  0  0      1   1  1  1     1  1  0  0     0  0  1  1 => 0xFC3
idx: 15 14 13 12     11  10  9  8     7  6  5  4     3  2  1  0
```
This corresponds to the -c command line argument.
Determine the number of memory channels:
```
idecode | grep Channel
Bank Locator: P0_Node0_Channel0_Dimm0
Bank Locator: P0_Node0_Channel1_Dimm0
Bank Locator: P0_Node0_Channel2_Dimm0
Bank Locator: P0_Node0_Channel3_Dimm0  <<<< the system has 4 channels (0-3)
```
The -n command line argument should be usually set to the max number of channels available in the system.

WARP17 should be using 32G of memory in this example so the -m command line argument should be set to 32768.

In order for WARP17 to use the default core to port mapping while maximizing the number of transmit queues the --qmap-default command line argument should be set to max-q.
Optional: the startup commands file can be specified through the --cmd-file command line argument.

For our example this translates into the following command:

ild/warp17 -c FC3 -n 4  -m 32768 -- --qmap-default max-q --tcb-pool-sz 32768 --cmd-file cfg.txt

which will start WARP17 with:

8 cores to be used by the application (-c FC3):
- 2 cores will be used by CLI and MGMT
- 6 cores for processing packets
4 mem channels (-n 4)
32G of available memory (-m 32768)
all 6 PKT cores will process all physical ports (--qmap-default max-q)
allocates 32 million TCBs (--tcb-pool-sz 32768): for the configs in the examples sections we need 20M TCBs, i.e., 10M clients and 10M servers.
will execute the CLI commands in file cfg.txt after starting WARP17

Using In-Memory-Ring-Based Interfaces

WARP17 can also be run when no physical interface is available. This is especially useful when developing new features as it removes the requirement of a specific hardware configuration. It also allows users to quickly try out WARP17 on their own laptop/VM.

In-Memory-Ring-Based Interfaces (let's just call them ring interfaces) are always created in pairs. The two interfaces in a pair act as if they would be physical interfaces connected back to back.

By default the support for ring interfaces is disabled. However the user can easily enable it by compiling WARP17 with the following command:

 all-ring-if

Using the --ring-if-pairs <number> command line argument the user can specify the number of ring interface pairs that WARP17 will create. Updating the previous command line example we end up with:

ild/warp17 -c FC3 -n 4  -m 32768 -- --qmap-default max-q --tcb-pool-sz 32768 --ring-if-pairs 1 --cmd-file cfg.txt

This will start WARP17 and add a pair of ring interfaces connected back to back.

The user can also use custom queue mappings for ring interfaces. The ring interface pairs are always created after physical interfaces. This means that their IDs will be allocated in order after physical IDs. For example:

ild/warp17 -c FC3 -n 4  -m 32768 -w 0000:82:00.0 -- --ring-if-pairs 1

This will start WARP17 with three interfaces (one physical and two ring interfaces). The physical interface (0000:82:00.0) will have ID 0 while the two ring interfaces will have IDs 1 and 2.

NOTE: There's a restriction in place when using ring interfaces: the user must make sure that the same number of TX/RX queues is created through qmaps for both ring interfaces in a pair. Otherwise the command line will be rejected.

Using Kernel Network Interface (KNI) Interfaces

WARP17 can also be run with a virtual interface into the Linux kernel. This is especially useful when developing a new protocol and you want to test it agains a known working server or client. See the HTTP example below.

By default the support for KNI interfaces is disabled. However the user can easily enable it by compiling WARP17 with the following command:

 all-kni-if

Using the --kni-ifs <number> command line argument the user can specify the number of KNI interfaces that WARP17 will create. Updating the previous command line example we end up with:

ild/warp17 -c FC3 -n 4  -m 32768 -- --qmap-default max-q --tcb-pool-sz 32768 --kni-ifs 2 --cmd-file cfg.txt

The user can also use custom queue mappings for KNI interfaces, however they can only be assigned to a single core. The KNI interfaces are always created after the physical and ring interfaces. This means that their IDs will be allocated in order after physical IDs. For example:

ild/warp17 -c FC3 -n 4  -m 32768 -w 0000:82:00.0 -- --ring-if-pairs 1 --kni-ifs 2

This will start WARP17 with five interfaces (one physical, two ring interfaces and two KNI interfaces). The physical interface (0000:82:00.0) will have ID 0, the two ring interfaces will have IDs 1 and 2, and the two KNI interfaces will have IDs 3 and 4.

For the example above the two Kernel interfaces will be named warp3 and warp4, so the naming convention is warp<eth_port>

The following example will show how to use the KNI interface to get some HTTP data from the built in HTTP server trough Linux. We assume no physical ports are configured, if you have them make sure you increase all the referenced ports:

Load the rte_kni DPDK module (if needed), either as shown below or by running the $RTE_SDK/tools/dpdk-setup.sh script and selecting option [4] Insert KNI module:

 modprobe rte_kni

Start WARP17 while blacklisting all physical devices (just for the purpose of this test as otherwise the KNI interface name might differ):

ild/warp17 -c FC3 -n 4  -m 32768 -w 0000:00:00.0 -- --kni-ifs 1

Configure the Linux kernel interface:

 ip link set warp0 up
 ip addr add 172.16.1.1/24 dev warp0

Configure WARP17 as follows:

tests l3_intf port 0 ip 172.16.1.2 mask 255.255.255.0
tests l3_gw port 0 gw 172.16.1.1
tests server tcp port 0 test-case-id 0 src 172.16.1.2 172.16.1.2 sport 80 80
tests server http port 0 test-case-id 0 200-OK resp-size 2000
t tests port 0

Now do a HTTP request using wget:

P17:~]$ wget 172.16.1.2
16-10-25 11:40:43--  http://172.16.1.2/
ecting to 172.16.1.2:80... connected.
 request sent, awaiting response... 200 OK
th: 2000 (2.0K)
ng to: ?index.html?

x.html                         100%[================================================================>]   1.95K  --.-KB/s   in 0s

-10-25 11:40:43 (478 MB/s) - ?index.html? saved [2000/2000]

CLI

Test configuration commands

NOTE: Only IPv4 is supported for now!

Configure Ethernet Port MTU:
```
tests mtu port <eth_port> <mtu-value>
```
Add L3 interfaces: configure an IP interface with the specified ip address and mask. Currently only 10 IP interfaces are supported per port.
```
tests l3_intf port <eth_port> ip <ip> mask <mask>
```
Add L3 default gateway: configure 'gw' as the default gateway for eth_port. There is only one default GW per port.
```
tests l3_gw port <eth_port> gw <gw_ip>
```
Add L3 interfaces with specific VLAN and GW: Configure interfaces (Upto 10) with a specified ip address/mask, vlan-id and gw. Each interface can be in a different subnet and an unique vlan-id and gateway can be configured for each.
```
tests l3_intf port <eth_port> ip <ip> mask <mask> vlan-id <vlan-id> gw <gw>
```
The Grat Arp Req/Reply for the interfaces will be sent using the vlan-id configured.
ARP request will be sent to the GW using the configured vlan-id.
ARP reply packets will use the vlan-id from the ARP req packet.
A per port per vlan-id GW table is maintained. The traffic streams will use this table for next-hop GW lookup based on its vlan-id configurations (vlan-options per test-case-id). A quick lookup is done on the per port per vlan id GW table; if no match is found then the default GW configured for the port will be used.
Currently only 10 IP interfaces are supported per port.
Configure server test cases: configure a server test case with ID test-case-id on eth_port. The underlying L4 traffic can be TCP or UDP. ip_range and port_range define the <ip:port> sockets on which the servers will be listening. By default, the application (L5-L7) traffic will be RAW traffic.
```
tests server tcp|udp port <eth_port> test-case-id <tcid>
             src <ip_range> sport <port_range>
```
Configure client test cases (per port): configure a client test case with ID test-case-id on eth_port. The underlying L4 traffic can be TCP or UDP. The source IP/l4-port and destination IP/l4-port ranges define the <src_ip, src_port:dst_ip, dst_port> TCP/UDP connections that will be established. By default, the application (L5-L7) traffic will be RAW traffic.
```
tests client tcp|udp port <eth_port> test-case-id <tcid>
             src <ip-range> sport <l4-ports>
             dest <ip-range> dport <l4-ports>
```
Configure multicast source test cases (per port): configure a multicast source test case with ID test-case-id on eth_port. The underlying L4 traffic can only be UDP. The source IP/l4-port and destination IP/l4-port ranges define the <src_ip, src_port:dst_ip, dst_port> UDP multicast streams that will be generated. By default, the application (L5-L7) traffic will be RAW traffic. Destination IP ranges must be valid IP Multicast ranges.
```
tests multicast-src udp port <eth_port> test-case-id <tcid>
                    src <ip-range> sport <l4-ports>
                    dest <ip-mcast-range> dport <l4-ports>
```
Configure test profile timeouts: each test has a specific timeout profile which is defined by the initial delay after which client connections are initiated, how long a connection should live and how long a connection should stay down (after closing) before the client reconnects.
- initial_delay: amount of time (in seconds) the clients defined in the test should wait before initiating a connection. infinite is allowed but doesn't really make sense for the initial delay as it would cause the clients to never initiate a connection.
```
tests timeouts port <eth_port> test-case-id <tcid> init <timeout>|infinite
```
- conn_uptime: amount of time (in seconds) the clients defined in the test should keep the connection up (and send application traffic) before initiating a close. infinite allows the clients to stay up forever.
```
tests timeouts port <eth_port> test-case-id <tcid> uptime <timeout>|infinite
```
- conn_downtime: amount of time (in seconds) the clients defined in the test should keep the connection down after a closebefore initiating a reconnect. infinite allows the clients to stay down forever.
```
tests timeouts port <eth_port> test-case-id <tcid> downtime <timeout>|infinite
```
Configure test profile rates: each test has a specific rate limiting profile which is defined by the connection open, close and send rate.
- setup rate: number of connections that the clients in the test are allowed to initiate per second. infinite removes any rate limiting for initiating sessions (i.e., WARP17 will try to do it as fast as possible).
```
tests rate port <eth_port> test-case-id <tcid> open <rate>|infinite
```
- close rate: number of connections that the clients in the test are allowed to close per second. infinite removes any rate limiting for closing sessions (i.e., WARP17 will try to do it as fast as possible).
```
tests rate port <eth_port> test-case-id <tcid> close <rate>|infinite
```
- data rate: number of connections that the clients in the test are allowed to send traffic on per second. infinite removes any rate limiting for sending traffic (i.e., WARP17 will try to do it as fast as possible).
```
tests rate port <eth_port> test-case-id <tcid> send <rate>|infinite
```
Configure test criteria: different criteria can be configured for each test case. The criteria will be used for declaring a test as PASSED or FAILED.
- run-time: declare the test case with ID tcid as PASSED after value seconds.
```
tests criteria port <eth_port> test-case-id <tcid> run-time <count>
```
- servers-up: declare the test case with ID tcid as PASSED when count servers are UP (listening for incoming connections).
```
tests criteria port <eth_port> test-case-id <tcid> servers-up <count>
```
- clients-up: declare the test case with ID tcid as PASSED when count clients are UP (ready to initiate a connection).
```
tests criteria port <eth_port> test-case-id <tcid> clients-up <count>
```
- clients-established: declare the test case with ID tcid as PASSED when count clients have established a connection.
```
tests criteria port <eth_port> test-case-id <tcid> clients-estab <count>
```
- data-MB: declare the test case with ID tcid as PASSED when count MB of data have been sent. NOTE: NOT supported yet!
```
tests criteria port <eth_port> test-case-id <tcid> data-MB <count>
```
Configure tests as asynchronous: if multiple test cases are defined on the same eth_port, by default, they will be executed in sequence (when a test case ends the next one is started). To change the behaviour the user can mark a test case as async forcing the test engine to advance to the next configured test case without waiting for the current one to finish.
```
tests async port <eth_port> test-case-id <tcid>
```
Delete test cases: delete a configured test case with ID tcid from port eth_port.

NOTE: if a test case is already running it has to be stopped before it can be deleted!
```
tests port <eth_port> test-case-id <tcid>
```
Start tests: start all the test cases configured on eth_port. Test cases will be started in sequence (after the previous test case ended) except for the ones that are marked as async.
```
t tests port <eth_port>
```
Stop tests: stop all the test cases currently running on eth_port.
```
 tests port <eth_port>
```
Customize TCP stack settings: customize the behavior of the TCP stack running on test case with ID tcid on port eth_port. The following settings are customizable:
- win-size: the size of the TCP send window.
```
tests tcp-options port <eth_port> test-case-id <tcid> win-size <size>
```
- syn-retry: number of times to retry sending SYN packets before aborting the connection.
```
tests tcp-options port <eth_port> test-case-id <tcid> syn-retry <cnt>
```
- syn-ack-retry: number of times to retry sending SYN + ACK packets before aborting the connection.
```
tests tcp-options port <eth_port> test-case-id <tcid> syn-ack-retry <cnt>
```
- data-retry: number of times to retry sending data packets before aborting the connection.
```
tests tcp-options port <eth_port> test-case-id <tcid> data-retry <cnt>
```
- retry: number of times to retry sending other control packets before aborting the connection.
```
tests tcp-options port <eth_port> test-case-id <tcid> retry <cnt>
```
- rto: retransmission timeout (in ms) to be used before retransmitting a packet.
```
tests tcp-options port <eth_port> test-case-id <tcid> rto <rto_ms>
```
- fin-to: FIN timeout (in ms) in order to avoid staying in state FIN-WAIT-II forever.
```
tests tcp-options port <eth_port> test-case-id <tcid> fin-to <fin_to_ms>
```
- twait-to: TIME-WAIT timeout (in ms) to wait before cleaning up the connection.
```
tests tcp-options port <eth_port> test-case-id <tcid> twait-to <twait_to_ms>
```
- orphan-to: ORPHAN timeout (in ms) in order to avoid staying in state FIN-WAIT-I forever.
```
tests tcp-options port <eth_port> test-case-id <tcid> orphan-to <orphan_to_us>
```
- twait_skip: boolean to decide if state TIME-WAIT should be skipped or not.
```
tests tcp-options port <eth_port> test-case-id <tcid> twait-skip <1|0>
```
- ack-delay: boolean to decide if ACK should be delayed (according to RFC1122, section 4.2.3.2) or not. By default ACK delay will be disabled.
```
tests tcp-options port <eth_port> test-case-id <tcid> ack-delay <1|0>
```
Customize IPv4 stack settings: customize the behavior of the IPv4 layer running on test case with ID tcid on port eth_port. The following settings are customizable:
- tos: the TOS field of the IPv4 header
```
tests ipv4-options port <eth_port> test-case-id <tcid> tos <tos-value>
```
- dscp and ecn: the DSCP/ECN field of the IPv4 header
```
tests ipv4-options port <eth_port> test-case-id <tcid> dscp <dscp-name> ecn <ecn-name>
```
- tx-timestamp and rx-timestamp: allow warp17 to write/read timestamp option in the IPv4 header (Warp17 will store timestamps according to RFC791, section 3.1). When RX timestamping is enabled, latency statistics will also be computed.
  
  NOTE: This might incur a small performance penalty.
```
tests ipv4-options port <eth_port> test-case-id <tcid> tx-timestamp|rx-timestamp <0|1>"
```
Latency: latency computation can be enabled on top of all the application types using IPv4 options or RAW timestamping. The latency config consists of the following optional fields:
- max latency threshold: all incoming packets with a measured latency higher than the configured max will be counted as threshold violations.
- max-avg latency threshold: every time the average measured latency is over the configured max-avg a new threshold violation will be counted.
- samples count: the number of recent samples used for computing recent statistics. Global statistics are computed per test case using all the received samples (not only the most recent ones).
```
latency port <eth_port> test-case-id <tcid> max <value> max-avg <value> samples <value>
```
NOTE: Latency configs make sense only if RX timestamping is enabled for the same test case.

Customize Vlan Settings: By default VLAN fields will not be set. VLANs can be enabled on top of all the application types using vlan-options configuration cli. Please also take a look at Add L3 (sub)interfaces with specific VLAN and GW section as well. The vlan-id of the testcases should match with any of the interface level l3_intf vlan-id configurations.

The vlan-options config consists of the following fields.

vlan-id : VLAN id to be used; Value from 1-4094 can be provided.

vlan-pri : VLAN priority field to be set; Value from 0-7 can be provided.

set tests vlan-options port <eth_port> test-case-id <tcid> vlan-id <1-4094> vlan-pri <0-7>"

NOTE-1: `show tests vlan-options port <eth_port> test-case-id <tcid>` can be used to check the current vlan configuration.

NOTE-1: `show tests config port <eth_port>` can be used to check the current vlan configuration on the l3_intf as well as testcases.

__Example-1: To give vlan id of 100 to all the configured sessions on port 0 test-case 0

      please use the configuration as shown below.__

 tests l3_intf port 0 ip 19.1.1.2 mask 255.255.255.0 vlan-id 100 gw 19.1.1.1
 tests client udp port 0 test-case-id 0 src 2.1.1.1 2.1.1.1 sport 1026 1026 dest 3.1.1.1 3.1.1.1 dport 1026 1026
 tests client raw port 0 test-case-id 0 data-req-plen 64 data-resp-plen 0
 tests vlan-options port 0 test-case-id 0 vlan-id 100 vlan-pri 7

__Example-2: To send all the configured sessions on port 0 test-case 0

      without any VLANs using default-gw, please use the configuration as shown below.__

 tests l3_intf port 0 ip 12.1.1.2 mask 255.255.255.0
 tests l3_gw port 0 gw 12.1.1.1
 tests client udp port 0 test-case-id 0 src 2.1.1.1 2.1.1.1 sport 1026 1026 dest 3.1.1.1 3.1.1.1 dport 1026 1026
 tests client raw port 0 test-case-id 0 data-req-plen 64 data-resp-plen 0

Example-3: Alternate way of doing Example-2.

 tests l3_intf port 0 ip 12.1.1.2 mask 255.255.255.0 vlan-id 0 gw 12.1.1.1
 tests client udp port 0 test-case-id 0 src 2.1.1.1 2.1.1.1 sport 1026 1026 dest 3.1.1.1 3.1.1.1 dport 1026 1026
 tests client raw port 0 test-case-id 0 data-req-plen 64 data-resp-plen 0

__Example-4: To make each test-case-id 0 to use GW without a VLAN (12.1.1.1), test-case-id 1 to

      use GW with vlan-id 1000 (13.1.1.1), and test-case-id 1 to use GW with
      vlan-id 1001 (14.1.1.1)__

 tests l3_intf port 0 ip 12.1.1.2 mask 255.255.255.0
 tests l3_gw port 0 gw 12.1.1.1
 tests l3_intf port 0 ip 13.1.1.2 mask 255.255.255.0 vlan-id 1000 gw 13.1.1.1
 tests l3_intf port 0 ip 14.1.1.2 mask 255.255.255.0 vlan-id 1001 gw 14.1.1.1
 tests client udp port 0 test-case-id 0 src 2.1.1.1 2.1.1.1 sport 1026 1026 dest 3.1.1.1 3.1.1.1 dport 1026 1026
 tests client raw port 0 test-case-id 0 data-req-plen 64 data-resp-plen 0
 tests client udp port 0 test-case-id 1 src 4.1.1.1 4.1.1.1 sport 1026 1026 dest 5.1.1.1 5.1.1.1 dport 1026 1026
 tests client raw port 0 test-case-id 1 data-req-plen 64 data-resp-plen 0
 tests vlan-options port 0 test-case-id 1 vlan-id 1000 vlan-pri 7
 tests client udp port 0 test-case-id 2 src 7.1.1.1 7.1.1.1 sport 1026 1026 dest 8.1.1.1 8.1.1.1 dport 1026 1026
 tests client raw port 0 test-case-id 2 data-req-plen 64 data-resp-plen 0
 tests vlan-options port 0 test-case-id 2 vlan-id 1001 vlan-pri 7

Application configuration and statistics commands

Currently only RAW TCP (L5-L7 payload is random) and a sub-set of HTTP 1.1 (GET/HEAD and 200 OK/404 NOT FOUND) traffic is supported.

Before configuring the application behavior the user must have previously defined the client or server test cases.

HTTP 1.1 application traffic: the HTTP 1.1 application allows the user to simulate different types of HTTP requests (for clients) and responses (for servers):
- HTTP 1.1 client configuration: GET/HEAD requests are supported. A req-size must also be specified (0 is also valid) in order to define the size of the body of the HTTP request.
```
tests client http port <eth_port> test-case-id <tcid> GET|HEAD <host-name> <obj-name> req-size <req-size>
```
- HTTP 1.1 request fields: Any user specified fields can be added to the HTTP request. The only constraint is that Content-Length cannot be explicitly set by the user. Use a set command for each of the HTTP fields that need to be set:
```
tests client http port <eth_port> test-case-id <tcid> http-field <plain text HTTP field>
```
- HTTP 1.1 server configuration: 200 OK/404 NOT FOUND responses are supported. A resp-size must also be specified (0 is also valid) in order to define the size of the body of the HTTP response.
```
tests server http port <eth_port> test-case-id <tcid> 200-OK|404-NOT-FOUND resp-size <resp-size>
```
- HTTP 1.1 response fields: Any user specified fields can be added to the HTTP response. The only constraint is that Content-Length cannot be explicitly set by the user. Use a set command for each of the HTTP fields that need to be set:
```
tests server http port <eth_port> test-case-id <tcid> http-field <plain text HTTP field>
```
- HTTP 1.1 global stats: display (detailed) statistics for the ethernet ports currently in use (e.g., allocation errors/parse errors). If detailed stats are requested then the information is displayed per port + lcore.
```
 http statistics [details]
```
RAW application traffic: the RAW application emulates request and response traffic. The client sends a request packet of a fixed configured size and waits for a fixed size response packet from the server. The user should configure the request/response size for both client and server test cases.

NOTE: the user has to make sure that the request/response sizes match between clients and servers!
```
tests client raw port <eth_port> test-case-id <tcid>data-req-plen <len> data-resp-plen <len> [rx-timestamp] [tx-timestamp]
```
```
tests server raw port <eth_port> test-case-id <tcid>data-req-plen <len> data-resp-plen <len> [rx-timestamp] [tx-timestamp]
```
Both CLI commands support additional RX/TX timestamping options. If rx-timestamp is set, the Warp17 traffic engine will timestamp packets at ingress and the RAW application will compute latency statistics when incoming packets have TX timestamp information embedded in their payload. If tx-timestamp is set RAW application clients will embed TX timestamps in the first 16 bytes of the application payload. The RX/TX timestamps are both computed early in the packet loop in order to be as precise as possible when measuring latency.
IMIX application traffic: multiple application configurations can be grouped in IMIX groups (i.e., Internet Mix traffic). Each application within a group has an associated weight. When an IMIX test starts the weights of each application are taken into account for determining how often the application will be represented in the final traffic profile. Test case rates (i.e., open, send, close) are also enforced according to the weights of each application.
```
tests imix app-index <app-index> <application-configuration>
```
The user can associate an application configuration to an IMIX application index. Then the user should associate a weight with the given IMIX application index:
```
tests imix app-index <app-index> weight <weight>
```
Finally, multiple IMIX application indices can be used within an IMIX group:
```
tests imix-id <imix-group-id> app <list-of-imix-app-indices>
```
The IMIX group can be used as an application configuration for a traffic test case:
```
tests imix port <eth-port> test-case-id <test-case-id> imix-id <imix-group-id>
```

Displaying test information

Current test configuration: the current test configuration (including per port L3 interfaces and default gateway) will be displayed for a given ethernet port.
```
 tests config port <eth_port>
```
Current test state: the current test state (including per test case quick statistics) will be displayed for a given ethernet port.
```
 tests state port <eth_port>
```
Detailed test statistics: the detailed test staistics will be displayed for a given ethernet port and test-case.
```
 tests stats port <eth_port> test-case-id <tcid>
```

Statistics and operational information

Different types of statistics can be dumped from the CLI. Currently all these stats are not directly linked to any test case ID but they are aggregate per ethernet port.

Port information and statistics
- Port information: display general port information.
```
 port info
```
- Port-core mappings: display the mappings between ethernet port RX/TX queues and lcore IDs. The socket IDs of the ports and lcores are also displayed.
  
  NOTE: Having lcores handling ports that have their PCI bus on a different socket than the lcore will affect performance!
```
 port map
```
- Port link information: display the current link status of the ethernet ports.
```
 port link
```
- Port statistics: display (detailed) statistics for the ethernet ports currently in use (e.g., received/transmitted packets/bytes). If detailed stats are requested then the information is displayed per port + lcore.
```
 port statistics [details]
```
Ethernet statistics: display (detailed) statistics regarding the Ethernet layer processing (e.g., ethernet type, errors). If detailed stats are requested then the information is displayed per port + lcore.
```
 ethernet statistics [details]
```
ARP information and statistics
- ARP tables: display the ARP tables for each ethernet port currently in use. For now L3 interfaces are defined per ethernet port and not per test case ID. This enforces a unique ARP table per port.
  
  NOTE: The current ARP implementation is limited in the sense that whenever tests are started on a port, gratuituous ARPs are sent for all the L3 interfaces that were defined on that port and an ARP request is sent for the default gateway. All ARP requests and replies are properly processed but there is no timeout mechanism in place for aging entries!
```
 arp entries
```
- ARP statistics: display (detailed) statistics regarding ARP processing (e.g., request/response count, errors). If detailed stats are requested then the information is displayed per port + lcore.
```
 arp statistics [details]
```
Route statistics: display (detailed) statistics for the routing module (e.g., interface/gateway creation/deletion count, errors). The current routing implementation is minimal and only handles L3 interface creation/deletion and default gateways.
```
 route statistics [details]
```
IPv4 statistics: display (detailed) statistics regarding IPv4 processing (e.g., received packet/bytes counts, per L4 protocol counters, errors). If detailed stats are requested then the information is displayed per port + lcore.
```
 ipv4 statistics [details]
```
TCP statistics: display (detailed) statistics regarding TCP processing (e.g., received packets/bytes counts, sent control/data packets/bytes counts, allocation counters, errors). If detailed stats are requested then the information is displayed per port + lcore.
```
 tcp statistics [details]
```
TCP state machine statistics: display (detailed) statistics regarding TCP state machine processing (e.g., per TCP state counters, retransmission counters, missing sequence counters). If detailed stats are requested then the information is displayed per port + lcore.
```
 tsm statistics [details]
```
UDP statistics: display (detailed) statistics regarding UDP processing (e.g., received packets/bytes counts, sent packets/bytes counts, allocation counters, errors). If detailed stats are requested then the information is displayed per port + lcore.
```
 udp statistics [details]
```
Timer statistics: there are currently three types of supported timers: fast retransmission timers, slow timers (e.g., TIME-WAIT) and test timers. Test_ timers are used by the test engine and the others are used by the TCP/UDP stack implementations. The command displays (detailed) statistics regarding these types of timers. If detailed stats are requested then the information is displayed per port + lcore.
```
 timer statistics [details]
```

Infrastructure statistics

Message queues statistics: all communication between lcores (PKT or CLI) is done by means of message passing. Each lcore has two message queues (a local and a global queue storing messages based on the originator of the message). The command displays (detailed) statistics regarding the message queues (e.g., messages sent/received/allocated, errors). If detailed stats are requested then the information is displayed per port + lcore.
```
 msg statistics [details]
```
Memory statistics: most of the memory used during the tests is allocated from different mempools (mbufs, TCP/UDP control blocks). The command displays (detailed) statistics regarding the usage of the memory pools. If detailed stats are requested then the information is displayed per port + lcore.
```
 memory statistics [details]
```
Modifying Log Levels: allow the user to change the syslog verbosity.
```
syslog <level>
```
Available log levels (corresponding to DPDK log levels):
- EMERG: System is unusable.
- ALERT: Action must be taken immediately.
- CRIT: Critical conditions.
- ERR: Error conditions.
- WARNING: Warning conditions.
- NOTICE: Normal but significant condition.
- INFO: Informational.
- DEBUG: Debug-level messages.

UI

show tests ui displays an UI which allows monitoring the test execution.

The UI is split in 4 main areas:

test status area
detailed test statistics area: Open/Closed/Send statistics and Application statistics are displayed for the currently selected test case.
detailed test configuration area: display the complete configuration of the currently selected test case. The user can navigate between test cases by pressing n for moving to the next test case and b for moving to the previous test case. Switching between the configuration view and statistics view can be done using the c and s keys.
statistics area: for each of the ethernet ports various statistics will be displayed for all levels of the TCP/IP stack.

Example run

Some example configuration files can be found in the examples/ directory. The configuration files can either be passed as a command-line argument, --cmd-file=<file>, when running WARP17 or executed directly in the CLI.

examples/test_1_raw_tcp_connection.cfg: single TCP client-server connection on a back to back setup using RAW application data (requests of size 100 and responses of size 200 bytes). The client connects immediately when the test starts and sends requests continuously (and waits for responses) until the uptime expires (5 seconds), closes the connection and reconnects after the downtime expires (15 seconds).
examples/test_2_raw_udp_connection.cfg: single UDP client-server connection on a back to back setup using RAW application data (requests of size 100 and responses of size 200 bytes). The client connects with a delay of 10 seconds (init) then sends requests continuously (and waits for responses) until the uptime expires (5 seconds), closes the connection and reconnects downtime expires (15 seconds).
examples/test_3_http_multiple.cfg: two client test cases each with a single HTTP client. The first client test case sends GET requests while the second one sends HEAD requests. The first test case is marked as async which will determine WARP17 to start both of them in parallel. The HTTP server test case is configured to reply with 200 OK.
examples/test_4_http_10M_sessions.cfg: single test case per port configuring 10M HTTP sessions. The test case on port 0 will establish connections from 10.0.0.1:[10000, 60000) to 10.0.0.253:[6000, 6200). On each of those connections HTTP GET requests will be sent continuously until the uptime of 30 seconds expires. Then the connections are closed. After another 15 seconds of downtime the clients reconnect and start over.
examples/test_5_raw_10M_sessions.cfg: single test case per port configuring 10M RAW sessions. The test case on port 0 will establish connections from 10.0.0.1:[10000, 60000) to 10.0.0.253:[6000, 6200). On each of those connections RAW requests of size 1K will be sent continuously. uptime is configured as infinite so the clients will stay UP forever. If the connection goes down (e.g., TCP session fails) then the client will reconnect after a downtime of 10 seconds. The RAW servers reply with responses of size 4K. The clients are also rate limited to 1M sessions/s open and 900K sess/s send rate (clients will)
examples/test_6_http_40M_sessions.cfg: single test case per port configuring 40M HTTP sessions. The test case on port 0 will establish connections from [10.0.0.1, 10.0.0.4]:[10000, 60000) to 10.0.0.253:[6000, 6200). On each of those connections HTTP GET requests will be sent continuously.
examples/test_7_routing_raw_8M_sesssions.cfg: example config to be used when having (multiple) routers in between the client and server ports.
examples/test_8_http_fields.cfg: example showing how to configure various HTTP fields in the requests/responses (e.g., Content-Type).
examples/test_9_ipv4_tos.cfg: example showing how to configure various TOS or DSCP/ECN values as part of the IPv4 options of the test cases.
examples/test_10_ipv4_mcast.cfg: example showing how to configure UDP Multicast Source test cases. The example combines UDP Unicast traffic with UDP Multicast traffic.
examples/test_11_ipv4_latency.cfg: example showing how to configure latency measurement on a TCP test case. Maximum and average thresholds are configured.
examples/test_12_raw_latency.cfg: example showing how to configure latency measurement using application layer timestamping on TCP and UDP test cases.
examples/test_13_vlan_udp.cfg: example showing how to configure vlan information and per vlan gateways.
examples/test_14_imix.cfg: example showing how to combine multiple L5 applications inside IMIX groups. Applications have different weights which are used for computing how often the apps will be represented in the traffic profile.

Python scripting API

WARP17 offers an RPC-based API which allows users to write scripts and automate the tests that WARP17 would run. WARP17 listens to incoming RPC connections on TCP port 42424.

The RPC object definitions can be found in the api/*.proto files. The main RPC interface is defined in api/warp17-service.proto. All *.proto files are compiled into Python classes when building WARP17. The generated code is saved in the api/generated/py directory (one .py file for each .proto definition file).

A short example about how to use the Python API can be found in examples/python/test_1_http_4M.py. The example sets up 4M HTTP clients and servers, polls for statistics and stops the tests after a while.

Perl scripting API

WARP17 can also be scripted through Perl by using the Inline::Python module. A short example about how to use Perl to script WARP17 can be found in examples/perl/test_1_http_4M.pl. Requirements for running the Perl scripts:

 apt-get install python2.7-dev cpanminus
 cpanm Inline::Python


 perl -I ./perl/ examples/perl/test_1_http_4M.pl

Contributing a new L7 Application implementation

WARP17 currently supports RAW TCP and HTTP 1.1 application traffic. Even though we are currently working on adding support for more application implementations, external contributions are welcome.

As a future development WARP17 will offer a socket API in order to allow applications to be easily plugged in. Until then any new application must be directly added to the WARP17 code. As an example, a good starting point is the HTTP 1.1 implementation itself.

In general, an application called foo should implement the following:

warp17-app-foo.proto definition file in api/: should contain the application configuration definitions (for clients and servers) and preferably application specific statistics definitions.

warp17-app-foo.proto should be included in warp17-app.proto and the application the App structure:

age App {
quired AppProto app_proto = 1;

/* Add different App configs below as optionals. */
tional RawClient  app_raw_client  = 2 [(warp17_union_anon) = true];
tional RawServer  app_raw_server  = 3 [(warp17_union_anon) = true];
tional HttpClient app_http_client = 4 [(warp17_union_anon) = true];
tional HttpServer app_http_server = 5 [(warp17_union_anon) = true];
tional Imix       app_imix        = 6 [(warp17_union_anon) = true];
optional Foo            app_foo                 = 7 [(warp17_union_anon) = true];

the application specific statistics should also be added to the AppStats definition:

age AppStats {
 The user will do the translation. */
tion (warp17_xlate_tpg) = false;

tional RawStats  as_raw  = 1 [(warp17_union_anon) = true];
tional HttpStats as_http = 2 [(warp17_union_anon) = true];
tional ImixStats as_imix = 3 [(warp17_union_anon) = true];
optional FooStats  as_foo  = 4 [(warp17_union_anon) = true];

a new entry for the application type should be added to the AppProto enum in warp17-common.proto:

 AppProto {
RAW_CLIENT    = 0;
RAW_SERVER    = 1;
HTTP_CLIENT   = 2;
HTTP_SERVER   = 3;
IMIX          = 4;
    FOO           = 5;
APP_PROTO_MAX = 6;

the new protocol buffer file (warp17-app-foo.proto) should also be added to api/Makefile.api:

O-SRCS += warp17-app-raw.proto
O-SRCS += warp17-app-http.proto
O-SRCS += warp17-app-foo.proto

the file Makefile.dpdk should also be updated to include the new application implementation:

-y += tpg_test_app.c
-y += tpg_test_http_1_1_app.c
-y += tpg_test_raw_app.c
-y += tpg_test_foo_app.c

include warp17-app-foo.proto in tcp_generator.h:

lude "warp17-app-raw.proto.xlate.h"
lude "warp17-app-http.proto.xlate.h"
lude "warp17-app-foo.proto.xlate.h"

RPC WARP17 to protobuf translation code:

a new case entry in tpg_xlate_tpg_union_App where the application translation function should be called:

 APP_PROTO__HTTP_SERVER:
t->app_http_server =
  rte_zmalloc("TPG_RPC_GEN", sizeof(*out->app_http_server), 0);
 (!out->app_http_server)
  return -ENOMEM;

g_xlate_tpg_HttpServer(&in->app_http_server, out->app_http_server);
eak;
 APP_PROTO__FOO:
out->app_foo = rte_zmalloc("TPG_RPC_GEN", sizeof(*out->app_foo), 0);
if (!out->app_foo)
    return -ENOMEM;

tpg_xlate_tpg_Foo(&in->app_foo, out->app_foo);
break;

a new case entry in tpg_xlate_tpgTestAppStats_by_proto when translating application statistics:

 APP_PROTO__HTTP_SERVER:
t->as_http = rte_zmalloc("TPG_RPC_GEN", sizeof(*out->as_http), 0);
 (!out->as_http)
 return -ENOMEM;

r = tpg_xlate_tpg_HttpStats(&in->as_http, out->as_http);
 (err)
  return err;
eak;
 APP_PROTO__FOO:
t->as_foo = rte_zmalloc("TPG_RPC_GEN", sizeof(*out->as_foo), 0);
 (!out->as_foo)
  return -ENOMEM;

r = tpg_xlate_tpg_FooStats(&in->as_foo, out->as_foo);
 (err)
  return err;
eak;

appl/tpg_test_app.h interface implementation:
- application foo should be added to the app_data_t definition in inc/appl/tpg_test_app.h. Type foo_app_t should be defined in the application headers and should represent a state storage for the foo application. The state is part of the L4 control block structures (TCB/UCB).
```
def struct app_data_s {

[...]

union {
    raw_app_t     ad_raw;
    http_app_t    ad_http;
    foo_app_t     ad_foo;
    generic_app_t ad_generic;
};

p_data_t;
```
- foo must also provide callback functions corresponding to the callback types defined in inc/appl/tpg_test_app.h. The callbacks should be added to the callback arrays in src/appl/tpg_test_app.c. These functions will be called by the test engine whenever application intervention is required:
  - app_default_cfg_cb_t: should initialize the foo application config to default values
  - app_validate_cfg_cb_t: should validate the config corresponding to the foo application
  - app_print_cfg_cb_t: should display the part of the configuration corresponding to the foo application by using the supplied printer
  - app_add_cfg_cb_t: will be called whenever a test case is added so foo initialize everything needed for the test case.
  - app_delete_cfg_cb_t: will be called whenever a test case is deleted so foo can cleanup anything it initialized for the test case.
  - app_pkts_per_send_cb_t: will be called when the test case is started to determine how many packets (on average) will foo be sending for a single data message
  - app_init_cb_t: will be called whenever a session is initialized and should initialize the application state.
  - app_tc_start_stop_cb_t: foo should define two callbacks (for test case start and stop). The application should initialize and cleanup any data that is required during the test case (e.g., predefined static data headers)
  - app_conn_up_cb_t: will be called whenever a session has been established
  - app_conn_down_cb_t: will be called whenever a session closed ( either because the underlying connection went down or because the application itself decided to close the connection)
  - app_deliver_cb_t: will be called whenever there was data received for the application to process. The received data is passed as an mbuf chain. The callback should return the total number of bytes that were consumed. For example, in the case of TCP these bytes will be ACK-ed and removed from the receive window.
  - app_send_cb_t: will be called whenever the test engine can send data on the current connection. The application can decide at any time that it would like to start sending or stop sending data by notifying the test engine through the TEST_NOTIF_APP_CLIENT/SERVER_SEND_STOP/START notifications. The app_send_cb_t callback should return an mbuf chain pointing to the data it would like to send. In general, freeing the mbuf upon sending is the job of the TCP/IP stack so the application must make sure that it doesn't continue using the mbuf after passing it to the test engine. NOTE: However, for some applications, in order to avoid building packets every time, the implementation might prefer to reuse data templates (e.g., HTTP requests can be easily prebuilt when the test case is started). In such a situation the application can mark the mbufs as STATIC through the DATA_SET_STATIC call which will inform the test engine that it shouldn't free the data itself. The application must ensure in such a case that the data itself is never freed during the execution of the test case!
  - app_data_sent_cb_t: will be called to notify the application that (part of) the data was sent. It might happen that not all the data could be sent in one shot so the application should return true if what was sent corresponds to a complete message
  - app_stats_add_cb_t: should aggregate application specific statistics
  - app_stats_print_cb_t: should print application specific statistics using the supplied printer
- the foo application can request the test engine to perform operations by sending the following notifications:
  - TEST_NOTIF_APP_CLIENT/SERVER_SEND_START: notifies the test engine that the application would like to send data (when possible) on the current connection
  - TEST_NOTIF_APP_CLIENT/SERVER_SEND_STOP: notifies the test engine that the application has finished sending data (for now) on the current connection
  - TEST_NOTIF_APP_CLIENT/SERVER_CLOSE: notifies the test engine that the application would like to close the connection
CLI: the foo application can define it's own CLI commands using the DPDK cmdline infrastructure. These can be added to a local cli_ctx which can be registered with the main CLI through a call to cli_add_main_ctx.
module initialization: the foo application must implement two module init functions:
- foo_init: should initialize global data to be used by the application (e.g., CLI, statistics storage). foo_init should be called directly from the main WARP17 function where all modules are initialized.
- foo_lcore_init: should initalize per core global data to be used by the application (e.g., per core pointers to the statistics corresponding to the current core). foo_lcore_init should be called from pkt_receive_loop where all modules are initialized.
example config: ideally, applications should also provide some configuration examples which could go to the examples/ directory.
.dot file: applications will most likely be implemented as state machines. A .dot file describing the state machine should be added to the dot/ directory
tests: any new application shouldn't break any existing tests and must have it's own tests:
- a configuration and functionality test file in ut/test_foo.py which should try to extensively cover all the code introduced by the application
- one or more scaling test entries (method) in ut/test_perf.py (class TestPerf) which should define the desired performance/scalability values.
commit messages: please make sure that commit messages follow the .git-commit.template provided in the repository. In order to enforce this template locally you can execute the following command:

config commit.template ./.git-commit.template

Release notes

For a summary of the currently supported functionalities please check the RELEASE_NOTES file.

Roadmap for future releases

Additional L7 application implementations (e.g., FTP, TFTP, SIP).
Socket API.
Fault injection at various levels in the L2-L7 stack.

Contact

Feel free to also check out the WARP17 google group.

For a list of maintainers and contributors please check the MAINTAINERS and CONTRIBUTORS files.

License

WARP17 is released under BSD 3-Clause license. The license file can be found here.

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.