From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from tyo201.gate.nec.co.jp (TYO201.gate.nec.co.jp [210.143.35.51]) by dpdk.org (Postfix) with ESMTP id B80535902 for ; Tue, 30 Sep 2014 13:09:35 +0200 (CEST) Received: from mailgate3.nec.co.jp ([10.7.69.197]) by tyo201.gate.nec.co.jp (8.13.8/8.13.4) with ESMTP id s8UBGE3Y026314 for ; Tue, 30 Sep 2014 20:16:14 +0900 (JST) Received: from mailsv3.nec.co.jp (imss61.nec.co.jp [10.7.69.156]) by mailgate3.nec.co.jp (8.11.7/3.7W-MAILGATE-NEC) with ESMTP id s8UBGER11156 for ; Tue, 30 Sep 2014 20:16:14 +0900 (JST) Received: from mail02.kamome.nec.co.jp (mail02.kamome.nec.co.jp [10.25.43.5]) by mailsv3.nec.co.jp (8.13.8/8.13.4) with ESMTP id s8UBGEwK009540 for ; Tue, 30 Sep 2014 20:16:14 +0900 (JST) Received: from bpxc99gp.gisp.nec.co.jp ([10.38.151.143] [10.38.151.143]) by mail01b.kamome.nec.co.jp with ESMTP id BT-MMP-2225085; Tue, 30 Sep 2014 20:11:57 +0900 Received: from BPXM14GP.gisp.nec.co.jp ([169.254.1.136]) by BPXC15GP.gisp.nec.co.jp ([10.38.151.143]) with mapi id 14.03.0174.002; Tue, 30 Sep 2014 20:11:57 +0900 From: Hiroshi Shimamoto To: "dev@dpdk.org" Thread-Topic: [memnic PATCH v2 1/7] guest: memnic-tester: PMD benchmark in guest Thread-Index: Ac/cn1GaJK4M6E+ETjyl6SzagP1LNw== Date: Tue, 30 Sep 2014 11:11:56 +0000 Message-ID: <7F861DC0615E0C47A872E6F3C5FCDDBD02AE2629@BPXM14GP.gisp.nec.co.jp> Accept-Language: ja-JP, en-US Content-Language: ja-JP X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.205.5.123] Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Hayato Momma Subject: [dpdk-dev] [memnic PATCH v2 1/7] guest: memnic-tester: PMD benchmark in guest X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Sep 2014 11:09:36 -0000 From: Hiroshi Shimamoto Introduce memnic-tester which benchmarks MEMNIC PMD performance in guest. It starts with two threads, one thread produces and consumes packets, other thread receives packets and directly transmits the received packets. This evaluates MEMNIC PMD running cost. memnic-tester is a benchmark tool to measure performance of MEMNIC PMD itse= lf. The master thread forward packets with Rx and Tx bursts. The slave thread fills and clears packets in the lightest way. It doesn't g= et packet out of VM because it would increase jitter and hide PMD performance. Throughput (number of forwarded packets per second) is given for each frame= size. The master thread does rx_burst and tx_burst through MEMNIC PMD. +---------+ | master | +---------+ rx_burst ^ | tx_burst | V +------+------+ | up | down | MEMNIC shared memory +------+------+ set flag ^ | unset flag | V +---------+ | slave | +---------+ The slave thread emulates packet-in/out by setting flag on/off. It shows that throughputs in different frame size. 64, 128, 256, 512, 1024, 1280, 1518 Signed-off-by: Hiroshi Shimamoto Reviewed-by: Hayato Momma --- guest/Makefile | 20 ++++ guest/README.rst | 93 +++++++++++++++++ guest/memnic-tester.c | 281 ++++++++++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 394 insertions(+) create mode 100644 guest/Makefile create mode 100644 guest/README.rst create mode 100644 guest/memnic-tester.c diff --git a/guest/Makefile b/guest/Makefile new file mode 100644 index 0000000..3c90350 --- /dev/null +++ b/guest/Makefile @@ -0,0 +1,20 @@ +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overriden by command line or environment +ifeq ($(RTE_TARGET),) +$(error "Please define RTE_TARGET environment variable") +endif + +include $(RTE_SDK)/mk/rte.vars.mk + +COMMON_INC_OPT =3D -I $(PWD)/../common + +APP =3D memnic-tester + +CFLAGS +=3D -Wall -g -O3 $(COMMON_INC_OPT) + +SRCS-y :=3D memnic-tester.c + +include $(RTE_SDK)/mk/rte.extapp.mk diff --git a/guest/README.rst b/guest/README.rst new file mode 100644 index 0000000..eb230b0 --- /dev/null +++ b/guest/README.rst @@ -0,0 +1,93 @@ +.. Copyright 2014 NEC + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + - Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + - Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE + COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, + INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + OF THE POSSIBILITY OF SUCH DAMAGE. + +MEMNIC TESTER +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +DESCRIPTION +----------- + +It is a simple benchmark test of MEMNIC PMD in guest. + +It have two threads, one thread produces and consumes packets, +other thread receives packets and directly transmits the received +packets back in MEMNIC interface. This evaluates MEMNIC PMD running cost. + +memnic-tester is a benchmark tool to measure performance of MEMNIC PMD its= elf. +The master thread forward packets with Rx and Tx bursts. +The slave thread fills and clears packets in the lightest way. It doesn't = get +packet out of VM because it would increase jitter and hide PMD performance= . +Throughput (number of forwarded packets per second) is given for each fram= e size. + +The master thread does rx_burst and tx_burst through MEMNIC PMD. + +---------+ + | master | + +---------+ + rx_burst ^ | tx_burst + | V + +------+------+ + | up | down | MEMNIC shared memory + +------+------+ + set flag ^ | unset flag + | V + +---------+ + | slave | + +---------+ +The slave thread emulates packet-in/out by setting flag on/off. + +Like RFC2544, evaluations are performed the below frame size packets. + 64, 128, 256, 512, 1024, 1280, 1518 + +It shows the result as packets per second number of each frame size. + +HOW TO BUILD +------------ + +DPDK and DPDK MEMNIC PMD must be built first like below:: + + cd /path/to/dpdk + make install T=3Dx86_64-native-linuxapp-gcc + cd /path/to/memnic/pmd + make RTE_INCLUDE=3D/path/to/dpdk/x86_64-native-linuxapp-gcc/include + cd /path/to/memnic/guest + make RTE_SDK=3D/path/to/dpdk RTE_TARGET=3Dx86_64-native-linuxapp-gcc + +The file ``memnic-tester`` is generated under ``build`` directory. + +HOW TO RUN +---------- + +On host the MEMNIC device must be initialized with proper program. +``memnic-host-sim`` should take care about it:: + + [host]# ./memnic-host-sim /dev/shm/ivshm + +Then stop the ``memnic-host-sim`` by CTRL-C. + +Run ``memnic-tester`` in guest:: + + [guest]# echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-20= 48kB/nr_hugepages + [guest]# mount -t hugetlbfs nodev /mnt/huge + [guest]# ./build/memnic-tester -c 0x6 -n 4 -d /path/to/librte_pmd_memnic= _copy.so + +The result shows how much packets are handled by MEMNIC PMD per second. diff --git a/guest/memnic-tester.c b/guest/memnic-tester.c new file mode 100644 index 0000000..10e304b --- /dev/null +++ b/guest/memnic-tester.c @@ -0,0 +1,281 @@ +/* + * BSD LICENSE + * + * Copyright(c) 2014 NEC All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "memnic.h" + +#define PKTS_BURST_SIZE (32) +#define TEST_DURATION (10) + +static const struct rte_eth_conf port_conf =3D { + .rxmode =3D { + .jumbo_frame =3D 0, + }, + .txmode =3D { + .mq_mode =3D ETH_MQ_TX_NONE, + }, +}; + +static const struct rte_eth_rxconf rx_conf =3D { + .rx_thresh =3D { + .pthresh =3D 8, + .hthresh =3D 8, + .wthresh =3D 4, + }, +}; + +static const struct rte_eth_txconf tx_conf =3D { + .tx_thresh =3D { + .pthresh =3D 36, + .hthresh =3D 0, + .wthresh =3D 0, + }, +}; + +static const unsigned nr_rxdesc =3D 128; +static const unsigned nr_txdesc =3D 512; + +static struct rte_mempool *mempool; + +static struct memnic_area *memnic; + +static void init_port(unsigned portid) +{ + if (rte_eth_dev_configure(portid, 1, 1, &port_conf) < 0) { + rte_exit(EXIT_FAILURE, "failed to configure port %u\n", + portid); + } + if (rte_eth_rx_queue_setup(portid, 0, nr_rxdesc, + rte_eth_dev_socket_id(portid), &rx_conf, + mempool) < 0) { + rte_exit(EXIT_FAILURE, + "failed to configure rx queue port %u\n", + portid); + } + if (rte_eth_tx_queue_setup(portid, 0, nr_txdesc, + rte_eth_dev_socket_id(portid), &tx_conf) < 0) { + rte_exit(EXIT_FAILURE, + "failed to configure tx queue port %u\n", + portid); + } + + rte_eth_promiscuous_enable(portid); +} + +static void reset_memnic(void) +{ + struct memnic_header *hdr =3D &memnic->hdr; + struct memnic_data *up =3D &memnic->up; + struct memnic_data *down =3D &memnic->down; + int i; + + /* prepare packet data */ + for (i =3D 0; i < MEMNIC_NR_PACKET; i++) { + struct memnic_packet *p =3D &up->packets[i]; + + p->status =3D MEMNIC_PKT_ST_FREE; + p->len =3D 60; /* short packet */ + + /* don't care about content */ + } + + /* clear packet data */ + for (i =3D 0; i < MEMNIC_NR_PACKET; i++) { + struct memnic_packet *p =3D &down->packets[i]; + + p->status =3D MEMNIC_PKT_ST_FREE; + } + + /* use default framesz */ + hdr->framesz =3D MEMNIC_MAX_FRAME_LEN; + + rte_compiler_barrier(); + + hdr->reset =3D 0; + hdr->valid =3D 1; +} + +static void slave(void) +{ + struct memnic_header *hdr =3D &memnic->hdr; + int up_idx, down_idx; + uint64_t hz =3D rte_get_tsc_hz(), next; + uint64_t count; + /* RFC2544 like */ + uint32_t testset[] =3D {64, 128, 256, 512, 1024, 1280, 1518}; + int n, nr_tests =3D sizeof(testset) / sizeof(uint32_t); + + /* wait to turn reset flag on */ + while (hdr->reset =3D=3D 0) + rte_pause(); + + /* wait a sec to confirm no one handles this MEMNIC in host side */ + next =3D rte_rdtsc() + hz; + while (next < rte_rdtsc()) { + if (ACCESS_ONCE(hdr->valid)) + rte_exit(EXIT_FAILURE, "MEMNIC is active\n"); + } + + reset_memnic(); + + up_idx =3D down_idx =3D 0; + + for (n =3D 0; n < nr_tests; n++) { + struct memnic_data *up =3D &memnic->up; + struct memnic_data *down =3D &memnic->down; + struct memnic_packet *p; + int i; + + /* prepare incoming packet */ + for (i =3D 0; i < MEMNIC_NR_PACKET; i++) { + p =3D &up->packets[i]; + p->len =3D testset[n] - 4; /* remove FCS */ + } + + count =3D 0; + next =3D rte_rdtsc() + hz * TEST_DURATION; + while (next > rte_rdtsc()) { + /* put packets */ + for (i =3D 0; i < PKTS_BURST_SIZE; i++) { +xmit_retry: + p =3D &up->packets[up_idx]; + if (ACCESS_ONCE(p->status) !=3D MEMNIC_PKT_ST_FREE) + goto xmit_retry; + if (++up_idx >=3D MEMNIC_NR_PACKET) + up_idx =3D 0; + p->status =3D MEMNIC_PKT_ST_FILLED; + } + /* get packets */ + for (i =3D 0; i < PKTS_BURST_SIZE; i++) { +recv_retry: + p =3D &down->packets[down_idx]; + if (ACCESS_ONCE(p->status) !=3D MEMNIC_PKT_ST_FILLED) + goto recv_retry; + if (++down_idx >=3D MEMNIC_NR_PACKET) + down_idx =3D 0; + p->status =3D MEMNIC_PKT_ST_FREE; + } + ++count; + } + printf("frame size %u throughput %lu pps\n", + testset[n], (count * PKTS_BURST_SIZE) / TEST_DURATION); + } + + /* finish the test */ + rte_exit(EXIT_SUCCESS, "Test done\n"); +} + +static void master(void) +{ + if (rte_eth_dev_start(0) < 0) + rte_exit(EXIT_FAILURE, "failed to start device\n"); + + /* infinity loop back */ + for (;;) { + struct rte_mbuf *bufs[PKTS_BURST_SIZE]; + int rx, tx; + + rx =3D rte_eth_rx_burst(0, 0, bufs, PKTS_BURST_SIZE); + tx =3D 0; + while (rx !=3D tx) + tx +=3D rte_eth_tx_burst(0, 0, &bufs[tx], rx - tx); + } +} + +static int lcore_main(void *p) +{ + if (rte_lcore_id() =3D=3D rte_get_master_lcore()) + master(); + else + slave(); + + /* never reach here */ + return 0; +} + +int main(int argc, char **argv) +{ + struct rte_eth_dev *dev; + struct memnic_adapter { + struct memnic_area *nic; + } *adapter; + int ret; + unsigned lcore_id; + + ret =3D rte_eal_init(argc, argv); + if (ret < 0) + exit(1); + + argc -=3D ret; + argv +=3D ret; + + if (rte_lcore_count() !=3D 2) + rte_exit(EXIT_FAILURE, "Need just 2 lcores\n"); + + /* alloc mempool */ + mempool =3D rte_mempool_create("pkt_mempool", 8192, 2048, 32, + sizeof(struct rte_pktmbuf_pool_private), + rte_pktmbuf_pool_init, NULL, + rte_pktmbuf_init, NULL, + rte_socket_id(), 0); + + if (rte_eal_pci_probe() < 0) + rte_exit(EXIT_FAILURE, "failed to probe PCI\n"); + + /* get MEMNIC data from ether device */ + dev =3D &rte_eth_devices[0]; + adapter =3D (struct memnic_adapter *)(dev->data->dev_private); + + memnic =3D adapter->nic; + + if (memnic->hdr.magic !=3D MEMNIC_MAGIC) + rte_exit(EXIT_FAILURE, "Not a MEMNIC device\n"); + + /* port 0 must initialize MEMNIC */ + init_port(0); + + rte_eal_mp_remote_launch(lcore_main, NULL, CALL_MASTER); + RTE_LCORE_FOREACH_SLAVE(lcore_id) { + if (rte_eal_wait_lcore(lcore_id) < 0) + return -1; + } + + return 0; +} --=20 1.8.3.1