From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 017FC1B03F for ; Tue, 16 Jan 2018 13:38:39 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2018 04:38:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,368,1511856000"; d="scan'208";a="166484667" Received: from irsmsx108.ger.corp.intel.com ([163.33.3.3]) by orsmga004.jf.intel.com with ESMTP; 16 Jan 2018 04:38:37 -0800 Received: from irsmsx155.ger.corp.intel.com (163.33.192.3) by IRSMSX108.ger.corp.intel.com (163.33.3.3) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 16 Jan 2018 12:38:36 +0000 Received: from irsmsx105.ger.corp.intel.com ([169.254.7.236]) by irsmsx155.ger.corp.intel.com ([169.254.14.169]) with mapi id 14.03.0319.002; Tue, 16 Jan 2018 12:38:36 +0000 From: "Ananyev, Konstantin" To: "wei.guo.simon@gmail.com" , "Lu, Wenzhuo" CC: "dev@dpdk.org" , Thomas Monjalon Thread-Topic: [dpdk-dev] [PATCH v5] app/testpmd: add option ring-bind-lcpu to bind Q with CPU Thread-Index: AQHTjm9RpGgyYska406FLNCRTPlp06N2b09A Date: Tue, 16 Jan 2018 12:38:35 +0000 Message-ID: <2601191342CEEE43887BDE71AB9772588627E492@irsmsx105.ger.corp.intel.com> References: <6A0DE07E22DDAD4C9103DF62FEBC09093B7109A8@shsmsx102.ccr.corp.intel.com> <1515810914-18762-1-git-send-email-wei.guo.simon@gmail.com> In-Reply-To: <1515810914-18762-1-git-send-email-wei.guo.simon@gmail.com> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZDZhMjQ2NDMtNjYxNC00NzM0LWI5YzktNzhjNDA4YmFjZWU5IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IkVRaGpcL01PRE0wd1JIYlI5NDluVDhxVFwvd25GejlYRWp2bFVyRlNGTlwvMWs9In0= x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v5] app/testpmd: add option ring-bind-lcpu to bind Q with CPU X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jan 2018 12:38:40 -0000 > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of wei.guo.simon@gmail.= com > Sent: Saturday, January 13, 2018 2:35 AM > To: Lu, Wenzhuo > Cc: dev@dpdk.org; Thomas Monjalon ; Simon Guo > Subject: [dpdk-dev] [PATCH v5] app/testpmd: add option ring-bind-lcpu to = bind Q with CPU >=20 > From: Simon Guo >=20 > Currently the rx/tx queue is allocated from the buffer pool on socket of: > - port's socket if --port-numa-config specified > - or ring-numa-config setting per port >=20 > All the above will "bind" queue to single socket per port configuration. > But it can actually archieve better performance if one port's queue can > be spread across multiple NUMA nodes, and the rx/tx queue is allocated > per lcpu socket. >=20 > This patch adds a new option "--ring-bind-lcpu"(no parameter). With > this, testpmd can utilize the PCI-e bus bandwidth on another NUMA > nodes. >=20 > When --port-numa-config or --ring-numa-config option is specified, this > --ring-bind-lcpu option will be suppressed. Instead of introducing one more option - wouldn't it be better to=20 allow user manually to define flows and assign them to particular lcores? Then the user will be able to create any FWD configuration he/she likes. Something like: lcore X add flow rxq N,Y txq M,Z Which would mean - on lcore X recv packets from port=3DN, rx_queue=3DY, and send them through port=3DM,tx_queue=3DZ. Konstantin =20 =20 >=20 > Test result: > 64bytes package, running in PowerPC with Mellanox > CX-4 card, single port(100G), with 8 cores, fw mode: > - Without this patch: 52.5Mpps throughput > - With this patch: 66Mpps throughput > ~25% improvement >=20 > Signed-off-by: Simon Guo > --- > app/test-pmd/parameters.c | 14 ++++- > app/test-pmd/testpmd.c | 112 ++++++++++++++++++++++++----= ------ > app/test-pmd/testpmd.h | 7 +++ > doc/guides/testpmd_app_ug/run_app.rst | 6 ++ > 4 files changed, 107 insertions(+), 32 deletions(-) >=20 > diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c > index 304b98d..1dba92e 100644 > --- a/app/test-pmd/parameters.c > +++ b/app/test-pmd/parameters.c > @@ -104,6 +104,10 @@ > "(flag: 1 for RX; 2 for TX; 3 for RX and TX).\n"); > printf(" --socket-num=3DN: set socket from which all memory is allocat= ed " > "in NUMA mode.\n"); > + printf(" --ring-bind-lcpu: " > + "specify TX/RX rings will be allocated on local socket of lcpu." > + "It will be ignored if ring-numa-config or port-numa-config is used. " > + "As a result, it allows one port binds to multiple NUMA nodes.\n"); > printf(" --mbuf-size=3DN: set the data size of mbuf to N bytes.\n"); > printf(" --total-num-mbufs=3DN: set the number of mbufs to be allocate= d " > "in mbuf pools.\n"); > @@ -544,6 +548,7 @@ > { "interactive", 0, 0, 0 }, > { "cmdline-file", 1, 0, 0 }, > { "auto-start", 0, 0, 0 }, > + { "ring-bind-lcpu", 0, 0, 0 }, > { "eth-peers-configfile", 1, 0, 0 }, > { "eth-peer", 1, 0, 0 }, > #endif > @@ -676,6 +681,10 @@ > stats_period =3D n; > break; > } > + if (!strcmp(lgopts[opt_idx].name, "ring-bind-lcpu")) { > + ring_bind_lcpu |=3D RBL_BIND_LOCAL_MASK; > + break; > + } > if (!strcmp(lgopts[opt_idx].name, > "eth-peers-configfile")) { > if (init_peer_eth_addrs(optarg) !=3D 0) > @@ -739,11 +748,14 @@ > if (parse_portnuma_config(optarg)) > rte_exit(EXIT_FAILURE, > "invalid port-numa configuration\n"); > + ring_bind_lcpu |=3D RBL_PORT_NUMA_MASK; > } > - if (!strcmp(lgopts[opt_idx].name, "ring-numa-config")) > + if (!strcmp(lgopts[opt_idx].name, "ring-numa-config")) { > if (parse_ringnuma_config(optarg)) > rte_exit(EXIT_FAILURE, > "invalid ring-numa configuration\n"); > + ring_bind_lcpu |=3D RBL_RING_NUMA_MASK; > + } > if (!strcmp(lgopts[opt_idx].name, "socket-num")) { > n =3D atoi(optarg); > if (!new_socket_id((uint8_t)n)) { > diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c > index 9414d0e..e9e89d0 100644 > --- a/app/test-pmd/testpmd.c > +++ b/app/test-pmd/testpmd.c > @@ -68,6 +68,9 @@ > uint8_t interactive =3D 0; > uint8_t auto_start =3D 0; > uint8_t tx_first; > + > +uint8_t ring_bind_lcpu; > + > char cmdline_filename[PATH_MAX] =3D {0}; >=20 > /* > @@ -1410,6 +1413,43 @@ static int eth_event_callback(portid_t port_id, > return 1; > } >=20 > +static int find_local_socket(queueid_t qi, int is_rxq) > +{ > + /* > + * try to find the local socket with following logic: > + * 1) Find the correct stream for the queue; > + * 2) Find the correct lcore for the stream; > + * 3) Find the correct socket for the lcore; > + */ > + int i, j, socket =3D NUMA_NO_CONFIG; > + > + /* find the stream based on queue no*/ > + for (i =3D 0; i < nb_fwd_streams; i++) { > + if (is_rxq) { > + if (fwd_streams[i]->rx_queue =3D=3D qi) > + break; > + } else { > + if (fwd_streams[i]->tx_queue =3D=3D qi) > + break; > + } > + } > + if (i =3D=3D nb_fwd_streams) > + return NUMA_NO_CONFIG; > + > + /* find the lcore based on stream idx */ > + for (j =3D 0; j < nb_lcores; j++) { > + if (fwd_lcores[j]->stream_idx =3D=3D i) > + break; > + } > + if (j =3D=3D nb_lcores) > + return NUMA_NO_CONFIG; > + > + /* find the socket for the lcore */ > + socket =3D rte_lcore_to_socket_id(fwd_lcores_cpuids[j]); > + > + return socket; > +} > + > int > start_port(portid_t pid) > { > @@ -1469,14 +1509,18 @@ static int eth_event_callback(portid_t port_id, > port->need_reconfig_queues =3D 0; > /* setup tx queues */ > for (qi =3D 0; qi < nb_txq; qi++) { > + int socket =3D port->socket_id; > if ((numa_support) && > (txring_numa[pi] !=3D NUMA_NO_CONFIG)) > - diag =3D rte_eth_tx_queue_setup(pi, qi, > - nb_txd,txring_numa[pi], > - &(port->tx_conf)); > - else > - diag =3D rte_eth_tx_queue_setup(pi, qi, > - nb_txd,port->socket_id, > + socket =3D txring_numa[pi]; > + else if (ring_bind_lcpu) { > + int ret =3D find_local_socket(qi, 0); > + if (ret !=3D NUMA_NO_CONFIG) > + socket =3D ret; > + } > + > + diag =3D rte_eth_tx_queue_setup(pi, qi, > + nb_txd, socket, > &(port->tx_conf)); >=20 > if (diag =3D=3D 0) > @@ -1495,35 +1539,28 @@ static int eth_event_callback(portid_t port_id, > } > /* setup rx queues */ > for (qi =3D 0; qi < nb_rxq; qi++) { > + int socket =3D port->socket_id; > if ((numa_support) && > - (rxring_numa[pi] !=3D NUMA_NO_CONFIG)) { > - struct rte_mempool * mp =3D > - mbuf_pool_find(rxring_numa[pi]); > - if (mp =3D=3D NULL) { > - printf("Failed to setup RX queue:" > - "No mempool allocation" > - " on the socket %d\n", > - rxring_numa[pi]); > - return -1; > - } > - > - diag =3D rte_eth_rx_queue_setup(pi, qi, > - nb_rxd,rxring_numa[pi], > - &(port->rx_conf),mp); > - } else { > - struct rte_mempool *mp =3D > - mbuf_pool_find(port->socket_id); > - if (mp =3D=3D NULL) { > - printf("Failed to setup RX queue:" > + (rxring_numa[pi] !=3D NUMA_NO_CONFIG)) > + socket =3D rxring_numa[pi]; > + else if (ring_bind_lcpu) { > + int ret =3D find_local_socket(qi, 1); > + if (ret !=3D NUMA_NO_CONFIG) > + socket =3D ret; > + } > + > + struct rte_mempool *mp =3D > + mbuf_pool_find(socket); > + if (mp =3D=3D NULL) { > + printf("Failed to setup RX queue:" > "No mempool allocation" > " on the socket %d\n", > - port->socket_id); > - return -1; > - } > - diag =3D rte_eth_rx_queue_setup(pi, qi, > - nb_rxd,port->socket_id, > - &(port->rx_conf), mp); > + socket); > + return -1; > } > + diag =3D rte_eth_rx_queue_setup(pi, qi, > + nb_rxd, socket, > + &(port->rx_conf), mp); > if (diag =3D=3D 0) > continue; >=20 > @@ -2414,6 +2451,19 @@ uint8_t port_is_bonding_slave(portid_t slave_pid) > "but nb_txq=3D%d will prevent to fully test it.\n", > nb_rxq, nb_txq); >=20 > + if (ring_bind_lcpu & RBL_BIND_LOCAL_MASK) { > + if (ring_bind_lcpu & > + (RBL_RING_NUMA_MASK | RBL_PORT_NUMA_MASK)) { > + printf("ring-bind-lcpu option is suppressed by " > + "ring-numa-config or port-numa-config option\n"); > + ring_bind_lcpu =3D 0; > + } else { > + printf("RingBuffer bind with local CPU selected\n"); > + ring_bind_lcpu =3D 1; > + } > + } else > + ring_bind_lcpu =3D 0; > + > init_config(); > if (start_port(RTE_PORT_ALL) !=3D 0) > rte_exit(EXIT_FAILURE, "Start ports failed\n"); > diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h > index 2a266fd..99e55b2 100644 > --- a/app/test-pmd/testpmd.h > +++ b/app/test-pmd/testpmd.h > @@ -328,6 +328,13 @@ struct queue_stats_mappings { > extern uint8_t interactive; > extern uint8_t auto_start; > extern uint8_t tx_first; > + > +/* for --ring-bind-lcpu option checking, RBL means Ring Bind Lcpu relate= d */ > +#define RBL_BIND_LOCAL_MASK (1 << 0) > +#define RBL_RING_NUMA_MASK (1 << 1) > +#define RBL_PORT_NUMA_MASK (1 << 2) > +extern uint8_t ring_bind_lcpu; > + > extern char cmdline_filename[PATH_MAX]; /**< offline commands file */ > extern uint8_t numa_support; /**< set by "--numa" parameter */ > extern uint16_t port_topology; /**< set by "--port-topology" parameter *= / > diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_a= pp_ug/run_app.rst > index 4c0d2ce..b88f099 100644 > --- a/doc/guides/testpmd_app_ug/run_app.rst > +++ b/doc/guides/testpmd_app_ug/run_app.rst > @@ -240,6 +240,12 @@ The commandline options are: > Specify the socket on which the TX/RX rings for the port will be all= ocated. > Where flag is 1 for RX, 2 for TX, and 3 for RX and TX. >=20 > +* ``--ring-bind-lcpu`` > + > + specify TX/RX rings will be allocated on local socket of lcpu. > + It will be ignored if ring-numa-config or port-numa-config is used. > + As a result, it allows one port binds to multiple NUMA nodes. > + > * ``--socket-num=3DN`` >=20 > Set the socket from which all memory is allocated in NUMA mode, > -- > 1.8.3.1