From: "Lu, Wenzhuo" <wenzhuo.lu@intel.com>
To: Luca Boccassi <lboccass@Brocade.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v6 0/4] support reset of VF link
Date: Wed, 6 Jul 2016 00:45:18 +0000 [thread overview]
Message-ID: <6A0DE07E22DDAD4C9103DF62FEBC09090348DB05@shsmsx102.ccr.corp.intel.com> (raw)
In-Reply-To: <1467712351.32466.11.camel@brocade.com>
[-- Attachment #1: Type: text/plain, Size: 5088 bytes --]
Hi Luca,
> -----Original Message-----
> From: Luca Boccassi [mailto:lboccass@Brocade.com]
> Sent: Tuesday, July 5, 2016 5:53 PM
> To: Lu, Wenzhuo
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v6 0/4] support reset of VF link
>
> On Tue, 2016-07-05 at 00:52 +0000, Lu, Wenzhuo wrote:
> > Hi Luca,
> >
> >
> > > -----Original Message-----
> > > From: Luca Boccassi [mailto:lboccass@Brocade.com]
> > > Sent: Monday, July 4, 2016 11:48 PM
> > > To: Lu, Wenzhuo
> > > Cc: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH v6 0/4] support reset of VF link
> > >
> > > On Mon, 2016-06-20 at 14:24 +0800, Wenzhuo Lu wrote:
> > > > If the PF link is down and up, VF link will not work accordingly.
> > > > This patch set addes the support of VF link reset. So, when VF
> > > > receices the messges of physical link down/up. APP can reset the
> > > > VF link and let it recover.
> > > >
> > > > PS: This patch set is splitted from a previous patch set,
> > > > *automatic link recovery on ixgbe/igb VF*, and it's base on the
> > > > patch set *support mailbox interruption on ixgbe/igb VF*.
> > > >
> > > > Wenzhuo Lu (3):
> > > > lib/librte_ether: support device reset
> > > > ixgbe: implement device reset on VF
> > > > igb: implement device reset on VF
> > > >
> > > > Zhe Tao (1):
> > > > i40e: implement device reset on VF
> > > >
> > > > v1:
> > > > - Added the implementation for the VF reset functionality.
> > > > v2:
> > > > - Changed the i40e related operations during VF reset.
> > > > v3:
> > > > - Resent the patches because of the mail sent issue.
> > > > v4:
> > > > - Removed some VF reset emulation code.
> > > > v5:
> > > > - Removed all the code related with lock.
> > > > v6:
> > > > - Updated the NIC feature overview matrix.
> > > > - Added more explanation in the doxygen comment of reset API.
> > > >
> > > > doc/guides/nics/overview.rst | 1 +
> > > > doc/guides/rel_notes/release_16_07.rst | 13 ++++++
> > > > drivers/net/e1000/igb_ethdev.c | 59 ++++++++++++++++++++++++
> > > > drivers/net/i40e/i40e_ethdev.h | 4 ++
> > > > drivers/net/i40e/i40e_ethdev_vf.c | 83
> > > ++++++++++++++++++++++++++++++++++
> > > > drivers/net/i40e/i40e_rxtx.c | 10 ++++
> > > > drivers/net/i40e/i40e_rxtx.h | 4 ++
> > > > drivers/net/ixgbe/ixgbe_ethdev.c | 64 +++++++++++++++++++++++++-
> > > > drivers/net/ixgbe/ixgbe_ethdev.h | 2 +-
> > > > drivers/net/ixgbe/ixgbe_rxtx.c | 12 +++--
> > > > lib/librte_ether/rte_ethdev.c | 17 +++++++
> > > > lib/librte_ether/rte_ethdev.h | 24 ++++++++++
> > > > lib/librte_ether/rte_ether_version.map | 7 +++
> > > > 13 files changed, 295 insertions(+), 5 deletions(-)
> > >
> > > Hello Wenzhuo,
> > >
> > > I'm testing this patchset, but I am sporadically running into an
> > > issue where the VFs reset fails after the PF flaps.
> > >
> > > I have a VM running on a KVM box with a X540-AT2, passing 2 VFs in.
> > >
> > > I am using calling rte_eth_dev_reset in response to a
> > > RTE_ETH_EVENT_INTR_RESET callback, and the following errors appear
> > > in the
> > > log:
> > >
> > > PMD: ixgbevf_dev_reset(): Ixgbe VF reset: Failed to update link.
> > > PMD: ixgbe_alloc_rx_queue_mbufs(): RX mbuf alloc failed queue_id=0
> > > PMD: ixgbevf_dev_start(): Unable to initialize RX hardware (-12)
> > > PMD: ixgbevf_dev_reset(): Ixgbe VF reset: Failed to start device.
> > >
> > > Jumping in with GDB, it seems that the rte_rxmbuf_alloc call in
> > > ixgbe_alloc_rx_queue_mbufs returns NULL at iteration 64 out of 2048.
> > > The application has ~500 2MB hugepages, and there's 2GB of free
> > > memory available on top of that.
> > >
> > > Have you seen this before? Any pointer or suggestion for debugging?
> > >
> > > Thanks!
> > >
> > > --
> > > Kind regards,
> > > Luca Boccassi
> > I think the problem is the mbuf occupied by the packets is not released. This
> memory has to be released by the APP, so my patches haven’t covered this.
> Actually an example is needed to show how to use the reset API. I plan to modify
> the testpmd.
> > You may notice this feature is postponed to 16.11. Would you like to wait for
> the new version that will include an example?
>
> Hi,
>
> Unfortunately we need the VF reset working sooner than that, so one way or
> the other I'll need to sort it out. Given I've got a use case where this is happening,
> if it can be helpful for you I'm more than happy to help as a guinea pig. If you
> could please give some guidance/guidelines with regards to which API to use to
> sort the mbuf problem, I can try it out and give back some feedback.
>
> Thanks!
I made a stupid mistake and deleted all my code. So, I have to take some time to rewrite it :(
Attached the example I used to test the reset API. It's modified from the l2fwd example. So you can compare it with l2fwd to see what need to be added.
Hopefully it can help :)
>
> --
> Kind regards,
> Luca Boccassi
[-- Attachment #2: main.c --]
[-- Type: text/plain, Size: 20990 bytes --]
/*-
* BSD LICENSE
*
* Copyright(c) 2010-2016 Intel Corporation. All rights reserved.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <inttypes.h>
#include <sys/types.h>
#include <sys/queue.h>
#include <netinet/in.h>
#include <setjmp.h>
#include <stdarg.h>
#include <ctype.h>
#include <errno.h>
#include <getopt.h>
#include <signal.h>
#include <stdbool.h>
#include <rte_common.h>
#include <rte_log.h>
#include <rte_malloc.h>
#include <rte_memory.h>
#include <rte_memcpy.h>
#include <rte_memzone.h>
#include <rte_eal.h>
#include <rte_per_lcore.h>
#include <rte_launch.h>
#include <rte_atomic.h>
#include <rte_cycles.h>
#include <rte_prefetch.h>
#include <rte_lcore.h>
#include <rte_per_lcore.h>
#include <rte_branch_prediction.h>
#include <rte_interrupts.h>
#include <rte_pci.h>
#include <rte_random.h>
#include <rte_debug.h>
#include <rte_ether.h>
#include <rte_ethdev.h>
#include <rte_ring.h>
#include <rte_mempool.h>
#include <rte_mbuf.h>
static volatile bool force_quit;
#define RTE_LOGTYPE_L2FWD RTE_LOGTYPE_USER1
#define NB_MBUF 8192
#define MAX_PKT_BURST 32
#define BURST_TX_DRAIN_US 100 /* TX drain every ~100us */
/*
* Configurable number of RX/TX ring descriptors
*/
#define RTE_TEST_RX_DESC_DEFAULT 128
#define RTE_TEST_TX_DESC_DEFAULT 512
static uint16_t nb_rxd = RTE_TEST_RX_DESC_DEFAULT;
static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT;
/* ethernet addresses of ports */
static struct ether_addr l2fwd_ports_eth_addr[RTE_MAX_ETHPORTS];
/* mask of enabled ports */
static uint32_t l2fwd_enabled_port_mask;
/* list of enabled ports */
static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
static unsigned int l2fwd_rx_queue_per_lcore = 1;
#define MAX_RX_QUEUE_PER_LCORE 16
#define MAX_TX_QUEUE_PER_PORT 16
struct lcore_queue_conf {
unsigned n_rx_port;
unsigned rx_port_list[MAX_RX_QUEUE_PER_LCORE];
} __rte_cache_aligned;
struct lcore_queue_conf lcore_queue_conf[RTE_MAX_LCORE];
static struct rte_eth_dev_tx_buffer *tx_buffer[RTE_MAX_ETHPORTS];
static const struct rte_eth_conf port_conf = {
.rxmode = {
.split_hdr_size = 0,
.header_split = 0, /**< Header Split disabled */
.hw_ip_checksum = 0, /**< IP checksum offload disabled */
.hw_vlan_filter = 0, /**< VLAN filtering disabled */
.jumbo_frame = 0, /**< Jumbo Frame Support disabled */
.hw_strip_crc = 0, /**< CRC stripped by hardware */
},
.txmode = {
.mq_mode = ETH_MQ_TX_NONE,
},
};
struct rte_mempool *l2fwd_pktmbuf_pool;
/* Per-port statistics struct */
struct l2fwd_port_statistics {
uint64_t tx;
uint64_t rx;
uint64_t dropped;
} __rte_cache_aligned;
struct l2fwd_port_statistics port_statistics[RTE_MAX_ETHPORTS];
/* A tsc-based timer responsible for triggering statistics printout */
#define TIMER_MILLISECOND 2000000ULL /* around 1ms at 2 Ghz */
#define MAX_TIMER_PERIOD 86400 /* 1 day max */
/* default period is 10 seconds */
static int64_t timer_period = 10 * TIMER_MILLISECOND * 1000;
static uint32_t stop_forwarding;
static uint8_t reset_port;
static rte_spinlock_t ports_rx_lock[RTE_MAX_ETHPORTS];
static rte_spinlock_t ports_tx_lock[RTE_MAX_ETHPORTS];
/* Print out statistics on packets dropped */
static void
print_stats(void)
{
uint64_t total_packets_dropped, total_packets_tx, total_packets_rx;
unsigned portid;
total_packets_dropped = 0;
total_packets_tx = 0;
total_packets_rx = 0;
const char clr[] = { 27, '[', '2', 'J', '\0' };
const char topLeft[] = { 27, '[', '1', ';', '1', 'H', '\0' };
/* Clear screen and move to top left */
printf("%s%s", clr, topLeft);
printf("\nPort statistics ====================================");
for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) {
/* skip disabled ports */
if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
continue;
printf("\nStatistics for port %u ------------------------------"
"\nPackets sent: %24"PRIu64
"\nPackets received: %20"PRIu64
"\nPackets dropped: %21"PRIu64,
portid,
port_statistics[portid].tx,
port_statistics[portid].rx,
port_statistics[portid].dropped);
total_packets_dropped += port_statistics[portid].dropped;
total_packets_tx += port_statistics[portid].tx;
total_packets_rx += port_statistics[portid].rx;
}
printf("\nAggregate statistics ==============================="
"\nTotal packets sent: %18"PRIu64
"\nTotal packets received: %14"PRIu64
"\nTotal packets dropped: %15"PRIu64,
total_packets_tx,
total_packets_rx,
total_packets_dropped);
printf("\n====================================================\n");
}
static void
l2fwd_simple_forward(struct rte_mbuf *m, unsigned portid)
{
struct ether_hdr *eth;
void *tmp;
unsigned dst_port;
int sent;
struct rte_eth_dev_tx_buffer *buffer;
dst_port = l2fwd_dst_ports[portid];
eth = rte_pktmbuf_mtod(m, struct ether_hdr *);
/* 02:00:00:00:00:xx */
tmp = ð->d_addr.addr_bytes[0];
*((uint64_t *)tmp) = 0x000000000002 + ((uint64_t)dst_port << 40);
/* src addr */
ether_addr_copy(&l2fwd_ports_eth_addr[dst_port], ð->s_addr);
buffer = tx_buffer[dst_port];
if (!rte_spinlock_trylock(&ports_tx_lock[dst_port])) {
rte_eth_tx_buffer_drop_callback
(buffer->pkts, buffer->length, 0);
} else {
sent = rte_eth_tx_buffer(dst_port, 0, buffer, m);
if (sent)
port_statistics[dst_port].tx += sent;
rte_spinlock_unlock(&ports_tx_lock[dst_port]);
}
}
/* main processing loop */
static void
l2fwd_main_loop(void)
{
struct rte_mbuf *pkts_burst[MAX_PKT_BURST];
struct rte_mbuf *m;
int sent;
unsigned lcore_id;
uint64_t prev_tsc, diff_tsc, cur_tsc, timer_tsc;
unsigned i, j, portid, nb_rx;
struct lcore_queue_conf *qconf;
const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /
US_PER_S * BURST_TX_DRAIN_US;
struct rte_eth_dev_tx_buffer *buffer;
prev_tsc = 0;
timer_tsc = 0;
lcore_id = rte_lcore_id();
qconf = &lcore_queue_conf[lcore_id];
if (qconf->n_rx_port == 0) {
RTE_LOG(INFO, L2FWD, "lcore %u has nothing to do\n", lcore_id);
return;
}
RTE_LOG(INFO, L2FWD, "entering main loop on lcore %u\n", lcore_id);
for (i = 0; i < qconf->n_rx_port; i++) {
portid = qconf->rx_port_list[i];
RTE_LOG(INFO, L2FWD, " -- lcoreid=%u portid=%u\n", lcore_id,
portid);
}
while (!force_quit) {
cur_tsc = rte_rdtsc();
/*
* TX burst queue drain
*/
diff_tsc = cur_tsc - prev_tsc;
if (unlikely(diff_tsc > drain_tsc)) {
for (i = 0; i < qconf->n_rx_port; i++) {
portid =
l2fwd_dst_ports[qconf->rx_port_list[i]];
buffer = tx_buffer[portid];
if (!rte_spinlock_trylock(&ports_tx_lock[portid])) {
rte_eth_tx_buffer_drop_callback
(buffer->pkts, buffer->length, 0);
continue;
}
sent = rte_eth_tx_buffer_flush(portid,
0,
buffer);
if (sent)
port_statistics[portid].tx += sent;
rte_spinlock_unlock(&ports_tx_lock[portid]);
}
/* if timer is enabled */
if (timer_period > 0) {
/* advance the timer */
timer_tsc += diff_tsc;
/* if timer has reached its timeout */
if (unlikely(timer_tsc >=
(uint64_t) timer_period)) {
/* do this only on master core */
if (lcore_id ==
rte_get_master_lcore()) {
print_stats();
/* reset the timer */
timer_tsc = 0;
}
}
}
prev_tsc = cur_tsc;
}
/*
* Read packet from RX queues
*/
for (i = 0; i < qconf->n_rx_port; i++) {
portid = qconf->rx_port_list[i];
if (!rte_spinlock_trylock(&ports_rx_lock[portid]))
continue;
nb_rx = rte_eth_rx_burst((uint8_t) portid, 0,
pkts_burst, MAX_PKT_BURST);
port_statistics[portid].rx += nb_rx;
rte_spinlock_unlock(&ports_rx_lock[portid]);
for (j = 0; j < nb_rx; j++) {
m = pkts_burst[j];
rte_prefetch0(rte_pktmbuf_mtod(m, void *));
l2fwd_simple_forward(m, portid);
}
}
}
}
static int
l2fwd_launch_one_lcore(__attribute__((unused)) void *dummy)
{
l2fwd_main_loop();
return 0;
}
/* display usage */
static void
l2fwd_usage(const char *prgname)
{
printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
" -p PORTMASK: hexadecimal bitmask of ports to configure\n"
" -q NQ: number of queue (=ports) per lcore (default is 1)\n"
" -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default, 86400 maximum)\n",
prgname);
}
static int
l2fwd_parse_portmask(const char *portmask)
{
char *end = NULL;
unsigned long pm;
/* parse hexadecimal string */
pm = strtoul(portmask, &end, 16);
if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
return -1;
if (pm == 0)
return -1;
return pm;
}
static unsigned int
l2fwd_parse_nqueue(const char *q_arg)
{
char *end = NULL;
unsigned long n;
/* parse hexadecimal string */
n = strtoul(q_arg, &end, 10);
if ((q_arg[0] == '\0') || (end == NULL) || (*end != '\0'))
return 0;
if (n == 0)
return 0;
if (n >= MAX_RX_QUEUE_PER_LCORE)
return 0;
return n;
}
static int
l2fwd_parse_timer_period(const char *q_arg)
{
char *end = NULL;
int n;
/* parse number string */
n = strtol(q_arg, &end, 10);
if ((q_arg[0] == '\0') || (end == NULL) || (*end != '\0'))
return -1;
if (n >= MAX_TIMER_PERIOD)
return -1;
return n;
}
/* Parse the argument given in the command line of the application */
static int
l2fwd_parse_args(int argc, char **argv)
{
int opt, ret;
char **argvopt;
int option_index;
char *prgname = argv[0];
static struct option lgopts[] = {
{NULL, 0, 0, 0}
};
argvopt = argv;
while ((opt = getopt_long(argc, argvopt, "p:q:T:",
lgopts, &option_index)) != EOF) {
switch (opt) {
/* portmask */
case 'p':
l2fwd_enabled_port_mask = l2fwd_parse_portmask(optarg);
if (l2fwd_enabled_port_mask == 0) {
printf("invalid portmask\n");
l2fwd_usage(prgname);
return -1;
}
break;
/* nqueue */
case 'q':
l2fwd_rx_queue_per_lcore = l2fwd_parse_nqueue(optarg);
if (l2fwd_rx_queue_per_lcore == 0) {
printf("invalid queue number\n");
l2fwd_usage(prgname);
return -1;
}
break;
/* timer period */
case 'T':
timer_period = l2fwd_parse_timer_period(optarg) *
1000 * TIMER_MILLISECOND;
if (timer_period < 0) {
printf("invalid timer period\n");
l2fwd_usage(prgname);
return -1;
}
break;
/* long options */
case 0:
l2fwd_usage(prgname);
return -1;
default:
l2fwd_usage(prgname);
return -1;
}
}
if (optind >= 0)
argv[optind-1] = prgname;
ret = optind-1;
optind = 0; /* reset getopt lib */
return ret;
}
/* Check the link status of all ports in up to 9s, and print them finally */
static void
check_all_ports_link_status(uint8_t port_num, uint32_t port_mask)
{
#define CHECK_INTERVAL 100 /* 100ms */
#define MAX_CHECK_TIME 90 /* 9s (90 * 100ms) in total */
uint8_t portid, count, all_ports_up, print_flag = 0;
struct rte_eth_link link;
printf("\nChecking link status");
fflush(stdout);
for (count = 0; count <= MAX_CHECK_TIME; count++) {
if (force_quit)
return;
all_ports_up = 1;
for (portid = 0; portid < port_num; portid++) {
if (force_quit)
return;
if ((port_mask & (1 << portid)) == 0)
continue;
memset(&link, 0, sizeof(link));
rte_eth_link_get_nowait(portid, &link);
/* print link status if flag set */
if (print_flag == 1) {
if (link.link_status)
printf("Port %d Link Up - speed %u "
"Mbps - %s\n", (uint8_t)portid,
(unsigned)link.link_speed,
(link.link_duplex == ETH_LINK_FULL_DUPLEX) ?
("full-duplex") : ("half-duplex\n"));
else
printf("Port %d Link Down\n",
(uint8_t)portid);
continue;
}
/* clear all_ports_up flag if any link down */
if (link.link_status == ETH_LINK_DOWN) {
all_ports_up = 0;
break;
}
}
/* after finally printing all link status, get out */
if (print_flag == 1)
break;
if (all_ports_up == 0) {
printf(".");
fflush(stdout);
rte_delay_ms(CHECK_INTERVAL);
}
/* set the print_flag if all ports up or timeout */
if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) {
print_flag = 1;
printf("done\n");
}
}
}
static void
signal_handler(int signum)
{
if (signum == SIGINT || signum == SIGTERM) {
printf("\n\nSignal %d received, preparing to exit...\n",
signum);
force_quit = true;
}
}
static void
reset_event_callback(uint8_t port_id, enum rte_eth_event_type type, void *param)
{
RTE_SET_USED(param);
printf("\n\nIn registered callback...\n");
printf("Event type: %s on port %d\n",
type == RTE_ETH_EVENT_INTR_RESET ? "RESET interrupt" :
"unknown event", port_id);
reset_port = port_id;
rte_compiler_barrier(); /* prevent compiler reordering */
stop_forwarding = 1;
}
int
main(int argc, char **argv)
{
struct lcore_queue_conf *qconf;
struct rte_eth_dev_info dev_info;
int ret;
uint8_t nb_ports;
uint8_t nb_ports_available;
uint8_t portid, last_port;
unsigned lcore_id, rx_lcore_id;
unsigned nb_ports_in_mask = 0;
/* init EAL */
ret = rte_eal_init(argc, argv);
if (ret < 0)
rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n");
argc -= ret;
argv += ret;
force_quit = false;
signal(SIGINT, signal_handler);
signal(SIGTERM, signal_handler);
/* parse application arguments (after the EAL ones) */
ret = l2fwd_parse_args(argc, argv);
if (ret < 0)
rte_exit(EXIT_FAILURE, "Invalid L2FWD arguments\n");
/* create the mbuf pool */
l2fwd_pktmbuf_pool = rte_pktmbuf_pool_create("mbuf_pool", NB_MBUF, 32,
0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
if (l2fwd_pktmbuf_pool == NULL)
rte_exit(EXIT_FAILURE, "Cannot init mbuf pool\n");
nb_ports = rte_eth_dev_count();
if (nb_ports == 0)
rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
if (nb_ports > RTE_MAX_ETHPORTS)
nb_ports = RTE_MAX_ETHPORTS;
/* reset l2fwd_dst_ports */
for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++)
l2fwd_dst_ports[portid] = 0;
last_port = 0;
/* init ports rx/tx lock */
for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) {
rte_spinlock_init(&ports_rx_lock[portid]);
rte_spinlock_init(&ports_tx_lock[portid]);
}
/*
* Each logical core is assigned a dedicated TX queue on each port.
*/
for (portid = 0; portid < nb_ports; portid++) {
/* skip ports that are not enabled */
if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
continue;
if (nb_ports_in_mask % 2) {
l2fwd_dst_ports[portid] = last_port;
l2fwd_dst_ports[last_port] = portid;
} else
last_port = portid;
nb_ports_in_mask++;
rte_eth_dev_info_get(portid, &dev_info);
}
if (nb_ports_in_mask % 2) {
printf("Notice: odd number of ports in portmask.\n");
l2fwd_dst_ports[last_port] = last_port;
}
rx_lcore_id = 1;
qconf = NULL;
/* Initialize the port/queue configuration of each logical core */
for (portid = 0; portid < nb_ports; portid++) {
/* skip ports that are not enabled */
if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
continue;
/* get the lcore_id for this port */
while (rte_lcore_is_enabled(rx_lcore_id) == 0 ||
lcore_queue_conf[rx_lcore_id].n_rx_port ==
l2fwd_rx_queue_per_lcore) {
rx_lcore_id++;
if (rx_lcore_id >= RTE_MAX_LCORE)
rte_exit(EXIT_FAILURE, "Not enough cores\n");
}
if (qconf != &lcore_queue_conf[rx_lcore_id])
/* Assigned a new logical core in the loop above. */
qconf = &lcore_queue_conf[rx_lcore_id];
qconf->rx_port_list[qconf->n_rx_port] = portid;
qconf->n_rx_port++;
printf("Lcore %u: RX port %u\n",
rx_lcore_id,
(unsigned) portid);
}
nb_ports_available = nb_ports;
/* Initialise each port */
for (portid = 0; portid < nb_ports; portid++) {
/* skip ports that are not enabled */
if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
printf("Skipping disabled port %u\n",
(unsigned) portid);
nb_ports_available--;
continue;
}
/* init port */
printf("Initializing port %u... ", (unsigned) portid);
fflush(stdout);
ret = rte_eth_dev_configure(portid, 1, 1, &port_conf);
if (ret < 0)
rte_exit(EXIT_FAILURE,
"Cannot configure device: err=%d, port=%u\n",
ret, (unsigned) portid);
/* register reset interrupt callback */
rte_eth_dev_callback_register(portid,
RTE_ETH_EVENT_INTR_RESET, reset_event_callback, NULL);
rte_eth_macaddr_get(portid, &l2fwd_ports_eth_addr[portid]);
/* init one RX queue */
fflush(stdout);
ret = rte_eth_rx_queue_setup(portid, 0, nb_rxd,
rte_eth_dev_socket_id(portid),
NULL,
l2fwd_pktmbuf_pool);
if (ret < 0)
rte_exit(EXIT_FAILURE,
"rte_eth_rx_queue_setup:err=%d, port=%u\n",
ret, (unsigned) portid);
/* init one TX queue on each port */
fflush(stdout);
ret = rte_eth_tx_queue_setup(portid, 0, nb_txd,
rte_eth_dev_socket_id(portid),
NULL);
if (ret < 0)
rte_exit(EXIT_FAILURE,
"rte_eth_tx_queue_setup:err=%d, port=%u\n",
ret, (unsigned) portid);
/* Initialize TX buffers */
tx_buffer[portid] = rte_zmalloc_socket("tx_buffer",
RTE_ETH_TX_BUFFER_SIZE(MAX_PKT_BURST), 0,
rte_eth_dev_socket_id(portid));
if (tx_buffer[portid] == NULL)
rte_exit(EXIT_FAILURE,
"Cannot allocate buffer for tx on port %u\n",
(unsigned) portid);
rte_eth_tx_buffer_init(tx_buffer[portid], MAX_PKT_BURST);
ret = rte_eth_tx_buffer_set_err_callback(tx_buffer[portid],
rte_eth_tx_buffer_count_callback,
&port_statistics[portid].dropped);
if (ret < 0)
rte_exit(EXIT_FAILURE, "Cannot set error callback for "
"tx buffer on port %u\n", (unsigned) portid);
/* Start device */
ret = rte_eth_dev_start(portid);
if (ret < 0)
rte_exit(EXIT_FAILURE,
"rte_eth_dev_start:err=%d, port=%u\n",
ret, (unsigned) portid);
printf("done:\n");
rte_eth_promiscuous_enable(portid);
printf("Port %u, MAC address: "
"%02X:%02X:%02X:%02X:%02X:%02X\n\n",
(unsigned) portid,
l2fwd_ports_eth_addr[portid].addr_bytes[0],
l2fwd_ports_eth_addr[portid].addr_bytes[1],
l2fwd_ports_eth_addr[portid].addr_bytes[2],
l2fwd_ports_eth_addr[portid].addr_bytes[3],
l2fwd_ports_eth_addr[portid].addr_bytes[4],
l2fwd_ports_eth_addr[portid].addr_bytes[5]);
/* initialize port stats */
memset(&port_statistics, 0, sizeof(port_statistics));
}
if (!nb_ports_available) {
rte_exit(EXIT_FAILURE,
"All available ports are disabled. Please set portmask.\n");
}
check_all_ports_link_status(nb_ports, l2fwd_enabled_port_mask);
ret = 0;
/* launch per-lcore init on every lcore */
rte_eal_mp_remote_launch(l2fwd_launch_one_lcore, NULL, SKIP_MASTER);
printf("\nwaiting..");
while (1) {
rte_delay_ms(1000);
printf("..");
if (stop_forwarding == 1) {
printf("\nreset port %u\n", reset_port);
rte_spinlock_lock(&ports_rx_lock[reset_port]);
rte_spinlock_lock(&ports_tx_lock[reset_port]);
rte_eth_dev_reset(reset_port);
rte_spinlock_unlock(&ports_rx_lock[reset_port]);
rte_spinlock_unlock(&ports_tx_lock[reset_port]);
stop_forwarding = 0;
}
if (force_quit)
break;
}
RTE_LCORE_FOREACH_SLAVE(lcore_id) {
if (rte_eal_wait_lcore(lcore_id) < 0) {
ret = -1;
break;
}
}
for (portid = 0; portid < nb_ports; portid++) {
if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
continue;
printf("Closing port %d...", portid);
rte_eth_dev_stop(portid);
rte_eth_dev_close(portid);
printf(" Done\n");
}
printf("Bye...\n");
return ret;
}
next prev parent reply other threads:[~2016-07-06 0:45 UTC|newest]
Thread overview: 72+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-06 5:40 [dpdk-dev] [PATCH 0/8] " Wenzhuo Lu
2016-06-06 5:40 ` [dpdk-dev] [PATCH 1/8] lib/librte_ether: support device reset Wenzhuo Lu
2016-06-06 5:40 ` [dpdk-dev] [PATCH 2/8] lib/librte_ether: defind RX/TX lock mode Wenzhuo Lu
2016-06-08 2:15 ` Stephen Hemminger
2016-06-08 7:34 ` Lu, Wenzhuo
2016-06-09 7:50 ` Olivier Matz
2016-06-12 5:25 ` Lu, Wenzhuo
2016-06-10 18:12 ` Stephen Hemminger
2016-06-12 5:27 ` Lu, Wenzhuo
2016-06-06 5:40 ` [dpdk-dev] [PATCH 3/8] ixgbe: RX/TX with lock on VF Wenzhuo Lu
2016-06-06 5:40 ` [dpdk-dev] [PATCH 4/8] ixgbe: implement device reset " Wenzhuo Lu
2016-06-06 5:40 ` [dpdk-dev] [PATCH 5/8] igb: RX/TX with lock " Wenzhuo Lu
2016-06-06 5:40 ` [dpdk-dev] [PATCH 6/8] igb: implement device reset " Wenzhuo Lu
2016-06-06 5:40 ` [dpdk-dev] [PATCH 7/8] i40e:RX/TX with lock " Wenzhuo Lu
2016-06-06 5:40 ` [dpdk-dev] [PATCH 8/8] i40e: implement device reset " Wenzhuo Lu
2016-06-15 3:03 ` [dpdk-dev] [PATCH v5 0/4] support reset of VF link Wenzhuo Lu
2016-06-15 3:03 ` [dpdk-dev] [PATCH v5 1/4] lib/librte_ether: support device reset Wenzhuo Lu
2016-06-16 15:31 ` Bruce Richardson
2016-06-16 15:36 ` Thomas Monjalon
2016-06-15 3:03 ` [dpdk-dev] [PATCH v5 2/4] ixgbe: implement device reset on VF Wenzhuo Lu
2016-06-15 3:03 ` [dpdk-dev] [PATCH v5 3/4] igb: " Wenzhuo Lu
2016-06-15 3:03 ` [dpdk-dev] [PATCH v5 4/4] i40e: " Wenzhuo Lu
2016-06-20 6:24 ` [dpdk-dev] [PATCH v6 0/4] support reset of VF link Wenzhuo Lu
2016-06-20 6:24 ` [dpdk-dev] [PATCH v6 1/4] lib/librte_ether: support device reset Wenzhuo Lu
2016-06-20 9:14 ` Jerin Jacob
2016-06-20 16:17 ` Stephen Hemminger
2016-06-21 3:51 ` Jerin Jacob
2016-06-21 6:14 ` Lu, Wenzhuo
2016-06-21 7:37 ` Jerin Jacob
2016-06-21 8:24 ` Lu, Wenzhuo
2016-06-21 8:55 ` Jerin Jacob
2016-06-21 9:26 ` Ananyev, Konstantin
2016-06-21 10:57 ` Jerin Jacob
2016-06-21 13:10 ` Ananyev, Konstantin
2016-06-21 13:30 ` Jerin Jacob
2016-06-21 14:03 ` Ananyev, Konstantin
2016-06-21 14:29 ` Jerin Jacob
2016-06-22 1:35 ` Lu, Wenzhuo
2016-06-22 2:37 ` Jerin Jacob
2016-06-22 3:32 ` Lu, Wenzhuo
2016-06-22 4:14 ` Jerin Jacob
2016-06-22 5:05 ` Lu, Wenzhuo
2016-06-22 6:10 ` Jerin Jacob
2016-06-22 6:42 ` Lu, Wenzhuo
2016-06-22 7:59 ` Jerin Jacob
2016-06-22 8:17 ` Thomas Monjalon
2016-06-22 8:25 ` Lu, Wenzhuo
2016-06-22 9:18 ` Thomas Monjalon
2016-06-22 11:06 ` Jerin Jacob
2016-06-23 0:45 ` Lu, Wenzhuo
2016-06-23 0:39 ` Lu, Wenzhuo
2016-06-21 0:51 ` Lu, Wenzhuo
2016-06-20 6:24 ` [dpdk-dev] [PATCH v6 2/4] ixgbe: implement device reset on VF Wenzhuo Lu
2016-06-20 6:24 ` [dpdk-dev] [PATCH v6 3/4] igb: " Wenzhuo Lu
2016-06-20 6:24 ` [dpdk-dev] [PATCH v6 4/4] i40e: " Wenzhuo Lu
2016-07-04 15:48 ` [dpdk-dev] [PATCH v6 0/4] support reset of VF link Luca Boccassi
2016-07-05 0:52 ` Lu, Wenzhuo
2016-07-05 9:52 ` Luca Boccassi
2016-07-06 0:45 ` Lu, Wenzhuo [this message]
2016-07-06 16:26 ` Luca Boccassi
[not found] ` <1467822182.32466.34.camel@brocade.com>
2016-07-07 1:09 ` Lu, Wenzhuo
2016-07-07 10:20 ` Luca Boccassi
2016-07-07 13:12 ` Lu, Wenzhuo
2016-07-07 16:19 ` Luca Boccassi
2016-07-08 0:14 ` Lu, Wenzhuo
2016-07-08 17:15 ` Luca Boccassi
2016-07-11 1:32 ` Lu, Wenzhuo
2016-07-11 12:02 ` Luca Boccassi
2016-07-11 15:43 ` Luca Boccassi
2016-07-12 1:19 ` Lu, Wenzhuo
2016-08-26 12:58 ` Luca Boccassi
2016-08-29 1:04 ` Lu, Wenzhuo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6A0DE07E22DDAD4C9103DF62FEBC09090348DB05@shsmsx102.ccr.corp.intel.com \
--to=wenzhuo.lu@intel.com \
--cc=dev@dpdk.org \
--cc=lboccass@Brocade.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).