From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) by dpdk.org (Postfix) with ESMTP id 363FB95A0 for ; Thu, 28 Jan 2016 01:57:34 +0100 (CET) Received: by mail-ig0-f172.google.com with SMTP id t15so2250686igr.0 for ; Wed, 27 Jan 2016 16:57:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=LgteHjYCd2WKt4nRMpIs62LH/IvESCm76BHi6hBbpO0=; b=0evZ+XTZojAUCUvHyl0TVhiFK0OcRCb8NuPy8N0AFf0zrY85se7x6ZJwJGr6eyubbQ fBO2rMWPMWhuabosSonqRTsiT2S9Zr2pCHZGsjGs0VsgpeTAsGVCi6p/qo7CAB6YUEmc Xvmsw522mOEp6/Hdem+vhc72zPRc4zcVrqujs0lLY+XwcIbvtENa6fGgYxUGp8VuLYV4 tKKsbvVdgkwMn+Uru16pMdYu54juLTmS5+48dYrfbvjgNahDLreY2bS3AKp3yGmezhmR 4v0BKVE0vXNXT66O6wSg5GgqWCAsxkw//FsriP1FJNtENknGf4Gwd+pPZpmiqCtcLOMf DzXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=LgteHjYCd2WKt4nRMpIs62LH/IvESCm76BHi6hBbpO0=; b=ISxixqcusDLwS4kTPJxs+gqMrRGDYIMF5QT3XfazdCjDuZREphReSrGdQqUNsgnTbF 8u1vKDmC33+fSFZwrzzYz8EHTGI/dP3w4CA9f292CXJkPUHTjBEdAitwVQTY67EPNzUu dqpMgRo5OYgFYoKH5sxLj5GBVfD+hNQ7GjSZmDiRDl+LfyQk0LOKRFrOzHt2LlxyIvHa FhrTYaqlz1aPhe+hDdM9sgY6FjLUyU8GlxsCJuIH57xLItXMg4kP6fcnUbxjgN4x3Hho g8nAdiaxvTsaNISrYRBuaKsljWAz7B046xzEEhccB6dDiRke7ZPfrHWAuF+O6MZ2MMPb uMfA== X-Gm-Message-State: AG10YOT1ZOQIFRUbMeX5BT7NFGh7EKJWP0DwEEoLmpXmuMfK/VBnrvk6zB4Qwyuhg9cjTbsycRyZx7SITLEvNg== MIME-Version: 1.0 X-Received: by 10.50.150.99 with SMTP id uh3mr132278igb.1.1453942653626; Wed, 27 Jan 2016 16:57:33 -0800 (PST) Received: by 10.79.120.66 with HTTP; Wed, 27 Jan 2016 16:57:33 -0800 (PST) In-Reply-To: References: <1451241098.2052.9.camel@guerby.net> <3FD3743F-667A-46AE-B144-8B5573E224F1@intel.com> <1451246999.2052.21.camel@guerby.net> <5D85DA4D-BB4A-4C5B-9EBD-90F609B0755F@intel.com> <1453878930.10396.78.camel@guerby.net> Date: Thu, 28 Jan 2016 09:57:33 +0900 Message-ID: From: Moon-Sang Lee To: "Zhang, Helin" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Errors Rx count increasing while pktgen doing nothing on Intel 82598EB 10G X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2016 00:57:34 -0000 Helin, I implemented my own sample application that is a kind of carrier grade NAT server. It works fine on 1G NIC (i.e. Intel Corporation 82576 Gigabit Network Connection (rev 01)) But, it does not receive packets on 10G NIC (i.e. Intel Corporation 82598EB 10-Gigabit AF Network Connection (rev 01)) as described in the previous email. According to my log messages, it seems that control register for RX DMA is not enabled. Here is some information about my environment. 1. HW & OS [mslee@centos7 ~]$ uname -a Linux centos7 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [mslee@centos7 ~]$ more /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz stepping : 5 microcode : 0x19 cpu MHz : 2262.000 cache size : 8192 KB physical id : 1 siblings : 8 core id : 0 cpu cores : 4 apicid : 16 initial apicid : 16 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm c onstant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 po pcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid bogomips : 4521.93 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ... 2. port ocnfigure parameter for rte_eth_dev_configure(): ret =3D rte_eth_dev_configure(port, NB_RXQ, NB_TXQ, &port_conf); where NB_RXQ=3D1, NB_TXQ=3D2, and struct rte_eth_conf port_conf =3D { .rxmode =3D { //.mq_mode =3D ETH_MQ_RX_RSS, .mq_mode =3D ETH_MQ_RX_NONE, // for 10G NIC .max_rx_pkt_len =3D ETHER_MAX_LEN, .split_hdr_size =3D 0, .header_split =3D 0, // Header Split disabled .hw_ip_checksum =3D 0, // IP checksum offload enabled .hw_vlan_filter =3D 0, // VLAN filtering disabled .jumbo_frame =3D 0, // Jumbo Frame Support disabled .hw_strip_crc =3D 0, // CRC stripped by hardware }, .rx_adv_conf =3D { .rss_conf =3D { .rss_key =3D NULL, .rss_hf =3D ETH_RSS_IP, }, }, .txmode =3D { .mq_mode =3D ETH_MQ_TX_NONE, }, }; 3. rx queue setup parameter ret =3D rte_eth_rx_queue_setup(port, RXQ_ID, NB_RXD, socket_id, NULL, pktmbuf_pool[socket_id]) where RXQ_ID =3D 0, NB_RXD =3D 128 4. config parameters in config/common_linuxapp # # Compile burst-oriented IXGBE PMD driver # CONFIG_RTE_LIBRTE_IXGBE_PMD=3Dy CONFIG_RTE_LIBRTE_IXGBE_DEBUG_INIT=3Dn CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX=3Dn CONFIG_RTE_LIBRTE_IXGBE_DEBUG_TX=3Dn CONFIG_RTE_LIBRTE_IXGBE_DEBUG_TX_FREE=3Dn CONFIG_RTE_LIBRTE_IXGBE_DEBUG_DRIVER=3Dn CONFIG_RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC=3Dn CONFIG_RTE_IXGBE_INC_VECTOR=3Dy CONFIG_RTE_IXGBE_RX_OLFLAGS_ENABLE=3Dy 5. where log message is printed dpdk-2.2.0/drivers/net/ixgbe/ixgbe_rxtx.c: /* Allocate buffers for descriptor rings */ if (ixgbe_alloc_rx_queue_mbufs(rxq) !=3D 0) { PMD_INIT_LOG(ERR, "Could not alloc mbuf for queue:%d", rx_queue_id); return -1; } rxdctl =3D IXGBE_READ_REG(hw, IXGBE_RXDCTL(rxq->reg_idx)); rxdctl |=3D IXGBE_RXDCTL_ENABLE; IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(rxq->reg_idx), rxdctl); /* Wait until RX Enable ready */ poll_ms =3D RTE_IXGBE_REGISTER_POLL_WAIT_10_MS; do { rte_delay_ms(1); rxdctl =3D IXGBE_READ_REG(hw, IXGBE_RXDCTL(rxq->reg_idx)); } while (--poll_ms && !(rxdctl & IXGBE_RXDCTL_ENABLE)); if (!poll_ms) PMD_INIT_LOG(ERR, "*Could not enable Rx Queue %d*", rx_queue_id); I'm going to update firmware of my NIC, but I'm not sure it helps. I appreciate any comment. On Wed, Jan 27, 2016 at 4:23 PM, Zhang, Helin wrote= : > Moon-Sang > > Were you using pktgen or else application? > Could you help to share with me the detailed steps of your reproducing > that issue? > We will find time on that soon later. Thanks! > > Regards, > Helin > > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Laurent GUERBY > Sent: Wednesday, January 27, 2016 3:16 PM > To: Moon-Sang Lee > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] Errors Rx count increasing while pktgen doing > nothing on Intel 82598EB 10G > > On Wed, 2016-01-27 at 15:50 +0900, Moon-Sang Lee wrote: > > > > > > Laurent, have you resolved this problem? > > I'm using the same NIC as yours (i.e. Intel 82598EB 10G NIC) and faced > > the same problem as you. > > Here is parts of my log and it says that PMD cannot enable RX queue > > for my NIC. > > I'm using DPDK 2.2.0 and used 'null' for the 4th parameter in calling > > rte_eth_rx_queue_setup(). > > (i.e. 'null' parameter provides the default rx_conf value.) > > Hi, > > I had to reuse my DPDK machines for another task, I will go back to it > after FOSDEM. > > The error you get is the same as mine. > > Sincerely, > > Laurent > > > > > Thanks. > > > > > > > > > > > > APP: initialising port 0 ... > > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=3D0x7f5f27258040 > > sw_sc_ring=3D0x7f5f27257b00 hw_ring=3D0x7f5f27258580 dma_addr=3D0x41f45= 8580 > > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=3D0x7f5f27245940 > > hw_ring=3D0x7f5f27247980 dma_addr=3D0x41f447980 > > PMD: ixgbe_set_tx_function(): Using simple tx code path > > PMD: ixgbe_set_tx_function(): Vector tx enabled. > > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=3D0x7f5f272337c0 > > hw_ring=3D0x7f5f27235800 dma_addr=3D0x41f435800 > > PMD: ixgbe_set_tx_function(): Using simple tx code path > > PMD: ixgbe_set_tx_function(): Vector tx enabled. > > PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX > > burst size no less than 4 (port=3D0). > > PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0 > > APP: port 0 has started > > APP: port 0 has entered in promiscuous mode > > APP: port 0 initialization is done. > > KNI: pci: 09:00:00 8086:10c7 > > APP: kni allocation is done for port 0. > > APP: initialising port 1 ... > > PMD: ixgbe_dev_rx_queue_setup(): sw_ring=3D0x7f5f27222dc0 > > sw_sc_ring=3D0x7f5f27222880 hw_ring=3D0x7f5f27223300 dma_addr=3D0x41f42= 3300 > > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=3D0x7f5f272106c0 > > hw_ring=3D0x7f5f27212700 dma_addr=3D0x41f412700 > > PMD: ixgbe_set_tx_function(): Using simple tx code path > > PMD: ixgbe_set_tx_function(): Vector tx enabled. > > PMD: ixgbe_dev_tx_queue_setup(): sw_ring=3D0x7f5f271fe540 > > hw_ring=3D0x7f5f27200580 dma_addr=3D0x41f400580 > > PMD: ixgbe_set_tx_function(): Using simple tx code path > > PMD: ixgbe_set_tx_function(): Vector tx enabled. > > PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX > > burst size no less than 4 (port=3D1). > > PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue 0 > > APP: port 1 has started > > APP: port 1 has entered in promiscuous mode > > APP: port 1 initialization is done. > > KNI: pci: 0a:00:00 8086:10c7 > > APP: kni allocation is done for port 1. > > > > > > checking link status > > .done > > Port 0 Link Up - speed 10000 Mbps - full-duplex Port 1 Link Up - speed > > 10000 Mbps - full-duplex > > > > > > > > On Mon, Dec 28, 2015 at 5:28 AM, Wiles, Keith > > wrote: > > On 12/27/15, 2:09 PM, "Laurent GUERBY" > > wrote: > > > > >On Sun, 2015-12-27 at 19:43 +0000, Wiles, Keith wrote: > > >> On 12/27/15, 12:31 PM, "dev on behalf of Laurent GUERBY" > > wrote: > > >> > > >> >Hi, > > >> > > > >> >I reported today an issue when using Pktgen-DPDK: > > >> >https://github.com/pktgen/Pktgen-DPDK/issues/52 > > >> > > > >> >But I think it's more in DPDK than pktgen > > >> > > > >> >two identical machines with SFP+ DA cable between them > > >> >DPDK 2.2.0 from tarball > > >> >Pktgen-DPDK from git > > >> >two identical machines: > > >> >core i7 2600 (sandy bridge 4C/8T), HT disabled in the BIOS > > >> >ASUS P8H67-M PRO BIOS 3904 (latest available) > > >> >Ethernet controller: Intel Corporation 82598EB 10-Gigabit > > AF Dual Port > > >> >Network Connection (rev 01) > > >> >01:00.0 0200: 8086:10f1 (rev 01) > > >> > Subsystem: 8086:a21f > > >> >boot kernel 3.16 unbutu 14.04 with isolcpus=3D2,3,4 > > >> > > > >> >When launching pktgen even with no TX asked the Errors RX > > counters keeps > > >> >going up by about 7.4 millions per second: > > >> > > > >> >Errors Rx/Tx : 7471857054/0 > > >> > > > >> >In the log I get "Could not enable Rx Queue", might be the > > >> >source of the issue? > > >> > > > >> >PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue > > 0 > > >> >PMD: ixgbe_dev_rx_queue_start(): Could not enable Rx Queue > > 1 > > >> > > > >> >When sending traffic single UDP src/dst/IP/MAC the setup > > >> >reaches 14204188 pps 64 bytes, the error counter is also > > >> >increasing. > > >> > > > >> >Any idea what to look for? > > >> > > >> One more suggestion is to run test_pmd on one machine and > > something > > >> like iperf on the other to verify the DPDK is working > > correct, which I > > >> assume will be true. Not sure the RX errors are reported in > > the > > >> test_pmd or you could use the l3fwd application too. > > > > > >Ok, I will check the test_pmd documentation and try to do > > this test: I'm > > >just starting on DPDK :). > > > > > >> Please also send me the 'lspci | grep Ethernet=E2=80=99 outp= ut. > > > > > >I included one line in my original email above (plus extract > > of lspci > > >-vn), here is the full output of the command: > > > > > >01:00.0 Ethernet controller: Intel Corporation 82598EB > > 10-Gigabit AF > > >Dual Port Network Connection (rev 01) > > >01:00.1 Ethernet controller: Intel Corporation 82598EB > > 10-Gigabit AF > > >Dual Port Network Connection (rev 01) > > >05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. > > >RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller > > (rev 06) > > > > > >(The realtek is used only for internet connectivity). > > > > > >> Also send me the command line. > > > > > >On the first machine t1 : > > >root@t1:~/pktgen-dpdk# > ./app/app/x86_64-native-linuxapp-gcc/pktgen \ > > >-c e -n 1 --proc-type auto -- -m '[2:3].1' -P -f t1-t3.pkt -N > > > > > >And on the other machine t3: > > >root@t3:~/pktgen-dpdk# > ./app/app/x86_64-native-linuxapp-gcc/pktgen \ > > >-c e -n 1 --proc-type auto -- -m '[2:3].1' -P -f t3-t1.pkt -N > > > > > > You need to always start with port numbering for Pktgen with > > zero. > > > > Change the [2:3].1 to [2:3].0 the reason is you removed one of > > the two ports and Pktgen starts port counting from zero for > > the first available port :-) > > > > Sorry, I did not spot that sooner. > > > > > >The two "-f" pkt files are attached to this email, I do > > "start 1" > > >manually at the pktgen prompt. > > > > > >Thanks for your time, > > > > > >Sincerely, > > > > > >Laurent > > > > > > > > >> > > > >> >Laurent > > >> > > > >> > > > >> > > >> > > >> Regards, > > >> Keith > > >> > > >> > > >> > > >> > > > > > > > > > > > > Regards, > > Keith > > > > > > > > > > > > > > > > > > -- > > Moon-Sang Lee, SW Engineer > > Email: sang0627@gmail.com > > Wisdom begins in wonder. *Socrates* > > > --=20 Moon-Sang Lee, SW Engineer Email: sang0627@gmail.com Wisdom begins in wonder. *Socrates*