[dpdk-dev] App acting strangely when Vector rx / FDIR is on

[dpdk-dev] App acting strangely when Vector rx / FDIR is on

[Dor Green]

I have an app which processes packets, and for a while I had FDIR in
signature mode. Since I no longer needed filtering, I turned it off
(which also turned on vector rx) which caused some strange behaviour.

I changed the app so that it simply prints the length of the packet
for each incoming one, then sent 10,000 packets at a very slow speed
(10Mbps) with only one core.
Most noticable was that even though the NIC stats showed the same
packets received (rte_eth_stats.ipackets), only 9111 packet lengths
were printed when vector/fdir was off (and the entire 10,000 were
printed when it was on).
The data lengths themselves were also different, but I expected some
reordering. When looking at a set of all the lengths, there were many
lengths that only appeared when fdir was on, and only one that
appeared when it was on but not when it was off.

Can anyone think of any explanation for this phenomenon? Any checks I could do?

Using a 10G 2P X520 adapter. These are the startup logs (when fdir is
off. When it is on, the "Vector rx enabled" line is removed):

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Detected lcore 6 as core 0 on socket 0
EAL: Detected lcore 7 as core 1 on socket 0
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 3 on socket 0
EAL: Detected lcore 10 as core 4 on socket 0
EAL: Detected lcore 11 as core 5 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 12 lcore(s)
EAL: Setting up memory...
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0x7f5700000000 (size = 0x400000000)
EAL: Requesting 16 pages of size 1024MB from socket 0
EAL: TSC frequency is ~2094901 KHz
EAL: Master core 0 is ready (tid=a40368c0)
EAL: Core 7 is ready (tid=10033700)
EAL: Core 6 is ready (tid=10834700)
EAL: Core 5 is ready (tid=11035700)
EAL: Core 4 is ready (tid=11836700)
EAL: Core 3 is ready (tid=12037700)
EAL: Core 2 is ready (tid=12838700)
EAL: Core 1 is ready (tid=13039700)
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL:   probe driver: 8086:154d rte_ixgbe_pmd
EAL:   0000:04:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:04:00.1 on NUMA socket 0
EAL:   probe driver: 8086:154d rte_ixgbe_pmd
EAL:   PCI memory mapped at 0x7f5ba3e84000
EAL:   PCI memory mapped at 0x7f5ba4045000
PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 11, SFP+: 4
PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x154d
MAIN: Finished DPDK memory setup
PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f5700af0e40
hw_ring=0x7f59555fae00 dma_addr=0xe155fae00
PMD: ixgbe_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are
satisfied. Rx Burst Bulk Alloc function will be used on port=0,
queue=0.
PMD: ixgbe_dev_rx_queue_setup(): Vector rx enabled, please make sure
RX burst size no less than 32.
PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5700af0900
hw_ring=0x7f595560ae80 dma_addr=0xe1560ae80
PMD: set_tx_function(): Using full-featured tx code path
PMD: set_tx_function():  - txq_flags = 0 [IXGBE_SIMPLE_FLAGS=f01]
PMD: set_tx_function():  - tx_rs_thresh = 32 [RTE_PMD_IXGBE_TX_MAX_BURST=32]


My (shortened) configuration:

static struct rte_eth_conf const ethconf = {
    .link_speed = 0,
    .link_duplex = 0,
    .rxmode = {
        .mq_mode = ETH_MQ_RX_RSS,
        .max_rx_pkt_len = 2000,
        .split_hdr_size = 0,
        .header_split = 0,
        .hw_ip_checksum = 0,
        .hw_vlan_filter = 0,
        .jumbo_frame = 1,
        .hw_strip_crc = 0,   /**< CRC stripped by hardware */
    },

    .rx_adv_conf = {
        .rss_conf = {
            .rss_key = NULL,
            .rss_hf = ETH_RSS_IPV4 | ETH_RSS_IPV6,
        }
    },

    .fdir_conf = {
        .mode = RTE_FDIR_MODE_NONE, // Or RTE_FDIR_MODE_SIGNATURE,
    },
};

static struct rte_eth_rxconf const rxconf = {
    .rx_thresh = {
        .pthresh = 8,
        .hthresh = 8,
        .wthresh = 0,
    },

    .rx_free_thresh = 300,
    .rx_drop_en = 0,
};

Thanks.

Re: [dpdk-dev] App acting strangely when Vector rx / FDIR is on

[Dor Green]

"Fixed" the problem.

I upgraded to DPDK 2.0, which did not allow vector rx to until I have
changed my RX hardware queue size to a power of two.
After I did that (and changed the rx_free_thresh accordingly), it simply worked.

On Sun, Apr 5, 2015 at 11:45 AM, Dor Green <dorgreen1@gmail.com> wrote:
> I have an app which processes packets, and for a while I had FDIR in
> signature mode. Since I no longer needed filtering, I turned it off
> (which also turned on vector rx) which caused some strange behaviour.
>
> I changed the app so that it simply prints the length of the packet
> for each incoming one, then sent 10,000 packets at a very slow speed
> (10Mbps) with only one core.
> Most noticable was that even though the NIC stats showed the same
> packets received (rte_eth_stats.ipackets), only 9111 packet lengths
> were printed when vector/fdir was off (and the entire 10,000 were
> printed when it was on).
> The data lengths themselves were also different, but I expected some
> reordering. When looking at a set of all the lengths, there were many
> lengths that only appeared when fdir was on, and only one that
> appeared when it was on but not when it was off.
>
> Can anyone think of any explanation for this phenomenon? Any checks I could do?
>
> Using a 10G 2P X520 adapter. These are the startup logs (when fdir is
> off. When it is on, the "Vector rx enabled" line is removed):
>
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Detected lcore 4 as core 4 on socket 0
> EAL: Detected lcore 5 as core 5 on socket 0
> EAL: Detected lcore 6 as core 0 on socket 0
> EAL: Detected lcore 7 as core 1 on socket 0
> EAL: Detected lcore 8 as core 2 on socket 0
> EAL: Detected lcore 9 as core 3 on socket 0
> EAL: Detected lcore 10 as core 4 on socket 0
> EAL: Detected lcore 11 as core 5 on socket 0
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 12 lcore(s)
> EAL: Setting up memory...
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0x7f5700000000 (size = 0x400000000)
> EAL: Requesting 16 pages of size 1024MB from socket 0
> EAL: TSC frequency is ~2094901 KHz
> EAL: Master core 0 is ready (tid=a40368c0)
> EAL: Core 7 is ready (tid=10033700)
> EAL: Core 6 is ready (tid=10834700)
> EAL: Core 5 is ready (tid=11035700)
> EAL: Core 4 is ready (tid=11836700)
> EAL: Core 3 is ready (tid=12037700)
> EAL: Core 2 is ready (tid=12838700)
> EAL: Core 1 is ready (tid=13039700)
> EAL: PCI device 0000:04:00.0 on NUMA socket -1
> EAL:   probe driver: 8086:154d rte_ixgbe_pmd
> EAL:   0000:04:00.0 not managed by UIO driver, skipping
> EAL: PCI device 0000:04:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:154d rte_ixgbe_pmd
> EAL:   PCI memory mapped at 0x7f5ba3e84000
> EAL:   PCI memory mapped at 0x7f5ba4045000
> PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 11, SFP+: 4
> PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x154d
> MAIN: Finished DPDK memory setup
> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f5700af0e40
> hw_ring=0x7f59555fae00 dma_addr=0xe155fae00
> PMD: ixgbe_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are
> satisfied. Rx Burst Bulk Alloc function will be used on port=0,
> queue=0.
> PMD: ixgbe_dev_rx_queue_setup(): Vector rx enabled, please make sure
> RX burst size no less than 32.
> PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f5700af0900
> hw_ring=0x7f595560ae80 dma_addr=0xe1560ae80
> PMD: set_tx_function(): Using full-featured tx code path
> PMD: set_tx_function():  - txq_flags = 0 [IXGBE_SIMPLE_FLAGS=f01]
> PMD: set_tx_function():  - tx_rs_thresh = 32 [RTE_PMD_IXGBE_TX_MAX_BURST=32]
>
>
> My (shortened) configuration:
>
> static struct rte_eth_conf const ethconf = {
>     .link_speed = 0,
>     .link_duplex = 0,
>     .rxmode = {
>         .mq_mode = ETH_MQ_RX_RSS,
>         .max_rx_pkt_len = 2000,
>         .split_hdr_size = 0,
>         .header_split = 0,
>         .hw_ip_checksum = 0,
>         .hw_vlan_filter = 0,
>         .jumbo_frame = 1,
>         .hw_strip_crc = 0,   /**< CRC stripped by hardware */
>     },
>
>     .rx_adv_conf = {
>         .rss_conf = {
>             .rss_key = NULL,
>             .rss_hf = ETH_RSS_IPV4 | ETH_RSS_IPV6,
>         }
>     },
>
>     .fdir_conf = {
>         .mode = RTE_FDIR_MODE_NONE, // Or RTE_FDIR_MODE_SIGNATURE,
>     },
> };
>
> static struct rte_eth_rxconf const rxconf = {
>     .rx_thresh = {
>         .pthresh = 8,
>         .hthresh = 8,
>         .wthresh = 0,
>     },
>
>     .rx_free_thresh = 300,
>     .rx_drop_en = 0,
> };
>
> Thanks.