From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 09AE241E25; Thu, 9 Mar 2023 08:43:29 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9131440ED7; Thu, 9 Mar 2023 08:43:28 +0100 (CET) Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by mails.dpdk.org (Postfix) with ESMTP id 6EF58400D7 for ; Thu, 9 Mar 2023 08:43:27 +0100 (CET) Received: by mail-pg1-f170.google.com with SMTP id s17so526579pgv.4 for ; Wed, 08 Mar 2023 23:43:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678347806; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0F5paxyuMvh7ZuYqyWoIlj08tnNtKDvuBFOjeC9UKX0=; b=NFxYSF0nNSP3vVuZl5MeME7Kv3nVkeTgEUf3aTBDqNGzmgmH2R1U5hH9UAa463UZKE NiYRguueL8RPKV/24rPbiZGahhCBI3JKs+Nb0T7xXPfJmDKwwblNE2cd+1FcmDd1KKzC aKgL/2V4OLz0tSkIo3oGhJnFhLw2/sHVZ1zNZ5Omw8/vag+84lYvZvbBta/8DNPrJN48 iqPgQH8U4M07ebOReU8F7dtP1b8Vanj/voTiL4y6UUxkmAHYLs5wFeOdckD+gfYsqgZq Dd9ygXC+kIz1nBtilyNln8b9kD+hW0dFxc1qpBhgxgMytIoqe4cDAZEdXRl0Td6KPOg/ TRgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678347806; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0F5paxyuMvh7ZuYqyWoIlj08tnNtKDvuBFOjeC9UKX0=; b=Eb2aj5HWZdiDEH9HPyC44ch3YcvoM0/hZ4tBSMxgkUEp8s9b8w2J53UN+9w3vOynyV hHHcVAoLrsPULQx7jaqhoUQHyzFo/x/6JVZxE+1rYH9mT2tUYXu6l6YeoRw/BOXjgOcd uTs/Ex6Qh0rrPROLUk0ebD9DMgUnfkztpYGegvL5ClMlw8WqozKGDmY+FhvpeR5WgbV7 KRl7LKGiExbMn4Zrf6M/14D8ke3GyXN1l47g/GoZmNlsIr7e5qdeYYXxFK58CBi12jg9 HwoI+PSPkopxTipjE4Qo8gX10lXRIs+B1rCjlEMu4RjCkUTMAdvvC6xw+HaCfew5BwOR ROEw== X-Gm-Message-State: AO0yUKVaB0DIZrvShmWVCCIBPlVwycqC6Wc0IUBioaUuFs1Mb36CVkCS liyofVAqoxXiDr01vWyK1oWxOHK5+O/PwAEHxyUpT+CKjVM= X-Google-Smtp-Source: AK7set9u7zra7Fl2rF3oZU0Eu7ZRjBX8q/WAw82Bwd6OtSuiDfUu8oZRB78kJhe33/ntaHIQNePijfMMFIPNL+1DjO0= X-Received: by 2002:a62:ce43:0:b0:593:dc61:2161 with SMTP id y64-20020a62ce43000000b00593dc612161mr8567123pfg.2.1678347806443; Wed, 08 Mar 2023 23:43:26 -0800 (PST) MIME-Version: 1.0 References: <20230308094404.33ed0adf@hermes.local> In-Reply-To: <20230308094404.33ed0adf@hermes.local> From: Rajasekhar Pulluru Date: Thu, 9 Mar 2023 13:13:14 +0530 Message-ID: Subject: Re: ixgbe rxq interrupt not working To: Stephen Hemminger Cc: Honnappa Nagarahalli , "dev@dpdk.org" , nd Content-Type: multipart/alternative; boundary="000000000000d00f6d05f672cc64" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000d00f6d05f672cc64 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Stephen, ixgbe interfaces works correctly when not used with dpdk. # cat /proc/interrupts | grep enp2s0f1 109: 436 0 0 0 IR-PCI-MSI 1050624-edge enp2s0f1-TxRx-0 110: 0 167 0 0 IR-PCI-MSI 1050625-edge enp2s0f1-TxRx-1 111: 0 0 306 0 IR-PCI-MSI 1050626-edge enp2s0f1-TxRx-2 112: 0 0 0 173 IR-PCI-MSI 1050627-edge enp2s0f1-TxRx-3 113: 0 0 1 0 IR-PCI-MSI 1050628-edge enp2s0f1 dmesg doesn't seem to offer any clue. # dmesg | grep ixgbe [ 7.680989] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver [ 7.680996] ixgbe: Copyright (c) 1999-2016 Intel Corporation. [ 8.318606] ixgbe 0000:02:00.0: Multiqueue Enabled: Rx Queue count =3D 4= , Tx Queue count =3D 4 XDP Queue count =3D 0 [ 8.477642] ixgbe 0000:02:00.0: MAC: 6, PHY: 27, PBA No: 000700-000 [ 8.477654] ixgbe 0000:02:00.0: 00:e0:ed:db:33:69 [ 8.539077] ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection [ 8.888011] ixgbe 0000:02:00.1: Multiqueue Enabled: Rx Queue count =3D 4= , Tx Queue count =3D 4 XDP Queue count =3D 0 [ 9.019755] ixgbe 0000:02:00.1: MAC: 6, PHY: 27, PBA No: 000700-000 [ 9.019760] ixgbe 0000:02:00.1: 00:e0:ed:db:33:68 [ 9.063320] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection Thanks & Regards, Rajasekhar On Wed, Mar 8, 2023 at 11:14=E2=80=AFPM Stephen Hemminger < stephen@networkplumber.org> wrote: > On Wed, 8 Mar 2023 22:54:12 +0530 > Rajasekhar Pulluru wrote: > > > No Honnappa. > > > > Thanks & Regards, > > Rajasekhar > > > > On Wed, Mar 8, 2023 at 5:49=E2=80=AFAM Honnappa Nagarahalli < > > Honnappa.Nagarahalli@arm.com> wrote: > > > > > > > > > > > From: Rajasekhar Pulluru > > > Sent: Tuesday, March 7, 2023 12:52 PM > > > To: dev@dpdk.org > > > Subject: ixgbe rxq interrupt not working > > > > > > Hi Team, > > > > > > Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of them > driven > > > by igb and the rest of the 4 ports driven by ixgbe. > > > [Honnappa] Do you have packets crossing between the 2 drivers? > > > > > > > > > I am following the below sequence to initialize these ports: > > > > > > dev_conf.intr_conf.lsc =3D 1; //Enable link state change interrupt > > > dev_conf.intr_conf.rxq =3D 1; //Enable RX Queue Interrupt > > > dev_conf.rxmode.mq_mode =3D RTE_ETH_MQ_RX_NONE; > > > dev_conf.rxmode.offloads =3D 0; > > > dev_conf.txmode.mq_mode =3D RTE_ETH_MQ_TX_NONE; > > > dev_conf.txmode.offloads =3D 0; > > > > > > rte_eth_dev_configure > > > rte_eth_rx_queue_setup > > > rte_eth_tx_queue_setup > > > rte_eth_dev_start > > > data =3D port_id << CHAR_BIT | queue_id; > > > rte_eth_dev_rx_intr_ctl_q(port_id, queue_id, RTE_EPOLL_PER_THREAD, > > > RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data)); > > > rte_eth_dev_rx_intr_enable(port_id, queue_id); > > > > > > And then main loop repeats the below: > > > > > > rte_epoll_wait(RTE_EPOLL_PER_THREAD, event, 1, timeout /* 200micro-se= c > > > */); /* ignore return value */ > > > rte_eth_dev_rx_intr_disable(port_id, queue_id); > > > rte_eth_rx_burst(port_id, queue_id, pkts, num_pkts); > > > rte_eth_dev_rx_intr_enable(port_id, queue_id); > > > > > > The code is same for all the ports, igb ports are able to come-up and > rx > > > packets, where-as the ixgbe ports are not able to rx packets at all. > > > cat /proc/interrupts dumps vfio-msix counters for ixgbe as 0, where-a= s > > > it's non-zero for igb. > > > If I don't use/enable rxq interrupt for ixgbe (and remove epoll wait, > > > interrupt enable/disable from while loop) and simply poll for > > > rte_eth_rx_burst in a loop, ixgbe ports are able to rx packets. > > > > > > What could be wrong here? Appreciate any help. > > > > > > I would also like to know if there's an asynchronous rxq interrupt > > > notification to the application instead of rte_epoll_wait (and sleep)= . > > > > > > Thanks & Regards, > > > Rajasekhar > > > > > Does the device work as expected when not used with DPDK? > I.e does the kernel driver handle it correctly. > > > Also check the kernel dmesg log, for any relevant info. > There maybe VFIO or other overlap involved. > --000000000000d00f6d05f672cc64 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Stephen,

ixgbe interfaces work= s correctly when not used with dpdk.=C2=A0

# cat /proc/interru= pts | grep enp2s0f1
109: =C2=A0 =C2=A0 =C2=A0 =C2=A0436 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A00 =C2=A0IR-PCI-MSI 1050624-edge =C2=A0 =C2=A0 =C2=A0enp2s0= f1-TxRx-0
110: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 = =C2=A0167 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A00 =C2=A0IR-PCI-MSI 1050625-edge =C2=A0 =C2=A0 =C2=A0enp2s0f1-TxRx-1111: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0306 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2= =A0IR-PCI-MSI 1050626-edge =C2=A0 =C2=A0 =C2=A0enp2s0f1-TxRx-2
112: =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0173 =C2=A0IR-PCI-M= SI 1050627-edge =C2=A0 =C2=A0 =C2=A0enp2s0f1-TxRx-3
113: =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A01 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0IR-PCI-MSI 105= 0628-edge =C2=A0 =C2=A0 =C2=A0enp2s0f1

dmesg d= oesn't seem to offer any clue.

# dmesg | grep = ixgbe
[ =C2=A0 =C2=A07.680989] ixgbe: Intel(R) 10 Gigabit PCI Express Ne= twork Driver
[ =C2=A0 =C2=A07.680996] ixgbe: Copyright (c) 1999-2016 Int= el Corporation.
[ =C2=A0 =C2=A08.318606] ixgbe 0000:02:00.0: Multiqueue = Enabled: Rx Queue count =3D 4, Tx Queue count =3D 4 XDP Queue count =3D 0[ =C2=A0 =C2=A08.477642] ixgbe 0000:02:00.0: MAC: 6, PHY: 27, PBA No: 000= 700-000
[ =C2=A0 =C2=A08.477654] ixgbe 0000:02:00.0: 00:e0:ed:db:33:69[ =C2=A0 =C2=A08.539077] ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network = Connection
[ =C2=A0 =C2=A08.888011] ixgbe 0000:02:00.1: Multiqueue Enabl= ed: Rx Queue count =3D 4, Tx Queue count =3D 4 XDP Queue count =3D 0
[ = =C2=A0 =C2=A09.019755] ixgbe 0000:02:00.1: MAC: 6, PHY: 27, PBA No: 000700-= 000
[ =C2=A0 =C2=A09.019760] ixgbe 0000:02:00.1: 00:e0:ed:db:33:68
[ = =C2=A0 =C2=A09.063320] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Conn= ection

Thanks & Regards,
Rajasek= har

On Wed, Mar 8, 2023 at 11:14=E2=80=AFPM Stephen Hemminger &= lt;stephen@networkplumber.org= > wrote:
= On Wed, 8 Mar 2023 22:54:12 +0530
Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:

> No Honnappa.
>
> Thanks & Regards,
> Rajasekhar
>
> On Wed, Mar 8, 2023 at 5:49=E2=80=AFAM Honnappa Nagarahalli <
> Honn= appa.Nagarahalli@arm.com> wrote:=C2=A0
>
> >
> >
> > From: Rajasekhar Pulluru <pullururajasekhar@gmail.com>
> > Sent: Tuesday, March 7, 2023 12:52 PM
> > To: dev@dpdk.or= g
> > Subject: ixgbe rxq interrupt not working
> >
> > Hi Team,
> >
> > Bringing-up dpdk-22.07 on an intel machine with 8 ports, 4 of the= m driven
> > by igb and the rest of the 4 ports driven by ixgbe.
> > [Honnappa]=C2=A0 Do you have packets crossing between the 2 drive= rs?
> >
> >
> > I am following the below sequence to initialize these ports:
> >
> > dev_conf.intr_conf.lsc =3D 1; //Enable link state change interrup= t
> > dev_conf.intr_conf.rxq =3D 1; //Enable RX Queue Interrupt
> > dev_conf.rxmode.mq_mode =3D RTE_ETH_MQ_RX_NONE;
> > dev_conf.rxmode.offloads =3D 0;
> > dev_conf.txmode.mq_mode =3D RTE_ETH_MQ_TX_NONE;
> > dev_conf.txmode.offloads =3D 0;
> >
> > rte_eth_dev_configure
> > rte_eth_rx_queue_setup
> > rte_eth_tx_queue_setup
> > rte_eth_dev_start
> > data =3D port_id << CHAR_BIT | queue_id;
> > rte_eth_dev_rx_intr_ctl_q(port_id, queue_id, RTE_EPOLL_PER_THREAD= ,
> > RTE_INTR_EVENT_ADD, (void *)((uintptr_t)data));
> > rte_eth_dev_rx_intr_enable(port_id, queue_id);
> >
> > And then main loop repeats the below:
> >
> > rte_epoll_wait(RTE_EPOLL_PER_THREAD, event, 1, timeout /* 200micr= o-sec
> > */); /* ignore return value */
> > rte_eth_dev_rx_intr_disable(port_id, queue_id);
> > rte_eth_rx_burst(port_id, queue_id, pkts, num_pkts);
> > rte_eth_dev_rx_intr_enable(port_id, queue_id);
> >
> > The code is same for all the ports, igb ports are able to come-up= and rx
> > packets, where-as the ixgbe ports are not able to rx packets at a= ll.
> > cat /proc/interrupts dumps vfio-msix counters for ixgbe as 0, whe= re-as
> > it's non-zero for igb.
> > If I don't use/enable rxq interrupt for ixgbe (and remove epo= ll wait,
> > interrupt enable/disable from while loop) and simply poll for
> > rte_eth_rx_burst in a loop, ixgbe ports are able to rx packets. > >
> > What could be wrong here? Appreciate any help.
> >
> > I would also like to know if there's an asynchronous rxq inte= rrupt
> > notification to the application instead of rte_epoll_wait (and sl= eep).
> >
> > Thanks & Regards,
> > Rajasekhar
> >=C2=A0

Does the device work as expected when not used with DPDK?
I.e does the kernel driver handle it correctly.


Also check the kernel dmesg log, for any relevant info.
There maybe VFIO or other overlap involved.
--000000000000d00f6d05f672cc64--