From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 83122429D2 for ; Mon, 24 Apr 2023 10:35:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E757E410EC; Mon, 24 Apr 2023 10:35:45 +0200 (CEST) Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by mails.dpdk.org (Postfix) with ESMTP id 62854410DE for ; Mon, 24 Apr 2023 10:35:44 +0200 (CEST) Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-63b73203e0aso25945874b3a.1 for ; Mon, 24 Apr 2023 01:35:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682325343; x=1684917343; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=wHxjOyxrALjc0tjs9IED4/WXl9mU2eQGCsqU6TNx81I=; b=AUQs3gj0tRN1nQw2bH6zDboMoCczPYQ+kMduavrAjgCkdV21OAs4T/FwVKEU7qMKXF YFe5MOCuzJALRIJLq4OYVw9W4GSsTSjMJalY9HcwPkUnY8GRAmbNhK7/WpNhNpHlDo1d ETWd5wIUQLA3QAruaY3FCkU6jIMEutoZI5WOGoEtTbU248QhpX3226KUN6UmDHg8tfmK p07uH2cnE2EsO8dUIk+dn2qZxhJe6sDpzLziakaEuUyxriJ4T0/kHjDjwyRDPpoXnJnK Xu6tYd/sCz+gkPGG9tfqaoTGIoyNef9K/Wo1lljHmQeAhvCHxjdevmmMQ/MyYXGrFiFB ab+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682325343; x=1684917343; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wHxjOyxrALjc0tjs9IED4/WXl9mU2eQGCsqU6TNx81I=; b=bLyBza0KIHVd2+3zvlRJUx4JZbgAghjEI6kA2RpzsAblV1gJIum9q7ZgnuKU9DM4Ag JuOGwkPvTZHYpd2qdIrpManW8BRoEJiwVQiJJBhduhcsp7T6t28SfO5jyHQhe/FZ53CD F5tHrDiBG4KIC45dMD720179Giytk1C7FEl5pXdmGxbvA3X1m8RfUksNOqXBL4FcpupA wMpqAVaCcTWbT8VgbImS2kdSQtJlcU6+W1U6RpU3tiH7rM9XXEcvWgQLHX+zdO7dEuG1 x6E68KmXBR/dFDbfLxY2q67ayV0JC/XpuXE0qlzlRA48O9O8eOqy6t7cSOTB/yxuPTFy uinQ== X-Gm-Message-State: AAQBX9dZC6zygNveEx0f2/vyrww3eug3gRb0K3KkL80N3Ggw4cd1oi0X t3LCn7LDoIfcrsPszZUSLuFD4f+oV+Q75JSMgak7ZGvKPtc= X-Google-Smtp-Source: AKy350bjE23qppwhjyePqy3+7llba0Fa5ztUH7RomQNBFxY0wEEosSZRYg/SDBMwEnjLQZhjmnMM4ObwpqH9NwJtlu0= X-Received: by 2002:a17:902:e5c5:b0:1a6:48e6:ea8e with SMTP id u5-20020a170902e5c500b001a648e6ea8emr21329691plf.4.1682325343298; Mon, 24 Apr 2023 01:35:43 -0700 (PDT) MIME-Version: 1.0 References: <20230421140608.35327af3@hermes.local> In-Reply-To: From: Igor de Paula Date: Mon, 24 Apr 2023 09:37:48 +0100 Message-ID: Subject: Re: Issue with rte_eth_dev_start To: Verghis Koshi Cc: stephen@networkplumber.org, users@dpdk.org Content-Type: multipart/alternative; boundary="0000000000007bea4a05fa10e410" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --0000000000007bea4a05fa10e410 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Invalid argument is the same error I get from the ioctl that I get (in the diable though). Does the function get called twice in a row? On Sat, Apr 22, 2023 at 2:16=E2=80=AFAM Verghis Koshi wr= ote: > Looks like I have a somewhat similar issue while running basicfwd, no= t > sure why. The code > is just trying to enable MSI-X interrupts, but perror() says 'Invalid > argument'. > Any thoughts? > > fio_enable_msix(), ../lib/eal/linux/eal_interrupts.c:306, ret -1 > Invalid argument > EAL: Error enabling MSI-X interrupts for fd 19 > Port 0 MAC: 08 00 27 0d 34 ff > > Verghis > > On Fri, Apr 21, 2023 at 2:12=E2=80=AFPM Igor de Paula wrote: > >> I am enabling lsc interrupts. >> I don't explicitly start/stop the queues. >> Should I call link up/down before the start/stop? From the doc I saw it >> shuts down the queues. >> From what I saw it calls the msix disable function in the start and the >> stop functions. In the second time it seem to cause the error. >> >> >> On Fri, Apr 21, 2023, 22:06 Stephen Hemminger >> wrote: >> >>> On Fri, 21 Apr 2023 13:39:18 +0100 >>> Igor de Paula wrote: >>> > From what I can tell, dpdk calles the function vfio_disable_msix in >>> the >>> > stop and start function. The reason it does not happen in start up is >>> that >>> > I don't call stop before. >>> > Calling stop and then start calls the function twice. Which maybe >>> shouldn't >>> > happen... Is it a bug? >>> > >>> > On Thu, Apr 20, 2023 at 12:33=E2=80=AFPM Igor de Paula >>> wrote: >>> > >>> > > Hi, >>> > > I am having trouble with restarting a HW port allocated to DPDK. >>> > > I am running the DPDK version: 21.08.0 and Ubuntu 20.04.3 LTS. >>> > > The driver is : net_e100igb >>> > > After I start the port with no issues I try to call rte_eth_dev_sto= p >>> to >>> > > stop it. >>> > > When I am ready I call rte_eth_dev_start to start it again an I get >>> the >>> > > following message: >>> > > EAL: Error disabling MSI-X interrupts for fd 46. >>> > > I am not sure what this error is coming from and what it causes. >>> > > I found little information online. If someone could explain, I woul= d >>> > > really appreciate it. >>> > > This is a copy print on setup with no issues: >>> > > 2023-04-20 11:09:51.703542: Driver: net_e1000_igb >>> > > 2023-04-20 11:09:51.703546: Bus Id: 0000:01:00.0 >>> > > 2023-04-20 11:09:51.703551: rx offload cap: 280e (92e0f) >>> > > 2023-04-20 11:09:51.703555: tx offload cap: 8002 (803f) >>> > > 2023-04-20 11:09:51.703559: NUMA Socket: 0 >>> > > 2023-04-20 11:09:51.703563: MAC Address: b4:96:91:63:62:40 >>> > > 2023-04-20 11:09:51.703568: Max Rx Queue: 8 >>> > > 2023-04-20 11:09:51.703572: Max Tx Queue: 8 >>> > > 2023-04-20 11:09:51.703577: Max Rx Descriptors: 4096 >>> > > 2023-04-20 11:09:51.703581: Max Tx Descriptors: 4096 >>> > > 2023-04-20 11:09:51.703585: Max Rx Packet Length: 16383 >>> > > 2023-04-20 11:09:51.703589: Available Link Speeds: 10Mb/s >>> 100Mb/s 1Gb/s >>> > > 2023-04-20 11:09:51.703608: Fixed Link Speed: Auto >>> > > 2023-04-20 11:09:51.703616: Fixed Duplex: Auto >>> > > 2023-04-20 11:09:51.703623: MTU Set to: 16383 >>> > > 2023-04-20 11:09:51.703632: Actual Rx Descriptors: 4096 >>> > > 2023-04-20 11:09:51.703636: Actual Tx Descriptors: 4096 >>> > > 2023-04-20 11:09:51.703695: Set up 1 send queues >>> > > 2023-04-20 11:09:51.703700: Actual MTU: 16383 >>> > > 2023-04-20 11:09:51.703704: Actual Linkspeed: 0 >>> > > 2023-04-20 11:09:51.703708: Actual duplex: 0 >>> > > 2023-04-20 11:09:51.703713: Successfully set port interrupt eve= nt >>> > > 2023-04-20 11:09:51.818085: Flow control turned off for Port 0 >>> > > 2023-04-20 11:09:51.844179: Port 0 up and running >>> > > 2023-04-20 11:09:51.844303: Event type: LSC interrupt >>> > > 2023-04-20 11:09:51.844382: Port 0 Link Down >>> > > >>> > > And after I stop and start: >>> > > 2023-04-20 11:11:00.492428: Driver: net_e1000_igb >>> > > 2023-04-20 11:11:00.492476: Bus Id: 0000:01:00.0 >>> > > 2023-04-20 11:11:00.492526: rx offload cap: 280e (92e0f) >>> > > 2023-04-20 11:11:00.492576: tx offload cap: 8002 (803f) >>> > > 2023-04-20 11:11:00.492624: NUMA Socket: 0 >>> > > 2023-04-20 11:11:00.492672: MAC Address: b4:96:91:63:62:40 >>> > > 2023-04-20 11:11:00.492721: Max Rx Queue: 8 >>> > > 2023-04-20 11:11:00.492770: Max Tx Queue: 8 >>> > > 2023-04-20 11:11:00.492813: Max Rx Descriptors: 4096 >>> > > 2023-04-20 11:11:00.492851: Max Tx Descriptors: 4096 >>> > > 2023-04-20 11:11:00.492889: Max Rx Packet Length: 16383 >>> > > 2023-04-20 11:11:00.492958: Available Link Speeds: 10Mb/s >>> 100Mb/s 1Gb/s >>> > > 2023-04-20 11:11:00.493207: Fixed Link Speed: Auto >>> > > 2023-04-20 11:11:00.493301: Fixed Duplex: Full >>> > > 2023-04-20 11:11:00.493822: MTU Set to: 16383 >>> > > 2023-04-20 11:11:00.493889: Actual Rx Descriptors: 4096 >>> > > 2023-04-20 11:11:00.493936: Actual Tx Descriptors: 4096 >>> > > 2023-04-20 11:11:00.494190: Actual MTU: 16383 >>> > > 2023-04-20 11:11:00.494229: Actual Linkspeed: 0 >>> > > 2023-04-20 11:11:00.494266: Actual duplex: 1 >>> > > 2023-04-20 11:11:00.494305: Successfully set port interrupt eve= nt >>> > > *EAL: Error disabling MSI-X interrupts for fd 46* >>> > > 2023-04-20 11:11:00.603181: Flow control turned off for Port 0 >>> > > 2023-04-20 11:11:00.629151: Port 0 up and running >>> > > 2023-04-20 11:11:00.629222: Event type: LSC interrupt >>> > > 2023-04-20 11:11:00.629273: Port 0 Link Down >>> > > >>> > > >>> > > Thanks, >>> > > Igor >>> >>> Are you using Link State (LSC) or receive interrupts? >>> Did you start/stop the tx and rx queues. Could be a device bug where >>> it assumes all queues were stopped. >>> >>> Also check kernel dmesg output; VFIO might print an error message there= . >>> >> --0000000000007bea4a05fa10e410 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Invalid argument is the same error I get from the ioctl th= at I get (in the diable though).
Does the function get called twice in a= row?

On Sat, Apr 22, 2023 at 2:16=E2=80=AFAM Verghis Koshi <vkoshi@gmail.com> wrote:
<= div dir=3D"ltr">
=C2=A0=C2=A0=C2=A0 Looks like I have = a somewhat similar issue while running basicfwd, not sure why.=C2=A0 The co= de
is just trying to enable MSI-X interrupts, but perror() says &= #39;Invalid argument'.
=C2=A0=C2=A0=C2=A0 Any thoughts?
=

= fio_enable_msix(), ../lib/eal/linux/eal_interrupts.c:306, ret -1
<Sni= p> Invalid argument
EAL: Error enabling MSI-X interrupts for fd 19Port 0 MAC: 08 00 27 0d 34 ff

Vergh= is

On Fri, Apr 21, 2023 at 2:12=E2=80=AFPM Igor = de Paula <igordp= tx@gmail.com> wrote:
I am enabling lsc interrupts.
I don't=C2=A0 explicitly start/stop the queues.
Should I call link up/down before the start/stop? From the doc= I saw it shuts down the queues.
From what I saw it = calls the msix disable function in the start and the stop functions. In the= second time it seem to cause the error.


= On Fri, Apr 21, 2023, 22:06 Stephen Hemminger <stephen@networkplumber.org> w= rote:
On Fri, 21= Apr 2023 13:39:18 +0100
Igor de Paula <igordptx@gmail.com> wrote:
> From what I can tell, dpdk calles the function=C2=A0 vfio_disable_msix= =C2=A0 in the
> stop and start function. The reason it does not happen in start up is = that
> I don't call stop before.
> Calling stop and then start calls the function twice. Which maybe shou= ldn't
> happen... Is it a bug?
>
> On Thu, Apr 20, 2023 at 12:33=E2=80=AFPM Igor de Paula <igordptx@gm= ail.com> wrote:
>
> > Hi,
> > I am having trouble with restarting a HW port allocated to DPDK.<= br> > > I am running the DPDK version: 21.08.0 and Ubuntu 20.04.3 LTS. > > The driver is : net_e100igb
> > After I start the port with no issues I try to call rte_eth_dev_s= top to
> > stop it.
> > When I am ready I call rte_eth_dev_start to start it again an I g= et the
> > following message:
> > EAL: Error disabling MSI-X interrupts for fd 46.
> > I am not sure what this error is coming from and what it causes.<= br> > > I found little information online. If someone could explain, I wo= uld
> > really appreciate it.
> > This is a copy print on setup with no issues:
> > 2023-04-20 11:09:51.703542:=C2=A0 =C2=A0 =C2=A0Driver: net_e1000_= igb
> > 2023-04-20 11:09:51.703546:=C2=A0 =C2=A0 =C2=A0Bus Id: 0000:01:00= .0
> > 2023-04-20 11:09:51.703551:=C2=A0 =C2=A0 =C2=A0rx offload cap: 28= 0e (92e0f)
> > 2023-04-20 11:09:51.703555:=C2=A0 =C2=A0 =C2=A0tx offload cap: 80= 02 (803f)
> > 2023-04-20 11:09:51.703559:=C2=A0 =C2=A0 =C2=A0NUMA Socket: 0
> > 2023-04-20 11:09:51.703563:=C2=A0 =C2=A0 =C2=A0MAC Address: b4:96= :91:63:62:40
> > 2023-04-20 11:09:51.703568:=C2=A0 =C2=A0 =C2=A0Max Rx Queue: 8 > > 2023-04-20 11:09:51.703572:=C2=A0 =C2=A0 =C2=A0Max Tx Queue: 8 > > 2023-04-20 11:09:51.703577:=C2=A0 =C2=A0 =C2=A0Max Rx Descriptors= : 4096
> > 2023-04-20 11:09:51.703581:=C2=A0 =C2=A0 =C2=A0Max Tx Descriptors= : 4096
> > 2023-04-20 11:09:51.703585:=C2=A0 =C2=A0 =C2=A0Max Rx Packet Leng= th: 16383
> > 2023-04-20 11:09:51.703589:=C2=A0 =C2=A0 =C2=A0Available Link Spe= eds: 10Mb/s 100Mb/s 1Gb/s
> > 2023-04-20 11:09:51.703608:=C2=A0 =C2=A0 =C2=A0Fixed Link Speed: = Auto
> > 2023-04-20 11:09:51.703616:=C2=A0 =C2=A0 =C2=A0Fixed Duplex: Auto=
> > 2023-04-20 11:09:51.703623:=C2=A0 =C2=A0 =C2=A0MTU Set to: 16383<= br> > > 2023-04-20 11:09:51.703632:=C2=A0 =C2=A0 =C2=A0Actual Rx Descript= ors: 4096
> > 2023-04-20 11:09:51.703636:=C2=A0 =C2=A0 =C2=A0Actual Tx Descript= ors: 4096
> > 2023-04-20 11:09:51.703695:=C2=A0 =C2=A0 =C2=A0Set up 1 send queu= es
> > 2023-04-20 11:09:51.703700:=C2=A0 =C2=A0 =C2=A0Actual MTU: 16383<= br> > > 2023-04-20 11:09:51.703704:=C2=A0 =C2=A0 =C2=A0Actual Linkspeed: = 0
> > 2023-04-20 11:09:51.703708:=C2=A0 =C2=A0 =C2=A0Actual duplex: 0 > > 2023-04-20 11:09:51.703713:=C2=A0 =C2=A0 =C2=A0Successfully set p= ort interrupt event
> > 2023-04-20 11:09:51.818085:=C2=A0 =C2=A0 =C2=A0Flow control turne= d off for Port 0
> > 2023-04-20 11:09:51.844179:=C2=A0 =C2=A0 =C2=A0Port 0 up and runn= ing
> > 2023-04-20 11:09:51.844303: Event type: LSC interrupt
> > 2023-04-20 11:09:51.844382:=C2=A0 =C2=A0 =C2=A0Port 0 Link Down > >
> > And after I stop and start:
> > 2023-04-20 11:11:00.492428:=C2=A0 =C2=A0 =C2=A0Driver: net_e1000_= igb
> > 2023-04-20 11:11:00.492476:=C2=A0 =C2=A0 =C2=A0Bus Id: 0000:01:00= .0
> > 2023-04-20 11:11:00.492526:=C2=A0 =C2=A0 =C2=A0rx offload cap: 28= 0e (92e0f)
> > 2023-04-20 11:11:00.492576:=C2=A0 =C2=A0 =C2=A0tx offload cap: 80= 02 (803f)
> > 2023-04-20 11:11:00.492624:=C2=A0 =C2=A0 =C2=A0NUMA Socket: 0
> > 2023-04-20 11:11:00.492672:=C2=A0 =C2=A0 =C2=A0MAC Address: b4:96= :91:63:62:40
> > 2023-04-20 11:11:00.492721:=C2=A0 =C2=A0 =C2=A0Max Rx Queue: 8 > > 2023-04-20 11:11:00.492770:=C2=A0 =C2=A0 =C2=A0Max Tx Queue: 8 > > 2023-04-20 11:11:00.492813:=C2=A0 =C2=A0 =C2=A0Max Rx Descriptors= : 4096
> > 2023-04-20 11:11:00.492851:=C2=A0 =C2=A0 =C2=A0Max Tx Descriptors= : 4096
> > 2023-04-20 11:11:00.492889:=C2=A0 =C2=A0 =C2=A0Max Rx Packet Leng= th: 16383
> > 2023-04-20 11:11:00.492958:=C2=A0 =C2=A0 =C2=A0Available Link Spe= eds: 10Mb/s 100Mb/s 1Gb/s
> > 2023-04-20 11:11:00.493207:=C2=A0 =C2=A0 =C2=A0Fixed Link Speed: = Auto
> > 2023-04-20 11:11:00.493301:=C2=A0 =C2=A0 =C2=A0Fixed Duplex: Full=
> > 2023-04-20 11:11:00.493822:=C2=A0 =C2=A0 =C2=A0MTU Set to: 16383<= br> > > 2023-04-20 11:11:00.493889:=C2=A0 =C2=A0 =C2=A0Actual Rx Descript= ors: 4096
> > 2023-04-20 11:11:00.493936:=C2=A0 =C2=A0 =C2=A0Actual Tx Descript= ors: 4096
> > 2023-04-20 11:11:00.494190:=C2=A0 =C2=A0 =C2=A0Actual MTU: 16383<= br> > > 2023-04-20 11:11:00.494229:=C2=A0 =C2=A0 =C2=A0Actual Linkspeed: = 0
> > 2023-04-20 11:11:00.494266:=C2=A0 =C2=A0 =C2=A0Actual duplex: 1 > > 2023-04-20 11:11:00.494305:=C2=A0 =C2=A0 =C2=A0Successfully set p= ort interrupt event
> > *EAL: Error disabling MSI-X interrupts for fd 46*
> > 2023-04-20 11:11:00.603181:=C2=A0 =C2=A0 =C2=A0Flow control turne= d off for Port 0
> > 2023-04-20 11:11:00.629151:=C2=A0 =C2=A0 =C2=A0Port 0 up and runn= ing
> > 2023-04-20 11:11:00.629222: Event type: LSC interrupt
> > 2023-04-20 11:11:00.629273:=C2=A0 =C2=A0 =C2=A0Port 0 Link Down > >
> >
> > Thanks,
> > Igor

Are you using Link State (LSC) or receive interrupts?
Did you start/stop the tx and rx queues. Could be a device bug where
it assumes all queues were stopped.

Also check kernel dmesg output; VFIO might print an error message there.
--0000000000007bea4a05fa10e410--