[dpdk-users] Does vmxnet3 PMD supports LSC=1 ?

[dpdk-users] Does vmxnet3 PMD supports LSC=1 ?

[madhukar mythri]

Hi,

Does vmxnet3 PMD support LSC=1(i.e with interrupt mode) for link changes ?

When i enable LSC=1 the functionality works fine, but, when pumping traffic
i'm seeing increasing in CPU load on some cores which is running
"eal-intr-thread" epoll_wait() function for more CPU-time.

Actually, interrupt should come only when Link changes, but, we are seeing
interrupt for each incoming Rx-packet and also a lot of spurious interrupts.
=============
~ # cat /proc/interrupts |grep igb
 58:    1254293          0          0          0   PCI-MSI 1572864-edge
 igb_uio
 59:    1278105          0          0          0   PCI-MSI 5767168-edge
 igb_uio
~ # cat /proc/irq/58/spurious
count 98035
unhandled 0
last_unhandled 0 ms
~ #
==============

Does anyone tried LSC=1 in vmxnet3 PMD based apps and faced similar issues
? If so, please let me know.

Tried with DPDK-18.11, DPDK-19.11 and DPDK-20.05.

Thanks,
Madhukar.

Re: [dpdk-users] Does vmxnet3 PMD supports LSC=1 ?

[Thomas Monjalon]

+Cc Yong Wang, maintainer of this PMD.

14/01/2021 18:30, madhukar mythri:
> Hi,
> 
> Does vmxnet3 PMD support LSC=1(i.e with interrupt mode) for link changes ?
> 
> When i enable LSC=1 the functionality works fine, but, when pumping traffic
> i'm seeing increasing in CPU load on some cores which is running
> "eal-intr-thread" epoll_wait() function for more CPU-time.
> 
> Actually, interrupt should come only when Link changes, but, we are seeing
> interrupt for each incoming Rx-packet and also a lot of spurious interrupts.
> =============
> ~ # cat /proc/interrupts |grep igb
>  58:    1254293          0          0          0   PCI-MSI 1572864-edge
>  igb_uio
>  59:    1278105          0          0          0   PCI-MSI 5767168-edge
>  igb_uio
> ~ # cat /proc/irq/58/spurious
> count 98035
> unhandled 0
> last_unhandled 0 ms
> ~ #
> ==============
> 
> Does anyone tried LSC=1 in vmxnet3 PMD based apps and faced similar issues
> ? If so, please let me know.
> 
> Tried with DPDK-18.11, DPDK-19.11 and DPDK-20.05.
> 
> Thanks,
> Madhukar.

Re: [dpdk-users] Does vmxnet3 PMD supports LSC=1 ?

[Yong Wang]

Vmxnet3 driver supports lsc and your observation is a known behavior with vmxnet3 running with INTx/MSI interrupt mode.  Enabling or disabling lsc intr in this mode will enable/disable RX interrupts as well.  This is a device behavior and there is no known way to mask just lsc or rx interrupts as the mask is per vector.  To work around this, either disable interrupt or move to msi-x. Currently the upstream dpdk vmxnet3 driver does not support msix.  We have plans to upstream vmxnet3 msix support in the future.

Yong

-----Original Message-----
From: Thomas Monjalon <thomas@monjalon.net>
Date: Thursday, January 14, 2021 at 10:50 AM
To: madhukar mythri <madhukar.mythri@gmail.com>
Cc: "users@dpdk.org" <users@dpdk.org>, Yong Wang <yongwang@vmware.com>
Subject: Re: [dpdk-users] Does vmxnet3 PMD supports LSC=1 ?

    +Cc Yong Wang, maintainer of this PMD.

    14/01/2021 18:30, madhukar mythri:
    > Hi,
    > 
    > Does vmxnet3 PMD support LSC=1(i.e with interrupt mode) for link changes ?
    > 
    > When i enable LSC=1 the functionality works fine, but, when pumping traffic
    > i'm seeing increasing in CPU load on some cores which is running
    > "eal-intr-thread" epoll_wait() function for more CPU-time.
    > 
    > Actually, interrupt should come only when Link changes, but, we are seeing
    > interrupt for each incoming Rx-packet and also a lot of spurious interrupts.
    > =============
    > ~ # cat /proc/interrupts |grep igb
    >  58:    1254293          0          0          0   PCI-MSI 1572864-edge
    >  igb_uio
    >  59:    1278105          0          0          0   PCI-MSI 5767168-edge
    >  igb_uio
    > ~ # cat /proc/irq/58/spurious
    > count 98035
    > unhandled 0
    > last_unhandled 0 ms
    > ~ #
    > ==============
    > 
    > Does anyone tried LSC=1 in vmxnet3 PMD based apps and faced similar issues
    > ? If so, please let me know.
    > 
    > Tried with DPDK-18.11, DPDK-19.11 and DPDK-20.05.
    > 
    > Thanks,
    > Madhukar.

Re: [dpdk-users] Does vmxnet3 PMD supports LSC=1 ?

[madhukar mythri]

Hi Yong,

Thanks for quick response.

As per "vmxnet3" PCI bus device information, we got MSI-X based
interrupt-line as mentioned below in the "sysfs":
we are able to get link-changes interrupts, but, on top it we get all other
interrupts also, which is taking more CPU-time when traffic load.
================
~ # cat /sys/bus/pci/devices/0000\:03\:00.0/msi_irqs/58
msix
~ #

~ # lspci -s 03:00.0 -vv
03:00.0 Class 0200: Device 15ad:07b0 (rev 01)
        Subsystem: Device 15ad:07b0
        Physical Slot: 160
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at fd4fc000 (32-bit, non-prefetchable) [size=4K]
        Region 1: Memory at fd4fd000 (32-bit, non-prefetchable) [size=4K]
        Region 2: Memory at fd4fe000 (32-bit, non-prefetchable) [size=8K]
        Region 3: I/O ports at 4000 [size=16]
        [virtual] Expansion ROM at fd400000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x32, ASPM L0s, Exit
Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x32, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-,
LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-
                         Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
        Capabilities: [84] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [9c] MSI-X: Enable+ Count=25 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00001000
        Capabilities: [100 v1] Device Serial Number ff-29-0c-00-d6-72-89-fe
        Kernel driver in use: igb_uio

~ #
==============

So, in vmxnet3 LSC=0(poll-mode) is recommended ?

We are using ESXi-6.7 Hyper-visor.

Thanks,
Madhuker.


On Fri, Jan 15, 2021 at 12:28 AM Yong Wang <yongwang@vmware.com> wrote:

> Vmxnet3 driver supports lsc and your observation is a known behavior with
> vmxnet3 running with INTx/MSI interrupt mode.  Enabling or disabling lsc
> intr in this mode will enable/disable RX interrupts as well.  This is a
> device behavior and there is no known way to mask just lsc or rx interrupts
> as the mask is per vector.  To work around this, either disable interrupt
> or move to msi-x. Currently the upstream dpdk vmxnet3 driver does not
> support msix.  We have plans to upstream vmxnet3 msix support in the future.
>
> Yong
>
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Date: Thursday, January 14, 2021 at 10:50 AM
> To: madhukar mythri <madhukar.mythri@gmail.com>
> Cc: "users@dpdk.org" <users@dpdk.org>, Yong Wang <yongwang@vmware.com>
> Subject: Re: [dpdk-users] Does vmxnet3 PMD supports LSC=1 ?
>
>     +Cc Yong Wang, maintainer of this PMD.
>
>     14/01/2021 18:30, madhukar mythri:
>     > Hi,
>     >
>     > Does vmxnet3 PMD support LSC=1(i.e with interrupt mode) for link
> changes ?
>     >
>     > When i enable LSC=1 the functionality works fine, but, when pumping
> traffic
>     > i'm seeing increasing in CPU load on some cores which is running
>     > "eal-intr-thread" epoll_wait() function for more CPU-time.
>     >
>     > Actually, interrupt should come only when Link changes, but, we are
> seeing
>     > interrupt for each incoming Rx-packet and also a lot of spurious
> interrupts.
>     > =============
>     > ~ # cat /proc/interrupts |grep igb
>     >  58:    1254293          0          0          0   PCI-MSI
> 1572864-edge
>     >  igb_uio
>     >  59:    1278105          0          0          0   PCI-MSI
> 5767168-edge
>     >  igb_uio
>     > ~ # cat /proc/irq/58/spurious
>     > count 98035
>     > unhandled 0
>     > last_unhandled 0 ms
>     > ~ #
>     > ==============
>     >
>     > Does anyone tried LSC=1 in vmxnet3 PMD based apps and faced similar
> issues
>     > ? If so, please let me know.
>     >
>     > Tried with DPDK-18.11, DPDK-19.11 and DPDK-20.05.
>     >
>     > Thanks,
>     > Madhukar.
>
>
>
>