Re: [dpdk-users] Does vmxnet3 PMD supports LSC=1 ?

[madhukar mythri <madhukar.mythri@gmail.com>]

Hi Yong,

Thanks for quick response.

As per "vmxnet3" PCI bus device information, we got MSI-X based
interrupt-line as mentioned below in the "sysfs":
we are able to get link-changes interrupts, but, on top it we get all other
interrupts also, which is taking more CPU-time when traffic load.
================
~ # cat /sys/bus/pci/devices/0000\:03\:00.0/msi_irqs/58
msix
~ #

~ # lspci -s 03:00.0 -vv
03:00.0 Class 0200: Device 15ad:07b0 (rev 01)
        Subsystem: Device 15ad:07b0
        Physical Slot: 160
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at fd4fc000 (32-bit, non-prefetchable) [size=4K]
        Region 1: Memory at fd4fd000 (32-bit, non-prefetchable) [size=4K]
        Region 2: Memory at fd4fe000 (32-bit, non-prefetchable) [size=8K]
        Region 3: I/O ports at 4000 [size=16]
        [virtual] Expansion ROM at fd400000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x32, ASPM L0s, Exit
Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x32, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-,
LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-
                         Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
        Capabilities: [84] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [9c] MSI-X: Enable+ Count=25 Masked-
                Vector table: BAR=2 offset=00000000
                PBA: BAR=2 offset=00001000
        Capabilities: [100 v1] Device Serial Number ff-29-0c-00-d6-72-89-fe
        Kernel driver in use: igb_uio

~ #
==============

So, in vmxnet3 LSC=0(poll-mode) is recommended ?

We are using ESXi-6.7 Hyper-visor.

Thanks,
Madhuker.


On Fri, Jan 15, 2021 at 12:28 AM Yong Wang <yongwang@vmware.com> wrote:

> Vmxnet3 driver supports lsc and your observation is a known behavior with
> vmxnet3 running with INTx/MSI interrupt mode.  Enabling or disabling lsc
> intr in this mode will enable/disable RX interrupts as well.  This is a
> device behavior and there is no known way to mask just lsc or rx interrupts
> as the mask is per vector.  To work around this, either disable interrupt
> or move to msi-x. Currently the upstream dpdk vmxnet3 driver does not
> support msix.  We have plans to upstream vmxnet3 msix support in the future.
>
> Yong
>
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Date: Thursday, January 14, 2021 at 10:50 AM
> To: madhukar mythri <madhukar.mythri@gmail.com>
> Cc: "users@dpdk.org" <users@dpdk.org>, Yong Wang <yongwang@vmware.com>
> Subject: Re: [dpdk-users] Does vmxnet3 PMD supports LSC=1 ?
>
>     +Cc Yong Wang, maintainer of this PMD.
>
>     14/01/2021 18:30, madhukar mythri:
>     > Hi,
>     >
>     > Does vmxnet3 PMD support LSC=1(i.e with interrupt mode) for link
> changes ?
>     >
>     > When i enable LSC=1 the functionality works fine, but, when pumping
> traffic
>     > i'm seeing increasing in CPU load on some cores which is running
>     > "eal-intr-thread" epoll_wait() function for more CPU-time.
>     >
>     > Actually, interrupt should come only when Link changes, but, we are
> seeing
>     > interrupt for each incoming Rx-packet and also a lot of spurious
> interrupts.
>     > =============
>     > ~ # cat /proc/interrupts |grep igb
>     >  58:    1254293          0          0          0   PCI-MSI
> 1572864-edge
>     >  igb_uio
>     >  59:    1278105          0          0          0   PCI-MSI
> 5767168-edge
>     >  igb_uio
>     > ~ # cat /proc/irq/58/spurious
>     > count 98035
>     > unhandled 0
>     > last_unhandled 0 ms
>     > ~ #
>     > ==============
>     >
>     > Does anyone tried LSC=1 in vmxnet3 PMD based apps and faced similar
> issues
>     > ? If so, please let me know.
>     >
>     > Tried with DPDK-18.11, DPDK-19.11 and DPDK-20.05.
>     >
>     > Thanks,
>     > Madhukar.
>
>
>
>