VPP main thread blocked by DPDK APIs

VPP main thread blocked by DPDK APIs

[Ruitao Gong]

[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]

hi DEVers,

Send this mail for some help about VPP and DPDK.
rte_eth_stats_get (xd->port_id, &xd->stats)
rte_eth_xstats_get (xd->port_id, xd->xstats, len)
    APIs in dpdk_update_counters() node "dpdk_process" will cost about 140ms each physical interface every 10s (DPDK_STATS_POLL_INTERVAL).
    main_thread will blocked N*140 ms when N physical interface in up state.
    if govpp send heartbeat message to VPP, main_thread will have chance delay to read msg and reply, when main_thread blocked by doing update_counters.
    link between GoVPP and VPP will turn down when delay time more than 250ms. It is easy to reproduce the scenario.
    is there any plan to fix it or some suggestion to avoid it? Many thanks!

env details:
    GoVPP:v0.3.5
    DPDK version: 20.05
    VPP version:20.09
    centos7 + Intel X710
expansion-rom-version:
bus-info: 0000:19:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

    startup.conf dpdk uio-driver vfio_pci
    driver: i40evf version: 3.2.2-k

B&R,
Ruitao
________________________________
This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to security@mavenir.com. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

[-- Attachment #2: Type: text/html, Size: 6564 bytes --]

Re: VPP main thread blocked by DPDK APIs

[Stephen Hemminger]

On Wed, 22 Dec 2021 08:25:49 +0000
Ruitao Gong <ruitao.gong@mavenir.com> wrote:

> hi DEVers,
> 
> Send this mail for some help about VPP and DPDK.
> rte_eth_stats_get (xd->port_id, &xd->stats)
> rte_eth_xstats_get (xd->port_id, xd->xstats, len)
>     APIs in dpdk_update_counters() node "dpdk_process" will cost about 140ms each physical interface every 10s (DPDK_STATS_POLL_INTERVAL).
>     main_thread will blocked N*140 ms when N physical interface in up state.
>     if govpp send heartbeat message to VPP, main_thread will have chance delay to read msg and reply, when main_thread blocked by doing update_counters.
>     link between GoVPP and VPP will turn down when delay time more than 250ms. It is easy to reproduce the scenario.
>     is there any plan to fix it or some suggestion to avoid it? Many thanks!
> 
> env details:
>     GoVPP:v0.3.5
>     DPDK version: 20.05
>     VPP version:20.09
>     centos7 + Intel X710
> expansion-rom-version:
> bus-info: 0000:19:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
> 
>     startup.conf dpdk uio-driver vfio_pci
>     driver: i40evf version: 3.2.2-k
> 
> B&R,
> Ruitao

Polling for statistics may be expensive based on the hardware, it requires the device driver
to query the hardware for each queue. These requests may involve setting a flag and polling
for completion of the request.

We had similar issues, with bnxt; ended up doing small changes to driver to avoid
polling for queue information for unused queues and that helped. But getting statistics
can be still be too expensive.


> ________________________________
> This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to security@mavenir.com. This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

Please lose the header or suppress it. Either your main should never be on mailing list because
it violates that clause, or the clause is irrelevant.