From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A12A043EB0 for ; Fri, 19 Apr 2024 14:44:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 14E3C40633; Fri, 19 Apr 2024 14:44:52 +0200 (CEST) Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by mails.dpdk.org (Postfix) with ESMTP id D0CFF402DF for ; Fri, 19 Apr 2024 14:44:50 +0200 (CEST) Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-516d264d0e4so2926983e87.0 for ; Fri, 19 Apr 2024 05:44:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bisdn-de.20230601.gappssmtp.com; s=20230601; t=1713530690; x=1714135490; darn=dpdk.org; h=content-transfer-encoding:content-language:in-reply-to:from :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=OpCK7/1jd2DdobqGdCd9ohLesTutxSre3sMzgckBOI4=; b=PNaSGbpjvnWhWzWcuEB6TAhjAkbgdIYeOK484kNg+px2cno9kwS7Q0lsxsqFCq9goA wTYwn1tMPvMKM/yLqMsJmgCfAb8HclLL23wh3f0VcqMWTPrKP6hZnDNMwGgirOo9cDlS 8b0BDmJNs9wwyYmjDyi5cw/HC9QMY7EkM1ZH6ZYQ5DrwGnBXpFlxlTYyG0zaqAr6EzTq 6NK+t6vgUAgdNurmKDgRaoT/p2/e+B1BbUFAnXzlhRq6d63wm/6yV3zMPJ4fmapMXxYF yZdL6YcU2ywt4LIA64Nehw12HfDLnER1mNKZlBxRaJ1UmulnS80j89VTMxa8sDjDeSZy R9+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713530690; x=1714135490; h=content-transfer-encoding:content-language:in-reply-to:from :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OpCK7/1jd2DdobqGdCd9ohLesTutxSre3sMzgckBOI4=; b=hvVLFKw4sJvtdbdT/WY4Z8SpYPDsK9qwpNJEytXScQjlIRdZ0cFLeBBBnz5p5AGb0e 5skR5K8H5m8MQ1KbegY0idBZVOCs5Y2lMMtT1AWIMq79IpgK5IQ3Io8licQwVqtx5DwA TD8hki/jcxWbyESYo7f/uVN6xdR8d7fz4NWRwr9SEWoTlz61oA2daMappF5j/0tKxN7d t0vEQ7S+N+8bkk3zVw5o1oXS69vDftSat+5gZvfWTSLk0y6WJioAa2T6KH0mg7EfMk7f tzIMEterQwaXYjeX2WYFIOHL0nYkNEoXt2Qq7UTcViyI5Pb2ppLgcsjbirZbHe5pNeld Le0g== X-Gm-Message-State: AOJu0YyDgcsmLs8OLRgdJFXJPQM4+aR9gbtNul5yMql+Ar3Oayh5magg HT2511PhuV+tLIo/RR+cCQZwv0AyVt1gT57YVVcNsoOJfBf3mEHVIAvIBKW4hIczr7AYZz8l1xI keijxrK0jkFS0LTyCIob5FGx1tgoMEk3FVn5R X-Google-Smtp-Source: AGHT+IE1BwiRvqcI6O55FK6w9PTaYD0WM1+1SrzDakJEfsQbHjtJ/JRQCjUFXzkapLnSv0QfolCQuQ== X-Received: by 2002:ac2:4850:0:b0:519:2d70:c5c5 with SMTP id 16-20020ac24850000000b005192d70c5c5mr703799lfy.7.1713530690072; Fri, 19 Apr 2024 05:44:50 -0700 (PDT) Received: from [10.101.18.19] ([185.224.57.161]) by smtp.gmail.com with ESMTPSA id a3-20020a19ca03000000b0051887bac843sm693407lfg.300.2024.04.19.05.44.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 19 Apr 2024 05:44:49 -0700 (PDT) Message-ID: <19d34638-e378-7b20-99d6-5c8d0ef2696e@bisdn.de> Date: Fri, 19 Apr 2024 14:44:44 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Trex with mlx5 driver - Multiple streams with different VLAN priority causes high CPU utilization To: Dariusz Sosnowski Cc: "users@dpdk.org" References: <7efda351-e554-4120-5a61-3b0a534189e2@bisdn.de> <0afcef7d-08a5-5696-ef13-a1ba4ad9ace2@bisdn.de> From: Rubens Figueiredo In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Thank you for the help! I ran the commands and traffic is being generated correctly. Best regards, Rubens On 4/19/24 14:31, Dariusz Sosnowski wrote: > Thank you for the info. > > You might have run into an issue, where NIC generates backpressure to the= SW because of very frequent switching between different VLAN priorities on= Tx datapath in the NIC. > > Could you please apply the following QoS configuration on all interfaces = and rerun the test with different VLAN priorities? > > sudo mlnx_qos -i --trust=3Ddscp > for dscp in {0..63}; do sudo mlnx_qos -i --dscp2prio set,$dscp,0;= sleep 0.001;done > > These commands will map internally all priorities to priority 0. This wor= karound should reduce the backpressure without affecting the generated traf= fic. > > Best regards, > Dariusz Sosnowski > >> -----Original Message----- >> From: Rubens Figueiredo >> Sent: Thursday, April 18, 2024 15:21 >> To: Dariusz Sosnowski >> Cc: users@dpdk.org >> Subject: Re: Trex with mlx5 driver - Multiple streams with different VLA= N >> priority causes high CPU utilization >> >> External email: Use caution opening links or attachments >> >> >> Hi Dariusz, >> >> Thank you for the help. >> >> If the two parallel stream use the same VLAN priority then the issue is = not >> visible anymore. Trex output visible below. >> >> Different priority >> CPU util. |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 96.97% |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0.0% | Tx bps L2 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 3.59 Gbps |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0.29 >> bps | 3.59 Gbps Tx bps L1 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 3.64 Gbps |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0.= 37 bps | 3.64 Gbps >> Tx pps |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 298.49 Kpps |=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 pps | 298= .49 Kpps >> Line Util. |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 3.64 % |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 % | >> >> Same priority >> CPU util. |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0.54% |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0.0% | Tx bps L2 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 23.98 Gbps |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 >> bps | 23.98 Gbps Tx bps L1 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 24.3 Gbps |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0 bps | 24.3 Gbps >> Tx pps |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1.99 Mpps |= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 = pps | 1.99 Mpps >> Line Util. |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 24.3 % |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 0 % | >> >> I have attached the requested output to the email. >> >> Best, >> Rubens >> >> On 4/18/24 14:48, Dariusz Sosnowski wrote: >>> Hi Rubens, >>> >>> Would you be able to provide the output of "ethtool -S " for bot= h VFs >> before and after the test? >>> Does the same issue appear on this system if both parallel streams use = the >> same VLAN priority? >>> Best regards, >>> Dariusz Sosnowski >>> >>>> From: Rubens Figueiredo >>>> Sent: Wednesday, April 17, 2024 19:07 >>>> To: users@dpdk.org >>>> Subject: Trex with mlx5 driver - Multiple streams with different VLAN >>>> priority causes high CPU utilization >>>> >>>> Hello community, >>>> I am facing a strange issue in the Trex stateless code, version v3.02 = and >> v3.04. I am using the Mellanox Cx-5, and have created two VFs on top of = the >> PF 0. The mlx5_core version I am using is the 5.7-1.0.2, and the ofed ve= rsion is >> MLNX_OFED_LINUX-5.7-1.0.2.0 (OFED-5.7-1.0.2). >>>> I have created the following issue in the trex-core repository >> [here](https://github.com/cisco-system-traffic-generator/trex- >> core/issues/1124), and was recommended to post the issue in here. In the >> github issue you see screenshots of the issue I am facing. >>>> I am trying to create two parallel streams with different VLAN priorit= ies, but >> the load generated is not what I expect it to be, and CPU util. seems in= credibly >> high (~99%). >>>> I have reproduced this issue with the --software and non software vers= ion. >>>> The script I used is below. >>>> import stl_path >>>> from trex.stl.api import * >>>> >>>> import time >>>> import pprint >>>> from ipaddress import ip_address, ip_network >>>> >>>> import argparse >>>> import configparser >>>> import os >>>> import json >>>> >>>> >>>> def get_packet(tos, mac_dst, ip_src, size): >>>> # pkt =3D Ether(src=3D"02:00:00:00:00:01",dst=3D"00:00:00:01:00:= 01") / >>>> IP(src=3D"10.0.0.2", tos=3Dtos) / UDP(sport=3D4444, dport=3D4444) >>>> >>>> pkt =3D ( >>>> Ether(src=3D"00:01:00:00:00:02", dst=3Dmac_dst) >>>> # Ether(dst=3D"11:11:11:11:11:11") >>>> # / Dot1AD(vlan=3D0) >>>> / Dot1Q(vlan=3D0, prio=3Dtos) >>>> / IP(src=3Dip_src) >>>> / UDP(sport=3D4444, dport=3D4444) >>>> ) >>>> pad =3D max(0, size - len(pkt)) * "x" >>>> >>>> return pkt / pad >>>> >>>> def main(): >>>> """ """ >>>> tx_port =3D 0 >>>> rx_port =3D 1 >>>> >>>> c =3D STLClient() >>>> >>>> # connect to server >>>> c.connect() >>>> >>>> # prepare our ports >>>> c.reset(ports=3D[tx_port, rx_port]) >>>> >>>> streams =3D [] >>>> s =3D STLStream( >>>> packet=3DSTLPktBuilder( >>>> pkt=3Dget_packet(4,"00:11:22:33:44:55", "10.1.0.2",512), >>>> # vm =3D vm, >>>> ), >>>> isg=3D0 * 1000000, >>>> mode=3DSTLTXCont(pps=3D1.2*10**6), >>>> # flow_stats =3D STLFlowLatencyStats(pg_id =3D 0) >>>> flow_stats =3D STLFlowStats(pg_id=3D0), >>>> ) >>>> >>>> streams.append(s) >>>> >>>> s2 =3D STLStream( >>>> packet=3DSTLPktBuilder( >>>> pkt=3Dget_packet(2,"00:11:22:33:44:55", "10.1.0.2",512), >>>> # vm =3D vm, >>>> ), >>>> isg=3D0 * 1000000, >>>> mode=3DSTLTXCont(pps=3D1.2*10**6), >>>> # flow_stats =3D STLFlowLatencyStats(pg_id =3D 0) >>>> flow_stats =3D STLFlowStats(pg_id=3D1), >>>> ) >>>> >>>> streams.append(s2) >>>> >>>> c.add_streams(streams, ports=3D[tx_port]) >>>> >>>> c.clear_stats() >>>> >>>> c.start(ports=3D[tx_port], duration=3D60, mult=3D"25gbpsl1") >>>> >>>> c.wait_on_traffic(ports=3D[tx_port, rx_port]) >>>> >>>> stats =3D c.get_stats() >>>> print(stats) >>>> >>>> if __name__ =3D=3D "__main__": >>>> main() >>>> >>>> >>>> And the configuration is >>>> - port_limit: 2 >>>> version: 2 >>>> port_bandwidth_gb: 100 >>>> interfaces: ["3b:00.2", "3b:00.3"] >>>> port_info: >>>> - dest_mac: 00:00:00:00:00:01 >>>> src_mac: 00:01:00:00:00:01 >>>> - dest_mac: 00:00:00:00:00:02 >>>> src_mac: 00:01:00:00:00:02 >>>> c: 14 >>>> platform: >>>> master_thread_id: 8 >>>> latency_thread_id: 27 >>>> dual_if: >>>> - socket: 0 >>>> threads: >>>> [9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26] >>>> >>>> >>>> BISDN GmbH >>>> K=C3=B6rnerstra=C3=9Fe 7-10 >>>> 10785 Berlin >>>> Germany >>>> >>>> Phone: +49-30-6108-1-6100 >>>> >>>> Managing Directors: >>>> Dr.-Ing. Hagen Woesner, Andreas K=C3=B6psel >>>> >>>> Commercial register: >>>> Amtsgericht Berlin-Charlottenburg HRB 141569 B VAT ID No: >> DE283257294 >>>> ________________________________________ >> -- >> BISDN GmbH >> K=C3=B6rnerstra=C3=9Fe 7-10 >> 10785 Berlin >> Germany >> >> >> Phone: >> +49-30-6108-1-6100 >> >> >> Managing Directors: >> Dr.-Ing. Hagen Woesner, Andreas >> K=C3=B6psel >> >> >> Commercial register: >> Amtsgericht Berlin-Charlottenburg HRB 141569 B VAT ID No: DE283257294 --=20 BISDN GmbH K=C3=B6rnerstra=C3=9Fe 7-10 10785 Berlin Germany Phone:=20 +49-30-6108-1-6100 Managing Directors:=C2=A0 Dr.-Ing. Hagen Woesner, Andreas=20 K=C3=B6psel Commercial register:=C2=A0 Amtsgericht Berlin-Charlottenburg HRB 141569=20 B VAT ID No:=C2=A0DE283257294