From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B5E514576F; Thu, 8 Aug 2024 22:36:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 346FA42E0B; Thu, 8 Aug 2024 22:36:32 +0200 (CEST) Received: from mail-pl1-f228.google.com (mail-pl1-f228.google.com [209.85.214.228]) by mails.dpdk.org (Postfix) with ESMTP id C140C4278D for ; Thu, 8 Aug 2024 22:36:29 +0200 (CEST) Received: by mail-pl1-f228.google.com with SMTP id d9443c01a7336-1fc491f9b55so12194075ad.3 for ; Thu, 08 Aug 2024 13:36:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1723149389; x=1723754189; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u/Hr06ndUJRxkooQeKlwgZ9Kcu+fk3z4vOjNx7Un+c4=; b=isRj6SIbthbrTJ1vNQr2Y9MV4XeIoGfbu0xat74QEr5JKG2+jSANdKIkas4UW0l0vw /wZ0cq/pPJbMiwMZmqarpBei91TjisJV0Eq8drYOuhDq/dTr4ikti9UfCeyh0x1P9GRG MFOahULG5RMqgeyn+x9/PZF6Zeqen83DJHrSQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723149389; x=1723754189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u/Hr06ndUJRxkooQeKlwgZ9Kcu+fk3z4vOjNx7Un+c4=; b=a5YsZI2yK/GDKE72Nv9kXDh4aTIXrVH7TIAifFpzYHn+YcpwbN9Lm4G3q4xY1VKaeb q9NEdMy9J1dWNWb5X25/+9SI6fbWUjsP2cS4XSwffB8SPYmasy0g+H4kmhoGl2qX1y32 sWNz9BuLSFQCBqorwVGXF2VhPi2PSvdOthsU6rR8ATSZcMb5o5oLjzZn/a1jh0hsQEQY ldYPuE7ypiAsO1ihOxr+FzuF+Szec0g+RoyUBOzQB/FIyqgiCKDYpPL/B1MRyFcJYwb/ Iq0gCtFXGfPOJxDccVsu5OuFI4ASHDE4JF/ycCTTOBrfgqyWa1eRQwCPfjKw4dyQyyHa cKXw== X-Gm-Message-State: AOJu0YxmFMw+dLoN1Okk+9zE0IVZaCVREeIf40jc375jQb7FR4XmYUJ2 umxDaM47mj/KyLv+FFO6XsH3jaMtMobqfqjeL9eEt3xlqDWbb9CrVAIUSlvksrr7ZGMxGRS7/Gi gYJiCP6tcTIE7rj5fBzctXYkVAkbV+ys1 X-Google-Smtp-Source: AGHT+IHl90cuETHGKt46TAZ5hmRXVuS06ovNea+B6n52LUSfe8paxw99EYCNCNRl9yeedgMucbdj0aAp0Rot X-Received: by 2002:a17:902:dac4:b0:1fb:8797:beff with SMTP id d9443c01a7336-200952706fbmr41555495ad.36.1723149388817; Thu, 08 Aug 2024 13:36:28 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [132.177.123.84]) by smtp-relay.gmail.com with ESMTPS id d9443c01a7336-1ff592907e3sm10519795ad.135.2024.08.08.13.36.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Aug 2024 13:36:28 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1257::1083]) by postal.iol.unh.edu (Postfix) with ESMTP id 87B2F605C351; Thu, 8 Aug 2024 16:36:27 -0400 (EDT) From: jspewock@iol.unh.edu To: thomas@monjalon.net, yoan.picchi@foss.arm.com, paul.szczepanek@arm.com, Honnappa.Nagarahalli@arm.com, probb@iol.unh.edu, wathsala.vithanage@arm.com, Luca.Vizzarro@arm.com, juraj.linkes@pantheon.tech, npratte@iol.unh.edu, alex.chapman@arm.com Cc: dev@dpdk.org, Jeremy Spewock Subject: [PATCH v3 1/1] dts: add text parser for testpmd verbose output Date: Thu, 8 Aug 2024 16:36:12 -0400 Message-ID: <20240808203612.329540-2-jspewock@iol.unh.edu> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240808203612.329540-1-jspewock@iol.unh.edu> References: <20240729203955.267942-1-jspewock@iol.unh.edu> <20240808203612.329540-1-jspewock@iol.unh.edu> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Jeremy Spewock Multiple test suites from the old DTS framework rely on being able to consume and interpret the verbose output of testpmd. The new framework doesn't have an elegant way for handling the verbose output, but test suites are starting to be written that rely on it. This patch creates a TextParser class that can be used to extract the verbose information from any testpmd output and also adjusts the `stop` method of the shell to return all output that it collected. Signed-off-by: Jeremy Spewock --- dts/framework/parser.py | 30 ++ dts/framework/remote_session/testpmd_shell.py | 405 +++++++++++++++++- dts/framework/utils.py | 1 + 3 files changed, 434 insertions(+), 2 deletions(-) diff --git a/dts/framework/parser.py b/dts/framework/parser.py index 741dfff821..0b39025a48 100644 --- a/dts/framework/parser.py +++ b/dts/framework/parser.py @@ -160,6 +160,36 @@ def _find(text: str) -> Any: return ParserFn(TextParser_fn=_find) + @staticmethod + def find_all( + pattern: str | re.Pattern[str], + flags: re.RegexFlag = re.RegexFlag(0), + ) -> ParserFn: + """Makes a parser function that finds all of the regular expression matches in the text. + + If there are no matches found in the text than None will be returned, otherwise a list + containing all matches will be returned. Patterns that contain multiple groups will pack + the matches for each group into a tuple. + + Args: + pattern: The regular expression pattern. + flags: The regular expression flags. Ignored if the given pattern is already compiled. + + Returns: + A :class:`ParserFn` that can be used as metadata for a dataclass field. + """ + if isinstance(pattern, str): + pattern = re.compile(pattern, flags) + + def _find_all(text: str) -> list[str] | None: + m = pattern.findall(text) + if len(m) == 0: + return None + + return m + + return ParserFn(TextParser_fn=_find_all) + @staticmethod def find_int( pattern: str | re.Pattern[str], diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framework/remote_session/testpmd_shell.py index 43e9f56517..7d0b5a374c 100644 --- a/dts/framework/remote_session/testpmd_shell.py +++ b/dts/framework/remote_session/testpmd_shell.py @@ -31,7 +31,7 @@ from framework.settings import SETTINGS from framework.testbed_model.cpu import LogicalCoreCount, LogicalCoreList from framework.testbed_model.sut_node import SutNode -from framework.utils import StrEnum +from framework.utils import REGEX_FOR_MAC_ADDRESS, StrEnum class TestPmdDevice: @@ -577,6 +577,377 @@ class TestPmdPortStats(TextParser): tx_bps: int = field(metadata=TextParser.find_int(r"Tx-bps:\s+(\d+)")) +class OLFlag(Flag): + """Flag representing the Packet Offload Features Flags in DPDK. + + Values in this class are taken from the definitions in the RTE MBUF core library in DPDK. + """ + + # RX flags + #: + RTE_MBUF_F_RX_RSS_HASH = auto() + + #: + RTE_MBUF_F_RX_L4_CKSUM_GOOD = auto() + #: + RTE_MBUF_F_RX_L4_CKSUM_BAD = auto() + #: + RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN = auto() + #: + RTE_MBUF_F_RX_L4_CKSUM_NONE = auto() + + #: + RTE_MBUF_F_RX_IP_CKSUM_GOOD = auto() + #: + RTE_MBUF_F_RX_IP_CKSUM_BAD = auto() + #: + RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN = auto() + #: + RTE_MBUF_F_RX_IP_CKSUM_NONE = auto() + + #: + RTE_MBUF_F_RX_OUTER_L4_CKSUM_GOOD = auto() + #: + RTE_MBUF_F_RX_OUTER_L4_CKSUM_BAD = auto() + #: + RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN = auto() + #: + RTE_MBUF_F_RX_OUTER_L4_CKSUM_INVALID = auto() + + #: + RTE_MBUF_F_RX_VLAN = auto() + #: + RTE_MBUF_F_RX_FDIR = auto() + #: + RTE_MBUF_F_RX_OUTER_IP_CKSUM_BAD = auto() + #: + RTE_MBUF_F_RX_VLAN_STRIPPED = auto() + #: RX IEEE1588 L2 Ethernet PT Packet. + RTE_MBUF_F_RX_IEEE1588_PTP = auto() + #: RX IEEE1588 L2/L4 timestamped packet. + RTE_MBUF_F_RX_IEEE1588_TMST = auto() + #: FD id reported if FDIR match. + RTE_MBUF_F_RX_FDIR_ID = auto() + #: Flexible bytes reported if FDIR match. + RTE_MBUF_F_RX_FDIR_FLX = auto() + #: + RTE_MBUF_F_RX_QINQ_STRIPPED = auto() + #: + RTE_MBUF_F_RX_LRO = auto() + #: + RTE_MBUF_F_RX_SEC_OFFLOAD_FAILED = auto() + #: + RTE_MBUF_F_RX_QINQ = auto() + + # TX flags + #: + RTE_MBUF_F_TX_OUTER_UDP_CKSUM = auto() + #: + RTE_MBUF_F_TX_UDP_SEG = auto() + #: + RTE_MBUF_F_TX_SEC_OFFLOAD = auto() + #: + RTE_MBUF_F_TX_MACSEC = auto() + + #: + RTE_MBUF_F_TX_TUNNEL_VXLAN = auto() + #: + RTE_MBUF_F_TX_TUNNEL_GRE = auto() + #: + RTE_MBUF_F_TX_TUNNEL_IPIP = auto() + #: + RTE_MBUF_F_TX_TUNNEL_GENEVE = auto() + #: + RTE_MBUF_F_TX_TUNNEL_MPLSINUDP = auto() + #: + RTE_MBUF_F_TX_TUNNEL_VXLAN_GPE = auto() + #: + RTE_MBUF_F_TX_TUNNEL_GTP = auto() + #: + RTE_MBUF_F_TX_TUNNEL_ESP = auto() + #: + RTE_MBUF_F_TX_TUNNEL_IP = auto() + #: + RTE_MBUF_F_TX_TUNNEL_UDP = auto() + + #: + RTE_MBUF_F_TX_QINQ = auto() + #: + RTE_MBUF_F_TX_TCP_SEG = auto() + #: TX IEEE1588 packet to timestamp. + RTE_MBUF_F_TX_IEEE1588_TMST = auto() + + #: + RTE_MBUF_F_TX_L4_NO_CKSUM = auto() + #: + RTE_MBUF_F_TX_TCP_CKSUM = auto() + #: + RTE_MBUF_F_TX_SCTP_CKSUM = auto() + #: + RTE_MBUF_F_TX_UDP_CKSUM = auto() + #: + RTE_MBUF_F_TX_L4_MASK = auto() + #: + RTE_MBUF_F_TX_IP_CKSUM = auto() + #: + RTE_MBUF_F_TX_OUTER_IP_CKSUM = auto() + + #: + RTE_MBUF_F_TX_IPV4 = auto() + #: + RTE_MBUF_F_TX_IPV6 = auto() + #: + RTE_MBUF_F_TX_VLAN = auto() + #: + RTE_MBUF_F_TX_OUTER_IPV4 = auto() + #: + RTE_MBUF_F_TX_OUTER_IPV6 = auto() + + @classmethod + def from_str_list(cls, arr: list[str]) -> Self: + """Makes an instance from a list containing the flag members. + + Args: + arr: A list of strings containing ol_flag values. + + Returns: + A new instance of the flag. + """ + flag = cls(0) + for name in arr: + if name in cls.__members__: + flag |= cls[name] + return flag + + @classmethod + def make_parser(cls) -> ParserFn: + """Makes a parser function. + + Returns: + ParserFn: A dictionary for the `dataclasses.field` metadata argument containing a + parser function that makes an instance of this flag from text. + """ + return TextParser.wrap( + TextParser.wrap(TextParser.find(r"ol_flags: ([^\n]+)"), str.split), + cls.from_str_list, + ) + + +class RtePTypes(Flag): + """Flag representing possible packet types in DPDK verbose output.""" + + # L2 + #: + L2_ETHER = auto() + #: + L2_ETHER_TIMESYNC = auto() + #: + L2_ETHER_ARP = auto() + #: + L2_ETHER_LLDP = auto() + #: + L2_ETHER_NSH = auto() + #: + L2_ETHER_VLAN = auto() + #: + L2_ETHER_QINQ = auto() + #: + L2_ETHER_PPPOE = auto() + #: + L2_ETHER_FCOE = auto() + #: + L2_ETHER_MPLS = auto() + #: + L2_UNKNOWN = auto() + + # L3 + #: + L3_IPV4 = auto() + #: + L3_IPV4_EXT = auto() + #: + L3_IPV6 = auto() + #: + L3_IPV4_EXT_UNKNOWN = auto() + #: + L3_IPV6_EXT = auto() + #: + L3_IPV6_EXT_UNKNOWN = auto() + #: + L3_UNKNOWN = auto() + + # L4 + #: + L4_TCP = auto() + #: + L4_UDP = auto() + #: + L4_FRAG = auto() + #: + L4_SCTP = auto() + #: + L4_ICMP = auto() + #: + L4_NONFRAG = auto() + #: + L4_IGMP = auto() + #: + L4_UNKNOWN = auto() + + # Tunnel + #: + TUNNEL_IP = auto() + #: + TUNNEL_GRE = auto() + #: + TUNNEL_VXLAN = auto() + #: + TUNNEL_NVGRE = auto() + #: + TUNNEL_GENEVE = auto() + #: + TUNNEL_GRENAT = auto() + #: + TUNNEL_GTPC = auto() + #: + TUNNEL_GTPU = auto() + #: + TUNNEL_ESP = auto() + #: + TUNNEL_L2TP = auto() + #: + TUNNEL_VXLAN_GPE = auto() + #: + TUNNEL_MPLS_IN_UDP = auto() + #: + TUNNEL_MPLS_IN_GRE = auto() + #: + TUNNEL_UNKNOWN = auto() + + # Inner L2 + #: + INNER_L2_ETHER = auto() + #: + INNER_L2_ETHER_VLAN = auto() + #: + INNER_L2_ETHER_QINQ = auto() + #: + INNER_L2_UNKNOWN = auto() + + # Inner L3 + #: + INNER_L3_IPV4 = auto() + #: + INNER_L3_IPV4_EXT = auto() + #: + INNER_L3_IPV6 = auto() + #: + INNER_L3_IPV4_EXT_UNKNOWN = auto() + #: + INNER_L3_IPV6_EXT = auto() + #: + INNER_L3_IPV6_EXT_UNKNOWN = auto() + #: + INNER_L3_UNKNOWN = auto() + + # Inner L4 + #: + INNER_L4_TCP = auto() + #: + INNER_L4_UDP = auto() + #: + INNER_L4_FRAG = auto() + #: + INNER_L4_SCTP = auto() + #: + INNER_L4_ICMP = auto() + #: + INNER_L4_NONFRAG = auto() + #: + INNER_L4_UNKNOWN = auto() + + @classmethod + def from_str_list(cls, arr: list[str]) -> Self: + """Makes an instance from a list containing the flag members. + + Args: + arr: A list of strings containing ol_flag values. + + Returns: + A new instance of the flag. + """ + flag = cls(0) + for name in arr: + if name in cls.__members__: + flag |= cls[name] + return flag + + @classmethod + def make_parser(cls, hw: bool) -> ParserFn: + """Makes a parser function. + + Args: + hw: Whether to make a parser for hardware ptypes or software ptypes. If :data:`True` + hardware ptypes will be collected, otherwise software pytpes will. + + Returns: + ParserFn: A dictionary for the `dataclasses.field` metadata argument containing a + parser function that makes an instance of this flag from text. + """ + return TextParser.wrap( + TextParser.wrap(TextParser.find(f"{'hw' if hw else 'sw'} ptype: ([^-]+)"), str.split), + cls.from_str_list, + ) + + +@dataclass +class TestPmdVerbosePacket(TextParser): + """Packet information provided by verbose output in Testpmd. + + This dataclass expects that packet information be prepended with the starting line of packet + bursts. Specifically, the line that reads "port X/queue Y: sent/received Z packets". + """ + + #: ID of the port that handled the packet. + port_id: int = field(metadata=TextParser.find_int(r"port (\d+)/queue \d+")) + #: ID of the queue that handled the packet. + queue_id: int = field(metadata=TextParser.find_int(r"port \d+/queue (\d+)")) + #: Whether the packet was received or sent by the queue/port. + was_received: bool = field(metadata=TextParser.find(r"received \d+ packets")) + #: + src_mac: str = field(metadata=TextParser.find(f"src=({REGEX_FOR_MAC_ADDRESS})")) + #: + dst_mac: str = field(metadata=TextParser.find(f"dst=({REGEX_FOR_MAC_ADDRESS})")) + #: Memory pool the packet was handled on. + pool: str = field(metadata=TextParser.find(r"pool=(\S+)")) + #: Packet type in hex. + p_type: int = field(metadata=TextParser.find_int(r"type=(0x[a-fA-F\d]+)")) + #: + length: int = field(metadata=TextParser.find_int(r"length=(\d+)")) + #: Number of segments in the packet. + nb_segs: int = field(metadata=TextParser.find_int(r"nb_segs=(\d+)")) + #: Hardware packet type. + hw_ptype: RtePTypes = field(metadata=RtePTypes.make_parser(hw=True)) + #: Software packet type. + sw_ptype: RtePTypes = field(metadata=RtePTypes.make_parser(hw=False)) + #: + l2_len: int = field(metadata=TextParser.find_int(r"l2_len=(\d+)")) + #: + ol_flags: OLFlag = field(metadata=OLFlag.make_parser()) + #: RSS has of the packet in hex. + rss_hash: int | None = field( + default=None, metadata=TextParser.find_int(r"RSS hash=(0x[a-fA-F\d]+)") + ) + #: RSS queue that handled the packet in hex. + rss_queue: int | None = field( + default=None, metadata=TextParser.find_int(r"RSS queue=(0x[a-fA-F\d]+)") + ) + #: + l3_len: int | None = field(default=None, metadata=TextParser.find_int(r"l3_len=(\d+)")) + #: + l4_len: int | None = field(default=None, metadata=TextParser.find_int(r"l4_len=(\d+)")) + + class TestPmdShell(DPDKShell): """Testpmd interactive shell. @@ -645,7 +1016,7 @@ def start(self, verify: bool = True) -> None: "Not all ports came up after starting packet forwarding in testpmd." ) - def stop(self, verify: bool = True) -> None: + def stop(self, verify: bool = True) -> str: """Stop packet forwarding. Args: @@ -656,6 +1027,9 @@ def stop(self, verify: bool = True) -> None: Raises: InteractiveCommandExecutionError: If `verify` is :data:`True` and the command to stop forwarding results in an error. + + Returns: + Output gathered from sending the stop command. """ stop_cmd_output = self.send_command("stop") if verify: @@ -665,6 +1039,7 @@ def stop(self, verify: bool = True) -> None: ): self._logger.debug(f"Failed to stop packet forwarding: \n{stop_cmd_output}") raise InteractiveCommandExecutionError("Testpmd failed to stop packet forwarding.") + return stop_cmd_output def get_devices(self) -> list[TestPmdDevice]: """Get a list of device names that are known to testpmd. @@ -806,6 +1181,32 @@ def show_port_stats(self, port_id: int) -> TestPmdPortStats: return TestPmdPortStats.parse(output) + @staticmethod + def extract_verbose_output(output: str) -> list[TestPmdVerbosePacket]: + """Extract the verbose information present in given testpmd output. + + This method extracts sections of verbose output that begin with the line + "port X/queue Y: sent/received Z packets" and end with the ol_flags of a packet. + + Args: + output: Testpmd output that contains verbose information + + Returns: + List of parsed packet information gathered from verbose information in `output`. + """ + out: list[TestPmdVerbosePacket] = [] + prev_header: str = "" + iter = re.finditer( + r"(?P
(?:port \d+/queue \d+: received \d packets)?)\s*" + r"(?Psrc=[\w\s=:-]+?ol_flags: [\w ]+)", + output, + ) + for match in iter: + if match.group("HEADER"): + prev_header = match.group("HEADER") + out.append(TestPmdVerbosePacket.parse(f"{prev_header}\n{match.group('PACKET')}")) + return out + def _close(self) -> None: """Overrides :meth:`~.interactive_shell.close`.""" self.stop() diff --git a/dts/framework/utils.py b/dts/framework/utils.py index 6b5d5a805f..9c64cf497f 100644 --- a/dts/framework/utils.py +++ b/dts/framework/utils.py @@ -27,6 +27,7 @@ from .exception import ConfigurationError REGEX_FOR_PCI_ADDRESS: str = "/[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}.[0-9]{1}/" +REGEX_FOR_MAC_ADDRESS: str = r"(?:[\da-fA-F]{2}:){5}[\da-fA-F]{2}" def expand_range(range_str: str) -> list[int]: -- 2.45.2