From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9F8CE424C3; Mon, 10 Jun 2024 22:07:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 746C6402CA; Mon, 10 Jun 2024 22:07:13 +0200 (CEST) Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by mails.dpdk.org (Postfix) with ESMTP id 6F303402A9 for ; Mon, 10 Jun 2024 22:07:11 +0200 (CEST) Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-704261a1f67so406923b3a.3 for ; Mon, 10 Jun 2024 13:07:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1718050030; x=1718654830; darn=dpdk.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=a/kZgFMHAuqKRfbqVl73FF9eoALpUr/hW82YkeCczmQ=; b=QQS1pEkWAs0PjjoUxd7NrEFpS57j/EFK1UF53mUkF1943Ko9lOhWAI0mHoVkD/HteY dOoOR3TWDMkoMY1fuPCZyW7h+JFmI+iY/4bZcOpXUNF/3tsjJVFJyadv2bkE6tA0yQ+e 44/0xesDtC3+QxYmVvgAMunPiUpn1voQwbA3c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718050030; x=1718654830; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a/kZgFMHAuqKRfbqVl73FF9eoALpUr/hW82YkeCczmQ=; b=JlskSZwkQS0PPrQ96hvjMoD5GfzoExcmeU3oPoXv4ibkH6T6u4h85zeOIXrwTPxM/o Q4VqXKMvVO0bVzzX05dNi6ML1HCoFSV5Na2+cP2XHg5eHgIBgASCP9tGEWTe9DOkpbYd nZdfcH/UKL/OAHjrYFJo662Oc71FUSdp3jbrkLJSgd8/0oQ0jPcftOFy5qwf51eilruC DET3X+LOlfOnvc78SwLEguK+n4TgvDbdnJEGSXKsGNWx9w3G82bOFPkQpjkL3mbB/qze UIaE0BSjfhqBcs4hMXn4zDcq40op0s5zvaMurbFrqc4YlGD2dxigaztanZVPxGJ1MUfP Pcjw== X-Forwarded-Encrypted: i=1; AJvYcCV8YmFeTN3TTxpLKRHW/WCOOveLE1SFoPQsH6pam450K1ipkMo4LqLMTWQT17RUT+yf4SFBYK0dmJ9BMDg= X-Gm-Message-State: AOJu0Yyk41w8f0KluXdW2a3JWsuJgTOYZX4fDZ+6TOc6c2XNWgxqL6z0 kfcybTCpL4fWAx4TueK43stuYVMqVDdOPA8VfHXQeEMO8ZhNqk/mnmuus9HfyNdX5lk09U2qMv9 RG6WJo9vSDCNfQUZwifTFYmLvGPyWJNs7tStnkA== X-Google-Smtp-Source: AGHT+IFzp94cT4hWKGsyuhkQcVh7fjnbJodlEqgsf+fscK9/uvhfliZm2SnvSdW2E1Nj0HIp5wEyh5VYDTn3ATc3vsA= X-Received: by 2002:a05:6a20:12c7:b0:1b8:391f:dfa9 with SMTP id adf61e73a8af0-1b8391fe400mr2624574637.25.1718050030205; Mon, 10 Jun 2024 13:07:10 -0700 (PDT) MIME-Version: 1.0 References: <20240514201436.2496-1-jspewock@iol.unh.edu> <20240605213148.21371-1-jspewock@iol.unh.edu> <20240605213148.21371-3-jspewock@iol.unh.edu> <5639da83-922d-4872-8b90-089048e46908@pantheon.tech> In-Reply-To: <5639da83-922d-4872-8b90-089048e46908@pantheon.tech> From: Jeremy Spewock Date: Mon, 10 Jun 2024 16:06:58 -0400 Message-ID: Subject: Re: [PATCH v3 2/4] dts: add context manager for interactive shells To: =?UTF-8?Q?Juraj_Linke=C5=A1?= Cc: Honnappa.Nagarahalli@arm.com, probb@iol.unh.edu, paul.szczepanek@arm.com, thomas@monjalon.net, wathsala.vithanage@arm.com, npratte@iol.unh.edu, yoan.picchi@foss.arm.com, Luca.Vizzarro@arm.com, dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Overall, my thoughts are that it's definitely an interesting idea to make the normal shell subclass the critical. I explain more below, but basically I think it makes sense as long as we are fine with the normal shells having a context manager which likely won't really be used since it doesn't really serve a purpose for them. On Mon, Jun 10, 2024 at 10:31=E2=80=AFAM Juraj Linke=C5=A1 wrote: > > It seems to me the patch would benefit from Luca's testpmd changes, > mainly how the Shell is created. Not sure if we actually want to do that > with this series, but it sound enticing. It definitely would make it more sleek. I would vouch for it, but just because this also depends on the capabilities patch it makes me hesitant to wait on another (it already has formatting warnings without Luca's mypy changes), but I guess ideally it would get merged after Luca's so that I can rebase and use his changes here. > > > diff --git a/dts/framework/remote_session/critical_interactive_shell.py= b/dts/framework/remote_session/critical_interactive_shell.py > > new file mode 100644 > > index 0000000000..26bd891267 > > --- /dev/null > > +++ b/dts/framework/remote_session/critical_interactive_shell.py > > @@ -0,0 +1,93 @@ > > +r"""Wrapper around :class:`~.interactive_shell.InteractiveShell` that = handles critical applications. > > + > > +Critical applications are defined as applications that require explici= t clean-up before another > > +instance of some application can be started. In DPDK these are referre= d to as "primary > > +applications" and these applications take out a lock which stops other= primary applications from > > +running. > > Sounds like this is implemented in both classes. In this class, we > ensure that the instance is closed when we're done with it and in the > superclass we make sure we keep trying to connect in case a previous > instance has not yet been cleaned up. This results in a name that's not > very accurate. This is a good point. I ended up adding the retry functionality to try to address this problem first, and then still found it useful after adding the context manager so I figured I'd leave it in the top level class. In hindsight what you are saying makes sense that this doesn't need to be in applications that don't rely on others being stopped, so there isn't much of a point to having it in all interactive shells. The only difficulty with adding it here is that there would be a lot more code duplication since I would have to do the whole _start_application method over again in this class. Unless, of course, we go the other route of making the normal shell a subclass of this one, in which case the normal shell would still need a retry... I guess the easiest way to handle this would just be making the number of retries a parameter to the method and the normal shells don't allow for any. That or I could just pull out the connection part like I did with _init_channels and modify that. > > > Much like :class:`~.interactive_shell.InteractiveShell`\s, > > +:class:`CriticalInteractiveShell` is meant to be extended by subclasse= s that implement application > > +specific functionality and should never be instantiated directly. > > +""" > > + > > +from typing import Callable > > + > > +from paramiko import SSHClient # type: ignore[import] > > +from typing_extensions import Self > > + > > +from framework.logger import DTSLogger > > +from framework.settings import SETTINGS > > + > > +from .interactive_shell import InteractiveShell > > + > > + > > +class CriticalInteractiveShell(InteractiveShell): > > + """The base class for interactive critical applications. > > + > > + This class is a wrapper around :class:`~.interactive_shell.Interac= tiveShell` and should always > > This actually sounds backwards to me. This should be the base class with > InteractiveShell adding the ability to start the shell without the > context manager (either right away or explicitly after creating the objec= t). > I guess I kind of see the context manager as the additional feature rather than the ability to start and stop automatically. I actually even deliberately did it this way because I figured that using normal shells as a context manager wasn't really useful, so I didn't add the ability to. It's an interesting idea and it might shorten some of the code like you mention in other places. > If we change this, then the name (CriticalInteractiveShell) starts to > make sense. The base class is just for critical applications and the > subclass offers more, so a more generic name makes sense. The only thing I guess I was thinking of "critical" in the name being more like "important" rather than like, "necessary" or as a "base" set of applications if that makes sense. > is that we chose a different name for something already defined in DPDK > (critical vs primary; I don't see why we should use a different term). > With this in mind, I'd just call this class PrimaryAppInteractiveShell > or maybe just ContextInteractiveShell. I only really deviated from the DPDK language because I didn't want it to be like, this is a class for DPDK primary applications, as much as I was thinking of it as generically just a class that can be used for any application that there can only be one instance of at a time. I guess it will mostly just be DPDK applications in this context, so just following the DPDK way of stating it might make sense. > > > + implement the exact same functionality with the primary difference= being how the application > > + is started and stopped. In contrast to normal interactive shells, = this class does not start the > > + application upon initialization of the class. Instead, the applica= tion is handled through a > > + context manager. This allows for more explicit starting and stoppi= ng of the application, and > > + more guarantees for when the application is cleaned up which are n= ot present with normal > > + interactive shells that get cleaned up upon garbage collection. > > + """ > > + > > + _get_privileged_command: Callable[[str], str] | None > > + > > + def __init__( > > + self, > > + interactive_session: SSHClient, > > + logger: DTSLogger, > > + get_privileged_command: Callable[[str], str] | None, > > + app_args: str =3D "", > > + timeout: float =3D SETTINGS.timeout, > > + ) -> None > + """Store parameters for creating an interacti= ve shell, but > do not start the application. > > + > > + Note that this method also does not create the channel for the= application, as this is > > + something that isn't needed until the application starts. > > + > > + Args: > > + interactive_session: The SSH session dedicated to interact= ive shells. > > + logger: The logger instance this session will use. > > + get_privileged_command: A method for modifying a command t= o allow it to use > > + elevated privileges. If :data:`None`, the application = will not be started > > + with elevated privileges. > > + app_args: The command line arguments to be passed to the a= pplication on startup. > > + timeout: The timeout used for the SSH channel that is dedi= cated to this interactive > > + shell. This timeout is for collecting output, so if re= ading from the buffer > > + and no output is gathered within the timeout, an excep= tion is thrown. The default > > + value for this argument may be modified using the :opt= ion:`--timeout` command-line > > + argument or the :envvar:`DTS_TIMEOUT` environment vari= able. > > + """ > > + self._interactive_session =3D interactive_session > > + self._logger =3D logger > > + self._timeout =3D timeout > > + self._app_args =3D app_args > > + self._get_privileged_command =3D get_privileged_command > > We see here why it's backwards. We're duplicating this part of the code > and if the class relation is the other way around we can just call > super().__init__(). I agree, this method does make it seem a little backwards. > > > + > > + def __enter__(self) -> Self: > > diff --git a/dts/framework/remote_session/testpmd_shell.py b/dts/framew= ork/remote_session/testpmd_shell.py > > index 284412e82c..ca30aac264 100644 > > --- a/dts/framework/remote_session/testpmd_shell.py > > +++ b/dts/framework/remote_session/testpmd_shell.py > > > @@ -253,6 +253,15 @@ def get_capas_rxq( > > else: > > unsupported_capabilities.add(NicCapability.scatte= red_rx) > > > > + def __exit__(self, *_) -> None: > > + """Overrides :meth:`~.critical_interactive_shell.CriticalInter= activeShell.__exit__`. > > + > > + Ensures that when the context is exited packet forwarding is s= topped before closing the > > + application. > > + """ > > + self.stop() > > + super().__exit__() > > + > > I think it would more sense to add this to self.close(). Ack. > > > > > class NicCapability(Enum): > > """A mapping between capability names and the associated :class:`= TestPmdShell` methods. > > > diff --git a/dts/tests/TestSuite_pmd_buffer_scatter.py b/dts/tests/Test= Suite_pmd_buffer_scatter.py > > index 3701c47408..41f6090a7e 100644 > > --- a/dts/tests/TestSuite_pmd_buffer_scatter.py > > +++ b/dts/tests/TestSuite_pmd_buffer_scatter.py > > "--mbcache=3D200 " > > @@ -112,17 +112,21 @@ def pmd_scatter(self, mbsize: int) -> None: > > ), > > privileged=3DTrue, > > ) > > - testpmd.set_forward_mode(TestPmdForwardingModes.mac) > > - testpmd.start() > > - > > - for offset in [-1, 0, 1, 4, 5]: > > - recv_payload =3D self.scatter_pktgen_send_packet(mbsize + = offset) > > - self._logger.debug(f"Payload of scattered packet after for= warding: \n{recv_payload}") > > - self.verify( > > - ("58 " * 8).strip() in recv_payload, > > - f"Payload of scattered packet did not match expected p= ayload with offset {offset}.", > > - ) > > - testpmd.stop() > > + with testpmd_shell as testpmd: > > + testpmd.set_forward_mode(TestPmdForwardingModes.mac) > > + testpmd.start() > > + > > + for offset in [-1, 0, 1, 4, 5]: > > + recv_payload =3D self.scatter_pktgen_send_packet(mbsiz= e + offset) > > + self._logger.debug( > > + f"Payload of scattered packet after forwarding: \n= {recv_payload}" > > + ) > > + self.verify( > > + ("58 " * 8).strip() in recv_payload, > > + "Payload of scattered packet did not match expecte= d payload with offset " > > + f"{offset}.", > > + ) > > + testpmd.stop() > > This is now not needed since you added this to __exit__(), right? Right, we don't need it here. I left it just because I like being a little more explicit, but I can remove it since it is just an unneeded extra line. > > But we should consider removing this (stopping forwarding) altogether > since you mentioned we don't really need this. I'm not sure what it adds > or what the rationale is - testpmd is going to handle this just fine, > right? And we're not doing any other cleanup, we're leaving all of that > to testpmd. I don't think we should remove it entirely, there is something beneficial that can come from explicitly stopping forwarding. When the method returns None (like it does now) I agree that it is useless, but when you stop forwarding it prints the statistics for each port. I modified the stop method in another series that isn't out yet actually for adding another test suite and use its output for validation. > > > > > def test_scatter_mbuf_2048(self) -> None: > > """Run the :meth:`pmd_scatter` test with `mbsize` set to 2048= .""" >