From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 65C57456A2; Wed, 24 Jul 2024 16:09:32 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A2A9C43445; Wed, 24 Jul 2024 16:09:26 +0200 (CEST) Received: from mail-il1-f232.google.com (mail-il1-f232.google.com [209.85.166.232]) by mails.dpdk.org (Postfix) with ESMTP id 5D09743445 for ; Wed, 24 Jul 2024 16:09:25 +0200 (CEST) Received: by mail-il1-f232.google.com with SMTP id e9e14a558f8ab-396eb81a1cfso27719915ab.2 for ; Wed, 24 Jul 2024 07:09:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1721830164; x=1722434964; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2jMdKCpCGz2EZQRwqvVFeYUYQgJ4MklAuyxUbMYnjew=; b=T4bBJ0e6ErsiIEz0yT9BceRLbYIRvTa6vsNTMDHYo2tRw040hQn1p9/Cl2ptLR2as0 BhtNoGzAi0Pe2RqmL9popTWWJ+Uv9AExorB8hFDINQgO01B1Susm1nYLD46qS7YBlQsG YqZWhZVtde5yV9noCdRkniMAvRkEXI9uoUMbc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721830164; x=1722434964; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2jMdKCpCGz2EZQRwqvVFeYUYQgJ4MklAuyxUbMYnjew=; b=soOxTAtIg6eGICLLcFt4TCXdMTXAa5vXQ7tJMa+GGp0Hj5FHegvJCl4n3oFpb7mheu r7KK4ipti36ICK+QGpqvtnI1fN9X8KvC/I1PyZH3bFDcvifoujim3vod9ztl3K0dYZzp sgn6PQ5FgSQcHBuH8KNyzvQ18cV/hpUORvFH5ip5UZbgkU3hYb/ueWpm4LxH51KTRz0a xtjlL9THsZhna/LBtvF9EVlnt9s4npGVcfUbd/wuBKISSDibztYH4YFRP83gpkAOMXV0 wqI8pNDhZnEuQAStovikOpjL1JyYdrSghW7IfW8wD0S6EY3AvW3+KKDycGjXfrljtRHK ujoQ== X-Gm-Message-State: AOJu0YzPd2QXC8DVcJCi7vkUVi7+YsqwuiANY0G4lK3bzk+oLrQNpbwB ToOvrd0upzu1CsRc5nthjmA4JxY4sJo5fGsObcwPywJ88vK46SSoyaWN6axvYB4sDUqrC6+dsUD VIxKNiucN/cUVWBptp+buFggn1ACsFetw X-Google-Smtp-Source: AGHT+IGEGpOxLNo35ghA9IH/wHF5IkBm7/JKC8Z3DjZtQYjFA8kpzkO6AXQqIUyMKr4sN3p2CIzRxdXBZx7Z X-Received: by 2002:a05:6e02:1a81:b0:396:ba86:318f with SMTP id e9e14a558f8ab-39a194d7b69mr25516365ab.24.1721830164577; Wed, 24 Jul 2024 07:09:24 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [132.177.123.84]) by smtp-relay.gmail.com with ESMTPS id e9e14a558f8ab-39a187d9dd5sm790235ab.55.2024.07.24.07.09.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Jul 2024 07:09:24 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1257::1083]) by postal.iol.unh.edu (Postfix) with ESMTP id BF6D360526E2; Wed, 24 Jul 2024 10:09:22 -0400 (EDT) From: jspewock@iol.unh.edu To: Honnappa.Nagarahalli@arm.com, npratte@iol.unh.edu, Luca.Vizzarro@arm.com, paul.szczepanek@arm.com, wathsala.vithanage@arm.com, probb@iol.unh.edu, thomas@monjalon.net, yoan.picchi@foss.arm.com, juraj.linkes@pantheon.tech Cc: dev@dpdk.org, Jeremy Spewock , Luca Vizzarro Subject: [PATCH v5 1/3] dts: Improve output gathering in interactive shells Date: Wed, 24 Jul 2024 10:08:36 -0400 Message-ID: <20240724140838.171012-2-jspewock@iol.unh.edu> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240724140838.171012-1-jspewock@iol.unh.edu> References: <20240501161623.26672-1-jspewock@iol.unh.edu> <20240724140838.171012-1-jspewock@iol.unh.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Jeremy Spewock The current implementation of consuming output from interactive shells relies on being able to find an expected prompt somewhere within the output buffer after sending the command. This is useful in situations where the prompt does not appear in the output itself, but in some practical cases (such as the starting of an XML-RPC server for scapy) the prompt exists in one of the commands sent to the shell and this can cause the command to exit early and creates a race condition between the server starting and the first command being sent to the server. This patch addresses this problem by searching for a line that strictly ends with the provided prompt, rather than one that simply contains it, so that the detection that a command is finished is more consistent. It also adds a catch to detect when a command times out before finding the prompt or the underlying SSH session dies so that the exception can be wrapped into a more explicit one and be more consistent with the non-interactive shells. Bugzilla ID: 1359 Fixes: 88489c0501af ("dts: add smoke tests") Signed-off-by: Jeremy Spewock Reviewed-by: Juraj Linkeš Reviewed-by: Luca Vizzarro Reviewed-by: Nicholas Pratte --- dts/framework/exception.py | 66 ++++++++++++------- .../single_active_interactive_shell.py | 49 ++++++++++---- 2 files changed, 79 insertions(+), 36 deletions(-) diff --git a/dts/framework/exception.py b/dts/framework/exception.py index 74fd2af3b6..f45f789825 100644 --- a/dts/framework/exception.py +++ b/dts/framework/exception.py @@ -51,26 +51,6 @@ class DTSError(Exception): severity: ClassVar[ErrorSeverity] = ErrorSeverity.GENERIC_ERR -class SSHTimeoutError(DTSError): - """The SSH execution of a command timed out.""" - - #: - severity: ClassVar[ErrorSeverity] = ErrorSeverity.SSH_ERR - _command: str - - def __init__(self, command: str): - """Define the meaning of the first argument. - - Args: - command: The executed command. - """ - self._command = command - - def __str__(self) -> str: - """Add some context to the string representation.""" - return f"{self._command} execution timed out." - - class SSHConnectionError(DTSError): """An unsuccessful SSH connection.""" @@ -98,8 +78,42 @@ def __str__(self) -> str: return message -class SSHSessionDeadError(DTSError): - """The SSH session is no longer alive.""" +class _SSHTimeoutError(DTSError): + """The execution of a command via SSH timed out. + + This class is private and meant to be raised as its interactive and non-interactive variants. + """ + + #: + severity: ClassVar[ErrorSeverity] = ErrorSeverity.SSH_ERR + _command: str + + def __init__(self, command: str): + """Define the meaning of the first argument. + + Args: + command: The executed command. + """ + self._command = command + + def __str__(self) -> str: + """Add some context to the string representation.""" + return f"{self._command} execution timed out." + + +class SSHTimeoutError(_SSHTimeoutError): + """The execution of a command on a non-interactive SSH session timed out.""" + + +class InteractiveSSHTimeoutError(_SSHTimeoutError): + """The execution of a command on an interactive SSH session timed out.""" + + +class _SSHSessionDeadError(DTSError): + """The SSH session is no longer alive. + + This class is private and meant to be raised as its interactive and non-interactive variants. + """ #: severity: ClassVar[ErrorSeverity] = ErrorSeverity.SSH_ERR @@ -118,6 +132,14 @@ def __str__(self) -> str: return f"SSH session with {self._host} has died." +class SSHSessionDeadError(_SSHSessionDeadError): + """Non-interactive SSH session has died.""" + + +class InteractiveSSHSessionDeadError(_SSHSessionDeadError): + """Interactive SSH session as died.""" + + class ConfigurationError(DTSError): """An invalid configuration.""" diff --git a/dts/framework/remote_session/single_active_interactive_shell.py b/dts/framework/remote_session/single_active_interactive_shell.py index 38094c0fe2..0e5a04885f 100644 --- a/dts/framework/remote_session/single_active_interactive_shell.py +++ b/dts/framework/remote_session/single_active_interactive_shell.py @@ -27,7 +27,11 @@ from paramiko import Channel, channel # type: ignore[import-untyped] from typing_extensions import Self -from framework.exception import InteractiveCommandExecutionError +from framework.exception import ( + InteractiveCommandExecutionError, + InteractiveSSHSessionDeadError, + InteractiveSSHTimeoutError, +) from framework.logger import DTSLogger from framework.params import Params from framework.settings import SETTINGS @@ -71,7 +75,10 @@ class SingleActiveInteractiveShell(ABC): #: Extra characters to add to the end of every command #: before sending them. This is often overridden by subclasses and is - #: most commonly an additional newline character. + #: most commonly an additional newline character. This additional newline + #: character is used to force the line that is currently awaiting input + #: into the stdout buffer so that it can be consumed and checked against + #: the expected prompt. _command_extra_chars: ClassVar[str] = "" #: Path to the executable to start the interactive application. @@ -175,6 +182,9 @@ def send_command( Raises: InteractiveCommandExecutionError: If attempting to send a command to a shell that is not currently running. + InteractiveSSHSessionDeadError: The session died while executing the command. + InteractiveSSHTimeoutError: If command was sent but prompt could not be found in + the output before the timeout. """ if not self.is_alive: raise InteractiveCommandExecutionError( @@ -183,19 +193,30 @@ def send_command( self._logger.info(f"Sending: '{command}'") if prompt is None: prompt = self._default_prompt - self._stdin.write(f"{command}{self._command_extra_chars}\n") - self._stdin.flush() out: str = "" - for line in self._stdout: - if skip_first_line: - skip_first_line = False - continue - if prompt in line and not line.rstrip().endswith( - command.rstrip() - ): # ignore line that sent command - break - out += line - self._logger.debug(f"Got output: {out}") + try: + self._stdin.write(f"{command}{self._command_extra_chars}\n") + self._stdin.flush() + for line in self._stdout: + if skip_first_line: + skip_first_line = False + continue + if line.rstrip().endswith(prompt): + break + out += line + except TimeoutError as e: + self._logger.exception(e) + self._logger.debug( + f"Prompt ({prompt}) was not found in output from command before timeout." + ) + raise InteractiveSSHTimeoutError(command) from e + except OSError as e: + self._logger.exception(e) + raise InteractiveSSHSessionDeadError( + self._node.main_session.interactive_session.hostname + ) from e + finally: + self._logger.debug(f"Got output: {out}") return out def _close(self) -> None: -- 2.45.2