DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Juraj Linkeš" <juraj.linkes@pantheon.tech>
To: thomas@monjalon.net, david.marchand@redhat.com,
	jerinjacobk@gmail.com, ronan.randles@intel.com,
	Honnappa.Nagarahalli@arm.com, ohilyard@iol.unh.edu,
	lijuan.tu@intel.com
Cc: dev@dpdk.org, "Juraj Linkeš" <juraj.linkes@pantheon.tech>
Subject: [PATCH v1 2/8] dts: add locks for parallel node connections
Date: Wed, 22 Jun 2022 12:14:42 +0000	[thread overview]
Message-ID: <20220622121448.3304251-3-juraj.linkes@pantheon.tech> (raw)
In-Reply-To: <20220622121448.3304251-1-juraj.linkes@pantheon.tech>

Each lock is held per node. The lock assures that multiple connections
to the same node don't execute anything at the same time, removing the
possibility of race conditions.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
---
 dts/framework/ssh_pexpect.py | 15 +++++--
 dts/framework/utils.py       | 81 ++++++++++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+), 4 deletions(-)

diff --git a/dts/framework/ssh_pexpect.py b/dts/framework/ssh_pexpect.py
index bccc6fae94..cbdbb91b64 100644
--- a/dts/framework/ssh_pexpect.py
+++ b/dts/framework/ssh_pexpect.py
@@ -3,7 +3,7 @@
 from pexpect import pxssh
 
 from .exception import SSHConnectionException, SSHSessionDeadException, TimeoutException
-from .utils import GREEN, RED
+from .utils import GREEN, RED, parallel_lock
 
 """
 Module handles ssh sessions to TG and SUT.
@@ -12,7 +12,7 @@
 
 
 class SSHPexpect:
-    def __init__(self, node, username, password):
+    def __init__(self, node, username, password, sut_id):
         self.magic_prompt = "MAGIC PROMPT"
         self.logger = None
 
@@ -20,11 +20,18 @@ def __init__(self, node, username, password):
         self.username = username
         self.password = password
 
-        self._connect_host()
+        self._connect_host(sut_id=sut_id)
 
-    def _connect_host(self):
+    @parallel_lock(num=8)
+    def _connect_host(self, sut_id=0):
         """
         Create connection to assigned node.
+        Parameter sut_id will be used in parallel_lock thus can assure
+        isolated locks for each node.
+        Parallel ssh connections are limited to MaxStartups option in SSHD
+        configuration file. By default concurrent number is 10, so default
+        threads number is limited to 8 which less than 10. Lock number can
+        be modified along with MaxStartups value.
         """
         retry_times = 10
         try:
diff --git a/dts/framework/utils.py b/dts/framework/utils.py
index 0ffd992952..a8e739f7b2 100644
--- a/dts/framework/utils.py
+++ b/dts/framework/utils.py
@@ -2,6 +2,87 @@
 # Copyright(c) 2010-2014 Intel Corporation
 #
 
+import threading
+from functools import wraps
+
+
+def parallel_lock(num=1):
+    """
+    Wrapper function for protect parallel threads, allow multiple threads
+    share one lock. Locks are created based on function name. Thread locks are
+    separated between SUTs according to argument 'sut_id'.
+    Parameter:
+        num: Number of parallel threads for the lock
+    """
+    global locks_info
+
+    def decorate(func):
+        @wraps(func)
+        def wrapper(*args, **kwargs):
+            if "sut_id" in kwargs:
+                sut_id = kwargs["sut_id"]
+            else:
+                sut_id = 0
+
+            # in case function arguments is not correct
+            if sut_id >= len(locks_info):
+                sut_id = 0
+
+            lock_info = locks_info[sut_id]
+            uplock = lock_info["update_lock"]
+
+            name = func.__name__
+            uplock.acquire()
+
+            if name not in lock_info:
+                lock_info[name] = dict()
+                lock_info[name]["lock"] = threading.RLock()
+                lock_info[name]["current_thread"] = 1
+            else:
+                lock_info[name]["current_thread"] += 1
+
+            lock = lock_info[name]["lock"]
+
+            # make sure when owned global lock, should also own update lock
+            if lock_info[name]["current_thread"] >= num:
+                if lock._is_owned():
+                    print(
+                        RED(
+                            "SUT%d %s waiting for func lock %s"
+                            % (sut_id, threading.current_thread().name, func.__name__)
+                        )
+                    )
+                lock.acquire()
+            else:
+                uplock.release()
+
+            try:
+                ret = func(*args, **kwargs)
+            except Exception as e:
+                if not uplock._is_owned():
+                    uplock.acquire()
+
+                if lock._is_owned():
+                    lock.release()
+                    lock_info[name]["current_thread"] = 0
+                uplock.release()
+                raise e
+
+            if not uplock._is_owned():
+                uplock.acquire()
+
+            if lock._is_owned():
+                lock.release()
+                lock_info[name]["current_thread"] = 0
+
+            uplock.release()
+
+            return ret
+
+        return wrapper
+
+    return decorate
+
 
 def RED(text):
     return "\x1B[" + "31;1m" + str(text) + "\x1B[" + "0m"
-- 
2.20.1


  parent reply	other threads:[~2022-06-22 12:15 UTC|newest]

Thread overview: 105+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-06-22 12:14 [PATCH v1 0/8] dts: ssh connection to a node Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 1/8] dts: add ssh pexpect library Juraj Linkeš
2022-06-22 12:14 ` Juraj Linkeš [this message]
2022-06-22 12:14 ` [PATCH v1 3/8] dts: add ssh connection extension Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 4/8] dts: add basic logging facility Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 5/8] dts: add Node base class Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 6/8] dts: add config parser module Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 7/8] dts: add dts runtime workflow module Juraj Linkeš
2022-06-22 12:14 ` [PATCH v1 8/8] dts: add main script for running dts Juraj Linkeš
2022-07-11 14:51 ` [PATCH v2 0/8] ssh connection to a node Juraj Linkeš
2022-07-11 14:51   ` [PATCH v2 1/8] dts: add basic logging facility Juraj Linkeš
2022-07-11 14:51   ` [PATCH v2 2/8] dts: add ssh pexpect library Juraj Linkeš
2022-07-11 14:51   ` [PATCH v2 3/8] dts: add locks for parallel node connections Juraj Linkeš
2022-07-11 14:51   ` [PATCH v2 4/8] dts: add ssh connection extension Juraj Linkeš
2022-07-11 14:51   ` [PATCH v2 5/8] dts: add config parser module Juraj Linkeš
2022-07-11 14:51   ` [PATCH v2 6/8] dts: add Node base class Juraj Linkeš
2022-07-11 14:51   ` [PATCH v2 7/8] dts: add dts workflow module Juraj Linkeš
2022-07-11 14:51   ` [PATCH v2 8/8] dts: add dts executable script Juraj Linkeš
2022-07-28 10:00   ` [PATCH v3 0/9] dts: ssh connection to a node Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 1/9] dts: add project tools config Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 2/9] dts: add developer tools Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 3/9] dts: add basic logging facility Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 4/9] dts: add ssh pexpect library Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 5/9] dts: add ssh connection extension Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 6/9] dts: add config parser module Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 7/9] dts: add Node base class Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 8/9] dts: add dts workflow module Juraj Linkeš
2022-07-28 10:00     ` [PATCH v3 9/9] dts: add dts executable script Juraj Linkeš
2022-07-29 10:55     ` [PATCH v4 0/9] dts: ssh connection to a node Juraj Linkeš
2022-07-29 10:55       ` [PATCH v4 1/9] dts: add project tools config Juraj Linkeš
2022-08-10  6:30         ` Tu, Lijuan
2022-09-07 16:16         ` Bruce Richardson
2022-09-09 13:38           ` Juraj Linkeš
2022-09-09 13:52             ` Bruce Richardson
2022-09-09 14:13               ` Juraj Linkeš
2022-09-12 14:06                 ` Owen Hilyard
2022-09-12 15:15                   ` Bruce Richardson
2022-09-13 12:08                     ` Juraj Linkeš
2022-09-13 14:18                       ` Bruce Richardson
2022-09-13 19:03                     ` Honnappa Nagarahalli
2022-09-13 19:19                 ` Honnappa Nagarahalli
2022-09-14  9:37                   ` Thomas Monjalon
2022-09-14 12:55                     ` Juraj Linkeš
2022-09-14 13:11                       ` Bruce Richardson
2022-09-14 14:28                         ` Thomas Monjalon
2022-09-21 10:49                           ` Juraj Linkeš
2022-09-13 19:11             ` Honnappa Nagarahalli
2022-07-29 10:55       ` [PATCH v4 2/9] dts: add developer tools Juraj Linkeš
2022-08-10  6:30         ` Tu, Lijuan
2022-09-07 16:37         ` Bruce Richardson
2022-09-13 12:38           ` Juraj Linkeš
2022-09-13 20:38             ` Honnappa Nagarahalli
2022-09-14  7:37               ` Bruce Richardson
2022-09-14 12:45               ` Juraj Linkeš
2022-09-14 13:13                 ` Bruce Richardson
2022-09-14 14:26                   ` Thomas Monjalon
2022-09-14 19:08                     ` Honnappa Nagarahalli
2022-09-20 12:14                       ` Juraj Linkeš
2022-09-20 12:22                         ` Tu, Lijuan
2022-07-29 10:55       ` [PATCH v4 3/9] dts: add basic logging facility Juraj Linkeš
2022-08-10  6:31         ` Tu, Lijuan
2022-09-08  8:31         ` Bruce Richardson
2022-09-13 12:52           ` Juraj Linkeš
2022-09-13 23:31             ` Honnappa Nagarahalli
2022-09-14 12:51               ` Juraj Linkeš
2022-07-29 10:55       ` [PATCH v4 4/9] dts: add ssh pexpect library Juraj Linkeš
2022-08-10  6:31         ` Tu, Lijuan
2022-09-08  9:53         ` Bruce Richardson
2022-09-13 13:36           ` Juraj Linkeš
2022-09-13 14:23             ` Bruce Richardson
2022-09-13 14:59         ` Stanislaw Kardach
2022-09-13 17:23           ` Owen Hilyard
2022-09-14  0:03             ` Honnappa Nagarahalli
2022-09-14  7:42               ` Bruce Richardson
2022-09-14  7:58                 ` Stanislaw Kardach
2022-09-14 19:57                 ` Honnappa Nagarahalli
2022-09-19 14:21                   ` Owen Hilyard
2022-09-20 17:54                     ` Honnappa Nagarahalli
2022-09-21  1:01                       ` Tu, Lijuan
2022-09-21  5:37                       ` Jerin Jacob
2022-09-22  9:03                         ` Juraj Linkeš
2022-09-14  9:42         ` Stanislaw Kardach
2022-09-22  9:41           ` Juraj Linkeš
2022-09-22 14:32             ` Stanislaw Kardach
2022-09-23  7:22               ` Juraj Linkeš
2022-09-23  8:15                 ` Bruce Richardson
2022-09-23 10:18                   ` Stanislaw Kardach
2022-07-29 10:55       ` [PATCH v4 5/9] dts: add ssh connection extension Juraj Linkeš
2022-08-10  6:32         ` Tu, Lijuan
2022-09-13 17:04         ` Bruce Richardson
2022-09-13 17:32           ` Owen Hilyard
2022-09-14  7:46             ` Bruce Richardson
2022-09-14 12:02               ` Owen Hilyard
2022-09-14 13:15                 ` Bruce Richardson
2022-07-29 10:55       ` [PATCH v4 6/9] dts: add config parser module Juraj Linkeš
2022-08-10  6:33         ` Tu, Lijuan
2022-09-13 17:19         ` Bruce Richardson
2022-09-13 17:47           ` Owen Hilyard
2022-09-14  7:48             ` Bruce Richardson
2022-07-29 10:55       ` [PATCH v4 7/9] dts: add Node base class Juraj Linkeš
2022-08-10  6:33         ` Tu, Lijuan
2022-07-29 10:55       ` [PATCH v4 8/9] dts: add dts workflow module Juraj Linkeš
2022-08-10  6:34         ` Tu, Lijuan
2022-07-29 10:55       ` [PATCH v4 9/9] dts: add dts executable script Juraj Linkeš
2022-08-10  6:35         ` Tu, Lijuan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220622121448.3304251-3-juraj.linkes@pantheon.tech \
    --to=juraj.linkes@pantheon.tech \
    --cc=Honnappa.Nagarahalli@arm.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=jerinjacobk@gmail.com \
    --cc=lijuan.tu@intel.com \
    --cc=ohilyard@iol.unh.edu \
    --cc=ronan.randles@intel.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).