Re: [dts] [PATCH v6 01/22] framework/packet: support packet sniffer to specify running machine

["Phil Yang (Arm Technology China)" <Phil.Yang@arm.com>]

Hi Lijuan,

I have sent out the 7th version of this single patch. Other patches in this series kept on the 6th version.
Please review it.

Thanks,
Phil Yang

> -----Original Message-----
> From: dts <dts-bounces@dpdk.org> On Behalf Of Phil Yang (Arm Technology
> China)
> Sent: Thursday, October 18, 2018 6:00 PM
> To: Tu, Lijuan <lijuan.tu@intel.com>; dts@dpdk.org
> Cc: nd <nd@arm.com>
> Subject: Re: [dts] [PATCH v6 01/22] framework/packet: support packet sniffer to
> specify running machine
> 
> Hi Lijuan,
> 
> Agree.
> 
> I can sanity the patch in the new version.
> 
> Thanks,
> Phil Yang
> 
> > -----Original Message-----
> > From: Tu, Lijuan <lijuan.tu@intel.com>
> > Sent: Thursday, October 18, 2018 5:15 PM
> > To: Phil Yang (Arm Technology China) <Phil.Yang@arm.com>; dts@dpdk.org
> > Cc: nd <nd@arm.com>
> > Subject: RE: [PATCH v6 01/22] framework/packet: support packet sniffer
> > to specify running machine
> >
> > Hi Phil,
> > Poll is an attribute of Popen, but not pxssh.
> >
> > To make sniff function more clear, I prefer to remove the old code
> > instead of keeping them.
> > As in dts, we always pass the parameter target, the tester
> > information, we could raise exception when finding target is None.
> >
> > > -----Original Message-----
> > > From: phil.yang@arm.com [mailto:phil.yang@arm.com]
> > > Sent: Tuesday, October 16, 2018 2:20 PM
> > > To: dts@dpdk.org
> > > Cc: nd@arm.com; Tu, Lijuan <lijuan.tu@intel.com>
> > > Subject: [PATCH v6 01/22] framework/packet: support packet sniffer
> > > to specify running machine
> > >
> > > By default, the Tester is supposed to be the server which running
> > > DTS and the packet sniff methods are running on it.
> > > However, if DTS was not running on the Tester and the Tester is
> > > another remote server, so packet sniff methods cannot sniff Tester's
> > > packets correctly.
> > >
> > > This patch adds support for packet sniff methods to specify running
> > > machine and implements sniff packet methods in class tester.
> > >
> > > 1. Add parameter to sniff_packet and load_sniff_pcap methods to
> > > specify the running machine.
> > > 2. Remove load_sniff_packets method in packet.py.
> > > 3. Add tcpdump_sniff_packets, load_tcpdump_sniff_pcap and
> > > load_tcpdump_sniff_packets for class tester with the new sniff_packet API.
> > > 4. Update tester.check_random_pkts method with
> > > tester.tcpdump_sniff_packets and tester.load_tcpdump_sniff_packets
> > > to make it execution on the Tester.
> > >
> > > Signed-off-by: Phil Yang <phil.yang@arm.com>
> > > Suggested-by: Marvin Liu <yong.liu@intel.com>
> > > Suggested-by: Tu Lijuan <lijuan.tu@intel.com>
> > > ---
> > >  framework/packet.py | 100
> > > +++++++++++++++++++++++++++-------------------------
> > >  framework/tester.py |  48 ++++++++++++++++++++++---
> > >  2 files changed, 95 insertions(+), 53 deletions(-)
> > >
> > > diff --git a/framework/packet.py b/framework/packet.py index
> > > 976b82b..517c93d 100755
> > > --- a/framework/packet.py
> > > +++ b/framework/packet.py
> > > @@ -33,7 +33,6 @@
> > >  Generic packet create, transmit and analyze module  Base on
> > > scapy(python program for packet manipulation)  """
> > > -
> > >  import os
> > >  import time
> > >  import sys
> > > @@ -61,6 +60,7 @@ from scapy.route import *  from scapy.packet
> > > import bind_layers, Raw  from scapy.sendrecv import sendp  from
> > > scapy.arch import get_if_hwaddr
> > > +from pexpect import pxssh
> > >
> > >  # load extension layers
> > >  exec_file = os.path.realpath(__file__) @@ -812,15 +812,31 @@ def
> > > get_filter_cmd(filters=[]):
> > >          return ""
> > >
> > >
> > > -def sniff_packets(intf, count=0, timeout=5, filters=[]):
> > > +def sniff_packets(intf, count=0, timeout=5, filters=[], target=[]):
> > >      """
> > >      sniff all packets for certain port in certain seconds
> > >      """
> > >      param = ""
> > >      direct_param = r"(\s+)\[ (\S+) in\|out\|inout \]"
> > > -    tcpdump_help = subprocess.check_output("tcpdump -h; echo 0",
> > > -
> > > stderr=subprocess.STDOUT,
> > > -                                           shell=True)
> > > +    remote_terminate = 0 # session remote terminate flag
> > > +
> > > +    # target[] contain the remote machine info for ssh connection
> > > +    # target[0]: username
> > > +    # target[1]: ip address
> > > +    # target[2]: pass word
> > > +    if target:
> > > +	tcpdump_help_session=pxssh.pxssh()
> > > +	tcpdump_help_session.login(server=target[1], username=target[0],
> > > +                                   password=target[2],
> > > original_prompt='[$#>]',
> > > +                                   login_timeout=20)
> > > +	tcpdump_help_session.sendline('tcpdump -h')
> > > +	tcpdump_help_session.prompt()
> > > +	tcpdump_help=tcpdump_help_session.before
> > > +	tcpdump_help_session.logout()
> > > +    else:
> > > +        tcpdump_help = subprocess.check_output("tcpdump -h; echo 0",
> > > +                                    stderr=subprocess.STDOUT,
> > > +shell=True)
> > > +
> > >      for line in tcpdump_help.split('\n'):
> > >          m = re.match(direct_param, line)
> > >          if m:
> > > @@ -850,22 +866,31 @@ def sniff_packets(intf, count=0, timeout=5,
> > > filters=[]):
> > >      else:
> > >          cmd = sniff_cmd % options
> > >
> > > -    args = shlex.split(cmd)
> > > +    if target:
> > > +        tcpdump_session=pxssh.pxssh()
> > > +        tcpdump_session.login(server=target[1], username=target[0],
> > > +                              password=target[2],
> > > original_prompt='[$#>]',
> > > +                              login_timeout=20)
> > > +        tcpdump_session.sendline(cmd)
> > > +        pipe = tcpdump_session
> > > +        remote_terminate = 1
> > > +    else:
> > > +        args = shlex.split(cmd)
> > > +        pipe = subprocess.Popen(args)
> > >
> > > -    pipe = subprocess.Popen(args)
> > >      index = str(time.time())
> > > -    SNIFF_PIDS[index] = (pipe, intf, timeout)
> > > +    SNIFF_PIDS[index] = (pipe, intf, timeout, remote_terminate)
> > >      time.sleep(1)
> > >      return index
> > >
> > >
> > > -def load_sniff_pcap(index=''):
> > > +def load_sniff_pcap(index='', target=[]):
> > >      """
> > >      Stop sniffer and return pcap file
> > >      """
> > >      child_exit = False
> > >      if index in SNIFF_PIDS.keys():
> > > -        pipe, intf, timeout = SNIFF_PIDS[index]
> > > +        pipe, intf, timeout, remote_terminate = SNIFF_PIDS[index]
> > >          time_elapse = int(time.time() - float(index))
> > >          while time_elapse < timeout:
> > >              if pipe.poll() is not None:
> > [Lijuan] Exception happened here because poll is not an attribute of pxssh.
> > > @@ -876,8 +901,22 @@ def load_sniff_pcap(index=''):
> > >              time_elapse += 1
> > >
> > >          if not child_exit:
> > > -            pipe.send_signal(signal.SIGINT)
> > > -            pipe.wait()
> > > +            if remote_terminate == 1:
> > > +                tcpdump_quit_session=pxssh.pxssh()
> > > +                tcpdump_quit_session.login(server=target[1],
> > > username=target[0],
> > > +                                           password=target[2],
> > > original_prompt='[$#>]',
> > > +                                           login_timeout=20)
> > > +                tcpdump_quit_session.sendline('kill -2 $(pidof
> > > tcpdump)')
> > > +                tcpdump_quit_session.prompt()
> > > +                tcpdump_quit_session.logout()
> > > +                # Teminate the tcpdump_session
> > > +                pipe.prompt()
> > > +                pipe.logout()
> > > +                pipe = None
> > > +
> > > +            if pipe is not None:
> > > +                pipe.send_signal(signal.SIGINT)
> > > +                pipe.wait()
> > >
> > >          # wait pcap file ready
> > >          time.sleep(1)
> > > @@ -886,42 +925,6 @@ def load_sniff_pcap(index=''):
> > >      return ""
> > >
> > >
> > > -def load_sniff_packets(index=''):
> > > -    """
> > > -    Stop sniffer and return packet objects
> > > -    """
> > > -    pkts = []
> > > -    child_exit = False
> > > -    if index in SNIFF_PIDS.keys():
> > > -        pipe, intf, timeout = SNIFF_PIDS[index]
> > > -        time_elapse = int(time.time() - float(index))
> > > -        while time_elapse < timeout:
> > > -            if pipe.poll() is not None:
> > > -                child_exit = True
> > > -                break
> > > -
> > > -            time.sleep(1)
> > > -            time_elapse += 1
> > > -
> > > -        if not child_exit:
> > > -            pipe.send_signal(signal.SIGINT)
> > > -            pipe.wait()
> > > -
> > > -        # wait pcap file ready
> > > -        time.sleep(1)
> > > -        try:
> > > -            cap_pkts = rdpcap("/tmp/sniff_%s.pcap" % intf)
> > > -            for pkt in cap_pkts:
> > > -                # packet gen should be scapy
> > > -                packet = Packet(tx_port=intf)
> > > -                packet.pktgen.assign_pkt(pkt)
> > > -                pkts.append(packet)
> > > -        except:
> > > -            pass
> > > -
> > > -    return pkts
> > > -
> > > -
> > >  def load_pcapfile(filename=""):
> > >      pkts = []
> > >      try:
> > > @@ -983,7 +986,6 @@ if __name__ == "__main__":
> > >      inst = sniff_packets("lo", timeout=5)
> > >      pkt = Packet(pkt_type='UDP')
> > >      pkt.send_pkt(tx_port='lo')
> > > -    pkts = load_sniff_packets(inst)
> > >
> > >      pkt = Packet(pkt_type='UDP', pkt_len=1500, ran_payload=True)
> > >      pkt.send_pkt(tx_port='lo')
> > > diff --git a/framework/tester.py b/framework/tester.py index
> > > a775f68..c5b705d 100755
> > > --- a/framework/tester.py
> > > +++ b/framework/tester.py
> > > @@ -35,6 +35,7 @@ Interface for bulk traffic generators.
> > >
> > >  import re
> > >  import subprocess
> > > +import os
> > >  from time import sleep
> > >  from settings import NICS, load_global_setting, PERF_SETTING  from
> > > crb import Crb @@ -560,8 +561,6 @@ class Tester(Crb):
> > >          module = __import__("packet")
> > >          pkt_c = getattr(module, "Packet")
> > >          send_f = getattr(module, "send_packets")
> > > -        sniff_f = getattr(module, "sniff_packets")
> > > -        load_f = getattr(module, "load_sniff_packets")
> > >          compare_f = getattr(module, "compare_pktload")
> > >          strip_f = getattr(module, "strip_pktload")
> > >          save_f = getattr(module, "save_packets") @@ -606,7 +605,7
> > > @@ class Tester(Crb):
> > >
> > >              # send and sniff packets
> > >              save_f(pkts=pkts, filename="/tmp/%s_tx.pcap" % txIntf)
> > > -            inst = sniff_f(intf=rxIntf, count=pktnum, timeout=timeout,
> > > filters=
> > > +            inst = self.tcpdump_sniff_packets(intf=rxIntf,
> > > + count=pktnum, timeout=timeout, filters=
> > >                  [{'layer': 'network', 'config': {'srcport': '65535'}},
> > >                   {'layer': 'network', 'config': {'dstport': '65535'}}])
> > >              rx_inst[rxport] = inst
> > > @@ -627,7 +626,7 @@ class Tester(Crb):
> > >          # Verify all packets
> > >          prev_id = -1
> > >          for txport, rxport in portList:
> > > -            recv_pkts = load_f(rx_inst[rxport])
> > > +            recv_pkts =
> > > + self.load_tcpdump_sniff_packets(rx_inst[rxport])
> > >
> > >              # only report when recevied number not matched
> > >              if len(tx_pkts[txport]) > len(recv_pkts):
> > > @@ -704,6 +703,47 @@ class Tester(Crb):
> > >              self.proc.kill()
> > >              self.proc = None
> > >
> > > +    def tcpdump_sniff_packets(self, intf, count=0, timeout=5, filters=[]):
> > > +        """
> > > +        Wrapper for packet module sniff_packets
> > > +        """
> > > +        # load functions in packet module
> > > +        module = __import__("packet")
> > > +        sniff_f = getattr(module, "sniff_packets")
> > > +
> > > +        target=[]
> > > +        target.append(self.get_username())
> > > +        target.append(self.get_ip_address())
> > > +        target.append(self.get_password())
> > > +        return sniff_f(intf, count, timeout, filters, target)
> > > +
> > > +    def load_tcpdump_sniff_pcap(self, index=''):
> > > +        """
> > > +        Wrapper for packet module load_sniff_pcap
> > > +        """
> > > +        # load functions in packet module
> > > +        module = __import__("packet")
> > > +        load_pcap_f = getattr(module, "load_sniff_pcap")
> > > +
> > > +        target=[]
> > > +        target.append(self.get_username())
> > > +        target.append(self.get_ip_address())
> > > +        target.append(self.get_password())
> > > +        pcap = load_pcap_f(index, target)
> > > +        self.session.copy_file_from(pcap)
> > > +
> > > +        return pcap.split(os.sep)[-1]
> > > +
> > > +    def load_tcpdump_sniff_packets(self, index=''):
> > > +        """
> > > +        Wrapper for packet module load_pcapfile
> > > +        """
> > > +        # load functions in packet module
> > > +        packet = __import__("packet")
> > > +        file = self.load_tcpdump_sniff_pcap(index)
> > > +
> > > +        return packet.load_pcapfile(file)
> > > +
> > >      def kill_all(self, killall=False):
> > >          """
> > >          Kill all scapy process or DPDK application on tester.
> > > --
> > > 2.7.4