test suite reviews and discussions
 help / color / mirror / Atom feed
From: Yulong Pei <yulong.pei@intel.com>
To: dts@dpdk.org
Cc: qian.q.xu@intel.com, yong.liu@intel.com
Subject: [dts] [PATCH v3 1/2] support html report for nic performance result
Date: Thu, 11 May 2017 11:13:57 +0800	[thread overview]
Message-ID: <1494472438-128286-1-git-send-email-yulong.pei@intel.com> (raw)
In-Reply-To: <86228AFD5BCD8E4EBFD2B90117B5E81E62DCEDD4@SHSMSX103.ccr.corp.intel.com>

support html report for nic performance result and support to send it
out by email.

Signed-off-by: Yulong Pei <yulong.pei@intel.com>
---
 nics/perf_report.jinja |  82 ++++++++++++++++++++++++++++++
 nics/perf_report.py    | 111 +++++++++++++++++++++++++++++++++++++++++
 nics/system_info.py    | 133 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 326 insertions(+)
 create mode 100644 nics/perf_report.jinja
 create mode 100644 nics/perf_report.py
 create mode 100644 nics/system_info.py

diff --git a/nics/perf_report.jinja b/nics/perf_report.jinja
new file mode 100644
index 0000000..1a3f903
--- /dev/null
+++ b/nics/perf_report.jinja
@@ -0,0 +1,82 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="UTF-8" />
+<title>{{ title }}</title>
+<style>
+table, th, td {
+    border: 1px solid black;
+    border-collapse: collapse;
+}
+th, td {
+    padding: 5px;
+    text-align: left; 
+    background-color: #33A1C9;  
+}
+</style>
+</head>
+
+<body>
+
+<div id="dpdk_info">
+<h2> DPDK Git Information </h2>
+<p> Branch: {{ git_info['branch'] }} </p>
+<p> commit: {{ git_info['commit'] }} </p>
+<p> Author: {{ git_info['author'] }} </p>
+<p> Date:   {{ git_info['date'] }} </p>
+<p> Summary: {{ git_info['summary'] }} </p>
+</div>
+
+<div id="nic_info">
+  <h2>NIC Detail Information</h2>
+  <table>
+    <tr>
+      <th>Item</th>
+      <th>Description</th>
+    </tr>
+    {% for key in nic_infos %}
+    <tr>
+      <td>{{ key }}</td>
+      <td>{{ nic_infos[key] }}</td>
+    </tr>
+    {% endfor %}
+  </table>
+</div>
+
+<div id="test_result">
+  <h2>Test Result:</h2>
+  <table>
+    <tr>
+      <th>Frame_size(bytes)</th>
+      <th>Throughput(Mpps)</th>
+      <th>Line rate%</th>
+    </tr>
+    {% for result in test_results %}
+    <tr>
+      <td>{{ result[0] }}</td>
+      <td>{{ result[1] }}</td>
+      <td>{{ result[2] }}</td>
+    </tr>
+    {% endfor %}
+  </table>
+</div>
+
+<div id="system_info">
+  <h2>Hardware and Software Ingredients</h2>
+  <table>
+    <tr>
+      <th>Item</th>
+      <th>Description</th>
+    </tr>
+    {% for key in system_infos %}
+    <tr>
+      <td>{{ key }}</td>
+      <td>{{ system_infos[key] }}</td>
+    </tr>
+    {% endfor %}
+  </table>
+</div>
+
+</body>
+
+</html>
diff --git a/nics/perf_report.py b/nics/perf_report.py
new file mode 100644
index 0000000..e833d6c
--- /dev/null
+++ b/nics/perf_report.py
@@ -0,0 +1,111 @@
+# BSD LICENSE
+#
+# Copyright(c) 2010-2017 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import os
+import time
+import re
+
+import jinja2
+import smtplib
+
+from email.mime.text import MIMEText
+from email.mime.multipart import MIMEMultipart
+from collections import OrderedDict
+#install GitPython
+from git import Repo
+from system_info import SystemInfo
+import utils
+
+def get_dpdk_git_info(repo_dir="/root/dpdk"):
+
+    if not os.path.exists(repo_dir):
+        return None
+
+    commit = OrderedDict()
+
+    git_repo = Repo(repo_dir)
+    assert not git_repo.bare
+
+    latest_commit = git_repo.active_branch.commit
+    commit['branch'] = str(git_repo.active_branch)
+    commit['commit'] = str(latest_commit)
+    commit['author'] = latest_commit.author
+    commit['date'] = time.ctime(latest_commit.authored_date)
+    commit['summary'] = latest_commit.summary
+    return commit
+
+def generate_html_report(file_tpl, perf_data, git_info, nic_info, system_info):
+   
+    if not os.path.exists(file_tpl):
+        return None
+
+    templateLoader = jinja2.FileSystemLoader(searchpath = "/")
+    templateEnv = jinja2.Environment(loader=templateLoader)
+    template = templateEnv.get_template(file_tpl)
+ 
+    templateVars = { "title" : "Daily Performance Test Report", \
+                     "test_results" : perf_data, \
+                     "system_infos" : system_info, \
+                     "nic_infos" : nic_info, \
+                     "git_info" : git_info \
+                   }
+
+    output = template.render(templateVars)
+    return output
+
+#sender = 'zzz@intel.com'
+#mailto = ['xxx@intel.com', 'yyy@intel.com']
+def html_message(sender, mailto, subject, html_msg):
+
+    msg = MIMEMultipart('alternative')
+    msg['From'] = sender
+    msg['to'] = ";".join(mailto)
+    msg['Subject'] = subject
+
+    msg.attach(MIMEText(html_msg, 'html'))
+
+    return msg
+
+#smtp = smtplib.SMTP('smtp.intel.com')
+def send_email(sender, mailto, message, smtp_server):
+
+    try:
+        smtp = smtplib.SMTP(smtp_server)
+        smtp.sendmail(sender, mailto, message.as_string())
+        smtp.quit()
+        print utils.GREEN("Email sent successfully.")
+    except Exception, e:
+        print utils.RED("Failed to send email " + str(e))
+
+def send_html_report(sender, mailto, subject, html_msg, smtp_server):
+    
+    message = html_message(sender, mailto, subject, html_msg)
+    send_email(sender, mailto, message, smtp_server)
diff --git a/nics/system_info.py b/nics/system_info.py
new file mode 100644
index 0000000..1f473d8
--- /dev/null
+++ b/nics/system_info.py
@@ -0,0 +1,133 @@
+# BSD LICENSE
+#
+# Copyright(c) 2010-2017 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+#   * Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#   * Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in
+#     the documentation and/or other materials provided with the
+#     distribution.
+#   * Neither the name of Intel Corporation nor the names of its
+#     contributors may be used to endorse or promote products derived
+#     from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import os
+import time
+import re
+
+from collections import OrderedDict
+#install GitPython
+from git import Repo
+
+class SystemInfo(object):
+
+    def __init__(self, dut, pci_device_id):
+	self.dut = dut
+	self.pci_device_id = pci_device_id
+        self.session = self.dut.session
+        self.system_info =  OrderedDict()
+        self.nic_info = OrderedDict()
+
+    def get_system_info(self):
+
+        board = self.session.send_expect("dmidecode -s system-product-name", "# ")
+        self.system_info["Board"] = board
+
+        processors = self.session.send_expect("dmidecode -s processor-version", "# ")
+        processor = processors.split('\r\n')[0]
+        self.system_info["CPU"] =  processor
+
+        memories = self.session.send_expect("dmidecode -t memory", "]# ")
+        channels, size, speed = self._strip_memory(memories)
+        memory_info = "Total %d MBs in %d channels @ %s" %(size, channels, speed)
+        self.system_info["Memory"] = memory_info
+
+        release = self.session.send_expect("lsb_release -d |awk -F':' '{print $2}'", "# ")
+        self.system_info["Operating system"] = release
+
+        kernel = self.session.send_expect("uname -r", "# ")
+        self.system_info["Linux kernel version"] = kernel
+        
+        gcc_info = self.session.send_expect("gcc --version", "# ")
+        gcc = gcc_info.split('\r\n')[0]
+        self.system_info["GCC version"] = gcc
+
+        return self.system_info
+        
+    def _strip_memory(self, memories):
+        """
+        Size: 8192 MB Locator: DIMM_A1 Speed: 2133 MHz
+        """
+        s_regex = r"(\s+)Size: (\d+) MB"
+        l_regex= r"(\s+)Locator: DIMM_(\w+)"
+        speed_regex = r"(\s+)Speed: (.*)"
+        size = ""
+        locate = ""
+        speed = "Unknown"
+        memory_infos = []
+        memory_channel = set()
+        lines = memories.split('\r\n')
+        total_size = 0
+        for line in lines:
+            m = re.match(s_regex, line)
+            if m:
+                size = m.group(2)
+            l_m = re.match(l_regex, line)
+            if l_m:
+                locate = l_m.group(2)
+            s_m = re.match(speed_regex, line)
+            if s_m:
+                speed = s_m.group(2)
+            if speed != "Unknown":
+                memory={"Size": size, "Locate": locate, "Speed": speed}
+                memory_infos.append(memory)
+                speed = "Unknown"
+                total_size += int(size)
+                memory_channel.add(locate[0])
+
+        return len(memory_channel), total_size, memory_infos[0]["Speed"]
+
+    def get_nic_info(self):
+
+        cmd = "cat /sys/bus/pci/devices/%s/vendor" % self.pci_device_id
+        vendor = self.session.send_expect(cmd, "# ")
+        if "No such" in vendor:
+            return None
+
+        cmd = "cat /sys/bus/pci/devices/%s/device" % self.pci_device_id
+        device = self.session.send_expect(cmd, "# ")
+        if "No such" in device:
+            return None
+
+        cmd = "ls --color=never /sys/bus/pci/devices/%s/net" % self.pci_device_id
+        interface = self.session.send_expect(cmd, "# ")
+        if "No such" in interface:
+            return None
+        cmd = "ethtool -i %s | grep --color=never firmware |awk -F':' '{print $2}'" % interface
+        firmware = self.session.send_expect(cmd, "# ")
+        if "No such" in firmware:
+            return None
+        cmd = "lspci -vmmks %s |grep -i ^device |awk -F':' '{print $2}'" % self.pci_device_id
+        self.nic_info['nic_name'] = self.session.send_expect(cmd, "# ")
+        self.nic_info['device_id'] = vendor[2:] + ':' + device[2:]
+        self.nic_info['firmware-version'] = firmware
+        return self.nic_info
+
-- 
2.5.0

  parent reply	other threads:[~2017-05-12  2:21 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-05-10  4:59 [dts] [PATCH " Yulong Pei
2017-05-10  4:59 ` [dts] [PATCH 2/2] tests: support html report for single core performance test Yulong Pei
2017-05-11  5:50   ` Liu, Yong
2017-05-11  5:42 ` [dts] [PATCH 1/2] support html report for nic performance result Liu, Yong
2017-05-10 10:27   ` [dts] [PATCH v2 " Yulong Pei
2017-05-10 10:27     ` [dts] [PATCH v2 2/2] tests: support html report for single core performance test Yulong Pei
2017-05-11  3:13   ` Yulong Pei [this message]
2017-05-11  3:13     ` [dts] [PATCH v3 " Yulong Pei
2017-05-12  3:03       ` Xu, Qian Q
2017-05-12  5:35         ` Pei, Yulong
2017-05-12  9:19     ` [dts] [PATCH v3 1/2] support html report for nic performance result Liu, Yong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1494472438-128286-1-git-send-email-yulong.pei@intel.com \
    --to=yulong.pei@intel.com \
    --cc=dts@dpdk.org \
    --cc=qian.q.xu@intel.com \
    --cc=yong.liu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).