[dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[Maxime Coquelin]

Having reference benchmarks is important in order to obtain
reproducible performance figures.

This patch describes required steps to configure a PVP setup
using testpmd in both host and guest.

Not relying on external vSwitch ease integration in a CI loop by
not being impacted by DPDK API changes.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---

Thanks to all the reviewers, this v2 should take all remarks into account.

 -- Maxime

 doc/guides/howto/img/pvp_2nics.svg           | 556 +++++++++++++++++++++++++++
 doc/guides/howto/index.rst                   |   1 +
 doc/guides/howto/pvp_reference_benchmark.rst | 395 +++++++++++++++++++
 3 files changed, 952 insertions(+)
 create mode 100644 doc/guides/howto/img/pvp_2nics.svg
 create mode 100644 doc/guides/howto/pvp_reference_benchmark.rst

diff --git a/doc/guides/howto/img/pvp_2nics.svg b/doc/guides/howto/img/pvp_2nics.svg
new file mode 100644
index 0000000..517a800
--- /dev/null
+++ b/doc/guides/howto/img/pvp_2nics.svg
@@ -0,0 +1,556 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="127.46428mm"
+   height="139.41411mm"
+   viewBox="0 0 451.64508 493.987"
+   id="svg2"
+   version="1.1"
+   inkscape:version="0.92pre2 r"
+   sodipodi:docname="pvp_2nics.svg"
+   inkscape:export-filename="/home/max/Pictures/dpdk/pvp/pvp.png"
+   inkscape:export-xdpi="90"
+   inkscape:export-ydpi="90">
+  <defs
+     id="defs4">
+    <marker
+       inkscape:stockid="Arrow1Lend"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="marker4760"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4762"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lend"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="marker4642"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4644"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lstart"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="marker10370"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         id="path10372"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(0.8,0,0,0.8,10,0)"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:isstock="true"
+       style="overflow:visible"
+       id="marker10306"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow1Lend">
+      <path
+         transform="matrix(-0.8,0,0,-0.8,-10,0)"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         id="path10308"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:isstock="true"
+       style="overflow:visible"
+       id="marker9757"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow1Lend"
+       inkscape:collect="always">
+      <path
+         transform="matrix(-0.8,0,0,-0.8,-10,0)"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         id="path9759"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lstart"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="Arrow1Lstart"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         id="path4224"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(0.8,0,0,0.8,10,0)"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lend"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="Arrow1Lend"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         id="path4227"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(-0.8,0,0,-0.8,-10,0)"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lend"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="Arrow1Lend-1"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4227-27"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lstart"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="Arrow1Lstart-9"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4224-3"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(0.8,0,0,0.8,10,0)" />
+    </marker>
+    <marker
+       inkscape:isstock="true"
+       style="overflow:visible"
+       id="marker9757-0"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow1Lend">
+      <path
+         transform="matrix(-0.8,0,0,-0.8,-10,0)"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         id="path9759-6"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lstart"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="Arrow1Lstart-6"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4224-0"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(0.8,0,0,0.8,10,0)" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lend"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="Arrow1Lend-62"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4227-6"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lstart"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="marker10370-7"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path10372-9"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(0.8,0,0,0.8,10,0)" />
+    </marker>
+    <marker
+       inkscape:isstock="true"
+       style="overflow:visible"
+       id="marker9757-2"
+       refX="0"
+       refY="0"
+       orient="auto"
+       inkscape:stockid="Arrow1Lend">
+      <path
+         transform="matrix(-0.8,0,0,-0.8,-10,0)"
+         style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         id="path9759-0"
+         inkscape:connector-curvature="0" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lstart"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="Arrow1Lstart-9-2"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4224-3-3"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(0.8,0,0,0.8,10,0)" />
+    </marker>
+    <marker
+       inkscape:stockid="Arrow1Lend"
+       orient="auto"
+       refY="0"
+       refX="0"
+       id="Arrow1Lend-1-7"
+       style="overflow:visible"
+       inkscape:isstock="true">
+      <path
+         inkscape:connector-curvature="0"
+         id="path4227-27-5"
+         d="M 0,0 5,-5 -12.5,0 5,5 Z"
+         style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+    </marker>
+  </defs>
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="0.49497475"
+     inkscape:cx="206.7485"
+     inkscape:cy="227.93958"
+     inkscape:document-units="px"
+     inkscape:current-layer="layer1"
+     showgrid="false"
+     inkscape:object-nodes="true"
+     inkscape:window-width="1916"
+     inkscape:window-height="1040"
+     inkscape:window-x="0"
+     inkscape:window-y="38"
+     inkscape:window-maximized="0"
+     inkscape:snap-grids="true"
+     inkscape:snap-to-guides="true"
+     inkscape:snap-others="false"
+     inkscape:snap-nodes="false"
+     inkscape:snap-global="false" />
+  <metadata
+     id="metadata7">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-5.3301459,-7.348317)">
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.78969002;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4140"
+       width="434.38919"
+       height="75.295639"
+       x="21.691195"
+       y="404.59354" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="421.47873"
+       y="501.3353"
+       id="text4142"><tspan
+         sodipodi:role="line"
+         id="tspan4144"
+         x="421.47873"
+         y="501.3353"
+         style="font-size:25px;line-height:125%">TE</tspan></text>
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4146"
+       width="92.934036"
+       height="32.324883"
+       x="182.57764"
+       y="372.03574" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="183.5878"
+       y="397.28958"
+       id="text4148"><tspan
+         sodipodi:role="line"
+         id="tspan4150"
+         x="183.5878"
+         y="397.28958"
+         style="font-size:25px">10G NIC</tspan></text>
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="166.92024"
+       y="451.33276"
+       id="text4152"><tspan
+         sodipodi:role="line"
+         id="tspan4154"
+         x="166.92024"
+         y="451.33276">Moongen</tspan></text>
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.39882457;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4156"
+       width="449.73071"
+       height="244.32167"
+       x="6.0295582"
+       y="29.046324" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="405.31628"
+       y="25.048317"
+       id="text4158"><tspan
+         sodipodi:role="line"
+         id="tspan4160"
+         x="405.31628"
+         y="25.048317"
+         style="font-size:25px">DUT</tspan></text>
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.14168489;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4162"
+       width="418.69415"
+       height="107.50462"
+       x="19.038134"
+       y="41.044758" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="395.38812"
+       y="66.496857"
+       id="text4164"><tspan
+         sodipodi:role="line"
+         id="tspan4166"
+         x="395.38812"
+         y="66.496857"
+         style="font-size:25px">VM</tspan></text>
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4146-3"
+       width="92.934036"
+       height="32.324883"
+       x="183.0827"
+       y="274.05093" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="184.09286"
+       y="299.30475"
+       id="text4148-6"><tspan
+         sodipodi:role="line"
+         id="tspan4150-7"
+         x="184.09286"
+         y="299.30475"
+         style="font-size:25px">10G NIC</tspan></text>
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:2.4804399;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4189"
+       width="398.00476"
+       height="65.451302"
+       x="26.901583"
+       y="82.647781" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:25px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="30.683046"
+       y="108.31288"
+       id="text4191"><tspan
+         sodipodi:role="line"
+         id="tspan4193"
+         x="30.683046"
+         y="108.31288">TestPMD</tspan><tspan
+         sodipodi:role="line"
+         x="30.683046"
+         y="139.56288"
+         id="tspan10476">(macswap)</tspan></text>
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:2.49124122;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4189-5"
+       width="397.22263"
+       height="66.152573"
+       x="29.743357"
+       y="207.6543" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:25px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="42.720772"
+       y="231.14902"
+       id="text4191-3"><tspan
+         sodipodi:role="line"
+         id="tspan4193-5"
+         x="42.720772"
+         y="231.14902">TestPMD </tspan><tspan
+         sodipodi:role="line"
+         x="42.720772"
+         y="262.39902"
+         id="tspan9747">(io)</tspan></text>
+    <path
+       style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend)"
+       d="M 202.56669,371.44487 V 308.37034"
+       id="path4218"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart)"
+       d="M 252.03098,369.63533 V 307.25568"
+       id="path4218-9"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-1)"
+       d="M 198.63811,207.44389 V 150.47507"
+       id="path4218-0"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-9)"
+       d="M 255.56859,206.9303 V 147.01008"
+       id="path4218-9-6"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker9757)"
+       d="M 199.50513,271.00921 V 207.3696"
+       id="path9749"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#marker10370)"
+       d="M 255.56859,270.56991 V 206.9303"
+       id="path9749-2"
+       inkscape:connector-curvature="0" />
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4146-36"
+       width="92.934036"
+       height="32.324883"
+       x="304.05591"
+       y="372.52954" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="305.06607"
+       y="397.78339"
+       id="text4148-7"><tspan
+         sodipodi:role="line"
+         id="tspan4150-5"
+         x="305.06607"
+         y="397.78339"
+         style="font-size:25px">10G NIC</tspan></text>
+    <rect
+       style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+       id="rect4146-3-3"
+       width="92.934036"
+       height="32.324883"
+       x="306.07623"
+       y="273.53461" />
+    <text
+       xml:space="preserve"
+       style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+       x="307.0864"
+       y="298.78842"
+       id="text4148-6-5"><tspan
+         sodipodi:role="line"
+         id="tspan4150-7-6"
+         x="307.0864"
+         y="298.78842"
+         style="font-size:25px">10G NIC</tspan></text>
+    <path
+       style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-62)"
+       d="M 323.7504,370.24835 V 307.17382"
+       id="path4218-1"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-6)"
+       d="M 373.21469,368.43881 V 306.05916"
+       id="path4218-9-8"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-1-7)"
+       d="M 324.93036,207.24894 V 150.28012"
+       id="path4218-0-9"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-9-2)"
+       d="M 381.86084,206.73535 V 146.81513"
+       id="path4218-9-6-2"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker9757-2)"
+       d="M 325.79738,270.81426 V 207.17465"
+       id="path9749-28"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#marker10370-7)"
+       d="M 381.86084,270.37496 V 206.73535"
+       id="path9749-2-9"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker4642)"
+       d="M 198.57143,148.79077 V 95.93363 h 182.85714 v 50"
+       id="path3748"
+       inkscape:connector-curvature="0" />
+    <path
+       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.01005316;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.02010632, 1.01005316;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker4760)"
+       d="m 325.70774,148.78714 v -32.84999 h -70.7012 v 30.70761"
+       id="path4634"
+       inkscape:connector-curvature="0" />
+  </g>
+</svg>
diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index 5575b27..712a9f3 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -38,3 +38,4 @@ HowTo Guides
     lm_bond_virtio_sriov
     lm_virtio_vhost_user
     flow_bifurcation
+    pvp_reference_benchmark
diff --git a/doc/guides/howto/pvp_reference_benchmark.rst b/doc/guides/howto/pvp_reference_benchmark.rst
new file mode 100644
index 0000000..c5bbd32
--- /dev/null
+++ b/doc/guides/howto/pvp_reference_benchmark.rst
@@ -0,0 +1,395 @@
+..  BSD LICENSE
+    Copyright(c) 2016 Red Hat, Inc. All rights reserved.
+    All rights reserved.
+
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions
+    are met:
+
+    * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+    * Neither the name of Intel Corporation nor the names of its
+    contributors may be used to endorse or promote products derived
+    from this software without specific prior written permission.
+
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+PVP reference benchmark setup using testpmd
+===========================================
+
+This guide lists the steps required to setup a PVP benchmark using testpmd as
+a simple forwarder between NICs and Vhost interfaces. The goal of this setup
+is to have a reference PVP benchmark without using external vSwitches (OVS,
+VPP, ...) to make it easier to obtain reproducible results and to facilitate
+continuous integration testing.
+
+The guide covers two ways of launching the VM, either by directly calling the
+QEMU command line, or by relying on libvirt. It has been tested with DPDK
+v16.11 using RHEL7 for both host and guest.
+
+
+Setup overview
+--------------
+
+.. _figure_pvp_2nics:
+
+.. figure:: img/pvp_2nics.*
+
+   PVP setup using 2 NICs
+
+In this diagram, each red arrow represents one logical core. This use-case
+requires 6 dedicated logical cores. A forwarding configuration with a single
+NIC is also possible, requiring 3 logical cores.
+
+
+Host setup
+----------
+
+In this setup, we isolate 6 cores (from CPU2 to CPU7) on the same NUMA
+node. Two cores are assigned to the VM vCPUs running testpmd and four are
+assigned to testpmd on the host.
+
+
+Host tuning
+~~~~~~~~~~~
+
+#. On BIOS, disable turbo-boost and hyper-threads.
+
+#. Append these options to Kernel command line:
+
+   .. code-block:: console
+
+      intel_pstate=disable mce=ignore_ce default_hugepagesz=1G hugepagesz=1G hugepages=6 isolcpus=2-7 rcu_nocbs=2-7 nohz_full=2-7 iommu=pt intel_iommu=on
+
+#. Disable hyper-threads at runtime if necessary or if BIOS is not accessible:
+
+   .. code-block:: console
+
+      cat /sys/devices/system/cpu/cpu*[0-9]/topology/thread_siblings_list \
+          | sort | uniq \
+          | awk -F, '{system("echo 0 > /sys/devices/system/cpu/cpu"$2"/online")}'
+
+#. Disable NMIs:
+
+   .. code-block:: console
+
+      echo 0 > /proc/sys/kernel/nmi_watchdog
+
+#. Exclude isolated CPUs from the writeback cpumask:
+
+   .. code-block:: console
+
+      echo ffffff03 > /sys/bus/workqueue/devices/writeback/cpumask
+
+#. Isolate CPUs from IRQs:
+
+   .. code-block:: console
+
+      clear_mask=0xfc #Isolate CPU2 to CPU7 from IRQs
+      for i in /proc/irq/*/smp_affinity
+      do
+        echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
+      done
+
+
+Qemu build
+~~~~~~~~~~
+
+Build Qemu:
+
+    .. code-block:: console
+
+       git clone git://git.qemu.org/qemu.git
+       cd qemu
+       mkdir bin
+       cd bin
+       ../configure --target-list=x86_64-softmmu
+
+
+DPDK build
+~~~~~~~~~~
+
+Build DPDK:
+
+   .. code-block:: console
+
+      git clone git://dpdk.org/dpdk
+      cd dpdk
+      export RTE_SDK=$PWD
+      make install T=x86_64-native-linuxapp-gcc DESTDIR=install
+
+
+Testpmd launch
+~~~~~~~~~~~~~~
+
+#. Assign NICs to DPDK:
+
+   .. code-block:: console
+
+      modprobe vfio-pci
+      $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0 0000:11:00.1
+
+   .. Note::
+
+      The Sandy Bridge family seems to have some IOMMU limitations giving poor
+      performance results. To achieve good performance on these machines
+      consider using UIO instead.
+
+#. Launch the testpmd application:
+
+   .. code-block:: console
+
+      $RTE_SDK/install/bin/testpmd -l 0,2,3,4,5 --socket-mem=1024 -n 4 \
+          --vdev 'net_vhost0,iface=/tmp/vhost-user1' \
+          --vdev 'net_vhost1,iface=/tmp/vhost-user2' -- \
+          --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1
+          --nb-cores=4 --forward-mode=io
+
+With this command, isolated CPUs 2 to 5 will be used as lcores for PMD threads. 
+
+#. In testpmd interactive mode, set the portlist to obtain the correct port
+   chaining:
+
+   .. code-block:: console
+
+      set portlist 0,2,1,3
+      start
+
+
+VM launch
+~~~~~~~~~
+
+The VM may be launched either by calling QEMU directly, or by using libvirt.
+
+#. Qemu way:
+
+Launch QEMU with two Virtio-net devices paired to the vhost-user sockets
+created by testpmd. Below example uses default Virtio-net options, but options
+may be specified, for example to disable mergeable buffers or indirect
+descriptors.
+
+   .. code-block:: console
+
+      <QEMU path>/bin/x86_64-softmmu/qemu-system-x86_64 \
+          -enable-kvm -cpu host -m 3072 -smp 3 \
+          -chardev socket,id=char0,path=/tmp/vhost-user1 \
+          -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
+          -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01,addr=0x10 \
+          -chardev socket,id=char1,path=/tmp/vhost-user2 \
+          -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \
+          -device virtio-net-pci,netdev=mynet2,mac=52:54:00:02:d9:02,addr=0x11 \
+          -object memory-backend-file,id=mem,size=3072M,mem-path=/dev/hugepages,share=on \
+          -numa node,memdev=mem -mem-prealloc \
+          -net user,hostfwd=tcp::1002$1-:22 -net nic \
+          -qmp unix:/tmp/qmp.socket,server,nowait \
+          -monitor stdio <vm_image>.qcow2
+
+You can use this `qmp-vcpu-pin <https://patchwork.kernel.org/patch/9361617/>`_
+script to pin vCPUs.
+
+It can be used as follows, for example to pin 3 vCPUs to CPUs 1, 6 and 7,
+where isolated CPUs 6 and 7 will be used as lcores for Virtio PMDs:
+
+   .. code-block:: console
+
+      export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp
+      ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7
+
+#. Libvirt way:
+
+Some initial steps are required for libvirt to be able to connect to testpmd's
+sockets.
+
+First, SELinux policy needs to be set to permissive, since testpmd is
+generally run as root (note, as reboot is required):
+
+   .. code-block:: console
+
+      cat /etc/selinux/config
+
+      # This file controls the state of SELinux on the system.
+      # SELINUX= can take one of these three values:
+      #     enforcing  - SELinux security policy is enforced.
+      #     permissive - SELinux prints warnings instead of enforcing.
+      #     disabled   - No SELinux policy is loaded.
+      SELINUX=permissive
+
+      # SELINUXTYPE= can take one of three two values:
+      #     targeted - Targeted processes are protected,
+      #     minimum  - Modification of targeted policy.
+      #                Only selected processes are protected.
+      #     mls      - Multi Level Security protection.
+      SELINUXTYPE=targeted
+
+
+Also, Qemu needs to be run as root, which has to be specified in
+``/etc/libvirt/qemu.conf``:
+
+   .. code-block:: console
+
+      user = "root"
+
+Once the domain created, the following snippet is an extract of he most
+important information (hugepages, vCPU pinning, Virtio PCI devices):
+
+   .. code-block:: xml
+
+      <domain type='kvm'>
+        <memory unit='KiB'>3145728</memory>
+        <currentMemory unit='KiB'>3145728</currentMemory>
+        <memoryBacking>
+          <hugepages>
+            <page size='1048576' unit='KiB' nodeset='0'/>
+          </hugepages>
+          <locked/>
+        </memoryBacking>
+        <vcpu placement='static'>3</vcpu>
+        <cputune>
+          <vcpupin vcpu='0' cpuset='1'/>
+          <vcpupin vcpu='1' cpuset='6'/>
+          <vcpupin vcpu='2' cpuset='7'/>
+          <emulatorpin cpuset='0'/>
+        </cputune>
+        <numatune>
+          <memory mode='strict' nodeset='0'/>
+        </numatune>
+        <os>
+          <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
+          <boot dev='hd'/>
+        </os>
+        <cpu mode='host-passthrough'>
+          <topology sockets='1' cores='3' threads='1'/>
+          <numa>
+            <cell id='0' cpus='0-2' memory='3145728' unit='KiB' memAccess='shared'/>
+          </numa>
+        </cpu>
+        <devices>
+          <interface type='vhostuser'>
+            <mac address='56:48:4f:53:54:01'/>
+            <source type='unix' path='/tmp/vhost-user1' mode='client'/>
+            <model type='virtio'/>
+            <driver name='vhost' rx_queue_size='256' />
+            <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
+          </interface>
+          <interface type='vhostuser'>
+            <mac address='56:48:4f:53:54:02'/>
+            <source type='unix' path='/tmp/vhost-user2' mode='client'/>
+            <model type='virtio'/>
+            <driver name='vhost' rx_queue_size='256' />
+            <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
+          </interface>
+        </devices>
+      </domain>
+
+
+Guest setup
+-----------
+
+
+Guest tuning
+~~~~~~~~~~~~
+
+#. Append these options to the Kernel command line:
+
+   .. code-block:: console
+
+      default_hugepagesz=1G hugepagesz=1G hugepages=1 intel_iommu=on iommu=pt isolcpus=1,2 rcu_nocbs=1,2 nohz_full=1,2
+
+#. Disable NMIs:
+
+   .. code-block:: console
+
+      echo 0 > /proc/sys/kernel/nmi_watchdog
+
+#. Exclude isolated CPU1 and CPU2 from the writeback cpumask:
+
+   .. code-block:: console
+
+      echo 1 > /sys/bus/workqueue/devices/writeback/cpumask
+
+#. Isolate CPUs from IRQs:
+
+   .. code-block:: console
+
+      clear_mask=0x6 #Isolate CPU1 and CPU2 from IRQs
+      for i in /proc/irq/*/smp_affinity
+      do
+        echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
+      done
+
+
+DPDK build
+~~~~~~~~~~
+
+Build DPDK:
+
+   .. code-block:: console
+
+      git clone git://dpdk.org/dpdk
+      cd dpdk
+      export RTE_SDK=$PWD
+      make install T=x86_64-native-linuxapp-gcc DESTDIR=install
+
+
+Testpmd launch
+~~~~~~~~~~~~~~
+
+Probe vfio module without iommu:
+
+   .. code-block:: console
+
+      modprobe -r vfio_iommu_type1
+      modprobe -r vfio
+      modprobe  vfio enable_unsafe_noiommu_mode=1
+      cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
+      modprobe vfio-pci
+
+Bind the virtio-net devices to DPDK:
+
+   .. code-block:: console
+
+      $RTE_SDK/tools/dpdk-devbind.py -b vfio-pci 0000:00:10.0 0000:00:11.0
+
+Start testpmd:
+
+   .. code-block:: console
+
+      $RTE_SDK/install/bin/testpmd -l 0,1,2 --socket-mem 1024 -n 4 \
+          --proc-type auto --file-prefix pg -- \
+          --portmask=3 --forward-mode=macswap --port-topology=chained \
+          --disable-hw-vlan --disable-rss -i --rxq=1 --txq=1 \
+          --rxd=256 --txd=256 --nb-cores=2 --auto-start
+
+Results template
+----------------
+
+Below template should be used when sharing results:
+
+   .. code-block:: none
+
+      Traffic Generator: <Test equipment (e.g. IXIA, Moongen, ...)>
+      Acceptable Loss: <n>%
+      Validation run time: <n>min
+      Host DPDK version/commit: <version, SHA-1>
+      Guest DPDK version/commit: <version, SHA-1>
+      Patches applied: <link to patchwork>
+      QEMU version/commit: <version>
+      Virtio features: <features (e.g. mrg_rxbuf='off', leave empty if default)>
+      CPU: <CPU model>, <CPU frequency>
+      NIC: <NIC model>
+      Result: <n> Mpps
-- 
2.9.3

Re: [dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[Yuanhan Liu]

On Tue, Dec 06, 2016 at 01:24:40PM +0100, Maxime Coquelin wrote:
> Having reference benchmarks is important in order to obtain
> reproducible performance figures.
> 
> This patch describes required steps to configure a PVP setup
> using testpmd in both host and guest.
> 
> Not relying on external vSwitch ease integration in a CI loop by
> not being impacted by DPDK API changes.
> 
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Reviewed-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

Thanks for the doc!

	--yliu

Re: [dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[Kevin Traynor]

On 12/06/2016 12:24 PM, Maxime Coquelin wrote:
> Having reference benchmarks is important in order to obtain
> reproducible performance figures.
> 
> This patch describes required steps to configure a PVP setup
> using testpmd in both host and guest.
> 
> Not relying on external vSwitch ease integration in a CI loop by
> not being impacted by DPDK API changes.
> 
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> 
> Thanks to all the reviewers, this v2 should take all remarks into account.
> 
>  -- Maxime

Acked-by: Kevin Traynor <ktraynor@redhat.com>

Re: [dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[Mcnamara, John]

> -----Original Message-----
> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> Sent: Tuesday, December 6, 2016 12:25 PM
> To: yuanhan.liu@linux.intel.com; thomas.monjalon@6wind.com; Mcnamara, John
> <john.mcnamara@intel.com>; Yang, Zhiyong <zhiyong.yang@intel.com>;
> ktraynor@redhat.com; dev@dpdk.org
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
> Subject: [PATCH v2] doc: introduce PVP reference benchmark
> 
> Having reference benchmarks is important in order to obtain reproducible
> performance figures.
> 
> This patch describes required steps to configure a PVP setup using testpmd
> in both host and guest.
> 
> Not relying on external vSwitch ease integration in a CI loop by not being
> impacted by DPDK API changes.
> 
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>

There is one trailing whitespace warning but apart from that:

Acked-by: John McNamara <john.mcnamara@intel.com>



> ---
> 
> Thanks to all the reviewers, this v2 should take all remarks into account.
> 
>  -- Maxime
> 
>  doc/guides/howto/img/pvp_2nics.svg           | 556
> +++++++++++++++++++++++++++
>  doc/guides/howto/index.rst                   |   1 +
>  doc/guides/howto/pvp_reference_benchmark.rst | 395 +++++++++++++++++++
>  3 files changed, 952 insertions(+)
>  create mode 100644 doc/guides/howto/img/pvp_2nics.svg
>  create mode 100644 doc/guides/howto/pvp_reference_benchmark.rst
> 
> diff --git a/doc/guides/howto/img/pvp_2nics.svg
> b/doc/guides/howto/img/pvp_2nics.svg
> new file mode 100644
> index 0000000..517a800
> --- /dev/null
> +++ b/doc/guides/howto/img/pvp_2nics.svg
> @@ -0,0 +1,556 @@
> +<?xml version="1.0" encoding="UTF-8" standalone="no"?>
> +<!-- Created with Inkscape (http://www.inkscape.org/) -->
> +
> +<svg
> +   xmlns:dc="http://purl.org/dc/elements/1.1/"
> +   xmlns:cc="http://creativecommons.org/ns#"
> +   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
> +   xmlns:svg="http://www.w3.org/2000/svg"
> +   xmlns="http://www.w3.org/2000/svg"
> +   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
> +   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
> +   width="127.46428mm"
> +   height="139.41411mm"
> +   viewBox="0 0 451.64508 493.987"
> +   id="svg2"
> +   version="1.1"
> +   inkscape:version="0.92pre2 r"
> +   sodipodi:docname="pvp_2nics.svg"
> +   inkscape:export-filename="/home/max/Pictures/dpdk/pvp/pvp.png"
> +   inkscape:export-xdpi="90"
> +   inkscape:export-ydpi="90">
> +  <defs
> +     id="defs4">
> +    <marker
> +       inkscape:stockid="Arrow1Lend"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="marker4760"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path4762"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#ff0000;fill-opacity:1;fill-
> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lend"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="marker4642"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path4644"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#ff0000;fill-opacity:1;fill-
> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lstart"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="marker10370"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         id="path10372"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#ff0000;fill-opacity:1;fill-
> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(0.8,0,0,0.8,10,0)"
> +         inkscape:connector-curvature="0" />
> +    </marker>
> +    <marker
> +       inkscape:isstock="true"
> +       style="overflow:visible"
> +       id="marker10306"
> +       refX="0"
> +       refY="0"
> +       orient="auto"
> +       inkscape:stockid="Arrow1Lend">
> +      <path
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
> +         style="fill:#ff0000;fill-opacity:1;fill-
> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         id="path10308"
> +         inkscape:connector-curvature="0" />
> +    </marker>
> +    <marker
> +       inkscape:isstock="true"
> +       style="overflow:visible"
> +       id="marker9757"
> +       refX="0"
> +       refY="0"
> +       orient="auto"
> +       inkscape:stockid="Arrow1Lend"
> +       inkscape:collect="always">
> +      <path
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
> +         style="fill:#ff0000;fill-opacity:1;fill-
> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         id="path9759"
> +         inkscape:connector-curvature="0" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lstart"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="Arrow1Lstart"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         id="path4224"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(0.8,0,0,0.8,10,0)"
> +         inkscape:connector-curvature="0" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lend"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="Arrow1Lend"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         id="path4227"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
> +         inkscape:connector-curvature="0" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lend"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="Arrow1Lend-1"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path4227-27"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lstart"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="Arrow1Lstart-9"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path4224-3"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(0.8,0,0,0.8,10,0)" />
> +    </marker>
> +    <marker
> +       inkscape:isstock="true"
> +       style="overflow:visible"
> +       id="marker9757-0"
> +       refX="0"
> +       refY="0"
> +       orient="auto"
> +       inkscape:stockid="Arrow1Lend">
> +      <path
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
> +         style="fill:#ff0000;fill-opacity:1;fill-
> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         id="path9759-6"
> +         inkscape:connector-curvature="0" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lstart"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="Arrow1Lstart-6"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path4224-0"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(0.8,0,0,0.8,10,0)" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lend"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="Arrow1Lend-62"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path4227-6"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lstart"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="marker10370-7"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path10372-9"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#ff0000;fill-opacity:1;fill-
> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(0.8,0,0,0.8,10,0)" />
> +    </marker>
> +    <marker
> +       inkscape:isstock="true"
> +       style="overflow:visible"
> +       id="marker9757-2"
> +       refX="0"
> +       refY="0"
> +       orient="auto"
> +       inkscape:stockid="Arrow1Lend">
> +      <path
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
> +         style="fill:#ff0000;fill-opacity:1;fill-
> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         id="path9759-0"
> +         inkscape:connector-curvature="0" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lstart"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="Arrow1Lstart-9-2"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path4224-3-3"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(0.8,0,0,0.8,10,0)" />
> +    </marker>
> +    <marker
> +       inkscape:stockid="Arrow1Lend"
> +       orient="auto"
> +       refY="0"
> +       refX="0"
> +       id="Arrow1Lend-1-7"
> +       style="overflow:visible"
> +       inkscape:isstock="true">
> +      <path
> +         inkscape:connector-curvature="0"
> +         id="path4227-27-5"
> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
> +         style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
> +    </marker>
> +  </defs>
> +  <sodipodi:namedview
> +     id="base"
> +     pagecolor="#ffffff"
> +     bordercolor="#666666"
> +     borderopacity="1.0"
> +     inkscape:pageopacity="0.0"
> +     inkscape:pageshadow="2"
> +     inkscape:zoom="0.49497475"
> +     inkscape:cx="206.7485"
> +     inkscape:cy="227.93958"
> +     inkscape:document-units="px"
> +     inkscape:current-layer="layer1"
> +     showgrid="false"
> +     inkscape:object-nodes="true"
> +     inkscape:window-width="1916"
> +     inkscape:window-height="1040"
> +     inkscape:window-x="0"
> +     inkscape:window-y="38"
> +     inkscape:window-maximized="0"
> +     inkscape:snap-grids="true"
> +     inkscape:snap-to-guides="true"
> +     inkscape:snap-others="false"
> +     inkscape:snap-nodes="false"
> +     inkscape:snap-global="false" />
> +  <metadata
> +     id="metadata7">
> +    <rdf:RDF>
> +      <cc:Work
> +         rdf:about="">
> +        <dc:format>image/svg+xml</dc:format>
> +        <dc:type
> +           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
> +        <dc:title></dc:title>
> +      </cc:Work>
> +    </rdf:RDF>
> +  </metadata>
> +  <g
> +     inkscape:label="Layer 1"
> +     inkscape:groupmode="layer"
> +     id="layer1"
> +     transform="translate(-5.3301459,-7.348317)">
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:1.78969002;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4140"
> +       width="434.38919"
> +       height="75.295639"
> +       x="21.691195"
> +       y="404.59354" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="421.47873"
> +       y="501.3353"
> +       id="text4142"><tspan
> +         sodipodi:role="line"
> +         id="tspan4144"
> +         x="421.47873"
> +         y="501.3353"
> +         style="font-size:25px;line-height:125%">TE</tspan></text>
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4146"
> +       width="92.934036"
> +       height="32.324883"
> +       x="182.57764"
> +       y="372.03574" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="183.5878"
> +       y="397.28958"
> +       id="text4148"><tspan
> +         sodipodi:role="line"
> +         id="tspan4150"
> +         x="183.5878"
> +         y="397.28958"
> +         style="font-size:25px">10G NIC</tspan></text>
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="166.92024"
> +       y="451.33276"
> +       id="text4152"><tspan
> +         sodipodi:role="line"
> +         id="tspan4154"
> +         x="166.92024"
> +         y="451.33276">Moongen</tspan></text>
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:1.39882457;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4156"
> +       width="449.73071"
> +       height="244.32167"
> +       x="6.0295582"
> +       y="29.046324" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="405.31628"
> +       y="25.048317"
> +       id="text4158"><tspan
> +         sodipodi:role="line"
> +         id="tspan4160"
> +         x="405.31628"
> +         y="25.048317"
> +         style="font-size:25px">DUT</tspan></text>
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:1.14168489;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4162"
> +       width="418.69415"
> +       height="107.50462"
> +       x="19.038134"
> +       y="41.044758" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="395.38812"
> +       y="66.496857"
> +       id="text4164"><tspan
> +         sodipodi:role="line"
> +         id="tspan4166"
> +         x="395.38812"
> +         y="66.496857"
> +         style="font-size:25px">VM</tspan></text>
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4146-3"
> +       width="92.934036"
> +       height="32.324883"
> +       x="183.0827"
> +       y="274.05093" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="184.09286"
> +       y="299.30475"
> +       id="text4148-6"><tspan
> +         sodipodi:role="line"
> +         id="tspan4150-7"
> +         x="184.09286"
> +         y="299.30475"
> +         style="font-size:25px">10G NIC</tspan></text>
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:2.4804399;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4189"
> +       width="398.00476"
> +       height="65.451302"
> +       x="26.901583"
> +       y="82.647781" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:25px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="30.683046"
> +       y="108.31288"
> +       id="text4191"><tspan
> +         sodipodi:role="line"
> +         id="tspan4193"
> +         x="30.683046"
> +         y="108.31288">TestPMD</tspan><tspan
> +         sodipodi:role="line"
> +         x="30.683046"
> +         y="139.56288"
> +         id="tspan10476">(macswap)</tspan></text>
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:2.49124122;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4189-5"
> +       width="397.22263"
> +       height="66.152573"
> +       x="29.743357"
> +       y="207.6543" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:25px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="42.720772"
> +       y="231.14902"
> +       id="text4191-3"><tspan
> +         sodipodi:role="line"
> +         id="tspan4193-5"
> +         x="42.720772"
> +         y="231.14902">TestPMD </tspan><tspan
> +         sodipodi:role="line"
> +         x="42.720772"
> +         y="262.39902"
> +         id="tspan9747">(io)</tspan></text>
> +    <path
> +       style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-
> linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-
> end:url(#Arrow1Lend)"
> +       d="M 202.56669,371.44487 V 308.37034"
> +       id="path4218"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-
> width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> opacity:1;marker-start:url(#Arrow1Lstart)"
> +       d="M 252.03098,369.63533 V 307.25568"
> +       id="path4218-9"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-
> linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-
> end:url(#Arrow1Lend-1)"
> +       d="M 198.63811,207.44389 V 150.47507"
> +       id="path4218-0"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-
> width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> opacity:1;marker-start:url(#Arrow1Lstart-9)"
> +       d="M 255.56859,206.9303 V 147.01008"
> +       id="path4218-9-6"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-
> opacity:1;marker-end:url(#marker9757)"
> +       d="M 199.50513,271.00921 V 207.3696"
> +       id="path9749"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-
> opacity:1;marker-start:url(#marker10370)"
> +       d="M 255.56859,270.56991 V 206.9303"
> +       id="path9749-2"
> +       inkscape:connector-curvature="0" />
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4146-36"
> +       width="92.934036"
> +       height="32.324883"
> +       x="304.05591"
> +       y="372.52954" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="305.06607"
> +       y="397.78339"
> +       id="text4148-7"><tspan
> +         sodipodi:role="line"
> +         id="tspan4150-5"
> +         x="305.06607"
> +         y="397.78339"
> +         style="font-size:25px">10G NIC</tspan></text>
> +    <rect
> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
> width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-
> dashoffset:0;stroke-opacity:1"
> +       id="rect4146-3-3"
> +       width="92.934036"
> +       height="32.324883"
> +       x="306.07623"
> +       y="273.53461" />
> +    <text
> +       xml:space="preserve"
> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
> height:125%;font-family:sans-serif;letter-spacing:0px;word-
> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
> +       x="307.0864"
> +       y="298.78842"
> +       id="text4148-6-5"><tspan
> +         sodipodi:role="line"
> +         id="tspan4150-7-6"
> +         x="307.0864"
> +         y="298.78842"
> +         style="font-size:25px">10G NIC</tspan></text>
> +    <path
> +       style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-
> linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-
> end:url(#Arrow1Lend-62)"
> +       d="M 323.7504,370.24835 V 307.17382"
> +       id="path4218-1"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-
> width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> opacity:1;marker-start:url(#Arrow1Lstart-6)"
> +       d="M 373.21469,368.43881 V 306.05916"
> +       id="path4218-9-8"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:#000000;fill-opacity:1;fill-
> rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-
> linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-
> end:url(#Arrow1Lend-1-7)"
> +       d="M 324.93036,207.24894 V 150.28012"
> +       id="path4218-0-9"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-
> width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> opacity:1;marker-start:url(#Arrow1Lstart-9-2)"
> +       d="M 381.86084,206.73535 V 146.81513"
> +       id="path4218-9-6-2"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-
> opacity:1;marker-end:url(#marker9757-2)"
> +       d="M 325.79738,270.81426 V 207.17465"
> +       id="path9749-28"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-
> opacity:1;marker-start:url(#marker10370-7)"
> +       d="M 381.86084,270.37496 V 206.73535"
> +       id="path9749-2-9"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> miterlimit:4;stroke-dasharray:2, 1;stroke-dashoffset:0;stroke-
> opacity:1;marker-end:url(#marker4642)"
> +       d="M 198.57143,148.79077 V 95.93363 h 182.85714 v 50"
> +       id="path3748"
> +       inkscape:connector-curvature="0" />
> +    <path
> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
> width:1.01005316;stroke-linecap:butt;stroke-linejoin:miter;stroke-
> miterlimit:4;stroke-dasharray:2.02010632, 1.01005316;stroke-
> dashoffset:0;stroke-opacity:1;marker-end:url(#marker4760)"
> +       d="m 325.70774,148.78714 v -32.84999 h -70.7012 v 30.70761"
> +       id="path4634"
> +       inkscape:connector-curvature="0" />
> +  </g>
> +</svg>
> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst index
> 5575b27..712a9f3 100644
> --- a/doc/guides/howto/index.rst
> +++ b/doc/guides/howto/index.rst
> @@ -38,3 +38,4 @@ HowTo Guides
>      lm_bond_virtio_sriov
>      lm_virtio_vhost_user
>      flow_bifurcation
> +    pvp_reference_benchmark
> diff --git a/doc/guides/howto/pvp_reference_benchmark.rst
> b/doc/guides/howto/pvp_reference_benchmark.rst
> new file mode 100644
> index 0000000..c5bbd32
> --- /dev/null
> +++ b/doc/guides/howto/pvp_reference_benchmark.rst
> @@ -0,0 +1,395 @@
> +..  BSD LICENSE
> +    Copyright(c) 2016 Red Hat, Inc. All rights reserved.
> +    All rights reserved.
> +
> +    Redistribution and use in source and binary forms, with or without
> +    modification, are permitted provided that the following conditions
> +    are met:
> +
> +    * Redistributions of source code must retain the above copyright
> +    notice, this list of conditions and the following disclaimer.
> +    * Redistributions in binary form must reproduce the above copyright
> +    notice, this list of conditions and the following disclaimer in
> +    the documentation and/or other materials provided with the
> +    distribution.
> +    * Neither the name of Intel Corporation nor the names of its
> +    contributors may be used to endorse or promote products derived
> +    from this software without specific prior written permission.
> +
> +    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> +    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> +    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> +    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> +    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> +    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> +    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> +    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> +    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> +    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +
> +PVP reference benchmark setup using testpmd
> +===========================================
> +
> +This guide lists the steps required to setup a PVP benchmark using
> +testpmd as a simple forwarder between NICs and Vhost interfaces. The
> +goal of this setup is to have a reference PVP benchmark without using
> +external vSwitches (OVS, VPP, ...) to make it easier to obtain
> +reproducible results and to facilitate continuous integration testing.
> +
> +The guide covers two ways of launching the VM, either by directly
> +calling the QEMU command line, or by relying on libvirt. It has been
> +tested with DPDK
> +v16.11 using RHEL7 for both host and guest.
> +
> +
> +Setup overview
> +--------------
> +
> +.. _figure_pvp_2nics:
> +
> +.. figure:: img/pvp_2nics.*
> +
> +   PVP setup using 2 NICs
> +
> +In this diagram, each red arrow represents one logical core. This
> +use-case requires 6 dedicated logical cores. A forwarding configuration
> +with a single NIC is also possible, requiring 3 logical cores.
> +
> +
> +Host setup
> +----------
> +
> +In this setup, we isolate 6 cores (from CPU2 to CPU7) on the same NUMA
> +node. Two cores are assigned to the VM vCPUs running testpmd and four
> +are assigned to testpmd on the host.
> +
> +
> +Host tuning
> +~~~~~~~~~~~
> +
> +#. On BIOS, disable turbo-boost and hyper-threads.
> +
> +#. Append these options to Kernel command line:
> +
> +   .. code-block:: console
> +
> +      intel_pstate=disable mce=ignore_ce default_hugepagesz=1G
> + hugepagesz=1G hugepages=6 isolcpus=2-7 rcu_nocbs=2-7 nohz_full=2-7
> + iommu=pt intel_iommu=on
> +
> +#. Disable hyper-threads at runtime if necessary or if BIOS is not
> accessible:
> +
> +   .. code-block:: console
> +
> +      cat /sys/devices/system/cpu/cpu*[0-9]/topology/thread_siblings_list
> \
> +          | sort | uniq \
> +          | awk -F, '{system("echo 0 >
> /sys/devices/system/cpu/cpu"$2"/online")}'
> +
> +#. Disable NMIs:
> +
> +   .. code-block:: console
> +
> +      echo 0 > /proc/sys/kernel/nmi_watchdog
> +
> +#. Exclude isolated CPUs from the writeback cpumask:
> +
> +   .. code-block:: console
> +
> +      echo ffffff03 > /sys/bus/workqueue/devices/writeback/cpumask
> +
> +#. Isolate CPUs from IRQs:
> +
> +   .. code-block:: console
> +
> +      clear_mask=0xfc #Isolate CPU2 to CPU7 from IRQs
> +      for i in /proc/irq/*/smp_affinity
> +      do
> +        echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
> +      done
> +
> +
> +Qemu build
> +~~~~~~~~~~
> +
> +Build Qemu:
> +
> +    .. code-block:: console
> +
> +       git clone git://git.qemu.org/qemu.git
> +       cd qemu
> +       mkdir bin
> +       cd bin
> +       ../configure --target-list=x86_64-softmmu
> +
> +
> +DPDK build
> +~~~~~~~~~~
> +
> +Build DPDK:
> +
> +   .. code-block:: console
> +
> +      git clone git://dpdk.org/dpdk
> +      cd dpdk
> +      export RTE_SDK=$PWD
> +      make install T=x86_64-native-linuxapp-gcc DESTDIR=install
> +
> +
> +Testpmd launch
> +~~~~~~~~~~~~~~
> +
> +#. Assign NICs to DPDK:
> +
> +   .. code-block:: console
> +
> +      modprobe vfio-pci
> +      $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0
> + 0000:11:00.1
> +
> +   .. Note::
> +
> +      The Sandy Bridge family seems to have some IOMMU limitations giving
> poor
> +      performance results. To achieve good performance on these machines
> +      consider using UIO instead.
> +
> +#. Launch the testpmd application:
> +
> +   .. code-block:: console
> +
> +      $RTE_SDK/install/bin/testpmd -l 0,2,3,4,5 --socket-mem=1024 -n 4 \
> +          --vdev 'net_vhost0,iface=/tmp/vhost-user1' \
> +          --vdev 'net_vhost1,iface=/tmp/vhost-user2' -- \
> +          --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1
> +          --nb-cores=4 --forward-mode=io
> +
> +With this command, isolated CPUs 2 to 5 will be used as lcores for PMD
> threads.
> +
> +#. In testpmd interactive mode, set the portlist to obtain the correct
> port
> +   chaining:
> +
> +   .. code-block:: console
> +
> +      set portlist 0,2,1,3
> +      start
> +
> +
> +VM launch
> +~~~~~~~~~
> +
> +The VM may be launched either by calling QEMU directly, or by using
> libvirt.
> +
> +#. Qemu way:
> +
> +Launch QEMU with two Virtio-net devices paired to the vhost-user
> +sockets created by testpmd. Below example uses default Virtio-net
> +options, but options may be specified, for example to disable mergeable
> +buffers or indirect descriptors.
> +
> +   .. code-block:: console
> +
> +      <QEMU path>/bin/x86_64-softmmu/qemu-system-x86_64 \
> +          -enable-kvm -cpu host -m 3072 -smp 3 \
> +          -chardev socket,id=char0,path=/tmp/vhost-user1 \
> +          -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
> +          -device virtio-net-
> pci,netdev=mynet1,mac=52:54:00:02:d9:01,addr=0x10 \
> +          -chardev socket,id=char1,path=/tmp/vhost-user2 \
> +          -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \
> +          -device virtio-net-
> pci,netdev=mynet2,mac=52:54:00:02:d9:02,addr=0x11 \
> +          -object memory-backend-file,id=mem,size=3072M,mem-
> path=/dev/hugepages,share=on \
> +          -numa node,memdev=mem -mem-prealloc \
> +          -net user,hostfwd=tcp::1002$1-:22 -net nic \
> +          -qmp unix:/tmp/qmp.socket,server,nowait \
> +          -monitor stdio <vm_image>.qcow2
> +
> +You can use this `qmp-vcpu-pin
> +<https://patchwork.kernel.org/patch/9361617/>`_
> +script to pin vCPUs.
> +
> +It can be used as follows, for example to pin 3 vCPUs to CPUs 1, 6 and
> +7, where isolated CPUs 6 and 7 will be used as lcores for Virtio PMDs:
> +
> +   .. code-block:: console
> +
> +      export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp
> +      ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7
> +
> +#. Libvirt way:
> +
> +Some initial steps are required for libvirt to be able to connect to
> +testpmd's sockets.
> +
> +First, SELinux policy needs to be set to permissive, since testpmd is
> +generally run as root (note, as reboot is required):
> +
> +   .. code-block:: console
> +
> +      cat /etc/selinux/config
> +
> +      # This file controls the state of SELinux on the system.
> +      # SELINUX= can take one of these three values:
> +      #     enforcing  - SELinux security policy is enforced.
> +      #     permissive - SELinux prints warnings instead of enforcing.
> +      #     disabled   - No SELinux policy is loaded.
> +      SELINUX=permissive
> +
> +      # SELINUXTYPE= can take one of three two values:
> +      #     targeted - Targeted processes are protected,
> +      #     minimum  - Modification of targeted policy.
> +      #                Only selected processes are protected.
> +      #     mls      - Multi Level Security protection.
> +      SELINUXTYPE=targeted
> +
> +
> +Also, Qemu needs to be run as root, which has to be specified in
> +``/etc/libvirt/qemu.conf``:
> +
> +   .. code-block:: console
> +
> +      user = "root"
> +
> +Once the domain created, the following snippet is an extract of he most
> +important information (hugepages, vCPU pinning, Virtio PCI devices):
> +
> +   .. code-block:: xml
> +
> +      <domain type='kvm'>
> +        <memory unit='KiB'>3145728</memory>
> +        <currentMemory unit='KiB'>3145728</currentMemory>
> +        <memoryBacking>
> +          <hugepages>
> +            <page size='1048576' unit='KiB' nodeset='0'/>
> +          </hugepages>
> +          <locked/>
> +        </memoryBacking>
> +        <vcpu placement='static'>3</vcpu>
> +        <cputune>
> +          <vcpupin vcpu='0' cpuset='1'/>
> +          <vcpupin vcpu='1' cpuset='6'/>
> +          <vcpupin vcpu='2' cpuset='7'/>
> +          <emulatorpin cpuset='0'/>
> +        </cputune>
> +        <numatune>
> +          <memory mode='strict' nodeset='0'/>
> +        </numatune>
> +        <os>
> +          <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
> +          <boot dev='hd'/>
> +        </os>
> +        <cpu mode='host-passthrough'>
> +          <topology sockets='1' cores='3' threads='1'/>
> +          <numa>
> +            <cell id='0' cpus='0-2' memory='3145728' unit='KiB'
> memAccess='shared'/>
> +          </numa>
> +        </cpu>
> +        <devices>
> +          <interface type='vhostuser'>
> +            <mac address='56:48:4f:53:54:01'/>
> +            <source type='unix' path='/tmp/vhost-user1' mode='client'/>
> +            <model type='virtio'/>
> +            <driver name='vhost' rx_queue_size='256' />
> +            <address type='pci' domain='0x0000' bus='0x00' slot='0x10'
> function='0x0'/>
> +          </interface>
> +          <interface type='vhostuser'>
> +            <mac address='56:48:4f:53:54:02'/>
> +            <source type='unix' path='/tmp/vhost-user2' mode='client'/>
> +            <model type='virtio'/>
> +            <driver name='vhost' rx_queue_size='256' />
> +            <address type='pci' domain='0x0000' bus='0x00' slot='0x11'
> function='0x0'/>
> +          </interface>
> +        </devices>
> +      </domain>
> +
> +
> +Guest setup
> +-----------
> +
> +
> +Guest tuning
> +~~~~~~~~~~~~
> +
> +#. Append these options to the Kernel command line:
> +
> +   .. code-block:: console
> +
> +      default_hugepagesz=1G hugepagesz=1G hugepages=1 intel_iommu=on
> + iommu=pt isolcpus=1,2 rcu_nocbs=1,2 nohz_full=1,2
> +
> +#. Disable NMIs:
> +
> +   .. code-block:: console
> +
> +      echo 0 > /proc/sys/kernel/nmi_watchdog
> +
> +#. Exclude isolated CPU1 and CPU2 from the writeback cpumask:
> +
> +   .. code-block:: console
> +
> +      echo 1 > /sys/bus/workqueue/devices/writeback/cpumask
> +
> +#. Isolate CPUs from IRQs:
> +
> +   .. code-block:: console
> +
> +      clear_mask=0x6 #Isolate CPU1 and CPU2 from IRQs
> +      for i in /proc/irq/*/smp_affinity
> +      do
> +        echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
> +      done
> +
> +
> +DPDK build
> +~~~~~~~~~~
> +
> +Build DPDK:
> +
> +   .. code-block:: console
> +
> +      git clone git://dpdk.org/dpdk
> +      cd dpdk
> +      export RTE_SDK=$PWD
> +      make install T=x86_64-native-linuxapp-gcc DESTDIR=install
> +
> +
> +Testpmd launch
> +~~~~~~~~~~~~~~
> +
> +Probe vfio module without iommu:
> +
> +   .. code-block:: console
> +
> +      modprobe -r vfio_iommu_type1
> +      modprobe -r vfio
> +      modprobe  vfio enable_unsafe_noiommu_mode=1
> +      cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
> +      modprobe vfio-pci
> +
> +Bind the virtio-net devices to DPDK:
> +
> +   .. code-block:: console
> +
> +      $RTE_SDK/tools/dpdk-devbind.py -b vfio-pci 0000:00:10.0
> + 0000:00:11.0
> +
> +Start testpmd:
> +
> +   .. code-block:: console
> +
> +      $RTE_SDK/install/bin/testpmd -l 0,1,2 --socket-mem 1024 -n 4 \
> +          --proc-type auto --file-prefix pg -- \
> +          --portmask=3 --forward-mode=macswap --port-topology=chained \
> +          --disable-hw-vlan --disable-rss -i --rxq=1 --txq=1 \
> +          --rxd=256 --txd=256 --nb-cores=2 --auto-start
> +
> +Results template
> +----------------
> +
> +Below template should be used when sharing results:
> +
> +   .. code-block:: none
> +
> +      Traffic Generator: <Test equipment (e.g. IXIA, Moongen, ...)>
> +      Acceptable Loss: <n>%
> +      Validation run time: <n>min
> +      Host DPDK version/commit: <version, SHA-1>
> +      Guest DPDK version/commit: <version, SHA-1>
> +      Patches applied: <link to patchwork>
> +      QEMU version/commit: <version>
> +      Virtio features: <features (e.g. mrg_rxbuf='off', leave empty if
> default)>
> +      CPU: <CPU model>, <CPU frequency>
> +      NIC: <NIC model>
> +      Result: <n> Mpps
> --
> 2.9.3

Re: [dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[Maxime Coquelin]

On 12/07/2016 10:07 PM, Mcnamara, John wrote:
>> -----Original Message-----
>> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
>> Sent: Tuesday, December 6, 2016 12:25 PM
>> To: yuanhan.liu@linux.intel.com; thomas.monjalon@6wind.com; Mcnamara, John
>> <john.mcnamara@intel.com>; Yang, Zhiyong <zhiyong.yang@intel.com>;
>> ktraynor@redhat.com; dev@dpdk.org
>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
>> Subject: [PATCH v2] doc: introduce PVP reference benchmark
>>
>> Having reference benchmarks is important in order to obtain reproducible
>> performance figures.
>>
>> This patch describes required steps to configure a PVP setup using testpmd
>> in both host and guest.
>>
>> Not relying on external vSwitch ease integration in a CI loop by not being
>> impacted by DPDK API changes.
>>
>> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>
> There is one trailing whitespace warning but apart from that:
>
> Acked-by: John McNamara <john.mcnamara@intel.com>

Thanks John,

Do you want me to send a v3, fixing the trailing whitespace &
collecting the acks?

  -- Maxime

>
>
>
>> ---
>>
>> Thanks to all the reviewers, this v2 should take all remarks into account.
>>
>>  -- Maxime
>>
>>  doc/guides/howto/img/pvp_2nics.svg           | 556
>> +++++++++++++++++++++++++++
>>  doc/guides/howto/index.rst                   |   1 +
>>  doc/guides/howto/pvp_reference_benchmark.rst | 395 +++++++++++++++++++
>>  3 files changed, 952 insertions(+)
>>  create mode 100644 doc/guides/howto/img/pvp_2nics.svg
>>  create mode 100644 doc/guides/howto/pvp_reference_benchmark.rst
>>
>> diff --git a/doc/guides/howto/img/pvp_2nics.svg
>> b/doc/guides/howto/img/pvp_2nics.svg
>> new file mode 100644
>> index 0000000..517a800
>> --- /dev/null
>> +++ b/doc/guides/howto/img/pvp_2nics.svg
>> @@ -0,0 +1,556 @@
>> +<?xml version="1.0" encoding="UTF-8" standalone="no"?>
>> +<!-- Created with Inkscape (http://www.inkscape.org/) -->
>> +
>> +<svg
>> +   xmlns:dc="http://purl.org/dc/elements/1.1/"
>> +   xmlns:cc="http://creativecommons.org/ns#"
>> +   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
>> +   xmlns:svg="http://www.w3.org/2000/svg"
>> +   xmlns="http://www.w3.org/2000/svg"
>> +   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
>> +   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
>> +   width="127.46428mm"
>> +   height="139.41411mm"
>> +   viewBox="0 0 451.64508 493.987"
>> +   id="svg2"
>> +   version="1.1"
>> +   inkscape:version="0.92pre2 r"
>> +   sodipodi:docname="pvp_2nics.svg"
>> +   inkscape:export-filename="/home/max/Pictures/dpdk/pvp/pvp.png"
>> +   inkscape:export-xdpi="90"
>> +   inkscape:export-ydpi="90">
>> +  <defs
>> +     id="defs4">
>> +    <marker
>> +       inkscape:stockid="Arrow1Lend"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="marker4760"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path4762"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#ff0000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lend"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="marker4642"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path4644"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#ff0000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lstart"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="marker10370"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         id="path10372"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#ff0000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(0.8,0,0,0.8,10,0)"
>> +         inkscape:connector-curvature="0" />
>> +    </marker>
>> +    <marker
>> +       inkscape:isstock="true"
>> +       style="overflow:visible"
>> +       id="marker10306"
>> +       refX="0"
>> +       refY="0"
>> +       orient="auto"
>> +       inkscape:stockid="Arrow1Lend">
>> +      <path
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
>> +         style="fill:#ff0000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         id="path10308"
>> +         inkscape:connector-curvature="0" />
>> +    </marker>
>> +    <marker
>> +       inkscape:isstock="true"
>> +       style="overflow:visible"
>> +       id="marker9757"
>> +       refX="0"
>> +       refY="0"
>> +       orient="auto"
>> +       inkscape:stockid="Arrow1Lend"
>> +       inkscape:collect="always">
>> +      <path
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
>> +         style="fill:#ff0000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         id="path9759"
>> +         inkscape:connector-curvature="0" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lstart"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="Arrow1Lstart"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         id="path4224"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(0.8,0,0,0.8,10,0)"
>> +         inkscape:connector-curvature="0" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lend"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="Arrow1Lend"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         id="path4227"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
>> +         inkscape:connector-curvature="0" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lend"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="Arrow1Lend-1"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path4227-27"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lstart"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="Arrow1Lstart-9"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path4224-3"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(0.8,0,0,0.8,10,0)" />
>> +    </marker>
>> +    <marker
>> +       inkscape:isstock="true"
>> +       style="overflow:visible"
>> +       id="marker9757-0"
>> +       refX="0"
>> +       refY="0"
>> +       orient="auto"
>> +       inkscape:stockid="Arrow1Lend">
>> +      <path
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
>> +         style="fill:#ff0000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         id="path9759-6"
>> +         inkscape:connector-curvature="0" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lstart"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="Arrow1Lstart-6"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path4224-0"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(0.8,0,0,0.8,10,0)" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lend"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="Arrow1Lend-62"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path4227-6"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lstart"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="marker10370-7"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path10372-9"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#ff0000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(0.8,0,0,0.8,10,0)" />
>> +    </marker>
>> +    <marker
>> +       inkscape:isstock="true"
>> +       style="overflow:visible"
>> +       id="marker9757-2"
>> +       refX="0"
>> +       refY="0"
>> +       orient="auto"
>> +       inkscape:stockid="Arrow1Lend">
>> +      <path
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)"
>> +         style="fill:#ff0000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         id="path9759-0"
>> +         inkscape:connector-curvature="0" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lstart"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="Arrow1Lstart-9-2"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path4224-3-3"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(0.8,0,0,0.8,10,0)" />
>> +    </marker>
>> +    <marker
>> +       inkscape:stockid="Arrow1Lend"
>> +       orient="auto"
>> +       refY="0"
>> +       refX="0"
>> +       id="Arrow1Lend-1-7"
>> +       style="overflow:visible"
>> +       inkscape:isstock="true">
>> +      <path
>> +         inkscape:connector-curvature="0"
>> +         id="path4227-27-5"
>> +         d="M 0,0 5,-5 -12.5,0 5,5 Z"
>> +         style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
>> +         transform="matrix(-0.8,0,0,-0.8,-10,0)" />
>> +    </marker>
>> +  </defs>
>> +  <sodipodi:namedview
>> +     id="base"
>> +     pagecolor="#ffffff"
>> +     bordercolor="#666666"
>> +     borderopacity="1.0"
>> +     inkscape:pageopacity="0.0"
>> +     inkscape:pageshadow="2"
>> +     inkscape:zoom="0.49497475"
>> +     inkscape:cx="206.7485"
>> +     inkscape:cy="227.93958"
>> +     inkscape:document-units="px"
>> +     inkscape:current-layer="layer1"
>> +     showgrid="false"
>> +     inkscape:object-nodes="true"
>> +     inkscape:window-width="1916"
>> +     inkscape:window-height="1040"
>> +     inkscape:window-x="0"
>> +     inkscape:window-y="38"
>> +     inkscape:window-maximized="0"
>> +     inkscape:snap-grids="true"
>> +     inkscape:snap-to-guides="true"
>> +     inkscape:snap-others="false"
>> +     inkscape:snap-nodes="false"
>> +     inkscape:snap-global="false" />
>> +  <metadata
>> +     id="metadata7">
>> +    <rdf:RDF>
>> +      <cc:Work
>> +         rdf:about="">
>> +        <dc:format>image/svg+xml</dc:format>
>> +        <dc:type
>> +           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
>> +        <dc:title></dc:title>
>> +      </cc:Work>
>> +    </rdf:RDF>
>> +  </metadata>
>> +  <g
>> +     inkscape:label="Layer 1"
>> +     inkscape:groupmode="layer"
>> +     id="layer1"
>> +     transform="translate(-5.3301459,-7.348317)">
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:1.78969002;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4140"
>> +       width="434.38919"
>> +       height="75.295639"
>> +       x="21.691195"
>> +       y="404.59354" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="421.47873"
>> +       y="501.3353"
>> +       id="text4142"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4144"
>> +         x="421.47873"
>> +         y="501.3353"
>> +         style="font-size:25px;line-height:125%">TE</tspan></text>
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4146"
>> +       width="92.934036"
>> +       height="32.324883"
>> +       x="182.57764"
>> +       y="372.03574" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="183.5878"
>> +       y="397.28958"
>> +       id="text4148"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4150"
>> +         x="183.5878"
>> +         y="397.28958"
>> +         style="font-size:25px">10G NIC</tspan></text>
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="166.92024"
>> +       y="451.33276"
>> +       id="text4152"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4154"
>> +         x="166.92024"
>> +         y="451.33276">Moongen</tspan></text>
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:1.39882457;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4156"
>> +       width="449.73071"
>> +       height="244.32167"
>> +       x="6.0295582"
>> +       y="29.046324" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="405.31628"
>> +       y="25.048317"
>> +       id="text4158"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4160"
>> +         x="405.31628"
>> +         y="25.048317"
>> +         style="font-size:25px">DUT</tspan></text>
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:1.14168489;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4162"
>> +       width="418.69415"
>> +       height="107.50462"
>> +       x="19.038134"
>> +       y="41.044758" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="395.38812"
>> +       y="66.496857"
>> +       id="text4164"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4166"
>> +         x="395.38812"
>> +         y="66.496857"
>> +         style="font-size:25px">VM</tspan></text>
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4146-3"
>> +       width="92.934036"
>> +       height="32.324883"
>> +       x="183.0827"
>> +       y="274.05093" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="184.09286"
>> +       y="299.30475"
>> +       id="text4148-6"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4150-7"
>> +         x="184.09286"
>> +         y="299.30475"
>> +         style="font-size:25px">10G NIC</tspan></text>
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:2.4804399;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4189"
>> +       width="398.00476"
>> +       height="65.451302"
>> +       x="26.901583"
>> +       y="82.647781" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:25px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="30.683046"
>> +       y="108.31288"
>> +       id="text4191"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4193"
>> +         x="30.683046"
>> +         y="108.31288">TestPMD</tspan><tspan
>> +         sodipodi:role="line"
>> +         x="30.683046"
>> +         y="139.56288"
>> +         id="tspan10476">(macswap)</tspan></text>
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:2.49124122;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4189-5"
>> +       width="397.22263"
>> +       height="66.152573"
>> +       x="29.743357"
>> +       y="207.6543" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:25px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="42.720772"
>> +       y="231.14902"
>> +       id="text4191-3"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4193-5"
>> +         x="42.720772"
>> +         y="231.14902">TestPMD </tspan><tspan
>> +         sodipodi:role="line"
>> +         x="42.720772"
>> +         y="262.39902"
>> +         id="tspan9747">(io)</tspan></text>
>> +    <path
>> +       style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-
>> linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-
>> end:url(#Arrow1Lend)"
>> +       d="M 202.56669,371.44487 V 308.37034"
>> +       id="path4218"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-
>> width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> opacity:1;marker-start:url(#Arrow1Lstart)"
>> +       d="M 252.03098,369.63533 V 307.25568"
>> +       id="path4218-9"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-
>> linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-
>> end:url(#Arrow1Lend-1)"
>> +       d="M 198.63811,207.44389 V 150.47507"
>> +       id="path4218-0"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-
>> width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> opacity:1;marker-start:url(#Arrow1Lstart-9)"
>> +       d="M 255.56859,206.9303 V 147.01008"
>> +       id="path4218-9-6"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
>> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-
>> opacity:1;marker-end:url(#marker9757)"
>> +       d="M 199.50513,271.00921 V 207.3696"
>> +       id="path9749"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
>> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-
>> opacity:1;marker-start:url(#marker10370)"
>> +       d="M 255.56859,270.56991 V 206.9303"
>> +       id="path9749-2"
>> +       inkscape:connector-curvature="0" />
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4146-36"
>> +       width="92.934036"
>> +       height="32.324883"
>> +       x="304.05591"
>> +       y="372.52954" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="305.06607"
>> +       y="397.78339"
>> +       id="text4148-7"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4150-5"
>> +         x="305.06607"
>> +         y="397.78339"
>> +         style="font-size:25px">10G NIC</tspan></text>
>> +    <rect
>> +       style="fill:none;fill-opacity:1;stroke:#000000;stroke-
>> width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-
>> dashoffset:0;stroke-opacity:1"
>> +       id="rect4146-3-3"
>> +       width="92.934036"
>> +       height="32.324883"
>> +       x="306.07623"
>> +       y="273.53461" />
>> +    <text
>> +       xml:space="preserve"
>> +       style="font-style:normal;font-weight:normal;font-size:40px;line-
>> height:125%;font-family:sans-serif;letter-spacing:0px;word-
>> spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-
>> width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
>> +       x="307.0864"
>> +       y="298.78842"
>> +       id="text4148-6-5"><tspan
>> +         sodipodi:role="line"
>> +         id="tspan4150-7-6"
>> +         x="307.0864"
>> +         y="298.78842"
>> +         style="font-size:25px">10G NIC</tspan></text>
>> +    <path
>> +       style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-
>> linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-
>> end:url(#Arrow1Lend-62)"
>> +       d="M 323.7504,370.24835 V 307.17382"
>> +       id="path4218-1"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-
>> width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> opacity:1;marker-start:url(#Arrow1Lstart-6)"
>> +       d="M 373.21469,368.43881 V 306.05916"
>> +       id="path4218-9-8"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:#000000;fill-opacity:1;fill-
>> rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-
>> linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-
>> end:url(#Arrow1Lend-1-7)"
>> +       d="M 324.93036,207.24894 V 150.28012"
>> +       id="path4218-0-9"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-
>> width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> opacity:1;marker-start:url(#Arrow1Lstart-9-2)"
>> +       d="M 381.86084,206.73535 V 146.81513"
>> +       id="path4218-9-6-2"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
>> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-
>> opacity:1;marker-end:url(#marker9757-2)"
>> +       d="M 325.79738,270.81426 V 207.17465"
>> +       id="path9749-28"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
>> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-
>> opacity:1;marker-start:url(#marker10370-7)"
>> +       d="M 381.86084,270.37496 V 206.73535"
>> +       id="path9749-2-9"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
>> width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> miterlimit:4;stroke-dasharray:2, 1;stroke-dashoffset:0;stroke-
>> opacity:1;marker-end:url(#marker4642)"
>> +       d="M 198.57143,148.79077 V 95.93363 h 182.85714 v 50"
>> +       id="path3748"
>> +       inkscape:connector-curvature="0" />
>> +    <path
>> +       style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-
>> width:1.01005316;stroke-linecap:butt;stroke-linejoin:miter;stroke-
>> miterlimit:4;stroke-dasharray:2.02010632, 1.01005316;stroke-
>> dashoffset:0;stroke-opacity:1;marker-end:url(#marker4760)"
>> +       d="m 325.70774,148.78714 v -32.84999 h -70.7012 v 30.70761"
>> +       id="path4634"
>> +       inkscape:connector-curvature="0" />
>> +  </g>
>> +</svg>
>> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst index
>> 5575b27..712a9f3 100644
>> --- a/doc/guides/howto/index.rst
>> +++ b/doc/guides/howto/index.rst
>> @@ -38,3 +38,4 @@ HowTo Guides
>>      lm_bond_virtio_sriov
>>      lm_virtio_vhost_user
>>      flow_bifurcation
>> +    pvp_reference_benchmark
>> diff --git a/doc/guides/howto/pvp_reference_benchmark.rst
>> b/doc/guides/howto/pvp_reference_benchmark.rst
>> new file mode 100644
>> index 0000000..c5bbd32
>> --- /dev/null
>> +++ b/doc/guides/howto/pvp_reference_benchmark.rst
>> @@ -0,0 +1,395 @@
>> +..  BSD LICENSE
>> +    Copyright(c) 2016 Red Hat, Inc. All rights reserved.
>> +    All rights reserved.
>> +
>> +    Redistribution and use in source and binary forms, with or without
>> +    modification, are permitted provided that the following conditions
>> +    are met:
>> +
>> +    * Redistributions of source code must retain the above copyright
>> +    notice, this list of conditions and the following disclaimer.
>> +    * Redistributions in binary form must reproduce the above copyright
>> +    notice, this list of conditions and the following disclaimer in
>> +    the documentation and/or other materials provided with the
>> +    distribution.
>> +    * Neither the name of Intel Corporation nor the names of its
>> +    contributors may be used to endorse or promote products derived
>> +    from this software without specific prior written permission.
>> +
>> +    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
>> +    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
>> +    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>> +    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
>> +    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
>> +    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
>> +    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
>> +    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
>> +    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> +    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
>> +    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>> +
>> +
>> +PVP reference benchmark setup using testpmd
>> +===========================================
>> +
>> +This guide lists the steps required to setup a PVP benchmark using
>> +testpmd as a simple forwarder between NICs and Vhost interfaces. The
>> +goal of this setup is to have a reference PVP benchmark without using
>> +external vSwitches (OVS, VPP, ...) to make it easier to obtain
>> +reproducible results and to facilitate continuous integration testing.
>> +
>> +The guide covers two ways of launching the VM, either by directly
>> +calling the QEMU command line, or by relying on libvirt. It has been
>> +tested with DPDK
>> +v16.11 using RHEL7 for both host and guest.
>> +
>> +
>> +Setup overview
>> +--------------
>> +
>> +.. _figure_pvp_2nics:
>> +
>> +.. figure:: img/pvp_2nics.*
>> +
>> +   PVP setup using 2 NICs
>> +
>> +In this diagram, each red arrow represents one logical core. This
>> +use-case requires 6 dedicated logical cores. A forwarding configuration
>> +with a single NIC is also possible, requiring 3 logical cores.
>> +
>> +
>> +Host setup
>> +----------
>> +
>> +In this setup, we isolate 6 cores (from CPU2 to CPU7) on the same NUMA
>> +node. Two cores are assigned to the VM vCPUs running testpmd and four
>> +are assigned to testpmd on the host.
>> +
>> +
>> +Host tuning
>> +~~~~~~~~~~~
>> +
>> +#. On BIOS, disable turbo-boost and hyper-threads.
>> +
>> +#. Append these options to Kernel command line:
>> +
>> +   .. code-block:: console
>> +
>> +      intel_pstate=disable mce=ignore_ce default_hugepagesz=1G
>> + hugepagesz=1G hugepages=6 isolcpus=2-7 rcu_nocbs=2-7 nohz_full=2-7
>> + iommu=pt intel_iommu=on
>> +
>> +#. Disable hyper-threads at runtime if necessary or if BIOS is not
>> accessible:
>> +
>> +   .. code-block:: console
>> +
>> +      cat /sys/devices/system/cpu/cpu*[0-9]/topology/thread_siblings_list
>> \
>> +          | sort | uniq \
>> +          | awk -F, '{system("echo 0 >
>> /sys/devices/system/cpu/cpu"$2"/online")}'
>> +
>> +#. Disable NMIs:
>> +
>> +   .. code-block:: console
>> +
>> +      echo 0 > /proc/sys/kernel/nmi_watchdog
>> +
>> +#. Exclude isolated CPUs from the writeback cpumask:
>> +
>> +   .. code-block:: console
>> +
>> +      echo ffffff03 > /sys/bus/workqueue/devices/writeback/cpumask
>> +
>> +#. Isolate CPUs from IRQs:
>> +
>> +   .. code-block:: console
>> +
>> +      clear_mask=0xfc #Isolate CPU2 to CPU7 from IRQs
>> +      for i in /proc/irq/*/smp_affinity
>> +      do
>> +        echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
>> +      done
>> +
>> +
>> +Qemu build
>> +~~~~~~~~~~
>> +
>> +Build Qemu:
>> +
>> +    .. code-block:: console
>> +
>> +       git clone git://git.qemu.org/qemu.git
>> +       cd qemu
>> +       mkdir bin
>> +       cd bin
>> +       ../configure --target-list=x86_64-softmmu
>> +
>> +
>> +DPDK build
>> +~~~~~~~~~~
>> +
>> +Build DPDK:
>> +
>> +   .. code-block:: console
>> +
>> +      git clone git://dpdk.org/dpdk
>> +      cd dpdk
>> +      export RTE_SDK=$PWD
>> +      make install T=x86_64-native-linuxapp-gcc DESTDIR=install
>> +
>> +
>> +Testpmd launch
>> +~~~~~~~~~~~~~~
>> +
>> +#. Assign NICs to DPDK:
>> +
>> +   .. code-block:: console
>> +
>> +      modprobe vfio-pci
>> +      $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0
>> + 0000:11:00.1
>> +
>> +   .. Note::
>> +
>> +      The Sandy Bridge family seems to have some IOMMU limitations giving
>> poor
>> +      performance results. To achieve good performance on these machines
>> +      consider using UIO instead.
>> +
>> +#. Launch the testpmd application:
>> +
>> +   .. code-block:: console
>> +
>> +      $RTE_SDK/install/bin/testpmd -l 0,2,3,4,5 --socket-mem=1024 -n 4 \
>> +          --vdev 'net_vhost0,iface=/tmp/vhost-user1' \
>> +          --vdev 'net_vhost1,iface=/tmp/vhost-user2' -- \
>> +          --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1
>> +          --nb-cores=4 --forward-mode=io
>> +
>> +With this command, isolated CPUs 2 to 5 will be used as lcores for PMD
>> threads.
>> +
>> +#. In testpmd interactive mode, set the portlist to obtain the correct
>> port
>> +   chaining:
>> +
>> +   .. code-block:: console
>> +
>> +      set portlist 0,2,1,3
>> +      start
>> +
>> +
>> +VM launch
>> +~~~~~~~~~
>> +
>> +The VM may be launched either by calling QEMU directly, or by using
>> libvirt.
>> +
>> +#. Qemu way:
>> +
>> +Launch QEMU with two Virtio-net devices paired to the vhost-user
>> +sockets created by testpmd. Below example uses default Virtio-net
>> +options, but options may be specified, for example to disable mergeable
>> +buffers or indirect descriptors.
>> +
>> +   .. code-block:: console
>> +
>> +      <QEMU path>/bin/x86_64-softmmu/qemu-system-x86_64 \
>> +          -enable-kvm -cpu host -m 3072 -smp 3 \
>> +          -chardev socket,id=char0,path=/tmp/vhost-user1 \
>> +          -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
>> +          -device virtio-net-
>> pci,netdev=mynet1,mac=52:54:00:02:d9:01,addr=0x10 \
>> +          -chardev socket,id=char1,path=/tmp/vhost-user2 \
>> +          -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \
>> +          -device virtio-net-
>> pci,netdev=mynet2,mac=52:54:00:02:d9:02,addr=0x11 \
>> +          -object memory-backend-file,id=mem,size=3072M,mem-
>> path=/dev/hugepages,share=on \
>> +          -numa node,memdev=mem -mem-prealloc \
>> +          -net user,hostfwd=tcp::1002$1-:22 -net nic \
>> +          -qmp unix:/tmp/qmp.socket,server,nowait \
>> +          -monitor stdio <vm_image>.qcow2
>> +
>> +You can use this `qmp-vcpu-pin
>> +<https://patchwork.kernel.org/patch/9361617/>`_
>> +script to pin vCPUs.
>> +
>> +It can be used as follows, for example to pin 3 vCPUs to CPUs 1, 6 and
>> +7, where isolated CPUs 6 and 7 will be used as lcores for Virtio PMDs:
>> +
>> +   .. code-block:: console
>> +
>> +      export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp
>> +      ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7
>> +
>> +#. Libvirt way:
>> +
>> +Some initial steps are required for libvirt to be able to connect to
>> +testpmd's sockets.
>> +
>> +First, SELinux policy needs to be set to permissive, since testpmd is
>> +generally run as root (note, as reboot is required):
>> +
>> +   .. code-block:: console
>> +
>> +      cat /etc/selinux/config
>> +
>> +      # This file controls the state of SELinux on the system.
>> +      # SELINUX= can take one of these three values:
>> +      #     enforcing  - SELinux security policy is enforced.
>> +      #     permissive - SELinux prints warnings instead of enforcing.
>> +      #     disabled   - No SELinux policy is loaded.
>> +      SELINUX=permissive
>> +
>> +      # SELINUXTYPE= can take one of three two values:
>> +      #     targeted - Targeted processes are protected,
>> +      #     minimum  - Modification of targeted policy.
>> +      #                Only selected processes are protected.
>> +      #     mls      - Multi Level Security protection.
>> +      SELINUXTYPE=targeted
>> +
>> +
>> +Also, Qemu needs to be run as root, which has to be specified in
>> +``/etc/libvirt/qemu.conf``:
>> +
>> +   .. code-block:: console
>> +
>> +      user = "root"
>> +
>> +Once the domain created, the following snippet is an extract of he most
>> +important information (hugepages, vCPU pinning, Virtio PCI devices):
>> +
>> +   .. code-block:: xml
>> +
>> +      <domain type='kvm'>
>> +        <memory unit='KiB'>3145728</memory>
>> +        <currentMemory unit='KiB'>3145728</currentMemory>
>> +        <memoryBacking>
>> +          <hugepages>
>> +            <page size='1048576' unit='KiB' nodeset='0'/>
>> +          </hugepages>
>> +          <locked/>
>> +        </memoryBacking>
>> +        <vcpu placement='static'>3</vcpu>
>> +        <cputune>
>> +          <vcpupin vcpu='0' cpuset='1'/>
>> +          <vcpupin vcpu='1' cpuset='6'/>
>> +          <vcpupin vcpu='2' cpuset='7'/>
>> +          <emulatorpin cpuset='0'/>
>> +        </cputune>
>> +        <numatune>
>> +          <memory mode='strict' nodeset='0'/>
>> +        </numatune>
>> +        <os>
>> +          <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
>> +          <boot dev='hd'/>
>> +        </os>
>> +        <cpu mode='host-passthrough'>
>> +          <topology sockets='1' cores='3' threads='1'/>
>> +          <numa>
>> +            <cell id='0' cpus='0-2' memory='3145728' unit='KiB'
>> memAccess='shared'/>
>> +          </numa>
>> +        </cpu>
>> +        <devices>
>> +          <interface type='vhostuser'>
>> +            <mac address='56:48:4f:53:54:01'/>
>> +            <source type='unix' path='/tmp/vhost-user1' mode='client'/>
>> +            <model type='virtio'/>
>> +            <driver name='vhost' rx_queue_size='256' />
>> +            <address type='pci' domain='0x0000' bus='0x00' slot='0x10'
>> function='0x0'/>
>> +          </interface>
>> +          <interface type='vhostuser'>
>> +            <mac address='56:48:4f:53:54:02'/>
>> +            <source type='unix' path='/tmp/vhost-user2' mode='client'/>
>> +            <model type='virtio'/>
>> +            <driver name='vhost' rx_queue_size='256' />
>> +            <address type='pci' domain='0x0000' bus='0x00' slot='0x11'
>> function='0x0'/>
>> +          </interface>
>> +        </devices>
>> +      </domain>
>> +
>> +
>> +Guest setup
>> +-----------
>> +
>> +
>> +Guest tuning
>> +~~~~~~~~~~~~
>> +
>> +#. Append these options to the Kernel command line:
>> +
>> +   .. code-block:: console
>> +
>> +      default_hugepagesz=1G hugepagesz=1G hugepages=1 intel_iommu=on
>> + iommu=pt isolcpus=1,2 rcu_nocbs=1,2 nohz_full=1,2
>> +
>> +#. Disable NMIs:
>> +
>> +   .. code-block:: console
>> +
>> +      echo 0 > /proc/sys/kernel/nmi_watchdog
>> +
>> +#. Exclude isolated CPU1 and CPU2 from the writeback cpumask:
>> +
>> +   .. code-block:: console
>> +
>> +      echo 1 > /sys/bus/workqueue/devices/writeback/cpumask
>> +
>> +#. Isolate CPUs from IRQs:
>> +
>> +   .. code-block:: console
>> +
>> +      clear_mask=0x6 #Isolate CPU1 and CPU2 from IRQs
>> +      for i in /proc/irq/*/smp_affinity
>> +      do
>> +        echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
>> +      done
>> +
>> +
>> +DPDK build
>> +~~~~~~~~~~
>> +
>> +Build DPDK:
>> +
>> +   .. code-block:: console
>> +
>> +      git clone git://dpdk.org/dpdk
>> +      cd dpdk
>> +      export RTE_SDK=$PWD
>> +      make install T=x86_64-native-linuxapp-gcc DESTDIR=install
>> +
>> +
>> +Testpmd launch
>> +~~~~~~~~~~~~~~
>> +
>> +Probe vfio module without iommu:
>> +
>> +   .. code-block:: console
>> +
>> +      modprobe -r vfio_iommu_type1
>> +      modprobe -r vfio
>> +      modprobe  vfio enable_unsafe_noiommu_mode=1
>> +      cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
>> +      modprobe vfio-pci
>> +
>> +Bind the virtio-net devices to DPDK:
>> +
>> +   .. code-block:: console
>> +
>> +      $RTE_SDK/tools/dpdk-devbind.py -b vfio-pci 0000:00:10.0
>> + 0000:00:11.0
>> +
>> +Start testpmd:
>> +
>> +   .. code-block:: console
>> +
>> +      $RTE_SDK/install/bin/testpmd -l 0,1,2 --socket-mem 1024 -n 4 \
>> +          --proc-type auto --file-prefix pg -- \
>> +          --portmask=3 --forward-mode=macswap --port-topology=chained \
>> +          --disable-hw-vlan --disable-rss -i --rxq=1 --txq=1 \
>> +          --rxd=256 --txd=256 --nb-cores=2 --auto-start
>> +
>> +Results template
>> +----------------
>> +
>> +Below template should be used when sharing results:
>> +
>> +   .. code-block:: none
>> +
>> +      Traffic Generator: <Test equipment (e.g. IXIA, Moongen, ...)>
>> +      Acceptable Loss: <n>%
>> +      Validation run time: <n>min
>> +      Host DPDK version/commit: <version, SHA-1>
>> +      Guest DPDK version/commit: <version, SHA-1>
>> +      Patches applied: <link to patchwork>
>> +      QEMU version/commit: <version>
>> +      Virtio features: <features (e.g. mrg_rxbuf='off', leave empty if
>> default)>
>> +      CPU: <CPU model>, <CPU frequency>
>> +      NIC: <NIC model>
>> +      Result: <n> Mpps
>> --
>> 2.9.3
>

Re: [dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[Mcnamara, John]

> -----Original Message-----
> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> Sent: Friday, December 9, 2016 6:30 AM
> To: Mcnamara, John <john.mcnamara@intel.com>; yuanhan.liu@linux.intel.com;
> thomas.monjalon@6wind.com; Yang, Zhiyong <zhiyong.yang@intel.com>;
> ktraynor@redhat.com; dev@dpdk.org
> Subject: Re: [PATCH v2] doc: introduce PVP reference benchmark
> 
> 
> 
> On 12/07/2016 10:07 PM, Mcnamara, John wrote:
> >> -----Original Message-----
> >> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> >> Sent: Tuesday, December 6, 2016 12:25 PM
> >> To: yuanhan.liu@linux.intel.com; thomas.monjalon@6wind.com; Mcnamara,
> >> John <john.mcnamara@intel.com>; Yang, Zhiyong
> >> <zhiyong.yang@intel.com>; ktraynor@redhat.com; dev@dpdk.org
> >> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
> >> Subject: [PATCH v2] doc: introduce PVP reference benchmark
> >>
> >> Having reference benchmarks is important in order to obtain
> >> reproducible performance figures.
> >>
> >> This patch describes required steps to configure a PVP setup using
> >> testpmd in both host and guest.
> >>
> >> Not relying on external vSwitch ease integration in a CI loop by not
> >> being impacted by DPDK API changes.
> >>
> >> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> >
> > There is one trailing whitespace warning but apart from that:
> >
> > Acked-by: John McNamara <john.mcnamara@intel.com>
> 
> Thanks John,
> 
> Do you want me to send a v3, fixing the trailing whitespace & collecting
> the acks?
> 
>   -- Maxime


No need (unless the tree maintainer says otherwise). It was one trailing whitespace. Just something to look out for in future. 


P.S., We both should have truncated the text in this mail before replying. :-/

  http://dpdk.org/dev/patchwork/patch/17704/

Re: [dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[Thomas Monjalon]

> > >> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> > >
> > > There is one trailing whitespace warning but apart from that:
> > >
> > > Acked-by: John McNamara <john.mcnamara@intel.com>
> > 
> > Thanks John,
> > 
> > Do you want me to send a v3, fixing the trailing whitespace & collecting
> > the acks?
> 
> No need (unless the tree maintainer says otherwise).
> It was one trailing whitespace. Just something to look out for in future. 

Removed when applying.
Fixed some heading issues also:

--- a/doc/guides/howto/pvp_reference_benchmark.rst
+++ b/doc/guides/howto/pvp_reference_benchmark.rst
@@ -160,7 +160,7 @@ Testpmd launch
           --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1
           --nb-cores=4 --forward-mode=io
 
-With this command, isolated CPUs 2 to 5 will be used as lcores for PMD threads.
+   With this command, isolated CPUs 2 to 5 will be used as lcores for PMD threads.
 
 #. In testpmd interactive mode, set the portlist to obtain the correct port
    chaining:
@@ -176,7 +176,8 @@ VM launch
 
 The VM may be launched either by calling QEMU directly, or by using libvirt.
 
-#. Qemu way:
+Qemu way
+^^^^^^^^
 
 Launch QEMU with two Virtio-net devices paired to the vhost-user sockets
 created by testpmd. Below example uses default Virtio-net options, but options
@@ -210,7 +211,8 @@ where isolated CPUs 6 and 7 will be used as lcores for Virtio PMDs:
       export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp
       ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7
 
-#. Libvirt way:
+Libvirt way
+^^^^^^^^^^^
 
 Some initial steps are required for libvirt to be able to connect to testpmd's
 sockets.


Thanks

Re: [dpdk-dev] [PATCH v2] doc: introduce PVP reference benchmark

[Maxime Coquelin]

On 12/20/2016 11:03 AM, Thomas Monjalon wrote:
>>>>> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>>>>
>>>> There is one trailing whitespace warning but apart from that:
>>>>
>>>> Acked-by: John McNamara <john.mcnamara@intel.com>
>>>
>>> Thanks John,
>>>
>>> Do you want me to send a v3, fixing the trailing whitespace & collecting
>>> the acks?
>>
>> No need (unless the tree maintainer says otherwise).
>> It was one trailing whitespace. Just something to look out for in future.
>
> Removed when applying.
> Fixed some heading issues also:
>
> --- a/doc/guides/howto/pvp_reference_benchmark.rst
> +++ b/doc/guides/howto/pvp_reference_benchmark.rst
> @@ -160,7 +160,7 @@ Testpmd launch
>            --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1
>            --nb-cores=4 --forward-mode=io
>
> -With this command, isolated CPUs 2 to 5 will be used as lcores for PMD threads.
> +   With this command, isolated CPUs 2 to 5 will be used as lcores for PMD threads.
>
>  #. In testpmd interactive mode, set the portlist to obtain the correct port
>     chaining:
> @@ -176,7 +176,8 @@ VM launch
>
>  The VM may be launched either by calling QEMU directly, or by using libvirt.
>
> -#. Qemu way:
> +Qemu way
> +^^^^^^^^
>
>  Launch QEMU with two Virtio-net devices paired to the vhost-user sockets
>  created by testpmd. Below example uses default Virtio-net options, but options
> @@ -210,7 +211,8 @@ where isolated CPUs 6 and 7 will be used as lcores for Virtio PMDs:
>        export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp
>        ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7
>
> -#. Libvirt way:
> +Libvirt way
> +^^^^^^^^^^^
>
>  Some initial steps are required for libvirt to be able to connect to testpmd's
>  sockets.

Thanks Thomas for handling these fixes.

Maxime