* [dpdk-dev] [PATCH v3 1/2] doc: add svg for debug and troubleshoot guide
@ 2018-11-26 7:08 Vipin Varghese
2018-11-26 7:08 ` [dpdk-dev] [PATCH v3 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
0 siblings, 1 reply; 23+ messages in thread
From: Vipin Varghese @ 2018-11-26 7:08 UTC (permalink / raw)
To: dev, thomas, marko.kovacevic, honnappa.nagarahalli,
cristian.dumitrescu, anatoly.burakov, bruce.richardson,
olivier.matz
Cc: john.mcnamara, amol.patel, Vipin Varghese
Add svg for debug and troubleshoot guide for single or multi stage
applications.
Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
---
V3:
- reorder for removing warning in 'make doc-guides-html' - Thomas Monjalon
V2:
- add ACK from Marko - Vipin Varghese
---
doc/guides/howto/img/dtg_consumer_ring.svg | 19 ++++
doc/guides/howto/img/dtg_crypto.svg | 16 +++
.../howto/img/dtg_distributor_worker.svg | 31 ++++++
doc/guides/howto/img/dtg_mempool.svg | 22 ++++
doc/guides/howto/img/dtg_pdump.svg | 28 +++++
doc/guides/howto/img/dtg_producer_ring.svg | 19 ++++
doc/guides/howto/img/dtg_qos_tx.svg | 24 ++++
doc/guides/howto/img/dtg_rx_rate.svg | 20 ++++
doc/guides/howto/img/dtg_rx_tx_drop.svg | 28 +++++
doc/guides/howto/img/dtg_sample_app_model.svg | 105 ++++++++++++++++++
doc/guides/howto/img/dtg_service.svg | 15 +++
11 files changed, 327 insertions(+)
create mode 100644 doc/guides/howto/img/dtg_consumer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_crypto.svg
create mode 100644 doc/guides/howto/img/dtg_distributor_worker.svg
create mode 100644 doc/guides/howto/img/dtg_mempool.svg
create mode 100644 doc/guides/howto/img/dtg_pdump.svg
create mode 100644 doc/guides/howto/img/dtg_producer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_qos_tx.svg
create mode 100644 doc/guides/howto/img/dtg_rx_rate.svg
create mode 100644 doc/guides/howto/img/dtg_rx_tx_drop.svg
create mode 100644 doc/guides/howto/img/dtg_sample_app_model.svg
create mode 100644 doc/guides/howto/img/dtg_service.svg
diff --git a/doc/guides/howto/img/dtg_consumer_ring.svg b/doc/guides/howto/img/dtg_consumer_ring.svg
new file mode 100644
index 000000000..d0e0b30f8
--- /dev/null
+++ b/doc/guides/howto/img/dtg_consumer_ring.svg
@@ -0,0 +1,19 @@
+<svg width="634" height="97" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>consumer ring</title>
+ <rect fill="#fff" id="canvas_background" height="99" width="636" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="95" width="135" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_2" height="95" width="135" y="2" x="252" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_3" height="95" width="135" y="2" x="499" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="45" x="10" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="52" x="268" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_6" y="53" x="517" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 3</text>
+ <path transform="rotate(90.19324493408203 177.864990234375,48.89100646972658) " id="svg_7" d="m155.364991,48.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path transform="rotate(90.19324493408203 434.864990234375,48.89100646972656) " id="svg_8" d="m412.364991,48.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path id="svg_10" d="m225.937481,47.95781l0,0c0,-10.493411 8.282734,-19 18.500005,-19l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564971c3.469424,3.563191 5.418523,8.395917 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999998 -18.5,18.999998l0,0c-10.217271,0 -18.500005,-8.506591 -18.500005,-18.999998zm18.500005,-19l0,37.999998m-18.500005,-18.999998l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ <path id="svg_11" d="m475.937481,46.95781l0,0c0,-10.493411 8.282734,-19.000001 18.500005,-19.000001l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564972c3.469424,3.563191 5.418523,8.395916 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999997 -18.5,18.999997l0,0c-10.217271,0 -18.500005,-8.50659 -18.500005,-18.999997zm18.500005,-19.000001l0,37.999998m-18.500005,-18.999997l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_crypto.svg b/doc/guides/howto/img/dtg_crypto.svg
new file mode 100644
index 000000000..7687aaa4a
--- /dev/null
+++ b/doc/guides/howto/img/dtg_crypto.svg
@@ -0,0 +1,16 @@
+<svg width="180" height="230" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>crypto</title>
+ <rect fill="#fff" id="canvas_background" height="232" width="182" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="1" width="0" y="369.4375" x="838.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="222.4375" x="39.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core 7</text>
+ <rect stroke="#000" id="svg_26" height="44.000003" width="155.000001" y="151.437497" x="6.499997" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_27" y="179.4375" x="13.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">CRYPTO PMD</text>
+ <ellipse stroke="#000" ry="41.5" rx="66.500001" id="svg_28" cy="48.9375" cx="87.999997" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_29" y="57.4375" x="45.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Device</text>
+ <path stroke="#000" transform="rotate(-90.19718170166016 85.76751708984375,119.45785522460936) " id="svg_30" d="m40.743289,119.457895l26.106598,-13.915756l0,6.95786l37.835289,0l0,-6.95786l26.106586,13.915756l-26.106586,13.915682l0,-6.957842l-37.835289,0l0,6.957842l-26.106598,-13.915682z" stroke-opacity="null" stroke-width="0" fill="#845335"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_distributor_worker.svg b/doc/guides/howto/img/dtg_distributor_worker.svg
new file mode 100644
index 000000000..3df758fa6
--- /dev/null
+++ b/doc/guides/howto/img/dtg_distributor_worker.svg
@@ -0,0 +1,31 @@
+<svg width="600" height="235" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>dsitributor and worker</title>
+ <rect fill="#fff" id="canvas_background" height="237" width="602" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_1" height="89.999991" width="202.000004" y="21.000003" x="4" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start"
+ font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="47" x="7"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">PKT classify</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_12" y="96.5" x="25.992188" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Distribute</text>
+ <path id="svg_13" d="m334.224445,98.046256l16.252783,-5.432074l-16.252783,-5.429993l-16.256262,-5.427952l0,6.038517c-4.264321,-0.004069 -11.331234,-0.004069 -13.334561,-0.004069c-5.919233,-0.081711 -9.804243,-1.153783 -13.847174,-3.073397c-6.02597,-2.865095 -11.16889,-8.021466 -17.371313,-12.989922c-3.154554,-2.470933 -6.669049,-4.921502 -11.349751,-6.892119c-4.633926,-1.978838 -10.68815,-3.351144 -17.41807,-3.324591l-29.170102,0l0,9.646983l29.170102,0c4.056131,0.042884 6.623169,0.808657 10.124445,2.70786c5.179989,2.824261 10.230311,8.223602 17.195784,13.339136c6.826056,5.066481 17.329865,10.182006 32.564612,10.231017l13.436029,0l0,6.0385l16.256262,-5.427896zm-68.759273,-36.535597c0.184389,0.075572 0.379369,0.14089 0.559297,0.218526c5.559349,2.340242 9.573988,5.211493 12.751502,7.709001c0.730423,0.584013 1.377934,1.145611 2.071302,1.719456l37.120911,0l0,6.040581l16.256262,-5.42798l16.252783,-5.429962l-16.252783,-5.432037l-16.256262,-5.434093l0,6.036509l-52.503011,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFAC8D"/>
+ <rect stroke="#000" id="svg_17" height="110.999999" width="180" y="3.4375" x="353.5"
+ stroke-opacity="null" stroke-width="0" fill="#FABA6F"/>
+ <rect stroke="#000" id="svg_14" height="110.999999" width="180" y="31.4375" x="369.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_15" height="110.999999" width="180" y="58.4375" x="392.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFF093"/>
+ <rect stroke="#000" id="svg_16" height="110.999999" width="180" y="90.4375" x="413.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,142.60348030924797,64.2037038393319) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="71.490132" x="413.769676" stroke-opacity="null" stroke-width="0" fill="#000000">worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="101.332237" x="406.404514" stroke-opacity="null" stroke-width="0" fill="#000000">worker 2</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_20" y="58.700658" x="376.943866" stroke-opacity="null" stroke-width="0" fill="#000000">worker 3</text>
+ <text style="cursor: move;" stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_22" y="18.911184" x="350.429283" stroke-opacity="null" stroke-width="0" fill="#000000">worker 4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="139.4375" x="49.5"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_24" y="226.4375" x="404.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4,5</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_mempool.svg b/doc/guides/howto/img/dtg_mempool.svg
new file mode 100644
index 000000000..f3d31a96e
--- /dev/null
+++ b/doc/guides/howto/img/dtg_mempool.svg
@@ -0,0 +1,22 @@
+<svg width="955" height="185" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>mempool</title>
+ <rect fill="#fff" id="canvas_background" height="187" width="957" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="1" width="0" y="370.4375" x="839.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <rect stroke="#000" id="svg_31" height="181.000002" width="951.000029" y="2.4375" x="3.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_32" y="167.4375" x="14.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">MBUF pool</text>
+ <rect id="svg_38" height="71" width="859" y="15.4375" x="18.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_39" height="71" width="859" y="32.4375" x="33.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_40" height="71" width="859" y="47.4375" x="52.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_41" height="71" width="859" y="62.4375" x="71.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect stroke="#000" id="svg_42" height="55.999999" width="403.000005" y="69.4375" x="486.499995" stroke-opacity="null" stroke-width="0" fill="#FFAC8D"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_43" y="103.4375" x="631.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Payload</text>
+ <rect stroke="#000" id="svg_44" height="56" width="140.999999" y="68.4375" x="292.500001" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_45" y="105.4375" x="303.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Metadata</text>
+ <rect stroke="#000" id="svg_46" height="53" width="185.999997" y="70.4375" x="90.5" stroke-opacity="null" stroke-width="0" fill="#EDFFFF"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_47" y="104.4375" x="105.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">struct mbuf</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_pdump.svg b/doc/guides/howto/img/dtg_pdump.svg
new file mode 100644
index 000000000..1dc63bed7
--- /dev/null
+++ b/doc/guides/howto/img/dtg_pdump.svg
@@ -0,0 +1,28 @@
+<svg width="860" height="140" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>pdump</title>
+ <rect fill="#fff" id="canvas_background" height="142" width="862" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_15" height="127.999992" width="568" y="7" x="8.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#B1FFFF"/>
+ <rect id="svg_2" height="51" width="100" y="23" x="22.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_3" height="51" width="100" y="26" x="455.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CC7C65"/>
+ <path id="svg_7" d="m208.364769,78.968848l12.704559,-5.623238l12.697917,-5.625551l-12.704644,-5.629525l-12.697836,-5.625202l0,6.255891c-3.112384,0 -6.729051,0 -7.940704,0c-5.776899,0.124913 -12.907107,-3.365352 -20.492742,-7.864703c-2.827801,-1.577599 -5.680695,-3.313223 -8.704363,-4.88042c3.023667,-1.569118 5.872475,-3.302747 8.704363,-4.880295c7.585635,-4.493132 14.715839,-7.98967 20.485292,-7.862546l7.94815,-0.002103l0,6.264275l12.697836,-5.627447l12.704644,-5.631775l-12.704644,-5.637807l-12.697836,-5.629547l0,6.260273l-7.944067,0c-11.799214,0.12679 -20.61742,5.506774 -28.333828,9.879239c-7.660201,4.613909 -14.545738,8.087381 -18.539115,7.862617l-22.780531,0l0,9.993806l22.788007,0c3.989952,-0.226969 10.878944,3.248761 18.53841,7.86254c7.71303,4.374678 16.52785,9.750196 28.330427,9.879312l7.940704,0l0,6.262203l0,0l0,0.000003z" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#559E84"/>
+ <path id="svg_8" d="m439.457308,43.77545l-10.478696,-7.306254l0,8.12971l-18.805519,0c-3.29272,0.296972 -8.977707,-4.212008 -15.298608,-10.195089c-6.365074,-5.675375 -13.639418,-12.638687 -23.37938,-12.824984l-6.552957,0l0,12.959904l6.552957,0c4.770151,-0.167389 10.652051,4.360514 16.914219,10.203329c2.334186,2.041106 4.688512,4.292919 7.183208,6.326016c-2.494696,2.035766 -4.846223,4.290275 -7.183208,6.331424c-6.262164,5.826551 -12.144067,10.365242 -16.908056,10.200515l-6.559117,0l0,12.978808l6.555749,0c9.719269,-0.1646 16.996426,-7.14678 23.382185,-12.814082c6.32147,-5.983193 12.003677,-10.494803 15.299177,-10.203221l18.799357,0l0,8.129634l10.478696,-7.298036l10.484823,-7.303408l-10.484823,-7.314263l-0.000006,-0.000004z" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#559E84"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="54" x="53.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_10" y="57" x="486.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_11" y="40" x="243.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_12" y="77" x="243.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="82" x="326.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="40" x="327.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q3</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="114.86203" x="234.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Primary</text>
+ <rect stroke="#000" id="svg_18" height="90.000001" width="201.999994" y="29.437499" x="654.5" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="102.4375" x="684.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Secondary</text>
+ <rect stroke="#000" id="svg_20" height="42.000001" width="184.000001" y="34.437499" x="664.499994" stroke-opacity="null" stroke-width="0" fill="#A0D58A"/>
+ <rect id="svg_21" height="1" width="0" y="366.4375" x="835.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_22" y="59.4375" x="668.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Ring BufferQ</text>
+ <path stroke="#000" transform="rotate(89.78996276855469 615.4766845703124,74.49395751953122) " id="svg_24" d="m599.023803,74.414257l17.255478,-33.667333l15.650356,33.667333l-8.226459,0l0,33.826726l-16.452915,0l0,-33.826726l-8.226459,0z" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="22.4375" x="714.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core 0</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_producer_ring.svg b/doc/guides/howto/img/dtg_producer_ring.svg
new file mode 100644
index 000000000..5657702e5
--- /dev/null
+++ b/doc/guides/howto/img/dtg_producer_ring.svg
@@ -0,0 +1,19 @@
+<svg width="634" height="97" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>producer ring</title>
+ <rect fill="#fff" id="canvas_background" height="99" width="636" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="95" width="135" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_2" height="95" width="135" y="2" x="252" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_3" height="95" width="135" y="2" x="499" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="45" x="10" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="52" x="268" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_6" y="53" x="517" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 3</text>
+ <path transform="rotate(90.19324493408203 197.864990234375,42.89100646972657) " id="svg_7" d="m175.364991,42.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path transform="rotate(90.19324493408203 445.864990234375,47.89100646972657) " id="svg_8" d="m423.364991,47.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path id="svg_10" d="m120.937481,42.95781l0,0c0,-10.493411 8.282734,-19 18.500005,-19l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564971c3.469424,3.563191 5.418523,8.395917 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999998 -18.5,18.999998l0,0c-10.217271,0 -18.500005,-8.506591 -18.500005,-18.999998zm18.500005,-19l0,37.999998m-18.500005,-18.999998l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ <path id="svg_11" d="m373.937481,45.95781l0,0c0,-10.493411 8.282734,-19.000001 18.500005,-19.000001l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564972c3.469424,3.563191 5.418523,8.395916 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999997 -18.5,18.999997l0,0c-10.217271,0 -18.500005,-8.50659 -18.500005,-18.999997zm18.500005,-19.000001l0,37.999998m-18.500005,-18.999997l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_qos_tx.svg b/doc/guides/howto/img/dtg_qos_tx.svg
new file mode 100644
index 000000000..6e46d188a
--- /dev/null
+++ b/doc/guides/howto/img/dtg_qos_tx.svg
@@ -0,0 +1,24 @@
+<svg width="287" height="180" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>qos</title>
+ <rect fill="#fff" id="canvas_background" height="182" width="289" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="180" width="287" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CBFFFF"/>
+ <rect id="svg_10" height="82" width="78" y="17.808165" x="152.25" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_11" height="82" width="78" y="36.808165" x="171.25" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_12" height="82" width="78" y="59.808165" x="188.25" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="113.808165" x="211.25" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="161.808165" x="161.25" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core1</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_15" y="235.775764" x="531.631651" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.3615568424500273,0,0,0.9358925757041695,106.13283890389599,-107.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="173.881425" x="192.8023" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_17" y="66.670315" x="-70.838675" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ <rect id="svg_18" height="103" width="104" y="26" x="26" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FABA6F"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="79" x="50" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">QoS</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_20" y="153" x="27" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 10</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_rx_rate.svg b/doc/guides/howto/img/dtg_rx_rate.svg
new file mode 100644
index 000000000..311c9c43f
--- /dev/null
+++ b/doc/guides/howto/img/dtg_rx_rate.svg
@@ -0,0 +1,20 @@
+<svg width="116" height="152" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>rx rate</title>
+ <rect fill="#fff" id="canvas_background" height="154" width="118" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="82" width="78" y="0.812613" x="0.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_2" height="82" width="78" y="19.812613" x="19.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_3" height="82" width="78" y="42.812613" x="36.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="96.812613" x="59.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="144.812613" x="9.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core0</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_7" y="214.621694" x="210.788594" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.36155684245002745,0,0,0.9358925757041695,-60.867161096103985,-178.0100101290174) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_8" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="42.29845" x="-417.420836" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_rx_tx_drop.svg b/doc/guides/howto/img/dtg_rx_tx_drop.svg
new file mode 100644
index 000000000..2f4e9d587
--- /dev/null
+++ b/doc/guides/howto/img/dtg_rx_tx_drop.svg
@@ -0,0 +1,28 @@
+<svg width="275" height="154" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>rx-tx drops</title>
+ <rect fill="#fff" id="canvas_background" height="156" width="277" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="82" width="78" y="0.808165" x="0.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_2" height="82" width="78" y="19.808165" x="19.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_3" height="82" width="78" y="42.808165" x="36.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="96.808165" x="59.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="144.808165" x="9.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core0</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_7" y="214.616159" x="210.788594" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.3615568424500274,0,0,0.9358925757041695,-60.867161096103985,-178.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_8" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="42.292073" x="-417.420836" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ <rect id="svg_10" height="82" width="78" y="2.808165" x="159.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_11" height="82" width="78" y="21.808165" x="178.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_12" height="82" width="78" y="44.808165" x="195.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="98.808165" x="218.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="146.808165" x="168.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core1</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_15" y="217.105524" x="547.514971" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.36155684245002734,0,0,0.9358925757041695,98.13283890389599,-176.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_17" y="45.160101" x="-53.681142" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_sample_app_model.svg b/doc/guides/howto/img/dtg_sample_app_model.svg
new file mode 100644
index 000000000..ad3b0995d
--- /dev/null
+++ b/doc/guides/howto/img/dtg_sample_app_model.svg
@@ -0,0 +1,105 @@
+<svg width="1418" height="379" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>sample application model</title>
+ <rect fill="#fff" id="canvas_background" height="381" width="1420" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_84" height="378.999996" width="1417.999937" y="0" x="0" stroke-opacity="null" stroke-width="0" fill="#B1FFFF"/>
+ <rect stroke="#000" id="svg_1" height="79.999993" width="187.000011" y="267" x="635.499996" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <rect stroke="#000" id="svg_14" height="28.000001" width="175.000003" y="271.437495" x="641" stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_16" height="36.000002" width="142" y="304.437495" x="659" stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(0.7497134942573729,0,0,0.7607963681117937,149.70768863149087,72.0554119542491) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="335.740664" x="684.214296"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Health Check</text>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="368.437495" x="685"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 7</text>
+ <text stroke="#000" transform="matrix(0.7446371555386645,0,0,0.7004599746900311,157.05848471617847,107.2548065316271) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="261.052167" x="658.840014"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Stats Collector</text>
+ <rect id="svg_26" height="91" width="88" y="89.4375" x="2"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFA9A2"/>
+ <rect id="svg_27" height="91" width="88" y="107.4375" x="24"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CE7975"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_28" y="166.4375" x="52"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6666666865348816,14.972222477197647,39.14583100005984) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_30" y="96.937501" x="-11.409091"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 1</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6666666865348816,14.972222477197647,39.14583100005984) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_33" y="128.4375" x="27.863636"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_34" y="216.4375" x="26"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core0</text>
+ <rect id="svg_35" height="91" width="88" y="55.4375" x="1303"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFA9A2"/>
+ <rect id="svg_36" height="91" width="88" y="73.4375" x="1325"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CE7975"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_37" y="132.4375" x="1352"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_40" y="184.4375" x="1327"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core1</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6296296119689941,200.86111453175545,24.236112266778946) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_43" y="74.319853" x="1811.681832"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 1</text>
+ <text stroke="#000" transform="matrix(0.6958672408102909,0,0,0.6184720487972513,537.7539486343405,-28.040291137315034) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_44" y="191.620936" x="1140.58332"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 2</text>
+ <rect stroke="#000" id="svg_45" height="62" width="83.000003" y="101.4375" x="1096"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_46" y="135.4375" x="1115"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">QoS</text>
+ <rect stroke="#000" id="svg_48" height="61" width="107" y="110.4375" x="649"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_49" y="145.4375" x="661"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">Crypto</text>
+ <rect stroke="#000" id="svg_50" height="70.999998" width="84.000002" y="65.4375" x="422"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <rect stroke="#000" id="svg_51" height="74.000003" width="90.000002" y="92.4375" x="444"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_52" height="71.999998" width="92" y="120.4375" x="468"
+ stroke-opacity="null" stroke-width="0" fill="#C48157"/>
+ <text stroke="#000" transform="matrix(0.6528342962265015,0,0,0.5925925970077515,63.70490664243698,168.43749817460775) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_53" y="-145.374996" x="553.425418"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.6666666865348816,166.69894686341286,-165.52084343507886) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_54" y="417.437503" x="420.257696" stroke-opacity="null" stroke-width="0" fill="#000000">Worker 2</text>
+ <text stroke="#000" transform="matrix(0.7483048439025879,0,0,0.7407407164573669,76.38947987556458,153.33566251024604) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_55" y="-19.012521" x="530.011964"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 3</text>
+ <rect stroke="#000" id="svg_62" height="70.999998" width="84.000002" y="67.4375" x="865"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <rect stroke="#000" id="svg_63" height="74.000003" width="90.000002" y="94.4375" x="887"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_64" height="71.999998" width="92" y="122.4375" x="911"
+ stroke-opacity="null" stroke-width="0" fill="#C48157"/>
+ <text stroke="#000" transform="matrix(0.6528342962265015,0,0,0.5925925970077515,63.70490664243698,168.43749817460775) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_65" y="-143.687496" x="1238.132093"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.6666666865348816,166.69894686341286,-165.52084343507886) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_66" y="420.437503" x="1072.811052"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 2</text>
+ <text stroke="#000" transform="matrix(0.7483048439025879,0,0,0.7407407164573669,76.38947987556458,153.33566251024604) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_67" y="-16.312521" x="1122.016685"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 3</text>
+ <path stroke="#000" id="svg_68" d="m653.747516,12.862181l19.36745,-10.781877l77.632553,0l0,53.999988l-97.000004,0l0,-43.218111z"
+ stroke-opacity="null" stroke-width="0" fill="#805064"/>
+ <text stroke="#000" transform="matrix(0.733815550804138,0,0,0.9629629850387573,306.63544338941574,-1.3912058547139168) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_69" y="33.322117" x="496.534253"
+ stroke-opacity="null" stroke-width="0" fill="#ffffff">Device</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_70" y="217.4375" x="457"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_71" y="197.4375" x="653"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 5</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_72" y="186.4375" x="1097"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 6</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_73" y="220.4375" x="885"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4</text>
+ <rect stroke="#000" id="svg_74" height="88.000002" width="110.999999" y="104" x="206.000001"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text stroke="#000" transform="matrix(0.5661651903991256,0,0,1,107.56695064249905,0) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_75" y="133" x="182.901935"
+ stroke-opacity="null" stroke-width="0" fill="#000000">PKT classify</text>
+ <text stroke="#000" transform="matrix(0.5444634556770325,0,0,1,115.93405053019524,0) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_76" y="170" x="196.644877"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Distribute</text>
+ <path id="svg_77" d="m189.755816,142.38346l-9.615746,-5.572568l0,6.200627l-17.256831,0c-3.021555,0.226504 -8.238368,-3.212549 -14.038725,-7.775916c-5.840892,-4.328677 -12.516173,-9.639678 -21.454022,-9.781769l-6.013302,0l0,9.884674l6.013302,0c4.377316,-0.12767 9.774825,3.325816 15.521285,7.782201c2.14196,1.556776 4.3024,3.274261 6.59165,4.824928c-2.28925,1.552703 -4.447123,3.272244 -6.59165,4.829053c-5.746457,4.44398 -11.143969,7.905694 -15.51563,7.780055l-6.018955,0l0,9.899092l6.015865,0c8.91886,-0.125542 15.596722,-5.450935 21.456596,-9.773453c5.800879,-4.563453 11.01514,-8.004512 14.039247,-7.782119l17.251176,0l0,6.200569l9.615746,-5.566299l9.621368,-5.570396l-9.621368,-5.578676l-0.000005,-0.000003z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path id="svg_78" d="m417.79301,181.072084l13.604178,-5.205736l-13.604178,-5.203742l-13.60709,-5.201786l0,5.786911c-3.569394,-0.003899 -9.48466,-0.003899 -11.161519,-0.003899c-4.954616,-0.078306 -8.206513,-1.105708 -11.590595,-2.945338c-5.043959,-2.745716 -9.348773,-7.687237 -14.54043,-12.448672c-2.640478,-2.367977 -5.58224,-4.716439 -9.50016,-6.604946c-3.878767,-1.896386 -8.946375,-3.211512 -14.579566,-3.186066l-24.41645,0l0,9.245023l24.41645,0c3.395131,0.041097 5.543836,0.774963 8.474533,2.595032c4.335842,2.706583 8.563147,7.88095 14.393505,12.783335c5.71366,4.855376 14.505736,9.757754 27.25778,9.804723l11.246452,0l0,5.786895l13.60709,-5.201732zm-57.554046,-35.013272c0.15434,0.072423 0.317546,0.135019 0.468152,0.209421c4.65338,2.242732 8.013781,4.994346 10.673477,7.387791c0.611391,0.559679 1.153382,1.097877 1.733756,1.647811l31.071571,0l0,5.788888l13.60709,-5.201813l13.604178,-5.203713l-13.604178,-5.205701l-13.60709,-5.207671l0,5.784986l-43.946955,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path stroke="#000" transform="rotate(89.46277618408203 605.7476806640625,144.42587280273435) " id="svg_79" d="m590.864808,144.339395l14.882874,-35.951591l14.882874,35.951591l-7.441438,0l0,36.124583l-14.882873,0l0,-36.124583l-7.441438,0z" stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ <path stroke="#000" transform="rotate(89.74066925048828 697.6382446289062,81.67762756347659) " id="svg_80" d="m661.638136,81.677665l20.874103,-13.000079l0,6.500022l30.252035,0l0,-6.500022l20.874093,13.000079l-20.874093,13.000009l0,-6.500005l-30.252035,0l0,6.500005l-20.874103,-13.000009z" fill-opacity="null"
+ stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ <path id="svg_81" d="m858.79301,172.072084l13.604178,-5.205736l-13.604178,-5.203742l-13.60709,-5.201786l0,5.786911c-3.569394,-0.0039 -9.48466,-0.0039 -11.161519,-0.0039c-4.954616,-0.078305 -8.206513,-1.105708 -11.590595,-2.945338c-5.043959,-2.745715 -9.348773,-7.687236 -14.54043,-12.448672c-2.640477,-2.367977 -5.58224,-4.716438 -9.50016,-6.604945c-3.878767,-1.896387 -8.946375,-3.211513 -14.579566,-3.186067l-24.416451,0l0,9.245023l24.416451,0c3.395131,0.041097 5.543836,0.774963 8.474533,2.595032c4.335842,2.706583 8.563147,7.88095 14.393505,12.783335c5.713661,4.855376 14.505736,9.757754 27.257781,9.804723l11.246451,0l0,5.786894l13.60709,-5.201732zm-57.554045,-35.013272c0.15434,0.072424 0.317546,0.135019 0.468152,0.209421c4.65338,2.242731 8.013781,4.994346 10.673477,7.38779c0.61139,0.559679 1.153381,1.097878 1.733756,1.647812l31.07157,0l0,5.788888l13.60709,-5.201812l13.604178,-5.203713l-13.604178,-5.205701l-13.60709,-5.207671l0,5.784986l-43.946955,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path id="svg_82" d="m1080.755816,127.38346l-9.615746,-5.572567l0,6.200627l-17.256831,0c-3.021555,0.226504 -8.238368,-3.212549 -14.038725,-7.775916c-5.840892,-4.328677 -12.516173,-9.639678 -21.454022,-9.78177l-6.013302,0l0,9.884674l6.013302,0c4.377316,-0.12767 9.774825,3.325816 15.521285,7.782201c2.141959,1.556776 4.3024,3.274261 6.59165,4.824928c-2.28925,1.552704 -4.447123,3.272244 -6.59165,4.829053c-5.746457,4.443981 -11.143969,7.905694 -15.51563,7.780055l-6.018955,0l0,9.899093l6.015865,0c8.91886,-0.125543 15.596723,-5.450935 21.456596,-9.773454c5.800879,-4.563453 11.01514,-8.004512 14.039246,-7.782119l17.251177,0l0,6.20057l9.615746,-5.566299l9.621368,-5.570397l-9.621368,-5.578676l-0.000006,-0.000003z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path stroke="#000" id="svg_83" d="m1269.963506,163.318859l13.567973,-4.653717l13.56088,-4.655631l-13.568064,-4.658919l-13.560793,-4.655342l0,5.177292c-3.323905,0 -7.186364,0 -8.480362,0c-6.169502,0.103377 -13.784287,-2.785121 -21.885448,-6.508723c-3.019981,-1.3056 -6.066761,-2.741979 -9.295919,-4.03897c3.229159,-1.298581 6.271574,-2.733309 9.295919,-4.038867c8.101162,-3.718456 15.715942,-6.612144 21.877492,-6.506938l8.488314,-0.001741l0,5.18423l13.560793,-4.6572l13.568064,-4.660782l-13.568064,-4.665774l-13.560793,-4.658938l0,5.180918l-8.483954,0c-12.6011,0.104929 -22.018599,4.557333 -30.259422,8.175926c-8.180795,3.81841 -15.53428,6.693008 -19.799052,6.506997l-24.328718,0l0,8.27074l24.336702,0c4.261113,-0.187837 11.618288,2.688631 19.798299,6.506933c8.237215,3.620425 17.651098,8.069132 30.25579,8.175987l8.480362,0l0,5.182516l0,0l0,0.000002z" fill-opacity="null"
+ stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_service.svg b/doc/guides/howto/img/dtg_service.svg
new file mode 100644
index 000000000..2d876a3a1
--- /dev/null
+++ b/doc/guides/howto/img/dtg_service.svg
@@ -0,0 +1,15 @@
+<svg width="254" height="145" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>service</title>
+ <rect fill="#fff" id="canvas_background" height="147" width="256" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_1" height="109.999987" width="254.000014" y="0" x="0" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <rect stroke="#000" id="svg_14" height="37.000001" width="225" y="7.437494" x="4.5" stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_16" height="45.000002" width="186.000003" y="58.437493" x="33.5" stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(1.0013854504218995,0,0,1.0926463039877063,73.76856116958965,-122.08861649089796) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="193.234194" x="-35.792216" stroke-opacity="null" stroke-width="0" fill="#000000">Health Check</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="138.437495" x="45.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 6</text>
+ <text stroke="#000" transform="matrix(1.0013854504218995,0,0,1.0926463039877063,73.76856116958965,-122.08861649089796) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="141.982476" x="-63.753477" stroke-opacity="null" stroke-width="0" fill="#000000">Stats Collector</text>
+ </g>
+</svg>
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v3 2/2] doc: add guide for debug and troubleshoot
2018-11-26 7:08 [dpdk-dev] [PATCH v3 1/2] doc: add svg for debug and troubleshoot guide Vipin Varghese
@ 2018-11-26 7:08 ` Vipin Varghese
2019-01-04 6:37 ` Shreyansh Jain
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 0/2] doci/howto: add debug and troubleshoot guide Vipin Varghese
0 siblings, 2 replies; 23+ messages in thread
From: Vipin Varghese @ 2018-11-26 7:08 UTC (permalink / raw)
To: dev, thomas, marko.kovacevic, honnappa.nagarahalli,
cristian.dumitrescu, anatoly.burakov, bruce.richardson,
olivier.matz
Cc: john.mcnamara, amol.patel, Vipin Varghese
Add user guide on debug and troubleshoot for common issues and bottleneck
found in various application models running on single or multi stages.
Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
---
V3:
- reorder for removing warning in 'make doc-guides-html' - Thomas Monjalon
V2:
- add offload flag check - Vipin Varghese
- change tab to space - Marko Kovacevic
- spelling correction - Marko Kovacevic
- remove extra characters - Marko Kovacevic
- add ACK by Marko - Vipn Varghese
---
doc/guides/howto/debug_troubleshoot_guide.rst | 351 ++++++++++++++++++
doc/guides/howto/index.rst | 1 +
2 files changed, 352 insertions(+)
create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
diff --git a/doc/guides/howto/debug_troubleshoot_guide.rst b/doc/guides/howto/debug_troubleshoot_guide.rst
new file mode 100644
index 000000000..55589085e
--- /dev/null
+++ b/doc/guides/howto/debug_troubleshoot_guide.rst
@@ -0,0 +1,351 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2018 Intel Corporation.
+
+.. _debug_troubleshoot_via_pmd:
+
+Debug & Troubleshoot guide via PMD
+==================================
+
+DPDK applications can be designed to run as single thread simple stage to
+multiple threads with complex pipeline stages. These application can use poll
+mode devices which helps in offloading CPU cycles. A few models are
+
+ * single primary
+ * multiple primary
+ * single primary single secondary
+ * single primary multiple secondary
+
+In all the above cases, it is a tedious task to isolate, debug and understand
+odd behaviour which can occurring random or periodic. The goal of guide is to
+share and explore a few commonly seen patterns and behaviour. Then isolate and
+identify the root cause via step by step debug at various processing stages.
+
+Application Overview
+--------------------
+
+Let us take up an example application as reference for explaining issues and
+patterns commonly seen. The sample application in discussion makes use of
+single primary model with various pipeline stages. The application uses PMD
+and libraries such as service cores, mempool, pkt mbuf, event, crypto, QoS
+and eth.
+
+The overview of an application modeled using PMD is shown in
+:numref:`dtg_sample_app_model`.
+
+.. _dtg_sample_app_model:
+
+.. figure:: img/dtg_sample_app_model.*
+
+ Overview of pipeline stage of an application
+
+Bottleneck Analysis
+-------------------
+
+To debug the bottleneck and performance issues the desired application
+is made to run in an environment matching as below
+
+#. Linux 64-bit|32-bit
+#. DPDK PMD and libraries are used
+#. Libraries and PMD are either static or shared. But not both
+#. Machine flag optimizations of gcc or compiler are made constant
+
+Is there mismatch in packet rate (received < send)?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RX Port and associated core :numref:`dtg_rx_rate`.
+
+.. _dtg_rx_rate:
+
+.. figure:: img/dtg_rx_rate.*
+
+ RX send rate compared against Received rate
+
+#. are generic configuration correct?
+ - What is port Speed, Duplex? rte_eth_link_get()
+ - Is packet of higher sizes are dropped? rte_eth_get_mtu()
+ - Are only specific MAC are received? rte_eth_promiscuous_get()
+
+#. are there NIC specific drops?
+ - Check rte_eth_rx_queue_info_get() for nb_desc, scattered_rx,
+ - Check rte_eth_dev_stats() for Stats per queue
+ - Is stats of other queues shows no change via
+ rte_eth_dev_dev_rss_hash_conf_get()
+ - Check if port offload and queue offload matches.
+
+#. If problem still persists, this might be at RX lcore thread
+ - Check if RX thread, distributor or event rx adapter is holding or
+ processing more than required
+ - try using rte_prefetch_non_temporal() to intimate the mbuf in pulled
+ to cache for temporary
+
+
+Are there packet drops (receive|transmit)?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RX-TX Port and associated cores :numref:`dtg_rx_tx_drop`.
+
+.. _dtg_rx_tx_drop:
+
+.. figure:: img/dtg_rx_tx_drop.*
+
+ RX-TX drops
+
+#. at RX
+ - Get the rx queues by rte_eth_dev_info_get() for nb_rx_queues
+ - Check for miss, errors, qerros by rte_eth_dev_stats() for imissed,
+ ierrors, q_erros, rx_nombuf, rte_mbuf_ref_count
+
+#. at TX
+ - Are we doing in bulk to reduce the TX descriptor overhead?
+ - Check rte_eth_dev_stats() for oerrors, qerros, rte_mbuf_ref_count
+ - Is the packet multi segmented? Check if port and queue offlaod is set.
+
+Are there object drops in producer point for ring?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Producer point for ring :numref:`dtg_producer_ring`.
+
+.. _dtg_producer_ring:
+
+.. figure:: img/dtg_producer_ring.*
+
+ Producer point for Rings
+
+#. Performance for Producer
+ - Fetch the type of RING 'rte_ring_dump()' for flags (RING_F_SP_ENQ)
+ - If '(burst enqueue - actual enqueue) > 0' check rte_ring_count() or
+ rte_ring_free_count()
+ - If 'burst or single enqueue is 0', then there is no more space check
+ rte_ring_full() or not
+
+Are there object drops in consumer point for ring?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Consumer point for ring :numref:`dtg_consumer_ring`.
+
+.. _dtg_consumer_ring:
+
+.. figure:: img/dtg_consumer_ring.*
+
+ Consumer point for Rings
+
+#. Performance for Consumer
+ - Fetch the type of RING – rte_ring_dump() for flags (RING_F_SC_DEQ)
+ - If '(burst dequeue - actual dequeue) > 0' for rte_ring_free_count()
+ - If 'burst or single enqueue' always results 0 check the ring is empty
+ via rte_ring_empty()
+
+Is packets or objects are not processed at desired rate?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Memory objects close to NUMA :numref:`dtg_mempool`.
+
+.. _dtg_mempool:
+
+.. figure:: img/dtg_mempool.*
+
+ Memory objects has to be close to device per NUMA
+
+#. Is the performance low?
+ - Are packets received from multiple NIC? rte_eth_dev_count_all()
+ - Are NIC interfaces on different socket? use rte_eth_dev_socket_id()
+ - Is mempool created with right socket? rte_mempool_create() or
+ rte_pktmbuf_pool_create()
+ - Are we seeing drop on specific socket? It might require more
+ mempool objects; try allocating more objects
+ - Is there single RX thread for multiple NIC? try having multiple
+ lcore to read from fixed interface or we might be hitting cache
+ limit, so Increase cache_size for pool_create()
+
+#. Are we are still seeing low performance
+ - Check if sufficient objects in mempool by rte_mempool_avail_count()
+ - Is failure in some pkt? we might be getting pkts with size > mbuf
+ data size. Check rte_pktmbuf_is_continguous()
+ - If user pthread is used to access object access
+ rte_mempool_cache_create()
+ - Try using 1GB huge pages instead of 2MB. If there is difference,
+ try then rte_mem_lock_page() for 2MB pages
+
+.. note::
+ Stall in release of MBUF can be because
+
+ * Processing pipeline is too heavy
+ * Number of stages are too many
+ * TX is not transferred at desired rate
+ * Multi segment is not offloaded at TX device.
+ * Application misuse scenarios can be
+ - not freeing packets
+ - invalid rte_pktmbuf_refcnt_set
+ - invalid rte_pktmbuf_prefree_seg
+
+Is there difference in performance for crypto?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Crypto device and PMD :numref:`dtg_crypto`.
+
+.. _dtg_crypto:
+
+.. figure:: img/dtg_crypto.*
+
+ CRYPTO and interaction with PMD device
+
+#. are generic configuration correct?
+ - Get total crypto devices – rte_cryptodev_count()
+ - Cross check SW or HW flags are configured properly
+ rte_cryptodev_info_get() for feature_flags
+
+#. If enqueue request > actual enqueue (drops)?
+ - Is the queue pair setup for proper node
+ rte_cryptodev_queue_pair_setup() for socket_id
+ - Is the session_pool created from same socket_id as queue pair?
+ - Is enqueue thread same socket_id?
+ - rte_cryptodev_stats() for drops err_count for enqueue or dequeue
+ - Are there multiple threads enqueue or dequeue from same queue pair?
+
+#. If enqueue rate > dequeue rate?
+ - Is dequeue lcore thread is same socket_id?
+ - If SW crypto is in use, check if the CRYPTO Library build with
+ right (SIMD) flags Or check if the queue pair using CPU ISA by
+ rte_cryptodev_info_get() for feature_flags for AVX|SSE
+ - If we are using HW crypto – Is the card on same NUMA socket as
+ queue pair and session pool?
+
+worker functions not giving performance?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Custom worker function :numref:`dtg_distributor_worker`.
+
+.. _dtg_distributor_worker:
+
+.. figure:: img/dtg_distributor_worker.*
+
+ Custom worker function performance drops
+
+#. Performance
+ - Threads context switches more frequently? Identify lcore with
+ rte_lcore() and lcore index mapping with rte_lcore_index(). Best
+ performance when mapping of thread and core is 1:1.
+ - Check lcore role type and state? rte_eal_lcore_role for
+ rte, off and service. User function on service core might be
+ sharing timeslots with other functions.
+ - Check the cpu core? check rte_thread_get_affinity() and
+ rte_eal_get_lcore_state() for run state.
+
+#. Debug
+ - Mode of operation? rte_eal_get_configuration() for master, fetch
+ lcore|service|numa count, process_type.
+ - Check lcore run mode? rte_eal_lcore_role() for rte, off, service.
+ - process details? rte_dump_stack(), rte_dump_registers() and
+ rte_memdump() will give insights.
+
+service functions are not frequent enough?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+service functions on service cores :numref:`dtg_service`.
+
+.. _dtg_service:
+
+.. figure:: img/dtg_service.*
+
+ functions running on service cores
+
+#. Performance
+ - Get service core count? rte_service_lcore_count() and compare with
+ result of rte_eal_get_configuration()
+ - Check if registered service is available?
+ rte_service_get_by_name(), rte_service_get_count() and
+ rte_service_get_name()
+ - Is given service running parallel on multiple lcores?
+ rte_service_probe_capability() and rte_service_map_lcore_get()
+ - Is service running? rte_service_runstate_get()
+
+#. Debug
+ - Find how many services are running on specific service lcore by
+ rte_service_lcore_count_services()
+ - Generic debug via rte_service_dump()
+
+Is there bottleneck in eventdev?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. are generic configuration correct?
+ - Get event_dev devices? rte_event_dev_count()
+ - Are they created on correct socket_id? - rte_event_dev_socket_id()
+ - Check if HW or SW capabilities? - rte_event_dev_info_get() for
+ event_qos, queue_all_types, burst_mode, multiple_queue_port,
+ max_event_queue|dequeue_depth
+ - Is packet stuck in queue? check for stages (event qeueue) where
+ packets are looped back to same or previous stages.
+
+#. Performance drops in enqueue (event count > actual enqueue)?
+ - Dump the event_dev information? rte_event_dev_dump()
+ - Check stats for queue and port for eventdev
+ - Check the inflight, current queue element for enqueue|deqeue
+
+How to debug QoS via TM?
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+TM on TX interface :numref:`dtg_qos_tx`.
+
+.. _dtg_qos_tx:
+
+.. figure:: img/dtg_qos_tx.*
+
+ Traffic Manager just before TX
+
+#. Is configuration right?
+ - Get current capabilities for DPDK port rte_tm_capabilities_get()
+ for max nodes, level, shaper_private, shaper_shared, sched_n_children
+ and stats_mask
+ - Check if current leaf are configured identically rte_tm_capabilities_get()
+ for lead_nodes_identicial
+ - Get leaf nodes for a dpdk port - rte_tn_get_number_of_leaf_node()
+ - Check level capabilities by rte_tm_level_capabilities_get for n_nodes
+ - Max, nonleaf_max, leaf_max
+ - identical, non_identical
+ - Shaper_private_supported
+ - Stats_mask
+ - Cman wred packet|byte supported
+ - Cman head drop supported
+ - Check node capabilities by rte_tm_node_capabilities_get for n_nodes
+ - Shaper_private_supported
+ - Stats_mask
+ - Cman wred packet|byte supported
+ - Cman head drop supported
+ - Debug via stats - rte_tm_stats_update() and rte_tm_node_stats_read()
+
+Packet is not of right format?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Packet capture before and after processing :numref:`dtg_pdump`.
+
+.. _dtg_pdump:
+
+.. figure:: img/dtg_pdump.*
+
+ Capture points of Traffic at RX-TX
+
+#. with primary enabling then secondary can access. Copies packets from
+ specific RX or TX queues to secondary process ring buffers.
+
+.. note::
+ Need to explore:
+ * if secondary shares same interface can we enable from secondary
+ for rx|tx happening on primary
+ * Specific PMD private data dump the details
+ * User private data if present, dump the details
+
+How to develop custom code to debug?
+------------------------------------
+
+- For single process - the debug functionality is to be added in same
+ process
+- For multiple process - the debug functionality can be added to
+ secondary multi process
+
+.. note::
+
+ Primary's Debug functions invoked via
+ #. Timer call-back
+ #. Service function under service core
+ #. USR1 or USR2 signal handler
+
diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a642a2be1..9527fa84d 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -18,3 +18,4 @@ HowTo Guides
virtio_user_as_exceptional_path
packet_capture_framework
telemetry
+ debug_troubleshoot_guide
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v3 2/2] doc: add guide for debug and troubleshoot
2018-11-26 7:08 ` [dpdk-dev] [PATCH v3 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
@ 2019-01-04 6:37 ` Shreyansh Jain
2019-01-04 7:01 ` Varghese, Vipin
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 0/2] doci/howto: add debug and troubleshoot guide Vipin Varghese
1 sibling, 1 reply; 23+ messages in thread
From: Shreyansh Jain @ 2019-01-04 6:37 UTC (permalink / raw)
To: Vipin Varghese
Cc: dev, thomas, marko.kovacevic, honnappa.nagarahalli,
cristian.dumitrescu, anatoly.burakov, bruce.richardson,
olivier.matz, john.mcnamara, amol.patel
Hello Vipin,
Some comments and lots of nitpicks inlined.
(I know this comes months late - apologies, just didn't stumble on this
earlier).
On Monday 26 November 2018 12:38 PM, Vipin Varghese wrote:
> Add user guide on debug and troubleshoot for common issues and bottleneck
> found in various application models running on single or multi stages.
>
> Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
> Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
> ---
>
> V3:
> - reorder for removing warning in 'make doc-guides-html' - Thomas Monjalon
>
> V2:
> - add offload flag check - Vipin Varghese
> - change tab to space - Marko Kovacevic
> - spelling correction - Marko Kovacevic
> - remove extra characters - Marko Kovacevic
> - add ACK by Marko - Vipn Varghese
> ---
> doc/guides/howto/debug_troubleshoot_guide.rst | 351 ++++++++++++++++++
> doc/guides/howto/index.rst | 1 +
> 2 files changed, 352 insertions(+)
> create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
>
> diff --git a/doc/guides/howto/debug_troubleshoot_guide.rst b/doc/guides/howto/debug_troubleshoot_guide.rst
> new file mode 100644
> index 000000000..55589085e
> --- /dev/null
> +++ b/doc/guides/howto/debug_troubleshoot_guide.rst
> @@ -0,0 +1,351 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright(c) 2018 Intel Corporation.
> +
> +.. _debug_troubleshoot_via_pmd:
> +
> +Debug & Troubleshoot guide via PMD
> +==================================
> +
> +DPDK applications can be designed to run as single thread simple stage to
> +multiple threads with complex pipeline stages. These application can use poll
> +mode devices which helps in offloading CPU cycles. A few models are
> +
> + * single primary
> + * multiple primary
> + * single primary single secondary
> + * single primary multiple secondary
> +
> +In all the above cases, it is a tedious task to isolate, debug and understand
> +odd behaviour which can occurring random or periodic. The goal of guide is to
^^^^^^^^^^^^^^^^^^^^^^ ^^^^^
either of: which can occur randomly or periodically ... this
which occurs randomly or periodically
> +share and explore a few commonly seen patterns and behaviour. Then isolate and
^^^^^^^
super nitpick: Then, isolate
> +identify the root cause via step by step debug at various processing stages.
> +
> +Application Overview
> +--------------------
> +
> +Let us take up an example application as reference for explaining issues and
> +patterns commonly seen. The sample application in discussion makes use of
> +single primary model with various pipeline stages. The application uses PMD
> +and libraries such as service cores, mempool, pkt mbuf, event, crypto, QoS
> +and eth.
> +
> +The overview of an application modeled using PMD is shown in
> +:numref:`dtg_sample_app_model`.
> +
> +.. _dtg_sample_app_model:
> +
> +.. figure:: img/dtg_sample_app_model.*
> +
> + Overview of pipeline stage of an application
> +
> +Bottleneck Analysis
> +-------------------
> +
> +To debug the bottleneck and performance issues the desired application
> +is made to run in an environment matching as below
> +
> +#. Linux 64-bit|32-bit
> +#. DPDK PMD and libraries are used
> +#. Libraries and PMD are either static or shared. But not both
> +#. Machine flag optimizations of gcc or compiler are made constant
> +
> +Is there mismatch in packet rate (received < send)?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +RX Port and associated core :numref:`dtg_rx_rate`.
> +
> +.. _dtg_rx_rate:
> +
> +.. figure:: img/dtg_rx_rate.*
> +
> + RX send rate compared against Received rate
> +
> +#. are generic configuration correct?
> + - What is port Speed, Duplex? rte_eth_link_get()
> + - Is packet of higher sizes are dropped? rte_eth_get_mtu()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Are packets of higher size dropped?
> + - Are only specific MAC are received? rte_eth_promiscuous_get()
^^^^^^^
can be removed
What about checking vlan-filters, if any? - that is, if packets have
some vlan stamped and hardware is configured to filter that.
> +
> +#. are there NIC specific drops?
> + - Check rte_eth_rx_queue_info_get() for nb_desc, scattered_rx,
> + - Check rte_eth_dev_stats() for Stats per queue
> + - Is stats of other queues shows no change via
> + rte_eth_dev_dev_rss_hash_conf_get()
Does stat for other queues shows no change via ...
Or, maybe you intend to say - check stats for all queues in case RSS is
configured and packets are being distributed to a queue not being
listened to.
> + - Check if port offload and queue offload matches.
> +
> +#. If problem still persists, this might be at RX lcore thread
> + - Check if RX thread, distributor or event rx adapter is holding or
> + processing more than required
^^^^^^^^^^^
Did you mean to say "...processing less that required.."? Because if
Rx doesn't match Tx (from a generator), it would be less processing.
> + - try using rte_prefetch_non_temporal() to intimate the mbuf in pulled
> + to cache for temporary
^^^^^^^
Maybe this statement is incomplete - or, need rephrasing.
> +
> +
> +Are there packet drops (receive|transmit)?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +RX-TX Port and associated cores :numref:`dtg_rx_tx_drop`.
> +
> +.. _dtg_rx_tx_drop:
> +
> +.. figure:: img/dtg_rx_tx_drop.*
> +
> + RX-TX drops
> +
> +#. at RX
> + - Get the rx queues by rte_eth_dev_info_get() for nb_rx_queues
^^^^
Or, you meant to write "Get the Rx queue info using
rte_eth_dev_info_get()". Or maybe "Get Rx queue count using info from
rte_eth_dev_info_get()"
> + - Check for miss, errors, qerros by rte_eth_dev_stats() for imissed,
> + ierrors, q_erros, rx_nombuf, rte_mbuf_ref_count
> +
> +#. at TX
> + - Are we doing in bulk to reduce the TX descriptor overhead?
^^^^^^
Assuming all your conversation until now is second person, 'you'.
> + - Check rte_eth_dev_stats() for oerrors, qerros, rte_mbuf_ref_count
> + - Is the packet multi segmented? Check if port and queue offlaod is set.
> +
> +Are there object drops in producer point for ring?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Producer point for ring :numref:`dtg_producer_ring`.
> +
> +.. _dtg_producer_ring:
> +
> +.. figure:: img/dtg_producer_ring.*
> +
> + Producer point for Rings
> +
> +#. Performance for Producer
> + - Fetch the type of RING 'rte_ring_dump()' for flags (RING_F_SP_ENQ)
> + - If '(burst enqueue - actual enqueue) > 0' check rte_ring_count() or
> + rte_ring_free_count()
> + - If 'burst or single enqueue is 0', then there is no more space check
> + rte_ring_full() or not
^^^^^^
This line needs rephrase; maybe like: If 'burst or single enqueue is 0',
then, if there is no more space, check rte_ring_full().
Or
... then, check if the ring is full or not using rte_ring_full()
(Or, your similar statement for rte_ring_empty(), below, looks better)
> +
> +Are there object drops in consumer point for ring?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Consumer point for ring :numref:`dtg_consumer_ring`.
> +
> +.. _dtg_consumer_ring:
> +
> +.. figure:: img/dtg_consumer_ring.*
> +
> + Consumer point for Rings
> +
> +#. Performance for Consumer
> + - Fetch the type of RING – rte_ring_dump() for flags (RING_F_SC_DEQ)
> + - If '(burst dequeue - actual dequeue) > 0' for rte_ring_free_count()
> + - If 'burst or single enqueue' always results 0 check the ring is empty
> + via rte_ring_empty()
> +
> +Is packets or objects are not processed at desired rate?
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Are packets or objects not processed at desired rate?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Memory objects close to NUMA :numref:`dtg_mempool`.
> +
> +.. _dtg_mempool:
> +
> +.. figure:: img/dtg_mempool.*
> +
> + Memory objects has to be close to device per NUMA
> +
> +#. Is the performance low?
> + - Are packets received from multiple NIC? rte_eth_dev_count_all()
> + - Are NIC interfaces on different socket? use rte_eth_dev_socket_id()
> + - Is mempool created with right socket? rte_mempool_create() or
> + rte_pktmbuf_pool_create()
> + - Are we seeing drop on specific socket? It might require more
^^^^^
Just like above, as your perspective has been second person, 'you'.
> + mempool objects; try allocating more object
So, Or check if mempool depletion levels are not being reached using
"rte_mempool_get_count()" or "rte_mempool_avail_count()". That might
help if mempool objects are falling short.
And, this might actually leads to drops rather than drop in desired rate
(which is a consequence)
(I do notice that you have mentioned this below - but I think that is
the first thing to check. Thats just my opinion - so, feel free to ignore)
> + - Is there single RX thread for multiple NIC? try having multiple
> + lcore to read from fixed interface or we might be hitting cache
> + limit, so Increase cache_size for pool_create()
^^^
Small 'i'
> +
> +#. Are we are still seeing low performance
> + - Check if sufficient objects in mempool by rte_mempool_avail_count()
> + - Is failure in some pkt? we might be getting pkts with size > mbuf
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1) Use packets rather than pkts or pkt
2) Did you mean "Is there failure in Rx of packets?" Or, "Packet drops
being observed?"
> + data size. Check rte_pktmbuf_is_continguous()
^^^^^^^^^^^^^^^^^^^^^^^^^
I think you intended: rte_pktmbuf_is_contiguous()
> + - If user pthread is used to access object access
> + rte_mempool_cache_create()
Maybe I didn't get the point - but, why using rte_mempool_cache_create
helps in case of external pthreads? Using cache would be applicable in
all cases (assuming mempool_create(..cache_size=0..).
> + - Try using 1GB huge pages instead of 2MB. If there is difference,
^^^^^^^^
If there is a difference or if there is no difference?
> + try then rte_mem_lock_page() for 2MB pages
^^^^^
Should be: 'then try ...'
> +
> +.. note::
> + Stall in release of MBUF can be because
> +
> + * Processing pipeline is too heavy
> + * Number of stages are too many
> + * TX is not transferred at desired rate
> + * Multi segment is not offloaded at TX device.
> + * Application misuse scenarios can be
> + - not freeing packets
> + - invalid rte_pktmbuf_refcnt_set
> + - invalid rte_pktmbuf_prefree_seg
> +
> +Is there difference in performance for crypto?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Crypto device and PMD :numref:`dtg_crypto`.
> +
> +.. _dtg_crypto:
> +
> +.. figure:: img/dtg_crypto.*
> +
> + CRYPTO and interaction with PMD device
> +
> +#. are generic configuration correct?
> + - Get total crypto devices – rte_cryptodev_count()
> + - Cross check SW or HW flags are configured properly
> + rte_cryptodev_info_get() for feature_flags
> +
> +#. If enqueue request > actual enqueue (drops)?
> + - Is the queue pair setup for proper node
> + rte_cryptodev_queue_pair_setup() for socket_id
> + - Is the session_pool created from same socket_id as queue pair?
> + - Is enqueue thread same socket_id? ^^^^^^^^^^^^^
... thread on same ...
> + - rte_cryptodev_stats() for drops err_count for enqueue or dequeue
> + - Are there multiple threads enqueue or dequeue from same queue pair?
^^^^^^^^^^^^^^^^^^^^^^^
I think you meant:
Are there multiple threads enqueuing/dequeuing from same...
Or
Do multiple threads enqueue or dequeue from same ....
> +
> +#. If enqueue rate > dequeue rate?
> + - Is dequeue lcore thread is same socket_id?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Do you mean dequeue lcore threads has same socket as enqueue lcore
thread? If so, can you elaborate
> + - If SW crypto is in use, check if the CRYPTO Library build with
^^^^^^^^^
...is built with...
> + right (SIMD) flags Or check if the queue pair using CPU ISA by
^^^^
or
> + rte_cryptodev_info_get() for feature_flags for AVX|SSE
> + - If we are using HW crypto – Is the card on same NUMA socket as
^^^^^^^^^^^^^
By 'card', you mean the hardware block? So, some form factors of SoC
might not even have 'cards'. Maybe, "Is hardware crypto block affined to
correct NUMA socket as per application configuration" can be a better check.
> + queue pair and session pool?
> +
> +worker functions not giving performance?
^^^^^
'W' in place of 'w'
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Custom worker function :numref:`dtg_distributor_worker`.
> +
> +.. _dtg_distributor_worker:
> +
> +.. figure:: img/dtg_distributor_worker.*
> +
> + Custom worker function performance drops
> +
> +#. Performance
> + - Threads context switches more frequently? Identify lcore with
^^^^^^^^^^^
frequent
> + rte_lcore() and lcore index mapping with rte_lcore_index(). Best
> + performance when mapping of thread and core is 1:1.
> + - Check lcore role type and state? rte_eal_lcore_role for
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is a suggestion, not a question, so the '?' should be removed.
> + rte, off and service. User function on service core might be
^^^^^^^^^^^^^^^^^^^^^
I think you mean: Use rte_eal_lcore_role for identifying role of core as
either of RTE, SERVICE or OFF. Otherwise, it looks more like notes than
a suggestion/hint.
> + sharing timeslots with other functions.
> + - Check the cpu core? check rte_thread_get_affinity() and
^^^^^^^^^^^^^^^^^^^
This is not a question - a suggestion, so '?' should be removed.
> + rte_eal_get_lcore_state() for run state.
> +
> +#. Debug
> + - Mode of operation? rte_eal_get_configuration() for master, fetch
^^^^^^^^^^^^^^^^^^^^^
"Check mode of operation" or "What is the mode of operation?"
> + lcore|service|numa count, process_type.
> + - Check lcore run mode? rte_eal_lcore_role() for rte, off, service.
^^^^^^^^^^^^^^^^^^^^^^
Same as above - it looks more like notes than a troubleshooting step.
> + - process details? rte_dump_stack(), rte_dump_registers() and
^^^^^^^^^^^^^^^^^
Same as above.
> + rte_memdump() will give insights.
> +
> +service functions are not frequent enough?
^^^^^^
'S'
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +service functions on service cores :numref:`dtg_service`.
> +
> +.. _dtg_service:
> +
> +.. figure:: img/dtg_service.*
> +
> + functions running on service cores
> +
> +#. Performance
> + - Get service core count? rte_service_lcore_count() and compare with
^^^^^^^^^^^^^^^^^^^^^^^^^^
Again, this is not a question, but a suggestion and should be without
'?'. And the statement is incoherent - "Get the service core count using
rte_service_lcore_count() API and compare with result of ..." maybe be
better.
> + result of rte_eal_get_configuration()
> + - Check if registered service is available?
^^^^^^^
Suggestion, not a question.
> + rte_service_get_by_name(), rte_service_get_count() and
> + rte_service_get_name()
> + - Is given service running parallel on multiple lcores?
> + rte_service_probe_capability() and rte_service_map_lcore_get()
> + - Is service running? rte_service_runstate_get()
> +
> +#. Debug
> + - Find how many services are running on specific service lcore by
> + rte_service_lcore_count_services()
> + - Generic debug via rte_service_dump()
> +
> +Is there bottleneck in eventdev?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +#. are generic configuration correct?
> + - Get event_dev devices? rte_event_dev_count()
> + - Are they created on correct socket_id? - rte_event_dev_socket_id()
> + - Check if HW or SW capabilities? - rte_event_dev_info_get() for
> + event_qos, queue_all_types, burst_mode, multiple_queue_port,
> + max_event_queue|dequeue_depth
> + - Is packet stuck in queue? check for stages (event qeueue) where
> + packets are looped back to same or previous stages.
> +
> +#. Performance drops in enqueue (event count > actual enqueue)?
> + - Dump the event_dev information? rte_event_dev_dump()
> + - Check stats for queue and port for eventdev
> + - Check the inflight, current queue element for enqueue|deqeue
> +
> +How to debug QoS via TM?
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +TM on TX interface :numref:`dtg_qos_tx`.
> +
> +.. _dtg_qos_tx:
> +
> +.. figure:: img/dtg_qos_tx.*
> +
> + Traffic Manager just before TX
> +
> +#. Is configuration right?
> + - Get current capabilities for DPDK port rte_tm_capabilities_get()
> + for max nodes, level, shaper_private, shaper_shared, sched_n_children
> + and stats_mask
> + - Check if current leaf are configured identically rte_tm_capabilities_get()
> + for lead_nodes_identicial
^^^^^^^^^^^^^^
identical
> + - Get leaf nodes for a dpdk port - rte_tn_get_number_of_leaf_node()
> + - Check level capabilities by rte_tm_level_capabilities_get for n_nodes
> + - Max, nonleaf_max, leaf_max
> + - identical, non_identical
> + - Shaper_private_supported
> + - Stats_mask
> + - Cman wred packet|byte supported
> + - Cman head drop supported
> + - Check node capabilities by rte_tm_node_capabilities_get for n_nodes
> + - Shaper_private_supported
> + - Stats_mask
> + - Cman wred packet|byte supported
> + - Cman head drop supported
> + - Debug via stats - rte_tm_stats_update() and rte_tm_node_stats_read()
> +
> +Packet is not of right format?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Packet capture before and after processing :numref:`dtg_pdump`.
> +
> +.. _dtg_pdump:
> +
> +.. figure:: img/dtg_pdump.*
> +
> + Capture points of Traffic at RX-TX
> +
> +#. with primary enabling then secondary can access. Copies packets from
> + specific RX or TX queues to secondary process ring buffers.
The statement above would be clearer with a rephrase. For example 'with
primary enabling then secondary can access' is incomplete (enable what
and access what?)
> +
> +.. note::
> + Need to explore:
Is this section for user to explore? Or, maybe it is a residual from
your notes?
> + * if secondary shares same interface can we enable from secondary
> + for rx|tx happening on primary
> + * Specific PMD private data dump the details
> + * User private data if present, dump the details
> +
> +How to develop custom code to debug?
> +------------------------------------
> +
> +- For single process - the debug functionality is to be added in same
> + process
> +- For multiple process - the debug functionality can be added to
> + secondary multi process
> +
> +.. note::
> +
> + Primary's Debug functions invoked via
> + #. Timer call-back
> + #. Service function under service core
> + #. USR1 or USR2 signal handler
> +
> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
> index a642a2be1..9527fa84d 100644
> --- a/doc/guides/howto/index.rst
> +++ b/doc/guides/howto/index.rst
> @@ -18,3 +18,4 @@ HowTo Guides
> virtio_user_as_exceptional_path
> packet_capture_framework
> telemetry
> + debug_troubleshoot_guide
>
Frankly, it is a good attempt, but I think this is more in shape of
notes right now.
I do understand that it is never going to be an easy task to list
troubleshooting steps - that is fairly dependent on env and use-case.
But, if maybe you can rephrase and add some context to some sections, it
would give real nice inputs/hints as a point to start with.
Again - just my take. And being so delayed, feel free to ignore.
-
Shreyansh
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v3 2/2] doc: add guide for debug and troubleshoot
2019-01-04 6:37 ` Shreyansh Jain
@ 2019-01-04 7:01 ` Varghese, Vipin
0 siblings, 0 replies; 23+ messages in thread
From: Varghese, Vipin @ 2019-01-04 7:01 UTC (permalink / raw)
To: Shreyansh Jain
Cc: dev, thomas, Kovacevic, Marko, honnappa.nagarahalli, Dumitrescu,
Cristian, Burakov, Anatoly, Richardson, Bruce, olivier.matz,
Mcnamara, John, Patel, Amol
Hi Shreyansh jain,
Thanks for the comment, please give me time to look into other comments.
snipped
> > + - Are only specific MAC are received? rte_eth_promiscuous_get()
> ^^^^^^^
> can be removed
>
> What about checking vlan-filters, if any? - that is, if packets have some vlan
> stamped and hardware is configured to filter that.
This is excellent thought; can we work together to add this part once the baseline is added to the guide?
>
> > +
> > +#. are there NIC specific drops?
> > + - Check rte_eth_rx_queue_info_get() for nb_desc, scattered_rx,
> > + - Check rte_eth_dev_stats() for Stats per queue
> > + - Is stats of other queues shows no change via
> > + rte_eth_dev_dev_rss_hash_conf_get()
>
> Does stat for other queues shows no change via ...
> Or, maybe you intend to say - check stats for all queues in case RSS is
> configured and packets are being distributed to a queue not being listened to.
Yes.
snipped
> > +#. Are we are still seeing low performance
> > + - Check if sufficient objects in mempool by rte_mempool_avail_count()
> > + - Is failure in some pkt? we might be getting pkts with size > mbuf
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 1) Use packets rather than pkts or pkt
> 2) Did you mean "Is there failure in Rx of packets?" Or, "Packet drops
> being observed?"
> > + data size. Check rte_pktmbuf_is_continguous()
> ^^^^^^^^^^^^^^^^^^^^^^^^^
> I think you intended: rte_pktmbuf_is_contiguous()
Well during field deployment with customer, failing to allow multi-segment has led to drops in virtio and NIC PMD. I will try to rephrase this in a better way.
>
> > + - If user pthread is used to access object access
> > + rte_mempool_cache_create()
>
> Maybe I didn't get the point - but, why using rte_mempool_cache_create
> helps in case of external pthreads? Using cache would be applicable in
> all cases (assuming mempool_create(..cache_size=0..).
The API ' rte_mempool_cache_create' is being created for non-eal threads. In use case scenarios where pthreads are created either before or after 'rte_eal_init', it would be useful to use this to isolate the defect is caused from starving the objects in master mempool.
snipped
> > +
> > +.. note::
> > + Need to explore:
>
> Is this section for user to explore? Or, maybe it is a residual from
> your notes?
As shared below, these would other possible areas for the user to explore in continuation of above.
snipped
>
>
> Frankly, it is a good attempt, but I think this is more in shape of
> notes right now.
>
> I do understand that it is never going to be an easy task to list
> troubleshooting steps - that is fairly dependent on env and use-case.
> But, if maybe you can rephrase and add some context to some sections, it
> would give real nice inputs/hints as a point to start with.
This is an attempt to capture and share points for debug and troubleshoot. As rightly pointed out this will be varied for different use cases and scenarios.
In my humble opinion an attempt is made to start with somewhere. If there more scenario and use cases that community can be benefited, we (dpdk community) is free to add the same.
>
> Again - just my take. And being so delayed, feel free to ignore.
Thanks for understanding. But I will try to accommodate.
>
> -
> Shreyansh
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v4 0/2] doci/howto: add debug and troubleshoot guide
2018-11-26 7:08 ` [dpdk-dev] [PATCH v3 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
2019-01-04 6:37 ` Shreyansh Jain
@ 2019-01-16 14:54 ` Vipin Varghese
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 1/2] doc: add svg for " Vipin Varghese
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
1 sibling, 2 replies; 23+ messages in thread
From: Vipin Varghese @ 2019-01-16 14:54 UTC (permalink / raw)
To: dev, shreyansh.jain, thomas
Cc: john.mcnamara, marko.kovacevic, amol.patel, sanjay.padubidri,
Vipin Varghese
This patch series is an attempt to add how to guide for commonly seen
issues in various application scenario. It also tries to suggest a step
by step guide on how to debug and troubleshoot with available stats and
counters.
Motivation
==========
DPDK allows to build application on top of custom off the shelf
components for user scenarios that are modeled with single or multi stage.
Each module interacts with PMD or rings with either software or hardware
offloads.
Identify the cause of issue or performance issues will be tedious and timie
consuming. The how to guide is an effort to consolidate such scenarios with
step by step focus to debug and troubleshoot.
Status
======
Reviews and changes accomadated. ACK received and udpated.
Future
======
1. Suggestion on vlan filters -
Change Log:
Vipin Varghese (2):
doc: add svg for debug and troubleshoot guide
doc: add guide for debug and troubleshoot
doc/guides/howto/debug_troubleshoot_guide.rst | 375 ++++++++++++++++++
doc/guides/howto/img/dtg_consumer_ring.svg | 19 +
doc/guides/howto/img/dtg_crypto.svg | 16 +
.../howto/img/dtg_distributor_worker.svg | 31 ++
doc/guides/howto/img/dtg_mempool.svg | 22 +
doc/guides/howto/img/dtg_pdump.svg | 28 ++
doc/guides/howto/img/dtg_producer_ring.svg | 19 +
doc/guides/howto/img/dtg_qos_tx.svg | 24 ++
doc/guides/howto/img/dtg_rx_rate.svg | 20 +
doc/guides/howto/img/dtg_rx_tx_drop.svg | 28 ++
doc/guides/howto/img/dtg_sample_app_model.svg | 105 +++++
doc/guides/howto/img/dtg_service.svg | 15 +
doc/guides/howto/index.rst | 1 +
13 files changed, 703 insertions(+)
create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
create mode 100644 doc/guides/howto/img/dtg_consumer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_crypto.svg
create mode 100644 doc/guides/howto/img/dtg_distributor_worker.svg
create mode 100644 doc/guides/howto/img/dtg_mempool.svg
create mode 100644 doc/guides/howto/img/dtg_pdump.svg
create mode 100644 doc/guides/howto/img/dtg_producer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_qos_tx.svg
create mode 100644 doc/guides/howto/img/dtg_rx_rate.svg
create mode 100644 doc/guides/howto/img/dtg_rx_tx_drop.svg
create mode 100644 doc/guides/howto/img/dtg_sample_app_model.svg
create mode 100644 doc/guides/howto/img/dtg_service.svg
V4:
- Added suggestion and corrections - Shreyansh Jain
V3:
- Reorder the commit order - Thomas Monjalon
V2:
- Added ACK - Marko Kovacevic
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v4 1/2] doc: add svg for debug and troubleshoot guide
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 0/2] doci/howto: add debug and troubleshoot guide Vipin Varghese
@ 2019-01-16 14:54 ` Vipin Varghese
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
1 sibling, 0 replies; 23+ messages in thread
From: Vipin Varghese @ 2019-01-16 14:54 UTC (permalink / raw)
To: dev, shreyansh.jain, thomas
Cc: john.mcnamara, marko.kovacevic, amol.patel, sanjay.padubidri,
Vipin Varghese
Add svg for debug and troubleshoot guide for single or multi stage
applications.
Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
---
doc/guides/howto/img/dtg_consumer_ring.svg | 19 ++++
doc/guides/howto/img/dtg_crypto.svg | 16 +++
.../howto/img/dtg_distributor_worker.svg | 31 ++++++
doc/guides/howto/img/dtg_mempool.svg | 22 ++++
doc/guides/howto/img/dtg_pdump.svg | 28 +++++
doc/guides/howto/img/dtg_producer_ring.svg | 19 ++++
doc/guides/howto/img/dtg_qos_tx.svg | 24 ++++
doc/guides/howto/img/dtg_rx_rate.svg | 20 ++++
doc/guides/howto/img/dtg_rx_tx_drop.svg | 28 +++++
doc/guides/howto/img/dtg_sample_app_model.svg | 105 ++++++++++++++++++
doc/guides/howto/img/dtg_service.svg | 15 +++
11 files changed, 327 insertions(+)
create mode 100644 doc/guides/howto/img/dtg_consumer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_crypto.svg
create mode 100644 doc/guides/howto/img/dtg_distributor_worker.svg
create mode 100644 doc/guides/howto/img/dtg_mempool.svg
create mode 100644 doc/guides/howto/img/dtg_pdump.svg
create mode 100644 doc/guides/howto/img/dtg_producer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_qos_tx.svg
create mode 100644 doc/guides/howto/img/dtg_rx_rate.svg
create mode 100644 doc/guides/howto/img/dtg_rx_tx_drop.svg
create mode 100644 doc/guides/howto/img/dtg_sample_app_model.svg
create mode 100644 doc/guides/howto/img/dtg_service.svg
diff --git a/doc/guides/howto/img/dtg_consumer_ring.svg b/doc/guides/howto/img/dtg_consumer_ring.svg
new file mode 100644
index 000000000..d0e0b30f8
--- /dev/null
+++ b/doc/guides/howto/img/dtg_consumer_ring.svg
@@ -0,0 +1,19 @@
+<svg width="634" height="97" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>consumer ring</title>
+ <rect fill="#fff" id="canvas_background" height="99" width="636" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="95" width="135" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_2" height="95" width="135" y="2" x="252" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_3" height="95" width="135" y="2" x="499" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="45" x="10" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="52" x="268" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_6" y="53" x="517" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 3</text>
+ <path transform="rotate(90.19324493408203 177.864990234375,48.89100646972658) " id="svg_7" d="m155.364991,48.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path transform="rotate(90.19324493408203 434.864990234375,48.89100646972656) " id="svg_8" d="m412.364991,48.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path id="svg_10" d="m225.937481,47.95781l0,0c0,-10.493411 8.282734,-19 18.500005,-19l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564971c3.469424,3.563191 5.418523,8.395917 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999998 -18.5,18.999998l0,0c-10.217271,0 -18.500005,-8.506591 -18.500005,-18.999998zm18.500005,-19l0,37.999998m-18.500005,-18.999998l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ <path id="svg_11" d="m475.937481,46.95781l0,0c0,-10.493411 8.282734,-19.000001 18.500005,-19.000001l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564972c3.469424,3.563191 5.418523,8.395916 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999997 -18.5,18.999997l0,0c-10.217271,0 -18.500005,-8.50659 -18.500005,-18.999997zm18.500005,-19.000001l0,37.999998m-18.500005,-18.999997l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_crypto.svg b/doc/guides/howto/img/dtg_crypto.svg
new file mode 100644
index 000000000..7687aaa4a
--- /dev/null
+++ b/doc/guides/howto/img/dtg_crypto.svg
@@ -0,0 +1,16 @@
+<svg width="180" height="230" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>crypto</title>
+ <rect fill="#fff" id="canvas_background" height="232" width="182" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="1" width="0" y="369.4375" x="838.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="222.4375" x="39.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core 7</text>
+ <rect stroke="#000" id="svg_26" height="44.000003" width="155.000001" y="151.437497" x="6.499997" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_27" y="179.4375" x="13.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">CRYPTO PMD</text>
+ <ellipse stroke="#000" ry="41.5" rx="66.500001" id="svg_28" cy="48.9375" cx="87.999997" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_29" y="57.4375" x="45.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Device</text>
+ <path stroke="#000" transform="rotate(-90.19718170166016 85.76751708984375,119.45785522460936) " id="svg_30" d="m40.743289,119.457895l26.106598,-13.915756l0,6.95786l37.835289,0l0,-6.95786l26.106586,13.915756l-26.106586,13.915682l0,-6.957842l-37.835289,0l0,6.957842l-26.106598,-13.915682z" stroke-opacity="null" stroke-width="0" fill="#845335"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_distributor_worker.svg b/doc/guides/howto/img/dtg_distributor_worker.svg
new file mode 100644
index 000000000..3df758fa6
--- /dev/null
+++ b/doc/guides/howto/img/dtg_distributor_worker.svg
@@ -0,0 +1,31 @@
+<svg width="600" height="235" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>dsitributor and worker</title>
+ <rect fill="#fff" id="canvas_background" height="237" width="602" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_1" height="89.999991" width="202.000004" y="21.000003" x="4" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start"
+ font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="47" x="7"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">PKT classify</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_12" y="96.5" x="25.992188" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Distribute</text>
+ <path id="svg_13" d="m334.224445,98.046256l16.252783,-5.432074l-16.252783,-5.429993l-16.256262,-5.427952l0,6.038517c-4.264321,-0.004069 -11.331234,-0.004069 -13.334561,-0.004069c-5.919233,-0.081711 -9.804243,-1.153783 -13.847174,-3.073397c-6.02597,-2.865095 -11.16889,-8.021466 -17.371313,-12.989922c-3.154554,-2.470933 -6.669049,-4.921502 -11.349751,-6.892119c-4.633926,-1.978838 -10.68815,-3.351144 -17.41807,-3.324591l-29.170102,0l0,9.646983l29.170102,0c4.056131,0.042884 6.623169,0.808657 10.124445,2.70786c5.179989,2.824261 10.230311,8.223602 17.195784,13.339136c6.826056,5.066481 17.329865,10.182006 32.564612,10.231017l13.436029,0l0,6.0385l16.256262,-5.427896zm-68.759273,-36.535597c0.184389,0.075572 0.379369,0.14089 0.559297,0.218526c5.559349,2.340242 9.573988,5.211493 12.751502,7.709001c0.730423,0.584013 1.377934,1.145611 2.071302,1.719456l37.120911,0l0,6.040581l16.256262,-5.42798l16.252783,-5.429962l-16.252783,-5.432037l-16.256262,-5.434093l0,6.036509l-52.503011,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFAC8D"/>
+ <rect stroke="#000" id="svg_17" height="110.999999" width="180" y="3.4375" x="353.5"
+ stroke-opacity="null" stroke-width="0" fill="#FABA6F"/>
+ <rect stroke="#000" id="svg_14" height="110.999999" width="180" y="31.4375" x="369.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_15" height="110.999999" width="180" y="58.4375" x="392.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFF093"/>
+ <rect stroke="#000" id="svg_16" height="110.999999" width="180" y="90.4375" x="413.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,142.60348030924797,64.2037038393319) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="71.490132" x="413.769676" stroke-opacity="null" stroke-width="0" fill="#000000">worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="101.332237" x="406.404514" stroke-opacity="null" stroke-width="0" fill="#000000">worker 2</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_20" y="58.700658" x="376.943866" stroke-opacity="null" stroke-width="0" fill="#000000">worker 3</text>
+ <text style="cursor: move;" stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_22" y="18.911184" x="350.429283" stroke-opacity="null" stroke-width="0" fill="#000000">worker 4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="139.4375" x="49.5"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_24" y="226.4375" x="404.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4,5</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_mempool.svg b/doc/guides/howto/img/dtg_mempool.svg
new file mode 100644
index 000000000..f3d31a96e
--- /dev/null
+++ b/doc/guides/howto/img/dtg_mempool.svg
@@ -0,0 +1,22 @@
+<svg width="955" height="185" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>mempool</title>
+ <rect fill="#fff" id="canvas_background" height="187" width="957" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="1" width="0" y="370.4375" x="839.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <rect stroke="#000" id="svg_31" height="181.000002" width="951.000029" y="2.4375" x="3.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_32" y="167.4375" x="14.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">MBUF pool</text>
+ <rect id="svg_38" height="71" width="859" y="15.4375" x="18.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_39" height="71" width="859" y="32.4375" x="33.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_40" height="71" width="859" y="47.4375" x="52.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_41" height="71" width="859" y="62.4375" x="71.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect stroke="#000" id="svg_42" height="55.999999" width="403.000005" y="69.4375" x="486.499995" stroke-opacity="null" stroke-width="0" fill="#FFAC8D"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_43" y="103.4375" x="631.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Payload</text>
+ <rect stroke="#000" id="svg_44" height="56" width="140.999999" y="68.4375" x="292.500001" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_45" y="105.4375" x="303.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Metadata</text>
+ <rect stroke="#000" id="svg_46" height="53" width="185.999997" y="70.4375" x="90.5" stroke-opacity="null" stroke-width="0" fill="#EDFFFF"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_47" y="104.4375" x="105.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">struct mbuf</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_pdump.svg b/doc/guides/howto/img/dtg_pdump.svg
new file mode 100644
index 000000000..1dc63bed7
--- /dev/null
+++ b/doc/guides/howto/img/dtg_pdump.svg
@@ -0,0 +1,28 @@
+<svg width="860" height="140" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>pdump</title>
+ <rect fill="#fff" id="canvas_background" height="142" width="862" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_15" height="127.999992" width="568" y="7" x="8.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#B1FFFF"/>
+ <rect id="svg_2" height="51" width="100" y="23" x="22.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_3" height="51" width="100" y="26" x="455.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CC7C65"/>
+ <path id="svg_7" d="m208.364769,78.968848l12.704559,-5.623238l12.697917,-5.625551l-12.704644,-5.629525l-12.697836,-5.625202l0,6.255891c-3.112384,0 -6.729051,0 -7.940704,0c-5.776899,0.124913 -12.907107,-3.365352 -20.492742,-7.864703c-2.827801,-1.577599 -5.680695,-3.313223 -8.704363,-4.88042c3.023667,-1.569118 5.872475,-3.302747 8.704363,-4.880295c7.585635,-4.493132 14.715839,-7.98967 20.485292,-7.862546l7.94815,-0.002103l0,6.264275l12.697836,-5.627447l12.704644,-5.631775l-12.704644,-5.637807l-12.697836,-5.629547l0,6.260273l-7.944067,0c-11.799214,0.12679 -20.61742,5.506774 -28.333828,9.879239c-7.660201,4.613909 -14.545738,8.087381 -18.539115,7.862617l-22.780531,0l0,9.993806l22.788007,0c3.989952,-0.226969 10.878944,3.248761 18.53841,7.86254c7.71303,4.374678 16.52785,9.750196 28.330427,9.879312l7.940704,0l0,6.262203l0,0l0,0.000003z" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#559E84"/>
+ <path id="svg_8" d="m439.457308,43.77545l-10.478696,-7.306254l0,8.12971l-18.805519,0c-3.29272,0.296972 -8.977707,-4.212008 -15.298608,-10.195089c-6.365074,-5.675375 -13.639418,-12.638687 -23.37938,-12.824984l-6.552957,0l0,12.959904l6.552957,0c4.770151,-0.167389 10.652051,4.360514 16.914219,10.203329c2.334186,2.041106 4.688512,4.292919 7.183208,6.326016c-2.494696,2.035766 -4.846223,4.290275 -7.183208,6.331424c-6.262164,5.826551 -12.144067,10.365242 -16.908056,10.200515l-6.559117,0l0,12.978808l6.555749,0c9.719269,-0.1646 16.996426,-7.14678 23.382185,-12.814082c6.32147,-5.983193 12.003677,-10.494803 15.299177,-10.203221l18.799357,0l0,8.129634l10.478696,-7.298036l10.484823,-7.303408l-10.484823,-7.314263l-0.000006,-0.000004z" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#559E84"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="54" x="53.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_10" y="57" x="486.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_11" y="40" x="243.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_12" y="77" x="243.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="82" x="326.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="40" x="327.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q3</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="114.86203" x="234.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Primary</text>
+ <rect stroke="#000" id="svg_18" height="90.000001" width="201.999994" y="29.437499" x="654.5" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="102.4375" x="684.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Secondary</text>
+ <rect stroke="#000" id="svg_20" height="42.000001" width="184.000001" y="34.437499" x="664.499994" stroke-opacity="null" stroke-width="0" fill="#A0D58A"/>
+ <rect id="svg_21" height="1" width="0" y="366.4375" x="835.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_22" y="59.4375" x="668.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Ring BufferQ</text>
+ <path stroke="#000" transform="rotate(89.78996276855469 615.4766845703124,74.49395751953122) " id="svg_24" d="m599.023803,74.414257l17.255478,-33.667333l15.650356,33.667333l-8.226459,0l0,33.826726l-16.452915,0l0,-33.826726l-8.226459,0z" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="22.4375" x="714.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core 0</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_producer_ring.svg b/doc/guides/howto/img/dtg_producer_ring.svg
new file mode 100644
index 000000000..5657702e5
--- /dev/null
+++ b/doc/guides/howto/img/dtg_producer_ring.svg
@@ -0,0 +1,19 @@
+<svg width="634" height="97" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>producer ring</title>
+ <rect fill="#fff" id="canvas_background" height="99" width="636" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="95" width="135" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_2" height="95" width="135" y="2" x="252" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_3" height="95" width="135" y="2" x="499" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="45" x="10" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="52" x="268" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_6" y="53" x="517" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 3</text>
+ <path transform="rotate(90.19324493408203 197.864990234375,42.89100646972657) " id="svg_7" d="m175.364991,42.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path transform="rotate(90.19324493408203 445.864990234375,47.89100646972657) " id="svg_8" d="m423.364991,47.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path id="svg_10" d="m120.937481,42.95781l0,0c0,-10.493411 8.282734,-19 18.500005,-19l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564971c3.469424,3.563191 5.418523,8.395917 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999998 -18.5,18.999998l0,0c-10.217271,0 -18.500005,-8.506591 -18.500005,-18.999998zm18.500005,-19l0,37.999998m-18.500005,-18.999998l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ <path id="svg_11" d="m373.937481,45.95781l0,0c0,-10.493411 8.282734,-19.000001 18.500005,-19.000001l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564972c3.469424,3.563191 5.418523,8.395916 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999997 -18.5,18.999997l0,0c-10.217271,0 -18.500005,-8.50659 -18.500005,-18.999997zm18.500005,-19.000001l0,37.999998m-18.500005,-18.999997l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_qos_tx.svg b/doc/guides/howto/img/dtg_qos_tx.svg
new file mode 100644
index 000000000..6e46d188a
--- /dev/null
+++ b/doc/guides/howto/img/dtg_qos_tx.svg
@@ -0,0 +1,24 @@
+<svg width="287" height="180" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>qos</title>
+ <rect fill="#fff" id="canvas_background" height="182" width="289" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="180" width="287" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CBFFFF"/>
+ <rect id="svg_10" height="82" width="78" y="17.808165" x="152.25" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_11" height="82" width="78" y="36.808165" x="171.25" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_12" height="82" width="78" y="59.808165" x="188.25" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="113.808165" x="211.25" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="161.808165" x="161.25" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core1</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_15" y="235.775764" x="531.631651" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.3615568424500273,0,0,0.9358925757041695,106.13283890389599,-107.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="173.881425" x="192.8023" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_17" y="66.670315" x="-70.838675" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ <rect id="svg_18" height="103" width="104" y="26" x="26" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FABA6F"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="79" x="50" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">QoS</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_20" y="153" x="27" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 10</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_rx_rate.svg b/doc/guides/howto/img/dtg_rx_rate.svg
new file mode 100644
index 000000000..311c9c43f
--- /dev/null
+++ b/doc/guides/howto/img/dtg_rx_rate.svg
@@ -0,0 +1,20 @@
+<svg width="116" height="152" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>rx rate</title>
+ <rect fill="#fff" id="canvas_background" height="154" width="118" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="82" width="78" y="0.812613" x="0.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_2" height="82" width="78" y="19.812613" x="19.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_3" height="82" width="78" y="42.812613" x="36.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="96.812613" x="59.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="144.812613" x="9.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core0</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_7" y="214.621694" x="210.788594" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.36155684245002745,0,0,0.9358925757041695,-60.867161096103985,-178.0100101290174) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_8" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="42.29845" x="-417.420836" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_rx_tx_drop.svg b/doc/guides/howto/img/dtg_rx_tx_drop.svg
new file mode 100644
index 000000000..2f4e9d587
--- /dev/null
+++ b/doc/guides/howto/img/dtg_rx_tx_drop.svg
@@ -0,0 +1,28 @@
+<svg width="275" height="154" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>rx-tx drops</title>
+ <rect fill="#fff" id="canvas_background" height="156" width="277" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="82" width="78" y="0.808165" x="0.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_2" height="82" width="78" y="19.808165" x="19.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_3" height="82" width="78" y="42.808165" x="36.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="96.808165" x="59.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="144.808165" x="9.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core0</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_7" y="214.616159" x="210.788594" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.3615568424500274,0,0,0.9358925757041695,-60.867161096103985,-178.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_8" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="42.292073" x="-417.420836" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ <rect id="svg_10" height="82" width="78" y="2.808165" x="159.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_11" height="82" width="78" y="21.808165" x="178.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_12" height="82" width="78" y="44.808165" x="195.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="98.808165" x="218.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="146.808165" x="168.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core1</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_15" y="217.105524" x="547.514971" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.36155684245002734,0,0,0.9358925757041695,98.13283890389599,-176.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_17" y="45.160101" x="-53.681142" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_sample_app_model.svg b/doc/guides/howto/img/dtg_sample_app_model.svg
new file mode 100644
index 000000000..ad3b0995d
--- /dev/null
+++ b/doc/guides/howto/img/dtg_sample_app_model.svg
@@ -0,0 +1,105 @@
+<svg width="1418" height="379" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>sample application model</title>
+ <rect fill="#fff" id="canvas_background" height="381" width="1420" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_84" height="378.999996" width="1417.999937" y="0" x="0" stroke-opacity="null" stroke-width="0" fill="#B1FFFF"/>
+ <rect stroke="#000" id="svg_1" height="79.999993" width="187.000011" y="267" x="635.499996" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <rect stroke="#000" id="svg_14" height="28.000001" width="175.000003" y="271.437495" x="641" stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_16" height="36.000002" width="142" y="304.437495" x="659" stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(0.7497134942573729,0,0,0.7607963681117937,149.70768863149087,72.0554119542491) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="335.740664" x="684.214296"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Health Check</text>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="368.437495" x="685"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 7</text>
+ <text stroke="#000" transform="matrix(0.7446371555386645,0,0,0.7004599746900311,157.05848471617847,107.2548065316271) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="261.052167" x="658.840014"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Stats Collector</text>
+ <rect id="svg_26" height="91" width="88" y="89.4375" x="2"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFA9A2"/>
+ <rect id="svg_27" height="91" width="88" y="107.4375" x="24"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CE7975"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_28" y="166.4375" x="52"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6666666865348816,14.972222477197647,39.14583100005984) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_30" y="96.937501" x="-11.409091"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 1</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6666666865348816,14.972222477197647,39.14583100005984) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_33" y="128.4375" x="27.863636"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_34" y="216.4375" x="26"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core0</text>
+ <rect id="svg_35" height="91" width="88" y="55.4375" x="1303"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFA9A2"/>
+ <rect id="svg_36" height="91" width="88" y="73.4375" x="1325"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CE7975"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_37" y="132.4375" x="1352"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_40" y="184.4375" x="1327"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core1</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6296296119689941,200.86111453175545,24.236112266778946) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_43" y="74.319853" x="1811.681832"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 1</text>
+ <text stroke="#000" transform="matrix(0.6958672408102909,0,0,0.6184720487972513,537.7539486343405,-28.040291137315034) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_44" y="191.620936" x="1140.58332"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 2</text>
+ <rect stroke="#000" id="svg_45" height="62" width="83.000003" y="101.4375" x="1096"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_46" y="135.4375" x="1115"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">QoS</text>
+ <rect stroke="#000" id="svg_48" height="61" width="107" y="110.4375" x="649"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_49" y="145.4375" x="661"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">Crypto</text>
+ <rect stroke="#000" id="svg_50" height="70.999998" width="84.000002" y="65.4375" x="422"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <rect stroke="#000" id="svg_51" height="74.000003" width="90.000002" y="92.4375" x="444"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_52" height="71.999998" width="92" y="120.4375" x="468"
+ stroke-opacity="null" stroke-width="0" fill="#C48157"/>
+ <text stroke="#000" transform="matrix(0.6528342962265015,0,0,0.5925925970077515,63.70490664243698,168.43749817460775) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_53" y="-145.374996" x="553.425418"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.6666666865348816,166.69894686341286,-165.52084343507886) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_54" y="417.437503" x="420.257696" stroke-opacity="null" stroke-width="0" fill="#000000">Worker 2</text>
+ <text stroke="#000" transform="matrix(0.7483048439025879,0,0,0.7407407164573669,76.38947987556458,153.33566251024604) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_55" y="-19.012521" x="530.011964"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 3</text>
+ <rect stroke="#000" id="svg_62" height="70.999998" width="84.000002" y="67.4375" x="865"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <rect stroke="#000" id="svg_63" height="74.000003" width="90.000002" y="94.4375" x="887"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_64" height="71.999998" width="92" y="122.4375" x="911"
+ stroke-opacity="null" stroke-width="0" fill="#C48157"/>
+ <text stroke="#000" transform="matrix(0.6528342962265015,0,0,0.5925925970077515,63.70490664243698,168.43749817460775) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_65" y="-143.687496" x="1238.132093"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.6666666865348816,166.69894686341286,-165.52084343507886) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_66" y="420.437503" x="1072.811052"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 2</text>
+ <text stroke="#000" transform="matrix(0.7483048439025879,0,0,0.7407407164573669,76.38947987556458,153.33566251024604) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_67" y="-16.312521" x="1122.016685"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 3</text>
+ <path stroke="#000" id="svg_68" d="m653.747516,12.862181l19.36745,-10.781877l77.632553,0l0,53.999988l-97.000004,0l0,-43.218111z"
+ stroke-opacity="null" stroke-width="0" fill="#805064"/>
+ <text stroke="#000" transform="matrix(0.733815550804138,0,0,0.9629629850387573,306.63544338941574,-1.3912058547139168) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_69" y="33.322117" x="496.534253"
+ stroke-opacity="null" stroke-width="0" fill="#ffffff">Device</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_70" y="217.4375" x="457"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_71" y="197.4375" x="653"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 5</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_72" y="186.4375" x="1097"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 6</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_73" y="220.4375" x="885"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4</text>
+ <rect stroke="#000" id="svg_74" height="88.000002" width="110.999999" y="104" x="206.000001"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text stroke="#000" transform="matrix(0.5661651903991256,0,0,1,107.56695064249905,0) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_75" y="133" x="182.901935"
+ stroke-opacity="null" stroke-width="0" fill="#000000">PKT classify</text>
+ <text stroke="#000" transform="matrix(0.5444634556770325,0,0,1,115.93405053019524,0) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_76" y="170" x="196.644877"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Distribute</text>
+ <path id="svg_77" d="m189.755816,142.38346l-9.615746,-5.572568l0,6.200627l-17.256831,0c-3.021555,0.226504 -8.238368,-3.212549 -14.038725,-7.775916c-5.840892,-4.328677 -12.516173,-9.639678 -21.454022,-9.781769l-6.013302,0l0,9.884674l6.013302,0c4.377316,-0.12767 9.774825,3.325816 15.521285,7.782201c2.14196,1.556776 4.3024,3.274261 6.59165,4.824928c-2.28925,1.552703 -4.447123,3.272244 -6.59165,4.829053c-5.746457,4.44398 -11.143969,7.905694 -15.51563,7.780055l-6.018955,0l0,9.899092l6.015865,0c8.91886,-0.125542 15.596722,-5.450935 21.456596,-9.773453c5.800879,-4.563453 11.01514,-8.004512 14.039247,-7.782119l17.251176,0l0,6.200569l9.615746,-5.566299l9.621368,-5.570396l-9.621368,-5.578676l-0.000005,-0.000003z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path id="svg_78" d="m417.79301,181.072084l13.604178,-5.205736l-13.604178,-5.203742l-13.60709,-5.201786l0,5.786911c-3.569394,-0.003899 -9.48466,-0.003899 -11.161519,-0.003899c-4.954616,-0.078306 -8.206513,-1.105708 -11.590595,-2.945338c-5.043959,-2.745716 -9.348773,-7.687237 -14.54043,-12.448672c-2.640478,-2.367977 -5.58224,-4.716439 -9.50016,-6.604946c-3.878767,-1.896386 -8.946375,-3.211512 -14.579566,-3.186066l-24.41645,0l0,9.245023l24.41645,0c3.395131,0.041097 5.543836,0.774963 8.474533,2.595032c4.335842,2.706583 8.563147,7.88095 14.393505,12.783335c5.71366,4.855376 14.505736,9.757754 27.25778,9.804723l11.246452,0l0,5.786895l13.60709,-5.201732zm-57.554046,-35.013272c0.15434,0.072423 0.317546,0.135019 0.468152,0.209421c4.65338,2.242732 8.013781,4.994346 10.673477,7.387791c0.611391,0.559679 1.153382,1.097877 1.733756,1.647811l31.071571,0l0,5.788888l13.60709,-5.201813l13.604178,-5.203713l-13.604178,-5.205701l-13.60709,-5.207671l0,5.784986l-43.946955,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path stroke="#000" transform="rotate(89.46277618408203 605.7476806640625,144.42587280273435) " id="svg_79" d="m590.864808,144.339395l14.882874,-35.951591l14.882874,35.951591l-7.441438,0l0,36.124583l-14.882873,0l0,-36.124583l-7.441438,0z" stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ <path stroke="#000" transform="rotate(89.74066925048828 697.6382446289062,81.67762756347659) " id="svg_80" d="m661.638136,81.677665l20.874103,-13.000079l0,6.500022l30.252035,0l0,-6.500022l20.874093,13.000079l-20.874093,13.000009l0,-6.500005l-30.252035,0l0,6.500005l-20.874103,-13.000009z" fill-opacity="null"
+ stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ <path id="svg_81" d="m858.79301,172.072084l13.604178,-5.205736l-13.604178,-5.203742l-13.60709,-5.201786l0,5.786911c-3.569394,-0.0039 -9.48466,-0.0039 -11.161519,-0.0039c-4.954616,-0.078305 -8.206513,-1.105708 -11.590595,-2.945338c-5.043959,-2.745715 -9.348773,-7.687236 -14.54043,-12.448672c-2.640477,-2.367977 -5.58224,-4.716438 -9.50016,-6.604945c-3.878767,-1.896387 -8.946375,-3.211513 -14.579566,-3.186067l-24.416451,0l0,9.245023l24.416451,0c3.395131,0.041097 5.543836,0.774963 8.474533,2.595032c4.335842,2.706583 8.563147,7.88095 14.393505,12.783335c5.713661,4.855376 14.505736,9.757754 27.257781,9.804723l11.246451,0l0,5.786894l13.60709,-5.201732zm-57.554045,-35.013272c0.15434,0.072424 0.317546,0.135019 0.468152,0.209421c4.65338,2.242731 8.013781,4.994346 10.673477,7.38779c0.61139,0.559679 1.153381,1.097878 1.733756,1.647812l31.07157,0l0,5.788888l13.60709,-5.201812l13.604178,-5.203713l-13.604178,-5.205701l-13.60709,-5.207671l0,5.784986l-43.946955,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path id="svg_82" d="m1080.755816,127.38346l-9.615746,-5.572567l0,6.200627l-17.256831,0c-3.021555,0.226504 -8.238368,-3.212549 -14.038725,-7.775916c-5.840892,-4.328677 -12.516173,-9.639678 -21.454022,-9.78177l-6.013302,0l0,9.884674l6.013302,0c4.377316,-0.12767 9.774825,3.325816 15.521285,7.782201c2.141959,1.556776 4.3024,3.274261 6.59165,4.824928c-2.28925,1.552704 -4.447123,3.272244 -6.59165,4.829053c-5.746457,4.443981 -11.143969,7.905694 -15.51563,7.780055l-6.018955,0l0,9.899093l6.015865,0c8.91886,-0.125543 15.596723,-5.450935 21.456596,-9.773454c5.800879,-4.563453 11.01514,-8.004512 14.039246,-7.782119l17.251177,0l0,6.20057l9.615746,-5.566299l9.621368,-5.570397l-9.621368,-5.578676l-0.000006,-0.000003z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path stroke="#000" id="svg_83" d="m1269.963506,163.318859l13.567973,-4.653717l13.56088,-4.655631l-13.568064,-4.658919l-13.560793,-4.655342l0,5.177292c-3.323905,0 -7.186364,0 -8.480362,0c-6.169502,0.103377 -13.784287,-2.785121 -21.885448,-6.508723c-3.019981,-1.3056 -6.066761,-2.741979 -9.295919,-4.03897c3.229159,-1.298581 6.271574,-2.733309 9.295919,-4.038867c8.101162,-3.718456 15.715942,-6.612144 21.877492,-6.506938l8.488314,-0.001741l0,5.18423l13.560793,-4.6572l13.568064,-4.660782l-13.568064,-4.665774l-13.560793,-4.658938l0,5.180918l-8.483954,0c-12.6011,0.104929 -22.018599,4.557333 -30.259422,8.175926c-8.180795,3.81841 -15.53428,6.693008 -19.799052,6.506997l-24.328718,0l0,8.27074l24.336702,0c4.261113,-0.187837 11.618288,2.688631 19.798299,6.506933c8.237215,3.620425 17.651098,8.069132 30.25579,8.175987l8.480362,0l0,5.182516l0,0l0,0.000002z" fill-opacity="null"
+ stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_service.svg b/doc/guides/howto/img/dtg_service.svg
new file mode 100644
index 000000000..2d876a3a1
--- /dev/null
+++ b/doc/guides/howto/img/dtg_service.svg
@@ -0,0 +1,15 @@
+<svg width="254" height="145" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>service</title>
+ <rect fill="#fff" id="canvas_background" height="147" width="256" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_1" height="109.999987" width="254.000014" y="0" x="0" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <rect stroke="#000" id="svg_14" height="37.000001" width="225" y="7.437494" x="4.5" stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_16" height="45.000002" width="186.000003" y="58.437493" x="33.5" stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(1.0013854504218995,0,0,1.0926463039877063,73.76856116958965,-122.08861649089796) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="193.234194" x="-35.792216" stroke-opacity="null" stroke-width="0" fill="#000000">Health Check</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="138.437495" x="45.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 6</text>
+ <text stroke="#000" transform="matrix(1.0013854504218995,0,0,1.0926463039877063,73.76856116958965,-122.08861649089796) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="141.982476" x="-63.753477" stroke-opacity="null" stroke-width="0" fill="#000000">Stats Collector</text>
+ </g>
+</svg>
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v4 2/2] doc: add guide for debug and troubleshoot
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 0/2] doci/howto: add debug and troubleshoot guide Vipin Varghese
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 1/2] doc: add svg for " Vipin Varghese
@ 2019-01-16 14:54 ` Vipin Varghese
2019-01-18 15:28 ` Kovacevic, Marko
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 0/2] doc/howto: add debug and troubleshoot guide Vipin Varghese
1 sibling, 2 replies; 23+ messages in thread
From: Vipin Varghese @ 2019-01-16 14:54 UTC (permalink / raw)
To: dev, shreyansh.jain, thomas
Cc: john.mcnamara, marko.kovacevic, amol.patel, sanjay.padubidri,
Vipin Varghese
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 15656 bytes --]
Add user guide on debug and troubleshoot for common issues and bottleneck
found in sample application model.
Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
---
doc/guides/howto/debug_troubleshoot_guide.rst | 375 ++++++++++++++++++
doc/guides/howto/index.rst | 1 +
2 files changed, 376 insertions(+)
create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
diff --git a/doc/guides/howto/debug_troubleshoot_guide.rst b/doc/guides/howto/debug_troubleshoot_guide.rst
new file mode 100644
index 000000000..f2e337bb1
--- /dev/null
+++ b/doc/guides/howto/debug_troubleshoot_guide.rst
@@ -0,0 +1,375 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2018 Intel Corporation.
+
+.. _debug_troubleshoot_via_pmd:
+
+Debug & Troubleshoot guide via PMD
+==================================
+
+DPDK applications can be designed to run as single thread simple stage to
+multiple threads with complex pipeline stages. These application can use poll
+mode devices which helps in offloading CPU cycles. A few models are
+
+ * single primary
+ * multiple primary
+ * single primary single secondary
+ * single primary multiple secondary
+
+In all the above cases, it is a tedious task to isolate, debug and understand
+odd behaviour which occurs randomly or periodically. The goal of guide is to
+share and explore a few commonly seen patterns and behaviour. Then, isolate
+and identify the root cause via step by step debug at various processing
+stages.
+
+Application Overview
+--------------------
+
+Let us take up an example application as reference for explaining issues and
+patterns commonly seen. The sample application in discussion makes use of
+single primary model with various pipeline stages. The application uses PMD
+and libraries such as service cores, mempool, pkt mbuf, event, crypto, QoS
+and eth.
+
+The overview of an application modeled using PMD is shown in
+:numref:`dtg_sample_app_model`.
+
+.. _dtg_sample_app_model:
+
+.. figure:: img/dtg_sample_app_model.*
+
+ Overview of pipeline stage of an application
+
+Bottleneck Analysis
+-------------------
+
+To debug the bottleneck and performance issues the desired application
+is made to run in an environment matching as below
+
+#. Linux 64-bit|32-bit
+#. DPDK PMD and libraries are used
+#. Libraries and PMD are either static or shared. But not both
+#. Machine flag optimizations of gcc or compiler are made constant
+
+Is there mismatch in packet rate (received < send)?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RX Port and associated core :numref:`dtg_rx_rate`.
+
+.. _dtg_rx_rate:
+
+.. figure:: img/dtg_rx_rate.*
+
+ RX send rate compared against Received rate
+
+#. Are generic configuration correct?
+ - What is port Speed, Duplex? rte_eth_link_get()
+ - Are packets of higher sizes are dropped? rte_eth_get_mtu()
+ - Are only specific MAC received? rte_eth_promiscuous_get()
+
+#. Are there NIC specific drops?
+ - Check rte_eth_rx_queue_info_get() for nb_desc and scattered_rx
+ - Is RSS enabled? rte_eth_dev_rss_hash_conf_get()
+ - Are packets spread on all queues? rte_eth_dev_stats()
+ - If stats for RX and drops updated on same queue? check receieve thread
+ - If packet does not reach PMD? check if offload for port and queue
+ matches to traffic pattern send.
+
+#. If problem still persists, this might be at RX lcore thread
+ - Check if RX thread, distributor or event rx adapter? these may be
+ processing less than required
+ - Is the application is build using processing pipeline with RX stage? If
+ there are multiple port-pair tied to a single RX core, try to debug by
+ using rte_prefetch_non_temporal(). This will intimate the mbuf in cache
+ is temporary.
+
+Are there packet drops (receive|transmit)?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RX-TX Port and associated cores :numref:`dtg_rx_tx_drop`.
+
+.. _dtg_rx_tx_drop:
+
+.. figure:: img/dtg_rx_tx_drop.*
+
+ RX-TX drops
+
+#. At RX
+ - Get RX queue count? nb_rx_queues using rte_eth_dev_info_get()
+ - Are there miss, errors, qerros? rte_eth_dev_stats() for imissed,
+ ierrors, q_erros, rx_nombuf, rte_mbuf_ref_count
+
+#. At TX
+ - Are you doing in bulk TX? check application for TX descriptor overhead.
+ - Are there TX errors? rte_eth_dev_stats() for oerrors and qerros
+ - Is specific scenarios not releasing mbuf? check rte_mbuf_ref_count of
+ those packets.
+ - Is the packet multi segmented? Check if port and queue offlaod is set.
+
+Are there object drops in producer point for ring?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Producer point for ring :numref:`dtg_producer_ring`.
+
+.. _dtg_producer_ring:
+
+.. figure:: img/dtg_producer_ring.*
+
+ Producer point for Rings
+
+#. Performance for Producer
+ - Fetch the type of RING 'rte_ring_dump()' for flags (RING_F_SP_ENQ)
+ - If '(burst enqueue - actual enqueue) > 0' check rte_ring_count() or
+ rte_ring_free_count()
+ - If 'burst or single enqueue returning always 0'? is rte_ring_full()
+ true then next stage is not pulling the content at desired rate.
+
+Are there object drops in consumer point for ring?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Consumer point for ring :numref:`dtg_consumer_ring`.
+
+.. _dtg_consumer_ring:
+
+.. figure:: img/dtg_consumer_ring.*
+
+ Consumer point for Rings
+
+#. Performance for Consumer
+ - Fetch the type of RING – rte_ring_dump() for flags (RING_F_SC_DEQ)
+ - If '(burst dequeue - actual dequeue) > 0' for rte_ring_free_count()
+ - If 'burst or single enqueue' always results 0 check the ring is empty
+ via rte_ring_empty()
+
+Are packets or objects are not processed at desired rate?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Memory objects close to NUMA :numref:`dtg_mempool`.
+
+.. _dtg_mempool:
+
+.. figure:: img/dtg_mempool.*
+
+ Memory objects has to be close to device per NUMA
+
+#. Is the performance low?
+ - Are packets received from multiple NIC? rte_eth_dev_count_all()
+ - Are NIC interfaces on different socket? use rte_eth_dev_socket_id()
+ - Is mempool created with right socket? rte_mempool_create() or
+ rte_pktmbuf_pool_create()
+ - Are drops on specific socket? If yes check if there are sufficent
+ objects by rte_mempool_get_count() or rte_mempool_avail_count()
+ - Is 'rte_mempool_get_count() or rte_mempool_avail_count()' zero?
+ application requires more objects hence reconfigure number of
+ elements in rte_mempool_create().
+ - Is there single RX thread for multiple NIC? try having multiple
+ lcore to read from fixed interface or we might be hitting cache
+ limit, so increase cache_size for pool_create().
+
+#. Is performance low for some sceanrios?
+ - Check if sufficient objects in mempool by rte_mempool_avail_count()
+ - Is failure seen in some packets? we might be getting packets with
+ 'size > mbuf data size'.
+ - Is NIC offload or application handling multi segment mbuf? check the
+ special packets are continuous with rte_pktmbuf_is_contiguous().
+ - If there separate user threads used to access mempool objects, use
+ rte_mempool_cache_create() for non DPDK threads.
+ - Is the error reproducible with 1GB hugepage? If no, then try debuging
+ the issue with lookup table or objects with rte_mem_lock_page().
+
+.. note::
+ Stall in release of MBUF can be because
+
+ * Processing pipeline is too heavy
+ * Number of stages are too many
+ * TX is not transferred at desired rate
+ * Multi segment is not offloaded at TX device.
+ * Application misuse scenarios can be
+ - not freeing packets
+ - invalid rte_pktmbuf_refcnt_set
+ - invalid rte_pktmbuf_prefree_seg
+
+Is there difference in performance for crypto?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Crypto device and PMD :numref:`dtg_crypto`.
+
+.. _dtg_crypto:
+
+.. figure:: img/dtg_crypto.*
+
+ CRYPTO and interaction with PMD device
+
+#. Are generic configuration correct?
+ - Get total crypto devices – rte_cryptodev_count()
+ - Cross check software or hardware flags are configured properly
+ rte_cryptodev_info_get() for feature_flags
+
+#. If enqueue request > actual enqueue (drops)?
+ - Is the queue pair setup for right NUMA? check for socket_id using
+ rte_cryptodev_queue_pair_setup().
+ - Is the session_pool created from same socket_id as queue pair? If no,
+ then create on same NUMA.
+ - Is enqueue thread on same socket_id as object? If no, then try
+ to put on same NUMA.
+ - Are there errors and drops? check err_count using rte_cryptodev_stats()
+ - Do multiple threads enqueue or dequeue from same queue pair? Try
+ debugging with separate threads.
+
+#. If enqueue rate > dequeue rate?
+ - Is dequeue lcore thread is same socket_id?
+ - If softwre crypto is in use, check if the CRYPTO Library is build with
+ right (SIMD) flags or check if the queue pair using CPU ISA for
+ feature_flags AVX|SSE|NEON using rte_cryptodev_info_get()
+ - If its hardware assited crypto showing performance variance? Check if
+ hardware is on same NUMA socket as queue pair and session pool.
+
+Worker functions not giving performance?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Custom worker function :numref:`dtg_distributor_worker`.
+
+.. _dtg_distributor_worker:
+
+.. figure:: img/dtg_distributor_worker.*
+
+ Custom worker function performance drops
+
+#. Performance
+ - Threads context switches are more frequent? Identify lcore with
+ rte_lcore() and lcore index mapping with rte_lcore_index(). Best
+ performance when mapping of thread and core is 1:1.
+ - What are lcore role (type or state)? fetch the roles like RTE, OFF and
+ SERVICE using rte_eal_lcore_role().
+ - Check if application has multiple functions running on same service
+ core? registered functions may be exceeeding the desired time slots
+ while running on same service core.
+ - Is function is running on RTE core? check if there are conflicting
+ functions running on same CPU core by rte_thread_get_affinity().
+
+#. Debug
+ - Check what is mode of operation? master core, lcore, service core,
+ and numa count can be fetched with rte_eal_get_configuration().
+ - Is it occurring on special scenario? Analyze run logic with
+ rte_dump_stack(), rte_dump_registers() and rte_memdump() for more
+ insights.
+ - Is 'perf' showing data process or memory stalls in functions? check
+ instruction being generated for functions using objdump.
+
+Service functions are not frequent enough?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+service functions on service cores :numref:`dtg_service`.
+
+.. _dtg_service:
+
+.. figure:: img/dtg_service.*
+
+ functions running on service cores
+
+#. Performance
+ - Get service core count using rte_service_lcore_count() and compare with
+ result of rte_eal_get_configuration()
+ - Check registered service is available using rte_service_get_by_name(),
+ rte_service_get_count() and rte_service_get_name()
+ - Is given service running parallel on multiple lcores?
+ rte_service_probe_capability() and rte_service_map_lcore_get()
+ - Is service running? rte_service_runstate_get()
+
+#. Debug
+ - Find how many services are running on specific service lcore by
+ rte_service_lcore_count_services()
+ - Generic debug via rte_service_dump()
+
+Is there bottleneck in eventdev?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. Are generic configuration correct?
+ - Get event_dev devices? rte_event_dev_count()
+ - Are they created on correct socket_id? - rte_event_dev_socket_id()
+ - Check if HW or SW capabilities? - rte_event_dev_info_get() for
+ event_qos, queue_all_types, burst_mode, multiple_queue_port,
+ max_event_queue|dequeue_depth
+ - Is packet stuck in queue? check for stages (event qeueue) where
+ packets are looped back to same or previous stages.
+
+#. Performance drops in enqueue (event count > actual enqueue)?
+ - Dump the event_dev information? rte_event_dev_dump()
+ - Check stats for queue and port for eventdev
+ - Check the inflight, current queue element for enqueue|deqeue
+
+How to debug QoS via TM?
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+TM on TX interface :numref:`dtg_qos_tx`.
+
+.. _dtg_qos_tx:
+
+.. figure:: img/dtg_qos_tx.*
+
+ Traffic Manager just before TX
+
+#. Is configuration right?
+ - Get current capabilities for DPDK port for max nodes, level, shaper
+ private, shaper shared, sched_n_children and stats_mask using
+ rte_tm_capabilities_get()
+ - Check if current leaf are configured identically by fetching
+ leaf_nodes_identical using rte_tm_capabilities_get()
+ - Get leaf nodes for a dpdk port - rte_tn_get_number_of_leaf_node()
+ - Check level capabilities by rte_tm_level_capabilities_get for n_nodes
+ - Max, nonleaf_max, leaf_max
+ - identical, non_identical
+ - Shaper_private_supported
+ - Stats_mask
+ - Cman wred packet|byte supported
+ - Cman head drop supported
+ - Check node capabilities by rte_tm_node_capabilities_get for n_nodes
+ - Shaper_private_supported
+ - Stats_mask
+ - Cman wred packet|byte supported
+ - Cman head drop supported
+ - Debug via stats - rte_tm_stats_update() and rte_tm_node_stats_read()
+
+Packet is not of right format?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Packet capture before and after processing :numref:`dtg_pdump`.
+
+.. _dtg_pdump:
+
+.. figure:: img/dtg_pdump.*
+
+ Capture points of Traffic at RX-TX
+
+#. Where to capture packets?
+ - Enable pdump in primary to allow secondary to access queue-pair for
+ ports. Thus packets are copied over in RX|TX callback by secondary
+ process using ring buffers.
+ - To capture packet in middle of pipeline stage, user specific hooks
+ or callback are to be used to copy the packets. These packets can
+ be shared to secodnary process via user defined custom rings.
+
+Issue still persists?
+~~~~~~~~~~~~~~~~~~~~~
+
+#. Are there custom or vendor specific offload meta data?
+ - From PMD, then check for META data error and drops.
+ - From application, then check for META data error and drops.
+#. Is multiprocess is used configuration and data processing?
+ - Check enabling or disabling features from secondary is supported or not?
+#. Is there drops for certain scenario for packets or obejcts?
+ - Check user private data in objects by dumping the details for debug.
+
+How to develop custom code to debug?
+------------------------------------
+
+- For single process - the debug functionality is to be added in same
+ process
+- For multiple process - the debug functionality can be added to
+ secondary multi process
+
+.. note::
+
+ Primary's Debug functions invoked via
+ #. Timer call-back
+ #. Service function under service core
+ #. USR1 or USR2 signal handler
diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a642a2be1..9527fa84d 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -18,3 +18,4 @@ HowTo Guides
virtio_user_as_exceptional_path
packet_capture_framework
telemetry
+ debug_troubleshoot_guide
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v4 2/2] doc: add guide for debug and troubleshoot
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
@ 2019-01-18 15:28 ` Kovacevic, Marko
2019-01-21 3:38 ` Varghese, Vipin
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 0/2] doc/howto: add debug and troubleshoot guide Vipin Varghese
1 sibling, 1 reply; 23+ messages in thread
From: Kovacevic, Marko @ 2019-01-18 15:28 UTC (permalink / raw)
To: Varghese, Vipin, dev, shreyansh.jain, thomas
Cc: Mcnamara, John, Patel, Amol, Padubidri, Sanjay A
After checking the patch again I found a few spelling mistakes
> Add user guide on debug and troubleshoot for common issues and
> bottleneck found in sample application model.
>
> Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
> Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
> ---
> doc/guides/howto/debug_troubleshoot_guide.rst | 375
> ++++++++++++++++++
> doc/guides/howto/index.rst | 1 +
> 2 files changed, 376 insertions(+)
> create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
>
<...>
receieve / receive
> + - If stats for RX and drops updated on same queue? check receieve
> thread
> + - If packet does not reach PMD? check if offload for port and queue
> + matches to traffic pattern send.
> +
<...>
Offlaod/ offload
> + - Is the packet multi segmented? Check if port and queue offlaod is set.
> +
> +Are there object drops in producer point for ring?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<...>
sufficent / sufficient
> + - Are drops on specific socket? If yes check if there are sufficent
> + objects by rte_mempool_get_count() or rte_mempool_avail_count()
> + - Is 'rte_mempool_get_count() or rte_mempool_avail_count()' zero?
> + application requires more objects hence reconfigure number of
> + elements in rte_mempool_create().
> + - Is there single RX thread for multiple NIC? try having multiple
> + lcore to read from fixed interface or we might be hitting cache
> + limit, so increase cache_size for pool_create().
> +
Sceanrios/ scenarios
> +#. Is performance low for some sceanrios?
> + - Check if sufficient objects in mempool by rte_mempool_avail_count()
> + - Is failure seen in some packets? we might be getting packets with
> + 'size > mbuf data size'.
> + - Is NIC offload or application handling multi segment mbuf? check the
> + special packets are continuous with rte_pktmbuf_is_contiguous().
> + - If there separate user threads used to access mempool objects, use
> + rte_mempool_cache_create() for non DPDK threads.
debuging / debugging
> + - Is the error reproducible with 1GB hugepage? If no, then try debuging
> + the issue with lookup table or objects with rte_mem_lock_page().
> +
> +.. note::
> + Stall in release of MBUF can be because
<...>
softwre / software
> + - If softwre crypto is in use, check if the CRYPTO Library is build with
> + right (SIMD) flags or check if the queue pair using CPU ISA for
> + feature_flags AVX|SSE|NEON using rte_cryptodev_info_get()
Assited/ assisted
> + - If its hardware assited crypto showing performance variance? Check if
> + hardware is on same NUMA socket as queue pair and session pool.
> +
<...>
exceeeding / exceeding
> + core? registered functions may be exceeeding the desired time slots
> + while running on same service core.
> + - Is function is running on RTE core? check if there are conflicting
> + functions running on same CPU core by rte_thread_get_affinity().
> +
<...>
> +#. Where to capture packets?
> + - Enable pdump in primary to allow secondary to access queue-pair for
> + ports. Thus packets are copied over in RX|TX callback by secondary
> + process using ring buffers.
> + - To capture packet in middle of pipeline stage, user specific hooks
> + or callback are to be used to copy the packets. These packets can
secodnary / secondary
> + be shared to secodnary process via user defined custom rings.
> +
> +Issue still persists?
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +#. Are there custom or vendor specific offload meta data?
> + - From PMD, then check for META data error and drops.
> + - From application, then check for META data error and drops.
> +#. Is multiprocess is used configuration and data processing?
> + - Check enabling or disabling features from secondary is supported or
> not?
Obejcts/ objects
> +#. Is there drops for certain scenario for packets or obejcts?
> + - Check user private data in objects by dumping the details for debug.
> +
<...>
Thanks,
Marko K
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v4 2/2] doc: add guide for debug and troubleshoot
2019-01-18 15:28 ` Kovacevic, Marko
@ 2019-01-21 3:38 ` Varghese, Vipin
0 siblings, 0 replies; 23+ messages in thread
From: Varghese, Vipin @ 2019-01-21 3:38 UTC (permalink / raw)
To: Kovacevic, Marko, dev, shreyansh.jain, thomas
Cc: Mcnamara, John, Patel, Amol, Padubidri, Sanjay A
Thanks Marko, I will spin v5 with the changes asap.
Note: Just wondering why 'devtools/checkpatches.sh' did not report any error.
Thanks
Vipin Varghese
> -----Original Message-----
> From: Kovacevic, Marko
> Sent: Friday, January 18, 2019 8:59 PM
> To: Varghese, Vipin <vipin.varghese@intel.com>; dev@dpdk.org;
> shreyansh.jain@nxp.com; thomas@monjalon.net
> Cc: Mcnamara, John <john.mcnamara@intel.com>; Patel, Amol
> <amol.patel@intel.com>; Padubidri, Sanjay A <sanjay.padubidri@intel.com>
> Subject: RE: [PATCH v4 2/2] doc: add guide for debug and troubleshoot
>
> After checking the patch again I found a few spelling mistakes
>
> > Add user guide on debug and troubleshoot for common issues and
> > bottleneck found in sample application model.
> >
> > Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
> > Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
> > ---
> > doc/guides/howto/debug_troubleshoot_guide.rst | 375
> > ++++++++++++++++++
> > doc/guides/howto/index.rst | 1 +
> > 2 files changed, 376 insertions(+)
> > create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
> >
>
> <...>
>
> receieve / receive
>
> > + - If stats for RX and drops updated on same queue? check
> > + receieve
> > thread
> > + - If packet does not reach PMD? check if offload for port and queue
> > + matches to traffic pattern send.
> > +
>
> <...>
>
> Offlaod/ offload
>
> > + - Is the packet multi segmented? Check if port and queue offlaod is set.
> > +
> > +Are there object drops in producer point for ring?
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> <...>
>
> sufficent / sufficient
>
> > + - Are drops on specific socket? If yes check if there are sufficent
> > + objects by rte_mempool_get_count() or rte_mempool_avail_count()
> > + - Is 'rte_mempool_get_count() or rte_mempool_avail_count()' zero?
> > + application requires more objects hence reconfigure number of
> > + elements in rte_mempool_create().
> > + - Is there single RX thread for multiple NIC? try having multiple
> > + lcore to read from fixed interface or we might be hitting cache
> > + limit, so increase cache_size for pool_create().
> > +
>
> Sceanrios/ scenarios
>
> > +#. Is performance low for some sceanrios?
> > + - Check if sufficient objects in mempool by rte_mempool_avail_count()
> > + - Is failure seen in some packets? we might be getting packets with
> > + 'size > mbuf data size'.
> > + - Is NIC offload or application handling multi segment mbuf? check the
> > + special packets are continuous with rte_pktmbuf_is_contiguous().
> > + - If there separate user threads used to access mempool objects, use
> > + rte_mempool_cache_create() for non DPDK threads.
>
> debuging / debugging
>
> > + - Is the error reproducible with 1GB hugepage? If no, then try debuging
> > + the issue with lookup table or objects with rte_mem_lock_page().
> > +
> > +.. note::
> > + Stall in release of MBUF can be because
>
> <...>
>
> softwre / software
>
> > + - If softwre crypto is in use, check if the CRYPTO Library is build with
> > + right (SIMD) flags or check if the queue pair using CPU ISA for
> > + feature_flags AVX|SSE|NEON using rte_cryptodev_info_get()
>
> Assited/ assisted
>
> > + - If its hardware assited crypto showing performance variance? Check if
> > + hardware is on same NUMA socket as queue pair and session pool.
> > +
>
> <...>
>
> exceeeding / exceeding
>
> > + core? registered functions may be exceeeding the desired time slots
> > + while running on same service core.
> > + - Is function is running on RTE core? check if there are conflicting
> > + functions running on same CPU core by rte_thread_get_affinity().
> > +
>
> <...>
>
> > +#. Where to capture packets?
> > + - Enable pdump in primary to allow secondary to access queue-pair for
> > + ports. Thus packets are copied over in RX|TX callback by secondary
> > + process using ring buffers.
> > + - To capture packet in middle of pipeline stage, user specific hooks
> > + or callback are to be used to copy the packets. These packets
> > +can
>
> secodnary / secondary
>
> > + be shared to secodnary process via user defined custom rings.
> > +
> > +Issue still persists?
> > +~~~~~~~~~~~~~~~~~~~~~
> > +
> > +#. Are there custom or vendor specific offload meta data?
> > + - From PMD, then check for META data error and drops.
> > + - From application, then check for META data error and drops.
> > +#. Is multiprocess is used configuration and data processing?
> > + - Check enabling or disabling features from secondary is
> > +supported or
> > not?
>
> Obejcts/ objects
>
> > +#. Is there drops for certain scenario for packets or obejcts?
> > + - Check user private data in objects by dumping the details for debug.
> > +
> <...>
>
> Thanks,
> Marko K
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v5 0/2] doc/howto: add debug and troubleshoot guide
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
2019-01-18 15:28 ` Kovacevic, Marko
@ 2019-01-21 10:41 ` Vipin Varghese
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 1/2] doc: add svg for " Vipin Varghese
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 " Vipin Varghese
1 sibling, 2 replies; 23+ messages in thread
From: Vipin Varghese @ 2019-01-21 10:41 UTC (permalink / raw)
To: dev, shreyansh.jain, thomas
Cc: john.mcnamara, marko.kovacevic, amol.patel, sanjay.padubidri,
Vipin Varghese
This patch series is an attempt to add how to guide for commonly seen
issues in various application scenario. It also tries to suggest a step
by step guide on how to debug and troubleshoot with available stats and
counters.
Motivation
==========
DPDK allows to build application on top of custom off the shelf
components for user scenarios that are modelled with single or multi stage.
Each module interacts with PMD or rings with either software or hardware
offloads.
Identify the cause of issue or performance issues will be tedious and time
consuming. The how to guide is an effort to consolidate such scenarios with
step by step focus to debug and troubleshoot.
Status
======
Reviews and changes accommodated. ACK received and updated.
Future
======
1. Suggestion on vlan filters - Shreyansh Jain
Change Log
==========
Vipin Varghese (2):
doc: add svg for debug and troubleshoot guide
doc: add guide for debug and troubleshoot
doc/guides/howto/debug_troubleshoot_guide.rst | 375 ++++++++++++++++++
doc/guides/howto/img/dtg_consumer_ring.svg | 19 +
doc/guides/howto/img/dtg_crypto.svg | 16 +
.../howto/img/dtg_distributor_worker.svg | 31 ++
doc/guides/howto/img/dtg_mempool.svg | 22 +
doc/guides/howto/img/dtg_pdump.svg | 28 ++
doc/guides/howto/img/dtg_producer_ring.svg | 19 +
doc/guides/howto/img/dtg_qos_tx.svg | 24 ++
doc/guides/howto/img/dtg_rx_rate.svg | 20 +
doc/guides/howto/img/dtg_rx_tx_drop.svg | 28 ++
doc/guides/howto/img/dtg_sample_app_model.svg | 105 +++++
doc/guides/howto/img/dtg_service.svg | 15 +
doc/guides/howto/index.rst | 1 +
13 files changed, 703 insertions(+)
create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
create mode 100644 doc/guides/howto/img/dtg_consumer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_crypto.svg
create mode 100644 doc/guides/howto/img/dtg_distributor_worker.svg
create mode 100644 doc/guides/howto/img/dtg_mempool.svg
create mode 100644 doc/guides/howto/img/dtg_pdump.svg
create mode 100644 doc/guides/howto/img/dtg_producer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_qos_tx.svg
create mode 100644 doc/guides/howto/img/dtg_rx_rate.svg
create mode 100644 doc/guides/howto/img/dtg_rx_tx_drop.svg
create mode 100644 doc/guides/howto/img/dtg_sample_app_model.svg
create mode 100644 doc/guides/howto/img/dtg_service.svg
V5:
- Corrections is spelling - Marko Kovacevic
V4:
- Added suggestion and corrections - Shreyansh Jain
V3:
- Reorder the commit order - Thomas Monjalon
V2:
- Added ACK - Marko Kovacevic
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v5 1/2] doc: add svg for debug and troubleshoot guide
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 0/2] doc/howto: add debug and troubleshoot guide Vipin Varghese
@ 2019-01-21 10:41 ` Vipin Varghese
2019-01-28 1:08 ` Thomas Monjalon
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 0/2] guide to debug and troubleshoot Vipin Varghese
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 " Vipin Varghese
1 sibling, 2 replies; 23+ messages in thread
From: Vipin Varghese @ 2019-01-21 10:41 UTC (permalink / raw)
To: dev, shreyansh.jain, thomas
Cc: john.mcnamara, marko.kovacevic, amol.patel, sanjay.padubidri,
Vipin Varghese
Add svg for debug and troubleshoot guide for single or multi stage
applications.
Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
---
doc/guides/howto/img/dtg_consumer_ring.svg | 19 ++++
doc/guides/howto/img/dtg_crypto.svg | 16 +++
.../howto/img/dtg_distributor_worker.svg | 31 ++++++
doc/guides/howto/img/dtg_mempool.svg | 22 ++++
doc/guides/howto/img/dtg_pdump.svg | 28 +++++
doc/guides/howto/img/dtg_producer_ring.svg | 19 ++++
doc/guides/howto/img/dtg_qos_tx.svg | 24 ++++
doc/guides/howto/img/dtg_rx_rate.svg | 20 ++++
doc/guides/howto/img/dtg_rx_tx_drop.svg | 28 +++++
doc/guides/howto/img/dtg_sample_app_model.svg | 105 ++++++++++++++++++
doc/guides/howto/img/dtg_service.svg | 15 +++
11 files changed, 327 insertions(+)
create mode 100644 doc/guides/howto/img/dtg_consumer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_crypto.svg
create mode 100644 doc/guides/howto/img/dtg_distributor_worker.svg
create mode 100644 doc/guides/howto/img/dtg_mempool.svg
create mode 100644 doc/guides/howto/img/dtg_pdump.svg
create mode 100644 doc/guides/howto/img/dtg_producer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_qos_tx.svg
create mode 100644 doc/guides/howto/img/dtg_rx_rate.svg
create mode 100644 doc/guides/howto/img/dtg_rx_tx_drop.svg
create mode 100644 doc/guides/howto/img/dtg_sample_app_model.svg
create mode 100644 doc/guides/howto/img/dtg_service.svg
diff --git a/doc/guides/howto/img/dtg_consumer_ring.svg b/doc/guides/howto/img/dtg_consumer_ring.svg
new file mode 100644
index 000000000..d0e0b30f8
--- /dev/null
+++ b/doc/guides/howto/img/dtg_consumer_ring.svg
@@ -0,0 +1,19 @@
+<svg width="634" height="97" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>consumer ring</title>
+ <rect fill="#fff" id="canvas_background" height="99" width="636" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="95" width="135" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_2" height="95" width="135" y="2" x="252" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_3" height="95" width="135" y="2" x="499" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="45" x="10" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="52" x="268" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_6" y="53" x="517" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 3</text>
+ <path transform="rotate(90.19324493408203 177.864990234375,48.89100646972658) " id="svg_7" d="m155.364991,48.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path transform="rotate(90.19324493408203 434.864990234375,48.89100646972656) " id="svg_8" d="m412.364991,48.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path id="svg_10" d="m225.937481,47.95781l0,0c0,-10.493411 8.282734,-19 18.500005,-19l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564971c3.469424,3.563191 5.418523,8.395917 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999998 -18.5,18.999998l0,0c-10.217271,0 -18.500005,-8.506591 -18.500005,-18.999998zm18.500005,-19l0,37.999998m-18.500005,-18.999998l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ <path id="svg_11" d="m475.937481,46.95781l0,0c0,-10.493411 8.282734,-19.000001 18.500005,-19.000001l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564972c3.469424,3.563191 5.418523,8.395916 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999997 -18.5,18.999997l0,0c-10.217271,0 -18.500005,-8.50659 -18.500005,-18.999997zm18.500005,-19.000001l0,37.999998m-18.500005,-18.999997l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_crypto.svg b/doc/guides/howto/img/dtg_crypto.svg
new file mode 100644
index 000000000..7687aaa4a
--- /dev/null
+++ b/doc/guides/howto/img/dtg_crypto.svg
@@ -0,0 +1,16 @@
+<svg width="180" height="230" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>crypto</title>
+ <rect fill="#fff" id="canvas_background" height="232" width="182" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="1" width="0" y="369.4375" x="838.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="222.4375" x="39.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core 7</text>
+ <rect stroke="#000" id="svg_26" height="44.000003" width="155.000001" y="151.437497" x="6.499997" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_27" y="179.4375" x="13.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">CRYPTO PMD</text>
+ <ellipse stroke="#000" ry="41.5" rx="66.500001" id="svg_28" cy="48.9375" cx="87.999997" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_29" y="57.4375" x="45.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Device</text>
+ <path stroke="#000" transform="rotate(-90.19718170166016 85.76751708984375,119.45785522460936) " id="svg_30" d="m40.743289,119.457895l26.106598,-13.915756l0,6.95786l37.835289,0l0,-6.95786l26.106586,13.915756l-26.106586,13.915682l0,-6.957842l-37.835289,0l0,6.957842l-26.106598,-13.915682z" stroke-opacity="null" stroke-width="0" fill="#845335"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_distributor_worker.svg b/doc/guides/howto/img/dtg_distributor_worker.svg
new file mode 100644
index 000000000..3df758fa6
--- /dev/null
+++ b/doc/guides/howto/img/dtg_distributor_worker.svg
@@ -0,0 +1,31 @@
+<svg width="600" height="235" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>dsitributor and worker</title>
+ <rect fill="#fff" id="canvas_background" height="237" width="602" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_1" height="89.999991" width="202.000004" y="21.000003" x="4" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start"
+ font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="47" x="7"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">PKT classify</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_12" y="96.5" x="25.992188" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Distribute</text>
+ <path id="svg_13" d="m334.224445,98.046256l16.252783,-5.432074l-16.252783,-5.429993l-16.256262,-5.427952l0,6.038517c-4.264321,-0.004069 -11.331234,-0.004069 -13.334561,-0.004069c-5.919233,-0.081711 -9.804243,-1.153783 -13.847174,-3.073397c-6.02597,-2.865095 -11.16889,-8.021466 -17.371313,-12.989922c-3.154554,-2.470933 -6.669049,-4.921502 -11.349751,-6.892119c-4.633926,-1.978838 -10.68815,-3.351144 -17.41807,-3.324591l-29.170102,0l0,9.646983l29.170102,0c4.056131,0.042884 6.623169,0.808657 10.124445,2.70786c5.179989,2.824261 10.230311,8.223602 17.195784,13.339136c6.826056,5.066481 17.329865,10.182006 32.564612,10.231017l13.436029,0l0,6.0385l16.256262,-5.427896zm-68.759273,-36.535597c0.184389,0.075572 0.379369,0.14089 0.559297,0.218526c5.559349,2.340242 9.573988,5.211493 12.751502,7.709001c0.730423,0.584013 1.377934,1.145611 2.071302,1.719456l37.120911,0l0,6.040581l16.256262,-5.42798l16.252783,-5.429962l-16.252783,-5.432037l-16.256262,-5.434093l0,6.036509l-52.503011,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFAC8D"/>
+ <rect stroke="#000" id="svg_17" height="110.999999" width="180" y="3.4375" x="353.5"
+ stroke-opacity="null" stroke-width="0" fill="#FABA6F"/>
+ <rect stroke="#000" id="svg_14" height="110.999999" width="180" y="31.4375" x="369.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_15" height="110.999999" width="180" y="58.4375" x="392.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFF093"/>
+ <rect stroke="#000" id="svg_16" height="110.999999" width="180" y="90.4375" x="413.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,142.60348030924797,64.2037038393319) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="71.490132" x="413.769676" stroke-opacity="null" stroke-width="0" fill="#000000">worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="101.332237" x="406.404514" stroke-opacity="null" stroke-width="0" fill="#000000">worker 2</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_20" y="58.700658" x="376.943866" stroke-opacity="null" stroke-width="0" fill="#000000">worker 3</text>
+ <text style="cursor: move;" stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_22" y="18.911184" x="350.429283" stroke-opacity="null" stroke-width="0" fill="#000000">worker 4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="139.4375" x="49.5"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_24" y="226.4375" x="404.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4,5</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_mempool.svg b/doc/guides/howto/img/dtg_mempool.svg
new file mode 100644
index 000000000..f3d31a96e
--- /dev/null
+++ b/doc/guides/howto/img/dtg_mempool.svg
@@ -0,0 +1,22 @@
+<svg width="955" height="185" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>mempool</title>
+ <rect fill="#fff" id="canvas_background" height="187" width="957" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="1" width="0" y="370.4375" x="839.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <rect stroke="#000" id="svg_31" height="181.000002" width="951.000029" y="2.4375" x="3.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_32" y="167.4375" x="14.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">MBUF pool</text>
+ <rect id="svg_38" height="71" width="859" y="15.4375" x="18.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_39" height="71" width="859" y="32.4375" x="33.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_40" height="71" width="859" y="47.4375" x="52.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_41" height="71" width="859" y="62.4375" x="71.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect stroke="#000" id="svg_42" height="55.999999" width="403.000005" y="69.4375" x="486.499995" stroke-opacity="null" stroke-width="0" fill="#FFAC8D"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_43" y="103.4375" x="631.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Payload</text>
+ <rect stroke="#000" id="svg_44" height="56" width="140.999999" y="68.4375" x="292.500001" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_45" y="105.4375" x="303.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Metadata</text>
+ <rect stroke="#000" id="svg_46" height="53" width="185.999997" y="70.4375" x="90.5" stroke-opacity="null" stroke-width="0" fill="#EDFFFF"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_47" y="104.4375" x="105.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">struct mbuf</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_pdump.svg b/doc/guides/howto/img/dtg_pdump.svg
new file mode 100644
index 000000000..1dc63bed7
--- /dev/null
+++ b/doc/guides/howto/img/dtg_pdump.svg
@@ -0,0 +1,28 @@
+<svg width="860" height="140" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>pdump</title>
+ <rect fill="#fff" id="canvas_background" height="142" width="862" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_15" height="127.999992" width="568" y="7" x="8.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#B1FFFF"/>
+ <rect id="svg_2" height="51" width="100" y="23" x="22.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_3" height="51" width="100" y="26" x="455.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CC7C65"/>
+ <path id="svg_7" d="m208.364769,78.968848l12.704559,-5.623238l12.697917,-5.625551l-12.704644,-5.629525l-12.697836,-5.625202l0,6.255891c-3.112384,0 -6.729051,0 -7.940704,0c-5.776899,0.124913 -12.907107,-3.365352 -20.492742,-7.864703c-2.827801,-1.577599 -5.680695,-3.313223 -8.704363,-4.88042c3.023667,-1.569118 5.872475,-3.302747 8.704363,-4.880295c7.585635,-4.493132 14.715839,-7.98967 20.485292,-7.862546l7.94815,-0.002103l0,6.264275l12.697836,-5.627447l12.704644,-5.631775l-12.704644,-5.637807l-12.697836,-5.629547l0,6.260273l-7.944067,0c-11.799214,0.12679 -20.61742,5.506774 -28.333828,9.879239c-7.660201,4.613909 -14.545738,8.087381 -18.539115,7.862617l-22.780531,0l0,9.993806l22.788007,0c3.989952,-0.226969 10.878944,3.248761 18.53841,7.86254c7.71303,4.374678 16.52785,9.750196 28.330427,9.879312l7.940704,0l0,6.262203l0,0l0,0.000003z" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#559E84"/>
+ <path id="svg_8" d="m439.457308,43.77545l-10.478696,-7.306254l0,8.12971l-18.805519,0c-3.29272,0.296972 -8.977707,-4.212008 -15.298608,-10.195089c-6.365074,-5.675375 -13.639418,-12.638687 -23.37938,-12.824984l-6.552957,0l0,12.959904l6.552957,0c4.770151,-0.167389 10.652051,4.360514 16.914219,10.203329c2.334186,2.041106 4.688512,4.292919 7.183208,6.326016c-2.494696,2.035766 -4.846223,4.290275 -7.183208,6.331424c-6.262164,5.826551 -12.144067,10.365242 -16.908056,10.200515l-6.559117,0l0,12.978808l6.555749,0c9.719269,-0.1646 16.996426,-7.14678 23.382185,-12.814082c6.32147,-5.983193 12.003677,-10.494803 15.299177,-10.203221l18.799357,0l0,8.129634l10.478696,-7.298036l10.484823,-7.303408l-10.484823,-7.314263l-0.000006,-0.000004z" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#559E84"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="54" x="53.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_10" y="57" x="486.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_11" y="40" x="243.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_12" y="77" x="243.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="82" x="326.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="40" x="327.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q3</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="114.86203" x="234.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Primary</text>
+ <rect stroke="#000" id="svg_18" height="90.000001" width="201.999994" y="29.437499" x="654.5" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="102.4375" x="684.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Secondary</text>
+ <rect stroke="#000" id="svg_20" height="42.000001" width="184.000001" y="34.437499" x="664.499994" stroke-opacity="null" stroke-width="0" fill="#A0D58A"/>
+ <rect id="svg_21" height="1" width="0" y="366.4375" x="835.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_22" y="59.4375" x="668.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Ring BufferQ</text>
+ <path stroke="#000" transform="rotate(89.78996276855469 615.4766845703124,74.49395751953122) " id="svg_24" d="m599.023803,74.414257l17.255478,-33.667333l15.650356,33.667333l-8.226459,0l0,33.826726l-16.452915,0l0,-33.826726l-8.226459,0z" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="22.4375" x="714.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core 0</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_producer_ring.svg b/doc/guides/howto/img/dtg_producer_ring.svg
new file mode 100644
index 000000000..5657702e5
--- /dev/null
+++ b/doc/guides/howto/img/dtg_producer_ring.svg
@@ -0,0 +1,19 @@
+<svg width="634" height="97" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>producer ring</title>
+ <rect fill="#fff" id="canvas_background" height="99" width="636" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="95" width="135" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_2" height="95" width="135" y="2" x="252" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_3" height="95" width="135" y="2" x="499" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="45" x="10" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="52" x="268" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_6" y="53" x="517" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 3</text>
+ <path transform="rotate(90.19324493408203 197.864990234375,42.89100646972657) " id="svg_7" d="m175.364991,42.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path transform="rotate(90.19324493408203 445.864990234375,47.89100646972657) " id="svg_8" d="m423.364991,47.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path id="svg_10" d="m120.937481,42.95781l0,0c0,-10.493411 8.282734,-19 18.500005,-19l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564971c3.469424,3.563191 5.418523,8.395917 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999998 -18.5,18.999998l0,0c-10.217271,0 -18.500005,-8.506591 -18.500005,-18.999998zm18.500005,-19l0,37.999998m-18.500005,-18.999998l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ <path id="svg_11" d="m373.937481,45.95781l0,0c0,-10.493411 8.282734,-19.000001 18.500005,-19.000001l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564972c3.469424,3.563191 5.418523,8.395916 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999997 -18.5,18.999997l0,0c-10.217271,0 -18.500005,-8.50659 -18.500005,-18.999997zm18.500005,-19.000001l0,37.999998m-18.500005,-18.999997l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_qos_tx.svg b/doc/guides/howto/img/dtg_qos_tx.svg
new file mode 100644
index 000000000..6e46d188a
--- /dev/null
+++ b/doc/guides/howto/img/dtg_qos_tx.svg
@@ -0,0 +1,24 @@
+<svg width="287" height="180" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>qos</title>
+ <rect fill="#fff" id="canvas_background" height="182" width="289" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="180" width="287" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CBFFFF"/>
+ <rect id="svg_10" height="82" width="78" y="17.808165" x="152.25" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_11" height="82" width="78" y="36.808165" x="171.25" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_12" height="82" width="78" y="59.808165" x="188.25" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="113.808165" x="211.25" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="161.808165" x="161.25" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core1</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_15" y="235.775764" x="531.631651" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.3615568424500273,0,0,0.9358925757041695,106.13283890389599,-107.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="173.881425" x="192.8023" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_17" y="66.670315" x="-70.838675" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ <rect id="svg_18" height="103" width="104" y="26" x="26" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FABA6F"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="79" x="50" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">QoS</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_20" y="153" x="27" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 10</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_rx_rate.svg b/doc/guides/howto/img/dtg_rx_rate.svg
new file mode 100644
index 000000000..311c9c43f
--- /dev/null
+++ b/doc/guides/howto/img/dtg_rx_rate.svg
@@ -0,0 +1,20 @@
+<svg width="116" height="152" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>rx rate</title>
+ <rect fill="#fff" id="canvas_background" height="154" width="118" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="82" width="78" y="0.812613" x="0.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_2" height="82" width="78" y="19.812613" x="19.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_3" height="82" width="78" y="42.812613" x="36.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="96.812613" x="59.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="144.812613" x="9.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core0</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_7" y="214.621694" x="210.788594" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.36155684245002745,0,0,0.9358925757041695,-60.867161096103985,-178.0100101290174) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_8" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="42.29845" x="-417.420836" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_rx_tx_drop.svg b/doc/guides/howto/img/dtg_rx_tx_drop.svg
new file mode 100644
index 000000000..2f4e9d587
--- /dev/null
+++ b/doc/guides/howto/img/dtg_rx_tx_drop.svg
@@ -0,0 +1,28 @@
+<svg width="275" height="154" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>rx-tx drops</title>
+ <rect fill="#fff" id="canvas_background" height="156" width="277" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="82" width="78" y="0.808165" x="0.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_2" height="82" width="78" y="19.808165" x="19.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_3" height="82" width="78" y="42.808165" x="36.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="96.808165" x="59.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="144.808165" x="9.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core0</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_7" y="214.616159" x="210.788594" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.3615568424500274,0,0,0.9358925757041695,-60.867161096103985,-178.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_8" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="42.292073" x="-417.420836" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ <rect id="svg_10" height="82" width="78" y="2.808165" x="159.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_11" height="82" width="78" y="21.808165" x="178.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_12" height="82" width="78" y="44.808165" x="195.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="98.808165" x="218.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="146.808165" x="168.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core1</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_15" y="217.105524" x="547.514971" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.36155684245002734,0,0,0.9358925757041695,98.13283890389599,-176.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_17" y="45.160101" x="-53.681142" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_sample_app_model.svg b/doc/guides/howto/img/dtg_sample_app_model.svg
new file mode 100644
index 000000000..ad3b0995d
--- /dev/null
+++ b/doc/guides/howto/img/dtg_sample_app_model.svg
@@ -0,0 +1,105 @@
+<svg width="1418" height="379" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>sample application model</title>
+ <rect fill="#fff" id="canvas_background" height="381" width="1420" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_84" height="378.999996" width="1417.999937" y="0" x="0" stroke-opacity="null" stroke-width="0" fill="#B1FFFF"/>
+ <rect stroke="#000" id="svg_1" height="79.999993" width="187.000011" y="267" x="635.499996" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <rect stroke="#000" id="svg_14" height="28.000001" width="175.000003" y="271.437495" x="641" stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_16" height="36.000002" width="142" y="304.437495" x="659" stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(0.7497134942573729,0,0,0.7607963681117937,149.70768863149087,72.0554119542491) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="335.740664" x="684.214296"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Health Check</text>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="368.437495" x="685"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 7</text>
+ <text stroke="#000" transform="matrix(0.7446371555386645,0,0,0.7004599746900311,157.05848471617847,107.2548065316271) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="261.052167" x="658.840014"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Stats Collector</text>
+ <rect id="svg_26" height="91" width="88" y="89.4375" x="2"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFA9A2"/>
+ <rect id="svg_27" height="91" width="88" y="107.4375" x="24"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CE7975"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_28" y="166.4375" x="52"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6666666865348816,14.972222477197647,39.14583100005984) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_30" y="96.937501" x="-11.409091"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 1</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6666666865348816,14.972222477197647,39.14583100005984) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_33" y="128.4375" x="27.863636"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_34" y="216.4375" x="26"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core0</text>
+ <rect id="svg_35" height="91" width="88" y="55.4375" x="1303"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFA9A2"/>
+ <rect id="svg_36" height="91" width="88" y="73.4375" x="1325"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CE7975"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_37" y="132.4375" x="1352"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_40" y="184.4375" x="1327"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core1</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6296296119689941,200.86111453175545,24.236112266778946) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_43" y="74.319853" x="1811.681832"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 1</text>
+ <text stroke="#000" transform="matrix(0.6958672408102909,0,0,0.6184720487972513,537.7539486343405,-28.040291137315034) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_44" y="191.620936" x="1140.58332"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 2</text>
+ <rect stroke="#000" id="svg_45" height="62" width="83.000003" y="101.4375" x="1096"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_46" y="135.4375" x="1115"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">QoS</text>
+ <rect stroke="#000" id="svg_48" height="61" width="107" y="110.4375" x="649"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_49" y="145.4375" x="661"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">Crypto</text>
+ <rect stroke="#000" id="svg_50" height="70.999998" width="84.000002" y="65.4375" x="422"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <rect stroke="#000" id="svg_51" height="74.000003" width="90.000002" y="92.4375" x="444"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_52" height="71.999998" width="92" y="120.4375" x="468"
+ stroke-opacity="null" stroke-width="0" fill="#C48157"/>
+ <text stroke="#000" transform="matrix(0.6528342962265015,0,0,0.5925925970077515,63.70490664243698,168.43749817460775) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_53" y="-145.374996" x="553.425418"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.6666666865348816,166.69894686341286,-165.52084343507886) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_54" y="417.437503" x="420.257696" stroke-opacity="null" stroke-width="0" fill="#000000">Worker 2</text>
+ <text stroke="#000" transform="matrix(0.7483048439025879,0,0,0.7407407164573669,76.38947987556458,153.33566251024604) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_55" y="-19.012521" x="530.011964"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 3</text>
+ <rect stroke="#000" id="svg_62" height="70.999998" width="84.000002" y="67.4375" x="865"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <rect stroke="#000" id="svg_63" height="74.000003" width="90.000002" y="94.4375" x="887"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_64" height="71.999998" width="92" y="122.4375" x="911"
+ stroke-opacity="null" stroke-width="0" fill="#C48157"/>
+ <text stroke="#000" transform="matrix(0.6528342962265015,0,0,0.5925925970077515,63.70490664243698,168.43749817460775) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_65" y="-143.687496" x="1238.132093"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.6666666865348816,166.69894686341286,-165.52084343507886) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_66" y="420.437503" x="1072.811052"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 2</text>
+ <text stroke="#000" transform="matrix(0.7483048439025879,0,0,0.7407407164573669,76.38947987556458,153.33566251024604) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_67" y="-16.312521" x="1122.016685"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 3</text>
+ <path stroke="#000" id="svg_68" d="m653.747516,12.862181l19.36745,-10.781877l77.632553,0l0,53.999988l-97.000004,0l0,-43.218111z"
+ stroke-opacity="null" stroke-width="0" fill="#805064"/>
+ <text stroke="#000" transform="matrix(0.733815550804138,0,0,0.9629629850387573,306.63544338941574,-1.3912058547139168) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_69" y="33.322117" x="496.534253"
+ stroke-opacity="null" stroke-width="0" fill="#ffffff">Device</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_70" y="217.4375" x="457"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_71" y="197.4375" x="653"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 5</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_72" y="186.4375" x="1097"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 6</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_73" y="220.4375" x="885"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4</text>
+ <rect stroke="#000" id="svg_74" height="88.000002" width="110.999999" y="104" x="206.000001"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text stroke="#000" transform="matrix(0.5661651903991256,0,0,1,107.56695064249905,0) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_75" y="133" x="182.901935"
+ stroke-opacity="null" stroke-width="0" fill="#000000">PKT classify</text>
+ <text stroke="#000" transform="matrix(0.5444634556770325,0,0,1,115.93405053019524,0) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_76" y="170" x="196.644877"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Distribute</text>
+ <path id="svg_77" d="m189.755816,142.38346l-9.615746,-5.572568l0,6.200627l-17.256831,0c-3.021555,0.226504 -8.238368,-3.212549 -14.038725,-7.775916c-5.840892,-4.328677 -12.516173,-9.639678 -21.454022,-9.781769l-6.013302,0l0,9.884674l6.013302,0c4.377316,-0.12767 9.774825,3.325816 15.521285,7.782201c2.14196,1.556776 4.3024,3.274261 6.59165,4.824928c-2.28925,1.552703 -4.447123,3.272244 -6.59165,4.829053c-5.746457,4.44398 -11.143969,7.905694 -15.51563,7.780055l-6.018955,0l0,9.899092l6.015865,0c8.91886,-0.125542 15.596722,-5.450935 21.456596,-9.773453c5.800879,-4.563453 11.01514,-8.004512 14.039247,-7.782119l17.251176,0l0,6.200569l9.615746,-5.566299l9.621368,-5.570396l-9.621368,-5.578676l-0.000005,-0.000003z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path id="svg_78" d="m417.79301,181.072084l13.604178,-5.205736l-13.604178,-5.203742l-13.60709,-5.201786l0,5.786911c-3.569394,-0.003899 -9.48466,-0.003899 -11.161519,-0.003899c-4.954616,-0.078306 -8.206513,-1.105708 -11.590595,-2.945338c-5.043959,-2.745716 -9.348773,-7.687237 -14.54043,-12.448672c-2.640478,-2.367977 -5.58224,-4.716439 -9.50016,-6.604946c-3.878767,-1.896386 -8.946375,-3.211512 -14.579566,-3.186066l-24.41645,0l0,9.245023l24.41645,0c3.395131,0.041097 5.543836,0.774963 8.474533,2.595032c4.335842,2.706583 8.563147,7.88095 14.393505,12.783335c5.71366,4.855376 14.505736,9.757754 27.25778,9.804723l11.246452,0l0,5.786895l13.60709,-5.201732zm-57.554046,-35.013272c0.15434,0.072423 0.317546,0.135019 0.468152,0.209421c4.65338,2.242732 8.013781,4.994346 10.673477,7.387791c0.611391,0.559679 1.153382,1.097877 1.733756,1.647811l31.071571,0l0,5.788888l13.60709,-5.201813l13.604178,-5.203713l-13.604178,-5.205701l-13.60709,-5.207671l0,5.784986l-43.946955,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path stroke="#000" transform="rotate(89.46277618408203 605.7476806640625,144.42587280273435) " id="svg_79" d="m590.864808,144.339395l14.882874,-35.951591l14.882874,35.951591l-7.441438,0l0,36.124583l-14.882873,0l0,-36.124583l-7.441438,0z" stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ <path stroke="#000" transform="rotate(89.74066925048828 697.6382446289062,81.67762756347659) " id="svg_80" d="m661.638136,81.677665l20.874103,-13.000079l0,6.500022l30.252035,0l0,-6.500022l20.874093,13.000079l-20.874093,13.000009l0,-6.500005l-30.252035,0l0,6.500005l-20.874103,-13.000009z" fill-opacity="null"
+ stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ <path id="svg_81" d="m858.79301,172.072084l13.604178,-5.205736l-13.604178,-5.203742l-13.60709,-5.201786l0,5.786911c-3.569394,-0.0039 -9.48466,-0.0039 -11.161519,-0.0039c-4.954616,-0.078305 -8.206513,-1.105708 -11.590595,-2.945338c-5.043959,-2.745715 -9.348773,-7.687236 -14.54043,-12.448672c-2.640477,-2.367977 -5.58224,-4.716438 -9.50016,-6.604945c-3.878767,-1.896387 -8.946375,-3.211513 -14.579566,-3.186067l-24.416451,0l0,9.245023l24.416451,0c3.395131,0.041097 5.543836,0.774963 8.474533,2.595032c4.335842,2.706583 8.563147,7.88095 14.393505,12.783335c5.713661,4.855376 14.505736,9.757754 27.257781,9.804723l11.246451,0l0,5.786894l13.60709,-5.201732zm-57.554045,-35.013272c0.15434,0.072424 0.317546,0.135019 0.468152,0.209421c4.65338,2.242731 8.013781,4.994346 10.673477,7.38779c0.61139,0.559679 1.153381,1.097878 1.733756,1.647812l31.07157,0l0,5.788888l13.60709,-5.201812l13.604178,-5.203713l-13.604178,-5.205701l-13.60709,-5.207671l0,5.784986l-43.946955,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path id="svg_82" d="m1080.755816,127.38346l-9.615746,-5.572567l0,6.200627l-17.256831,0c-3.021555,0.226504 -8.238368,-3.212549 -14.038725,-7.775916c-5.840892,-4.328677 -12.516173,-9.639678 -21.454022,-9.78177l-6.013302,0l0,9.884674l6.013302,0c4.377316,-0.12767 9.774825,3.325816 15.521285,7.782201c2.141959,1.556776 4.3024,3.274261 6.59165,4.824928c-2.28925,1.552704 -4.447123,3.272244 -6.59165,4.829053c-5.746457,4.443981 -11.143969,7.905694 -15.51563,7.780055l-6.018955,0l0,9.899093l6.015865,0c8.91886,-0.125543 15.596723,-5.450935 21.456596,-9.773454c5.800879,-4.563453 11.01514,-8.004512 14.039246,-7.782119l17.251177,0l0,6.20057l9.615746,-5.566299l9.621368,-5.570397l-9.621368,-5.578676l-0.000006,-0.000003z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path stroke="#000" id="svg_83" d="m1269.963506,163.318859l13.567973,-4.653717l13.56088,-4.655631l-13.568064,-4.658919l-13.560793,-4.655342l0,5.177292c-3.323905,0 -7.186364,0 -8.480362,0c-6.169502,0.103377 -13.784287,-2.785121 -21.885448,-6.508723c-3.019981,-1.3056 -6.066761,-2.741979 -9.295919,-4.03897c3.229159,-1.298581 6.271574,-2.733309 9.295919,-4.038867c8.101162,-3.718456 15.715942,-6.612144 21.877492,-6.506938l8.488314,-0.001741l0,5.18423l13.560793,-4.6572l13.568064,-4.660782l-13.568064,-4.665774l-13.560793,-4.658938l0,5.180918l-8.483954,0c-12.6011,0.104929 -22.018599,4.557333 -30.259422,8.175926c-8.180795,3.81841 -15.53428,6.693008 -19.799052,6.506997l-24.328718,0l0,8.27074l24.336702,0c4.261113,-0.187837 11.618288,2.688631 19.798299,6.506933c8.237215,3.620425 17.651098,8.069132 30.25579,8.175987l8.480362,0l0,5.182516l0,0l0,0.000002z" fill-opacity="null"
+ stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_service.svg b/doc/guides/howto/img/dtg_service.svg
new file mode 100644
index 000000000..2d876a3a1
--- /dev/null
+++ b/doc/guides/howto/img/dtg_service.svg
@@ -0,0 +1,15 @@
+<svg width="254" height="145" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>service</title>
+ <rect fill="#fff" id="canvas_background" height="147" width="256" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_1" height="109.999987" width="254.000014" y="0" x="0" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <rect stroke="#000" id="svg_14" height="37.000001" width="225" y="7.437494" x="4.5" stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_16" height="45.000002" width="186.000003" y="58.437493" x="33.5" stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(1.0013854504218995,0,0,1.0926463039877063,73.76856116958965,-122.08861649089796) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="193.234194" x="-35.792216" stroke-opacity="null" stroke-width="0" fill="#000000">Health Check</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="138.437495" x="45.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 6</text>
+ <text stroke="#000" transform="matrix(1.0013854504218995,0,0,1.0926463039877063,73.76856116958965,-122.08861649089796) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="141.982476" x="-63.753477" stroke-opacity="null" stroke-width="0" fill="#000000">Stats Collector</text>
+ </g>
+</svg>
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v5 2/2] doc: add guide for debug and troubleshoot
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 0/2] doc/howto: add debug and troubleshoot guide Vipin Varghese
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 1/2] doc: add svg for " Vipin Varghese
@ 2019-01-21 10:41 ` Vipin Varghese
2019-01-28 1:30 ` Thomas Monjalon
1 sibling, 1 reply; 23+ messages in thread
From: Vipin Varghese @ 2019-01-21 10:41 UTC (permalink / raw)
To: dev, shreyansh.jain, thomas
Cc: john.mcnamara, marko.kovacevic, amol.patel, sanjay.padubidri,
Vipin Varghese
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 15658 bytes --]
Add user guide on debug and troubleshoot for common issues and bottleneck
found in sample application model.
Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
---
doc/guides/howto/debug_troubleshoot_guide.rst | 375 ++++++++++++++++++
doc/guides/howto/index.rst | 1 +
2 files changed, 376 insertions(+)
create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
diff --git a/doc/guides/howto/debug_troubleshoot_guide.rst b/doc/guides/howto/debug_troubleshoot_guide.rst
new file mode 100644
index 000000000..868dc6e58
--- /dev/null
+++ b/doc/guides/howto/debug_troubleshoot_guide.rst
@@ -0,0 +1,375 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2018 Intel Corporation.
+
+.. _debug_troubleshoot_via_pmd:
+
+Debug & Troubleshoot guide via PMD
+==================================
+
+DPDK applications can be designed to run as single thread simple stage to
+multiple threads with complex pipeline stages. These application can use poll
+mode devices which helps in offloading CPU cycles. A few models are
+
+ * single primary
+ * multiple primary
+ * single primary single secondary
+ * single primary multiple secondary
+
+In all the above cases, it is a tedious task to isolate, debug and understand
+odd behaviour which occurs randomly or periodically. The goal of guide is to
+share and explore a few commonly seen patterns and behaviour. Then, isolate
+and identify the root cause via step by step debug at various processing
+stages.
+
+Application Overview
+--------------------
+
+Let us take up an example application as reference for explaining issues and
+patterns commonly seen. The sample application in discussion makes use of
+single primary model with various pipeline stages. The application uses PMD
+and libraries such as service cores, mempool, pkt mbuf, event, crypto, QoS
+and eth.
+
+The overview of an application modeled using PMD is shown in
+:numref:`dtg_sample_app_model`.
+
+.. _dtg_sample_app_model:
+
+.. figure:: img/dtg_sample_app_model.*
+
+ Overview of pipeline stage of an application
+
+Bottleneck Analysis
+-------------------
+
+To debug the bottleneck and performance issues the desired application
+is made to run in an environment matching as below
+
+#. Linux 64-bit|32-bit
+#. DPDK PMD and libraries are used
+#. Libraries and PMD are either static or shared. But not both
+#. Machine flag optimizations of gcc or compiler are made constant
+
+Is there mismatch in packet rate (received < send)?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RX Port and associated core :numref:`dtg_rx_rate`.
+
+.. _dtg_rx_rate:
+
+.. figure:: img/dtg_rx_rate.*
+
+ RX send rate compared against Received rate
+
+#. Are generic configuration correct?
+ - What is port Speed, Duplex? rte_eth_link_get()
+ - Are packets of higher sizes are dropped? rte_eth_get_mtu()
+ - Are only specific MAC received? rte_eth_promiscuous_get()
+
+#. Are there NIC specific drops?
+ - Check rte_eth_rx_queue_info_get() for nb_desc and scattered_rx
+ - Is RSS enabled? rte_eth_dev_rss_hash_conf_get()
+ - Are packets spread on all queues? rte_eth_dev_stats()
+ - If stats for RX and drops updated on same queue? check receive thread
+ - If packet does not reach PMD? check if offload for port and queue
+ matches to traffic pattern send.
+
+#. If problem still persists, this might be at RX lcore thread
+ - Check if RX thread, distributor or event rx adapter? these may be
+ processing less than required
+ - Is the application is build using processing pipeline with RX stage? If
+ there are multiple port-pair tied to a single RX core, try to debug by
+ using rte_prefetch_non_temporal(). This will intimate the mbuf in cache
+ is temporary.
+
+Are there packet drops (receive|transmit)?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RX-TX Port and associated cores :numref:`dtg_rx_tx_drop`.
+
+.. _dtg_rx_tx_drop:
+
+.. figure:: img/dtg_rx_tx_drop.*
+
+ RX-TX drops
+
+#. At RX
+ - Get RX queue count? nb_rx_queues using rte_eth_dev_info_get()
+ - Are there miss, errors, qerros? rte_eth_dev_stats() for imissed,
+ ierrors, q_erros, rx_nombuf, rte_mbuf_ref_count
+
+#. At TX
+ - Are you doing in bulk TX? check application for TX descriptor overhead.
+ - Are there TX errors? rte_eth_dev_stats() for oerrors and qerros
+ - Is specific scenarios not releasing mbuf? check rte_mbuf_ref_count of
+ those packets.
+ - Is the packet multi segmented? Check if port and queue offload is set.
+
+Are there object drops in producer point for ring?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Producer point for ring :numref:`dtg_producer_ring`.
+
+.. _dtg_producer_ring:
+
+.. figure:: img/dtg_producer_ring.*
+
+ Producer point for Rings
+
+#. Performance for Producer
+ - Fetch the type of RING 'rte_ring_dump()' for flags (RING_F_SP_ENQ)
+ - If '(burst enqueue - actual enqueue) > 0' check rte_ring_count() or
+ rte_ring_free_count()
+ - If 'burst or single enqueue returning always 0'? is rte_ring_full()
+ true then next stage is not pulling the content at desired rate.
+
+Are there object drops in consumer point for ring?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Consumer point for ring :numref:`dtg_consumer_ring`.
+
+.. _dtg_consumer_ring:
+
+.. figure:: img/dtg_consumer_ring.*
+
+ Consumer point for Rings
+
+#. Performance for Consumer
+ - Fetch the type of RING – rte_ring_dump() for flags (RING_F_SC_DEQ)
+ - If '(burst dequeue - actual dequeue) > 0' for rte_ring_free_count()
+ - If 'burst or single enqueue' always results 0 check the ring is empty
+ via rte_ring_empty()
+
+Are packets or objects are not processed at desired rate?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Memory objects close to NUMA :numref:`dtg_mempool`.
+
+.. _dtg_mempool:
+
+.. figure:: img/dtg_mempool.*
+
+ Memory objects has to be close to device per NUMA
+
+#. Is the performance low?
+ - Are packets received from multiple NIC? rte_eth_dev_count_all()
+ - Are NIC interfaces on different socket? use rte_eth_dev_socket_id()
+ - Is mempool created with right socket? rte_mempool_create() or
+ rte_pktmbuf_pool_create()
+ - Are drops on specific socket? If yes check if there are sufficient
+ objects by rte_mempool_get_count() or rte_mempool_avail_count()
+ - Is 'rte_mempool_get_count() or rte_mempool_avail_count()' zero?
+ application requires more objects hence reconfigure number of
+ elements in rte_mempool_create().
+ - Is there single RX thread for multiple NIC? try having multiple
+ lcore to read from fixed interface or we might be hitting cache
+ limit, so increase cache_size for pool_create().
+
+#. Is performance low for some scenarios?
+ - Check if sufficient objects in mempool by rte_mempool_avail_count()
+ - Is failure seen in some packets? we might be getting packets with
+ 'size > mbuf data size'.
+ - Is NIC offload or application handling multi segment mbuf? check the
+ special packets are continuous with rte_pktmbuf_is_contiguous().
+ - If there separate user threads used to access mempool objects, use
+ rte_mempool_cache_create() for non DPDK threads.
+ - Is the error reproducible with 1GB hugepage? If no, then try debugging
+ the issue with lookup table or objects with rte_mem_lock_page().
+
+.. note::
+ Stall in release of MBUF can be because
+
+ * Processing pipeline is too heavy
+ * Number of stages are too many
+ * TX is not transferred at desired rate
+ * Multi segment is not offloaded at TX device.
+ * Application misuse scenarios can be
+ - not freeing packets
+ - invalid rte_pktmbuf_refcnt_set
+ - invalid rte_pktmbuf_prefree_seg
+
+Is there difference in performance for crypto?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Crypto device and PMD :numref:`dtg_crypto`.
+
+.. _dtg_crypto:
+
+.. figure:: img/dtg_crypto.*
+
+ CRYPTO and interaction with PMD device
+
+#. Are generic configuration correct?
+ - Get total crypto devices – rte_cryptodev_count()
+ - Cross check software or hardware flags are configured properly
+ rte_cryptodev_info_get() for feature_flags
+
+#. If enqueue request > actual enqueue (drops)?
+ - Is the queue pair setup for right NUMA? check for socket_id using
+ rte_cryptodev_queue_pair_setup().
+ - Is the session_pool created from same socket_id as queue pair? If no,
+ then create on same NUMA.
+ - Is enqueue thread on same socket_id as object? If no, then try
+ to put on same NUMA.
+ - Are there errors and drops? check err_count using rte_cryptodev_stats()
+ - Do multiple threads enqueue or dequeue from same queue pair? Try
+ debugging with separate threads.
+
+#. If enqueue rate > dequeue rate?
+ - Is dequeue lcore thread is same socket_id?
+ - If software crypto is in use, check if the CRYPTO Library is build with
+ right (SIMD) flags or check if the queue pair using CPU ISA for
+ feature_flags AVX|SSE|NEON using rte_cryptodev_info_get()
+ - If its hardware assisted crypto showing performance variance? Check if
+ hardware is on same NUMA socket as queue pair and session pool.
+
+Worker functions not giving performance?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Custom worker function :numref:`dtg_distributor_worker`.
+
+.. _dtg_distributor_worker:
+
+.. figure:: img/dtg_distributor_worker.*
+
+ Custom worker function performance drops
+
+#. Performance
+ - Threads context switches are more frequent? Identify lcore with
+ rte_lcore() and lcore index mapping with rte_lcore_index(). Best
+ performance when mapping of thread and core is 1:1.
+ - What are lcore role (type or state)? fetch the roles like RTE, OFF and
+ SERVICE using rte_eal_lcore_role().
+ - Check if application has multiple functions running on same service
+ core? registered functions may be exceeding the desired time slots
+ while running on same service core.
+ - Is function is running on RTE core? check if there are conflicting
+ functions running on same CPU core by rte_thread_get_affinity().
+
+#. Debug
+ - Check what is mode of operation? master core, lcore, service core,
+ and numa count can be fetched with rte_eal_get_configuration().
+ - Is it occurring on special scenario? Analyze run logic with
+ rte_dump_stack(), rte_dump_registers() and rte_memdump() for more
+ insights.
+ - Is 'perf' showing data process or memory stalls in functions? check
+ instruction being generated for functions using objdump.
+
+Service functions are not frequent enough?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+service functions on service cores :numref:`dtg_service`.
+
+.. _dtg_service:
+
+.. figure:: img/dtg_service.*
+
+ functions running on service cores
+
+#. Performance
+ - Get service core count using rte_service_lcore_count() and compare with
+ result of rte_eal_get_configuration()
+ - Check registered service is available using rte_service_get_by_name(),
+ rte_service_get_count() and rte_service_get_name()
+ - Is given service running parallel on multiple lcores?
+ rte_service_probe_capability() and rte_service_map_lcore_get()
+ - Is service running? rte_service_runstate_get()
+
+#. Debug
+ - Find how many services are running on specific service lcore by
+ rte_service_lcore_count_services()
+ - Generic debug via rte_service_dump()
+
+Is there bottleneck in eventdev?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. Are generic configuration correct?
+ - Get event_dev devices? rte_event_dev_count()
+ - Are they created on correct socket_id? - rte_event_dev_socket_id()
+ - Check if HW or SW capabilities? - rte_event_dev_info_get() for
+ event_qos, queue_all_types, burst_mode, multiple_queue_port,
+ max_event_queue|dequeue_depth
+ - Is packet stuck in queue? check for stages (event qeueue) where
+ packets are looped back to same or previous stages.
+
+#. Performance drops in enqueue (event count > actual enqueue)?
+ - Dump the event_dev information? rte_event_dev_dump()
+ - Check stats for queue and port for eventdev
+ - Check the inflight, current queue element for enqueue|deqeue
+
+How to debug QoS via TM?
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+TM on TX interface :numref:`dtg_qos_tx`.
+
+.. _dtg_qos_tx:
+
+.. figure:: img/dtg_qos_tx.*
+
+ Traffic Manager just before TX
+
+#. Is configuration right?
+ - Get current capabilities for DPDK port for max nodes, level, shaper
+ private, shaper shared, sched_n_children and stats_mask using
+ rte_tm_capabilities_get()
+ - Check if current leaf are configured identically by fetching
+ leaf_nodes_identical using rte_tm_capabilities_get()
+ - Get leaf nodes for a dpdk port - rte_tn_get_number_of_leaf_node()
+ - Check level capabilities by rte_tm_level_capabilities_get for n_nodes
+ - Max, nonleaf_max, leaf_max
+ - identical, non_identical
+ - Shaper_private_supported
+ - Stats_mask
+ - Cman wred packet|byte supported
+ - Cman head drop supported
+ - Check node capabilities by rte_tm_node_capabilities_get for n_nodes
+ - Shaper_private_supported
+ - Stats_mask
+ - Cman wred packet|byte supported
+ - Cman head drop supported
+ - Debug via stats - rte_tm_stats_update() and rte_tm_node_stats_read()
+
+Packet is not of right format?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Packet capture before and after processing :numref:`dtg_pdump`.
+
+.. _dtg_pdump:
+
+.. figure:: img/dtg_pdump.*
+
+ Capture points of Traffic at RX-TX
+
+#. Where to capture packets?
+ - Enable pdump in primary to allow secondary to access queue-pair for
+ ports. Thus packets are copied over in RX|TX callback by secondary
+ process using ring buffers.
+ - To capture packet in middle of pipeline stage, user specific hooks
+ or callback are to be used to copy the packets. These packets can
+ be shared to secondary process via user defined custom rings.
+
+Issue still persists?
+~~~~~~~~~~~~~~~~~~~~~
+
+#. Are there custom or vendor specific offload meta data?
+ - From PMD, then check for META data error and drops.
+ - From application, then check for META data error and drops.
+#. Is multiprocess is used configuration and data processing?
+ - Check enabling or disabling features from secondary is supported or not?
+#. Is there drops for certain scenario for packets or objects?
+ - Check user private data in objects by dumping the details for debug.
+
+How to develop custom code to debug?
+------------------------------------
+
+- For single process - the debug functionality is to be added in same
+ process
+- For multiple process - the debug functionality can be added to
+ secondary multi process
+
+.. note::
+
+ Primary's Debug functions invoked via
+ #. Timer call-back
+ #. Service function under service core
+ #. USR1 or USR2 signal handler
diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a642a2be1..9527fa84d 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -18,3 +18,4 @@ HowTo Guides
virtio_user_as_exceptional_path
packet_capture_framework
telemetry
+ debug_troubleshoot_guide
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v5 1/2] doc: add svg for debug and troubleshoot guide
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 1/2] doc: add svg for " Vipin Varghese
@ 2019-01-28 1:08 ` Thomas Monjalon
2019-01-28 14:28 ` Varghese, Vipin
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 0/2] guide to debug and troubleshoot Vipin Varghese
1 sibling, 1 reply; 23+ messages in thread
From: Thomas Monjalon @ 2019-01-28 1:08 UTC (permalink / raw)
To: Vipin Varghese
Cc: dev, shreyansh.jain, john.mcnamara, marko.kovacevic, amol.patel,
sanjay.padubidri
Hi,
I don't know how I caught a typo in the middle of SVG files :)
> + <title>dsitributor and worker</title>
dsitributor -> distributor
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v5 2/2] doc: add guide for debug and troubleshoot
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 " Vipin Varghese
@ 2019-01-28 1:30 ` Thomas Monjalon
2019-01-28 14:51 ` Varghese, Vipin
0 siblings, 1 reply; 23+ messages in thread
From: Thomas Monjalon @ 2019-01-28 1:30 UTC (permalink / raw)
To: Vipin Varghese, john.mcnamara, marko.kovacevic
Cc: dev, shreyansh.jain, amol.patel, sanjay.padubidri
Hi,
I feel this doc will be updated to provide a complete debug checklist,
and will become useful to many users hopefully.
One general comment about documentation,
It is better to wrap lines logically, for example, always start sentences
at the beginning of a new line. It will make further update patches
simpler to review.
Few more nits below,
21/01/2019 11:41, Vipin Varghese:
> +.. _debug_troubleshoot_via_pmd:
No need of such anchor.
> +
> +Debug & Troubleshoot guide via PMD
> +==================================
Why "via PMD"? Do we use PMD for troubleshooting?
Or is it dedicated to troubleshoot the PMD behaviour?
> +
> +DPDK applications can be designed to run as single thread simple stage to
> +multiple threads with complex pipeline stages. These application can use poll
applications
> +mode devices which helps in offloading CPU cycles. A few models are
help
A colon would be nice at the end of the line before the list.
> +
> + * single primary
> + * multiple primary
> + * single primary single secondary
> + * single primary multiple secondary
> +
> +In all the above cases, it is a tedious task to isolate, debug and understand
> +odd behaviour which occurs randomly or periodically. The goal of guide is to
> +share and explore a few commonly seen patterns and behaviour. Then, isolate
> +and identify the root cause via step by step debug at various processing
> +stages.
I don't understand how this introduction is related to "via PMD" in the title.
> +
> +Application Overview
> +--------------------
> +
> +Let us take up an example application as reference for explaining issues and
> +patterns commonly seen. The sample application in discussion makes use of
> +single primary model with various pipeline stages. The application uses PMD
> +and libraries such as service cores, mempool, pkt mbuf, event, crypto, QoS
> +and eth.
"pkt mbuf" can be called simply mbuf, but event, crypto and eth
should be eventdev, cryptodev and ethdev.
> +
> +The overview of an application modeled using PMD is shown in
> +:numref:`dtg_sample_app_model`.
> +
> +.. _dtg_sample_app_model:
> +
> +.. figure:: img/dtg_sample_app_model.*
> +
> + Overview of pipeline stage of an application
> +
> +Bottleneck Analysis
> +-------------------
> +
> +To debug the bottleneck and performance issues the desired application
missing comma after "issues"?
> +is made to run in an environment matching as below
colon missing
> +
> +#. Linux 64-bit|32-bit
> +#. DPDK PMD and libraries are used
Isn't it always the case with DPDK?
> +#. Libraries and PMD are either static or shared. But not both
Strange assumption. Why would it be both?
> +#. Machine flag optimizations of gcc or compiler are made constant
What do you mean?
> +
> +Is there mismatch in packet rate (received < send)?
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +RX Port and associated core :numref:`dtg_rx_rate`.
> +
> +.. _dtg_rx_rate:
> +
> +.. figure:: img/dtg_rx_rate.*
> +
> + RX send rate compared against Received rate
RX send ?
> +
> +#. Are generic configuration correct?
Are -> Is
> + - What is port Speed, Duplex? rte_eth_link_get()
> + - Are packets of higher sizes are dropped? rte_eth_get_mtu()
are dropped -> dropped
> + - Are only specific MAC received? rte_eth_promiscuous_get()
> +
> +#. Are there NIC specific drops?
> + - Check rte_eth_rx_queue_info_get() for nb_desc and scattered_rx
> + - Is RSS enabled? rte_eth_dev_rss_hash_conf_get()
> + - Are packets spread on all queues? rte_eth_dev_stats()
> + - If stats for RX and drops updated on same queue? check receive thread
> + - If packet does not reach PMD? check if offload for port and queue
> + matches to traffic pattern send.
> +
> +#. If problem still persists, this might be at RX lcore thread
> + - Check if RX thread, distributor or event rx adapter? these may be
> + processing less than required
> + - Is the application is build using processing pipeline with RX stage? If
is build -> built
> + there are multiple port-pair tied to a single RX core, try to debug by
> + using rte_prefetch_non_temporal(). This will intimate the mbuf in cache
> + is temporary.
I stop nit-picking review here.
Marko, John, please could you check english grammar?
Thanks
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v5 1/2] doc: add svg for debug and troubleshoot guide
2019-01-28 1:08 ` Thomas Monjalon
@ 2019-01-28 14:28 ` Varghese, Vipin
0 siblings, 0 replies; 23+ messages in thread
From: Varghese, Vipin @ 2019-01-28 14:28 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, shreyansh.jain, Mcnamara, John, Kovacevic, Marko, Patel,
Amol, Padubidri, Sanjay A
Thanks Thomas,
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, January 28, 2019 6:38 AM
> To: Varghese, Vipin <vipin.varghese@intel.com>
> Cc: dev@dpdk.org; shreyansh.jain@nxp.com; Mcnamara, John
> <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; Patel, Amol <amol.patel@intel.com>;
> Padubidri, Sanjay A <sanjay.padubidri@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v5 1/2] doc: add svg for debug and
> troubleshoot guide
>
> Hi,
>
> I don't know how I caught a typo in the middle of SVG files :)
Indeed 'a keen eye for details.
>
> > + <title>dsitributor and worker</title>
>
> dsitributor -> distributor
Will rectify the same.
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v5 2/2] doc: add guide for debug and troubleshoot
2019-01-28 1:30 ` Thomas Monjalon
@ 2019-01-28 14:51 ` Varghese, Vipin
2019-01-28 15:59 ` Thomas Monjalon
0 siblings, 1 reply; 23+ messages in thread
From: Varghese, Vipin @ 2019-01-28 14:51 UTC (permalink / raw)
To: Thomas Monjalon, Mcnamara, John, Kovacevic, Marko
Cc: dev, shreyansh.jain, Patel, Amol, Padubidri, Sanjay A
Hi Thomas,
snipped
>
> I feel this doc will be updated to provide a complete debug checklist,
Attempt is made to capture commonly seen filed issue. Saying so, I am clear that I will not be able to identify all debug check list. As time, experience and sharing increases (from the community), I am certain sure this will grow
and will
snipped
> One general comment about documentation, It is better to wrap lines
> logically, for example, always start sentences at the beginning of a new line. It
> will make further update patches simpler to review.
>
> Few more nits below,
>
> 21/01/2019 11:41, Vipin Varghese:
> > +.. _debug_troubleshoot_via_pmd:
I need cross check with John or Marko on the same, as the PDF generator tool make a check for anchor and figure name.
>
> No need of such anchor
Please give me time to cross check.
.
>
> > +
> > +Debug & Troubleshoot guide via PMD
> > +==================================
>
> Why "via PMD"? Do we use PMD for troubleshooting?
I believe yes, we do collect information with enhanced procinfo tool.
> Or is it dedicated to troubleshoot the PMD behaviour?
I am not clear with this statement. Hence is the query 'Is this dedicated to troubleshooting Application. PMD and Library uses cases?'
>
> > +
> > +DPDK applications can be designed to run as single thread simple
> > +stage to multiple threads with complex pipeline stages. These
> > +application can use poll
>
> applications
Ok
>
> > +mode devices which helps in offloading CPU cycles. A few models are
>
> help
Ok
>
> A colon would be nice at the end of the line before the list.
>
> > +
> > + * single primary
> > + * multiple primary
> > + * single primary single secondary
> > + * single primary multiple secondary
> > +
> > +In all the above cases, it is a tedious task to isolate, debug and
> > +understand odd behaviour which occurs randomly or periodically. The
> > +goal of guide is to share and explore a few commonly seen patterns
> > +and behaviour. Then, isolate and identify the root cause via step by
> > +step debug at various processing stages.
>
> I don't understand how this introduction is related to "via PMD" in the title.
I believe the information is shared ```The goal of guide is to share and explore a few commonly seen patterns and behaviour. Then, isolate and identify the root cause via step by step debug at various processing stages.'```
There would multiple ways to design application for solving a same problem. These are depended on user, platform, scaling factor and target. These various combinations make use PMD and libraries. Misconfiguration and not taking care of platform will cause throttling and even drops.
Example: application designed to run on single is now been deployed to run on multi NUMA model.
snipped
>
> "pkt mbuf" can be called simply mbuf, but event, crypto and eth should be
> eventdev, cryptodev and ethdev.
Ok. I can make this change.
>
snipped
> > +To debug the bottleneck and performance issues the desired
> > +application
>
> missing comma after "issues"?
Ok
>
> > +is made to run in an environment matching as below
>
> colon missing
Ok
>
> > +
> > +#. Linux 64-bit|32-bit
> > +#. DPDK PMD and libraries are used
>
> Isn't it always the case with DPDK?
>
> > +#. Libraries and PMD are either static or shared. But not both
>
> Strange assumption. Why would it be both?
If applications are only build with DPDK libraries, then yes the assumption is correct. But when applications are build using DPDK as one of software layer (example DPDK network stack, DPDK suricata, DPDK hyperscan) as per my understanding this is not true.
>
> > +#. Machine flag optimizations of gcc or compiler are made constant
>
> What do you mean?
I can reword as ```DPDK and the application libraries are built with same flags. ```
>
snipped
> > +
> > + RX send rate compared against Received rate
>
> RX send ?
Thanks will correct this
>
> > +
> > +#. Are generic configuration correct?
>
> Are -> Is
>
> > + - What is port Speed, Duplex? rte_eth_link_get()
> > + - Are packets of higher sizes are dropped? rte_eth_get_mtu()
>
> are dropped -> dropped
Ok
snipped
> > + - Is the application is build using processing pipeline with RX
> > +stage? If
>
> is build -> built
Ok
>
> > + there are multiple port-pair tied to a single RX core, try to debug by
> > + using rte_prefetch_non_temporal(). This will intimate the mbuf in cache
> > + is temporary.
>
> I stop nit-picking review here.
Thanks as any form of correction is always good.
> Marko, John, please could you check english grammar?
> Thanks
>
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v5 2/2] doc: add guide for debug and troubleshoot
2019-01-28 14:51 ` Varghese, Vipin
@ 2019-01-28 15:59 ` Thomas Monjalon
2019-02-08 9:21 ` Varghese, Vipin
0 siblings, 1 reply; 23+ messages in thread
From: Thomas Monjalon @ 2019-01-28 15:59 UTC (permalink / raw)
To: Varghese, Vipin
Cc: Mcnamara, John, Kovacevic, Marko, dev, shreyansh.jain, Patel,
Amol, Padubidri, Sanjay A
28/01/2019 15:51, Varghese, Vipin:
> Hi Thomas,
>
> snipped
> >
> > I feel this doc will be updated to provide a complete debug checklist,
> Attempt is made to capture commonly seen filed issue. Saying so, I am clear that I will not be able to identify all debug check list. As time, experience and sharing increases (from the community), I am certain sure this will grow
Yes this is what I mean.
We just need to give a good start by explaining well the intent and context.
> > > +Debug & Troubleshoot guide via PMD
> > > +==================================
> >
> > Why "via PMD"? Do we use PMD for troubleshooting?
> I believe yes, we do collect information with enhanced procinfo tool.
>
> > Or is it dedicated to troubleshoot the PMD behaviour?
> I am not clear with this statement. Hence is the query 'Is this dedicated to troubleshooting Application. PMD and Library uses cases?'
Sorry I don't understand.
I think you can just remove "via PMD" in the title.
[...]
> > > + * single primary
> > > + * multiple primary
> > > + * single primary single secondary
> > > + * single primary multiple secondary
> > > +
> > > +In all the above cases, it is a tedious task to isolate, debug and
> > > +understand odd behaviour which occurs randomly or periodically. The
> > > +goal of guide is to share and explore a few commonly seen patterns
> > > +and behaviour. Then, isolate and identify the root cause via step by
> > > +step debug at various processing stages.
> >
> > I don't understand how this introduction is related to "via PMD" in the title.
> I believe the information is shared ```The goal of guide is to share and explore a few commonly seen patterns and behaviour. Then, isolate and identify the root cause via step by step debug at various processing stages.'```
>
> There would multiple ways to design application for solving a same problem. These are depended on user, platform, scaling factor and target. These various combinations make use PMD and libraries. Misconfiguration and not taking care of platform will cause throttling and even drops.
>
> Example: application designed to run on single is now been deployed to run on multi NUMA model.
Yes, so you are explaining there can be a lot of different scenarios.
[...]
> > > +#. Linux 64-bit|32-bit
> > > +#. DPDK PMD and libraries are used
> >
> > Isn't it always the case with DPDK?
> >
> > > +#. Libraries and PMD are either static or shared. But not both
> >
> > Strange assumption. Why would it be both?
> If applications are only build with DPDK libraries, then yes the assumption is correct. But when applications are build using DPDK as one of software layer (example DPDK network stack, DPDK suricata, DPDK hyperscan) as per my understanding this is not true.
Sorry I don't understand.
The DPDK libraries are either shared or static, but never mixed.
Anyway why is it significant here?
> > > +#. Machine flag optimizations of gcc or compiler are made constant
> >
> > What do you mean?
> I can reword as ```DPDK and the application libraries are built with same flags. ```
Why is it significant?
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v5 2/2] doc: add guide for debug and troubleshoot
2019-01-28 15:59 ` Thomas Monjalon
@ 2019-02-08 9:21 ` Varghese, Vipin
0 siblings, 0 replies; 23+ messages in thread
From: Varghese, Vipin @ 2019-02-08 9:21 UTC (permalink / raw)
To: Thomas Monjalon
Cc: Mcnamara, John, Kovacevic, Marko, dev, shreyansh.jain, Patel,
Amol, Padubidri, Sanjay A
Hi Thomas, John and Marko,
I am working on sharing the next version with grammar checks done with Grammarly. Will update ASAP.
Thanks
Vipin Varghese
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Monday, January 28, 2019 9:30 PM
> To: Varghese, Vipin <vipin.varghese@intel.com>
> Cc: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; dev@dpdk.org; shreyansh.jain@nxp.com;
> Patel, Amol <amol.patel@intel.com>; Padubidri, Sanjay A
> <sanjay.padubidri@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v5 2/2] doc: add guide for debug and
> troubleshoot
>
> 28/01/2019 15:51, Varghese, Vipin:
> > Hi Thomas,
> >
> > snipped
> > >
> > > I feel this doc will be updated to provide a complete debug
> > > checklist,
> > Attempt is made to capture commonly seen filed issue. Saying so, I am
> > clear that I will not be able to identify all debug check list. As
> > time, experience and sharing increases (from the community), I am
> > certain sure this will grow
>
> Yes this is what I mean.
> We just need to give a good start by explaining well the intent and context.
>
> > > > +Debug & Troubleshoot guide via PMD
> > > > +==================================
> > >
> > > Why "via PMD"? Do we use PMD for troubleshooting?
> > I believe yes, we do collect information with enhanced procinfo tool.
> >
> > > Or is it dedicated to troubleshoot the PMD behaviour?
> > I am not clear with this statement. Hence is the query 'Is this dedicated to
> troubleshooting Application. PMD and Library uses cases?'
>
> Sorry I don't understand.
> I think you can just remove "via PMD" in the title.
>
> [...]
> > > > + * single primary
> > > > + * multiple primary
> > > > + * single primary single secondary
> > > > + * single primary multiple secondary
> > > > +
> > > > +In all the above cases, it is a tedious task to isolate, debug
> > > > +and understand odd behaviour which occurs randomly or
> > > > +periodically. The goal of guide is to share and explore a few
> > > > +commonly seen patterns and behaviour. Then, isolate and identify
> > > > +the root cause via step by step debug at various processing stages.
> > >
> > > I don't understand how this introduction is related to "via PMD" in the
> title.
> > I believe the information is shared ```The goal of guide is to share
> > and explore a few commonly seen patterns and behaviour. Then, isolate
> > and identify the root cause via step by step debug at various
> > processing stages.'```
> >
> > There would multiple ways to design application for solving a same problem.
> These are depended on user, platform, scaling factor and target. These various
> combinations make use PMD and libraries. Misconfiguration and not taking
> care of platform will cause throttling and even drops.
> >
> > Example: application designed to run on single is now been deployed to run
> on multi NUMA model.
>
> Yes, so you are explaining there can be a lot of different scenarios.
>
> [...]
> > > > +#. Linux 64-bit|32-bit
> > > > +#. DPDK PMD and libraries are used
> > >
> > > Isn't it always the case with DPDK?
> > >
> > > > +#. Libraries and PMD are either static or shared. But not both
> > >
> > > Strange assumption. Why would it be both?
> > If applications are only build with DPDK libraries, then yes the assumption is
> correct. But when applications are build using DPDK as one of software layer
> (example DPDK network stack, DPDK suricata, DPDK hyperscan) as per my
> understanding this is not true.
>
> Sorry I don't understand.
> The DPDK libraries are either shared or static, but never mixed.
> Anyway why is it significant here?
>
> > > > +#. Machine flag optimizations of gcc or compiler are made
> > > > +constant
> > >
> > > What do you mean?
> > I can reword as ```DPDK and the application libraries are built with
> > same flags. ```
>
> Why is it significant?
>
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v6 0/2] guide to debug and troubleshoot.
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 1/2] doc: add svg for " Vipin Varghese
2019-01-28 1:08 ` Thomas Monjalon
@ 2019-02-19 15:57 ` Vipin Varghese
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 1/2] doc: add svg for debug and troubleshoot guide Vipin Varghese
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
1 sibling, 2 replies; 23+ messages in thread
From: Vipin Varghese @ 2019-02-19 15:57 UTC (permalink / raw)
To: john.mcnamara, marko.kovacevic, thomas, ferruh.yigit,
shreyansh.jain, dev
Cc: sanjay.padubidri, amol.patel, Vipin Varghese
The patch series adds a how-to guide for debugging and troubleshooting tips.
Motivation
==========
DPDK proc-info tool is been enhanced to accommodate the debug information for
the port, traffic manager crypto, ring and mempool contents. With these
additional information, it becomes easy to analyze issues and performance
variance.
But applications are designed based on the target platform, workload, poll mode
drivers, and multi-process. This raises variance in debugging and collecting
data. Hence attempt of patch series is identified such symptoms and share step
by step guide to cover the cases.
Not all possible cases could be covered in a single attempt. But with feedback
and support from the community, this can be expanded.
Status
======
Reviews and changes accommodated. ACK received for documentation and SVG files.
Change Log:
==========
V5:
- correction for word style and grammar - Thomas Monjalon
- add license for svg files - Vipin Varghese
V4:
- Correction for word style - Shreyansh Jain
V3:
- reorder for removing warning in 'make doc-guides-html' - Thomas Monjalon
V2:
- add offload flag check - Vipin Varghese
- change tab to space - Marko Kovacevic
- spelling correction - Marko Kovacevic
- remove extra characters - Marko Kovacevic
- add ACK by Marko - Vipn Varghese
- add ACK from Marko - Vipin Varghese
Vipin Varghese (2):
doc: add svg for debug and troubleshoot guide
doc: add guide for debug and troubleshoot
doc/guides/howto/debug_troubleshoot_guide.rst | 451 ++++++++++++++++++
doc/guides/howto/img/dtg_consumer_ring.svg | 24 +
doc/guides/howto/img/dtg_crypto.svg | 21 +
.../howto/img/dtg_distributor_worker.svg | 36 ++
doc/guides/howto/img/dtg_mempool.svg | 27 ++
doc/guides/howto/img/dtg_pdump.svg | 33 ++
doc/guides/howto/img/dtg_producer_ring.svg | 24 +
doc/guides/howto/img/dtg_qos_tx.svg | 29 ++
doc/guides/howto/img/dtg_rx_rate.svg | 25 +
doc/guides/howto/img/dtg_rx_tx_drop.svg | 33 ++
doc/guides/howto/img/dtg_sample_app_model.svg | 110 +++++
doc/guides/howto/img/dtg_service.svg | 20 +
doc/guides/howto/index.rst | 1 +
13 files changed, 834 insertions(+)
create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
create mode 100644 doc/guides/howto/img/dtg_consumer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_crypto.svg
create mode 100644 doc/guides/howto/img/dtg_distributor_worker.svg
create mode 100644 doc/guides/howto/img/dtg_mempool.svg
create mode 100644 doc/guides/howto/img/dtg_pdump.svg
create mode 100644 doc/guides/howto/img/dtg_producer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_qos_tx.svg
create mode 100644 doc/guides/howto/img/dtg_rx_rate.svg
create mode 100644 doc/guides/howto/img/dtg_rx_tx_drop.svg
create mode 100644 doc/guides/howto/img/dtg_sample_app_model.svg
create mode 100644 doc/guides/howto/img/dtg_service.svg
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v6 1/2] doc: add svg for debug and troubleshoot guide
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 0/2] guide to debug and troubleshoot Vipin Varghese
@ 2019-02-19 15:57 ` Vipin Varghese
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
1 sibling, 0 replies; 23+ messages in thread
From: Vipin Varghese @ 2019-02-19 15:57 UTC (permalink / raw)
To: john.mcnamara, marko.kovacevic, thomas, ferruh.yigit,
shreyansh.jain, dev
Cc: sanjay.padubidri, amol.patel, Vipin Varghese
Add svg images for debug and troubleshoot guide.
Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
---
---
doc/guides/howto/img/dtg_consumer_ring.svg | 24 ++++
doc/guides/howto/img/dtg_crypto.svg | 21 ++++
.../howto/img/dtg_distributor_worker.svg | 36 ++++++
doc/guides/howto/img/dtg_mempool.svg | 27 +++++
doc/guides/howto/img/dtg_pdump.svg | 33 ++++++
doc/guides/howto/img/dtg_producer_ring.svg | 24 ++++
doc/guides/howto/img/dtg_qos_tx.svg | 29 +++++
doc/guides/howto/img/dtg_rx_rate.svg | 25 ++++
doc/guides/howto/img/dtg_rx_tx_drop.svg | 33 ++++++
doc/guides/howto/img/dtg_sample_app_model.svg | 110 ++++++++++++++++++
doc/guides/howto/img/dtg_service.svg | 20 ++++
11 files changed, 382 insertions(+)
create mode 100644 doc/guides/howto/img/dtg_consumer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_crypto.svg
create mode 100644 doc/guides/howto/img/dtg_distributor_worker.svg
create mode 100644 doc/guides/howto/img/dtg_mempool.svg
create mode 100644 doc/guides/howto/img/dtg_pdump.svg
create mode 100644 doc/guides/howto/img/dtg_producer_ring.svg
create mode 100644 doc/guides/howto/img/dtg_qos_tx.svg
create mode 100644 doc/guides/howto/img/dtg_rx_rate.svg
create mode 100644 doc/guides/howto/img/dtg_rx_tx_drop.svg
create mode 100644 doc/guides/howto/img/dtg_sample_app_model.svg
create mode 100644 doc/guides/howto/img/dtg_service.svg
diff --git a/doc/guides/howto/img/dtg_consumer_ring.svg b/doc/guides/howto/img/dtg_consumer_ring.svg
new file mode 100644
index 000000000..d8cf0c292
--- /dev/null
+++ b/doc/guides/howto/img/dtg_consumer_ring.svg
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="634" height="97" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>consumer ring</title>
+ <rect fill="#fff" id="canvas_background" height="99" width="636" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="95" width="135" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_2" height="95" width="135" y="2" x="252" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_3" height="95" width="135" y="2" x="499" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="45" x="10" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="52" x="268" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_6" y="53" x="517" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 3</text>
+ <path transform="rotate(90.19324493408203 177.864990234375,48.89100646972658) " id="svg_7" d="m155.364991,48.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path transform="rotate(90.19324493408203 434.864990234375,48.89100646972656) " id="svg_8" d="m412.364991,48.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path id="svg_10" d="m225.937481,47.95781l0,0c0,-10.493411 8.282734,-19 18.500005,-19l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564971c3.469424,3.563191 5.418523,8.395917 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999998 -18.5,18.999998l0,0c-10.217271,0 -18.500005,-8.506591 -18.500005,-18.999998zm18.500005,-19l0,37.999998m-18.500005,-18.999998l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ <path id="svg_11" d="m475.937481,46.95781l0,0c0,-10.493411 8.282734,-19.000001 18.500005,-19.000001l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564972c3.469424,3.563191 5.418523,8.395916 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999997 -18.5,18.999997l0,0c-10.217271,0 -18.500005,-8.50659 -18.500005,-18.999997zm18.500005,-19.000001l0,37.999998m-18.500005,-18.999997l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_crypto.svg b/doc/guides/howto/img/dtg_crypto.svg
new file mode 100644
index 000000000..969f8b5d5
--- /dev/null
+++ b/doc/guides/howto/img/dtg_crypto.svg
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="180" height="230" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>crypto</title>
+ <rect fill="#fff" id="canvas_background" height="232" width="182" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="1" width="0" y="369.4375" x="838.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="222.4375" x="39.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core 7</text>
+ <rect stroke="#000" id="svg_26" height="44.000003" width="155.000001" y="151.437497" x="6.499997" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_27" y="179.4375" x="13.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">CRYPTO PMD</text>
+ <ellipse stroke="#000" ry="41.5" rx="66.500001" id="svg_28" cy="48.9375" cx="87.999997" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_29" y="57.4375" x="45.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Device</text>
+ <path stroke="#000" transform="rotate(-90.19718170166016 85.76751708984375,119.45785522460936) " id="svg_30" d="m40.743289,119.457895l26.106598,-13.915756l0,6.95786l37.835289,0l0,-6.95786l26.106586,13.915756l-26.106586,13.915682l0,-6.957842l-37.835289,0l0,6.957842l-26.106598,-13.915682z" stroke-opacity="null" stroke-width="0" fill="#845335"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_distributor_worker.svg b/doc/guides/howto/img/dtg_distributor_worker.svg
new file mode 100644
index 000000000..d2638ded8
--- /dev/null
+++ b/doc/guides/howto/img/dtg_distributor_worker.svg
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="600" height="235" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>distributor and worker</title>
+ <rect fill="#fff" id="canvas_background" height="237" width="602" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_1" height="89.999991" width="202.000004" y="21.000003" x="4" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start"
+ font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="47" x="7"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">PKT classify</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_12" y="96.5" x="25.992188" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Distribute</text>
+ <path id="svg_13" d="m334.224445,98.046256l16.252783,-5.432074l-16.252783,-5.429993l-16.256262,-5.427952l0,6.038517c-4.264321,-0.004069 -11.331234,-0.004069 -13.334561,-0.004069c-5.919233,-0.081711 -9.804243,-1.153783 -13.847174,-3.073397c-6.02597,-2.865095 -11.16889,-8.021466 -17.371313,-12.989922c-3.154554,-2.470933 -6.669049,-4.921502 -11.349751,-6.892119c-4.633926,-1.978838 -10.68815,-3.351144 -17.41807,-3.324591l-29.170102,0l0,9.646983l29.170102,0c4.056131,0.042884 6.623169,0.808657 10.124445,2.70786c5.179989,2.824261 10.230311,8.223602 17.195784,13.339136c6.826056,5.066481 17.329865,10.182006 32.564612,10.231017l13.436029,0l0,6.0385l16.256262,-5.427896zm-68.759273,-36.535597c0.184389,0.075572 0.379369,0.14089 0.559297,0.218526c5.559349,2.340242 9.573988,5.211493 12.751502,7.709001c0.730423,0.584013 1.377934,1.145611 2.071302,1.719456l37.120911,0l0,6.040581l16.256262,-5.42798l16.252783,-5.429962l-16.252783,-5.432037l-16.256262,-5.434093l0,6.036509l-52.503011,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFAC8D"/>
+ <rect stroke="#000" id="svg_17" height="110.999999" width="180" y="3.4375" x="353.5"
+ stroke-opacity="null" stroke-width="0" fill="#FABA6F"/>
+ <rect stroke="#000" id="svg_14" height="110.999999" width="180" y="31.4375" x="369.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_15" height="110.999999" width="180" y="58.4375" x="392.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFF093"/>
+ <rect stroke="#000" id="svg_16" height="110.999999" width="180" y="90.4375" x="413.5"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,142.60348030924797,64.2037038393319) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="71.490132" x="413.769676" stroke-opacity="null" stroke-width="0" fill="#000000">worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="101.332237" x="406.404514" stroke-opacity="null" stroke-width="0" fill="#000000">worker 2</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_20" y="58.700658" x="376.943866" stroke-opacity="null" stroke-width="0" fill="#000000">worker 3</text>
+ <text style="cursor: move;" stroke="#000" transform="matrix(0.678871691226959,0,0,0.7037037014961243,121.60348030924797,9.203703839331922) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_22" y="18.911184" x="350.429283" stroke-opacity="null" stroke-width="0" fill="#000000">worker 4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="139.4375" x="49.5"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_24" y="226.4375" x="404.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4,5</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_mempool.svg b/doc/guides/howto/img/dtg_mempool.svg
new file mode 100644
index 000000000..2f53a3ca8
--- /dev/null
+++ b/doc/guides/howto/img/dtg_mempool.svg
@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="955" height="185" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>mempool</title>
+ <rect fill="#fff" id="canvas_background" height="187" width="957" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="1" width="0" y="370.4375" x="839.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <rect stroke="#000" id="svg_31" height="181.000002" width="951.000029" y="2.4375" x="3.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_32" y="167.4375" x="14.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">MBUF pool</text>
+ <rect id="svg_38" height="71" width="859" y="15.4375" x="18.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_39" height="71" width="859" y="32.4375" x="33.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_40" height="71" width="859" y="47.4375" x="52.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect id="svg_41" height="71" width="859" y="62.4375" x="71.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8B4D4B"/>
+ <rect stroke="#000" id="svg_42" height="55.999999" width="403.000005" y="69.4375" x="486.499995" stroke-opacity="null" stroke-width="0" fill="#FFAC8D"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_43" y="103.4375" x="631.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Payload</text>
+ <rect stroke="#000" id="svg_44" height="56" width="140.999999" y="68.4375" x="292.500001" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_45" y="105.4375" x="303.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Metadata</text>
+ <rect stroke="#000" id="svg_46" height="53" width="185.999997" y="70.4375" x="90.5" stroke-opacity="null" stroke-width="0" fill="#EDFFFF"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_47" y="104.4375" x="105.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">struct mbuf</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_pdump.svg b/doc/guides/howto/img/dtg_pdump.svg
new file mode 100644
index 000000000..4b14fb7ce
--- /dev/null
+++ b/doc/guides/howto/img/dtg_pdump.svg
@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="860" height="140" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>pdump</title>
+ <rect fill="#fff" id="canvas_background" height="142" width="862" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_15" height="127.999992" width="568" y="7" x="8.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#B1FFFF"/>
+ <rect id="svg_2" height="51" width="100" y="23" x="22.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_3" height="51" width="100" y="26" x="455.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CC7C65"/>
+ <path id="svg_7" d="m208.364769,78.968848l12.704559,-5.623238l12.697917,-5.625551l-12.704644,-5.629525l-12.697836,-5.625202l0,6.255891c-3.112384,0 -6.729051,0 -7.940704,0c-5.776899,0.124913 -12.907107,-3.365352 -20.492742,-7.864703c-2.827801,-1.577599 -5.680695,-3.313223 -8.704363,-4.88042c3.023667,-1.569118 5.872475,-3.302747 8.704363,-4.880295c7.585635,-4.493132 14.715839,-7.98967 20.485292,-7.862546l7.94815,-0.002103l0,6.264275l12.697836,-5.627447l12.704644,-5.631775l-12.704644,-5.637807l-12.697836,-5.629547l0,6.260273l-7.944067,0c-11.799214,0.12679 -20.61742,5.506774 -28.333828,9.879239c-7.660201,4.613909 -14.545738,8.087381 -18.539115,7.862617l-22.780531,0l0,9.993806l22.788007,0c3.989952,-0.226969 10.878944,3.248761 18.53841,7.86254c7.71303,4.374678 16.52785,9.750196 28.330427,9.879312l7.940704,0l0,6.262203l0,0l0,0.000003z" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#559E84"/>
+ <path id="svg_8" d="m439.457308,43.77545l-10.478696,-7.306254l0,8.12971l-18.805519,0c-3.29272,0.296972 -8.977707,-4.212008 -15.298608,-10.195089c-6.365074,-5.675375 -13.639418,-12.638687 -23.37938,-12.824984l-6.552957,0l0,12.959904l6.552957,0c4.770151,-0.167389 10.652051,4.360514 16.914219,10.203329c2.334186,2.041106 4.688512,4.292919 7.183208,6.326016c-2.494696,2.035766 -4.846223,4.290275 -7.183208,6.331424c-6.262164,5.826551 -12.144067,10.365242 -16.908056,10.200515l-6.559117,0l0,12.978808l6.555749,0c9.719269,-0.1646 16.996426,-7.14678 23.382185,-12.814082c6.32147,-5.983193 12.003677,-10.494803 15.299177,-10.203221l18.799357,0l0,8.129634l10.478696,-7.298036l10.484823,-7.303408l-10.484823,-7.314263l-0.000006,-0.000004z" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#559E84"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="54" x="53.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_10" y="57" x="486.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_11" y="40" x="243.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_12" y="77" x="243.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="82" x="326.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="40" x="327.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Q3</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="114.86203" x="234.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Primary</text>
+ <rect stroke="#000" id="svg_18" height="90.000001" width="201.999994" y="29.437499" x="654.5" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="102.4375" x="684.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Secondary</text>
+ <rect stroke="#000" id="svg_20" height="42.000001" width="184.000001" y="34.437499" x="664.499994" stroke-opacity="null" stroke-width="0" fill="#A0D58A"/>
+ <rect id="svg_21" height="1" width="0" y="366.4375" x="835.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#A0D58A"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_22" y="59.4375" x="668.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Ring BufferQ</text>
+ <path stroke="#000" transform="rotate(89.78996276855469 615.4766845703124,74.49395751953122) " id="svg_24" d="m599.023803,74.414257l17.255478,-33.667333l15.650356,33.667333l-8.226459,0l0,33.826726l-16.452915,0l0,-33.826726l-8.226459,0z" fill-opacity="null" stroke-opacity="null" stroke-width="0" fill="#7AD6CA"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="22.4375" x="714.5" fill-opacity="null" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core 0</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_producer_ring.svg b/doc/guides/howto/img/dtg_producer_ring.svg
new file mode 100644
index 000000000..e300234cd
--- /dev/null
+++ b/doc/guides/howto/img/dtg_producer_ring.svg
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="634" height="97" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>producer ring</title>
+ <rect fill="#fff" id="canvas_background" height="99" width="636" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="95" width="135" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_2" height="95" width="135" y="2" x="252" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <rect id="svg_3" height="95" width="135" y="2" x="499" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="45" x="10" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 1</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="52" x="268" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_6" y="53" x="517" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Stage 3</text>
+ <path transform="rotate(90.19324493408203 197.864990234375,42.89100646972657) " id="svg_7" d="m175.364991,42.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path transform="rotate(90.19324493408203 445.864990234375,47.89100646972657) " id="svg_8" d="m423.364991,47.766198l22.499998,-51.875196l22.499998,51.875196l-11.25,0l0,52.12481l-22.499996,0l0,-52.12481l-11.25,0z" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#4D2425"/>
+ <path id="svg_10" d="m120.937481,42.95781l0,0c0,-10.493411 8.282734,-19 18.500005,-19l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564971c3.469424,3.563191 5.418523,8.395917 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999998 -18.5,18.999998l0,0c-10.217271,0 -18.500005,-8.506591 -18.500005,-18.999998zm18.500005,-19l0,37.999998m-18.500005,-18.999998l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ <path id="svg_11" d="m373.937481,45.95781l0,0c0,-10.493411 8.282734,-19.000001 18.500005,-19.000001l0,0c4.906504,0 9.612052,2.00178 13.081477,5.564972c3.469424,3.563191 5.418523,8.395916 5.418523,13.435029l0,0c0,10.493408 -8.282732,18.999997 -18.5,18.999997l0,0c-10.217271,0 -18.500005,-8.50659 -18.500005,-18.999997zm18.500005,-19.000001l0,37.999998m-18.500005,-18.999997l37.000005,0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#8CD79F"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_qos_tx.svg b/doc/guides/howto/img/dtg_qos_tx.svg
new file mode 100644
index 000000000..9f1d3b80d
--- /dev/null
+++ b/doc/guides/howto/img/dtg_qos_tx.svg
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="287" height="180" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>qos</title>
+ <rect fill="#fff" id="canvas_background" height="182" width="289" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_21" height="180" width="287" y="0" x="0" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CBFFFF"/>
+ <rect id="svg_10" height="82" width="78" y="17.808165" x="152.25" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_11" height="82" width="78" y="36.808165" x="171.25" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_12" height="82" width="78" y="59.808165" x="188.25" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="113.808165" x="211.25" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="161.808165" x="161.25" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core1</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_15" y="235.775764" x="531.631651" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.3615568424500273,0,0,0.9358925757041695,106.13283890389599,-107.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="173.881425" x="192.8023" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_17" y="66.670315" x="-70.838675" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ <rect id="svg_18" height="103" width="104" y="26" x="26" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FABA6F"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_19" y="79" x="50" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">QoS</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_20" y="153" x="27" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 10</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_rx_rate.svg b/doc/guides/howto/img/dtg_rx_rate.svg
new file mode 100644
index 000000000..02f8760c5
--- /dev/null
+++ b/doc/guides/howto/img/dtg_rx_rate.svg
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="116" height="152" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>rx rate</title>
+ <rect fill="#fff" id="canvas_background" height="154" width="118" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="82" width="78" y="0.812613" x="0.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_2" height="82" width="78" y="19.812613" x="19.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_3" height="82" width="78" y="42.812613" x="36.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="96.812613" x="59.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="144.812613" x="9.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core0</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_7" y="214.621694" x="210.788594" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.36155684245002745,0,0,0.9358925757041695,-60.867161096103985,-178.0100101290174) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_8" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="42.29845" x="-417.420836" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_rx_tx_drop.svg b/doc/guides/howto/img/dtg_rx_tx_drop.svg
new file mode 100644
index 000000000..ca1484862
--- /dev/null
+++ b/doc/guides/howto/img/dtg_rx_tx_drop.svg
@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="275" height="154" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>rx-tx drops</title>
+ <rect fill="#fff" id="canvas_background" height="156" width="277" y="-1" x="-1"/>
+ <g display="none" overflow="visible" y="0" x="0" height="100%" width="100%" id="canvasGrid">
+ <rect fill="url(#gridpattern)" stroke-width="0" y="0" x="0" height="100%" width="100%"/>
+ </g>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect id="svg_1" height="82" width="78" y="0.808165" x="0.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_2" height="82" width="78" y="19.808165" x="19.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_3" height="82" width="78" y="42.808165" x="36.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_4" y="96.808165" x="59.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_5" y="144.808165" x="9.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core0</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_7" y="214.616159" x="210.788594" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.3615568424500274,0,0,0.9358925757041695,-60.867161096103985,-178.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_8" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_9" y="42.292073" x="-417.420836" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ <rect id="svg_10" height="82" width="78" y="2.808165" x="159.75" stroke-width="1.5" stroke="#000" fill="#CC7C65"/>
+ <rect id="svg_11" height="82" width="78" y="21.808165" x="178.75" stroke-width="1.5" stroke="#000" fill="#CE7975"/>
+ <rect id="svg_12" height="82" width="78" y="44.808165" x="195.75" stroke-width="1.5" stroke="#000" fill="#8B4D4B"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_13" y="98.808165" x="218.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_14" y="146.808165" x="168.75" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">Core1</text>
+ <text stroke="#000" transform="matrix(0.47219348024375296,0,0,0.8034176367619965,-95.87709913037096,-156.57412588945806) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_15" y="217.105524" x="547.514971" stroke-opacity="null" stroke-width="0" fill="#000000">NIC1</text>
+ <text stroke="#000" transform="matrix(0.36155684245002734,0,0,0.9358925757041695,98.13283890389599,-176.01445723901463) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_16" y="231.580355" x="235.672461" stroke-opacity="null" stroke-width="0" fill="#000000">NIC2</text>
+ <text stroke="#000" transform="matrix(0.4371257876803068,0,0,0.6973431388181265,223.6990186363727,31.828115819425847) " font-style="italic" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_17" y="45.160101" x="-53.681142" stroke-opacity="null" stroke-width="0" fill="#000000">NIC3</text>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_sample_app_model.svg b/doc/guides/howto/img/dtg_sample_app_model.svg
new file mode 100644
index 000000000..b876aa3b6
--- /dev/null
+++ b/doc/guides/howto/img/dtg_sample_app_model.svg
@@ -0,0 +1,110 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="1418" height="379" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>sample application model</title>
+ <rect fill="#fff" id="canvas_background" height="381" width="1420" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_84" height="378.999996" width="1417.999937" y="0" x="0" stroke-opacity="null" stroke-width="0" fill="#B1FFFF"/>
+ <rect stroke="#000" id="svg_1" height="79.999993" width="187.000011" y="267" x="635.499996" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <rect stroke="#000" id="svg_14" height="28.000001" width="175.000003" y="271.437495" x="641" stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_16" height="36.000002" width="142" y="304.437495" x="659" stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(0.7497134942573729,0,0,0.7607963681117937,149.70768863149087,72.0554119542491) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="335.740664" x="684.214296"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Health Check</text>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="368.437495" x="685"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 7</text>
+ <text stroke="#000" transform="matrix(0.7446371555386645,0,0,0.7004599746900311,157.05848471617847,107.2548065316271) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="261.052167" x="658.840014"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Stats Collector</text>
+ <rect id="svg_26" height="91" width="88" y="89.4375" x="2"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFA9A2"/>
+ <rect id="svg_27" height="91" width="88" y="107.4375" x="24"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CE7975"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_28" y="166.4375" x="52"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">RX</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6666666865348816,14.972222477197647,39.14583100005984) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_30" y="96.937501" x="-11.409091"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 1</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6666666865348816,14.972222477197647,39.14583100005984) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_33" y="128.4375" x="27.863636"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 2</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_34" y="216.4375" x="26"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core0</text>
+ <rect id="svg_35" height="91" width="88" y="55.4375" x="1303"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#FFA9A2"/>
+ <rect id="svg_36" height="91" width="88" y="73.4375" x="1325"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#CE7975"/>
+ <text style="cursor: move;" xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_37" y="132.4375" x="1352"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">TX</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_40" y="184.4375" x="1327"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core1</text>
+ <text stroke="#000" transform="matrix(0.6111111044883728,0,0,0.6296296119689941,200.86111453175545,24.236112266778946) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_43" y="74.319853" x="1811.681832"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 1</text>
+ <text stroke="#000" transform="matrix(0.6958672408102909,0,0,0.6184720487972513,537.7539486343405,-28.040291137315034) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_44" y="191.620936" x="1140.58332"
+ stroke-opacity="null" stroke-width="0" fill="#000000">NIC 2</text>
+ <rect stroke="#000" id="svg_45" height="62" width="83.000003" y="101.4375" x="1096"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_46" y="135.4375" x="1115"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">QoS</text>
+ <rect stroke="#000" id="svg_48" height="61" width="107" y="110.4375" x="649"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_49" y="145.4375" x="661"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#ffffff">Crypto</text>
+ <rect stroke="#000" id="svg_50" height="70.999998" width="84.000002" y="65.4375" x="422"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <rect stroke="#000" id="svg_51" height="74.000003" width="90.000002" y="92.4375" x="444"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_52" height="71.999998" width="92" y="120.4375" x="468"
+ stroke-opacity="null" stroke-width="0" fill="#C48157"/>
+ <text stroke="#000" transform="matrix(0.6528342962265015,0,0,0.5925925970077515,63.70490664243698,168.43749817460775) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_53" y="-145.374996" x="553.425418"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.6666666865348816,166.69894686341286,-165.52084343507886) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_54" y="417.437503" x="420.257696" stroke-opacity="null" stroke-width="0" fill="#000000">Worker 2</text>
+ <text stroke="#000" transform="matrix(0.7483048439025879,0,0,0.7407407164573669,76.38947987556458,153.33566251024604) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_55" y="-19.012521" x="530.011964"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 3</text>
+ <rect stroke="#000" id="svg_62" height="70.999998" width="84.000002" y="67.4375" x="865"
+ stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <rect stroke="#000" id="svg_63" height="74.000003" width="90.000002" y="94.4375" x="887"
+ stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_64" height="71.999998" width="92" y="122.4375" x="911"
+ stroke-opacity="null" stroke-width="0" fill="#C48157"/>
+ <text stroke="#000" transform="matrix(0.6528342962265015,0,0,0.5925925970077515,63.70490664243698,168.43749817460775) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_65" y="-143.687496" x="1238.132093"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 1</text>
+ <text stroke="#000" transform="matrix(0.678871691226959,0,0,0.6666666865348816,166.69894686341286,-165.52084343507886) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_66" y="420.437503" x="1072.811052"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 2</text>
+ <text stroke="#000" transform="matrix(0.7483048439025879,0,0,0.7407407164573669,76.38947987556458,153.33566251024604) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_67" y="-16.312521" x="1122.016685"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Worker 3</text>
+ <path stroke="#000" id="svg_68" d="m653.747516,12.862181l19.36745,-10.781877l77.632553,0l0,53.999988l-97.000004,0l0,-43.218111z"
+ stroke-opacity="null" stroke-width="0" fill="#805064"/>
+ <text stroke="#000" transform="matrix(0.733815550804138,0,0,0.9629629850387573,306.63544338941574,-1.3912058547139168) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_69" y="33.322117" x="496.534253"
+ stroke-opacity="null" stroke-width="0" fill="#ffffff">Device</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_70" y="217.4375" x="457"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_71" y="197.4375" x="653"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 5</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_72" y="186.4375" x="1097"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 6</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_73" y="220.4375" x="885"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 2,3,4</text>
+ <rect stroke="#000" id="svg_74" height="88.000002" width="110.999999" y="104" x="206.000001"
+ stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <text stroke="#000" transform="matrix(0.5661651903991256,0,0,1,107.56695064249905,0) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_75" y="133" x="182.901935"
+ stroke-opacity="null" stroke-width="0" fill="#000000">PKT classify</text>
+ <text stroke="#000" transform="matrix(0.5444634556770325,0,0,1,115.93405053019524,0) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_76" y="170" x="196.644877"
+ stroke-opacity="null" stroke-width="0" fill="#000000">Distribute</text>
+ <path id="svg_77" d="m189.755816,142.38346l-9.615746,-5.572568l0,6.200627l-17.256831,0c-3.021555,0.226504 -8.238368,-3.212549 -14.038725,-7.775916c-5.840892,-4.328677 -12.516173,-9.639678 -21.454022,-9.781769l-6.013302,0l0,9.884674l6.013302,0c4.377316,-0.12767 9.774825,3.325816 15.521285,7.782201c2.14196,1.556776 4.3024,3.274261 6.59165,4.824928c-2.28925,1.552703 -4.447123,3.272244 -6.59165,4.829053c-5.746457,4.44398 -11.143969,7.905694 -15.51563,7.780055l-6.018955,0l0,9.899092l6.015865,0c8.91886,-0.125542 15.596722,-5.450935 21.456596,-9.773453c5.800879,-4.563453 11.01514,-8.004512 14.039247,-7.782119l17.251176,0l0,6.200569l9.615746,-5.566299l9.621368,-5.570396l-9.621368,-5.578676l-0.000005,-0.000003z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path id="svg_78" d="m417.79301,181.072084l13.604178,-5.205736l-13.604178,-5.203742l-13.60709,-5.201786l0,5.786911c-3.569394,-0.003899 -9.48466,-0.003899 -11.161519,-0.003899c-4.954616,-0.078306 -8.206513,-1.105708 -11.590595,-2.945338c-5.043959,-2.745716 -9.348773,-7.687237 -14.54043,-12.448672c-2.640478,-2.367977 -5.58224,-4.716439 -9.50016,-6.604946c-3.878767,-1.896386 -8.946375,-3.211512 -14.579566,-3.186066l-24.41645,0l0,9.245023l24.41645,0c3.395131,0.041097 5.543836,0.774963 8.474533,2.595032c4.335842,2.706583 8.563147,7.88095 14.393505,12.783335c5.71366,4.855376 14.505736,9.757754 27.25778,9.804723l11.246452,0l0,5.786895l13.60709,-5.201732zm-57.554046,-35.013272c0.15434,0.072423 0.317546,0.135019 0.468152,0.209421c4.65338,2.242732 8.013781,4.994346 10.673477,7.387791c0.611391,0.559679 1.153382,1.097877 1.733756,1.647811l31.071571,0l0,5.788888l13.60709,-5.201813l13.604178,-5.203713l-13.604178,-5.205701l-13.60709,-5.207671l0,5.784986l-43.946955,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path stroke="#000" transform="rotate(89.46277618408203 605.7476806640625,144.42587280273435) " id="svg_79" d="m590.864808,144.339395l14.882874,-35.951591l14.882874,35.951591l-7.441438,0l0,36.124583l-14.882873,0l0,-36.124583l-7.441438,0z" stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ <path stroke="#000" transform="rotate(89.74066925048828 697.6382446289062,81.67762756347659) " id="svg_80" d="m661.638136,81.677665l20.874103,-13.000079l0,6.500022l30.252035,0l0,-6.500022l20.874093,13.000079l-20.874093,13.000009l0,-6.500005l-30.252035,0l0,6.500005l-20.874103,-13.000009z" fill-opacity="null"
+ stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ <path id="svg_81" d="m858.79301,172.072084l13.604178,-5.205736l-13.604178,-5.203742l-13.60709,-5.201786l0,5.786911c-3.569394,-0.0039 -9.48466,-0.0039 -11.161519,-0.0039c-4.954616,-0.078305 -8.206513,-1.105708 -11.590595,-2.945338c-5.043959,-2.745715 -9.348773,-7.687236 -14.54043,-12.448672c-2.640477,-2.367977 -5.58224,-4.716438 -9.50016,-6.604945c-3.878767,-1.896387 -8.946375,-3.211513 -14.579566,-3.186067l-24.416451,0l0,9.245023l24.416451,0c3.395131,0.041097 5.543836,0.774963 8.474533,2.595032c4.335842,2.706583 8.563147,7.88095 14.393505,12.783335c5.713661,4.855376 14.505736,9.757754 27.257781,9.804723l11.246451,0l0,5.786894l13.60709,-5.201732zm-57.554045,-35.013272c0.15434,0.072424 0.317546,0.135019 0.468152,0.209421c4.65338,2.242731 8.013781,4.994346 10.673477,7.38779c0.61139,0.559679 1.153381,1.097878 1.733756,1.647812l31.07157,0l0,5.788888l13.60709,-5.201812l13.604178,-5.203713l-13.604178,-5.205701l-13.60709,-5.207671l0,5.784986l-43.946955,0z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path id="svg_82" d="m1080.755816,127.38346l-9.615746,-5.572567l0,6.200627l-17.256831,0c-3.021555,0.226504 -8.238368,-3.212549 -14.038725,-7.775916c-5.840892,-4.328677 -12.516173,-9.639678 -21.454022,-9.78177l-6.013302,0l0,9.884674l6.013302,0c4.377316,-0.12767 9.774825,3.325816 15.521285,7.782201c2.141959,1.556776 4.3024,3.274261 6.59165,4.824928c-2.28925,1.552704 -4.447123,3.272244 -6.59165,4.829053c-5.746457,4.443981 -11.143969,7.905694 -15.51563,7.780055l-6.018955,0l0,9.899093l6.015865,0c8.91886,-0.125543 15.596723,-5.450935 21.456596,-9.773454c5.800879,-4.563453 11.01514,-8.004512 14.039246,-7.782119l17.251177,0l0,6.20057l9.615746,-5.566299l9.621368,-5.570397l-9.621368,-5.578676l-0.000006,-0.000003z"
+ stroke-opacity="null" stroke-width="0" stroke="#000" fill="#619E73"/>
+ <path stroke="#000" id="svg_83" d="m1269.963506,163.318859l13.567973,-4.653717l13.56088,-4.655631l-13.568064,-4.658919l-13.560793,-4.655342l0,5.177292c-3.323905,0 -7.186364,0 -8.480362,0c-6.169502,0.103377 -13.784287,-2.785121 -21.885448,-6.508723c-3.019981,-1.3056 -6.066761,-2.741979 -9.295919,-4.03897c3.229159,-1.298581 6.271574,-2.733309 9.295919,-4.038867c8.101162,-3.718456 15.715942,-6.612144 21.877492,-6.506938l8.488314,-0.001741l0,5.18423l13.560793,-4.6572l13.568064,-4.660782l-13.568064,-4.665774l-13.560793,-4.658938l0,5.180918l-8.483954,0c-12.6011,0.104929 -22.018599,4.557333 -30.259422,8.175926c-8.180795,3.81841 -15.53428,6.693008 -19.799052,6.506997l-24.328718,0l0,8.27074l24.336702,0c4.261113,-0.187837 11.618288,2.688631 19.798299,6.506933c8.237215,3.620425 17.651098,8.069132 30.25579,8.175987l8.480362,0l0,5.182516l0,0l0,0.000002z" fill-opacity="null"
+ stroke-opacity="null" stroke-width="0" fill="#619E73"/>
+ </g>
+</svg>
diff --git a/doc/guides/howto/img/dtg_service.svg b/doc/guides/howto/img/dtg_service.svg
new file mode 100644
index 000000000..fa72de823
--- /dev/null
+++ b/doc/guides/howto/img/dtg_service.svg
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Method Draw (https://editor.method.ac/) -->
+<!-- SPDX-License-Identifier: BSD-3-Clause -->
+<!-- Copyright(c) 2018-2019 Intel Corporation -->
+
+<svg width="254" height="145" xmlns="http://www.w3.org/2000/svg">
+ <g>
+ <title>service</title>
+ <rect fill="#fff" id="canvas_background" height="147" width="256" y="-1" x="-1"/>
+ </g>
+ <g>
+ <title>Layer 1</title>
+ <rect stroke="#000" id="svg_1" height="109.999987" width="254.000014" y="0" x="0" stroke-opacity="null" stroke-width="0" fill="#9ACEE6"/>
+ <rect stroke="#000" id="svg_14" height="37.000001" width="225" y="7.437494" x="4.5" stroke-opacity="null" stroke-width="0" fill="#FFB27C"/>
+ <rect stroke="#000" id="svg_16" height="45.000002" width="186.000003" y="58.437493" x="33.5" stroke-opacity="null" stroke-width="0" fill="#FFE7A2"/>
+ <text stroke="#000" transform="matrix(1.0013854504218995,0,0,1.0926463039877063,73.76856116958965,-122.08861649089796) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_18" y="193.234194" x="-35.792216" stroke-opacity="null" stroke-width="0" fill="#000000">Health Check</text>
+ <text xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_23" y="138.437495" x="45.5" stroke-opacity="null" stroke-width="0" stroke="#000" fill="#000000">core 6</text>
+ <text stroke="#000" transform="matrix(1.0013854504218995,0,0,1.0926463039877063,73.76856116958965,-122.08861649089796) " xml:space="preserve" text-anchor="start" font-family="'Courier New', Courier, monospace" font-size="24" id="svg_25" y="141.982476" x="-63.753477" stroke-opacity="null" stroke-width="0" fill="#000000">Stats Collector</text>
+ </g>
+</svg>
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* [dpdk-dev] [PATCH v6 2/2] doc: add guide for debug and troubleshoot
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 0/2] guide to debug and troubleshoot Vipin Varghese
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 1/2] doc: add svg for debug and troubleshoot guide Vipin Varghese
@ 2019-02-19 15:57 ` Vipin Varghese
2019-02-25 11:34 ` Mcnamara, John
1 sibling, 1 reply; 23+ messages in thread
From: Vipin Varghese @ 2019-02-19 15:57 UTC (permalink / raw)
To: john.mcnamara, marko.kovacevic, thomas, ferruh.yigit,
shreyansh.jain, dev
Cc: sanjay.padubidri, amol.patel, Vipin Varghese
Add user guide on debugging and troubleshooting for common
issues and bottleneck found in the sample application model.
Signed-off-by: Vipin Varghese <vipin.varghese@intel.com>
Acked-by: Marko Kovacevic <marko.kovacevic@intel.com>
---
doc/guides/howto/debug_troubleshoot_guide.rst | 451 ++++++++++++++++++
doc/guides/howto/index.rst | 1 +
2 files changed, 452 insertions(+)
create mode 100644 doc/guides/howto/debug_troubleshoot_guide.rst
diff --git a/doc/guides/howto/debug_troubleshoot_guide.rst b/doc/guides/howto/debug_troubleshoot_guide.rst
new file mode 100644
index 000000000..89310121f
--- /dev/null
+++ b/doc/guides/howto/debug_troubleshoot_guide.rst
@@ -0,0 +1,451 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2018 Intel Corporation.
+
+.. _debug_troubleshoot_guide:
+
+Debug & Troubleshoot guide
+==========================
+
+DPDK applications can be designed to have simple or complex pipeline processing
+stages making use of single or multiple threads. Applications can use poll mode
+hardware devices which helps in offloading CPU cycles too. It is common to find
+solutions designed with
+
+* single or multiple primary processes
+
+* single primary and single secondary
+
+* single primary and multiple secondaries
+
+In all the above cases, it is tedious to isolate, debug, and understand various
+behaviors which occur randomly or periodically. The goal of the guide is to
+consolidate a few commonly seen issues for reference. Then, isolate to identify
+the root cause through step by step debug at various stages.
+
+..note::
+
+ It is difficult to cover all possible issues; in a single attempt. With
+ feedback and suggestions from the community, more cases can be covered.
+
+
+Application Overview
+--------------------
+
+By making use of the application model as a reference, we can discuss multiple
+causes of issues in the guide. Let us assume the sample makes use of a single
+primary process, with various processing stages running on multiple cores. The
+application may also make uses of Poll Mode Driver, and libraries like service
+cores, mempool, mbuf, eventdev, cryptodev, QoS, and ethdev.
+
+The overview of an application modeled using PMD is shown in
+:numref:`dtg_sample_app_model`.
+
+.. _dtg_sample_app_model:
+
+.. figure:: img/dtg_sample_app_model.*
+
+ Overview of pipeline stage of an application
+
+
+Bottleneck Analysis
+-------------------
+
+A couple of factors that lead the design decision could be the platform, scale
+factor, and target. This distinct preference leads to multiple combinations,
+that are built using PMD and libraries of DPDK. While the compiler, library
+mode, and optimization flags are the components are to be constant, that
+affects the application too.
+
+
+Is there mismatch in packet (received < desired) rate?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RX Port and associated core :numref:`dtg_rx_rate`.
+
+.. _dtg_rx_rate:
+
+.. figure:: img/dtg_rx_rate.*
+
+ RX packet rate compared against received rate.
+
+#. Is the configuration for the RX setup correctly?
+
+ - Identify if port Speed and Duplex is matching to desired values with
+ rte_eth_link_get.
+
+ - Check MTU value is set to the expected packet size to receive with
+ rte_eth_get_mtu if there are large packet drops.
+
+ - Check promiscuous mode if the drops do not occur for unique MAC address
+ with rte_eth_promiscuous_get.
+
+#. Is the drop isolated to certain NIC only?
+
+ - Make use of rte_eth_dev_stats to identify the drops cause.
+
+ - If there are mbuf drops, check nb_desc for RX descriptor as it might not
+ be sufficient for the application.
+
+ - If rte_eth_dev_stats shows drops are on specific RX queues, ensure RX
+ lcore threads has enough cycles for rte_eth_rx_burst on the port queue
+ pair.
+
+ - If there are redirect to a specific port queue pair with, ensure RX lcore
+ threads gets enough cycles.
+
+ - Check the RSS configuration rte_eth_dev_rss_hash_conf_get if the spread is
+ not even and causing drops.
+
+ - If PMD stats are not updating, then there might be offload or configuration
+ which is dropping the incoming traffic.
+
+#. Is there drops still seen?
+
+ - If there are multiple port queue pair, it might be the RX thread, RX
+ distributor, or event RX adapter not having enough cycles.
+
+ - If there are drops seen for RX adapter or RX distributor, try using
+ rte_prefetch_non_temporal() which intimates the core that the mbuf in the
+ cache is temporary.
+
+Is there packet drops at receive or transmit?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RX-TX port and associated cores :numref:`dtg_rx_tx_drop`.
+
+.. _dtg_rx_tx_drop:
+
+.. figure:: img/dtg_rx_tx_drop.*
+
+ RX-TX drops
+
+#. At RX
+
+ - Identify if there are multiple RX queue configured for port by nb_rx_queues
+ using rte_eth_dev_info_get.
+
+ - Using rte_eth_dev_stats shows drops in q_errors, check if RX thread is
+ configured to fetch packets from the port queue pair.
+
+ - Using rte_eth_dev_stats shows drops in rx_nombuf, check if RX thread has
+ enough cycles to consume the packets from the queue.
+
+#. At TX
+
+ - If the TX rate is falling behind the application fill rate, identify if
+ there are enough descriptors with rte_eth_dev_info_get for TX.
+
+ - Check the nb_pkt in rte_eth_tx_burst is done for multiple packets.
+
+ - Check rte_eth_tx_burst invokes the vector function call for the PMD.
+
+ - If oerrors are getting incremented, TX packet validations are failing.
+ Check if there queue specific offload failures.
+
+ - If the drops occur for large size packets, check MTU and multi-segment
+ support configured for NIC.
+
+Is there object drops in producer point for the ring library?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Producer point for ring :numref:`dtg_producer_ring`.
+
+.. _dtg_producer_ring:
+
+.. figure:: img/dtg_producer_ring.*
+
+ Producer point for Rings
+
+#. Performance issue isolation at producer
+
+ - Use rte_ring_dump to validate for all single producer flag is set to
+ RING_F_SP_ENQ.
+
+ - There should be sufficient rte_ring_free_count at any point in time.
+
+ - Extreme stalls in dequeue stage of the pipeline will cause rte_ring_full
+ to be true.
+
+Is there object drops in consumer point for the ring library?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Consumer point for ring :numref:`dtg_consumer_ring`.
+
+.. _dtg_consumer_ring:
+
+.. figure:: img/dtg_consumer_ring.*
+
+ Consumer point for Rings
+
+#. Performance issue isolation at consumer
+
+ - Use rte_ring_dump to validate for all single consumer flag is set to
+ RING_F_SC_DEQ.
+
+ - If the desired burst dequeue falls behind the actual dequeue, the enqueue
+ stage is not filling up the ring as required.
+
+ - Extreme stall in the enqueue will lead to rte_ring_empty to be true.
+
+Is there a variance in packet or object processing rate in the pipeline?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Memory objects close to NUMA :numref:`dtg_mempool`.
+
+.. _dtg_mempool:
+
+.. figure:: img/dtg_mempool.*
+
+ Memory objects have to be close to the device per NUMA.
+
+#. Stall in processing pipeline can be attributes of MBUF release delays.
+ These can be narrowed down to
+
+ - Heavy processing cycles at single or multiple processing stages.
+
+ - Cache is spread due to the increased stages in the pipeline.
+
+ - CPU thread responsible for TX is not able to keep up with the burst of
+ traffic.
+
+ - Extra cycles to linearize multi-segment buffer and software offload like
+ checksum, TSO, and VLAN strip.
+
+ - Packet buffer copy in fast path also results in stalls in MBUF release if
+ not done selectively.
+
+ - Application logic sets rte_pktmbuf_refcnt_set to higher than the desired
+ value and frequently uses rte_pktmbuf_prefree_seg and does not release
+ MBUF back to mempool.
+
+#. Lower performance between the pipeline processing stages can be
+
+ - The NUMA instance for packets or objects from NIC, mempool, and ring
+ should be the same.
+
+ - Drops on a specific socket are due to insufficient objects in the pool.
+ Use rte_mempool_get_count or rte_mempool_avail_count to monitor when
+ drops occurs.
+
+ - Try prefetching the content in processing pipeline logic to minimize the
+ stalls.
+
+#. Performance issue can be due to special cases
+
+ - Check if MBUF continuous with rte_pktmbuf_is_contiguous as certain offload
+ requires the same.
+
+ - Use rte_mempool_cache_create for user threads require access to mempool
+ objects.
+
+ - If the variance is absent for larger huge pages, then try rte_mem_lock_page
+ on the objects, packets, lookup tables to isolate the issue.
+
+Is there a variance in cryptodev performance?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Crypto device and PMD :numref:`dtg_crypto`.
+
+.. _dtg_crypto:
+
+.. figure:: img/dtg_crypto.*
+
+ CRYPTO and interaction with PMD device.
+
+#. Performance issue isolation for enqueue
+
+ - Ensure cryptodev, resources and enqueue is running on NUMA cores.
+
+ - Isolate if the cause of errors for err_count using rte_cryptodev_stats.
+
+ - Parallelize enqueue thread for varied multiple queue pair.
+
+#. Performance issue isolation for dequeue
+
+ - Ensure cryptodev, resources and dequeue are running on NUMA cores.
+
+ - Isolate if the cause of errors for err_count using rte_cryptodev_stats.
+
+ - Parallelize dequeue thread for varied multiple queue pair.
+
+#. Performance issue isolation for crypto operation
+
+ - If the cryptodev software-assist is in use, ensure the library is built
+ with right (SIMD) flags or check if the queue pair using CPU ISA for
+ feature_flags AVX|SSE|NEON using rte_cryptodev_info_get.
+
+ - If the cryptodev hardware-assist is in use, ensure both firmware and
+ drivers are up to date.
+
+#. Configuration issue isolation
+
+ - Identify cryptodev instances with rte_cryptodev_count and
+ rte_cryptodev_info_get.
+
+Is user functions performance is not as expected?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Custom worker function :numref:`dtg_distributor_worker`.
+
+.. _dtg_distributor_worker:
+
+.. figure:: img/dtg_distributor_worker.*
+
+ Custom worker function performance drops.
+
+#. Performance issue isolation
+
+ - The functions running on CPU cores without context switches are the
+ performing scenarios. Identify lcore with rte_lcore and lcore index
+ mapping with CPU using rte_lcore_index.
+
+ - The functions running on CPU cores without context switches are the
+ performing scenarios. Identify lcore with rte_lcore and lcore index
+ mapping with CPU using rte_lcore_index.
+
+ - Use rte_thread_get_affinity to isolate functions running on the same
+ CPU core.
+
+#. Configuration issue isolation
+
+ - Identify core role using rte_eal_lcore_role to identify RTE, OFF and
+ SERVICE. Check performance functions are mapped to run on the cores.
+
+ - For high-performance execution logic ensure running it on correct NUMA
+ and non-master core.
+
+ - Analyze run logic with rte_dump_stack, rte_dump_registers and rte_memdump
+ for more insights.
+
+ - Make use of 'objdump' to ensure opcode is matching to the desired state.
+
+Is the execution cycles for dynamic service functions are not frequent?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+service functions on service cores :numref:`dtg_service`.
+
+.. _dtg_service:
+
+.. figure:: img/dtg_service.*
+
+ functions running on service cores
+
+#. Performance issue isolation
+
+ - Services configured for parallel execution should have
+ rte_service_lcore_count should be equal to
+ rte_service_lcore_count_services.
+
+ - A service to run parallel on all cores should return
+ RTE_SERVICE_CAP_MT_SAFE for rte_service_probe_capability and
+ rte_service_map_lcore_get returns unique lcore.
+
+ - If service function execution cycles for dynamic service functions are
+ not frequent?
+
+ - If services share the same lcore should run in the overall CPU execution
+ cycle.
+
+#. Configuration issue isolation
+ - Check if service is running with rte_service_runstate_get.
+ - Generic debug via rte_service_dump.
+
+Is there a bottleneck in the performance of eventdev?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. Check for generic configuration
+
+ - Ensure the event devices created are right NUMA using rte_event_dev_count
+ and rte_event_dev_socket_id.
+
+ - Check for event stages if the events are looped back into the same queue.
+
+ - If the failure is on the enqueue stage for events, check if queue depth
+ with rte_event_dev_info_get.
+
+#. If there are performance drops in the enqueue stage
+
+ - Use rte_event_dev_dump to dump the eventdev information.
+
+ - Periodically checks stats for queue and port to identify the starvation.
+
+ - Check the in-flight events for the desired queue for enqueue and dequeue.
+
+Is there a variance in traffic manager?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Traffic Manager on TX interface :numref:`dtg_qos_tx`.
+
+.. _dtg_qos_tx:
+
+.. figure:: img/dtg_qos_tx.*
+
+ Traffic Manager just before TX.
+
+#. Identify the cause for a variance from expected behavior, is due to
+ insufficient CPU cycles. Use rte_tm_capabilities_get to fetch features for
+ hierarchies, WRED and priority schedulers to be offloaded hardware.
+
+#. Undesired flow drops can be narrowed down to WRED, priority, and rates
+ limiters.
+
+#. Isolate the flow in which the undesired drops occur. Use
+ rte_tn_get_number_of_leaf_node and flow table to ping down the leaf where
+ drops occur.
+
+#. Check the stats using rte_tm_stats_update and rte_tm_node_stats_read for
+ drops for hierarchy, schedulers and WRED configurations.
+
+Is the packet not in the unexpected format?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Packet capture before and after processing :numref:`dtg_pdump`.
+
+.. _dtg_pdump:
+
+.. figure:: img/dtg_pdump.*
+
+ Capture points of Traffic at RX-TX.
+
+#. To isolate the possible packet corruption in the processing pipeline,
+ carefully staged capture packets are to be implemented.
+
+ - First, isolate at NIC entry and exit. Use pdump in primary to allow
+ secondary to access port-queue pair. The packets get copied over in RX|TX
+ callback by the secondary process using ring buffers.
+
+ - Second, isolate at pipeline entry and exit. Using hooks or callbacks
+ capture the packet middle of the pipeline stage to copy the packets, which
+ can be shared to the secondary debug process via user-defined custom rings.
+
+..note::
+
+ Use similar analysis to objects and metadata corruption.
+
+Does the issue still persist?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The cause of the issue can be further narrowed down to the following causes.
+
+#. If there are vendor or application specific metadata, check for errors due
+ to META data error flags. Dumping private meta-data in the objects can give
+ insight into details for debugging.
+
+#. If there are multi-process for either data or configuration, check for
+ possible errors in the secondary process where the configuration fails and
+ possible data corruption in the data plane.
+
+#. Random drops in the RX or TX when opening other application is an indication
+ of the effect of a noisy neighbor. Try using the cache allocation technique
+ to minimize the effect between applications.
+
+
+How to develop a custom code to debug?
+-------------------------------------
+
+#. For an application that runs as the primary process only, debug functionality
+ is added in the same process. These can be invoked by timer call-back,
+ service core and signal handler.
+
+#. For the application that runs as multiple processes. debug functionality in
+ a standalone secondary process.
diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a642a2be1..9527fa84d 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -18,3 +18,4 @@ HowTo Guides
virtio_user_as_exceptional_path
packet_capture_framework
telemetry
+ debug_troubleshoot_guide
--
2.17.1
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v6 2/2] doc: add guide for debug and troubleshoot
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
@ 2019-02-25 11:34 ` Mcnamara, John
2019-02-25 15:46 ` Varghese, Vipin
0 siblings, 1 reply; 23+ messages in thread
From: Mcnamara, John @ 2019-02-25 11:34 UTC (permalink / raw)
To: Varghese, Vipin, Kovacevic, Marko, thomas, Yigit, Ferruh,
shreyansh.jain, dev
Cc: Padubidri, Sanjay A, Patel, Amol
> -----Original Message-----
> From: Varghese, Vipin
> Sent: Tuesday, February 19, 2019 3:57 PM
> To: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; thomas@monjalon.net; Yigit, Ferruh
> <ferruh.yigit@intel.com>; shreyansh.jain@nxp.com; dev@dpdk.org
> Cc: Padubidri, Sanjay A <sanjay.padubidri@intel.com>; Patel, Amol
> <amol.patel@intel.com>; Varghese, Vipin <vipin.varghese@intel.com>
> Subject: [PATCH v6 2/2] doc: add guide for debug and troubleshoot
>
> Add user guide on debugging and troubleshooting for common issues and
> bottleneck found in the sample application model.
>
Hi,
Thanks for this. It is a good How-to doc and also a FAQ from customers.
A couple of minor comments.
There is a warning due to one of the header underlines not being long
enough:
How to develop a custom code to debug?
-------------------------------------
/work/dpdk_docs/doc/guides/howto/debug_troubleshoot_guide.rst:444:
WARNING: Title underline too short.
Also, in 2 places the ".. note::" directive is used without a space
after .., like below:
> +
> +..note::
> +
> + It is difficult to cover all possible issues; in a single attempt.
Also, for the second note you may wish to indent it to the same level as
the previous paragraph.
But overall it looks good.
John
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [dpdk-dev] [PATCH v6 2/2] doc: add guide for debug and troubleshoot
2019-02-25 11:34 ` Mcnamara, John
@ 2019-02-25 15:46 ` Varghese, Vipin
0 siblings, 0 replies; 23+ messages in thread
From: Varghese, Vipin @ 2019-02-25 15:46 UTC (permalink / raw)
To: Mcnamara, John, Kovacevic, Marko, thomas, Yigit, Ferruh,
shreyansh.jain, dev
Cc: Padubidri, Sanjay A, Patel, Amol
Thanks John, I will look into the same ASAP.
> -----Original Message-----
> From: Mcnamara, John
> Sent: Monday, February 25, 2019 5:05 PM
> To: Varghese, Vipin <vipin.varghese@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; thomas@monjalon.net; Yigit, Ferruh
> <ferruh.yigit@intel.com>; shreyansh.jain@nxp.com; dev@dpdk.org
> Cc: Padubidri, Sanjay A <sanjay.padubidri@intel.com>; Patel, Amol
> <amol.patel@intel.com>
> Subject: RE: [PATCH v6 2/2] doc: add guide for debug and troubleshoot
>
>
>
> > -----Original Message-----
> > From: Varghese, Vipin
> > Sent: Tuesday, February 19, 2019 3:57 PM
> > To: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> > <marko.kovacevic@intel.com>; thomas@monjalon.net; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; shreyansh.jain@nxp.com; dev@dpdk.org
> > Cc: Padubidri, Sanjay A <sanjay.padubidri@intel.com>; Patel, Amol
> > <amol.patel@intel.com>; Varghese, Vipin <vipin.varghese@intel.com>
> > Subject: [PATCH v6 2/2] doc: add guide for debug and troubleshoot
> >
> > Add user guide on debugging and troubleshooting for common issues and
> > bottleneck found in the sample application model.
> >
>
> Hi,
>
> Thanks for this. It is a good How-to doc and also a FAQ from customers.
>
> A couple of minor comments.
>
>
> There is a warning due to one of the header underlines not being long
> enough:
>
> How to develop a custom code to debug?
> -------------------------------------
> /work/dpdk_docs/doc/guides/howto/debug_troubleshoot_guide.rst:444:
> WARNING: Title underline too short.
>
> Also, in 2 places the ".. note::" directive is used without a space after .., like
> below:
>
> > +
> > +..note::
> > +
> > + It is difficult to cover all possible issues; in a single attempt.
>
> Also, for the second note you may wish to indent it to the same level as the
> previous paragraph.
>
> But overall it looks good.
>
> John
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2019-02-25 15:46 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-26 7:08 [dpdk-dev] [PATCH v3 1/2] doc: add svg for debug and troubleshoot guide Vipin Varghese
2018-11-26 7:08 ` [dpdk-dev] [PATCH v3 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
2019-01-04 6:37 ` Shreyansh Jain
2019-01-04 7:01 ` Varghese, Vipin
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 0/2] doci/howto: add debug and troubleshoot guide Vipin Varghese
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 1/2] doc: add svg for " Vipin Varghese
2019-01-16 14:54 ` [dpdk-dev] [PATCH v4 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
2019-01-18 15:28 ` Kovacevic, Marko
2019-01-21 3:38 ` Varghese, Vipin
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 0/2] doc/howto: add debug and troubleshoot guide Vipin Varghese
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 1/2] doc: add svg for " Vipin Varghese
2019-01-28 1:08 ` Thomas Monjalon
2019-01-28 14:28 ` Varghese, Vipin
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 0/2] guide to debug and troubleshoot Vipin Varghese
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 1/2] doc: add svg for debug and troubleshoot guide Vipin Varghese
2019-02-19 15:57 ` [dpdk-dev] [PATCH v6 2/2] doc: add guide for debug and troubleshoot Vipin Varghese
2019-02-25 11:34 ` Mcnamara, John
2019-02-25 15:46 ` Varghese, Vipin
2019-01-21 10:41 ` [dpdk-dev] [PATCH v5 " Vipin Varghese
2019-01-28 1:30 ` Thomas Monjalon
2019-01-28 14:51 ` Varghese, Vipin
2019-01-28 15:59 ` Thomas Monjalon
2019-02-08 9:21 ` Varghese, Vipin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).