[dts][PATCH V1 1/2] test_plans/vdev_primary_secondary_test_plan: add hotplug testcase2

[dts][PATCH V1 1/2] test_plans/vdev_primary_secondary_test_plan: add hotplug testcase2

[Wei Ling]

Add hotplug testcase2.

Signed-off-by: Wei Ling <weix.ling@intel.com>
---
 .../vdev_primary_secondary_test_plan.rst      | 188 ++++++++----------
 1 file changed, 82 insertions(+), 106 deletions(-)

diff --git a/test_plans/vdev_primary_secondary_test_plan.rst b/test_plans/vdev_primary_secondary_test_plan.rst
index b98d2d04..8258a6d6 100644
--- a/test_plans/vdev_primary_secondary_test_plan.rst
+++ b/test_plans/vdev_primary_secondary_test_plan.rst
@@ -34,172 +34,148 @@
 Virtio-pmd primary/secondary process test plan
 ==============================================
 
-This test plan will test vdev primary/secondary with symmetric_mp, which demonstrates how a set of processes can run in parallel,
-with each process performing the same set of packet processing operations. Also test vdev primary/secondary with hotplug_mp example.
-
-Symmetric MP Application Description
-------------------------------------
-
-This test is a multi-process test which demonstrates how multiple processes can
-work together to perform packet I/O and packet processing in parallel, much as
-other example application work by using multiple threads. In this example, each
-process reads packets from all network ports being used - though from a different
-RX queue in each case. Those packets are then forwarded by each process which
-sends them out by writing them directly to a suitable TX queue.
+Description
+===========
+
+This test plan will test vdev primary/secondary with symmetric_mp,
+which demonstrates how a set of processes can run in parallel,
+with each process performing the same set of packet processing operations,
+each process reads packets from each of the network ports in use.
+Each process reads a different RX queue and writes outgoing packets to a different TX queue on each port.
+Also test vdev primary/secondary with hotplug_mp example.
+The primary process work as a resource management process, it will create/destroy virtual device at runtime,
+while the secondary process can deal with the network stuff with these devices,
+that means attach or detach a device on any process will broadcast to all other processes through mp channel
+then device information will be synchronized on all processes.
+For more about the Symmetric MP Application example,
+please refer to the dpdk docs: http://doc.dpdk.org/guides/sample_app_ug/multi_process.html
 
 Prerequisites
--------------
-
-Assuming that DPDK build has been set up and the multi-process sample
-applications have been built. It is also assumed that a traffic generator has
-been configured and plugged in to the NIC ports 0 and 1.
-
-Test Methodology
-----------------
-
-As with the simple_mp example, the first instance of the symmetric_mp process
-must be run as the primary instance, though with a number of other application
-specific parameters also provided after the EAL arguments. These additional
-parameters are:
+=============
 
-* -p <portmask>, where portmask is a hexadecimal bitmask of what ports on the
-  system are to be used. For example: -p 3 to use ports 0 and 1 only.
-* --num-procs <N>, where N is the total number of symmetric_mp instances that
-  will be run side-by-side to perform packet processing. This parameter is used to
-  configure the appropriate number of receive queues on each network port.
-* --proc-id <n>, where n is a numeric value in the range 0 <= n < N (number of
-  processes, specified above). This identifies which symmetric_mp instance is being
-  run, so that each process can read a unique receive queue on each network port.
+Topology
+--------
+Vhost-->Virtio-tmp
 
-The secondary symmetric_mp instances must also have these parameters specified,
-and the first two must be the same as those passed to the primary instance, or errors
-result.
+Hardware
+--------
+Supportted NICs: all
 
-For example,
-to build symmetric_mp::
+General set up
+--------------
+1. Compile DPDK in host and VM::
 
-    meson configure -Dexamples=multi_process/symmetric_mp <build_target>
-    ninja -C <build_target>
+    <dpdk dir># meson <dpdk build dir>
+    <dpdk dir># ninja -C <dpdk build dir> -j 110
 
-to run a set of four symmetric_mp instances, running on lcores 1-4, all
-performing level-2 forwarding of packets between ports 0 and 1, the following
-commands can be used (assuming run as root)::
+2. Compile symmetric_mp example application in VM::
 
-   ./<build_target>/examples/dpdk-symmetric_mp -c 2 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=0
-   ./<build_target>/examples/dpdk-symmetric_mp -c 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=1
-   ./<build_target>/examples/dpdk-symmetric_mp -c 8 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=2
-   ./<build_target>/examples/dpdk-symmetric_mp -c 10 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=3
+   <dpdk dir># meson configure -Dexamples=multi_process/symmetric_mp <dpdk build dir>
+   <dpdk dir># ninja -C <dpdk build dir> -j 110
 
-To run only 1 or 2 instances, the above parameters to the 1 or 2 instances being
-run should remain the same, except for the ``num-procs`` value, which should be
-adjusted appropriately.
+3. Compile hotplug_mp example application in VM::
 
-Hotplug MP Application Description
-----------------------------------
+    <dpdk dir># meson configure -Dexamples=multi_process/hotplug_mp <dpdk build dir>   
+    <dpdk dir># ninja -C <dpdk build dir> -j 110
 
-Currently secondary process will only sync ethdev from primary process at
-init stage, but it will not be aware if device is attached/detached on
-primary process at runtime.
+Test case
+=========
 
-While there is the requirement from application that take
-primary-secondary process model. The primary process work as a resource
-management process, it will create/destroy virtual device at runtime,
-while the secondary process deal with the network stuff with these devices.
+Common steps
+------------
+1. Bind the virtio port to vfio-pci in VM::
 
-So the orignial intention is to fix this gap, but beyond that the patch
-set provide a more comprehesive solution to handle different hotplug
-cases in multi-process situation, it cover below scenario:
+    <VM1 dpdk dir># modprobe vfio-pci
+    <VM1 dpdk dir># echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 
-* Attach a device from the primary
-* Detach a device from the primary
-* Attach a device from a secondary
-* Detach a device from a secondary
-
-In primary-secondary process model, we assume ethernet devices are shared
-by default, that means attach or detach a device on any process will
-broadcast to all other processes through mp channel then device
-information will be synchronized on all processes.
-
-Any failure during attaching process will cause inconsistent status
-between processes, so proper rollback action should be considered.
+    <VM1 dpdk dir># ./usertools/dpdk-devbind.py --bind=vfio-pci 0000:00:05.0 0000:00:06.0
 
 Test Case 1: Virtio-pmd primary and secondary process symmetric test
-====================================================================
+--------------------------------------------------------------------
+This case test vdev primary/secondary with two symmetric_mp process in VM, each process can reads a different RX queue on each port.
 
-1. Bind one port to vfio-pci, launch testpmd by below command::
+1. Launch testpmd with 2 vdev by below command::
 
-    ./<build_target>/app/dpdk-testpmd -l 1-6 -n 4 --file-prefix=vhost --vdev 'net_vhost,iface=vhost-net,queues=2,client=1' --vdev 'net_vhost1,iface=vhost-net1,queues=2,client=1'  -- -i --nb-cores=4 --rxq=2 --txq=2 --txd=1024 --rxd=1024
-    testpmd>set fwd txonly
-    testpmd>start
+    <dpdk dir># ./<dpdk build dir>/app/dpdk-testpmd -l 1-6 -n 4 --file-prefix=vhost \
+    --vdev 'net_vhost,iface=vhost-net,queues=2,client=1' --vdev 'net_vhost1,iface=vhost-net1,queues=2,client=1' \
+    -- -i --nb-cores=4 --rxq=2 --txq=2 --txd=1024 --rxd=1024
 
 2. Launch VM with two virtio ports, must set queues=2 as app receive packets from special queue which index same with proc-id::
 
-    qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
+    <dpdk dir># qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
     -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
-    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu16.img  \
+    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04.img \
     -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
     -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
-    -monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
+    -monitor unix:/tmp/vm2_monitor.sock,server,nowait \
+    -device e1000,netdev=nttsip1 \
     -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6002-:22 \
     -chardev socket,id=char,path=./vhost-net,server -netdev type=vhost-user,id=mynet1,chardev=char,vhostforce,queues=2 \
     -device virtio-net-pci,mac=52:54:00:00:00:02,netdev=mynet1,mrg_rxbuf=on,csum=on,mq=on,vectors=15  \
     -chardev socket,id=char1,path=./vhost-net1,server -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce,queues=2 \
     -device virtio-net-pci,mac=52:54:00:00:00:03,netdev=mynet2,mrg_rxbuf=on,csum=on,mq=on,vectors=15  -vnc :10 -daemonize
 
-3.  Bind virtio port to vfio-pci::
+3. In VM, get the virtio network device and bind bind it to vfio-pci as common steps 1.
+
+4. In Vm, Compile symmetric_mp and hotplug_mp example as General set up step 2-3.
 
-    ./usertools/dpdk-devbind.py --bind=vfio-pci xx:xx.x
-    ./usertools/dpdk-devbind.py --bind=vfio-pci xx:xx.x
+4. Launch two process by symmetric_mp example in VM, each process will using ports 0 and 1, primary process using queue 0 of each port, secondary process using queue 1 of each port::
 
-4. Launch two process by example::
+    <VM1 dpdk dir># ./<dpdk build dir>/examples/dpdk-symmetric_mp -l 1 -n 4 --proc-type=auto -- -p 3 --num-procs=2 --proc-id=0
+    <VM1 dpdk dir># ./<dpdk build dir>/examples/dpdk-symmetric_mp -l 2 -n 4 --proc-type=secondary -- -p 3 --num-procs=2 --proc-id=1
 
-    ./<build_target>/examples/dpdk-symmetric_mp -l 1 -n 4 --proc-type=auto -- -p 3 --num-procs=2 --proc-id=0
-    ./<build_target>/examples/dpdk-symmetric_mp -l 2 -n 4 --proc-type=secondary -- -p 3 --num-procs=2 --proc-id=1
+5. Send packets from vhost testpmd::
 
-5. Quit all process, check the packets number in rx/tx statistic like below for both primary process and secondary process::
+    testpmd> set fwd mac
+    testpmd> start tx_first
 
-    Port 0: RX - 27511680, TX - 256, Drop - 27499168
-    Port 1: RX - 27499424, TX - 256, Drop - 27511424
+6. Quit all process, check the packets number in rx/tx statistic like below for both primary process and secondary process::
+
+    Port 0: RX - 22322016, TX - 22321184, Drop - 0
+    Port 1: RX - 22321184, TX - 22322016, Drop - 0
 
 Test Case 2: Virtio-pmd primary and secondary process hotplug test
-==================================================================
+------------------------------------------------------------------
+This case use hotplug_mp example to test the MP Application hotplug, attach or detach a device on any process will broadcast to all other processes through mp channel, 
+then device information will be synchronized on all processes. it cover below scenario: attach a device from the primary, detach a device from the primary,
+attach a device from a secondary,detach a device from a secondary.
 
 1. Launch testpmd by below command::
 
-    ./<build_target>/app/dpdk-testpmd -l 1-6 -n 4 --file-prefix=vhost --vdev 'net_vhost,iface=vhost-net,queues=2,client=1' --vdev 'net_vhost1,iface=vhost-net1,queues=2,client=1'  -- -i --nb-cores=4 --rxq=2 --txq=2 --txd=1024 --rxd=1024
-    testpmd>set fwd txonly
-    testpmd>start
+    <dpdk dir> ./<dpdk build dir>/app/dpdk-testpmd -l 1-6 -n 4 --file-prefix=vhost \
+    --vdev 'net_vhost,iface=vhost-net,queues=2,client=1' --vdev 'net_vhost1,iface=vhost-net1,queues=2,client=1' \
+    -- -i --nb-cores=4 --rxq=2 --txq=2 --txd=1024 --rxd=1024
+    testpmd> set fwd txonly
+    testpmd> start
 
 2. Launch VM with two virtio ports, must set queues=2 as app receive packets from special queue which index same with proc-id::
 
-    qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
+    <dpdk dir># qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
     -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
-    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu16.img  \
+    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04.img  \
     -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
     -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
-    -monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
+    -monitor unix:/tmp/vm2_monitor.sock,server,nowait \
+    -device e1000,netdev=nttsip1 \
     -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6002-:22 \
     -chardev socket,id=char,path=./vhost-net,server -netdev type=vhost-user,id=mynet1,chardev=char,vhostforce,queues=2 \
     -device virtio-net-pci,mac=52:54:00:00:00:02,netdev=mynet1,mrg_rxbuf=on,csum=on,mq=on,vectors=15  \
     -chardev socket,id=char1,path=./vhost-net1,server -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce,queues=2 \
     -device virtio-net-pci,mac=52:54:00:00:00:03,netdev=mynet2,mrg_rxbuf=on,csum=on,mq=on,vectors=15  -vnc :10 -daemonize
 
-3.  Bind virtio port to vfio-pci::
-
-    ./usertools/dpdk-devbind.py --bind=vfio-pci xx:xx.x
-    ./usertools/dpdk-devbind.py --bind=vfio-pci xx:xx.x
+3. In VM, get the virtio network device and bind bind it to vfio-pci as common steps 1-2.
 
-4. Start sample code as primary process::
+4. Start sample code as primary process and list all etherdev::
 
-    ./<build_target>/examples/dpdk-symmetric_mp --proc-type=auto -- -p 3 --num-procs=2 --proc-id=0
+    <VM1 dpdk dir> ./<dpdk build dir>/examples/dpdk-hotplug_mp --proc-type=auto -- -p 3 --num-procs=2 --proc-id=0
     example> list
     list all etherdev
     0       0000:00:05.0
     1       0000:00:06.0
 
-5. Start sample code as secondary process::
+5. Start sample code as secondary process and list all etherdev::
 
-    ./<build_target>/examples/dpdk-symmetric_mp --proc-type=secondary -- -p 3 --num-procs=2 --proc-id=1
+    <VM1 dpdk dir> ./<dpdk build dir>/examples/dpdk-hotplug_mp --proc-type=secondary -- -p 3 --num-procs=2 --proc-id=1
     example> list
     list all etherdev
     0       0000:00:05.0
-- 
2.25.1

[dts][PATCH V1 1/2] test_plans/vdev_primary_secondary_test_plan: add hotplug testcase2

[Wei Ling]

Add hotplug testcase2 and modify testplan with new format.

Signed-off-by: Wei Ling <weix.ling@intel.com>
---
 .../vdev_primary_secondary_test_plan.rst      | 188 ++++++++----------
 1 file changed, 82 insertions(+), 106 deletions(-)

diff --git a/test_plans/vdev_primary_secondary_test_plan.rst b/test_plans/vdev_primary_secondary_test_plan.rst
index b98d2d04..9d3acbad 100644
--- a/test_plans/vdev_primary_secondary_test_plan.rst
+++ b/test_plans/vdev_primary_secondary_test_plan.rst
@@ -34,172 +34,148 @@
 Virtio-pmd primary/secondary process test plan
 ==============================================
 
-This test plan will test vdev primary/secondary with symmetric_mp, which demonstrates how a set of processes can run in parallel,
-with each process performing the same set of packet processing operations. Also test vdev primary/secondary with hotplug_mp example.
-
-Symmetric MP Application Description
-------------------------------------
-
-This test is a multi-process test which demonstrates how multiple processes can
-work together to perform packet I/O and packet processing in parallel, much as
-other example application work by using multiple threads. In this example, each
-process reads packets from all network ports being used - though from a different
-RX queue in each case. Those packets are then forwarded by each process which
-sends them out by writing them directly to a suitable TX queue.
+Description
+===========
+
+This test plan will test vdev primary/secondary with symmetric_mp, 
+which demonstrates how a set of processes can run in parallel,
+with each process performing the same set of packet processing operations, 
+each process reads packets from each of the network ports in use.
+Each process reads a different RX queue and writes outgoing packets to a different TX queue on each port. 
+Also test vdev primary/secondary with hotplug_mp example. 
+The primary process work as a resource management process, it will create/destroy virtual device at runtime, 
+while the secondary process can deal with the network stuff with these devices, 
+that means attach or detach a device on any process will broadcast to all other processes through mp channel 
+then device information will be synchronized on all processes.
+For more about the Symmetric MP Application example, 
+please refer to the dpdk docs: http://doc.dpdk.org/guides/sample_app_ug/multi_process.html
 
 Prerequisites
--------------
-
-Assuming that DPDK build has been set up and the multi-process sample
-applications have been built. It is also assumed that a traffic generator has
-been configured and plugged in to the NIC ports 0 and 1.
-
-Test Methodology
-----------------
-
-As with the simple_mp example, the first instance of the symmetric_mp process
-must be run as the primary instance, though with a number of other application
-specific parameters also provided after the EAL arguments. These additional
-parameters are:
+=============
 
-* -p <portmask>, where portmask is a hexadecimal bitmask of what ports on the
-  system are to be used. For example: -p 3 to use ports 0 and 1 only.
-* --num-procs <N>, where N is the total number of symmetric_mp instances that
-  will be run side-by-side to perform packet processing. This parameter is used to
-  configure the appropriate number of receive queues on each network port.
-* --proc-id <n>, where n is a numeric value in the range 0 <= n < N (number of
-  processes, specified above). This identifies which symmetric_mp instance is being
-  run, so that each process can read a unique receive queue on each network port.
+Topology
+--------
+Vhost-->Virtio-tmp
 
-The secondary symmetric_mp instances must also have these parameters specified,
-and the first two must be the same as those passed to the primary instance, or errors
-result.
+Hardware
+--------
+Supportted NICs: all
 
-For example,
-to build symmetric_mp::
+General set up
+--------------
+1. Compile DPDK in host and VM::
 
-    meson configure -Dexamples=multi_process/symmetric_mp <build_target>
-    ninja -C <build_target>
+    <dpdk dir># meson <dpdk build dir>
+    <dpdk dir># ninja -C <dpdk build dir> -j 110
 
-to run a set of four symmetric_mp instances, running on lcores 1-4, all
-performing level-2 forwarding of packets between ports 0 and 1, the following
-commands can be used (assuming run as root)::
+2. Compile symmetric_mp example application in VM::
 
-   ./<build_target>/examples/dpdk-symmetric_mp -c 2 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=0
-   ./<build_target>/examples/dpdk-symmetric_mp -c 4 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=1
-   ./<build_target>/examples/dpdk-symmetric_mp -c 8 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=2
-   ./<build_target>/examples/dpdk-symmetric_mp -c 10 --proc-type=auto -- -p 3 --num-procs=4 --proc-id=3
+   <dpdk dir># meson configure -Dexamples=multi_process/symmetric_mp <dpdk build dir>
+   <dpdk dir># ninja -C <dpdk build dir> -j 110
 
-To run only 1 or 2 instances, the above parameters to the 1 or 2 instances being
-run should remain the same, except for the ``num-procs`` value, which should be
-adjusted appropriately.
+3. Compile hotplug_mp example application in VM::
 
-Hotplug MP Application Description
-----------------------------------
+    <dpdk dir># meson configure -Dexamples=multi_process/hotplug_mp <dpdk build dir>   
+    <dpdk dir># ninja -C <dpdk build dir> -j 110
 
-Currently secondary process will only sync ethdev from primary process at
-init stage, but it will not be aware if device is attached/detached on
-primary process at runtime.
+Test case
+=========
 
-While there is the requirement from application that take
-primary-secondary process model. The primary process work as a resource
-management process, it will create/destroy virtual device at runtime,
-while the secondary process deal with the network stuff with these devices.
+Common steps
+------------
+1. Bind the virtio port to vfio-pci in VM::
 
-So the orignial intention is to fix this gap, but beyond that the patch
-set provide a more comprehesive solution to handle different hotplug
-cases in multi-process situation, it cover below scenario:
+    <VM1 dpdk dir># modprobe vfio-pci
+    <VM1 dpdk dir># echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 
-* Attach a device from the primary
-* Detach a device from the primary
-* Attach a device from a secondary
-* Detach a device from a secondary
-
-In primary-secondary process model, we assume ethernet devices are shared
-by default, that means attach or detach a device on any process will
-broadcast to all other processes through mp channel then device
-information will be synchronized on all processes.
-
-Any failure during attaching process will cause inconsistent status
-between processes, so proper rollback action should be considered.
+    <VM1 dpdk dir># ./usertools/dpdk-devbind.py --bind=vfio-pci 0000:00:05.0 0000:00:06.0
 
 Test Case 1: Virtio-pmd primary and secondary process symmetric test
-====================================================================
+--------------------------------------------------------------------
+This case test vdev primary/secondary with two symmetric_mp process in VM, each process can reads a different RX queue on each port.
 
-1. Bind one port to vfio-pci, launch testpmd by below command::
+1. Launch testpmd with 2 vdev by below command::
 
-    ./<build_target>/app/dpdk-testpmd -l 1-6 -n 4 --file-prefix=vhost --vdev 'net_vhost,iface=vhost-net,queues=2,client=1' --vdev 'net_vhost1,iface=vhost-net1,queues=2,client=1'  -- -i --nb-cores=4 --rxq=2 --txq=2 --txd=1024 --rxd=1024
-    testpmd>set fwd txonly
-    testpmd>start
+    <dpdk dir># ./<dpdk build dir>/app/dpdk-testpmd -l 1-6 -n 4 --file-prefix=vhost \
+    --vdev 'net_vhost,iface=vhost-net,queues=2,client=1' --vdev 'net_vhost1,iface=vhost-net1,queues=2,client=1' \
+    -- -i --nb-cores=4 --rxq=2 --txq=2 --txd=1024 --rxd=1024
 
 2. Launch VM with two virtio ports, must set queues=2 as app receive packets from special queue which index same with proc-id::
 
-    qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
+    <dpdk dir># qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
     -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
-    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu16.img  \
+    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04.img \
     -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
     -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
-    -monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
+    -monitor unix:/tmp/vm2_monitor.sock,server,nowait \
+    -device e1000,netdev=nttsip1 \
     -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6002-:22 \
     -chardev socket,id=char,path=./vhost-net,server -netdev type=vhost-user,id=mynet1,chardev=char,vhostforce,queues=2 \
     -device virtio-net-pci,mac=52:54:00:00:00:02,netdev=mynet1,mrg_rxbuf=on,csum=on,mq=on,vectors=15  \
     -chardev socket,id=char1,path=./vhost-net1,server -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce,queues=2 \
     -device virtio-net-pci,mac=52:54:00:00:00:03,netdev=mynet2,mrg_rxbuf=on,csum=on,mq=on,vectors=15  -vnc :10 -daemonize
 
-3.  Bind virtio port to vfio-pci::
+3. In VM, get the virtio network device and bind bind it to vfio-pci as common steps 1.
+
+4. In Vm, Compile symmetric_mp and hotplug_mp example as General set up step 2-3.
 
-    ./usertools/dpdk-devbind.py --bind=vfio-pci xx:xx.x
-    ./usertools/dpdk-devbind.py --bind=vfio-pci xx:xx.x
+4. Launch two process by symmetric_mp example in VM, each process will using ports 0 and 1, primary process using queue 0 of each port, secondary process using queue 1 of each port::
 
-4. Launch two process by example::
+    <VM1 dpdk dir># ./<dpdk build dir>/examples/dpdk-symmetric_mp -l 1 -n 4 --proc-type=auto -- -p 3 --num-procs=2 --proc-id=0
+    <VM1 dpdk dir># ./<dpdk build dir>/examples/dpdk-symmetric_mp -l 2 -n 4 --proc-type=secondary -- -p 3 --num-procs=2 --proc-id=1
 
-    ./<build_target>/examples/dpdk-symmetric_mp -l 1 -n 4 --proc-type=auto -- -p 3 --num-procs=2 --proc-id=0
-    ./<build_target>/examples/dpdk-symmetric_mp -l 2 -n 4 --proc-type=secondary -- -p 3 --num-procs=2 --proc-id=1
+5. Send packets from vhost testpmd::
 
-5. Quit all process, check the packets number in rx/tx statistic like below for both primary process and secondary process::
+    testpmd> set fwd mac
+    testpmd> start tx_first
 
-    Port 0: RX - 27511680, TX - 256, Drop - 27499168
-    Port 1: RX - 27499424, TX - 256, Drop - 27511424
+6. Quit all process, check the packets number in rx/tx statistic like below for both primary process and secondary process::
+
+    Port 0: RX - 22322016, TX - 22321184, Drop - 0
+    Port 1: RX - 22321184, TX - 22322016, Drop - 0
 
 Test Case 2: Virtio-pmd primary and secondary process hotplug test
-==================================================================
+------------------------------------------------------------------
+This case use hotplug_mp example to test the MP Application hotplug, attach or detach a device on any process will broadcast to all other processes through mp channel, 
+then device information will be synchronized on all processes. it cover below scenario: attach a device from the primary, detach a device from the primary,
+attach a device from a secondary,detach a device from a secondary.
 
 1. Launch testpmd by below command::
 
-    ./<build_target>/app/dpdk-testpmd -l 1-6 -n 4 --file-prefix=vhost --vdev 'net_vhost,iface=vhost-net,queues=2,client=1' --vdev 'net_vhost1,iface=vhost-net1,queues=2,client=1'  -- -i --nb-cores=4 --rxq=2 --txq=2 --txd=1024 --rxd=1024
-    testpmd>set fwd txonly
-    testpmd>start
+    <dpdk dir> ./<dpdk build dir>/app/dpdk-testpmd -l 1-6 -n 4 --file-prefix=vhost \
+    --vdev 'net_vhost,iface=vhost-net,queues=2,client=1' --vdev 'net_vhost1,iface=vhost-net1,queues=2,client=1' \
+    -- -i --nb-cores=4 --rxq=2 --txq=2 --txd=1024 --rxd=1024
+    testpmd> set fwd txonly
+    testpmd> start
 
 2. Launch VM with two virtio ports, must set queues=2 as app receive packets from special queue which index same with proc-id::
 
-    qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
+    <dpdk dir># qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
     -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
-    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu16.img  \
+    -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04.img  \
     -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
     -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
-    -monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
+    -monitor unix:/tmp/vm2_monitor.sock,server,nowait \
+    -device e1000,netdev=nttsip1 \
     -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6002-:22 \
     -chardev socket,id=char,path=./vhost-net,server -netdev type=vhost-user,id=mynet1,chardev=char,vhostforce,queues=2 \
     -device virtio-net-pci,mac=52:54:00:00:00:02,netdev=mynet1,mrg_rxbuf=on,csum=on,mq=on,vectors=15  \
     -chardev socket,id=char1,path=./vhost-net1,server -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce,queues=2 \
     -device virtio-net-pci,mac=52:54:00:00:00:03,netdev=mynet2,mrg_rxbuf=on,csum=on,mq=on,vectors=15  -vnc :10 -daemonize
 
-3.  Bind virtio port to vfio-pci::
-
-    ./usertools/dpdk-devbind.py --bind=vfio-pci xx:xx.x
-    ./usertools/dpdk-devbind.py --bind=vfio-pci xx:xx.x
+3. In VM, get the virtio network device and bind bind it to vfio-pci as common steps 1-2.
 
-4. Start sample code as primary process::
+4. Start sample code as primary process and list all etherdev::
 
-    ./<build_target>/examples/dpdk-symmetric_mp --proc-type=auto -- -p 3 --num-procs=2 --proc-id=0
+    <VM1 dpdk dir> ./<dpdk build dir>/examples/dpdk-hotplug_mp --proc-type=auto -- -p 3 --num-procs=2 --proc-id=0
     example> list
     list all etherdev
     0       0000:00:05.0
     1       0000:00:06.0
 
-5. Start sample code as secondary process::
+5. Start sample code as secondary process and list all etherdev::
 
-    ./<build_target>/examples/dpdk-symmetric_mp --proc-type=secondary -- -p 3 --num-procs=2 --proc-id=1
+    <VM1 dpdk dir> ./<dpdk build dir>/examples/dpdk-hotplug_mp --proc-type=secondary -- -p 3 --num-procs=2 --proc-id=1
     example> list
     list all etherdev
     0       0000:00:05.0
-- 
2.25.1