[dpdk-users] Possible memory corruption due to incorrect DMA shutdown

[dpdk-users] Possible memory corruption due to incorrect DMA shutdown

[George Prekas]

Hi. I've been using DPDK for a research project 
(https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay) 
for over 2 years and I'd like to report a behavior that puzzles me using 
DPDK.

The behavior leads to memory corruption and is caused by the incorrect 
shutdown of DMA. I can reproduce it after executing the following steps:

$ sudo modprobe uio
$ sudo insmod ./build/kmod/igb_uio.ko
$ sudo python ./tools/dpdk_nic_bind.py --bind=igb_uio 0000:42:00.1
$ sudo ./build/app/testpmd -- --forward-mode=icmpecho

Then I terminate the DPDK program (after populating the ARP cache of 
another host on the local network). After this, I can send UDP packets 
to the host and can observe their payload in host memory. Clearly, 
network packets are arriving to the network card and are written to RAM 
after DPDK has finished executing.

Am I doing something wrong? Is this behavior expected?

Regards,
George

Re: [dpdk-users] Possible memory corruption due to incorrect DMA shutdown

[George Prekas]

Any updates on this?

I can consistently reproduce this behavior following these steps:

On host A do:

$ sudo modprobe uio
$ sudo insmod ./build/kmod/igb_uio.ko
$ sudo python ./tools/dpdk_nic_bind.py --bind=igb_uio 0000:42:00.1
$ sudo ./build/app/testpmd -- --forward-mode=icmpecho

 From host B (which is on the same local network), do an arping to an 
arbitrary IP address C.C.C.C (that is in the same network and it doesn't 
belong to another host). DPDK will respond to any IP address. All you 
want is to populate host B's ARP cache. Then terminate DPDK and run the 
following Python program on host B:

import sys
import socket

A=sys.argv[3] * int(sys.argv[2])

for i in xrange(10000):
   sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
   sock.sendto(A, (sys.argv[1], 1))

as:

$ python go.py C.C.C.C 28 prekageo_was_here_

Then go back to host A and use the physical memory grep tool that you 
can find here https://github.com/prekageo/allmemscan

$ make
$ sudo insmod allmem.ko
$ sudo ./allmemscan 28 prekageo_was_here

Optionally:

$ sudo dd if=/dev/allmem bs=4K skip=PAGE count=1 | hexdump -C | less

On 11/03/2016 19:41, George Prekas wrote:
> Hi. I've been using DPDK for a research project 
> (https://www.usenix.org/conference/osdi14/technical-sessions/presentation/belay) 
> for over 2 years and I'd like to report a behavior that puzzles me 
> using DPDK.
>
> The behavior leads to memory corruption and is caused by the incorrect 
> shutdown of DMA. I can reproduce it after executing the following steps:
>
> $ sudo modprobe uio
> $ sudo insmod ./build/kmod/igb_uio.ko
> $ sudo python ./tools/dpdk_nic_bind.py --bind=igb_uio 0000:42:00.1
> $ sudo ./build/app/testpmd -- --forward-mode=icmpecho
>
> Then I terminate the DPDK program (after populating the ARP cache of 
> another host on the local network). After this, I can send UDP packets 
> to the host and can observe their payload in host memory. Clearly, 
> network packets are arriving to the network card and are written to 
> RAM after DPDK has finished executing.
>
> Am I doing something wrong? Is this behavior expected?
>
> Regards,
> George
>