[PATCH] app/dma-perf: fix physical address seg-fault

[PATCH] app/dma-perf: fix physical address seg-fault

[Vipin Varghese]

do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
the start of the virtual address for both src and dst.
But in case of iova mode set as PA, this results in seg-fault.
This is because rte_memcpy VA address and not PA.

this fix checks the iova mode and invokes rte_memcpy with the right
arguments.

Bugzilla ID: 1269
Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
Cc: cheng1.jiang@intel.com

Cc: stable@dpdk.org

Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
---

tested for both va and pa

CMD:
PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini

Log: fails for dc mode `EAL: invalid parameters for --iova-mode`

test.ini:
```
[case1]
type=CPU_MEM_COPY
mem_size=10
buf_size=64,8192,2,MUL
src_numa_node=0
dst_numa_node=0
cache_flush=0
test_seconds=2
lcore = 7
eal_args=--in-memory --no-pci
```
---
 app/test-dma-perf/benchmark.c | 39 +++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 9 deletions(-)

diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-perf/benchmark.c
index 0601e0d171..5573acc9f9 100644
--- a/app/test-dma-perf/benchmark.c
+++ b/app/test-dma-perf/benchmark.c
@@ -279,6 +279,10 @@ do_cpu_mem_copy(void *p)
 	struct rte_mbuf **srcs = para->srcs;
 	struct rte_mbuf **dsts = para->dsts;
 	uint32_t i;
+	bool isAddrPaMode = false;
+
+	if (rte_eal_iova_mode() == RTE_IOVA_PA)
+		isAddrPaMode = true;
 
 	worker_info->stop_flag = false;
 	worker_info->ready_flag = true;
@@ -286,16 +290,33 @@ do_cpu_mem_copy(void *p)
 	while (!worker_info->start_flag)
 		;
 
-	while (1) {
-		for (i = 0; i < nr_buf; i++) {
-			/* copy buffer form src to dst */
-			rte_memcpy((void *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
-				(void *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
-				(size_t)buf_size);
-			worker_info->total_cpl++;
+	if (true == isAddrPaMode) {
+		while (1) {
+			for (i = 0; i < nr_buf; i++) {
+				void *src = rte_pktmbuf_mtod(dsts[i], void *);
+				void *dst = rte_pktmbuf_mtod(srcs[i], void *);
+
+				/* copy buffer form src to dst */
+				rte_memcpy(dst,
+					src,
+					(size_t)buf_size);
+				worker_info->total_cpl++;
+			}
+			if (worker_info->stop_flag)
+				break;
+		}
+	} else {
+		while (1) {
+			for (i = 0; i < nr_buf; i++) {
+				/* copy buffer form src to dst */
+				rte_memcpy((void *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
+					(void *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
+					(size_t)buf_size);
+				worker_info->total_cpl++;
+			}
+			if (worker_info->stop_flag)
+				break;
 		}
-		if (worker_info->stop_flag)
-			break;
 	}
 
 	return 0;
-- 
2.34.1

RE: [EXT] [PATCH] app/dma-perf: fix physical address seg-fault

[Anoob Joseph]

Hi Vipin,

Thanks for the update. Please see inline.

Thanks,
Anoob

> -----Original Message-----
> From: Vipin Varghese <vipin.varghese@amd.com>
> Sent: Wednesday, August 16, 2023 12:48 PM
> To: thomas@monjalon.net; dev@dpdk.org
> Cc: Ferruh.Yigit@amd.com; cheng1.jiang@intel.com; stable@dpdk.org
> Subject: [EXT] [PATCH] app/dma-perf: fix physical address seg-fault
> 
> External Email
> 
> ----------------------------------------------------------------------
> do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return the start
> of the virtual address for both src and dst.
> But in case of iova mode set as PA, this results in seg-fault.
> This is because rte_memcpy VA address and not PA.
> 
> this fix checks the iova mode and invokes rte_memcpy with the right
> arguments.
> 
> Bugzilla ID: 1269
> Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> Cc: cheng1.jiang@intel.com
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> ---
> 
> tested for both va and pa
> 
> CMD:
> PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
> VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
> DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini
> 
> Log: fails for dc mode `EAL: invalid parameters for --iova-mode`
> 
> test.ini:
> ```
> [case1]
> type=CPU_MEM_COPY
> mem_size=10
> buf_size=64,8192,2,MUL
> src_numa_node=0
> dst_numa_node=0
> cache_flush=0
> test_seconds=2
> lcore = 7
> eal_args=--in-memory --no-pci
> ```
> ---
>  app/test-dma-perf/benchmark.c | 39 +++++++++++++++++++++++++++----
> ----
>  1 file changed, 30 insertions(+), 9 deletions(-)
> 
> diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-
> perf/benchmark.c index 0601e0d171..5573acc9f9 100644
> --- a/app/test-dma-perf/benchmark.c
> +++ b/app/test-dma-perf/benchmark.c
> @@ -279,6 +279,10 @@ do_cpu_mem_copy(void *p)
>  	struct rte_mbuf **srcs = para->srcs;
>  	struct rte_mbuf **dsts = para->dsts;
>  	uint32_t i;
> +	bool isAddrPaMode = false;
> +
> +	if (rte_eal_iova_mode() == RTE_IOVA_PA)
> +		isAddrPaMode = true;
> 
>  	worker_info->stop_flag = false;
>  	worker_info->ready_flag = true;
> @@ -286,16 +290,33 @@ do_cpu_mem_copy(void *p)
>  	while (!worker_info->start_flag)
>  		;
> 
> -	while (1) {
> -		for (i = 0; i < nr_buf; i++) {
> -			/* copy buffer form src to dst */
> -			rte_memcpy((void
> *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
> -				(void
> *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
> -				(size_t)buf_size);
> -			worker_info->total_cpl++;
> +	if (true == isAddrPaMode) {
> +		while (1) {
> +			for (i = 0; i < nr_buf; i++) {
> +				void *src = rte_pktmbuf_mtod(dsts[i], void
> *);
> +				void *dst = rte_pktmbuf_mtod(srcs[i], void
> *);
> +
> +				/* copy buffer form src to dst */
> +				rte_memcpy(dst,
> +					src,
> +					(size_t)buf_size);
> +				worker_info->total_cpl++;
> +			}
> +			if (worker_info->stop_flag)
> +				break;
> +		}
> +	} else {
> +		while (1) {
> +			for (i = 0; i < nr_buf; i++) {
> +				/* copy buffer form src to dst */
> +				rte_memcpy((void
> *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
> +					(void
> *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
> +					(size_t)buf_size);

[Anoob] Even in case of VA as IOVA, rte_pktmbuf_mtod() should work, right? I was imagining a simple replacement of ' rte_mbuf_data_iova' with 'rte_pktmbuf_mtod' as the fix.

> +				worker_info->total_cpl++;
> +			}
> +			if (worker_info->stop_flag)
> +				break;
>  		}
> -		if (worker_info->stop_flag)
> -			break;
>  	}
> 
>  	return 0;
> --
> 2.34.1

RE: [EXT] [PATCH] app/dma-perf: fix physical address seg-fault

[Varghese, Vipin]

[AMD Official Use Only - General]

> -----Original Message-----
> From: Anoob Joseph <anoobj@marvell.com>
> Sent: Wednesday, August 16, 2023 12:57 PM
> To: Varghese, Vipin <Vipin.Varghese@amd.com>
> Cc: Yigit, Ferruh <Ferruh.Yigit@amd.com>; cheng1.jiang@intel.com;
> stable@dpdk.org; thomas@monjalon.net; dev@dpdk.org; Jerin Jacob
> Kollanukkaran <jerinj@marvell.com>
> Subject: RE: [EXT] [PATCH] app/dma-perf: fix physical address seg-fault
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> Hi Vipin,
>
> Thanks for the update. Please see inline.
>
> Thanks,
> Anoob
>
> > -----Original Message-----
> > From: Vipin Varghese <vipin.varghese@amd.com>
> > Sent: Wednesday, August 16, 2023 12:48 PM
> > To: thomas@monjalon.net; dev@dpdk.org
> > Cc: Ferruh.Yigit@amd.com; cheng1.jiang@intel.com; stable@dpdk.org
> > Subject: [EXT] [PATCH] app/dma-perf: fix physical address seg-fault
> >
> > External Email
> >
> > ----------------------------------------------------------------------
> > do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return the start
> > of the virtual address for both src and dst.
> > But in case of iova mode set as PA, this results in seg-fault.
> > This is because rte_memcpy VA address and not PA.
> >
> > this fix checks the iova mode and invokes rte_memcpy with the right
> > arguments.
> >
> > Bugzilla ID: 1269
> > Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> > Cc: cheng1.jiang@intel.com
> >
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> > ---
> >
> > tested for both va and pa
> >
> > CMD:
> > PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
> > VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
> > DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini
> >
> > Log: fails for dc mode `EAL: invalid parameters for --iova-mode`
> >
> > test.ini:
> > ```
> > [case1]
> > type=CPU_MEM_COPY
> > mem_size=10
> > buf_size=64,8192,2,MUL
> > src_numa_node=0
> > dst_numa_node=0
> > cache_flush=0
> > test_seconds=2
> > lcore = 7
> > eal_args=--in-memory --no-pci
> > ```
> > ---
> >  app/test-dma-perf/benchmark.c | 39 +++++++++++++++++++++++++++----
> > ----
> >  1 file changed, 30 insertions(+), 9 deletions(-)
> >
> > diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-
> > perf/benchmark.c index 0601e0d171..5573acc9f9 100644
> > --- a/app/test-dma-perf/benchmark.c
> > +++ b/app/test-dma-perf/benchmark.c
> > @@ -279,6 +279,10 @@ do_cpu_mem_copy(void *p)
> >       struct rte_mbuf **srcs = para->srcs;
> >       struct rte_mbuf **dsts = para->dsts;
> >       uint32_t i;
> > +     bool isAddrPaMode = false;
> > +
> > +     if (rte_eal_iova_mode() == RTE_IOVA_PA)
> > +             isAddrPaMode = true;
> >
> >       worker_info->stop_flag = false;
> >       worker_info->ready_flag = true;
> > @@ -286,16 +290,33 @@ do_cpu_mem_copy(void *p)
> >       while (!worker_info->start_flag)
> >               ;
> >
> > -     while (1) {
> > -             for (i = 0; i < nr_buf; i++) {
> > -                     /* copy buffer form src to dst */
> > -                     rte_memcpy((void
> > *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
> > -                             (void
> > *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
> > -                             (size_t)buf_size);
> > -                     worker_info->total_cpl++;
> > +     if (true == isAddrPaMode) {
> > +             while (1) {
> > +                     for (i = 0; i < nr_buf; i++) {
> > +                             void *src = rte_pktmbuf_mtod(dsts[i],
> > + void
> > *);
> > +                             void *dst = rte_pktmbuf_mtod(srcs[i],
> > + void
> > *);
> > +
> > +                             /* copy buffer form src to dst */
> > +                             rte_memcpy(dst,
> > +                                     src,
> > +                                     (size_t)buf_size);
> > +                             worker_info->total_cpl++;
> > +                     }
> > +                     if (worker_info->stop_flag)
> > +                             break;
> > +             }
> > +     } else {
> > +             while (1) {
> > +                     for (i = 0; i < nr_buf; i++) {
> > +                             /* copy buffer form src to dst */
> > +                             rte_memcpy((void
> > *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
> > +                                     (void
> > *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
> > +                                     (size_t)buf_size);
>
> [Anoob] Even in case of VA as IOVA, rte_pktmbuf_mtod() should work, right? I
> was imagining a simple replacement of ' rte_mbuf_data_iova' with
> 'rte_pktmbuf_mtod' as the fix.

Ferruh and myself believe the rte_mbuf_data_iova should have been working for both PA and VA. But internally does not.

I think what you are recommending is for both PA and VA just use ` rte_pktmbuf_mtod` for both. Thus we can avoid the check itself.

>
> > +                             worker_info->total_cpl++;
> > +                     }
> > +                     if (worker_info->stop_flag)
> > +                             break;
> >               }
> > -             if (worker_info->stop_flag)
> > -                     break;
> >       }
> >
> >       return 0;
> > --
> > 2.34.1

[PATCH v2] app/dma-perf: fix physical address seg-fault

[Vipin Varghese]

do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
the start of the virtual address for both src and dst.
But in case of iova mode set as PA, this results in seg-fault.
This is because rte_memcpy uses VA address and not PA.

This fix invokes `rte_pktmbuf_mtod` for both src and dst.

Bugzilla ID: 1269
Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
Cc: cheng1.jiang@intel.com

Cc: stable@dpdk.org

Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
Suggested-by: Anoob Joseph <anoobj@marvell.com>
---

v2:
 - suggest use of pktmbuf_mtod for both va and pa.

tested for both va and pa

CMD:
PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini

Log: fails for dc mode `EAL: invalid parameters for --iova-mode`

test.ini:
```
[case1]
type=CPU_MEM_COPY
mem_size=10
buf_size=64,8192,2,MUL
src_numa_node=0
dst_numa_node=0
cache_flush=0
test_seconds=2
lcore = 7
eal_args=--in-memory --no-pci
```
---
 app/test-dma-perf/benchmark.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-perf/benchmark.c
index 0601e0d171..9059253057 100644
--- a/app/test-dma-perf/benchmark.c
+++ b/app/test-dma-perf/benchmark.c
@@ -288,9 +288,12 @@ do_cpu_mem_copy(void *p)
 
 	while (1) {
 		for (i = 0; i < nr_buf; i++) {
+			void *src = rte_pktmbuf_mtod(dsts[i], void *);
+			void *dst = rte_pktmbuf_mtod(srcs[i], void *);
+
 			/* copy buffer form src to dst */
-			rte_memcpy((void *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
-				(void *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
+			rte_memcpy(dst,
+				src,
 				(size_t)buf_size);
 			worker_info->total_cpl++;
 		}
-- 
2.34.1

[PATCH v2] app/dma-perf: fix physical address seg-fault

[Vipin Varghese]

do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
the start of the virtual address for both src and dst.
But in case of iova mode set as PA, this results in seg-fault.
This is because rte_memcpy uses VA address and not PA.

This fix invokes `rte_pktmbuf_mtod` for both src and dst.

Bugzilla ID: 1269
Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
Cc: cheng1.jiang@intel.com

Cc: stable@dpdk.org

Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
Suggested-by: Anoob Joseph <anoobj@marvell.com>
---

v2:
 - suggest use of pktmbuf_mtod for both va and pa.

tested for both va and pa

CMD:
PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini

Log: fails for dc mode `EAL: invalid parameters for --iova-mode`

test.ini:
```
[case1]
type=CPU_MEM_COPY
mem_size=10
buf_size=64,8192,2,MUL
src_numa_node=0
dst_numa_node=0
cache_flush=0
test_seconds=2
lcore = 7
eal_args=--in-memory --no-pci
```
---
 app/test-dma-perf/benchmark.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-perf/benchmark.c
index 0601e0d171..9059253057 100644
--- a/app/test-dma-perf/benchmark.c
+++ b/app/test-dma-perf/benchmark.c
@@ -288,9 +288,12 @@ do_cpu_mem_copy(void *p)
 
 	while (1) {
 		for (i = 0; i < nr_buf; i++) {
+			void *src = rte_pktmbuf_mtod(dsts[i], void *);
+			void *dst = rte_pktmbuf_mtod(srcs[i], void *);
+
 			/* copy buffer form src to dst */
-			rte_memcpy((void *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
-				(void *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
+			rte_memcpy(dst,
+				src,
 				(size_t)buf_size);
 			worker_info->total_cpl++;
 		}
-- 
2.34.1

[PATCH v2] app/dma-perf: fix physical address seg-fault

[Vipin Varghese]

do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
the start of the virtual address for both src and dst.
But in case of iova mode set as PA, this results in seg-fault.
This is because rte_memcpy uses VA address and not PA.

This fix invokes `rte_pktmbuf_mtod` for both src and dst.

Bugzilla ID: 1269
Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
Cc: cheng1.jiang@intel.com

Cc: stable@dpdk.org

Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
Suggested-by: Anoob Joseph <anoobj@marvell.com>
---

v2:
 - suggest use of pktmbuf_mtod for both va and pa.

tested for both va and pa

CMD:
PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini

Log: fails for dc mode `EAL: invalid parameters for --iova-mode`

test.ini:
```
[case1]
type=CPU_MEM_COPY
mem_size=10
buf_size=64,8192,2,MUL
src_numa_node=0
dst_numa_node=0
cache_flush=0
test_seconds=2
lcore = 7
eal_args=--in-memory --no-pci
```
---
 app/test-dma-perf/benchmark.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-perf/benchmark.c
index 0601e0d171..1d1c9bde99 100644
--- a/app/test-dma-perf/benchmark.c
+++ b/app/test-dma-perf/benchmark.c
@@ -288,10 +288,11 @@ do_cpu_mem_copy(void *p)
 
 	while (1) {
 		for (i = 0; i < nr_buf; i++) {
+			void *src = rte_pktmbuf_mtod(dsts[i], void *);
+			void *dst = rte_pktmbuf_mtod(srcs[i], void *);
+
 			/* copy buffer form src to dst */
-			rte_memcpy((void *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
-				(void *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
-				(size_t)buf_size);
+			rte_memcpy(dst, src, (size_t)buf_size);
 			worker_info->total_cpl++;
 		}
 		if (worker_info->stop_flag)
-- 
2.34.1

Re: [PATCH v2] app/dma-perf: fix physical address seg-fault

[Jerin Jacob]

On Wed, Aug 16, 2023 at 3:21 PM Vipin Varghese <vipin.varghese@amd.com> wrote:
>
> do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
> the start of the virtual address for both src and dst.
> But in case of iova mode set as PA, this results in seg-fault.
> This is because rte_memcpy uses VA address and not PA.
>
> This fix invokes `rte_pktmbuf_mtod` for both src and dst.
>
> Bugzilla ID: 1269
> Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> Cc: cheng1.jiang@intel.com
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> Suggested-by: Anoob Joseph <anoobj@marvell.com>
> ---
>
> v2:
>  - suggest use of pktmbuf_mtod for both va and pa.
>
> tested for both va and pa
>
> CMD:
> PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
> VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
> DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini
>
> Log: fails for dc mode `EAL: invalid parameters for --iova-mode`
>
> test.ini:
> ```
> [case1]
> type=CPU_MEM_COPY
> mem_size=10
> buf_size=64,8192,2,MUL
> src_numa_node=0
> dst_numa_node=0
> cache_flush=0
> test_seconds=2
> lcore = 7
> eal_args=--in-memory --no-pci
> ```
> ---
>  app/test-dma-perf/benchmark.c | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-perf/benchmark.c
> index 0601e0d171..1d1c9bde99 100644
> --- a/app/test-dma-perf/benchmark.c
> +++ b/app/test-dma-perf/benchmark.c
> @@ -288,10 +288,11 @@ do_cpu_mem_copy(void *p)
>
>         while (1) {
>                 for (i = 0; i < nr_buf; i++) {
> +                       void *src = rte_pktmbuf_mtod(dsts[i], void *);
> +                       void *dst = rte_pktmbuf_mtod(srcs[i], void *);

Use _const_ in fast path if src and dst in not changing as better
optimization hint to compiler.

RE: [EXT] [PATCH v2] app/dma-perf: fix physical address seg-fault

[Anoob Joseph]

> do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return the start
> of the virtual address for both src and dst.
> But in case of iova mode set as PA, this results in seg-fault.
> This is because rte_memcpy uses VA address and not PA.
> 
> This fix invokes `rte_pktmbuf_mtod` for both src and dst.
> 
> Bugzilla ID: 1269
> Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> Cc: cheng1.jiang@intel.com
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> Suggested-by: Anoob Joseph <anoobj@marvell.com>

Acked-by: Anoob Joseph <anoobj@marvell.com>

RE: [PATCH v2] app/dma-perf: fix physical address seg-fault

[Jiang, Cheng1]

Hi,

> -----Original Message-----
> From: Vipin Varghese <vipin.varghese@amd.com>
> Sent: Wednesday, August 16, 2023 5:42 PM
> To: thomas@monjalon.net; dev@dpdk.org; anoobj@marvell.com
> Cc: Ferruh.Yigit@amd.com; Jiang, Cheng1 <cheng1.jiang@intel.com>;
> stable@dpdk.org
> Subject: [PATCH v2] app/dma-perf: fix physical address seg-fault
> 
> do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return the start
> of the virtual address for both src and dst.
> But in case of iova mode set as PA, this results in seg-fault.
> This is because rte_memcpy uses VA address and not PA.
> 
> This fix invokes `rte_pktmbuf_mtod` for both src and dst.
> 
> Bugzilla ID: 1269
> Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> Cc: cheng1.jiang@intel.com
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> Suggested-by: Anoob Joseph <anoobj@marvell.com>
> ---
> 
> v2:
>  - suggest use of pktmbuf_mtod for both va and pa.
> 
> tested for both va and pa
> 
> CMD:
> PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
> VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
> DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini
> 
> Log: fails for dc mode `EAL: invalid parameters for --iova-mode`
> 
> test.ini:
> ```
> [case1]
> type=CPU_MEM_COPY
> mem_size=10
> buf_size=64,8192,2,MUL
> src_numa_node=0
> dst_numa_node=0
> cache_flush=0
> test_seconds=2
> lcore = 7
> eal_args=--in-memory --no-pci
> ```
> ---
>  app/test-dma-perf/benchmark.c | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-
> perf/benchmark.c index 0601e0d171..1d1c9bde99 100644
> --- a/app/test-dma-perf/benchmark.c
> +++ b/app/test-dma-perf/benchmark.c
> @@ -288,10 +288,11 @@ do_cpu_mem_copy(void *p)
> 
>  	while (1) {
>  		for (i = 0; i < nr_buf; i++) {
> +			void *src = rte_pktmbuf_mtod(dsts[i], void *);
> +			void *dst = rte_pktmbuf_mtod(srcs[i], void *);
> +

Thank you for fixing it.
Sorry that I missed the discussion earlier.

Reviewed-by: Cheng Jiang <cheng1.jiang@intel.com>

Thanks,
Cheng

RE: [PATCH v2] app/dma-perf: fix physical address seg-fault

[Varghese, Vipin]

[AMD Official Use Only - General]

Hi Jerrin,

Apologies for the delay, I missed on the comment

```
>                 for (i = 0; i < nr_buf; i++) {
> +                       void *src = rte_pktmbuf_mtod(dsts[i], void *);
> +                       void *dst = rte_pktmbuf_mtod(srcs[i], void *);

Use _const_ in fast path if src and dst in not changing as better optimization hint to compiler.
```

Thanks for the suggestion, but may I ask this differently `rte_memcpy` is defined with `always_inline` and all places where src used is with `const *`. Hence I have different view there as `setting const * for always_inline definition has the same effect too`.


> -----Original Message-----
> From: Jiang, Cheng1 <cheng1.jiang@intel.com>
> Sent: Thursday, September 21, 2023 7:58 AM
> To: Varghese, Vipin <Vipin.Varghese@amd.com>; thomas@monjalon.net;
> dev@dpdk.org; anoobj@marvell.com
> Cc: Yigit, Ferruh <Ferruh.Yigit@amd.com>; stable@dpdk.org
> Subject: RE: [PATCH v2] app/dma-perf: fix physical address seg-fault
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> Hi,
>
> > -----Original Message-----
> > From: Vipin Varghese <vipin.varghese@amd.com>
> > Sent: Wednesday, August 16, 2023 5:42 PM
> > To: thomas@monjalon.net; dev@dpdk.org; anoobj@marvell.com
> > Cc: Ferruh.Yigit@amd.com; Jiang, Cheng1 <cheng1.jiang@intel.com>;
> > stable@dpdk.org
> > Subject: [PATCH v2] app/dma-perf: fix physical address seg-fault
> >
> > do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return the start
> > of the virtual address for both src and dst.
> > But in case of iova mode set as PA, this results in seg-fault.
> > This is because rte_memcpy uses VA address and not PA.
> >
> > This fix invokes `rte_pktmbuf_mtod` for both src and dst.
> >
> > Bugzilla ID: 1269
> > Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> > Cc: cheng1.jiang@intel.com
> >
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> > Suggested-by: Anoob Joseph <anoobj@marvell.com>
> > ---
> >
> > v2:
> >  - suggest use of pktmbuf_mtod for both va and pa.
> >
> > tested for both va and pa
> >
> > CMD:
> > PA: dpdk-test-dma-perf --iova-mode=pa  -- --config test.ini
> > VA: dpdk-test-dma-perf --iova-mode=va  -- --config test.ini
> > DC: dpdk-test-dma-perf --iova-mode=dc  -- --config test.ini
> >
> > Log: fails for dc mode `EAL: invalid parameters for --iova-mode`
> >
> > test.ini:
> > ```
> > [case1]
> > type=CPU_MEM_COPY
> > mem_size=10
> > buf_size=64,8192,2,MUL
> > src_numa_node=0
> > dst_numa_node=0
> > cache_flush=0
> > test_seconds=2
> > lcore = 7
> > eal_args=--in-memory --no-pci
> > ```
> > ---
> >  app/test-dma-perf/benchmark.c | 7 ++++---
> >  1 file changed, 4 insertions(+), 3 deletions(-)
> >
> > diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-
> > perf/benchmark.c index 0601e0d171..1d1c9bde99 100644
> > --- a/app/test-dma-perf/benchmark.c
> > +++ b/app/test-dma-perf/benchmark.c
> > @@ -288,10 +288,11 @@ do_cpu_mem_copy(void *p)
> >
> >       while (1) {
> >               for (i = 0; i < nr_buf; i++) {
> > +                     void *src = rte_pktmbuf_mtod(dsts[i], void *);
> > +                     void *dst = rte_pktmbuf_mtod(srcs[i], void *);
> > +
>
> Thank you for fixing it.
> Sorry that I missed the discussion earlier.
>
> Reviewed-by: Cheng Jiang <cheng1.jiang@intel.com>
>
> Thanks,
> Cheng

Re: [PATCH v2] app/dma-perf: fix physical address seg-fault

[Jerin Jacob]

On Wed, Oct 4, 2023 at 3:46 PM Varghese, Vipin <Vipin.Varghese@amd.com> wrote:
>
> [AMD Official Use Only - General]
>
> Hi Jerrin,
>
> Apologies for the delay, I missed on the comment
>
> ```
> >                 for (i = 0; i < nr_buf; i++) {
> > +                       void *src = rte_pktmbuf_mtod(dsts[i], void *);
> > +                       void *dst = rte_pktmbuf_mtod(srcs[i], void *);
>
> Use _const_ in fast path if src and dst in not changing as better optimization hint to compiler.
> ```
>
> Thanks for the suggestion, but may I ask this differently `rte_memcpy` is defined with `always_inline` and all places where src used is with `const *`. Hence I have different view there as `setting const * for always_inline definition has the same effect too`.

Yeah, most likely. However, Is the above behavior is documented in
gcc/clang? if not, the compiler is free to change as it deemed to fit
based on the register allocation and optimization flags etc.

[PATCH v3] app/dma-perf: fix physical address seg-fault

[Vipin Varghese]

do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
the start of the virtual address for both src and dst.
But in case of iova mode set as PA, this results in seg-fault.
This is because rte_memcpy uses VA address and not PA.

This fix invokes `rte_pktmbuf_mtod` for both src and dst.

Bugzilla ID: 1269
Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
Cc: cheng1.jiang@intel.com

Cc: stable@dpdk.org

Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
Suggested-by: Anoob Joseph <anoobj@marvell.com>
Suggested-by: Jerin Jacob <jerinj@marvell.com>

---

v2:
 - suggest use of pktmbuf_mtod for both va and pa by Anoob.

V3:
 - add const to src suggested by Jerin.
---
 app/test-dma-perf/benchmark.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-perf/benchmark.c
index 0601e0d171..9b1f58c78c 100644
--- a/app/test-dma-perf/benchmark.c
+++ b/app/test-dma-perf/benchmark.c
@@ -288,10 +288,11 @@ do_cpu_mem_copy(void *p)
 
 	while (1) {
 		for (i = 0; i < nr_buf; i++) {
+			const void *src = rte_pktmbuf_mtod(dsts[i], void *);
+			void *dst = rte_pktmbuf_mtod(srcs[i], void *);
+
 			/* copy buffer form src to dst */
-			rte_memcpy((void *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
-				(void *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
-				(size_t)buf_size);
+			rte_memcpy(dst, src, (size_t)buf_size);
 			worker_info->total_cpl++;
 		}
 		if (worker_info->stop_flag)
-- 
2.34.1

Re: [PATCH v3] app/dma-perf: fix physical address seg-fault

[lihuisong (C)]

在 2023/10/19 12:19, Vipin Varghese 写道:
> do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
> the start of the virtual address for both src and dst.
> But in case of iova mode set as PA, this results in seg-fault.
> This is because rte_memcpy uses VA address and not PA.
>
> This fix invokes `rte_pktmbuf_mtod` for both src and dst.
>
> Bugzilla ID: 1269
> Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> Cc: cheng1.jiang@intel.com
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> Suggested-by: Anoob Joseph <anoobj@marvell.com>
> Suggested-by: Jerin Jacob <jerinj@marvell.com>
>
Acked-by: Huisong Li <lihuisong@huawei.com>

Re: [PATCH v3] app/dma-perf: fix physical address seg-fault

[Thomas Monjalon]

24/10/2023 04:16, lihuisong (C):
> 
> 在 2023/10/19 12:19, Vipin Varghese 写道:
> > do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
> > the start of the virtual address for both src and dst.
> > But in case of iova mode set as PA, this results in seg-fault.
> > This is because rte_memcpy uses VA address and not PA.
> >
> > This fix invokes `rte_pktmbuf_mtod` for both src and dst.
> >
> > Bugzilla ID: 1269
> > Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> > Cc: cheng1.jiang@intel.com
> >
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> > Suggested-by: Anoob Joseph <anoobj@marvell.com>
> > Suggested-by: Jerin Jacob <jerinj@marvell.com>
> >
> Acked-by: Huisong Li <lihuisong@huawei.com>

Adding some missing acks.

Applied, thanks.

[PATCH v3] app/dma-perf: fix physical address seg-fault

[Vipin Varghese]

do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
the start of the virtual address for both src and dst.
But in case of iova mode set as PA, this results in seg-fault.
This is because rte_memcpy uses VA address and not PA.

This fix invokes `rte_pktmbuf_mtod` for both src and dst.

Bugzilla ID: 1269
Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
Cc: cheng1.jiang@intel.com

Cc: stable@dpdk.org

Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
Suggested-by: Anoob Joseph <anoobj@marvell.com>
Suggested-by: Jerin Jacob <jerinj@marvell.com>

---

v2:
 - suggest use of pktmbuf_mtod for both va and pa by Anoob.

V3:
 - add const to src suggested by Jerin.
---
 app/test-dma-perf/benchmark.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-perf/benchmark.c
index 0601e0d171..9b1f58c78c 100644
--- a/app/test-dma-perf/benchmark.c
+++ b/app/test-dma-perf/benchmark.c
@@ -288,10 +288,11 @@ do_cpu_mem_copy(void *p)
 
 	while (1) {
 		for (i = 0; i < nr_buf; i++) {
+			const void *src = rte_pktmbuf_mtod(dsts[i], void *);
+			void *dst = rte_pktmbuf_mtod(srcs[i], void *);
+
 			/* copy buffer form src to dst */
-			rte_memcpy((void *)(uintptr_t)rte_mbuf_data_iova(dsts[i]),
-				(void *)(uintptr_t)rte_mbuf_data_iova(srcs[i]),
-				(size_t)buf_size);
+			rte_memcpy(dst, src, (size_t)buf_size);
 			worker_info->total_cpl++;
 		}
 		if (worker_info->stop_flag)
-- 
2.34.1

Re: [PATCH v3] app/dma-perf: fix physical address seg-fault

[fengchengwen]

Acked-by: Chengwen Feng <fengchengwen@huawei.com>

On 2023/10/19 12:16, Vipin Varghese wrote:
> do_cpu_mem_copy uses DPDK API rte_mbuf_data_iova to return
> the start of the virtual address for both src and dst.
> But in case of iova mode set as PA, this results in seg-fault.
> This is because rte_memcpy uses VA address and not PA.
> 
> This fix invokes `rte_pktmbuf_mtod` for both src and dst.
> 
> Bugzilla ID: 1269
> Fixes: 623dc9364dc6 ("app/dma-perf: introduce DMA performance test")
> Cc: cheng1.jiang@intel.com
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
> Suggested-by: Anoob Joseph <anoobj@marvell.com>
> Suggested-by: Jerin Jacob <jerinj@marvell.com>
> 
> ---

...