[dpdk-dev] MLX5: Array written to out-of-bounds when freeing memory

[dpdk-dev] MLX5: Array written to out-of-bounds when freeing memory

[Daniel Pharos]

Hi,

Recently I had the privilege of some play-time on a IBM Power 9 machine with a Mellanox MLX5-card in it. However, I encountered a seg fault problem using DPDK and DPDK-pktgen. It's the exact same one as somebody else encountered (also on a Power 9) here:
https://bugzilla.redhat.com/show_bug.cgi?id=1634159#c10
It looks like it's writing out-of-bounds on the "free"-array in DPDK.

Reverting the patch that added the bulk-free ( http://mails.dpdk.org/archives/dev/2017-June/069154.html ) indeed makes DPDK and pktgen function correctly. A better workaround I found is to change drivers/net/mlx5/mlx5_rxtx.h, line 580 to:
                        if (likely((m->pool == pool) && (blk_n != elts_n))) {

I'm not familiar enough with the code to understand why the array is being written to out-of-bounds, and why it's only happening on a Power 9 machine. Unfortunately, my play-time is now over, but I thought I'd report this issue anyway, so hopefully it can be investigated and fixed properly.


Kind regards,
DanielPharos

Re: [dpdk-dev] MLX5: Array written to out-of-bounds when freeing memory

[Yongseok Koh]

Appreciate your report.
We are aware of the issue and Dekel is investigating it.
As I wrote the code, I'm also looking at the issue.
Will keep you posted.

Thanks,
Yongseok

> On Jan 18, 2019, at 9:36 AM, Daniel Pharos <danielpharos@hotmail.com> wrote:
> 
> Hi,
> 
> Recently I had the privilege of some play-time on a IBM Power 9 machine with a Mellanox MLX5-card in it. However, I encountered a seg fault problem using DPDK and DPDK-pktgen. It's the exact same one as somebody else encountered (also on a Power 9) here:
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.redhat.com%2Fshow_bug.cgi%3Fid%3D1634159%23c10&amp;data=02%7C01%7Cyskoh%40mellanox.com%7Ce3e3546283b64a84e3b608d67d9acf8f%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636834501198813346&amp;sdata=722uRuNvcISF69NTrqwFldzIPC%2FP8Wz3CDKPutStWP8%3D&amp;reserved=0
> It looks like it's writing out-of-bounds on the "free"-array in DPDK.
> 
> Reverting the patch that added the bulk-free ( https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dpdk.org%2Farchives%2Fdev%2F2017-June%2F069154.html&amp;data=02%7C01%7Cyskoh%40mellanox.com%7Ce3e3546283b64a84e3b608d67d9acf8f%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636834501198813346&amp;sdata=73up%2FKm9SZHBTdu64f5guSojTGNTFRkY7ePrBal7Bvo%3D&amp;reserved=0 ) indeed makes DPDK and pktgen function correctly. A better workaround I found is to change drivers/net/mlx5/mlx5_rxtx.h, line 580 to:
>                        if (likely((m->pool == pool) && (blk_n != elts_n))) {
> 
> I'm not familiar enough with the code to understand why the array is being written to out-of-bounds, and why it's only happening on a Power 9 machine. Unfortunately, my play-time is now over, but I thought I'd report this issue anyway, so hopefully it can be investigated and fixed properly.
> 
> 
> Kind regards,
> DanielPharos

Re: [dpdk-dev] MLX5: Array written to out-of-bounds when freeing memory

[Daniel Pharos]

Hi Yongseok,

Thanks for the quick response! If there is any information you need that I may be able to provide, please just ask.


Kind regards,
DanielPharos

-----Original Message-----
From: Yongseok Koh <yskoh@mellanox.com> 
Sent: 19 January 2019 08:15
To: Daniel Pharos <danielpharos@hotmail.com>
Cc: dev@dpdk.org; Dekel Peled <dekelp@mellanox.com>; Olga Shern <olgas@mellanox.com>
Subject: Re: [dpdk-dev] MLX5: Array written to out-of-bounds when freeing memory

Appreciate your report.
We are aware of the issue and Dekel is investigating it.
As I wrote the code, I'm also looking at the issue.
Will keep you posted.

Thanks,
Yongseok

> On Jan 18, 2019, at 9:36 AM, Daniel Pharos <danielpharos@hotmail.com> wrote:
> 
> Hi,
> 
> Recently I had the privilege of some play-time on a IBM Power 9 machine with a Mellanox MLX5-card in it. However, I encountered a seg fault problem using DPDK and DPDK-pktgen. It's the exact same one as somebody else encountered (also on a Power 9) here:
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.redhat.com%2Fshow_bug.cgi%3Fid%3D1634159%23c10&amp;data=02%7C01%7Cyskoh%40mellanox.com%7Ce3e3546283b64a84e3b608d67d9acf8f%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636834501198813346&amp;sdata=722uRuNvcISF69NTrqwFldzIPC%2FP8Wz3CDKPutStWP8%3D&amp;reserved=0
> It looks like it's writing out-of-bounds on the "free"-array in DPDK.
> 
> Reverting the patch that added the bulk-free ( https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dpdk.org%2Farchives%2Fdev%2F2017-June%2F069154.html&amp;data=02%7C01%7Cyskoh%40mellanox.com%7Ce3e3546283b64a84e3b608d67d9acf8f%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636834501198813346&amp;sdata=73up%2FKm9SZHBTdu64f5guSojTGNTFRkY7ePrBal7Bvo%3D&amp;reserved=0 ) indeed makes DPDK and pktgen function correctly. A better workaround I found is to change drivers/net/mlx5/mlx5_rxtx.h, line 580 to:
>                        if (likely((m->pool == pool) && (blk_n != elts_n))) {
> 
> I'm not familiar enough with the code to understand why the array is being written to out-of-bounds, and why it's only happening on a Power 9 machine. Unfortunately, my play-time is now over, but I thought I'd report this issue anyway, so hopefully it can be investigated and fixed properly.
> 
> 
> Kind regards,
> DanielPharos