From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id C47CB1BA96 for ; Thu, 5 Jul 2018 19:49:57 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Jul 2018 10:49:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,313,1526367600"; d="scan'208";a="52262990" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.83]) ([10.237.221.83]) by fmsmga007.fm.intel.com with ESMTP; 05 Jul 2018 10:49:52 -0700 To: Mordechay Haimovsky , Shahaf Shuler Cc: Adrien Mazarguil , "dev@dpdk.org" , Olga Shern References: <1530169969-6708-1-git-send-email-motih@mellanox.com> <1530529900-27859-1-git-send-email-motih@mellanox.com> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= xsFNBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABzSVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+wsF+BBMBAgAoAhsDBgsJCAcDAgYVCAIJCgsE FgIDAQIeAQIXgAUCWZR3VQUJB33WBQAKCRD5M+tD3xNhH6DWEACVhEb8q1epPwZrUDoxzu7E TS1b8tmabOmnjXZRs6+EXgUVHkp2xxkCfDmL3pa5bC0G/74aJnWjNsdvE05V1cb4YK4kRQ62 FwDQ+hlrFrwFB3PtDZk1tpkzCRHvJgnIil+0MuEh32Y57ig6hy8yO8ql7Lohyrnpfk/nNpm4 jQGEF5qEeHcEFe1AZQlPHN/STno8NZSz2nl0b2cw+cujN1krmvB52Ah/2KugQ6pprVyrGrzB c34ZQO9OsmSjJlETCZk6EZzuhfe16iqBFbOSadi9sPcJRwaUQBid+xdFWl7GQ8qC3zNPibSF HmU43yBZUqJDZlhIcl6/cFpOSjv2sDWdtjEXTDn5y/0FsuY0mFE78ItC4kCTIVk17VZoywcd fmbbnwOSWzDq7hiUYuQGkIudJw5k/A1CMsyLkoUEGN3sLfsw6KASgS4XrrmPO4UVr3mH5bP1 yC7i1OVNpzvOxtahmzm481ID8sk72GC2RktTOHb0cX+qdoiMMfYgo3wRRDYCBt6YoGYUxF1p msjocXyqToKhhnFbXLaZlVfnQ9i2i8jsj9SKig+ewC2p3lkPj6ncye9q95bzhmUeJO6sFhJg Hiz6syOMg8yCcq60j07airybAuHIDNFWk0gaWAmtHZxLObZx2PVn2nv9kLYGohFekw0AOsIW ta++5m48dnCoAc7BTQRX1ky+ARAApzQNvXvE2q1LAS+Z+ni2R13Bb1cDS1ZYq1jgpR13+OKN ipzd8MPngRJilXxBaPTErhgzR0vGcNTYhjGMSyFIHVOoBq1VbP1a0Fi/NqWzJOowo/fDfgVy K4vuitc/gCJs+2se4hdZA4EQJxVlNM51lgYDNpjPGIA43MX15OLAip73+ho6NPBMuc5qse3X pAClNhBKfENRCWN428pi3WVkT+ABRTE0taxjJNP7bb+9TQYNRqGwnGzX5/XISv44asWIQCaq vOkXSUJLd//cdVNTqtL1wreCVVR5pMXj7VIrlk07fmmJVALCmGbFr53BMb8O+8dgK2A5mitM n44d+8KdJWOwziRxcaMk/LclmZS3Iv1TERtiWt98Y9AjeAtcgYPkA3ld0BcUKONogP8pHVz1 Ed3s5rDQ91yr1S0wuAzW91fxGUO4wY+uPmxCtFVuBgd9VT9NAKTUL0qHM7CDgCnZPe0TW6Zj 8OqtdCCyAfvU9cW5xWM7Icxhde6AtPxhDSBwE8fL2ZmrDmaA4jmUKXp3i4JxRPSX84S08b+s DWXHPxy10UFU5A7EK/BEbZAKBwn9ROfm+WK+6X5xOGLoRE++OqNuUudxC1GDyLOPaqCbBCS9 +P6HsTHzxsjyJa27n4jcrcuY3P9TEcFJYSZSeSDh8mVGvugi0exnSJrrBZDyVCcAEQEAAcLB ZQQYAQIADwIbDAUCWZR1ZwUJA59cIQAKCRD5M+tD3xNhH5b+D/9XG44Ci6STdcA5RO/ur05J EE3Ux1DCHZ5V7vNAtX/8Wg4l4GZfweauXwuJ1w7Sp7fklwcNC6wsceI+EmNjGMqfIaukGetG +jBGqsQ7moOZodfXUoCK98gblKgt/BPYMVidzlGC8Q/+lZg1+o29sPnwImW+MXt/Z5az/Z17 Qc265g+p5cqJHzq6bpQdnF7Fu6btKU/kv6wJghENvgMXBuyThqsyFReJWFh2wfaKyuix3Zyj ccq7/blkhzIKmtFWgDcgaSc2UAuJU+x9nuYjihW6WobpKP/nlUDu3BIsbIq09UEke+uE/QK+ FJ8PTJkAsXOf1Bc2C0XbW4Y2hf103+YY6L8weUCBsWC5VH5VtVmeuh26ENURclwfeXhWQ9Og 77yzpTXWr5g1Z0oLpYpWPv745J4bE7pv+dzxOrFdM1xNkzY2pvXph/A8OjxZNQklDkHQ7PIB Lki5L2F4XkEOddUUQchJwzMqTPsggPDmGjgLZrqgO+s4ECZK5+nLD3HEpAbPa3JLDaScy+90 Nu1lAqPUHSnP3vYZVw85ZYm6UCxHE4VLMnnJsN09ZhsOSVR+GyP5Nyw9rT1V3lcsuH7M5Naa 2Xobn9m7l9bRCD/Ji8kG15eV1WTxx1HXVQGjdUYDI7UwegBNbwMLh17XDy+3sn/6SgcqtECA Q6pZKA2mTQxEKMLBZQQYAQIADwIbDAUCWZR3hQUJA59eRwAKCRD5M+tD3xNhH4a/D/4jLAZu UhvU1swWcNEVVCELZ0D3LOV14XcY2MXa3QOpeZ9Bgq7YYJ4S5YXK+SBQS0FkRZdjGNvlGZoG ZdpU+NsQmQFhqHGwX0IT9MeTFM8uvKgxNKGwMVcV9g0IOqwBhGHne+BFboRA9362fgGW5AYQ zT0mzzRKEoOh4r3AQvbM6kLISxo0k1ujdYiI5nj/5WoKDqxTwwfuN1uDUHsWo3tzenRmpMyU NyW3Dc+1ajvXLyo09sRRq7BnM99Rix1EGL8Qhwy+j0YAv+FuspWxUX9FxXYho5PvGLHLsHfK FYQ7x/RRbpMjkJWVfIe/xVnfvn4kz+MTA5yhvsuNi678fLwY9hBP0y4lO8Ob2IhEPdfnTuIs tFVxXuelJ9xAe5TyqP0f+fQjf1ixsBZkqOohsBXDfje0iaUpYa/OQ/BBeej0dUdg2JEu4jAC x41HpVCnP9ipLpD0fYz1d/dX0F/VY2ovW6Eba/y/ngOSAR6C+u881m7oH2l0G47MTwkaQCBA bLGXPj4TCdX3lftqt4bcBPBJ+rFAnJmRHtUuyyaewBnZ81ZU2YAptqFM1kTh+aSvMvGhfVsQ qZL2rk2OPN1hg+KXhErlbTZ6oPtLCFhSHQmuxQ4oc4U147wBTUuOdwNjtnNatUhRCp8POc+3 XphVR5G70mnca1E2vzC77z+XSlTyRA== Message-ID: Date: Thu, 5 Jul 2018 18:49:52 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH v2] net/mlx5: add support for 32bit systems X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jul 2018 17:49:58 -0000 On 7/5/2018 6:07 PM, Mordechay Haimovsky wrote: > Hello Ferruh, > Here are my findings: > > 1. The error you've seen is definitely a bug in mlx5dv.h from rdma-core > (I'm emphasizing rdma-core since I cannot just send a fix for this file) > As it didn’t take into account that an address may be a 32bit one when performing the 32bit shift. > __m128i val = _mm_set_epi32((uint32_t)address, (uint32_t)(address >> 32), lkey, length); > 2. The reason we didn’t see it in our setups is due to the values assigned to the GCC predefined macros > We are using (from RH and UBUNTU). > When I run the following commands in our setups: > alias gccmacros='gcc -dM -E -x c /dev/null' > gccmacros -m32 | grep -E "(MMX|SSE|AVX|XOP)" > I get the following results: > On RH setup using gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) > #define __MMX__ 1 > #define __SSE2__ 1 > #define __SSE__ 1 > On Ubuntu setup using gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) > No flags are defined. > Since the "offending" routine is wrapped with #ifdef __SSE3__ the compiler just ignores it. > > ARs: > 1. Open a bug for fixing mlx5dv.h in rdma-core. - Moti H. > 2. Provide a workaround for the problem. - Moti H. > 3. Verify that this is actually the issue by running the above scripts > In Ferruh setup and verifying the SSE3 flag is set. - Ferruh Yigit I confirm SSE3 is set in my environment, but I think this will be true for all x86 because DPDK min required SIMD is SSE4.2. According wiki SSE3 introduced in 2004. We use -march=native in dpdk build, so: $ gcc -march=native -m32 -dM -E - > Moti H. > >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Mordechay >> Haimovsky >> Sent: Thursday, July 5, 2018 1:10 PM >> To: Ferruh Yigit ; Shahaf Shuler >> >> Cc: Adrien Mazarguil ; dev@dpdk.org >> Subject: Re: [dpdk-dev] [PATCH v2] net/mlx5: add support for 32bit systems >> >> Hi, >> Didn’t see it in our setups (not an excuse), Investigating .... >> >> Moti >> >>> -----Original Message----- >>> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com] >>> Sent: Wednesday, July 4, 2018 4:49 PM >>> To: Mordechay Haimovsky ; Shahaf Shuler >>> >>> Cc: Adrien Mazarguil ; dev@dpdk.org >>> Subject: Re: [dpdk-dev] [PATCH v2] net/mlx5: add support for 32bit >>> systems >>> >>> On 7/2/2018 12:11 PM, Moti Haimovsky wrote: >>>> This patch adds support for building and running mlx5 PMD on 32bit >>>> systems such as i686. >>>> >>>> The main issue to tackle was handling the 32bit access to the UAR as >>>> quoted from the mlx5 PRM: >>>> QP and CQ DoorBells require 64-bit writes. For best performance, it >>>> is recommended to execute the QP/CQ DoorBell as a single 64-bit >>>> write operation. For platforms that do not support 64 bit writes, it >>>> is possible to issue the 64 bits DoorBells through two consecutive >>>> writes, each write 32 bits, as described below: >>>> * The order of writing each of the Dwords is from lower to upper >>>> addresses. >>>> * No other DoorBell can be rung (or even start ringing) in the midst of >>>> an on-going write of a DoorBell over a given UAR page. >>>> The last rule implies that in a multi-threaded environment, the >>>> access to a UAR page (which can be accessible by all threads in the >>>> process) must be synchronized (for example, using a semaphore) >>>> unless an atomic write of 64 bits in a single bus operation is >>>> guaranteed. Such a synchronization is not required for when ringing >>>> DoorBells on different UAR pages. >>>> >>>> Signed-off-by: Moti Haimovsky >>>> --- >>>> v2: >>>> * Fixed coding style issues. >>>> * Modified documentation according to review inputs. >>>> * Fixed merge conflicts. >>>> --- >>>> doc/guides/nics/features/mlx5.ini | 1 + >>>> doc/guides/nics/mlx5.rst | 6 +++- >>>> drivers/net/mlx5/mlx5.c | 8 ++++- >>>> drivers/net/mlx5/mlx5.h | 5 +++ >>>> drivers/net/mlx5/mlx5_defs.h | 18 ++++++++-- >>>> drivers/net/mlx5/mlx5_rxq.c | 6 +++- >>>> drivers/net/mlx5/mlx5_rxtx.c | 22 +++++++------ >>>> drivers/net/mlx5/mlx5_rxtx.h | 69 >>> ++++++++++++++++++++++++++++++++++++++- >>>> drivers/net/mlx5/mlx5_txq.c | 13 +++++++- >>>> 9 files changed, 131 insertions(+), 17 deletions(-) >>>> >>>> diff --git a/doc/guides/nics/features/mlx5.ini >>>> b/doc/guides/nics/features/mlx5.ini >>>> index e75b14b..b28b43e 100644 >>>> --- a/doc/guides/nics/features/mlx5.ini >>>> +++ b/doc/guides/nics/features/mlx5.ini >>>> @@ -43,5 +43,6 @@ Multiprocess aware = Y >>>> Other kdrv = Y >>>> ARMv8 = Y >>>> Power8 = Y >>>> +x86-32 = Y >>>> x86-64 = Y >>>> Usage doc = Y >>>> diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst >>>> index >>>> 7dd9c1c..5fbad60 100644 >>>> --- a/doc/guides/nics/mlx5.rst >>>> +++ b/doc/guides/nics/mlx5.rst >>>> @@ -49,7 +49,7 @@ libibverbs. >>>> Features >>>> -------- >>>> >>>> -- Multi arch support: x86_64, POWER8, ARMv8. >>>> +- Multi arch support: x86_64, POWER8, ARMv8, i686. >>>> - Multiple TX and RX queues. >>>> - Support for scattered TX and RX frames. >>>> - IPv4, IPv6, TCPv4, TCPv6, UDPv4 and UDPv6 RSS on any number of >>> queues. >>>> @@ -477,6 +477,10 @@ RMDA Core with Linux Kernel >>>> - Minimal kernel version : v4.14 or the most recent 4.14-rc (see >>>> `Linux installation documentation`_) >>>> - Minimal rdma-core version: v15+ commit 0c5f5765213a ("Merge pull >>> request #227 from yishaih/tm") >>>> (see `RDMA Core installation documentation`_) >>>> +- When building for i686 use: >>>> + >>>> + - rdma-core version 18.0 or above built with 32bit support. >>> >>> related "or above" part, v19 giving build errors with mlx5, FYI. >>> >>> And with v18 getting build errors originated from rdma headers [1], am >>> I doing something wrong? >>> >>> [1] >>> In file included from .../dpdk/drivers/net/mlx5/mlx5_glue.c:20: >>> .../rdma-core/build32/include/infiniband/mlx5dv.h: In function >>> ‘mlx5dv_x86_set_data_seg’: >>> .../rdma-core/build32/include/infiniband/mlx5dv.h:787:69: error: right >>> shift count >= width of type [-Werror=shift-count-overflow] >>> __m128i val = _mm_set_epi32((uint32_t)address, (uint32_t)(address >>>>> 32), lkey, length); >>> >>> ^~