From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1C4ECA0562; Wed, 14 Apr 2021 17:31:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 98452161B69; Wed, 14 Apr 2021 17:31:21 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id BFC844013F for ; Wed, 14 Apr 2021 17:31:19 +0200 (CEST) IronPort-SDR: fsanRtS/NuivcYcFPGA+UQAgIwgyR2cjPhwodi3sCyh20CRIorTu1b6/OjK+bosjldSBvf3Yb7 3I73GVo8k7Fw== X-IronPort-AV: E=McAfee;i="6200,9189,9954"; a="192538684" X-IronPort-AV: E=Sophos;i="5.82,222,1613462400"; d="scan'208";a="192538684" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Apr 2021 08:31:18 -0700 IronPort-SDR: gLlojjN6oA9YNbMv0C9kw8byr1WlaGLecGjUEMuQhyUrlGMd+tXCWcRWhCibzqK2xoosOPXgng nPz/hAtvx+ww== X-IronPort-AV: E=Sophos;i="5.82,222,1613462400"; d="scan'208";a="600823399" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.213.237.43]) ([10.213.237.43]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Apr 2021 08:31:16 -0700 To: Bruce Richardson , Liang Ma Cc: dev@dpdk.org References: <20210413182441.GC1185@bricha3-MOBL.ger.corp.intel.com> <20210414104803.GA514@bricha3-MOBL.ger.corp.intel.com> From: "Burakov, Anatoly" Message-ID: Date: Wed, 14 Apr 2021 16:31:12 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: <20210414104803.GA514@bricha3-MOBL.ger.corp.intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 14-Apr-21 11:48 AM, Bruce Richardson wrote: > On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: >> On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: >>> On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: >>>> Hi Bruce, >>>> I look into current IOAT PMD. I found that looks only support the >>>> latest DSA device which is part of SPR(Please correct me if I'm wrong). >>>> However, there still is lots existing CPU that support previous CBDMA >>>> Intel Quick-DATA technology. SPDK enable those device with uio. >>>> Any hint to enable those CBDMA device with IOAT PMD ? >>>> >>> >>> Hi Liang, >>> >>> These devices are still supported by the driver. In the latest releases the >>> new device support was added but no older device support was removed. >>> Please let us know if you do hit any problems with running it on the CBDMA >>> HW devices, because it should all still be working as before. >>> >>> Regards, >>> /Bruce >> Hi Bruce, >> I hit a problem with CBDMA enabling. >> >> 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with >> dpdk-devbind.py >> >> 2. I run dpdk-test without any parameter then dpdk-test complain with >> >> EAL: VFIO support initialized >> EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU >> group bound to VFIO or unbound >> Requested device 0000:00:04.0 cannot be used >> >> 3. kernel boot parameter include the "intel_iommu=on" >> >> If I still need some extra iommu settings for CBDMA device ? >> > It's an IOMMU limitation, not just for CBDMA but for other devices like > NICs too - you need to ensure all subfunctions on a device are all bound to > the same IOMMU group. So for the CBDMA instance, to use one channel with > vfio, the other channels on the device also need to be bound to vfio, or > else not bound to any driver. If bound to the kernel driver, there will be > IOMMU conflicts as we can't split the channels between kernel and userspace > address spaces. > > /Bruce > Yep, and depending on HW architecture, that just might not be possible (i've seen this happen before, where all devices on the platform, including things like GPU, were behind one IOMMU group). -- Thanks, Anatoly