From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 43583A0562; Wed, 14 Apr 2021 17:33:16 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E7ECA161B6F; Wed, 14 Apr 2021 17:33:15 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 356A34013F for ; Wed, 14 Apr 2021 17:33:14 +0200 (CEST) IronPort-SDR: cfuqDWl2dcz7IMCWUh4e8LuCp/ijBiaQfuu+k9JtUfggKlo8jYEDmJGDG2oQxK7Wp8eWulN/Md aqG6zz1DxYwA== X-IronPort-AV: E=McAfee;i="6200,9189,9954"; a="215157017" X-IronPort-AV: E=Sophos;i="5.82,222,1613462400"; d="scan'208";a="215157017" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Apr 2021 08:33:12 -0700 IronPort-SDR: 9tuGr3l/lvVQuO/O67OuDySDXRTGZ3kGAy/Ybg5ZXj5D2WI9I8ZZUnxJhd9QMIZTsB1j8z20XG qPMzQ+DKRyrA== X-IronPort-AV: E=Sophos;i="5.82,222,1613462400"; d="scan'208";a="600824556" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.213.237.43]) ([10.213.237.43]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Apr 2021 08:33:11 -0700 To: Liang Ma , Bruce Richardson Cc: dev@dpdk.org References: <20210413182441.GC1185@bricha3-MOBL.ger.corp.intel.com> <20210414104803.GA514@bricha3-MOBL.ger.corp.intel.com> <20210414131301.GB514@bricha3-MOBL.ger.corp.intel.com> From: "Burakov, Anatoly" Message-ID: Date: Wed, 14 Apr 2021 16:33:10 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 14-Apr-21 2:45 PM, Liang Ma wrote: > On Wed, Apr 14, 2021 at 02:21:53PM +0100, Bruce Richardson wrote: >> On Wed, Apr 14, 2021 at 01:52:14PM +0100, Liang Ma wrote: >>> On Wed, Apr 14, 2021 at 11:48:03AM +0100, Bruce Richardson wrote: >>>> On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: >>>>> On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: >>>>>> On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: >>>>>>> Hi Bruce, >>>>>>> I look into current IOAT PMD. I found that looks only support the >>>>>>> latest DSA device which is part of SPR(Please correct me if I'm wrong). >>>>>>> However, there still is lots existing CPU that support previous CBDMA >>>>>>> Intel Quick-DATA technology. SPDK enable those device with uio. >>>>>>> Any hint to enable those CBDMA device with IOAT PMD ? >>>>>>> >>>>>> >>>>>> Hi Liang, >>>>>> >>>>>> These devices are still supported by the driver. In the latest releases the >>>>>> new device support was added but no older device support was removed. >>>>>> Please let us know if you do hit any problems with running it on the CBDMA >>>>>> HW devices, because it should all still be working as before. >>>>>> >>>>>> Regards, >>>>>> /Bruce >>>>> Hi Bruce, >>>>> I hit a problem with CBDMA enabling. >>>>> >>>>> 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with >>>>> dpdk-devbind.py >>>>> >>>>> 2. I run dpdk-test without any parameter then dpdk-test complain with >>>>> >>>>> EAL: VFIO support initialized >>>>> EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU >>>>> group bound to VFIO or unbound >>>>> Requested device 0000:00:04.0 cannot be used >>>>> >>>>> 3. kernel boot parameter include the "intel_iommu=on" >>>>> >>>>> If I still need some extra iommu settings for CBDMA device ? >>>>> >>>> It's an IOMMU limitation, not just for CBDMA but for other devices like >>>> NICs too - you need to ensure all subfunctions on a device are all bound to >>>> the same IOMMU group. So for the CBDMA instance, to use one channel with >>>> vfio, the other channels on the device also need to be bound to vfio, or >>>> else not bound to any driver. If bound to the kernel driver, there will be >>>> IOMMU conflicts as we can't split the channels between kernel and userspace >>>> address spaces. >>>> >>>> /Bruce >>> I bind whole device from 00:04.0 to 00:04.7. ioat PMD complain >>> with "ioat_rawdev_create(): ioat_rawdev_create: Channel appears >>> locked". Any hint? >>> Regards >>> Liang >> >> That should be a warning only, I think and the device should still be >> available. Is that not the case, or is there a later error? >> >> /Bruce > Good news, that's a warning only message, I checked the code. > after probe the device, rawdev_autotest all passed ! > Many thanks for your help ! > I strongly recommend to add the IOMMU hint of CBDMA(bind whole device) > into the Doc. I think DSA device don't need to do that right ? > Best Regards > Liang > This is documented in our VFIO docs [1]. [1] http://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules (see Warning section) -- Thanks, Anatoly