* [dpdk-dev] Question about enable CBDMA with ioat PMD ? @ 2021-04-13 17:36 Liang Ma 2021-04-13 18:24 ` Bruce Richardson 0 siblings, 1 reply; 11+ messages in thread From: Liang Ma @ 2021-04-13 17:36 UTC (permalink / raw) To: dev; +Cc: bruce.richardson Hi Bruce, I look into current IOAT PMD. I found that looks only support the latest DSA device which is part of SPR(Please correct me if I'm wrong). However, there still is lots existing CPU that support previous CBDMA Intel Quick-DATA technology. SPDK enable those device with uio. Any hint to enable those CBDMA device with IOAT PMD ? Many thanks Best Regards Liang ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-13 17:36 [dpdk-dev] Question about enable CBDMA with ioat PMD ? Liang Ma @ 2021-04-13 18:24 ` Bruce Richardson 2021-04-14 10:37 ` Liang Ma 0 siblings, 1 reply; 11+ messages in thread From: Bruce Richardson @ 2021-04-13 18:24 UTC (permalink / raw) To: Liang Ma; +Cc: dev On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: > Hi Bruce, > I look into current IOAT PMD. I found that looks only support the > latest DSA device which is part of SPR(Please correct me if I'm wrong). > However, there still is lots existing CPU that support previous CBDMA > Intel Quick-DATA technology. SPDK enable those device with uio. > Any hint to enable those CBDMA device with IOAT PMD ? > Hi Liang, These devices are still supported by the driver. In the latest releases the new device support was added but no older device support was removed. Please let us know if you do hit any problems with running it on the CBDMA HW devices, because it should all still be working as before. Regards, /Bruce ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-13 18:24 ` Bruce Richardson @ 2021-04-14 10:37 ` Liang Ma 2021-04-14 10:48 ` Bruce Richardson 0 siblings, 1 reply; 11+ messages in thread From: Liang Ma @ 2021-04-14 10:37 UTC (permalink / raw) To: Bruce Richardson; +Cc: dev On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: > On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: > > Hi Bruce, > > I look into current IOAT PMD. I found that looks only support the > > latest DSA device which is part of SPR(Please correct me if I'm wrong). > > However, there still is lots existing CPU that support previous CBDMA > > Intel Quick-DATA technology. SPDK enable those device with uio. > > Any hint to enable those CBDMA device with IOAT PMD ? > > > > Hi Liang, > > These devices are still supported by the driver. In the latest releases the > new device support was added but no older device support was removed. > Please let us know if you do hit any problems with running it on the CBDMA > HW devices, because it should all still be working as before. > > Regards, > /Bruce Hi Bruce, I hit a problem with CBDMA enabling. 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with dpdk-devbind.py 2. I run dpdk-test without any parameter then dpdk-test complain with EAL: VFIO support initialized EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU group bound to VFIO or unbound Requested device 0000:00:04.0 cannot be used 3. kernel boot parameter include the "intel_iommu=on" If I still need some extra iommu settings for CBDMA device ? Regards Liang ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-14 10:37 ` Liang Ma @ 2021-04-14 10:48 ` Bruce Richardson 2021-04-14 12:52 ` Liang Ma 2021-04-14 15:31 ` Burakov, Anatoly 0 siblings, 2 replies; 11+ messages in thread From: Bruce Richardson @ 2021-04-14 10:48 UTC (permalink / raw) To: Liang Ma; +Cc: dev On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: > On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: > > On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: > > > Hi Bruce, > > > I look into current IOAT PMD. I found that looks only support the > > > latest DSA device which is part of SPR(Please correct me if I'm wrong). > > > However, there still is lots existing CPU that support previous CBDMA > > > Intel Quick-DATA technology. SPDK enable those device with uio. > > > Any hint to enable those CBDMA device with IOAT PMD ? > > > > > > > Hi Liang, > > > > These devices are still supported by the driver. In the latest releases the > > new device support was added but no older device support was removed. > > Please let us know if you do hit any problems with running it on the CBDMA > > HW devices, because it should all still be working as before. > > > > Regards, > > /Bruce > Hi Bruce, > I hit a problem with CBDMA enabling. > > 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with > dpdk-devbind.py > > 2. I run dpdk-test without any parameter then dpdk-test complain with > > EAL: VFIO support initialized > EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU > group bound to VFIO or unbound > Requested device 0000:00:04.0 cannot be used > > 3. kernel boot parameter include the "intel_iommu=on" > > If I still need some extra iommu settings for CBDMA device ? > It's an IOMMU limitation, not just for CBDMA but for other devices like NICs too - you need to ensure all subfunctions on a device are all bound to the same IOMMU group. So for the CBDMA instance, to use one channel with vfio, the other channels on the device also need to be bound to vfio, or else not bound to any driver. If bound to the kernel driver, there will be IOMMU conflicts as we can't split the channels between kernel and userspace address spaces. /Bruce ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-14 10:48 ` Bruce Richardson @ 2021-04-14 12:52 ` Liang Ma 2021-04-14 13:21 ` Bruce Richardson 2021-04-14 15:31 ` Burakov, Anatoly 1 sibling, 1 reply; 11+ messages in thread From: Liang Ma @ 2021-04-14 12:52 UTC (permalink / raw) To: Bruce Richardson; +Cc: dev On Wed, Apr 14, 2021 at 11:48:03AM +0100, Bruce Richardson wrote: > On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: > > On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: > > > On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: > > > > Hi Bruce, > > > > I look into current IOAT PMD. I found that looks only support the > > > > latest DSA device which is part of SPR(Please correct me if I'm wrong). > > > > However, there still is lots existing CPU that support previous CBDMA > > > > Intel Quick-DATA technology. SPDK enable those device with uio. > > > > Any hint to enable those CBDMA device with IOAT PMD ? > > > > > > > > > > Hi Liang, > > > > > > These devices are still supported by the driver. In the latest releases the > > > new device support was added but no older device support was removed. > > > Please let us know if you do hit any problems with running it on the CBDMA > > > HW devices, because it should all still be working as before. > > > > > > Regards, > > > /Bruce > > Hi Bruce, > > I hit a problem with CBDMA enabling. > > > > 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with > > dpdk-devbind.py > > > > 2. I run dpdk-test without any parameter then dpdk-test complain with > > > > EAL: VFIO support initialized > > EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU > > group bound to VFIO or unbound > > Requested device 0000:00:04.0 cannot be used > > > > 3. kernel boot parameter include the "intel_iommu=on" > > > > If I still need some extra iommu settings for CBDMA device ? > > > It's an IOMMU limitation, not just for CBDMA but for other devices like > NICs too - you need to ensure all subfunctions on a device are all bound to > the same IOMMU group. So for the CBDMA instance, to use one channel with > vfio, the other channels on the device also need to be bound to vfio, or > else not bound to any driver. If bound to the kernel driver, there will be > IOMMU conflicts as we can't split the channels between kernel and userspace > address spaces. > > /Bruce I bind whole device from 00:04.0 to 00:04.7. ioat PMD complain with "ioat_rawdev_create(): ioat_rawdev_create: Channel appears locked". Any hint? Regards Liang ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-14 12:52 ` Liang Ma @ 2021-04-14 13:21 ` Bruce Richardson 2021-04-14 13:45 ` Liang Ma 0 siblings, 1 reply; 11+ messages in thread From: Bruce Richardson @ 2021-04-14 13:21 UTC (permalink / raw) To: Liang Ma; +Cc: dev On Wed, Apr 14, 2021 at 01:52:14PM +0100, Liang Ma wrote: > On Wed, Apr 14, 2021 at 11:48:03AM +0100, Bruce Richardson wrote: > > On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: > > > On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: > > > > On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: > > > > > Hi Bruce, > > > > > I look into current IOAT PMD. I found that looks only support the > > > > > latest DSA device which is part of SPR(Please correct me if I'm wrong). > > > > > However, there still is lots existing CPU that support previous CBDMA > > > > > Intel Quick-DATA technology. SPDK enable those device with uio. > > > > > Any hint to enable those CBDMA device with IOAT PMD ? > > > > > > > > > > > > > Hi Liang, > > > > > > > > These devices are still supported by the driver. In the latest releases the > > > > new device support was added but no older device support was removed. > > > > Please let us know if you do hit any problems with running it on the CBDMA > > > > HW devices, because it should all still be working as before. > > > > > > > > Regards, > > > > /Bruce > > > Hi Bruce, > > > I hit a problem with CBDMA enabling. > > > > > > 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with > > > dpdk-devbind.py > > > > > > 2. I run dpdk-test without any parameter then dpdk-test complain with > > > > > > EAL: VFIO support initialized > > > EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU > > > group bound to VFIO or unbound > > > Requested device 0000:00:04.0 cannot be used > > > > > > 3. kernel boot parameter include the "intel_iommu=on" > > > > > > If I still need some extra iommu settings for CBDMA device ? > > > > > It's an IOMMU limitation, not just for CBDMA but for other devices like > > NICs too - you need to ensure all subfunctions on a device are all bound to > > the same IOMMU group. So for the CBDMA instance, to use one channel with > > vfio, the other channels on the device also need to be bound to vfio, or > > else not bound to any driver. If bound to the kernel driver, there will be > > IOMMU conflicts as we can't split the channels between kernel and userspace > > address spaces. > > > > /Bruce > I bind whole device from 00:04.0 to 00:04.7. ioat PMD complain > with "ioat_rawdev_create(): ioat_rawdev_create: Channel appears > locked". Any hint? > Regards > Liang That should be a warning only, I think and the device should still be available. Is that not the case, or is there a later error? /Bruce ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-14 13:21 ` Bruce Richardson @ 2021-04-14 13:45 ` Liang Ma 2021-04-14 13:49 ` Bruce Richardson 2021-04-14 15:33 ` Burakov, Anatoly 0 siblings, 2 replies; 11+ messages in thread From: Liang Ma @ 2021-04-14 13:45 UTC (permalink / raw) To: Bruce Richardson; +Cc: dev On Wed, Apr 14, 2021 at 02:21:53PM +0100, Bruce Richardson wrote: > On Wed, Apr 14, 2021 at 01:52:14PM +0100, Liang Ma wrote: > > On Wed, Apr 14, 2021 at 11:48:03AM +0100, Bruce Richardson wrote: > > > On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: > > > > On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: > > > > > On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: > > > > > > Hi Bruce, > > > > > > I look into current IOAT PMD. I found that looks only support the > > > > > > latest DSA device which is part of SPR(Please correct me if I'm wrong). > > > > > > However, there still is lots existing CPU that support previous CBDMA > > > > > > Intel Quick-DATA technology. SPDK enable those device with uio. > > > > > > Any hint to enable those CBDMA device with IOAT PMD ? > > > > > > > > > > > > > > > > Hi Liang, > > > > > > > > > > These devices are still supported by the driver. In the latest releases the > > > > > new device support was added but no older device support was removed. > > > > > Please let us know if you do hit any problems with running it on the CBDMA > > > > > HW devices, because it should all still be working as before. > > > > > > > > > > Regards, > > > > > /Bruce > > > > Hi Bruce, > > > > I hit a problem with CBDMA enabling. > > > > > > > > 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with > > > > dpdk-devbind.py > > > > > > > > 2. I run dpdk-test without any parameter then dpdk-test complain with > > > > > > > > EAL: VFIO support initialized > > > > EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU > > > > group bound to VFIO or unbound > > > > Requested device 0000:00:04.0 cannot be used > > > > > > > > 3. kernel boot parameter include the "intel_iommu=on" > > > > > > > > If I still need some extra iommu settings for CBDMA device ? > > > > > > > It's an IOMMU limitation, not just for CBDMA but for other devices like > > > NICs too - you need to ensure all subfunctions on a device are all bound to > > > the same IOMMU group. So for the CBDMA instance, to use one channel with > > > vfio, the other channels on the device also need to be bound to vfio, or > > > else not bound to any driver. If bound to the kernel driver, there will be > > > IOMMU conflicts as we can't split the channels between kernel and userspace > > > address spaces. > > > > > > /Bruce > > I bind whole device from 00:04.0 to 00:04.7. ioat PMD complain > > with "ioat_rawdev_create(): ioat_rawdev_create: Channel appears > > locked". Any hint? > > Regards > > Liang > > That should be a warning only, I think and the device should still be > available. Is that not the case, or is there a later error? > > /Bruce Good news, that's a warning only message, I checked the code. after probe the device, rawdev_autotest all passed ! Many thanks for your help ! I strongly recommend to add the IOMMU hint of CBDMA(bind whole device) into the Doc. I think DSA device don't need to do that right ? Best Regards Liang ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-14 13:45 ` Liang Ma @ 2021-04-14 13:49 ` Bruce Richardson 2021-04-14 15:33 ` Burakov, Anatoly 1 sibling, 0 replies; 11+ messages in thread From: Bruce Richardson @ 2021-04-14 13:49 UTC (permalink / raw) To: Liang Ma; +Cc: dev On Wed, Apr 14, 2021 at 02:45:06PM +0100, Liang Ma wrote: > On Wed, Apr 14, 2021 at 02:21:53PM +0100, Bruce Richardson wrote: > > On Wed, Apr 14, 2021 at 01:52:14PM +0100, Liang Ma wrote: > > > On Wed, Apr 14, 2021 at 11:48:03AM +0100, Bruce Richardson wrote: > > > > On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: > > > > > On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: > > > > > > On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: > > > > > > > Hi Bruce, > > > > > > > I look into current IOAT PMD. I found that looks only support the > > > > > > > latest DSA device which is part of SPR(Please correct me if I'm wrong). > > > > > > > However, there still is lots existing CPU that support previous CBDMA > > > > > > > Intel Quick-DATA technology. SPDK enable those device with uio. > > > > > > > Any hint to enable those CBDMA device with IOAT PMD ? > > > > > > > > > > > > > > > > > > > Hi Liang, > > > > > > > > > > > > These devices are still supported by the driver. In the latest releases the > > > > > > new device support was added but no older device support was removed. > > > > > > Please let us know if you do hit any problems with running it on the CBDMA > > > > > > HW devices, because it should all still be working as before. > > > > > > > > > > > > Regards, > > > > > > /Bruce > > > > > Hi Bruce, > > > > > I hit a problem with CBDMA enabling. > > > > > > > > > > 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with > > > > > dpdk-devbind.py > > > > > > > > > > 2. I run dpdk-test without any parameter then dpdk-test complain with > > > > > > > > > > EAL: VFIO support initialized > > > > > EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU > > > > > group bound to VFIO or unbound > > > > > Requested device 0000:00:04.0 cannot be used > > > > > > > > > > 3. kernel boot parameter include the "intel_iommu=on" > > > > > > > > > > If I still need some extra iommu settings for CBDMA device ? > > > > > > > > > It's an IOMMU limitation, not just for CBDMA but for other devices like > > > > NICs too - you need to ensure all subfunctions on a device are all bound to > > > > the same IOMMU group. So for the CBDMA instance, to use one channel with > > > > vfio, the other channels on the device also need to be bound to vfio, or > > > > else not bound to any driver. If bound to the kernel driver, there will be > > > > IOMMU conflicts as we can't split the channels between kernel and userspace > > > > address spaces. > > > > > > > > /Bruce > > > I bind whole device from 00:04.0 to 00:04.7. ioat PMD complain > > > with "ioat_rawdev_create(): ioat_rawdev_create: Channel appears > > > locked". Any hint? > > > Regards > > > Liang > > > > That should be a warning only, I think and the device should still be > > available. Is that not the case, or is there a later error? > > > > /Bruce > Good news, that's a warning only message, I checked the code. > after probe the device, rawdev_autotest all passed ! > Many thanks for your help ! > I strongly recommend to add the IOMMU hint of CBDMA(bind whole device) > into the Doc. I think DSA device don't need to do that right ? Each DSA instance appears only as a single device so this would not apply as it can't be partially in kernel and partially not. However, as I said previously, the issue is not just relevant to CBDMA, it applies equally to e.g. ixgbe or i40e NICs with two ports, or any other device where different subfunctions must be in the same IOMMU group. /Bruce ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-14 13:45 ` Liang Ma 2021-04-14 13:49 ` Bruce Richardson @ 2021-04-14 15:33 ` Burakov, Anatoly 2021-04-18 15:00 ` Liang Ma 1 sibling, 1 reply; 11+ messages in thread From: Burakov, Anatoly @ 2021-04-14 15:33 UTC (permalink / raw) To: Liang Ma, Bruce Richardson; +Cc: dev On 14-Apr-21 2:45 PM, Liang Ma wrote: > On Wed, Apr 14, 2021 at 02:21:53PM +0100, Bruce Richardson wrote: >> On Wed, Apr 14, 2021 at 01:52:14PM +0100, Liang Ma wrote: >>> On Wed, Apr 14, 2021 at 11:48:03AM +0100, Bruce Richardson wrote: >>>> On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: >>>>> On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: >>>>>> On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: >>>>>>> Hi Bruce, >>>>>>> I look into current IOAT PMD. I found that looks only support the >>>>>>> latest DSA device which is part of SPR(Please correct me if I'm wrong). >>>>>>> However, there still is lots existing CPU that support previous CBDMA >>>>>>> Intel Quick-DATA technology. SPDK enable those device with uio. >>>>>>> Any hint to enable those CBDMA device with IOAT PMD ? >>>>>>> >>>>>> >>>>>> Hi Liang, >>>>>> >>>>>> These devices are still supported by the driver. In the latest releases the >>>>>> new device support was added but no older device support was removed. >>>>>> Please let us know if you do hit any problems with running it on the CBDMA >>>>>> HW devices, because it should all still be working as before. >>>>>> >>>>>> Regards, >>>>>> /Bruce >>>>> Hi Bruce, >>>>> I hit a problem with CBDMA enabling. >>>>> >>>>> 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with >>>>> dpdk-devbind.py >>>>> >>>>> 2. I run dpdk-test without any parameter then dpdk-test complain with >>>>> >>>>> EAL: VFIO support initialized >>>>> EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU >>>>> group bound to VFIO or unbound >>>>> Requested device 0000:00:04.0 cannot be used >>>>> >>>>> 3. kernel boot parameter include the "intel_iommu=on" >>>>> >>>>> If I still need some extra iommu settings for CBDMA device ? >>>>> >>>> It's an IOMMU limitation, not just for CBDMA but for other devices like >>>> NICs too - you need to ensure all subfunctions on a device are all bound to >>>> the same IOMMU group. So for the CBDMA instance, to use one channel with >>>> vfio, the other channels on the device also need to be bound to vfio, or >>>> else not bound to any driver. If bound to the kernel driver, there will be >>>> IOMMU conflicts as we can't split the channels between kernel and userspace >>>> address spaces. >>>> >>>> /Bruce >>> I bind whole device from 00:04.0 to 00:04.7. ioat PMD complain >>> with "ioat_rawdev_create(): ioat_rawdev_create: Channel appears >>> locked". Any hint? >>> Regards >>> Liang >> >> That should be a warning only, I think and the device should still be >> available. Is that not the case, or is there a later error? >> >> /Bruce > Good news, that's a warning only message, I checked the code. > after probe the device, rawdev_autotest all passed ! > Many thanks for your help ! > I strongly recommend to add the IOMMU hint of CBDMA(bind whole device) > into the Doc. I think DSA device don't need to do that right ? > Best Regards > Liang > This is documented in our VFIO docs [1]. [1] http://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules (see Warning section) -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-14 15:33 ` Burakov, Anatoly @ 2021-04-18 15:00 ` Liang Ma 0 siblings, 0 replies; 11+ messages in thread From: Liang Ma @ 2021-04-18 15:00 UTC (permalink / raw) To: Burakov, Anatoly; +Cc: Bruce Richardson, dev On Wed, Apr 14, 2021 at 04:33:10PM +0100, Burakov, Anatoly wrote: > On 14-Apr-21 2:45 PM, Liang Ma wrote: > > On Wed, Apr 14, 2021 at 02:21:53PM +0100, Bruce Richardson wrote: > > > On Wed, Apr 14, 2021 at 01:52:14PM +0100, Liang Ma wrote: > > > > On Wed, Apr 14, 2021 at 11:48:03AM +0100, Bruce Richardson wrote: > > > > > On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: > > > > > > On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: > > > > > > > On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: > > > > > > > > Hi Bruce, > > > > > > > > I look into current IOAT PMD. I found that looks only support the > > > > > > > > latest DSA device which is part of SPR(Please correct me if I'm wrong). > > > > > > > > However, there still is lots existing CPU that support previous CBDMA > > > > > > > > Intel Quick-DATA technology. SPDK enable those device with uio. > > > > > > > > Any hint to enable those CBDMA device with IOAT PMD ? > > > > > > > > > > > > > > Hi Liang, > > > > > > > > > > > > > > These devices are still supported by the driver. In the latest releases the > > > > > > > new device support was added but no older device support was removed. > > > > > > > Please let us know if you do hit any problems with running it on the CBDMA > > > > > > > HW devices, because it should all still be working as before. > > > > > > > > > > > > > > Regards, > > > > > > > /Bruce > > > > > > Hi Bruce, > > > > > > I hit a problem with CBDMA enabling. > > > > > > > > > > > > 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with > > > > > > dpdk-devbind.py > > > > > > 2. I run dpdk-test without any parameter then dpdk-test complain with > > > > > > > > > > > > EAL: VFIO support initialized > > > > > > EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU > > > > > > group bound to VFIO or unbound > > > > > > Requested device 0000:00:04.0 cannot be used > > > > > > > > > > > > 3. kernel boot parameter include the "intel_iommu=on" > > > > > > > > > > > > If I still need some extra iommu settings for CBDMA device ? > > > > > > > > > > > It's an IOMMU limitation, not just for CBDMA but for other devices like > > > > > NICs too - you need to ensure all subfunctions on a device are all bound to > > > > > the same IOMMU group. So for the CBDMA instance, to use one channel with > > > > > vfio, the other channels on the device also need to be bound to vfio, or > > > > > else not bound to any driver. If bound to the kernel driver, there will be > > > > > IOMMU conflicts as we can't split the channels between kernel and userspace > > > > > address spaces. > > > > > > > > > > /Bruce > > > > I bind whole device from 00:04.0 to 00:04.7. ioat PMD complain > > > > with "ioat_rawdev_create(): ioat_rawdev_create: Channel appears > > > > locked". Any hint? > > > > Regards > > > > Liang > > > > > > That should be a warning only, I think and the device should still be > > > available. Is that not the case, or is there a later error? > > > > > > /Bruce > > Good news, that's a warning only message, I checked the code. > > after probe the device, rawdev_autotest all passed ! > > Many thanks for your help ! > > I strongly recommend to add the IOMMU hint of CBDMA(bind whole device) > > into the Doc. I think DSA device don't need to do that right ? > > Best Regards > > Liang > > > > This is documented in our VFIO docs [1]. > > [1] http://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules Thanks Anatoly. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [dpdk-dev] Question about enable CBDMA with ioat PMD ? 2021-04-14 10:48 ` Bruce Richardson 2021-04-14 12:52 ` Liang Ma @ 2021-04-14 15:31 ` Burakov, Anatoly 1 sibling, 0 replies; 11+ messages in thread From: Burakov, Anatoly @ 2021-04-14 15:31 UTC (permalink / raw) To: Bruce Richardson, Liang Ma; +Cc: dev On 14-Apr-21 11:48 AM, Bruce Richardson wrote: > On Wed, Apr 14, 2021 at 11:37:48AM +0100, Liang Ma wrote: >> On Tue, Apr 13, 2021 at 07:24:41PM +0100, Bruce Richardson wrote: >>> On Tue, Apr 13, 2021 at 06:36:38PM +0100, Liang Ma wrote: >>>> Hi Bruce, >>>> I look into current IOAT PMD. I found that looks only support the >>>> latest DSA device which is part of SPR(Please correct me if I'm wrong). >>>> However, there still is lots existing CPU that support previous CBDMA >>>> Intel Quick-DATA technology. SPDK enable those device with uio. >>>> Any hint to enable those CBDMA device with IOAT PMD ? >>>> >>> >>> Hi Liang, >>> >>> These devices are still supported by the driver. In the latest releases the >>> new device support was added but no older device support was removed. >>> Please let us know if you do hit any problems with running it on the CBDMA >>> HW devices, because it should all still be working as before. >>> >>> Regards, >>> /Bruce >> Hi Bruce, >> I hit a problem with CBDMA enabling. >> >> 1. I follow the DPDK IOAT driver Doc, I bind 0000.04.0 device with >> dpdk-devbind.py >> >> 2. I run dpdk-test without any parameter then dpdk-test complain with >> >> EAL: VFIO support initialized >> EAL: 0000:00:04.0 VFIO group is not viable! Not all devices in IOMMU >> group bound to VFIO or unbound >> Requested device 0000:00:04.0 cannot be used >> >> 3. kernel boot parameter include the "intel_iommu=on" >> >> If I still need some extra iommu settings for CBDMA device ? >> > It's an IOMMU limitation, not just for CBDMA but for other devices like > NICs too - you need to ensure all subfunctions on a device are all bound to > the same IOMMU group. So for the CBDMA instance, to use one channel with > vfio, the other channels on the device also need to be bound to vfio, or > else not bound to any driver. If bound to the kernel driver, there will be > IOMMU conflicts as we can't split the channels between kernel and userspace > address spaces. > > /Bruce > Yep, and depending on HW architecture, that just might not be possible (i've seen this happen before, where all devices on the platform, including things like GPU, were behind one IOMMU group). -- Thanks, Anatoly ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-04-18 15:00 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-04-13 17:36 [dpdk-dev] Question about enable CBDMA with ioat PMD ? Liang Ma 2021-04-13 18:24 ` Bruce Richardson 2021-04-14 10:37 ` Liang Ma 2021-04-14 10:48 ` Bruce Richardson 2021-04-14 12:52 ` Liang Ma 2021-04-14 13:21 ` Bruce Richardson 2021-04-14 13:45 ` Liang Ma 2021-04-14 13:49 ` Bruce Richardson 2021-04-14 15:33 ` Burakov, Anatoly 2021-04-18 15:00 ` Liang Ma 2021-04-14 15:31 ` Burakov, Anatoly
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).