From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B494FA0032; Thu, 16 Dec 2021 20:09:04 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 45F7B40143; Thu, 16 Dec 2021 20:09:04 +0100 (CET) Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by mails.dpdk.org (Postfix) with ESMTP id 39D054013F for ; Thu, 16 Dec 2021 20:09:03 +0100 (CET) Received: by mail-yb1-f175.google.com with SMTP id f9so67226982ybq.10 for ; Thu, 16 Dec 2021 11:09:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=w1HPlJazqsVSyeS13s9BEa9j97C1JmpHBv+ifZMBmFw=; b=NLYASWIhCIYq6dHxzXE1/I8lo+ezeVO01nefsJpwZaZpVRODt1SVHEtqoEk8vPHT/p 4qcU3WtV6n4J+KDUkE152+bHACKA02lL8lh0F1h6O6hliyI95Mpfeocs9DJFuha02RRr GtzryD28G+iJiM9TTfa5g3Y21WX9sAmtl0ws0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=w1HPlJazqsVSyeS13s9BEa9j97C1JmpHBv+ifZMBmFw=; b=hCMXJ4bNtlYKkUMjjZyhpfgEjGHBtC3+MbWN5MF1QqkEIMY7wgKUB5O2KAz9uTs6PQ FiUBOuuqFL8gcWc58lkwfIHpAujT+mSlbotTFWtBfNJFjPq3OZ+Qm/cMAoYqQfJahc7X 2Dmc0ctwfcFDD+OC0MNhTBQ5zMjKbKK8JZCBiLQfdt2cZjs7OGz9uXKaQVtV2P+Xv3Pi s1NuZLfJmnT9yWXQt+GEQPfHcIJmQg53ymP207lTcTMiEvYGbZtgBlj120pfwwitwyPJ o8y0WcaK6oMICDjnuYl9YiEARoRl3eeb5Tg6aa8CODEq07rW462mff5+dTjcu6PqbL4G FqAQ== X-Gm-Message-State: AOAM533lh1UqD4v8Jpt/C5eu2WvhVaX9cl67FNAJULeU9cvuzHWC3Ao3 lgHKARU96o4Dx35VWGGPfMvSNvBrSEuNLO6igQdxDA== X-Google-Smtp-Source: ABdhPJzvKvNX2cbHa/GDvpElBM3L4N0WEfYmbmvQ4omHK6n9dJBThRUMMbuxvIaSP0Qb06K37Xw+hj6R+ILjWQeRizw= X-Received: by 2002:a05:6902:707:: with SMTP id k7mr17181438ybt.295.1639681742466; Thu, 16 Dec 2021 11:09:02 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Lance Richardson Date: Thu, 16 Dec 2021 14:08:51 -0500 Message-ID: Subject: Re: Using IOAT PMD To: Bruce Richardson Cc: dev , conor.walsh@intel.com, kevin.laatz@intel.com Content-Type: text/plain; charset="UTF-8" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Thu, Dec 16, 2021 at 12:33 PM Bruce Richardson wrote: > > On Thu, Dec 16, 2021 at 11:34:25AM -0500, Lance Richardson wrote: > > On Thu, Dec 16, 2021 at 11:20 AM Bruce Richardson > > wrote: > > > > > > On Thu, Dec 16, 2021 at 11:04:54AM -0500, Lance Richardson wrote: > > > > Hi Bruce, > > > > > > > > I've been looking into using the IOAT PMD, initially with dma_autotest > > > > and the dpdk-dma example application. These seem to work fine on > > > > SKX with the current main branch, but when I try the same procedure > > > > on ICX (binding all 8 devices to vfio-pci in both cases), I get the following > > > > output for each device when probed. Is something different needed when > > > > using IOAT on ICX vs. SKX? > > > > > > > > Thanks, > > > > Lance > > > > > > > > EAL: Probe PCI driver: dmadev_ioat (8086:b00) device: 0000:80:01.0 (socket 2) > > > > IOAT: ioat_dmadev_probe(): Init 0000:80:01.0 on NUMA node 2 > > > > IOAT: ioat_dmadev_create(): ioat_dmadev_create: Channel count == 255 > > > > > > > > IOAT: ioat_dmadev_create(): ioat_dmadev_create: Channel appears locked > > > > > > > > IOAT: ioat_dmadev_create(): ioat_dmadev_create: cannot reset device. > > > > CHANCMD=0xff, CHANSTS=0xffffffffffffffff, CHANERR=0xffffffff > > > > > > > > EAL: Releasing PCI mapped resource for 0000:80:01.0 > > > > EAL: Calling pci_unmap_resource for 0000:80:01.0 at 0x4102430000 > > > > EAL: Requested device 0000:80:01.0 cannot be used > > > > > > That is strange, the same PMD should work ok on both platforms. This is all > > > on latest branch, right? Let me attempt to reproduce and get back to you. > > > > Hi Bruce, > > > > That's correct, I'm using the current tip of the main branch, which > > seems to be identical to 21.11.0. > > > > > > /Bruce > > > > > > PS: Is this a 4-socket system you are running on, since I see "socket 2" > > > being described as the socket number for device 80:01.0? > > > > > It is a two-socket system with sub-NUMA enabled, so it appears as four > > NUMA nodes. I'm only binding the devices on the second socket. > > > > Ok, [not that that should affect anything to do with ioat, AFAIK] > > Tried quickly reproducing the issue on some of our systems and failed to do > so. Does this error appear consistently, especially after a reboot? > > Thanks, > /Bruce It fails consistently after every warm reboot or power cycle. The kernel ioatdma driver always loads successfully at boot time for both sockets, but it also fails once I have bound the devices to vfio-pci and attempted to run examples/dpdk-dma. The kernel log messages are similar, both seem to read all-ones values. However, I have found that it works when binding to igb_uio instead of vfio, so maybe that's some kind of clue (vfio does work for the NIC ports). I'll continue to experiment with igb_uio, but I'm happy to gather any debug info for the vfio case if that would help. Thanks, Lance