From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A072DA00C2 for ; Mon, 15 Aug 2022 17:32:34 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A6C040146; Mon, 15 Aug 2022 17:32:34 +0200 (CEST) Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) by mails.dpdk.org (Postfix) with ESMTP id 4FE2540143 for ; Mon, 15 Aug 2022 17:32:33 +0200 (CEST) Received: by mail-oi1-f171.google.com with SMTP id q184so8943560oif.1 for ; Mon, 15 Aug 2022 08:32:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc; bh=Q/ouxVrGOrmsMsSpqtAY3CsDLREv/olHytOql5jc0Lc=; b=bC2Bu1t+2AUYrThFPZQFslOPpQvnZWfG8OHuv34QeOxc6EZaUlVDJbnpF//YsD/jtv MNpX2D5NcgU432PXNyDRnXwWUXza4SaSFNW0duH+J1MjqwmdcXOTLXWaPQMTGMmaY6dz Skz40s79oJCa25ZyLjd+p1heWvRavnhLFQC2LbQ3JCMRmpNzGHISZxRoLHp7PSrHOaNR 68RG474N0d/6TLrF5Mb4nrlzfN8bNHP5mDKU8bVfec5EiALgtWRMAHnejib80m/wOlJa QSN8lWtwH0gOG76WDvqqSaNa43n03CJvP7gn8lFVExsw0hRBOxHPT/Iou8njWFKg9PA0 VxvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc; bh=Q/ouxVrGOrmsMsSpqtAY3CsDLREv/olHytOql5jc0Lc=; b=MjYh4PrAejN3LJIdxjnfE2WhH09vIHZjiFKMYicZrsFn/DM3A2jUNcyvg4E04u8uTu rsCejChkR9yN+y0WsIM5S271/sP7bZ3voSeek4v5n7lV8GVbC01fkvUBjJ2+yDkXrNGN Ev+vvvsoUqJf1kdt1ogngb+Fw3RvAriVhVqJ27tkNfa4n1Qfaw8G8yFx/WXnfyxC2GLX nKNU7yUvjFZnEdPj+aOnN0/RkQIVPIOwJOwGY8TdKO5V18XwmwuvnWEv5cHry8OewM0D YQnQbSeVsOf9gZFHOBM/APXbdSdb+1M0DxQEnN2fjzN+nNhpWHIWoNHTmAJyEaA7av6u CRSg== X-Gm-Message-State: ACgBeo2ytmRbt3udiVuRUuZcIQhvXgtwKofUmpQumDtvM9jxPayD6+ui pZrkstzqELNG85SrXfPvXhu++JjvsiuMBhXf3dGeCbqe/g8= X-Google-Smtp-Source: AA6agR7NwTZaQlO8EGf7um1zFH2pwbOdf4uDpFJ9a11WSKw7egB8l1zaXuN1e3Cb+BQxne8R9v0fcJ9NBHr9u7kLU98= X-Received: by 2002:aca:bb44:0:b0:343:6f16:f264 with SMTP id l65-20020acabb44000000b003436f16f264mr6182650oif.59.1660577552251; Mon, 15 Aug 2022 08:32:32 -0700 (PDT) MIME-Version: 1.0 From: Antonio Di Bacco Date: Mon, 15 Aug 2022 17:32:21 +0200 Message-ID: Subject: PMD for FPGA based network device To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org I need to write a PMD for an FPGA based network device based on Xilinx FPGA (Ultrascale). Just to start I tried to load a dummy bitstream that has a status register with a fixed value and a bunch of writable registers. Then, I bound the vfio-pci to the device (10ee:903f) because I plan to be able to use "user space DMA" in my PMD. I was able to mmap the fpga registers and read the status register. I can write and read back the writable registers but, when I stop my process and relaunch it, I find that the registers are all cleared (really bad). I tried to mmap the fpga using resource0 under sysfs and I'm able to write and read back the registers correctly and the registers are persistent (good). Then it is not a problem related to the bitstream but something wrong I make on the VFIO mmap or the device is reset when I start my C code to do the mmap. I also tried disabling IOMMU when loading the VFIO-PCI driver with no luck. I don't know if this is the right place to ask this question but I know there are many knowledgeable people reading this forum and I cannot find many examples of using VFIO. This is the script I use to bind vfio: VID="10ee" DID="903f" DBDF="0000:"`lspci -n | grep -E ${VID}:${DID} | cut -d ' ' -f1` echo ${DBDF} ROOT_DBDF="0000:3a:00.0" readlink /sys/bus/pci/devices/${DBDF}/iommu_group GROUP=`readlink /sys/bus/pci/devices/${DBDF}/iommu_group | rev | cut -d '/' -f1 | rev` echo "GROUP " ${GROUP} # unbind the ROOT device makes the group viable echo ${ROOT_DBDF} > /sys/bus/pci/devices/${ROOT_DBDF}/driver/unbind; sleep 1 echo ${DBDF} > /sys/bus/pci/devices/${DBDF}/driver/unbind; sleep 1 echo ${VID} ${DID} > /sys/bus/pci/drivers/vfio-pci/remove_id; sleep 1 echo ${VID} ${DID} > /sys/bus/pci/drivers/vfio-pci/new_id; sleep 1 echo 8086 2030 > /sys/bus/pci/drivers/vfio-pci/new_id; sleep 1 ls -l /sys/bus/pci/devices/${DBDF}/iommu_group/devices; sleep 1 chmod 660 /dev/vfio/vfio WHILE this is the C code I use to do the mmap: #include #include #include #include #include #include #include #include int main(int argc, char** argv) { int container, group, device, i; struct vfio_group_status group_status = { .argsz = sizeof(group_status) }; struct vfio_iommu_type1_info iommu_info = { .argsz = sizeof(iommu_info) }; struct vfio_iommu_type1_dma_map dma_map = { .argsz = sizeof(dma_map) }; struct vfio_device_info device_info = { .argsz = sizeof(device_info) }; /* Create a new container */ container = open("/dev/vfio/vfio", O_RDWR); if (ioctl(container, VFIO_GET_API_VERSION) != VFIO_API_VERSION) printf("Unknown API version\n"); /* Unknown API version */ if (!ioctl(container, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU)) printf("Doesn't support IOMMU driver we want\n"); /* Open the group */ group = open("/dev/vfio/69", O_RDWR); /* Test the group is viable and available */ ioctl(group, VFIO_GROUP_GET_STATUS, &group_status); if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) printf("Group is not viable\n"); /* Group is not viable (ie, not all devices bound for vfio) */ /* Add the group to the container */ ioctl(group, VFIO_GROUP_SET_CONTAINER, &container); /* Enable the IOMMU model we want */ ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU); /* Get addition IOMMU info */ ioctl(container, VFIO_IOMMU_GET_INFO, &iommu_info); /* Allocate some space and setup a DMA mapping */ /* dma_map.vaddr = mmap(0, 1024 * 1024, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, 0, 0); dma_map.size = 1024 * 1024; dma_map.iova = 0; dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE; ioctl(container, VFIO_IOMMU_MAP_DMA, &dma_map); */ /* Get a file descriptor for the device */ device = ioctl(group, VFIO_GROUP_GET_DEVICE_FD, "0000:3b:00.0"); /* Test and setup the device */ ioctl(device, VFIO_DEVICE_GET_INFO, &device_info); printf("NUM REGIONS %d\n", device_info.num_regions); struct vfio_region_info regs[64]; for (i = 0; i < device_info.num_regions; i++) { regs[i].argsz = sizeof(struct vfio_region_info); regs[i].index = i; ioctl(device, VFIO_DEVICE_GET_REGION_INFO, ®s[i]); printf("region %d flags %08x offset %lld size %lld\n", i, regs[i].flags, regs[i].offset, regs[i].size); /* Setup mappings... read/write offsets, mmaps * For PCI devices, config space is a region */ } volatile uint8_t* ptr = mmap(0, regs[0].size, PROT_READ | PROT_WRITE, MAP_SHARED, device, 0); printf("addr %p\n", ptr); printf("reg 0x38000 %08x\n", *(uint32_t*)(ptr + 0x38000)); { uint32_t ival = *(volatile uint32_t*) (ptr + 0x38008); *(volatile uint32_t*) (ptr + 0x38008) = ival + 0x1000; printf("%08x\n", *(volatile uint32_t*) (ptr + 0x38008)); } printf("reg 0x38008 %08x\n", *(volatile uint32_t*)(ptr + 0x38008)); }