From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C825D42EDA for ; Fri, 21 Jul 2023 14:14:54 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3953540E2D; Fri, 21 Jul 2023 14:14:54 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2079.outbound.protection.outlook.com [40.107.92.79]) by mails.dpdk.org (Postfix) with ESMTP id 7176840DDC for ; Fri, 21 Jul 2023 14:14:52 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fFB5terqCynLPWmFE7FQdnBv/FtQbSPoZSXODIBldq+3eGo0VBXrdM2fXJL5qE3LhX1VHAEuBLQDOVjKLe7qfGz47cPlh+yldsXijGDX3rZdG9nQceOV0xkitAOfzsO77PYs0U7MNntzpVHNaDYvf0ihI/3Z0Z1Ehbp6vhVO9kKYJYu99+T8p/Z6YoviWSLLq4uMMEdSp89QKC43CLktRprJE3qf7XoCy/TWWUtm64/8bnabOuTZuVuHXbjhhey7RwTg5uxHQJOeac5QDbc1auBoKMjAotdhj9VLhwQfWbCSBt1SpJGN7BqNTV+0CcHFG+TzpF1cWO+RJL9HiwxwvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ewA3WXz/1N+dJrRYJjMCcZ3DX36Rng3FweL7qk6Yt2g=; b=UTNxu2JrXfTejYXZQND88nDQ63x3l2kCMyMKWB7H7i20PCkGyX/q5zIA3uTW4nn2X22k4nZ8keQVVNHQ9cuntJLdeKe4pIyPGPhT6SlHoDzyalW6827kOcslEMgQ96jIDticXLUvKp631lKM94kF9kTymrdGcRUwvv7MHcxsPHMMYyVt2iXtdU09wSHF3u+J29wFoAJ3Dnkkmog4E3HVFaDWCNZaT3ZMywadY9SstSRGsVG6gxrQhg3hocRn+cRLjtYufkA6i7NVkAcaUExSSy+EH2lTrZHpe3syaQVusa1dH4X+/aJpxiwbxGFHIXVpwuDy3Azebd8FSdLxfOFN8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ewA3WXz/1N+dJrRYJjMCcZ3DX36Rng3FweL7qk6Yt2g=; b=2XiJuAzfcdr3N/ygvcEOdlByJCI4rGfYLSZpOJ1qCXSom1JBJoD9kYv4u+fiuksjEa4t+EiVlkmnETdaepbLyBfuLXVbFmWku4N/QOGt232tLUYrbRPt60m2q2Pu5/4FMbBNrOI2cM2lbEyrBTkH0Ed7ge/AbAfVLtXsf7HY/6A= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by DM4PR12MB5818.namprd12.prod.outlook.com (2603:10b6:8:62::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6609.23; Fri, 21 Jul 2023 12:14:50 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::15a9:4e83:4217:8b49]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::15a9:4e83:4217:8b49%7]) with mapi id 15.20.6609.026; Fri, 21 Jul 2023 12:14:49 +0000 Message-ID: <69c5475c-fe50-552d-7299-47d2781749aa@amd.com> Date: Fri, 21 Jul 2023 13:14:42 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Content-Language: en-US To: Igor de Paula Cc: Jochen Behrens , Thomas Monjalon , "users@dpdk.org" , Nipun Gupta , Nikhil Agarwal , Ronak Doshi , venkat.immanni@amd.com, "Varghese, Vipin" , Chenbo Xia References: <2663502.2IRrRt1zHL@thomas> <7070795.j6PcuT4dK6@thomas> <32a6b0b0-c76d-b535-e727-e62e796ca429@amd.com> From: Ferruh Yigit Subject: Re: help In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO4P265CA0109.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:2c3::13) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|DM4PR12MB5818:EE_ X-MS-Office365-Filtering-Correlation-Id: df402978-95d2-48e8-bece-08db89e41496 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5YsgHNkBm7+vz+KFe5r13q/6HblHz1i3Amz9lSSkcv2cZHXJVaN5xJm2ltQ2keGoesIdINuJtMnKexNFUxaBJ6vs6eBao+22w+h8Z5uNI5lrBciaE6pwB4fRBIbBJmtrdWLR5u2SiAUd86fkHzJ3+aTYMcd1OufdGC2tQZ10/dU8CtooEoE2yGgkDXalU7uVD6mRR8KLaRIYkdMce5o0lfdpoB6H1akMuEjy2wEjX2atJ1nqcLaGHuNYw/kegxqRpXmnsidLiG9TZ/mkO3L+yYI55SbZ9GC9QRGj5UM5oEeB9XvFgOr9jzth0upwEJKw/t2zR1tDWsqAmIb/o07B9PcgO2wLUcF2Bo6DJlfKWB11BbPkHXWpardwAz1KYPcdtsAVAU9Etep3MAZ0EISDW6X5Nh16vk+pL9osd6zUOxhNJHwlQm+zJKgpeddW/Chp0EpPIArTF4Az1Rd8hx7VVpAe+QyX0u+jUlKCW+9OjgwoKtc7p6GdLizf6x6K4J7jsPXtZu4a7+ZLI9WHcXmt5LwXmCea5Lzl0xdGF3sB5TEpAi0bxA1GMRde3u2uBb6IoCGJdLYfacvlpSG5gAYu1g/JOCbqEcm3dCPuzUHGVH67x+p9YYzmWtxpf7v76ST0DxLVhvks+j2t5yLytWXGWg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(136003)(366004)(346002)(39860400002)(376002)(396003)(451199021)(31686004)(6486002)(478600001)(54906003)(6666004)(83380400001)(31696002)(86362001)(3480700007)(2616005)(26005)(6512007)(7116003)(316002)(2906002)(186003)(36756003)(966005)(6506007)(53546011)(66476007)(30864003)(5660300002)(4326008)(66946007)(6916009)(8936002)(8676002)(41300700001)(66556008)(38100700002)(44832011)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QndnUXNabk9TbjN5eU1tT2hQR0NrOVNFV3lpTkVCNHB3dTErOU01dVBXUVF5?= =?utf-8?B?emJwUTVsZ3RKS0RSQTV3UlIrMWtEekxyekdaS3FzcDBaU3VjS244bFdCcDRi?= =?utf-8?B?M1FiVUhIOHNZZXMvbUROUll5SE9JcXZtNVAxTjZUOVhHZ3MrdU9tQUFIUVBE?= =?utf-8?B?UjRvUmJEZVplbCtEbmdET2FybERRNldLbHNCUWQ5cFFJTFVHbmVWR3ZNWlBl?= =?utf-8?B?MHJaSTh4d3YwQjVCOGxrQTFOUlEyN1Y0SEJFbHluSmRsc2FHMmFPRlBBQzlR?= =?utf-8?B?QWFMNlpFS002L0FFdis4OG9aYU5DblhGR2hVb1dEdTlHMnIraEFCSEZ0aGlL?= =?utf-8?B?azYzMThQTXdxcHA0SkNuQkFMZjFkTWxYcXJQeFNoNFJVdStTK3hOTVJ2YkNz?= =?utf-8?B?OG01UlN1emlZQ3FDMW5YNUNsWjJNU3orQ1lGRXBPTmVuSkZoeDd5V3lxdlBh?= =?utf-8?B?V1d3cWtWdGtDVHg3K3IySE1jWDVTd1haZ0tNM2w1dlRMd051S2ZwK3Z2RlZT?= =?utf-8?B?ZEF0WEp0bWFIMlJvWGhEM3A2OStpWVN0dUttWXpwdGZ6YmZwK1pjRmtiUS9Z?= =?utf-8?B?OWZvcExERXN4L2xCSjZYN3luc2cvREY2T1FGMXVlamtpenlNMjZySS9NWmts?= =?utf-8?B?bG05K0FwNkt0amZkM1B0N0FFTkl6dW5OK2x5YlN4M0d5ZFVScDhxcVJnbVl4?= =?utf-8?B?dTMrVnlPK3l3Z3I5cjI4QmpOMmowb0ZNS2srTE5JRVMrb0NBWER2ZFI5TGFn?= =?utf-8?B?aEdxK1FtdVJFYU9WL2tIa2dsMFJXRzRVTFFva3p4LzgzVUJoU3N3OW1sZVh2?= =?utf-8?B?YXFmSUI5Nms5ZFJvbkNjdFBjZk9BYmRNKzNUa2lFcHFHNkpLTjlvRXpKWit3?= =?utf-8?B?OUIzRzhqOTJVUlFGaU9TZ0NlOVllVVNZMFFoeGk1bmdnZkVXeGRVOEovV2lI?= =?utf-8?B?ZWhERHVyZjl1QkhiQU1iVXd3ekNHckJpbkZES05md1hBT01rdVIxNmVMckIr?= =?utf-8?B?c2ZEc3ZQbjNmMFRtSFIwZFhFMVI2SWNoQXNHbjY3VU96VXZGWnczbGV3RXA4?= =?utf-8?B?eHJNZEdTVGMyS1NyOCtaYjhYOHB0L3Y3Z2hKdHYxdXluYkd1SzRVbmxxV1Vy?= =?utf-8?B?YTBKNTNSWjRtNXMyVWRxVHE1T2FPMG51Z1d5cnRRTWZOdlF1dklPSnk4VWhV?= =?utf-8?B?MnE3WXBzWWhURXdWTWpta3duc2JCNVFsN0h0QWUzUGpsZVM5VERrbVNaZE4x?= =?utf-8?B?WjA0UVJHbDVtQS8ybXBpUnpweFdmdVBYNld1SUEyak5LeWVsMEpTRmhaMVM2?= =?utf-8?B?TE1sTFBmcnhPRmR1OHlNaEtzd2Mxd2NyazRxRmxkNjIyNEQwalZqUUVSdGxS?= =?utf-8?B?V2gra1BzNm9Hb0tzRkFXYVErdWZXM0Npc1dVVUtmRGdpWVNaUFUrZGRWdWIz?= =?utf-8?B?L2duUGRhM1FHblR1cUlpa3REKy82YWZab1FYV3BDOUcza2dTcXBFWXZVN0oy?= =?utf-8?B?a2R6Ri85NThTVnRsSFlzTzJmYTlyaXpFd0xWcHpXNVlvbkcyZWtPa2cxbjlK?= =?utf-8?B?VXZmd0FGelhWZ2JabFh0SWtwTmEwaC9iODhFdC9xYmxXUUxFNE9SQVNDYXgy?= =?utf-8?B?TTR4QlJWN01zanpjbVN4Y3h6dHJPREgrUmZOdFc4OFpEQVdqdHVmK05SZCtL?= =?utf-8?B?d25qS1lwa2cxd0FVOWFMWStDNWxCL1pqeGFvaE1ueUZ6RWwzR0VDK2dBemYz?= =?utf-8?B?OUxZU0Z5bThaeWxrdjEvZDBuUnJiNTdxZmoxMzdNZjN3bHFUNkxlVWIrVHFR?= =?utf-8?B?NTRvRUpYNkVBTmh1Wi9hQ2cvM0syS2dkdlJxQVRadDgzQlFGUTQxbSt6MFlq?= =?utf-8?B?VGlKTzRkczR2aDVKQTNRT2dQY1RHaXF1QWlPT2sxMUNDdE82M2lSZ255S1RN?= =?utf-8?B?d210a0RUV2s5d2FsOXlxUGRTeUJtT0hEaDkrdnBOK1Q3NWNNb1dwem5IcmFn?= =?utf-8?B?L0NPWWxWdEJ0QXlacm40SmtsamdHOWd2YW5DVjE4T1hGNlZFYXduUnZ4OHVv?= =?utf-8?B?WjJONWM5MkxOZEZvUUhHVVJva2RnY0pHSDVnYkd4N3JSQnZFK1MvWExESWRO?= =?utf-8?Q?TOsb/DgdRUJ1uDnFInIixAyvm?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: df402978-95d2-48e8-bece-08db89e41496 X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2023 12:14:49.6275 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rruza1ChhGTrdtZVz+5iqvNFVpRbKNafOI4uUdh0k9emB2FdsjBD2TJm8YYE+K3l X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5818 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On 7/21/2023 12:39 PM, Igor de Paula wrote: > I am trying to use virtio_user for an interface with the > kernel: https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html > I think this requires IOVA as va. > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo, he may know better. Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and '--iova-mode=pa'? > It does work with Intel host and IOMMU > enabled. Part of the negotiation when setting it up is getting the IOMMU > number so I thought it has to have IOMMU. > Yes, issue looks like related to the IOMMU, and it may be either related to HW support, or ESXi iommu driver support, we will check using below information you provided. > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > again, that didn't work. > ESXI version - VMware ESXi, 7.0.0, 16324942 > AMD:  AMD EPYC 7452 32-Core Processo > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > Regarding the logs I will try and attach it soon. > Thanks for info, waiting for logs. > > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit > wrote: > > Hi Igor, > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you > please try with: > - enable 'enable_unsafe_noiommu_mode' flag > - Force PA mode via '--iova-mode=pa' eal parameter > > > Also to be able to figure out AMD IOMMU support level, can you please > provide: > - AMD part number > - ESXi/hypervisor version > - ESXi & VM kernel logs > > > Thanks, > Ferruh > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > +Ronak from the ESX team > > > >   > > > >   > > > > In our usage, we do set amd_iommu=off in the boot command line from > > grub. (Or intel_iommu=off for Intel processors.) > > > >   > > > >                 Jochen > > > >   > > > > *From: *Thomas Monjalon > > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > *To: *Igor de Paula > > > *Cc: *users@dpdk.org >, Jochen Behrens > > >, Nipun Gupta > >, Nikhil Agarwal > > >, Ferruh > Yigit > > > *Subject: *Re: help > > > > !! External Email > > > > +Cc some AMD maintainers, they can have an idea about IOMMU settings. > > > > > > 20/07/2023 14:44, Igor de Paula: > >> I have enabled it in the host and in the BIOS for AMD... > >> In the Bios I changed to amd_iommu=on and in the host it's the > same for > >> either. > >> > >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon > > wrote: > >> > >> > 20/07/2023 11:35, Igor de Paula: > >> > > The weird thing is that it only happens when I am using a > host with an > >> > AMD > >> > > processor. It doesn't happen when I use a host with an Intel > processor. > >> > > >> > So it's probably a matter of BIOS settings for the IOMMU? > >> > > >> > > >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon > > > >> > > wrote: > >> > > > >> > > > +Cc the vmxnet3 maintainer. > >> > > > > >> > > > Please Jochen, do you have an idea what's wrong below? > >> > > > > >> > > > > >> > > > 20/07/2023 11:25, Igor de Paula: > >> > > > > This is because it can't negotiate the IOMMU type with > any port. > >> > > > > > >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > > >> > > > >> > > > wrote: > >> > > > > > >> > > > > > Hello, > >> > > > > > > >> > > > > > The first error is "Cause: Error: number of ports must > be even" > >> > > > > > > >> > > > > > > >> > > > > > 03/05/2023 18:13, Igor de Paula: > >> > > > > > > I am running a VM inside a VMWARE server (vSphere). > >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a > >> > > > virtio_user to > >> > > > > > > interact with the kernel stack. > >> > > > > > > In another app I have it working but instead of > virtio_user I am > >> > > > running > >> > > > > > > KNI, it works in IOVA-PA mode. > >> > > > > > > I am looking to replace the KNI. > >> > > > > > > > >> > > > > > > When I try to set up virtio_user port as in the doc: > >> > > > > > > > >> > > > > > > >> > > > > >> > > https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path > > >> > > > > > > I get a error it can't run in PA mode. > >> > > > > > > > >> > > > > > > > >> > > > > > > When I try to run as VA mode from a parameter, I get the > >> > following > >> > > > > > errors: > >> > > > > > > EAL: lib.eal log level changed from info to debug > >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > >> > > > > > > EAL: Support maximum 128 logical core(s) by > configuration. > >> > > > > > > EAL: Detected 2 lcore(s) > >> > > > > > > EAL: Detected 1 NUMA nodes > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so' > >> > > > > > > EAL: Detected static linkage of DPDK > >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes > >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000) > >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > >> > > > > > > EAL: DPAA Bus not present. Skipping. > >> > > > > > > EAL: VFIO PCI modules not loaded > >> > > > > > > EAL: Selected IOVA mode 'VA' > >> > > > > > > EAL: Probing VFIO support... > >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported > >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > >> > > > > > > EAL: VFIO support initialized > >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes > >> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000) > >> > > > > > > EAL: Setting up physically contiguous memory... > >> > > > > > > EAL: Setting maximum number of open files to 1048576 > >> > > > > > > EAL: Detected memory type: socket_id:0 > hugepage_sz:1073741824 > >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > >> > > > > > hugepage_sz:1073741824 > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > >> > > > > > > EAL: Memseg list allocated at socket 0, page size > 0x100000kB > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > >> > > > > > > EAL: Virtual area found at 0x140000000 (size = > 0x2000000000) > >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size > 2000000000 > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > >> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000) > >> > > > > > > EAL: Memseg list allocated at socket 0, page size > 0x100000kB > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > >> > > > > > > EAL: Virtual area found at 0x2180000000 (size = > 0x2000000000) > >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, > size 2000000000 > >> > > > > > > EAL: TSC frequency is ~2350000 KHz > >> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > >> > > > > > > EAL: Trying to obtain current memory policy. > >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > >> > > > > > > EAL: Restoring previous memory policy: 0 > >> > > > > > > EAL: request: mp_malloc_sync > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is > 'VA', not > >> > > > > > initializing > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL:   Expecting 'PA' IOVA mode but current mode is > 'VA', not > >> > > > > > initializing > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > >> > > > > > > EAL: Bus (pci) probe failed. > >> > > > > > > EAL: lib.telemetry log level changed from disabled to > warning > >> > > > > > > EAL: Error - exiting with code: 1 > >> > > > > > >   Cause: Error: number of ports must be even > >> > > > > > > EAL: request: mp_malloc_sync > >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > For some reason the HW ports won't setup. From what I > understand > >> > > > > > > net_vmxnet3 should work with VA mode. > >> > > > > > > I enabled I/OMUU for the VM. > >> > > > > > > The weird thing even when enabled, I still have the > >> > > > > > > enable_unsafe_noiommu_mode flag on. > >> > > > > > > And because it's on the this: > >> > > > > > > > >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev); > >> > > > > > > > >> > > > > > > return PA mode, and it fails. > >> > > > > > > > >> > > > > > > When I disable it by modifying > >> > > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get > >> > another > >> > > > > > error. > >> > > > > > > The error is that it doesn't find a suitable IOMMU type: > >> > > > > > > Just putting the relevant message: > >> > > > > > > > >> > > > > > > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No > such device) > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No > such device) > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No > such > >> > device) > >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > >> > > > > > > EAL:   probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No > such device) > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No > such device) > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No > such > >> > device) > >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > >> > > > > > > > >> > > > > > > > >> > > > > > > I can see where it fails: > >> > > > > > > > >> > > > > > > > >> > > > > > > const struct vfio_iommu_type * > >> > > > > > > vfio_set_iommu_type(int vfio_container_fd) > >> > > > > > > { > >> > > > > > > unsigned idx; > >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) { > >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx]; > >> > > > > > > > >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, > >> > > > > > > t->type_id); > >> > > > > > > if (!ret) { > >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > >> > > > > > > t->type_id, t->name); > >> > > > > > > return t; > >> > > > > > > } > >> > > > > > > /* not an error, there may be more supported IOMMU > types */ > >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, > error " > >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno, > >> > > > > > > strerror(errno)); > >> > > > > > > } > >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */ > >> > > > > > > return NULL; > >> > > > > > > } > >> > > > > > > > >> > > > > > > > >> > > > > > > The ioctl returns -1. > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > I tried many things. > >> > > > > > > Enabling/disabling IOMMU. > >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag. > >> > > > > > > Running in PA mode. > >> > > > > > > Any help will be much appreciated. > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > > > > > > > > > > > > !! External Email: This email originated from outside of the > > organization. Do not click links or open attachments unless you > > recognize the sender. > > >