From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0674CA00C5 for ; Tue, 19 Jul 2022 18:18:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 77B9740A8B; Tue, 19 Jul 2022 18:18:28 +0200 (CEST) Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) by mails.dpdk.org (Postfix) with ESMTP id BB2ED40A8A for ; Tue, 19 Jul 2022 18:18:27 +0200 (CEST) Received: by mail-ej1-f50.google.com with SMTP id ez10so28088462ejc.13 for ; Tue, 19 Jul 2022 09:18:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=LeheG5QyUzxjfY4QekJmS/qVwy1jexR2FoJY0LfYYzw=; b=WKSfQWn53QvcTE90Z8l/oNZxhmEmiEo89R2tbapbxPTCi2UiBpyVy0DVoPt3oEbJbr UUp2Oy6A6cCB/l8ypM17UzE/ZwzDyl0D1UYHvyrUcDjhX3O5YohDYlDzs6x3Rx9FB1n0 B+1vNOXgXyTUcrCPTweTZDAvNx9kr01o1rtPd/NRn1DgCQtjyKE0D1xaD9cuPTQbpfNT qzOyN++TtO7yhIaCGsN956gwei5aDsaWjVf5pGrUDNKwrgZ6DGk42nzuNoREMKGV+4HW tPUQgUUAfqma1f05wlMzROIjA6CMmYmOYI5sObXKDDYYn3cCzI916WsyJ+cZw2ULnQBK ZIpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=LeheG5QyUzxjfY4QekJmS/qVwy1jexR2FoJY0LfYYzw=; b=AGnfX79wCFJkQEK7NLwsjoouV1K/ceg7A8BcqnB19KF2Vg4r3phhnHbjxqNi5nxq/Q eeDmzG+mZywtT3N48QaFZ5MwIX5EtcKtgI43q9hsOCRaxZU6zWLp8gUtEgJoVksa9ujw npYQHO74WpUK/rEm2imGRR4HoqFdMlzo+xABCgyoiPfe5CjehojeeJNOi4HCw40jMLqW wxiZnNXNQraGZ2sWqja3hTBSdzJNOMYzwlumkJpsQ7tG1ihQ7GBbQhbbUNHW2avJ5YXp ms8mu+FEc5OROBWDd/fEx6cC1R3j7o2vFkSr4znNnda6jMkwU3cXiSL3HvWaLihT8wV4 MQ1w== X-Gm-Message-State: AJIora8fKE4vb9BvMoAve7qHQM9DdQsyD5l5WnAV3fdtX2w7Q344DM5G esIre1PIQCaW1WU2CD2GOM1NA2YL+oQ32VCc40L2QJfg55qFqw== X-Google-Smtp-Source: AGRyM1v4/Mi42j9e13UPviGkszLMtKUozFHs4UGPMHsQVt4hW4iODDLzZNUOBWCGonGQtg0bUz2U7nL8qFSZhu0Ts7Q= X-Received: by 2002:a17:906:cc4a:b0:72b:863e:ef7c with SMTP id mm10-20020a170906cc4a00b0072b863eef7cmr30185692ejb.686.1658247507033; Tue, 19 Jul 2022 09:18:27 -0700 (PDT) MIME-Version: 1.0 From: Nobin Mathew Date: Tue, 19 Jul 2022 21:48:15 +0530 Message-ID: Subject: VF is still resetting To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, We are running a dpdk app inside a pod, and orchestrating the app very frequently(test app). 1/100 or so we are getting an error: 2022-07-17T22:34:24.620291289+03:00 iavf_check_vf_reset_done(): reset VFR value: 3 2022-07-17T22:34:24.620310455+03:00 iavf_init_vf(): VF is still resetting 2022-07-17T22:34:24.620339697+03:00 iavf_dev_init(): Init vf failed 2022-07-17T22:34:24.620390802+03:00 EAL: Releasing PCI mapped resource for 0000:3b:0f.5 2022-07-17T22:34:24.620397381+03:00 EAL: Calling pci_unmap_resource for 0000:3b:0f.5 at 0x2101000000 2022-07-17T22:34:24.620442514+03:00 EAL: Calling pci_unmap_resource for 0000:3b:0f.5 at 0x2101010000 2022-07-17T22:34:24.729012277+03:00 EAL: Requested device 0000:3b:0f.5 cannot be used 2022-07-17T22:34:24.729028758+03:00 EAL: Bus (pci) probe failed. we added one log in dpdk lib to print the VFGEN_RSTAT register of the VF. In problematic cases, we are seeing the value 3 which maps to 0xDEADBEEF / VF reset states - these are written into the RSTAT register: * VFGEN_RSTAT on the VF * When the PF initiates a reset, it writes 0 * When the reset is complete, it writes 1 * When the PF detects that the VF has recovered, it writes 2 * VF checks this register periodically to determine if a reset has occurred, * then polls it to know when the reset is complete. * If either the PF or VF reads the register while the hardware * is in a reset state, it will return DEADBEEF, which, when masked * will result in 3. / enum virtchnl_vfr_states { VIRTCHNL_VFR_INPROGRESS = 0, VIRTCHNL_VFR_COMPLETED, VIRTCHNL_VFR_VFACTIVE, }; We tried this patch also, increasing the poll time, no help. https://github.com/DPDK/dpdk/commit/be7226980c9ad4963b92b489c8afb17f08899953 Details of the setup: DPDK library version 21.11 VF Driver:- intel-iavf version 4.0.1-3.2 PF driver:- sudo ethtool -i enp94s0f1 driver: i40e version: 2.14.13 firmware-version: 8.15 0x800096ca 20.0.17 Since we are seeing 0xDEADBEEF, I am assuming VF-PF reset mailbox msg is received by PF, and PF initiated the RESET sequence by writing VFSWR to VPGEN_VFRTRIG register. I am not seeing " dev_err(&pf->pdev->dev, "VF reset check timeout on VF %d\n", " anywhere in syslog. Any pointers?, why does this happen(why VF reset is not complete)?... One more question, what is the sequence of calls in the reset path? i40e_vc_process_vf_msg() -> VIRTCHNL_OP_RESET_VF i40e_vc_reset_vf() -> i40e_reset_vf() -> i40e_trigger_vf_reset() & i40e_cleanup_reset_vf() this one? -Nobin