From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot0-f178.google.com (mail-ot0-f178.google.com [74.125.82.178]) by dpdk.org (Postfix) with ESMTP id 4B14C133F for ; Wed, 22 Mar 2017 06:32:14 +0100 (CET) Received: by mail-ot0-f178.google.com with SMTP id y88so8738540ota.2 for ; Tue, 21 Mar 2017 22:32:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=pVe+Mnmf22TLOdpnl9tDELf1hbU8h82Og8Cv/5Dnzh0=; b=RPiW+W+aYSJACX3ug05nZCcTEyrN0nRRZG79wFcWbBczh7qXCfSHj1vE+buIoqr5ju pku9x4RLzDU9aTD6T9tazUPU7WUc3RxV5/bcPZh0n5wlKoZ+ZYmcHLVKjmbt4bzfvP4Q LjouKsfVmlUx1f3QTAJ1rHaUZa4GqEKjKrYqeyHT7KVBxuGKvKVzZ0dRnuzgugi1oH+b esw7A1t0gWzgtolnCqaEPo4OEG+QhvMxqvnnSphvkhsEbLt5EFGjO6Fv1SmZeJkCBzpB A444wXMQV/wgh3Jv8XSqCzRLr4/JcoRr2WV3Np11X+f/lEqsvClvZ2SSPReYak9RAxXv rNsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=pVe+Mnmf22TLOdpnl9tDELf1hbU8h82Og8Cv/5Dnzh0=; b=hU3vBrR0lPVZ9aXMFwp4l6K+eSNadAtiXyEAutwevCE68BB9vhWNddS/4ACbaHbU8o 3YVMY82tUlJkWnTy6O7B0ty2dCBkz5Gn3miCMqxNOfO7kfBkFWBjxm/40l1EV4m+p5RI IdYh8SX5GpxLLvCfqxWyv5MVsg4mi2kBzdwvR1npPzDEl9EH4jBA38w1a94rSrQqppA9 kwVMCOQQkP0KjkmEzfpbm+K3LbofSOY/z51N41d/I5KIl5Vo0vPrTXNl4v5qD03XXBxA sLcr2dD+mZsmM0GUTTgL/RnbiROLL4xy/fLnfgKfH1YX9bjn4ShZbIM6ytMw22iyrfZL P8mA== X-Gm-Message-State: AFeK/H329BdYhqEMNtlSijpJlCIbK5rht0O9SgaRhVZQWmcS2dU9vOKQN12Cxs6E65zqLVz8+KliTuuhpvlXIQ== X-Received: by 10.157.11.108 with SMTP id p41mr23304264otd.82.1490160733360; Tue, 21 Mar 2017 22:32:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.55.195 with HTTP; Tue, 21 Mar 2017 22:32:12 -0700 (PDT) In-Reply-To: References: <20170317020611.GV18844@yliu-dev.sh.intel.com> <20170317043526.GW18844@yliu-dev.sh.intel.com> <20170317051343.GY18844@yliu-dev.sh.intel.com> <20170317052433.GZ18844@yliu-dev.sh.intel.com> <20170317054036.GB18844@yliu-dev.sh.intel.com> From: Gopakumar Choorakkot Edakkunni Date: Tue, 21 Mar 2017 22:32:12 -0700 Message-ID: To: Yuanhan Liu Cc: dev@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2017 05:32:14 -0000 Also Yuanhuan, your suggestion about the hugepage mapping / clearing memory was great .. I tried a test where I just wrote random values into the entire hugepage area and that succesfully crashed the ovs on the host :). So thats a good test to generally ensure that the guest doesnt mess up the host ! Thanks again for your suggestions. Rgds, Gopa. On Sat, Mar 18, 2017 at 4:43 PM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e@gmail.com> wrote: > I ended up implementing a mechanism to do the equivalent of a > vtpci_reset() as soon as the dpdk-app dies and just before it comes back > up. I am "hoping" that is sufficient to let the host know that the virtio > rings etc.. are unconfigured, so that when the dpdk app comes up again in > guest and does hugepage init etc.., it in theory should not confuse the > host ovs ? > > Rgds, > Gopa. > > On Sat, Mar 18, 2017 at 2:37 PM, Gopakumar Choorakkot Edakkunni < > gopakumar.c.e@gmail.com> wrote: > >> I mean vtpci_reset is called from rte_eal_pci_probe() which is the *last* >> thing in rte_eal_init(), *after* hugepage init, so if I can somehow get >> that done *before* hugepage init maybe all will be well (because I cant do >> anything to fix the host side) >> >> Rgds, >> Gopa. >> >> On Sat, Mar 18, 2017 at 2:32 PM, Gopakumar Choorakkot Edakkunni < >> gopakumar.c.e@gmail.com> wrote: >> >>> Hi Yuan, >>> >>> As a "hack"/"workaround", in rte_eal_init(), if I can call vtpci_reset() >>> just before rte_eal_memory_init(), that should take care of the problem of >>> host zeroing out hugepages right ? As of today vtpci_reset() is called in >>> rte_eal_dev_init() which comes *after* rte_eal_memory_init() >>> >>> Rgds, >>> Gopa. >>> >>> On Thu, Mar 16, 2017 at 10:50 PM, Gopakumar Choorakkot Edakkunni < >>> gopakumar.c.e@gmail.com> wrote: >>> >>>> Thanks again Yuanhan, you are the true expert!! >>>> >>>> Rgds, >>>> Gopa. >>>> >>>> On Thu, Mar 16, 2017 at 10:40 PM, Yuanhan Liu < >>>> yuanhan.liu@linux.intel.com> wrote: >>>> >>>>> On Thu, Mar 16, 2017 at 10:30:09PM -0700, Gopakumar Choorakkot >>>>> Edakkunni wrote: >>>>> > Thanks for the confirmation, glad I reached the person who knows the >>>>> nuts and >>>>> > bolts of virtio :-). So if the host is not in our control (ie if I >>>>> am just >>>>> > running as a VM on host provided by thirdparty vendor), is there any >>>>> workaround >>>>> > I can do from the guest side to prevent problems from happening on a >>>>> guest >>>>> > restart ? >>>>> >>>>> Not too much. You might want to hack the guest DPDK EAL memory >>>>> initiation >>>>> part though, to not reset the hugepage memory on start. But that's too >>>>> hacky >>>>> that I will not recommend you to do so! >>>>> >>>>> > And if theres no workarounds at all and the host has to change, >>>>> instead of >>>>> > asking the third party vendor to do a wholesale upgrade to 16.04, is >>>>> there one/ >>>>> > few commits that can be added to the host ovs-dpdk to take care of >>>>> this guest >>>>> > restart virtio-reset-before opening case ? >>>>> >>>>> Yes, backporting the commits I have mentioned should be able to fix it. >>>>> But please note that I did some code refactorings before those fixes: >>>>> it >>>>> won't apply cleanly to DPDK v2.2. >>>>> >>>>> And if you want to upgrade, I'd suggest to upgrade to v16.11, which is >>>>> LTS release. >>>>> >>>>> --yliu >>>>> > >>>>> > Rgds, >>>>> > Gopa. >>>>> > >>>>> > On Thu, Mar 16, 2017 at 10:24 PM, Yuanhan Liu < >>>>> yuanhan.liu@linux.intel.com> >>>>> > wrote: >>>>> > >>>>> > On Thu, Mar 16, 2017 at 10:20:30PM -0700, Gopakumar Choorakkot >>>>> Edakkunni >>>>> > wrote: >>>>> > > >> When I was saying dpdk version, I meant the DPDK version >>>>> with OVS. >>>>> > > >>>>> > > Oh I see! My apologies for the misuderstanding. The dpdk >>>>> version used by >>>>> > host >>>>> > > ovs should be dpdk2.2, the guest process uses dpdk16.07. The >>>>> OVS process >>>>> > is not >>>>> > > getting restarted, what is getting restarted is the guest >>>>> process using >>>>> > > dpdk16.07 - so the above clarifications you had about virtio >>>>> being >>>>> > > reset-before-opened on guest restart - does that still hold >>>>> good or does >>>>> > that >>>>> > > need the HOST side dpdk to be 16.04 or above ? >>>>> > >>>>> > Yes, the HOST dpdk should be >= v16.04. >>>>> > >>>>> > --yliu >>>>> > > >>>>> > > >> And yes, the fixes are not included in the DPDK required >>>>> for OVS 2.4. >>>>> > > >>>>> > > Thanks for the info. >>>>> > > >>>>> > > Rgds, >>>>> > > Gopa. >>>>> > > >>>>> > > On Thu, Mar 16, 2017 at 10:13 PM, Yuanhan Liu < >>>>> > yuanhan.liu@linux.intel.com> >>>>> > > wrote: >>>>> > > >>>>> > > On Thu, Mar 16, 2017 at 09:56:01PM -0700, Gopakumar >>>>> Choorakkot >>>>> > Edakkunni >>>>> > > wrote: >>>>> > > > Hi Yuanhan, >>>>> > > > >>>>> > > > Thanks for the confirmation about not having to do >>>>> anything special >>>>> > to >>>>> > > close >>>>> > > > the ports on dpdk going down or coming up. >>>>> > > > >>>>> > > > As for the question about if I met any issue of ovs >>>>> getting stuck - >>>>> > yes, >>>>> > > my >>>>> > > > guest process runs dpdk 16.07 as I mentioned earlier - >>>>> and if I >>>>> > kill my >>>>> > > guest >>>>> > > > process, then the host OVS-dpdk on the host reports >>>>> stall ! The >>>>> > OVS-dpdk >>>>> > > and >>>>> > > > emu versions I use are as below. But maybe that is >>>>> because of the >>>>> > ovs >>>>> > > missing >>>>> > > > the fixes you mentioned ? >>>>> > > >>>>> > > When I was saying dpdk version, I meant the DPDK version >>>>> with OVS. >>>>> > > >>>>> > > > ~# ovs-vswitchd --version >>>>> > > > ovs-vswitchd (Open vSwitch) 2.4.1 >>>>> > > >>>>> > > And yes, the fixes are not included in the DPDK required >>>>> for OVS 2.4. >>>>> > > >>>>> > > --yliu >>>>> > > >>>>> > > > Compiled Nov 14 2016 06:53:31 >>>>> > > > # kvm --version >>>>> > > > QEMU emulator version 2.2.0, Copyright (c) 2003-2008 >>>>> Fabrice >>>>> > Bellard >>>>> > > > ~# >>>>> > > > >>>>> > > > >>>>> > > > Rgds, >>>>> > > > Gopa. >>>>> > > > >>>>> > > > On Thu, Mar 16, 2017 at 9:35 PM, Yuanhan Liu < >>>>> > yuanhan.liu@linux.intel.com >>>>> > > > >>>>> > > > wrote: >>>>> > > > >>>>> > > > On Thu, Mar 16, 2017 at 07:48:28PM -0700, Gopakumar >>>>> Choorakkot >>>>> > > Edakkunni >>>>> > > > wrote: >>>>> > > > > Thanks a lot for the response Yuanhan. I am using >>>>> dpdk >>>>> > v16.07. So >>>>> > > what >>>>> > > > you are >>>>> > > > > saying is that in 16.07, we dont really need to >>>>> call >>>>> > > rte_eth_dev_close() >>>>> > > > on >>>>> > > > > exit, >>>>> > > > >>>>> > > > It's not about "don't really need", it's more like >>>>> "it's hard >>>>> > to". >>>>> > > Just >>>>> > > > think that it may crash at any time. >>>>> > > > >>>>> > > > > because dpdk will ensure that it will do virtio >>>>> reset before >>>>> > init >>>>> > > when it >>>>> > > > > comes up right ? >>>>> > > > >>>>> > > > No, It just handles the abnormal case well when >>>>> guest APP >>>>> > restarts. >>>>> > > > >>>>> > > > > Regarding the vhost commits you mentioned - do we >>>>> still need >>>>> > those >>>>> > > fixes >>>>> > > > if we >>>>> > > > > have the "virtio reset before init" mechanism ? >>>>> > > > >>>>> > > > Yes, we still need them: just think some malicious >>>>> guest may >>>>> > also >>>>> > > forge >>>>> > > > data like that. >>>>> > > > >>>>> > > > I'm a bit confused then. Have you actually met any >>>>> issue (like >>>>> > got >>>>> > > stucked) >>>>> > > > with DPDK v16.07? >>>>> > > > >>>>> > > > --yliu >>>>> > > > >>>>> > > > > Or that is a seperate problem >>>>> > > > > altogether (and hence we would need those fixes) ? >>>>> > > > > >>>>> > > > > Rgds, >>>>> > > > > Gopa. >>>>> > > > > >>>>> > > > > On Thu, Mar 16, 2017 at 7:06 PM, Yuanhan Liu < >>>>> > > yuanhan.liu@linux.intel.com >>>>> > > > > >>>>> > > > > wrote: >>>>> > > > > >>>>> > > > > On Thu, Mar 16, 2017 at 12:39:16PM -0700, >>>>> Gopakumar >>>>> > Choorakkot >>>>> > > > Edakkunni >>>>> > > > > wrote: >>>>> > > > > > So the doc says we should call >>>>> rte_eth_dev_close() >>>>> > *before* >>>>> > > going >>>>> > > > down. >>>>> > > > > And I >>>>> > > > > > know that especially in dpdk-virtionet in >>>>> the guest + >>>>> > > ovs-dpdk in >>>>> > > > the >>>>> > > > > host, >>>>> > > > > > the ovs ends up getting stalled/stuck (!!) >>>>> if I dont >>>>> > close >>>>> > > the port >>>>> > > > > before >>>>> > > > > > starting() it when the guest dpdk process >>>>> comes back >>>>> > up. >>>>> > > > > >>>>> > > > > I'm assuming you were using an old version, >>>>> something >>>>> > like dpdk >>>>> > > v2.2? >>>>> > > > > IIRC, DPDK v16.04 should have fixed your issue. >>>>> > > > > >>>>> > > > > > Considering that this not done properly can >>>>> screw up >>>>> > the HOST >>>>> > > ovs, >>>>> > > > and I >>>>> > > > > want >>>>> > > > > > to do everything possible to avoid that, I >>>>> want to be >>>>> > 200% >>>>> > > sure >>>>> > > > that I >>>>> > > > > call >>>>> > > > > > close even if my process gets a kill -9 .. >>>>> So obviously >>>>> > the >>>>> > > only >>>>> > > > way of >>>>> > > > > doing >>>>> > > > > > that is to close the port when the dpdk >>>>> process comes >>>>> > back up >>>>> > > and >>>>> > > > > *before* we >>>>> > > > > > init the port. rte_eth_dev_close() is not >>>>> capable of >>>>> > doing >>>>> > > that as >>>>> > > > it >>>>> > > > > expects >>>>> > > > > > the port parameters to be initialized etc.. >>>>> before it >>>>> > can be >>>>> > > > called. >>>>> > > > > >>>>> > > > > We do virtio reset before init, which is >>>>> basically what >>>>> > > > rte_eth_dev_close() >>>>> > > > > mainly does. So I see no big issue here. >>>>> > > > > >>>>> > > > > The stuck issue is due to hugepage reset by >>>>> the guest >>>>> > DPDK >>>>> > > > application, >>>>> > > > > leading all virtio vring elements being mem >>>>> zeroed. The >>>>> > old >>>>> > > vhost >>>>> > > > doesn't >>>>> > > > > handle it well, as a result, it got stuck. And >>>>> here are >>>>> > some >>>>> > > relevant >>>>> > > > > commits: >>>>> > > > > >>>>> > > > > a436f53 vhost: avoid dead loop chain >>>>> > > > > c687b0b vhost: check for ring descriptors >>>>> overflow >>>>> > > > > 623bc47 vhost: do sanity check for ring >>>>> descriptor >>>>> > length >>>>> > > > > >>>>> > > > > --yliu >>>>> > > > > >>>>> > > > > > Any other >>>>> > > > > > suggestions on what can be done to close on >>>>> restart >>>>> > rather >>>>> > > than >>>>> > > > close on >>>>> > > > > going >>>>> > > > > > down ? Thought of bouncing this by the alias >>>>> before I >>>>> > add a >>>>> > > version >>>>> > > > of >>>>> > > > > close >>>>> > > > > > myself that can do this close-on-restart >>>>> > > > > >>>>> > > > > >>>>> > > > >>>>> > > > >>>>> > > >>>>> > > >>>>> > >>>>> > >>>>> >>>> >>>> >>> >> >