From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f44.google.com (mail-oi0-f44.google.com [209.85.218.44]) by dpdk.org (Postfix) with ESMTP id 263E820F for ; Tue, 16 May 2017 07:51:57 +0200 (CEST) Received: by mail-oi0-f44.google.com with SMTP id h4so12718570oib.3 for ; Mon, 15 May 2017 22:51:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Cq9HNXVgTkZklW8Ge3x2OH0ihUDFHwXMZSMfiAc/sHo=; b=ndUe8Og6jl8A3K5IiItJJwEFBpoQ1Jyfcq/CBcYfH3MFWFPnqw+1ROG9mT88ZbJzhU f/PqolOxH9oHUe+So5ptSiafgLgT44A5SGFWiLomcPfoZ6xxe8YJEo9VLpEc+iT4+Glt 3ZGqb8iKyoZD0e9fHC8yCxwZpyP14sPdw1Dlb88q/WdFElgA9mtjRMN1iFWEy3uzywEF 4wWDbsr6Vb0XNh810XPTweNxd1a9z0GeNeD7KB/gI1YawAQzwl9v7StZNP3GsJbMZ/6l wPRuyZ2pnyVtWS2FqjyQr3hvC3Eg74h98nfwazLy1yeTPD40tZe+j4BSmpyJdLE7Tdew 4FMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Cq9HNXVgTkZklW8Ge3x2OH0ihUDFHwXMZSMfiAc/sHo=; b=b8tdH8B5Sc9Hha+4wZw6JekCswKSB/Xz8RYd9Tx/9iI0UV2UwaAO8ANh5smxIUq10E Ij59ULkJ1dblfSeWzOLkH6exNTKt5WNk1KCLcfpFjdgmkFYu07FnMC8vu6nu3nqZb6Dq 3DmpYkDmG0PhIGh1JpbF+/jgiovwOjkZYFxyCxbzkXot6wWGp1YZjZSWPKo4oUlsqmk/ 5VlU7nSV/S/jNhtuBHIOy20DsoPppPof7e7MFZidgWeR/tEUNT4KwqMAwCCwPd8AMrUD LoFfClEzHfoMlsq0V+PwzhIyK9hCxU7sOzHlaBEsnh32QzY8wsNM3hR9diGF30SQKM4J HRog== X-Gm-Message-State: AODbwcA6SFiiSyluEz7S7+kuARv9C4APiUswhUIWm5VpJ0co4ze1MPjO vSClDNPIimdYP240ItkMGFH3+9a36K4G X-Received: by 10.202.1.14 with SMTP id 14mr1038989oib.150.1494913916463; Mon, 15 May 2017 22:51:56 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.63.215 with HTTP; Mon, 15 May 2017 22:51:56 -0700 (PDT) In-Reply-To: <80307F746F1522479831AB1253B7024E759820@IRSMSX102.ger.corp.intel.com> References: <348A99DA5F5B7549AA880327E580B435891F4159@IRSMSX101.ger.corp.intel.com> <80307F746F1522479831AB1253B7024E759820@IRSMSX102.ger.corp.intel.com> From: Chinmaya Dwibedy Date: Tue, 16 May 2017 11:21:56 +0530 Message-ID: To: "Kusztal, ArkadiuszX" Cc: "Trahe, Fiona" , "users@dpdk.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Memory requirements for crypto devices (QAT and AESNI) (using DPDK-17.02) X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 May 2017 05:51:57 -0000 Thank you Arek for corrections. On Mon, May 15, 2017 at 5:22 PM, Kusztal, ArkadiuszX < arkadiuszx.kusztal@intel.com> wrote: > Hi Chinmaya, > > > > Sorry for delayed answer. > > > > As for QAT. > > Few corrections. > > > > 2) Allocates memory for qat PMD op cookie pointer: 16384 bytes > > There will be as many cookie pointers as there is nb_descriptors (so in > this case 2048). > > Cookie pointer struct will take 704 bytes, 256 is needed for buffers in > in-place operation and 256 for out-of-place + 16 bytes which will be padded > due to 64B alignment constraint to 320 bytes and then align (320 + 320 + > 16) = 704 bytes. > > So it will be 704 * 2048 = 1441792B. > > > > Qat_session size should be of 568 bytes. > > > > Regards, > > Arek > > > > *From:* Chinmaya Dwibedy [mailto:ckdwibedy@gmail.com] > *Sent:* Wednesday, May 10, 2017 10:36 AM > *To:* Trahe, Fiona > *Cc:* users@dpdk.org; Kusztal, ArkadiuszX > > *Subject:* Re: [dpdk-users] Memory requirements for crypto devices (QAT > and AESNI) (using DPDK-17.02) > > > > Hi Fiona, > > Thanks a lot for your valuable feedback. Once again I reviewed the code > and figured out the memory requirement for crypto device. Kindly review the > below stated, feel free to suggest if something is wrong. > > *AESNI SW Crypto Device (dpdk 17.02)* > > During initialization of AESNI vdev (via rte_eal_vdev_init (), called by > DPDK application) > > 1) Allocates memzone for cryptodev data structure: 128 bytes. > > 2) Allocates memzone for cryptodev device private data: 12 bytes > > During configuration of a device (via rte_cryptodev_configure ()) > > 1) Allocates memzone for queue_pairs meta data: 8 byes. > > 2) Allocates memory required for session mempool: 2048*848= 1736704 > bytes > > (Note: Size of element: 848 bytes and Number of elements: 2048, Number of > queue pairs: 1 and Number of sessions: 2048) > > During queue pair setup (via rte_cryptodev_queue_pair_setup ()) > > 1) Allocates memzone for queue pair data structure: 52928 bytes. > > 2) Allocates memory for ring ( to place processed operations on) : > 52928 bytes > > *Total memory required per AESNI vdev: 1842708 bytes (1.757MB)* > > > > *QAT HW Crypto Device (dpdk 17.02)* > > During initialization of QAT device (via rte_cryptodev_pci_probe(),QAT > devices are discovered during the PCI probe of the EAL function which is > executed at DPDK initialization) > > 1) Allocates memzone for cryptodev data structure: 128 bytes. > > 2) Allocates memzone for cryptodev device private data: 80 bytes > > During configuring a device (via rte_cryptodev_configure ()) > > 1) Allocate memzone for queue_pairs meta data: 8 byes. > > 2) Allocates memory required for session mempool:: 2048*592= 1212416 > bytes > > (Note: Size of element: 592 bytes and Number of elements: 2048, Number of > queue pairs: 1 and Number of sessions: 2048) > > During setting up a queue pair (via rte_cryptodev_queue_pair_setup ()) > > 1) Allocates memzone for queue pair data structure:: 320 bytes. > > 2) Allocates memory for qat PMD op cookie pointer: 16384 bytes. > > 3) Allocates memory for Tx queue: 262144 bytes. > > 4) Allocates memory for Rx queue: 65536 bytes > > *Total memory required per QAT device: 1557008 bytes (1.484MB)* > > Regards, > > Chinmaya > > > > On Tue, May 9, 2017 at 9:22 PM, Trahe, Fiona > wrote: > > Hi Chinmaya, > > > > -----Original Message----- > > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Chinmaya > Dwibedy > > Sent: Monday, May 8, 2017 2:54 PM > > To: users@dpdk.org > > Subject: Re: [dpdk-users] Memory requirements for crypto devices (QAT and > > AESNI) (using DPDK-17.02) > > > > Hi, > > > > Can anyone please respond to this email ? Thank you in advance for your > > suggestion and time. > > > > Regards, > > Chinmaya > > > > On Fri, May 5, 2017 at 6:20 PM, Chinmaya Dwibedy > > wrote: > > > > > Hi All, > > > > > > > > > We are using DPK-17.02. We use crypto via hardware (QAT) and software > > > acceleration (AESNI). There is one to one mapping between crypto Dev > and > > > worker core. What are the memory requirements for the below stated > > > > > > 1) Creation of one physical Crypto device. > > > > > > 2) Creation of one AESNI MB virtual Crypto device. > > > > > > Thereafter we configure a device with the default number of queue > pairs to > > > set up for the device as shown below. > > > > > > > > > #define CDEV_MP_CACHE_SZ 64 > > > > > > rte_cryptodev_info_get(cdev_id, &info); > > > > > > dev_conf.nb_queue_pairs = info.max_nb_queue_pairs; > > > > > > dev_conf.session_mp.nb_objs = info.sym.max_nb_sessions; > > > > > > dev_conf.socket_id = SOCKET_ID_ANY; > > > > > > dev_conf.session_mp.cache_size = CDEV_MP_CACHE_SZ; > > > > > > rte_cryptodev_configure (cdev_id, &dev_conf); > > > > > > > > > How to calculate the minimum memory required to configure per HW and > per > > > SW crypto device. Then we allocate and set up a receive queue pair for > a > > > device as follows. As of now we use one queue per device and number of > > > descriptors per queue pair is set to 2k. If we increase the number of > > > descriptors, will it improve the performance in terms of throughput? > > > > > > > > [Fiona] > The QAT device can serve only a certain number of requests in parallel > which is far smaller than 2k. So increasing number of descriptors > won't speed up throughput. In fact 2k is probably excessive and could > lead to longer latency if the queue is being filled up. > I would suggest trying values of 1k, 512 and 256 and if you see no > reduction in > reduction in throughput you can use a smaller queue and save some memory. > The optimal size partly depends on how bursty your traffic is. > > > > #define CDEV_MP_NB_OBJS 2048 > > > > > > qp_conf.nb_descriptors = CDEV_MP_NB_OBJS; > > > > > > rte_cryptodev_queue_pair_setup (cdev_id, 0, &qp_conf, > dev_conf.socket_id) > > > > > > > [Fiona] Memory for each QAT queue pair (max 2 sym qps per QAT device). > TX queue = qp_conf.nb_descriptors * 128 bytes > + > RX queue = qp_conf.nb_descriptors * 32 bytes > + > op cookies (used for sgl meta-data) = qp_conf.nb_descriptors * 264 bytes > > op mempool size is totally up to the user and is not bound to any device > or PMD. > > > Session mempool is per device (though this will change in 17.08) > QAT session struct is 576 bytes long + memory for > bpi_ctx and inst pointers. > Number of sessions in the pool are passed in to rte_cryptodev_configure(). > This should be <= max_nb_sessions for that device which can be queried > using > rte_cryptodev_info_get() > > > > > > We create a session for symmetric cryptographic operations per IPsec > > > Security association. What is the memory required to hold session data > > > structure? > > > > > > > > > The intent behind this is to calculate the memory requirements in > advance > > > (before EAL initialization) and based upon the available memory, > figure out > > > how many crypto devices (note: our application initializes AESNI vdev > > > without using EAL command line option) can be initialized? Say there > are 24 > > > worker cores and we need 24 crypto AESNI vdevs. But there is no > sufficient > > > hugepage memory for creating 24 crypto AESNI vdevs. In such case, we > will > > > allocate more hugepages , then call rte_eal_init() and expect it to be > > > passed. > > > > > > > > > Thank you in advance for your suggestion and time. > > > > > > > > > > > > Regards, > > > > > > Chinmaya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >