From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9E276A00C3 for ; Tue, 25 Jan 2022 14:55:49 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6F56F426ED; Tue, 25 Jan 2022 14:55:49 +0100 (CET) Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by mails.dpdk.org (Postfix) with ESMTP id 2FAFB426E4 for ; Tue, 25 Jan 2022 14:55:48 +0100 (CET) Received: by mail-qv1-f53.google.com with SMTP id e20so17065280qvu.7 for ; Tue, 25 Jan 2022 05:55:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:from:mime-version:date:subject:message-id :to; bh=pGoCYcbTq3c10QyKmDGFzU4/jTKPg2xBnpdS+dVlrnY=; b=bEfkDi5//2yRjVlXFXRmvlIyDQHdUh7mR94yTgn/CmSeXziZ3YJzlaN+TK9vvqPrtu KLsKzhFpaEFx/EwlHeQUNLftR/WnKTpkR+GTMREO015REDQOiB7URk20S6m0l9FWutRI nYwxeUVoejUEZTkWr7hf2izQY/4jde8PW8pPZuVpvs9byg4nOQI6Z7uMWzvzZqgPVSmX gYglYCCZrBeFUjbSstD5BTtQxIcetGd3VNZfJstmIEj/Fywwx9+Je3DrKeGH0j3uYlPb 03kgrmJ9I6AajJvaTy8Rs5pXGvrfzic8gHIh6/2HjxksKHiS3yyk+TQ2fGXinjbgoNZv LUsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:content-transfer-encoding:from:mime-version:date :subject:message-id:to; bh=pGoCYcbTq3c10QyKmDGFzU4/jTKPg2xBnpdS+dVlrnY=; b=Upc12ObcC3Ywsjqz/DVZy8/RIzV0JjulLl12ngAXjOUgbdEAJ45L0ozrJ1j9eu3yhr cX7COE9peJQPitFjkwWzW6pU58ij6dAK/81IW3/imprPExwY07uxuXzLSMMzDSaA9pDQ n+DwCvUo6Nm5j6JHoFpG0jWIKqMOrHgGfJkYcmxJ5ur6RzDPIU8ihZaRKWu4fiZRMXYS zyyPB5AWpywbpRvsjvMI+M2TPqpNR07tLJfDYiKKVX8171mnCewbVTnEZluY6bf94afL oXviSNJ71cwC5uBTiBHBGeVPATwxdm0sGdjypeOj+251Y5e4SUfusS4tNECtLmQsQazO NH2w== X-Gm-Message-State: AOAM532A3E8CW8amDBrBW9Ve5ceC46gYhHDQGKt/3mu/ejNJLzmp2/gA XYkkEDlfWW3ud6SJ6bsicg5wRGj4sno= X-Google-Smtp-Source: ABdhPJwN5Jz8wdMKVStiGO0j8WZJuKPzlJbyCEVp07+a3GlZCUZI0Op1g9e/czXUhHrbbwbhlHlN+A== X-Received: by 2002:a05:6214:21e4:: with SMTP id p4mr13747283qvj.23.1643118947286; Tue, 25 Jan 2022 05:55:47 -0800 (PST) Received: from smtpclient.apple (pool-141-156-166-42.washdc.fios.verizon.net. [141.156.166.42]) by smtp.gmail.com with ESMTPSA id x14sm10105636qkp.105.2022.01.25.05.55.46 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 25 Jan 2022 05:55:46 -0800 (PST) Content-Type: multipart/alternative; boundary=Apple-Mail-D5A228DA-FE2B-4B19-ABD5-C0991E47DECD Content-Transfer-Encoding: 7bit From: Michael Mason Mime-Version: 1.0 (1.0) Date: Tue, 25 Jan 2022 08:55:46 -0500 Subject: Linux / MLX4 / ConnectX-3 / secondary process support Message-Id: <41BAAD20-8A6A-4008-BB98-B8257349A84C@gmail.com> To: users@dpdk.org X-Mailer: iPad Mail (18H107) X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --Apple-Mail-D5A228DA-FE2B-4B19-ABD5-C0991E47DECD Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable We=E2=80=99re running the configuration in the subject line but we need seco= ndary process support. however, runtime console messages and the mlx4 docs i= ndicate that mlx4 does not explicitly support this approach. (Even though I= was able to run the multi_process/simple_mp example successfully earlier to= day so that=E2=80=99s confusing =E2=80=A6) Our options appear to be: (1) use a different NIC (2) change sw architecture= (move away from using the primary/secondary process approach) There is a description of how to get around this in one of the commit messag= es for mlx4, and I=E2=80=99ve pasted it below: =E2=80=9C External memory unregistered in EAL memseg list cannot be used for= DMA unless such memory has been registered by mlx4_mr_update_ext_mp() in pr= imary process and remapped to the same virtual address in secondary process.= If the external memory is registered by primary process but has different v= irtual address in secondary process, unexpected error may happen=E2=80=9D I think I understand most of this, but does someone have an example for how t= o do this? Or could someone add some more details to this paragraph? Wrt r= emapping the address, is that just as simple as using the DPDK IPC to send a= message from primary->secondary containing the correct virtual address? La= stly, how would this additional process be created? Using something outside= of the DPDK environment? Thanks everyone!= --Apple-Mail-D5A228DA-FE2B-4B19-ABD5-C0991E47DECD Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable We=E2=80=99re running the configuration in t= he subject line but we need secondary process support.  however, runtim= e console messages and the mlx4 docs indicate that mlx4 does not explicitly s= upport this approach.  (Even though I was able to run the multi_process= /simple_mp example successfully earlier today so that=E2=80=99s confusing =E2= =80=A6)

Our options appear to be: (1) use a different NIC= (2) change sw architecture (move away from using the primary/secondary proc= ess approach)

There is a description of how to get a= round this in one of the commit messages for mlx4, and I=E2=80=99ve pasted i= t below:

=E2=80=9C External memory un= registered in EAL memseg list cannot be used for DMA unless such memory has b= een registered by mlx4_mr_update_ext= _mp() in primary process and remapped to the same= virtual address in secondary process. If the external memory is registered b= y primary process but has different virtual address in secondary process, un= expected error may happen=E2=80=9D


I think I understand most of this, but does someone have an example= for how to do this?  Or could someone add some more details to this pa= ragraph?   Wrt remapping the address, is that just as simple as using t= he DPDK IPC to send a message from primary->secondary containing the corr= ect virtual address?  Lastly, how would this additional process be crea= ted?  Using something outside of the DPDK environment?

Thanks everyone!
<= /body>= --Apple-Mail-D5A228DA-FE2B-4B19-ABD5-C0991E47DECD--