From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sesbmg22.ericsson.net (sesbmg22.ericsson.net [193.180.251.48]) by dpdk.org (Postfix) with ESMTP id 688D820F for ; Tue, 13 Dec 2016 14:15:24 +0100 (CET) X-AuditID: c1b4fb30-037da980000054c8-08-584ff46b1374 Received: from ESESSHC002.ericsson.se (Unknown_Domain [153.88.183.24]) by (Symantec Mail Security) with SMTP id AE.13.21704.B64FF485; Tue, 13 Dec 2016 14:15:23 +0100 (CET) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (153.88.183.145) by oa.msg.ericsson.com (153.88.183.24) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 13 Dec 2016 14:15:22 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.onmicrosoft.com; s=selector1-ericsson-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=tn4XSpx7vlN+pOtB4pKQD81AAkQaCZC2cwmwf8i619c=; b=BZv9Peg5g0y9IZIYJ3FKzDdvsI4vVT86TOKxNOxPL0RtGUXSUnk+wXINAF2nLEVHGItRxexcjXFYbkd6lyDUPiEBG/jXGq4GTtONdjMp35K8Kc8exDiy5K0m9MAEXJGDlQzCKFbQYuF9goBAOUMSmCU88HryPwEQ4GGOPfm4QdM= Received: from AM4PR0701MB2146.eurprd07.prod.outlook.com (10.167.132.143) by DB5PR07MB0997.eurprd07.prod.outlook.com (10.161.200.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.761.5; Tue, 13 Dec 2016 13:15:21 +0000 Received: from AM4PR0701MB2146.eurprd07.prod.outlook.com ([10.167.132.143]) by AM4PR0701MB2146.eurprd07.prod.outlook.com ([10.167.132.143]) with mapi id 15.01.0771.014; Tue, 13 Dec 2016 13:15:20 +0000 From: Jan Wickbom To: Yuanhan Liu CC: "dev@dpdk.org" , Patrik Andersson R Thread-Topic: [PATCH v3] vhost: allow for many vhost user ports Thread-Index: AQHSVJfs+jWRdIc55Uaa107fdKpE16EFmYsAgABCI/A= Date: Tue, 13 Dec 2016 13:15:20 +0000 Message-ID: References: <1480606010-6132-1-git-send-email-jan.wickbom@ericsson.com> <1481561434-28675-1-git-send-email-jan.wickbom@ericsson.com> <20161213091454.GB18991@yliu-dev.sh.intel.com> In-Reply-To: <20161213091454.GB18991@yliu-dev.sh.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=jan.wickbom@ericsson.com; x-originating-ip: [192.176.1.95] x-microsoft-exchange-diagnostics: 1; DB5PR07MB0997; 7:h+G7JlF8jtBIrs9R//MmCJC8C1RjOIMi5gZr+bYU3ZOVohkWIVfOIUbnChBPb7vGl2TaFb5nSxiQOdEcUBHlMEU/rCeOyLe6+tQ9WEqfXGLIMW7RWmxL7KUN0Zzsvpv3wm+x3Ae95yllTiNKcSCa+OC/hpy1YH9YYLKzzg88bH1uhNZkt3m29TUKZN5D/RIYKn7TIO57GMJdCV/umLp9hzlG5uS9qm2ndr2d8Lfe0EtOfFYIkTs4uteAIx4AettBpHahyS7BnLV/fsT0IxJr/irQdNEXglilNL58WqFcWcTN+MWtmI749esBBLOu5tGNIA8Yk4Qb3e2YzDyUTZMFOwL+XOrg0bMvIUhc04w4xDabFDRJm66tzUieS8Zhneg+m9+fykzLDJyNOBKyVHqf6oaP/JIe2vsHucZZ2WKrTYyd/hgoIvZTmUWoXLizOUFf3IvvQWbKMMlNzdXWg3fkpg== x-forefront-antispam-report: SFV:SKI; SCL:-1SFV:NSPM; SFS:(10009020)(6009001)(7916002)(39450400003)(39860400002)(39410400002)(39840400002)(39850400002)(24454002)(13464003)(189002)(51444003)(199003)(33656002)(101416001)(4001430100002)(107886002)(189998001)(4326007)(2906002)(5660300001)(81166006)(68736007)(3660700001)(3280700002)(66066001)(6436002)(76176999)(122556002)(74316002)(6506006)(110136003)(305945005)(8676002)(2900100001)(92566002)(81156014)(77096006)(8936002)(50986999)(86362001)(7696004)(229853002)(105586002)(38730400001)(15395725005)(106116001)(97736004)(2950100002)(7736002)(6916009)(76576001)(106356001)(102836003)(6116002)(54356999)(3846002)(9686002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR07MB0997; H:AM4PR0701MB2146.eurprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; x-ms-office365-filtering-correlation-id: 29180a97-e52b-46b9-650c-08d4235a172d x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DB5PR07MB0997; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(37575265505322)(228905959029699); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6041248)(20161123562025)(20161123555025)(20161123564025)(20161123560025)(6072148); SRVR:DB5PR07MB0997; BCL:0; PCL:0; RULEID:; SRVR:DB5PR07MB0997; x-forefront-prvs: 01559F388D received-spf: None (protection.outlook.com: ericsson.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Dec 2016 13:15:20.7300 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR07MB0997 X-OriginatorOrg: ericsson.com X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpjleLIzCtJLcpLzFFi42KZGbFdQjf7i3+EwfI+c4t3n7YzWVyfcIHV gcnj14KlrB7zTgYGMEVx2aSk5mSWpRbp2yVwZaw7tpOxYI9mxbJLM1kbGL/JdTFyckgImEjM nPqBqYuRi0NIYB2jxJqju9lAEkICJxglHm0OB0mwCPQyS7ztfwZVNZNJ4vLuNWwQzhlGiZe7 XjOBtLAJ6EisftvBCmKLCOhKPJ2zDsxmFoiU2HHgONhYYQE7idVvTrFA1NhL9EzdDFVvJdHc dAyshkVAVeJ23wUwm1cgQeLKz00sEMu2M0pcfrkObBmngLXEtRNHwAYxCohJfD+1hglimbjE rSfzmSCeE5BYsuc8M4QtKvHy8T9WiPpoiZX7nrBAxBUkOg+8AXtNQqCHReLf77ksEM5CNolJ 21uhJvlKfFt8ihnGvvPpGlR3tkTzp14o21tiat9cRojmLiaJ+2svMkIkZCQ2z9sJDdZUieVr WxkhYSElcfdKJ+MERq1ZSC6HsHUkFuz+xAZha0ssW/iaeRY4OAQlTs58wrKAkWUVo2hxanFS brqRkV5qUWZycXF+nl5easkmRmDqOLjlt8EOxpfPHQ8xCnAwKvHwFuz2ixBiTSwrrsw9xCjB wawkwvvkg3+EEG9KYmVValF+fFFpTmrxIUZpDhYlcV6zlffDhQTSE0tSs1NTC1KLYLJMHJxS DYwWkZ7ZOgtESrLSLhn/eKdvFJS6xttA4+/jYNeUpXxmU0pkPiddWbWBR6Yls+asmtakSWu1 LxjZTal/aLtca/4Kif+OhnwLW7rMnzxPUpBaUaMcs+HWHOenslcv6mQ059m/k7FZ9n/1qxCN FvGQisB5mT1yV8SXtunlPWsNKjOeErTn3/sEZiWW4oxEQy3mouJEAHgQxDYZAwAA Subject: Re: [dpdk-dev] [PATCH v3] vhost: allow for many vhost user ports X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2016 13:15:24 -0000 > -----Original Message----- > From: Yuanhan Liu [mailto:yuanhan.liu@linux.intel.com] > Sent: den 13 december 2016 10:15 > To: Jan Wickbom > Cc: dev@dpdk.org; Patrik Andersson R > Subject: Re: [PATCH v3] vhost: allow for many vhost user ports >=20 > On Mon, Dec 12, 2016 at 05:50:34PM +0100, Jan Wickbom wrote: > > Currently select() is used to monitor file descriptors for vhostuser > > ports. This limits the number of ports possible to create since the > > fd number is used as index in the fd_set and we have seen fds > 1023. > > This patch changes select() to poll(). This way we can keep an > > packed (pollfd) array for the fds, e.g. as many fds as the size of > > the array. > > > > Also see: > > http://dpdk.org/ml/archives/dev/2016-April/037024.html > > > > Signed-off-by: Jan Wickbom > > Reported-by: Patrik Andersson > ... > > +static struct pollfd rwfds[MAX_FDS]; >=20 > Though it's unlikely, but just assume we have multiple instance of > fdset_event_dispatch(pfdset), a global rwfds will not work. >=20 > Thought twice, and it's better to put it into the fdset struct: >=20 > struct fdset { > struct fdentry fd[MAX_FDS]; > struct pollfd rwfds[MAX_FDS]; > ... >=20 Done > > /** > > * This functions runs in infinite blocking loop until there is no fd = in > > * pfdset. It calls corresponding r/w handler if there is event on the= fd. > > @@ -229,55 +217,71 @@ > > void > > fdset_event_dispatch(struct fdset *pfdset) > > { > > - fd_set rfds, wfds; > > - int i, maxfds; > > + int i; > > struct fdentry *pfdentry; > > - int num =3D MAX_FDS; > > fd_cb rcb, wcb; > > void *dat; > > int fd; > > int remove1, remove2; > > - int ret; > > > > if (pfdset =3D=3D NULL) > > return; > > > > - while (1) { > > - struct timeval tv; > > - tv.tv_sec =3D 1; > > - tv.tv_usec =3D 0; > > - FD_ZERO(&rfds); > > - FD_ZERO(&wfds); > > - pthread_mutex_lock(&pfdset->fd_mutex); > > - > > - maxfds =3D fdset_fill(&rfds, &wfds, pfdset); > > - > > - pthread_mutex_unlock(&pfdset->fd_mutex); > > + memset(rwfds, 0, sizeof(rwfds)); > > > > + while (1) { > > /* > > - * When select is blocked, other threads might > unregister > > + * When poll is blocked, other threads might > unregister > > * listenfds from and register new listenfds into > fdset. > > - * When select returns, the entries for listenfds > in the fdset > > + * When poll returns, the entries for listenfds in > the fdset > > * might have been updated. It is ok if there is > unwanted call > > * for new listenfds. > > */ > > - ret =3D select(maxfds + 1, &rfds, &wfds, NULL, > &tv); > > - if (ret <=3D 0) > > - continue; > > + poll(rwfds, pfdset->num, 1000 /* millisecs */); > > > > - for (i =3D 0; i < num; i++) { > > - remove1 =3D remove2 =3D 0; > > + for (i =3D 0; i < pfdset->num; ) { > > pthread_mutex_lock(&pfdset- > >fd_mutex); > > + > > pfdentry =3D &pfdset->fd[i]; > > fd =3D pfdentry->fd; > > + > > + if (fd < 0) { > > + /* Removed during > poll */ > > + > > + > fdset_move_last(pfdset, i); > > + > fdset_shrink(pfdset); > > + > > + > pthread_mutex_unlock(&pfdset->fd_mutex); > > + > > + continue; > > + } > > + > > + if (!rwfds[i].revents) { > > + /* No revents, > maybe added during poll */ > > + > > + rwfds[i].fd =3D fd; > > + rwfds[i].events =3D > pfdentry->rcb ? POLLIN : 0; > > + rwfds[i].events |=3D > pfdentry->wcb ? POLLOUT : 0; > > + > pthread_mutex_unlock(&pfdset->fd_mutex); > > + > > + i++; > > + continue; >=20 > I think it's error-prone to manipulate the rwfds here. Besides, it > registers an fd repeatedly. >=20 > The way I think of is: >=20 > - set rwfds[i] at fdset_add(). >=20 > This also simply makes sure that pfdset->rwfds[i] and pfdset->fd[i] is > correctly bond. >=20 > fdset_add(fdset, fd, ...) { > lock(); >=20 > i =3D fdset_find_free_slot(..); >=20 > pfdset->fd[i]->fd =3D fd; > pfdset->fd[i]->rcb =3D rcb; > pfdset->fd[i]->...; >=20 > pfdset->rwfds[i]->fd =3D fd; > pfdset->rwfds[i]->events =3D ...; > pfdset->rwfds[i]->revents =3D 0; > } >=20 >=20 > - set pfdset->fd[i]->fd =3D -1 on fdset_del. Note we should not decrease > 'num' here, as we may be at poll. >=20 > fdset_del(fdset, fd) { > lock(); >=20 > i =3D fdset_find_fd(pfdset, fd); > pfdset->fd[i]->fd =3D -1; >=20 > ... > } >=20 >=20 >=20 > - log down pfdset->num before poll, because 'num' may increase during pol= l. >=20 > I think it's optional, since even 'num' is increased during poll, it ju= st > leads to few more rwfds entries will be accessed. But it's not tracked = by > kernel, and revents is initiated with 0, that there is no issue. >=20 >=20 > - shrink the fdset and rwfds (together) for those removed entries, > __outside__ > the for loop after poll. >=20 > Works to you? I did quite of a rework of this yesterday before reading your mail. I think That should work, please have a look. V4 coming in a minute >=20 > --yliu