DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] Running DPDK on Google Cloud Compute Engine (GCE)
@ 2017-09-13 22:53 Rex Shang
  2017-09-16  2:03 ` Tan, Jianfeng
  0 siblings, 1 reply; 6+ messages in thread
From: Rex Shang @ 2017-09-13 22:53 UTC (permalink / raw)
  To: users

Hi all

Just wondering if anyone has successfully have DPDK running on VM inside Google Cloud.  I’ve tried similar set up on my Mac Book Pro with Virtual Box and had success.  Using similar recipe, I can’t have testpmd pass the initialization.  Before I spend too much time, just wondering if I am working on mission impossible.  ;)  Your input is greatly appreciated.

Here is my setup:
Google Cloud Compute Engine
4 vCore with 8 GB of memory
Ubuntu 16.04 LTS
3 NIC with 2 used for DPDK (driver: net_virtio); I am a bit doubtful about this NIC but so far the NIC is up with “igb_uio"

Rex

------------

EAL: Detected 4 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:04.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1000 net_virtio
EAL: PCI device 0000:00:05.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1000 net_virtio
EAL: PCI device 0000:00:06.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1000 net_virtio
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
Configuring Port 0 (socket 0)
Fail to configure port 0
EAL: Error - exiting with code: 1
  Cause: Start ports failed

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-users] Running DPDK on Google Cloud Compute Engine (GCE)
  2017-09-13 22:53 [dpdk-users] Running DPDK on Google Cloud Compute Engine (GCE) Rex Shang
@ 2017-09-16  2:03 ` Tan, Jianfeng
  2017-09-16  7:00   ` Learner Study
  2017-09-17  2:08   ` Rex Shang
  0 siblings, 2 replies; 6+ messages in thread
From: Tan, Jianfeng @ 2017-09-16  2:03 UTC (permalink / raw)
  To: Rex Shang, users

Hi Rex,


On 9/14/2017 6:53 AM, Rex Shang wrote:
> Hi all
>
> Just wondering if anyone has successfully have DPDK running on VM inside Google Cloud.  I’ve tried similar set up on my Mac Book Pro with Virtual Box and had success.  Using similar recipe, I can’t have testpmd pass the initialization.  Before I spend too much time, just wondering if I am working on mission impossible.  ;)  Your input is greatly appreciated.
>
> Here is my setup:
> Google Cloud Compute Engine
> 4 vCore with 8 GB of memory
> Ubuntu 16.04 LTS
> 3 NIC with 2 used for DPDK (driver: net_virtio); I am a bit doubtful about this NIC but so far the NIC is up with “igb_uio"
>
> Rex
>
> ------------
>
> EAL: Detected 4 lcore(s)
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:00:04.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1000 net_virtio
> EAL: PCI device 0000:00:05.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1000 net_virtio
> EAL: PCI device 0000:00:06.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 1af4:1000 net_virtio
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
> Configuring Port 0 (socket 0)
> Fail to configure port 0
> EAL: Error - exiting with code: 1
>    Cause: Start ports failed

When you start testpmd, please make sure the option, --disable-hw-vlan, 
is added.

Besides, you can apply the below patch to enable the debug mode of 
virtio to see what's wrong:

diff --git a/config/common_base b/config/common_base
index 5e97a08..5863203 100644
--- a/config/common_base
+++ b/config/common_base
@@ -328,10 +328,10 @@ CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX_FREE=n
  # Compile burst-oriented VIRTIO PMD driver
  #
  CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
-CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=n
-CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
-CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
-CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=n
+CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
+CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=y
+CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=y
+CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
  CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n

  #

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-users] Running DPDK on Google Cloud Compute Engine (GCE)
  2017-09-16  2:03 ` Tan, Jianfeng
@ 2017-09-16  7:00   ` Learner Study
  2017-09-16  7:01     ` Learner Study
  2017-09-17  2:08   ` Rex Shang
  1 sibling, 1 reply; 6+ messages in thread
From: Learner Study @ 2017-09-16  7:00 UTC (permalink / raw)
  To: Tan, Jianfeng; +Cc: Rex Shang, users

Hello Rex,

Could you please provide steps u used to configure the interfaces and
how u r starting test_pmd?
I am having somewhat similar issues on Centos7.

Thanks a lot!

On Fri, Sep 15, 2017 at 7:03 PM, Tan, Jianfeng <jianfeng.tan@intel.com> wrote:
> Hi Rex,
>
>
> On 9/14/2017 6:53 AM, Rex Shang wrote:
>>
>> Hi all
>>
>> Just wondering if anyone has successfully have DPDK running on VM inside
>> Google Cloud.  I’ve tried similar set up on my Mac Book Pro with Virtual Box
>> and had success.  Using similar recipe, I can’t have testpmd pass the
>> initialization.  Before I spend too much time, just wondering if I am
>> working on mission impossible.  ;)  Your input is greatly appreciated.
>>
>> Here is my setup:
>> Google Cloud Compute Engine
>> 4 vCore with 8 GB of memory
>> Ubuntu 16.04 LTS
>> 3 NIC with 2 used for DPDK (driver: net_virtio); I am a bit doubtful about
>> this NIC but so far the NIC is up with “igb_uio"
>>
>> Rex
>>
>> ------------
>>
>> EAL: Detected 4 lcore(s)
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> EAL: VFIO support initialized
>> EAL: PCI device 0000:00:04.0 on NUMA socket -1
>> EAL:   Invalid NUMA socket, default to 0
>> EAL:   probe driver: 1af4:1000 net_virtio
>> EAL: PCI device 0000:00:05.0 on NUMA socket -1
>> EAL:   Invalid NUMA socket, default to 0
>> EAL:   probe driver: 1af4:1000 net_virtio
>> EAL: PCI device 0000:00:06.0 on NUMA socket -1
>> EAL:   Invalid NUMA socket, default to 0
>> EAL:   probe driver: 1af4:1000 net_virtio
>> Interactive-mode selected
>> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176,
>> socket=0
>> Configuring Port 0 (socket 0)
>> Fail to configure port 0
>> EAL: Error - exiting with code: 1
>>    Cause: Start ports failed
>
>
> When you start testpmd, please make sure the option, --disable-hw-vlan, is
> added.
>
> Besides, you can apply the below patch to enable the debug mode of virtio to
> see what's wrong:
>
> diff --git a/config/common_base b/config/common_base
> index 5e97a08..5863203 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -328,10 +328,10 @@ CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX_FREE=n
>  # Compile burst-oriented VIRTIO PMD driver
>  #
>  CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=n
> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=n
> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=y
> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=y
> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
>  CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n
>
>  #
>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-users] Running DPDK on Google Cloud Compute Engine (GCE)
  2017-09-16  7:00   ` Learner Study
@ 2017-09-16  7:01     ` Learner Study
  2017-09-17  0:59       ` Rex Shang
  0 siblings, 1 reply; 6+ messages in thread
From: Learner Study @ 2017-09-16  7:01 UTC (permalink / raw)
  To: Tan, Jianfeng; +Cc: Rex Shang, users

Forgot to show the error I see in my CentOS7 setup:

[root@localhost tools]# ./dpdk-devbind.py -b ixgbe 02:05.0
Warning - no supported modules(DPDK driver) are loaded
Routing table indicates that interface 0000:02:05.0 is active. Not modifying

Please help! Thanks!

On Sat, Sep 16, 2017 at 12:00 AM, Learner Study <learner.study@gmail.com> wrote:
> Hello Rex,
>
> Could you please provide steps u used to configure the interfaces and
> how u r starting test_pmd?
> I am having somewhat similar issues on Centos7.
>
> Thanks a lot!
>
> On Fri, Sep 15, 2017 at 7:03 PM, Tan, Jianfeng <jianfeng.tan@intel.com> wrote:
>> Hi Rex,
>>
>>
>> On 9/14/2017 6:53 AM, Rex Shang wrote:
>>>
>>> Hi all
>>>
>>> Just wondering if anyone has successfully have DPDK running on VM inside
>>> Google Cloud.  I’ve tried similar set up on my Mac Book Pro with Virtual Box
>>> and had success.  Using similar recipe, I can’t have testpmd pass the
>>> initialization.  Before I spend too much time, just wondering if I am
>>> working on mission impossible.  ;)  Your input is greatly appreciated.
>>>
>>> Here is my setup:
>>> Google Cloud Compute Engine
>>> 4 vCore with 8 GB of memory
>>> Ubuntu 16.04 LTS
>>> 3 NIC with 2 used for DPDK (driver: net_virtio); I am a bit doubtful about
>>> this NIC but so far the NIC is up with “igb_uio"
>>>
>>> Rex
>>>
>>> ------------
>>>
>>> EAL: Detected 4 lcore(s)
>>> EAL: No free hugepages reported in hugepages-1048576kB
>>> EAL: Probing VFIO support...
>>> EAL: VFIO support initialized
>>> EAL: PCI device 0000:00:04.0 on NUMA socket -1
>>> EAL:   Invalid NUMA socket, default to 0
>>> EAL:   probe driver: 1af4:1000 net_virtio
>>> EAL: PCI device 0000:00:05.0 on NUMA socket -1
>>> EAL:   Invalid NUMA socket, default to 0
>>> EAL:   probe driver: 1af4:1000 net_virtio
>>> EAL: PCI device 0000:00:06.0 on NUMA socket -1
>>> EAL:   Invalid NUMA socket, default to 0
>>> EAL:   probe driver: 1af4:1000 net_virtio
>>> Interactive-mode selected
>>> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176,
>>> socket=0
>>> Configuring Port 0 (socket 0)
>>> Fail to configure port 0
>>> EAL: Error - exiting with code: 1
>>>    Cause: Start ports failed
>>
>>
>> When you start testpmd, please make sure the option, --disable-hw-vlan, is
>> added.
>>
>> Besides, you can apply the below patch to enable the debug mode of virtio to
>> see what's wrong:
>>
>> diff --git a/config/common_base b/config/common_base
>> index 5e97a08..5863203 100644
>> --- a/config/common_base
>> +++ b/config/common_base
>> @@ -328,10 +328,10 @@ CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX_FREE=n
>>  # Compile burst-oriented VIRTIO PMD driver
>>  #
>>  CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
>> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=n
>> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
>> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
>> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=n
>> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
>> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=y
>> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=y
>> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
>>  CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n
>>
>>  #
>>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-users] Running DPDK on Google Cloud Compute Engine (GCE)
  2017-09-16  7:01     ` Learner Study
@ 2017-09-17  0:59       ` Rex Shang
  0 siblings, 0 replies; 6+ messages in thread
From: Rex Shang @ 2017-09-17  0:59 UTC (permalink / raw)
  To: Learner Study; +Cc: Tan, Jianfeng, users

I used the tool “DPDK_DIR/usertools/dpdk-setup.sh" to do all of the initialization.  It is quite intuitive to use as the steps are laid out.

Before binding, you need to bring down the link.  Looks to me your network device is “up”.  You should “sudo ip link set DEVICE down” before you bind.

Rex
> On Sep 16, 2017, at 12:01 AM, Learner Study <learner.study@gmail.com> wrote:
> 
> Forgot to show the error I see in my CentOS7 setup:
> 
> [root@localhost tools]# ./dpdk-devbind.py -b ixgbe 02:05.0
> Warning - no supported modules(DPDK driver) are loaded
> Routing table indicates that interface 0000:02:05.0 is active. Not modifying
> 
> Please help! Thanks!
> 
> On Sat, Sep 16, 2017 at 12:00 AM, Learner Study <learner.study@gmail.com> wrote:
>> Hello Rex,
>> 
>> Could you please provide steps u used to configure the interfaces and
>> how u r starting test_pmd?
>> I am having somewhat similar issues on Centos7.
>> 
>> Thanks a lot!
>> 
>> On Fri, Sep 15, 2017 at 7:03 PM, Tan, Jianfeng <jianfeng.tan@intel.com> wrote:
>>> Hi Rex,
>>> 
>>> 
>>> On 9/14/2017 6:53 AM, Rex Shang wrote:
>>>> 
>>>> Hi all
>>>> 
>>>> Just wondering if anyone has successfully have DPDK running on VM inside
>>>> Google Cloud.  I’ve tried similar set up on my Mac Book Pro with Virtual Box
>>>> and had success.  Using similar recipe, I can’t have testpmd pass the
>>>> initialization.  Before I spend too much time, just wondering if I am
>>>> working on mission impossible.  ;)  Your input is greatly appreciated.
>>>> 
>>>> Here is my setup:
>>>> Google Cloud Compute Engine
>>>> 4 vCore with 8 GB of memory
>>>> Ubuntu 16.04 LTS
>>>> 3 NIC with 2 used for DPDK (driver: net_virtio); I am a bit doubtful about
>>>> this NIC but so far the NIC is up with “igb_uio"
>>>> 
>>>> Rex
>>>> 
>>>> ------------
>>>> 
>>>> EAL: Detected 4 lcore(s)
>>>> EAL: No free hugepages reported in hugepages-1048576kB
>>>> EAL: Probing VFIO support...
>>>> EAL: VFIO support initialized
>>>> EAL: PCI device 0000:00:04.0 on NUMA socket -1
>>>> EAL:   Invalid NUMA socket, default to 0
>>>> EAL:   probe driver: 1af4:1000 net_virtio
>>>> EAL: PCI device 0000:00:05.0 on NUMA socket -1
>>>> EAL:   Invalid NUMA socket, default to 0
>>>> EAL:   probe driver: 1af4:1000 net_virtio
>>>> EAL: PCI device 0000:00:06.0 on NUMA socket -1
>>>> EAL:   Invalid NUMA socket, default to 0
>>>> EAL:   probe driver: 1af4:1000 net_virtio
>>>> Interactive-mode selected
>>>> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176,
>>>> socket=0
>>>> Configuring Port 0 (socket 0)
>>>> Fail to configure port 0
>>>> EAL: Error - exiting with code: 1
>>>>   Cause: Start ports failed
>>> 
>>> 
>>> When you start testpmd, please make sure the option, --disable-hw-vlan, is
>>> added.
>>> 
>>> Besides, you can apply the below patch to enable the debug mode of virtio to
>>> see what's wrong:
>>> 
>>> diff --git a/config/common_base b/config/common_base
>>> index 5e97a08..5863203 100644
>>> --- a/config/common_base
>>> +++ b/config/common_base
>>> @@ -328,10 +328,10 @@ CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX_FREE=n
>>> # Compile burst-oriented VIRTIO PMD driver
>>> #
>>> CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
>>> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=n
>>> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
>>> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
>>> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=n
>>> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
>>> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=y
>>> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=y
>>> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
>>> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n
>>> 
>>> #
>>> 

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [dpdk-users] Running DPDK on Google Cloud Compute Engine (GCE)
  2017-09-16  2:03 ` Tan, Jianfeng
  2017-09-16  7:00   ` Learner Study
@ 2017-09-17  2:08   ` Rex Shang
  1 sibling, 0 replies; 6+ messages in thread
From: Rex Shang @ 2017-09-17  2:08 UTC (permalink / raw)
  To: Tan, Jianfeng; +Cc: users

HI Jianfeng

Thanks for the ideas.  I’ve tried “--disable-hw-vlan”, with my “testpmd” binary, it is not recognizing this option…

With the debug options for VIRTIO, I’ve now gotten the following message.  I can’t figure out whether the “vlan filtering” is the fatal error.  Does anyone know?

Configuring Port 0 (socket 0)
PMD: virtio_dev_configure(): vlan filtering not available on this host
Fail to configure port 0

Rex


> On Sep 15, 2017, at 7:03 PM, Tan, Jianfeng <jianfeng.tan@intel.com> wrote:
> 
> Hi Rex,
> 
> 
> On 9/14/2017 6:53 AM, Rex Shang wrote:
>> Hi all
>> 
>> Just wondering if anyone has successfully have DPDK running on VM inside Google Cloud.  I’ve tried similar set up on my Mac Book Pro with Virtual Box and had success.  Using similar recipe, I can’t have testpmd pass the initialization.  Before I spend too much time, just wondering if I am working on mission impossible.  ;)  Your input is greatly appreciated.
>> 
>> Here is my setup:
>> Google Cloud Compute Engine
>> 4 vCore with 8 GB of memory
>> Ubuntu 16.04 LTS
>> 3 NIC with 2 used for DPDK (driver: net_virtio); I am a bit doubtful about this NIC but so far the NIC is up with “igb_uio"
>> 
>> Rex
>> 
>> ------------
>> 
>> EAL: Detected 4 lcore(s)
>> EAL: No free hugepages reported in hugepages-1048576kB
>> EAL: Probing VFIO support...
>> EAL: VFIO support initialized
>> EAL: PCI device 0000:00:04.0 on NUMA socket -1
>> EAL:   Invalid NUMA socket, default to 0
>> EAL:   probe driver: 1af4:1000 net_virtio
>> EAL: PCI device 0000:00:05.0 on NUMA socket -1
>> EAL:   Invalid NUMA socket, default to 0
>> EAL:   probe driver: 1af4:1000 net_virtio
>> EAL: PCI device 0000:00:06.0 on NUMA socket -1
>> EAL:   Invalid NUMA socket, default to 0
>> EAL:   probe driver: 1af4:1000 net_virtio
>> Interactive-mode selected
>> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
>> Configuring Port 0 (socket 0)
>> Fail to configure port 0
>> EAL: Error - exiting with code: 1
>>   Cause: Start ports failed
> 
> When you start testpmd, please make sure the option, --disable-hw-vlan, is added.
> 
> Besides, you can apply the below patch to enable the debug mode of virtio to see what's wrong:
> 
> diff --git a/config/common_base b/config/common_base
> index 5e97a08..5863203 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -328,10 +328,10 @@ CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX_FREE=n
> # Compile burst-oriented VIRTIO PMD driver
> #
> CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=n
> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
> -CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=n
> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y
> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=y
> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=y
> +CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y
> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n
> 
> #
> 

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2017-09-17  2:08 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-13 22:53 [dpdk-users] Running DPDK on Google Cloud Compute Engine (GCE) Rex Shang
2017-09-16  2:03 ` Tan, Jianfeng
2017-09-16  7:00   ` Learner Study
2017-09-16  7:01     ` Learner Study
2017-09-17  0:59       ` Rex Shang
2017-09-17  2:08   ` Rex Shang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).