* [dpdk-dev] Issues running Ethtool on KNI interfaces @ 2014-10-13 7:21 Deshvanth Mirle Jayaprakash (dmirleja) 2014-10-13 7:34 ` Zhang, Helin 0 siblings, 1 reply; 6+ messages in thread From: Deshvanth Mirle Jayaprakash (dmirleja) @ 2014-10-13 7:21 UTC (permalink / raw) To: dev [-- Attachment #1: Type: text/plain, Size: 1424 bytes --] Hi, Hypervisor: KVM VM: Linux OS with 2.6.32 Kernel VM Settings: 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4, Threads=1 10Gigi Passthrough Interfaces attached to VM: Intel X520 DPDK settings: DPDK Version: 1.6R2 echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages rmmod ixgbe insmod igb_uio.ko mkdir /mnt/hugepages mount -t hugetlbfs nodev /mnt/hugepages #./igb_uio_bind.py --status #./igb_uio_bind.py --bind=igb_uio 03:01.0 #./igb_uio_bind.py --bind=igb_uio 03:02.0 #./igb_uio_bind.py --status modprobe hwmon insmod rte_kni.ko ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3 --config="(0,1,3),(1,2,4)" & Having KNI process running in the background, creates two KNI interfaces called vEth0 and vEth1 which corresponds to IGB-UIO interfaces attached to the Userspace. The issue is that the Ethtool doesn't work with KNI interfaces. Is there anything that I am missing, How can I get Ethtool to work with KNI interfaces? Also, If I do not use --socket-mem OR -m EAL option, KNI application consumes all the huge pages(It doesn't mater 512/4096) assigned in the very first step.Is that expected behavior? Can I get KNI to work without assigning any Hugepages? Also, I have attached screen shots of KNI log file. Please take a look. Thanks, Desh [-- Attachment #2: KNI-SS4.png --] [-- Type: image/png, Size: 12209 bytes --] [-- Attachment #3: KNI-SS1.png --] [-- Type: image/png, Size: 15281 bytes --] [-- Attachment #4: KNI-SS2.png --] [-- Type: image/png, Size: 15239 bytes --] [-- Attachment #5: KNI-SS3.png --] [-- Type: image/png, Size: 12812 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Issues running Ethtool on KNI interfaces 2014-10-13 7:21 [dpdk-dev] Issues running Ethtool on KNI interfaces Deshvanth Mirle Jayaprakash (dmirleja) @ 2014-10-13 7:34 ` Zhang, Helin 2014-10-13 7:39 ` Deshvanth Mirle Jayaprakash (dmirleja) 0 siblings, 1 reply; 6+ messages in thread From: Zhang, Helin @ 2014-10-13 7:34 UTC (permalink / raw) To: Deshvanth Mirle Jayaprakash (dmirleja), dev Hi Desh You tried to use ethtool for KNI interfaces in VM, right? I don't think it is supported in VM. Currently it just supports ethtool for KNI interfaces in host for some igb and ixgbe NICs. Regards, Helin > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth Mirle > Jayaprakash (dmirleja) > Sent: Monday, October 13, 2014 3:21 PM > To: dev@dpdk.org > Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces > > Hi, > > Hypervisor: KVM > > VM: Linux OS with 2.6.32 Kernel > > VM Settings: > > 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config > (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4, Threads=1 > > 10Gigi Passthrough Interfaces attached to VM: Intel X520 > > > DPDK settings: > > > DPDK Version: 1.6R2 > > > echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > > rmmod ixgbe > > insmod igb_uio.ko > > mkdir /mnt/hugepages > > mount -t hugetlbfs nodev /mnt/hugepages > > #./igb_uio_bind.py --status > > #./igb_uio_bind.py --bind=igb_uio 03:01.0 > > #./igb_uio_bind.py --bind=igb_uio 03:02.0 > > #./igb_uio_bind.py --status > > modprobe hwmon > > insmod rte_kni.ko > > ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3 --config="(0,1,3),(1,2,4)" & > > > Having KNI process running in the background, creates two KNI interfaces > called vEth0 and vEth1 which corresponds to IGB-UIO interfaces attached to > the Userspace. The issue is that the Ethtool doesn't work with KNI interfaces. > Is there anything that I am missing, How can I get Ethtool to work with KNI > interfaces? Also, If I do not use --socket-mem OR -m EAL option, KNI application > consumes all the huge pages(It doesn't mater 512/4096) assigned in the very > first step.Is that expected behavior? Can I get KNI to work without assigning > any Hugepages? > > > > Also, I have attached screen shots of KNI log file. Please take a look. > > > Thanks, > > Desh ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Issues running Ethtool on KNI interfaces 2014-10-13 7:34 ` Zhang, Helin @ 2014-10-13 7:39 ` Deshvanth Mirle Jayaprakash (dmirleja) 2014-10-13 7:46 ` Zhang, Helin 0 siblings, 1 reply; 6+ messages in thread From: Deshvanth Mirle Jayaprakash (dmirleja) @ 2014-10-13 7:39 UTC (permalink / raw) To: Zhang, Helin, dev Thanks Helin, have been trying this for some time, Is there any other way I can pass IOCTLs to IGB-UIO interfaces. Can I use /dev/uio0 and /dev/uio1 ? Can I use IOCTLS on these references in Kernel to gather interface statistics, set MTU or bring up/down UIO interfaces. /Desh On 10/13/14 12:34 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote: >Hi Desh > >You tried to use ethtool for KNI interfaces in VM, right? I don't think >it is supported in VM. >Currently it just supports ethtool for KNI interfaces in host for some >igb and ixgbe NICs. > >Regards, >Helin > >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth Mirle >> Jayaprakash (dmirleja) >> Sent: Monday, October 13, 2014 3:21 PM >> To: dev@dpdk.org >> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces >> >> Hi, >> >> Hypervisor: KVM >> >> VM: Linux OS with 2.6.32 Kernel >> >> VM Settings: >> >> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config >> (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4, Threads=1 >> >> 10Gigi Passthrough Interfaces attached to VM: Intel X520 >> >> >> DPDK settings: >> >> >> DPDK Version: 1.6R2 >> >> >> echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages >> >> rmmod ixgbe >> >> insmod igb_uio.ko >> >> mkdir /mnt/hugepages >> >> mount -t hugetlbfs nodev /mnt/hugepages >> >> #./igb_uio_bind.py --status >> >> #./igb_uio_bind.py --bind=igb_uio 03:01.0 >> >> #./igb_uio_bind.py --bind=igb_uio 03:02.0 >> >> #./igb_uio_bind.py --status >> >> modprobe hwmon >> >> insmod rte_kni.ko >> >> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3 >>--config="(0,1,3),(1,2,4)" & >> >> >> Having KNI process running in the background, creates two KNI interfaces >> called vEth0 and vEth1 which corresponds to IGB-UIO interfaces attached >>to >> the Userspace. The issue is that the Ethtool doesn't work with KNI >>interfaces. >> Is there anything that I am missing, How can I get Ethtool to work with >>KNI >> interfaces? Also, If I do not use --socket-mem OR -m EAL option, KNI >>application >> consumes all the huge pages(It doesn't mater 512/4096) assigned in the >>very >> first step.Is that expected behavior? Can I get KNI to work without >>assigning >> any Hugepages? >> >> >> >> Also, I have attached screen shots of KNI log file. Please take a look. >> >> >> Thanks, >> >> Desh ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Issues running Ethtool on KNI interfaces 2014-10-13 7:39 ` Deshvanth Mirle Jayaprakash (dmirleja) @ 2014-10-13 7:46 ` Zhang, Helin 2014-10-17 7:58 ` Deshvanth Mirle Jayaprakash (dmirleja) 0 siblings, 1 reply; 6+ messages in thread From: Zhang, Helin @ 2014-10-13 7:46 UTC (permalink / raw) To: Deshvanth Mirle Jayaprakash (dmirleja), dev Hi Desh Actually KNI provide a path to exchange info/actions between user space and kernel space. You can read kni example application and KNI kernel module and KNI library for more details. It already supports setting MTU, link up/down the port, etc. Regards, Helin > -----Original Message----- > From: Deshvanth Mirle Jayaprakash (dmirleja) [mailto:dmirleja@cisco.com] > Sent: Monday, October 13, 2014 3:39 PM > To: Zhang, Helin; dev@dpdk.org > Subject: Re: Issues running Ethtool on KNI interfaces > > Thanks Helin, have been trying this for some time, Is there any other way I can > pass IOCTLs to IGB-UIO interfaces. Can I use /dev/uio0 and /dev/uio1 ? Can I > use IOCTLS on these references in Kernel to gather interface statistics, set > MTU or bring up/down UIO interfaces. > > /Desh > > On 10/13/14 12:34 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote: > > >Hi Desh > > > >You tried to use ethtool for KNI interfaces in VM, right? I don't think > >it is supported in VM. > >Currently it just supports ethtool for KNI interfaces in host for some > >igb and ixgbe NICs. > > > >Regards, > >Helin > > > >> -----Original Message----- > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth Mirle > >> Jayaprakash (dmirleja) > >> Sent: Monday, October 13, 2014 3:21 PM > >> To: dev@dpdk.org > >> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces > >> > >> Hi, > >> > >> Hypervisor: KVM > >> > >> VM: Linux OS with 2.6.32 Kernel > >> > >> VM Settings: > >> > >> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config > >> (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4, > >> Threads=1 > >> > >> 10Gigi Passthrough Interfaces attached to VM: Intel X520 > >> > >> > >> DPDK settings: > >> > >> > >> DPDK Version: 1.6R2 > >> > >> > >> echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > >> > >> rmmod ixgbe > >> > >> insmod igb_uio.ko > >> > >> mkdir /mnt/hugepages > >> > >> mount -t hugetlbfs nodev /mnt/hugepages > >> > >> #./igb_uio_bind.py --status > >> > >> #./igb_uio_bind.py --bind=igb_uio 03:01.0 > >> > >> #./igb_uio_bind.py --bind=igb_uio 03:02.0 > >> > >> #./igb_uio_bind.py --status > >> > >> modprobe hwmon > >> > >> insmod rte_kni.ko > >> > >> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3 > >>--config="(0,1,3),(1,2,4)" & > >> > >> > >> Having KNI process running in the background, creates two KNI > >>interfaces called vEth0 and vEth1 which corresponds to IGB-UIO > >>interfaces attached to the Userspace. The issue is that the Ethtool > >>doesn't work with KNI interfaces. > >> Is there anything that I am missing, How can I get Ethtool to work > >>with KNI interfaces? Also, If I do not use --socket-mem OR -m EAL > >>option, KNI application consumes all the huge pages(It doesn't mater > >>512/4096) assigned in the very first step.Is that expected behavior? > >>Can I get KNI to work without assigning any Hugepages? > >> > >> > >> > >> Also, I have attached screen shots of KNI log file. Please take a look. > >> > >> > >> Thanks, > >> > >> Desh ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Issues running Ethtool on KNI interfaces 2014-10-13 7:46 ` Zhang, Helin @ 2014-10-17 7:58 ` Deshvanth Mirle Jayaprakash (dmirleja) 2014-10-17 8:12 ` Zhang, Helin 0 siblings, 1 reply; 6+ messages in thread From: Deshvanth Mirle Jayaprakash (dmirleja) @ 2014-10-17 7:58 UTC (permalink / raw) To: Zhang, Helin, dev Hi Helin, We have been using KNI application to create Virtual interfaces. We would like to use KNI interfaces for control plane application only. As intended, We are still planning to use UIO interfaces to pass Data plane traffic which would bump up traffic throughput. We have assigned 4 vCPUs for KNI application (CoreMask is 0x1E (Core 1,2,3, and 4)). When I run top while KNI process is running in the background, see there is 100% CPU utilization on all 4 Cores. Is there a way to fix this or is this expected? In our case, the CPU utilization seems to cause lot of issues wrt to other processes running on Linux OS. Am I missing any EAL option in KNI.Also, I was wondering If there is a way to run KNI without Hugepages. The Command: ./kni -c 0x1E -n 4 --socket-mem 512 -- -p 0x3 --config="(0,1,3),(1,2,4)" & Thanks, Desh There is some issue with respect to Hugepage allocation that prevents other processes from On 10/13/14 12:46 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote: >Hi Desh > >Actually KNI provide a path to exchange info/actions between user space >and kernel space. You can read kni example application and KNI kernel >module and KNI library for more details. It already supports setting MTU, >link up/down the port, etc. > >Regards, >Helin > >> -----Original Message----- >> From: Deshvanth Mirle Jayaprakash (dmirleja) [mailto:dmirleja@cisco.com] >> Sent: Monday, October 13, 2014 3:39 PM >> To: Zhang, Helin; dev@dpdk.org >> Subject: Re: Issues running Ethtool on KNI interfaces >> >> Thanks Helin, have been trying this for some time, Is there any other >>way I can >> pass IOCTLs to IGB-UIO interfaces. Can I use /dev/uio0 and /dev/uio1 ? >>Can I >> use IOCTLS on these references in Kernel to gather interface >>statistics, set >> MTU or bring up/down UIO interfaces. >> >> /Desh >> >> On 10/13/14 12:34 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote: >> >> >Hi Desh >> > >> >You tried to use ethtool for KNI interfaces in VM, right? I don't think >> >it is supported in VM. >> >Currently it just supports ethtool for KNI interfaces in host for some >> >igb and ixgbe NICs. >> > >> >Regards, >> >Helin >> > >> >> -----Original Message----- >> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth Mirle >> >> Jayaprakash (dmirleja) >> >> Sent: Monday, October 13, 2014 3:21 PM >> >> To: dev@dpdk.org >> >> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces >> >> >> >> Hi, >> >> >> >> Hypervisor: KVM >> >> >> >> VM: Linux OS with 2.6.32 Kernel >> >> >> >> VM Settings: >> >> >> >> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config >> >> (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4, >> >> Threads=1 >> >> >> >> 10Gigi Passthrough Interfaces attached to VM: Intel X520 >> >> >> >> >> >> DPDK settings: >> >> >> >> >> >> DPDK Version: 1.6R2 >> >> >> >> >> >> echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages >> >> >> >> rmmod ixgbe >> >> >> >> insmod igb_uio.ko >> >> >> >> mkdir /mnt/hugepages >> >> >> >> mount -t hugetlbfs nodev /mnt/hugepages >> >> >> >> #./igb_uio_bind.py --status >> >> >> >> #./igb_uio_bind.py --bind=igb_uio 03:01.0 >> >> >> >> #./igb_uio_bind.py --bind=igb_uio 03:02.0 >> >> >> >> #./igb_uio_bind.py --status >> >> >> >> modprobe hwmon >> >> >> >> insmod rte_kni.ko >> >> >> >> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3 >> >>--config="(0,1,3),(1,2,4)" & >> >> >> >> >> >> Having KNI process running in the background, creates two KNI >> >>interfaces called vEth0 and vEth1 which corresponds to IGB-UIO >> >>interfaces attached to the Userspace. The issue is that the Ethtool >> >>doesn't work with KNI interfaces. >> >> Is there anything that I am missing, How can I get Ethtool to work >> >>with KNI interfaces? Also, If I do not use --socket-mem OR -m EAL >> >>option, KNI application consumes all the huge pages(It doesn't mater >> >>512/4096) assigned in the very first step.Is that expected behavior? >> >>Can I get KNI to work without assigning any Hugepages? >> >> >> >> >> >> >> >> Also, I have attached screen shots of KNI log file. Please take a >>look. >> >> >> >> >> >> Thanks, >> >> >> >> Desh > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Issues running Ethtool on KNI interfaces 2014-10-17 7:58 ` Deshvanth Mirle Jayaprakash (dmirleja) @ 2014-10-17 8:12 ` Zhang, Helin 0 siblings, 0 replies; 6+ messages in thread From: Zhang, Helin @ 2014-10-17 8:12 UTC (permalink / raw) To: Deshvanth Mirle Jayaprakash (dmirleja), dev Hi Desh > -----Original Message----- > From: Deshvanth Mirle Jayaprakash (dmirleja) [mailto:dmirleja@cisco.com] > Sent: Friday, October 17, 2014 3:58 PM > To: Zhang, Helin; dev@dpdk.org > Subject: Re: Issues running Ethtool on KNI interfaces > > Hi Helin, > > We have been using KNI application to create Virtual interfaces. We would > like to use KNI interfaces for control plane application only. As intended, We are > still planning to use UIO interfaces to pass Data plane traffic which would bump > up traffic throughput. We have assigned 4 vCPUs for KNI application (CoreMask > is 0x1E (Core 1,2,3, and 4)). When I run top while KNI process is running in the > background, see there is 100% CPU utilization on all 4 Cores. Yes, all user space threads are 100% utilized in DPDK, including KNI user space threads. I guess somebody is thinking of a good solution for this issue, but for now, it is what we can expected. > > Is there a way to fix this or is this expected? In our case, the CPU utilization > seems to cause lot of issues wrt to other processes running on Linux OS. Am I > missing any EAL option in KNI.Also, I was wondering If there is a way to run KNI > without Hugepages. See my answer above, KNI user space app is actually a special DPDK application, it is based on all DPDK technologies, including hugepages. > > The Command: > > ./kni -c 0x1E -n 4 --socket-mem 512 -- -p 0x3 --config="(0,1,3),(1,2,4)" & > > Thanks, > Desh > > There is some issue with respect to Hugepage allocation that prevents other > processes from On 10/13/14 12:46 AM, "Zhang, Helin" > <helin.zhang@intel.com> wrote: > > >Hi Desh > > > >Actually KNI provide a path to exchange info/actions between user space > >and kernel space. You can read kni example application and KNI kernel > >module and KNI library for more details. It already supports setting > >MTU, link up/down the port, etc. > > > >Regards, > >Helin > > > >> -----Original Message----- > >> From: Deshvanth Mirle Jayaprakash (dmirleja) > >> [mailto:dmirleja@cisco.com] > >> Sent: Monday, October 13, 2014 3:39 PM > >> To: Zhang, Helin; dev@dpdk.org > >> Subject: Re: Issues running Ethtool on KNI interfaces > >> > >> Thanks Helin, have been trying this for some time, Is there any other > >>way I can pass IOCTLs to IGB-UIO interfaces. Can I use /dev/uio0 and > >>/dev/uio1 ? > >>Can I > >> use IOCTLS on these references in Kernel to gather interface > >>statistics, set MTU or bring up/down UIO interfaces. > >> > >> /Desh > >> > >> On 10/13/14 12:34 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote: > >> > >> >Hi Desh > >> > > >> >You tried to use ethtool for KNI interfaces in VM, right? I don't > >> >think it is supported in VM. > >> >Currently it just supports ethtool for KNI interfaces in host for > >> >some igb and ixgbe NICs. > >> > > >> >Regards, > >> >Helin > >> > > >> >> -----Original Message----- > >> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth > >> >> Mirle Jayaprakash (dmirleja) > >> >> Sent: Monday, October 13, 2014 3:21 PM > >> >> To: dev@dpdk.org > >> >> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces > >> >> > >> >> Hi, > >> >> > >> >> Hypervisor: KVM > >> >> > >> >> VM: Linux OS with 2.6.32 Kernel > >> >> > >> >> VM Settings: > >> >> > >> >> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU > >> >> Config (SandyBridge), Manually set CPU topology: Sockets=2, > >> >> Cores=4, > >> >> Threads=1 > >> >> > >> >> 10Gigi Passthrough Interfaces attached to VM: Intel X520 > >> >> > >> >> > >> >> DPDK settings: > >> >> > >> >> > >> >> DPDK Version: 1.6R2 > >> >> > >> >> > >> >> echo 512 > > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > >> >> > >> >> rmmod ixgbe > >> >> > >> >> insmod igb_uio.ko > >> >> > >> >> mkdir /mnt/hugepages > >> >> > >> >> mount -t hugetlbfs nodev /mnt/hugepages > >> >> > >> >> #./igb_uio_bind.py --status > >> >> > >> >> #./igb_uio_bind.py --bind=igb_uio 03:01.0 > >> >> > >> >> #./igb_uio_bind.py --bind=igb_uio 03:02.0 > >> >> > >> >> #./igb_uio_bind.py --status > >> >> > >> >> modprobe hwmon > >> >> > >> >> insmod rte_kni.ko > >> >> > >> >> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3 > >> >>--config="(0,1,3),(1,2,4)" & > >> >> > >> >> > >> >> Having KNI process running in the background, creates two KNI > >> >>interfaces called vEth0 and vEth1 which corresponds to IGB-UIO > >> >>interfaces attached to the Userspace. The issue is that the > >> >>Ethtool doesn't work with KNI interfaces. > >> >> Is there anything that I am missing, How can I get Ethtool to work > >> >>with KNI interfaces? Also, If I do not use --socket-mem OR -m EAL > >> >>option, KNI application consumes all the huge pages(It doesn't > >> >>mater > >> >>512/4096) assigned in the very first step.Is that expected behavior? > >> >>Can I get KNI to work without assigning any Hugepages? > >> >> > >> >> > >> >> > >> >> Also, I have attached screen shots of KNI log file. Please take a > >>look. > >> >> > >> >> > >> >> Thanks, > >> >> > >> >> Desh > > Regards, Helin ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-10-17 8:05 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-10-13 7:21 [dpdk-dev] Issues running Ethtool on KNI interfaces Deshvanth Mirle Jayaprakash (dmirleja) 2014-10-13 7:34 ` Zhang, Helin 2014-10-13 7:39 ` Deshvanth Mirle Jayaprakash (dmirleja) 2014-10-13 7:46 ` Zhang, Helin 2014-10-17 7:58 ` Deshvanth Mirle Jayaprakash (dmirleja) 2014-10-17 8:12 ` Zhang, Helin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).