DPDK patches and discussions
 help / color / mirror / Atom feed
* RE: TSC for dmaengine API batch support for DSA (linux-idxd-ka2 @ 211126)
@ 2021-11-29 19:11 Ananyev, Konstantin
  0 siblings, 0 replies; only message in thread
From: Ananyev, Konstantin @ 2021-11-29 19:11 UTC (permalink / raw)
  To: Ananyev, Konstantin; +Cc: dev


 1.7 GHz SPR, 'D' stepping
 1 DSA device, 1 WQ (size=128, 1 engine, mode="dedicated", type="kernel", name="dmaengine")
 io_uring-bench with some mods:
 1 thread, 1 io_uring context (sq_ring=64,cq_ring=128), random reads over one 1GB file on tmpfs

 Without touching data in user-space:
                  SW copy (KIOPS)       DMA copy (irq) (KIOPS)        DMA copy (poll) (KIOPS)
1KB		1488.8			583.9			1189.0
2KB		1300.7			582.0			1170.8
4KB		1045.5			580.1			1214.8
8KB		  709.7			532.2			  952.3
16KB		  393.3			441.1			  690.7
32KB		  203.2			313.6			  434.2
64KB		    91.5 			229.3			  266.8
128KB		    46.1			128.9			  155.2
 
With touching data in user-space:
                  SW copy (KIOPS)       DMA copy (irq) (KIOPS)        DMA copy (poll) (KIOPS)
1KB		1281.4			586.8			1157.0
2KB		1129.0			603.9			1058.1
4K		  888.9			615.7			  935.6
8KB		  506.2   		511.8			  757.8
16KB		  309.4			363.1			  472.6
32KB		  131.1			221.4			  275.2
64KB		    65.7			147.2			  152.8
128KB		    31.4			  79.5			    76.2

FS layer					1KB	4KB	8KB	16KB	32KB	128KB
-----------
read_iter (get_page + sw copy)		986	1367	2194	3994	8237	35441	
read_iter (get_page)			442	459	652	1089	1852	  5599
sw copy (deduced)			542	908	1542	2905	6385	29842

io_uring layer
-------------------
alloc dma_copy_vec			222	214	244	248	284	381
dma_map_sg				130	132	146	210	296	1034
cleanup dma_copy_vec			208	303	375	451	700	2108
(dma_unmap_sg, put_page, kfree)

dma driver layer
----------------------
dma_memcpy_sg_prep			345	344	329	479	713	2611
dma_txd_submit			108	109	108	109	109	  111
dma_tx_status				327	349	431	534	797	2346
idxd_dma_tx_status (deduced)		119	  41	  56	  83	  97	  238   ???

SW copy
=======

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -- /tmp/xz1
[  152.636953] _io_ring_call_stat_print(read_iter): nb_call=37514528,nb_cycle=37008954492,cycle/call=986;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -- /tmp/xz1
[   70.028931] _io_ring_call_stat_print(read_iter): nb_call=28191936,nb_cycle=38541193956,cycle/call=1367;

taskset -c 1 /bin/bash -x /mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -- /tmp/xz1
[  218.785124] _io_ring_call_stat_print(read_iter): nb_call=19109312,nb_cycle=41928352830,cycle/call=2194;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 16384 -d 64 -- /tmp/xz1
[  337.191163] _io_ring_call_stat_print(read_iter): nb_call=11452608,nb_cycle=45751699798,cycle/call=3994;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 32768 -d 64 -- /tmp/xz1
[  294.087806] _io_ring_call_stat_print(read_iter): nb_call=5717280,nb_cycle=47097616396,cycle/call=8237;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 131072 -d 64 -- /tmp/xz1
[  152.502022] _io_ring_call_stat_print(read_iter): nb_call=1404480,nb_cycle=49777091724,cycle/call=35441;

DMA copy (ipoll)
==============

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -p 1 -- /tmp/xz1
[  838.747911] _io_ring_call_stat_print(read_iter): nb_call=24296523,nb_cycle=10747261984,cycle/call=442;
[  838.749273] _io_ring_call_stat_print(dma_copy): nb_call=24296523,nb_cycle=10502724540,cycle/call=432;
[  838.750632] _io_ring_call_stat_print(dma_tx_status): nb_call=24296523,nb_cycle=7600041778,cycle/call=312;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -p 1 -- /tmp/xz1
[ 1007.960910] _io_ring_call_stat_print(read_iter): nb_call=23038754,nb_cycle=10597803630,cycle/call=459;
[ 1007.962291] _io_ring_call_stat_print(dma_copy): nb_call=23038754,nb_cycle=10078534928,cycle/call=437;
[ 1007.963670] _io_ring_call_stat_print(dma_tx_status): nb_call=23038754,nb_cycle=8797313170,cycle/call=381;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[ 1111.263944] _io_ring_call_stat_print(read_iter): nb_call=19890966,nb_cycle=12980210200,cycle/call=652;
[ 1111.265306] _io_ring_call_stat_print(dma_copy): nb_call=19890966,nb_cycle=10791559664,cycle/call=542;
[ 1111.266677] _io_ring_call_stat_print(dma_tx_status): nb_call=19890966,nb_cycle=8551330946,cycle/call=429;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 16384 -d 64 -p 1 -- /tmp/xz1
[ 1185.235939] _io_ring_call_stat_print(read_iter): nb_call=14168913,nb_cycle=15437025706,cycle/call=1089;
[ 1185.237311] _io_ring_call_stat_print(dma_copy): nb_call=14168913,nb_cycle=13240559576,cycle/call=934;
[ 1185.238681] _io_ring_call_stat_print(dma_tx_status): nb_call=14168913,nb_cycle=7891900714,cycle/call=556;

extra stats enabled:
--------------------------

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -p 1 -- /tmp/xz1
[  165.854097] _io_ring_call_stat_print(read_iter): nb_call=20243146,nb_cycle=9360535452,cycle/call=462;
[  165.855472] _io_ring_call_stat_print(dma_map_page): nb_call=20243146,nb_cycle=2645942826,cycle/call=130;
[  165.856866] _io_ring_call_stat_print(dma_map_prep): nb_call=20243146,nb_cycle=6984741092,cycle/call=345;
[  165.858238] _io_ring_call_stat_print(dma_txd_submit): nb_call=20243146,nb_cycle=2204089220,cycle/call=108;
[  165.859657] _io_ring_call_stat_print(dma_tx_status): nb_call=20243146,nb_cycle=6626478424,cycle/call=327;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -p 1 -- /tmp/xz1
[  275.031263] _io_ring_call_stat_print(read_iter): nb_call=19959625,nb_cycle=9161716298,cycle/call=459;
[  275.033058] _io_ring_call_stat_print(dma_map_page): nb_call=19959625,nb_cycle=2635932496,cycle/call=132;
[  275.034480] _io_ring_call_stat_print(dma_map_prep): nb_call=19959625,nb_cycle=6870646042,cycle/call=344;
[  275.035862] _io_ring_call_stat_print(dma_txd_submit): nb_call=19959625,nb_cycle=2176350008,cycle/call=109;
[  275.037256] _io_ring_call_stat_print(dma_tx_status): nb_call=19959625,nb_cycle=6977679726,cycle/call=349;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[  401.383119] _io_ring_call_stat_print(read_iter): nb_call=18062188,nb_cycle=11587080420,cycle/call=641;
[  401.384486] _io_ring_call_stat_print(dma_map_page): nb_call=18062188,nb_cycle=2637550686,cycle/call=146;
[  401.385889] _io_ring_call_stat_print(dma_map_prep): nb_call=18062188,nb_cycle=5952760396,cycle/call=329;
[  401.387271] _io_ring_call_stat_print(dma_txd_submit): nb_call=18062188,nb_cycle=1965616416,cycle/call=108;
[  401.388691] _io_ring_call_stat_print(dma_tx_status): nb_call=18062188,nb_cycle=7790097454,cycle/call=431;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 16384 -d 64 -p 1 -- /tmp/xz1
[  627.571132] _io_ring_call_stat_print(read_iter): nb_call=13945661,nb_cycle=14898802216,cycle/call=1068;
[  627.572494] _io_ring_call_stat_print(dma_map_page): nb_call=13945661,nb_cycle=2929960406,cycle/call=210;
[  627.573883] _io_ring_call_stat_print(dma_map_prep): nb_call=13945661,nb_cycle=6693123644,cycle/call=479;
[  627.575250] _io_ring_call_stat_print(dma_txd_submit): nb_call=13945661,nb_cycle=1520918608,cycle/call=109;
[  627.576649] _io_ring_call_stat_print(dma_tx_status): nb_call=13945661,nb_cycle=7452250324,cycle/call=534;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 32768 -d 64 -p 1 -- /tmp/xz1
[  778.798192] _io_ring_call_stat_print(read_iter): nb_call=10047303,nb_cycle=17626174618,cycle/call=1754;
[  778.799583] _io_ring_call_stat_print(dma_map_page): nb_call=10047303,nb_cycle=2975384344,cycle/call=296;
[  778.801000] _io_ring_call_stat_print(dma_map_prep): nb_call=10047303,nb_cycle=7166354834,cycle/call=713;
[  778.802384] _io_ring_call_stat_print(dma_txd_submit): nb_call=10047303,nb_cycle=1098932882,cycle/call=109;
[  778.803807] _io_ring_call_stat_print(dma_tx_status): nb_call=10047303,nb_cycle=8010400466,cycle/call=797;

even more stats enabled:
----------------------------------

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -p 1 -- /tmp/xz1
[  227.487680] _io_ring_call_stat_print(read_iter): nb_call=18531697,nb_cycle=9091525924,cycle/call=490;
[  227.489095] _io_ring_call_stat_print(dma_map_page): nb_call=18531697,nb_cycle=2642274344,cycle/call=142;
[  227.490470] _io_ring_call_stat_print(dma_map_prep): nb_call=18531697,nb_cycle=6435793506,cycle/call=347;
[  227.491841] _io_ring_call_stat_print(dma_txd_submit): nb_call=18531697,nb_cycle=2023143092,cycle/call=109;
[  227.493257] _io_ring_call_stat_print(dma_tx_status): nb_call=18531697,nb_cycle=8965236398,cycle/call=483;
[  227.494650] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=18531697,nb_cycle=3868579740,cycle/call=208;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -p 1 -- /tmp/xz1
[  335.794733] _io_ring_call_stat_print(read_iter): nb_call=18385330,nb_cycle=8541024350,cycle/call=464;
[  335.796064] _io_ring_call_stat_print(dma_map_page): nb_call=18385330,nb_cycle=2000742552,cycle/call=108;
[  335.797471] _io_ring_call_stat_print(dma_map_prep): nb_call=18385330,nb_cycle=6616304978,cycle/call=359;
[  335.798869] _io_ring_call_stat_print(dma_txd_submit): nb_call=18385330,nb_cycle=2006059222,cycle/call=109;
[  335.800289] _io_ring_call_stat_print(dma_tx_status): nb_call=18385330,nb_cycle=10568171772,cycle/call=574;
[  335.801702] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=18385330,nb_cycle=5578989814,cycle/call=303;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[  935.314721] _io_ring_call_stat_print(read_iter): nb_call=16560375,nb_cycle=10774634416,cycle/call=650;
[  935.316063] _io_ring_call_stat_print(dma_map_page): nb_call=16560375,nb_cycle=2548383934,cycle/call=153;
[  935.317423] _io_ring_call_stat_print(dma_map_prep): nb_call=16560375,nb_cycle=5842722324,cycle/call=352;
[  935.318786] _io_ring_call_stat_print(dma_txd_submit): nb_call=16560375,nb_cycle=1802839382,cycle/call=108;
[  935.320174] _io_ring_call_stat_print(dma_tx_status): nb_call=16560375,nb_cycle=10670571864,cycle/call=644;
[  935.321607] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=16560375,nb_cycle=6226614110,cycle/call=375;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[ 1051.589712] _io_ring_call_stat_print(read_iter): nb_call=13569679,nb_cycle=13826817960,cycle/call=1018;
[ 1051.591105] _io_ring_call_stat_print(dma_map_page): nb_call=13569679,nb_cycle=2828194164,cycle/call=208;
[ 1051.592483] _io_ring_call_stat_print(dma_map_prep): nb_call=13569679,nb_cycle=6270020674,cycle/call=462;
[ 1051.593857] _io_ring_call_stat_print(dma_txd_submit): nb_call=13569679,nb_cycle=1482618160,cycle/call=109;
[ 1051.595262] _io_ring_call_stat_print(dma_tx_status): nb_call=13569679,nb_cycle=9804848154,cycle/call=722;
[ 1051.596671] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=13569679,nb_cycle=6125278794,cycle/call=451;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 32768 -d 64 -p 1 -- /tmp/xz1
[ 1167.621755] _io_ring_call_stat_print(read_iter): nb_call=9356189,nb_cycle=17348247116,cycle/call=1854;
[ 1167.623112] _io_ring_call_stat_print(dma_map_page): nb_call=9356189,nb_cycle=3255963092,cycle/call=348;
[ 1167.624480] _io_ring_call_stat_print(dma_map_prep): nb_call=9356189,nb_cycle=7065992002,cycle/call=755;
[ 1167.625861] _io_ring_call_stat_print(dma_txd_submit): nb_call=9356189,nb_cycle=1023755914,cycle/call=109;
[ 1167.627276] _io_ring_call_stat_print(dma_tx_status): nb_call=9356189,nb_cycle=9096623304,cycle/call=972;
[ 1167.628657] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=9356189,nb_cycle=6554425776,cycle/call=700;

and even more stats enabled:
----------------------------------------

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 1024 -d 64 -p 1 -- /tmp/xz1
[  413.308768] _io_ring_call_stat_print(read_iter): nb_call=17436862,nb_cycle=8339876898,cycle/call=478;
[  413.310115] _io_ring_call_stat_print(dma_map_page): nb_call=17436862,nb_cycle=2397826664,cycle/call=137;
[  413.311477] _io_ring_call_stat_print(dma_map_prep): nb_call=17436862,nb_cycle=5871816456,cycle/call=336;
[  413.312827] _io_ring_call_stat_print(dma_txd_submit): nb_call=17436862,nb_cycle=1930111314,cycle/call=110;
[  413.314212] _io_ring_call_stat_print(dma_tx_status): nb_call=17436862,nb_cycle=9608856944,cycle/call=551;
[  413.315580] _io_ring_call_stat_print(dma_alloc_copy_vec): nb_call=17436862,nb_cycle=3873159708,cycle/call=222;
[  413.317007] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=17436862,nb_cycle=4812252522,cycle/call=275;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 4096 -d 64 -p 1 -- /tmp/xz1
[  491.803957] _io_ring_call_stat_print(read_iter): nb_call=17431464,nb_cycle=7976750478,cycle/call=457;
[  491.805309] _io_ring_call_stat_print(dma_map_page): nb_call=17431464,nb_cycle=2506301976,cycle/call=143;
[  491.806670] _io_ring_call_stat_print(dma_map_prep): nb_call=17431464,nb_cycle=6338791238,cycle/call=363;
[  491.808039] _io_ring_call_stat_print(dma_txd_submit): nb_call=17431464,nb_cycle=1944660584,cycle/call=111;
[  491.809436] _io_ring_call_stat_print(dma_tx_status): nb_call=17431464,nb_cycle=9289975568,cycle/call=532;
[  491.810802] _io_ring_call_stat_print(dma_alloc_copy_vec): nb_call=17431464,nb_cycle=3738193802,cycle/call=214;
[  491.812247] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=17431464,nb_cycle=4546959358,cycle/call=260;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 8192 -d 64 -p 1 -- /tmp/xz1
[  596.820790] _io_ring_call_stat_print(read_iter): nb_call=15143794,nb_cycle=10155218664,cycle/call=670;
[  596.822172] _io_ring_call_stat_print(dma_map_page): nb_call=15143794,nb_cycle=2554489340,cycle/call=168;
[  596.823568] _io_ring_call_stat_print(dma_map_prep): nb_call=15143794,nb_cycle=5858304540,cycle/call=386;
[  596.824943] _io_ring_call_stat_print(dma_txd_submit): nb_call=15143794,nb_cycle=1679767784,cycle/call=110;
[  596.826358] _io_ring_call_stat_print(dma_tx_status): nb_call=15143794,nb_cycle=10300601004,cycle/call=680;
[  596.827758] _io_ring_call_stat_print(dma_alloc_copy_vec): nb_call=15143794,nb_cycle=3702212752,cycle/call=244;
[  596.829219] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=15143794,nb_cycle=6186738794,cycle/call=408;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 16384 -d 64 -p 1 -- /tmp/xz1
[  673.975774] _io_ring_call_stat_print(read_iter): nb_call=12651137,nb_cycle=13348866092,cycle/call=1055;
[  673.977163] _io_ring_call_stat_print(dma_map_page): nb_call=12651137,nb_cycle=2657725706,cycle/call=210;
[  673.978553] _io_ring_call_stat_print(dma_map_prep): nb_call=12651137,nb_cycle=6720093690,cycle/call=531;
[  673.979918] _io_ring_call_stat_print(dma_txd_submit): nb_call=12651137,nb_cycle=1403316410,cycle/call=110;
[  673.981316] _io_ring_call_stat_print(dma_tx_status): nb_call=12651137,nb_cycle=9043971908,cycle/call=714;
[  673.982718] _io_ring_call_stat_print(dma_alloc_copy_vec): nb_call=12651137,nb_cycle=3138149536,cycle/call=248;
[  673.984144] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=12651137,nb_cycle=5639966438,cycle/call=445;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 32768 -d 64 -p 1 -- /tmp/xz1
[  763.673818] _io_ring_call_stat_print(read_iter): nb_call=8953781,nb_cycle=16585739778,cycle/call=1852;
[  763.675148] _io_ring_call_stat_print(dma_map_page): nb_call=8953781,nb_cycle=2772644866,cycle/call=309;
[  763.676504] _io_ring_call_stat_print(dma_map_prep): nb_call=8953781,nb_cycle=7655287758,cycle/call=854;
[  763.677845] _io_ring_call_stat_print(dma_txd_submit): nb_call=8953781,nb_cycle=985388208,cycle/call=110;
[  763.679198] _io_ring_call_stat_print(dma_tx_status): nb_call=8953781,nb_cycle=9035643372,cycle/call=1009;
[  763.680572] _io_ring_call_stat_print(dma_alloc_copy_vec): nb_call=8953781,nb_cycle=2548060728,cycle/call=284;
[  763.681994] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=8953781,nb_cycle=6571959652,cycle/call=733;

taskset -c 1 /bin/bash -x /mnt/share0/run_bg_task.sh 30 /mnt/share0/buildroot-prev1/build/images/io_uring -b 131072 -d 64 -p 1 -- /tmp/xz1
[  851.217379] _io_ring_call_stat_print(read_iter): nb_call=3744307,nb_cycle=20966938344,cycle/call=5599;
[  851.218765] _io_ring_call_stat_print(dma_map_page): nb_call=3744307,nb_cycle=3872112822,cycle/call=1034;
[  851.220061] _io_ring_call_stat_print(dma_map_prep): nb_call=3744307,nb_cycle=9778748434,cycle/call=2611;
[  851.221416] _io_ring_call_stat_print(dma_txd_submit): nb_call=3744307,nb_cycle=417579118,cycle/call=111;
[  851.222755] _io_ring_call_stat_print(dma_tx_status): nb_call=11076628,nb_cycle=9711661782,cycle/call=876;
[  851.224114] _io_ring_call_stat_print(dma_alloc_copy_vec): nb_call=3744307,nb_cycle=1430044398,cycle/call=381;
[  851.225592] _io_ring_call_stat_print(dma_cb_free_copy_vec): nb_call=3744307,nb_cycle=7895268808,cycle/call=2108;

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2021-11-29 19:11 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-29 19:11 TSC for dmaengine API batch support for DSA (linux-idxd-ka2 @ 211126) Ananyev, Konstantin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).