JavaScriptを有効にしてください

【摩訶不思議】azcopyはなぜ速い?

 ·  ☕ 3 min read

TL;DR

  • azcopyとは, Azure内のblobを操作するためのツールなのだが, これを使えば摩訶不思議. なぜかwgetよりも快適にblobをダウンロードできる.
  • azcopyがなぜこんなにも速いのか誰か教えてくれ
    • 仮説: azcopyはコネクションを大量に張る + サーバを分散させている

導入: wgetが遅い問題

  • ORTの再現実験のため, https://imagecaption.blob.core.windows.net/imagecaption/trainval.zipwgetしようとしたら, 何度もエラーでリトライされる
  • それに, あまりにもスループットが低すぎる
    • 一晩続けてみると28回もリトライ…
    • どうにかできないかなと思っていたところ, URLをよく見ると, どこかで見たことあるオリジン.
    • 記憶をたぐってみると, アレレ VinVLで指定されてたものにソックリ
    • ということでazcopyでダウンロードしてみるとあら不思議!!!
      • 全くリトライされないどころか, めちゃくちゃ高速にダウンロードしてやがる
      • azcopy copyがなぜこんなにも速いのか誰か教えてくれ

azcopyはなぜ速い

  • netstatすると大量にコネクションが張られていますね
    • azcopy copyはただのTCPマシマシスパゲッティ🍝だったってこと…?
    • てことはサーバ分散 + コネクション並列化しただけってことなのかもしれない
    • だとしたら aria2とかでも同じくらいスループット出るのかな?今度時間がある時に実験してみましょう
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 d15:55414               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55280               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54960               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55426               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55646               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55270               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55556               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55506               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55180               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54872               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55508               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55622               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55184               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54846               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54874               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55092               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55454               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55652               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55258               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55148               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55078               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55394               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55328               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54858               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55604               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55060               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54964               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55160               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55288               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55562               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55066               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55132               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55526               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54934               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55592               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54826               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55518               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55522               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55416               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55648               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55380               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55074               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55370               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55118               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55102               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54870               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55442               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54950               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55006               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55142               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54970               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54902               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55574               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55126               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55338               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55614               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55070               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54914               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55256               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54910               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55390               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55492               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55054               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55190               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55088               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55348               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55350               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55124               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55154               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55120               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55076               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54966               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55354               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55058               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55086               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54852               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55084               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55590               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55400               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54860               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54904               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55600               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55290               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54828               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55170               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55196               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54896               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:54900               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55130               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55346               52.239.161.42:https     ESTABLISHED
tcp        0      0 d15:55334               52.239.161.42:https     ESTABLISHED
共有

YuWd (Yuiga Wada)
著者
YuWd (Yuiga Wada)
機械学習・競プロ・iOS・Web