Object Storage Networking Tools as Analyzer
ping
how to use
ping is default network tools in most of linux, you shouldn't yum or apt install it. Here is an ping cli example for ping with package size around MTU e.g. 1500 see how that happen
$ ping -s <package size; e.g. 1480 > -c <count; e.g. : 3> or -i <wait>
Here is the example
$ ping 192.168.202.10 -s 1480 -c 10
PING 192.168.202.10 (192.168.202.10): 1480 data bytes
1488 bytes from 192.168.202.10: icmp_seq=0 ttl=63 time=8.118 ms
1488 bytes from 192.168.202.10: icmp_seq=1 ttl=63 time=6.077 ms
1488 bytes from 192.168.202.10: icmp_seq=2 ttl=63 time=3.317 ms
1488 bytes from 192.168.202.10: icmp_seq=3 ttl=63 time=3.430 ms
1488 bytes from 192.168.202.10: icmp_seq=4 ttl=63 time=3.659 ms
1488 bytes from 192.168.202.10: icmp_seq=5 ttl=63 time=3.277 ms
1488 bytes from 192.168.202.10: icmp_seq=6 ttl=63 time=3.220 ms
1488 bytes from 192.168.202.10: icmp_seq=7 ttl=63 time=3.617 ms
1488 bytes from 192.168.202.10: icmp_seq=8 ttl=63 time=3.707 ms
1488 bytes from 192.168.202.10: icmp_seq=9 ttl=63 time=3.024 ms
--- 192.168.202.10 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.024/4.145/8.118/1.558 ms
you can see at the end of summary, it shows how many packages transmitted and how many packages received at remote end, and total package loss.
package loss
package loss is kinds of import, here is bad example
more info regarding package loss affect on your network.
Packet loss is almost always bad when it occurs at the final destination. Packet loss happens when a packet doesn't make it there and back again. Anything over 2% packet loss over a period of time is a strong indicator of problems. Most internet protocols can correct for some packet loss, so you really shouldn't expect to see a lot of impact from packet loss until that loss starts to approach 5% and higher. Anything less than this is showing a possible problem, but one that is probably not impacting your experience significantly at present (unless you're an online gamer or something similar that requires 'twitch' reflexes).
latency
latency estimation from ping result
e.g. 1488 bytes from 192.168.202.10: icmp_seq=9 ttl=63 time=20.024 ms
latency : 20.024 ms = 0.020024 seconds
If your network latency larger than 0.07, then we think this latency might be potential to hurt your object storage
trouble shoot when you can't run ping.
Please do make sure your firewall doesn't block icmp
dstat
dstat is not usually default install, you might need to install first, however installation is pretty straightforward.
dstat installation
# yum install dstat -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.hmc.edu
* extras: mirror.sjc02.svwh.net
* updates: mirror.linuxfix.com
Resolving Dependencies
--> Running transaction check
---> Package dstat.noarch 0:0.7.2-12.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================================================================================================================================================
Installing:
dstat noarch 0.7.2-12.el7 base 163 k
Transaction Summary
================================================================================================================================================================================================================================================================================
Install 1 Package
Total download size: 163 k
Installed size: 752 k
Downloading packages:
dstat-0.7.2-12.el7.noarch.rpm | 163 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : dstat-0.7.2-12.el7.noarch 1/1
Verifying : dstat-0.7.2-12.el7.noarch 1/1
Installed:
dstat.noarch 0:0.7.2-12.el7
Complete!
how to use dstat in general
I usually doesn't add too many parameters for dstat.
$ dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
4 1 95 0 0 0| 31k 138k| 0 0 |5335B 16k| 706 1480
0 1 99 0 0 0| 0 0 |1774B 998B| 0 0 | 242 594
1 0 99 0 0 0| 0 0 |1668B 1422B| 0 0 | 241 592
2 0 98 0 0 0| 0 16k| 827B 1065B| 0 0 | 277 749
1 1 99 0 0 0| 0 0 | 891B 350B| 0 0 | 242 616
1 1 98 0 0 0| 0 0 | 666B 358B| 0 0 | 236 602
1 0 99 0 0 0| 0 0 | 771B 350B| 0 0 | 217 581
1 1 99 0 0 0| 0 0 |1355B 1320B| 0 0 | 265 612
2 0 98 0 0 0| 0 12k| 831B 1065B| 0 0 | 267 755
0 0 99 0 0 1| 0 0 | 666B 350B| 0 0 | 251 625
1 1 98 0 0 0| 0 0 | 891B 358B| 0 0 | 249 638
1 1 99 0 0 0| 0 0 |1601B 350B| 0 0 | 261 607
1 0 99 0 0 0| 0 0 |1563B 1874B| 0 0 | 247 623
2 0 98 0 0 0| 0 32k| 953B 1167B| 0 0 | 284 756
how to use dstat with more parameters
if you would like to specific drives or NICs, you can try as below. This dstat will list specific drives or NICs and collect info within 20 seconds as a batch.
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 39.5G 0 part
├─centos-root 253:0 0 38.5G 0 lvm /
└─centos-swap 253:1 0 1G 0 lvm [SWAP]
sdb 8:16 0 8G 0 disk /srv/node/d0
sdc 8:32 0 8G 0 disk /srv/node/d1
sdd 8:48 0 8G 0 disk /srv/node/d2
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:0c:4e:dc brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 86285sec preferred_lft 86285sec
inet6 fe80::a00:27ff:fe0c:4edc/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:5c:d7:41 brd ff:ff:ff:ff:ff:ff
inet 172.28.128.42/24 brd 172.28.128.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe5c:d741/64 scope link
valid_lft forever preferred_lft forever
4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:be:6e:03 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a00:27ff:febe:6e03/64 scope link
valid_lft forever preferred_lft forever
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
link/none
inet 10.123.0.4 peer 10.123.0.1/32 scope global tun0
valid_lft forever preferred_lft forever
# dstat -D sda,sdb,sdc,sdd -N enp0s3,enp0s8,enp0s9,tun0 5
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- --dsk/sda-----dsk/sdb-----dsk/sdc-----dsk/sdd-- -net/enp0s3--net/enp0s8--net/enp0s9---net/tun0- ---paging-- ---system--
usr sys idl wai hiq siq| read writ: read writ: read writ: read writ| recv send: recv send: recv send: recv send| in out | int csw
10 7 82 1 0 0| 618k 64k: 14k 13k: 18k 9707B: 18k 9659B| 0 0 : 0 0 : 0 0 : 0 0 | 0 0 |1584 2081
11 8 81 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 60B 164B: 46B 57B: 0 0 : 10B 22B| 0 0 |1864 2192
8 8 83 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 60B 135B:1099B 4922B: 0 0 : 421B 11k| 0 0 |1836 2326
5 1 93 0 0 0| 0 819B: 29k 12k: 34k 8192B: 34k 8397B| 129B 200B: 46B 38B: 0 0 : 10B 22B| 0 0 |1174 1822
17 15 68 0 0 0| 0 9318B:5734B 29k: 0 19k: 0 19k| 96B 182B: 290B 296B: 0 0 : 110B 135B| 0 0 |2681 2983
3 1 96 0 0 0| 0 289k: 0 0 : 0 0 : 0 0 | 60B 132B: 46B 57B: 0 0 : 10B 22B| 0 0 | 955 1417
7 5 88 0 0 0| 0 2458B: 0 0 : 0 0 : 0 0 | 78B 140B: 237B 296B: 0 0 : 110B 135B| 0 0 |1491 1920
9 7 84 0 0 0| 0 1638B: 0 0 : 0 0 : 0 0 | 60B 119B: 58B 69B: 0 0 : 10B 22B| 0 0 |1523 1847
3 1 96 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 60B 124B:1007B 4910B: 0 0 : 421B 11k| 0 0 |1041 1603
12 7 80 0 0 0| 0 0 : 0 10k: 0 3789B: 0 5530B| 127B 185B: 46B 38B: 0 0 : 10B 22B| 0 0 |1815 2292
19 13 68 0 0 0| 0 15k: 0 25k: 0 13k: 0 11k| 60B 122B: 283B 308B: 0 0 : 110B 135B| 0 0 |2219 2395
6 1 93 0 0 0| 0 4915B: 0 0 : 0 0 : 0 0 | 60B 117B: 83B 57B: 0 0 : 10B 22B| 0 0 | 915 1267
8 5 87 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 60B 127B: 310B 370B: 0 0 : 150B 177B| 0 0 |1507 1936
7 6 87 0 0 0| 0 287k: 0 0 : 0 0 : 0 0 | 60B 120B: 46B 57B: 0 0 : 10B 22B| 0 0 |1434 1801
2 1 97 0 0 0| 0 3277B: 0 0 : 0 0 : 0 0 | 60B 114B:1019B 4997B: 0 0 : 421B 11k| 0 0 | 907 1489
10 6 83 0 0 0| 0 0 : 0 9114B: 22k 4710B: 0 4710B| 93B 152B: 46B 38B: 0 0 : 10B 22B| 0 0 |1881 2385
15 13 72 0 0 0| 0 3686B: 34k 25k: 13k 19k: 34k 17k| 96B 182B: 387B 323B: 0 0 : 131B 146B| 0 0 |2161 2406
2 0 98 0 0 0| 0 1638B: 0 819B: 0 0 : 0 0 | 78B 138B: 46B 57B: 0 0 : 10B 22B| 0 0 | 753 1272
10 9 81 0 0 0| 0 2458B: 0 0 : 0 0 : 0 0 | 60B 137B: 303B 335B: 0 0 : 131B 146B| 0 0 |1897 2255
6 3 91 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 | 60B 117B: 46B 57B: 0 0 : 10B 22B| 0 0 |1148 1533
3 1 97 0 0 0| 0 288k: 0 0 : 0 0 : 0 0 | 78B 138B: 980B 4841B: 0 0 : 411B 11k| 0 0 | 909 1433
13 11 75 0 0 0| 0 0 : 0 15k: 0 5222B: 0 8294B| 109B 186B: 95B 50B: 0 0 : 10B 22B| 0 0 |2012 2392
iperf
install iperf
iperf is usually not default install in the Linux, you might need to install it first before you start to use it.
$ yum install iperf3
trouble shoot for running iperf
From swift node to swift node, sometimes, iptables block iperf 5001 testing
Solution 1 , disable firewalld temporary for iperf
For CentOS
=== flush out iptables ===
$ iptables -F
=== bring it back once you finish
CentOS 6
$ service iptables restart
CentOS 7
$ service firewalld restart
Solution 2 , add rule in firewalld permeantly
$ iptables -I INPUT -p udp -m udp --dport 5001 -j ACCEPT
how to use iperf in general
In general how to run iperf is very straightforward. Server end for listening the packages from iperf clients.
$ iperf -s
Client end for generate the packages to iperf server.
$ iperf -c <server_ip>
how to use iperf example with more detail
The main purpose for me to use iperf is to measure bandwidth, here is an example to test against with TCP protocol.
$ iperf -c 192.168.201.239
------------------------------------------------------------
Client connecting to 192.168.201.239, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.201.153 port 56476 connected with 192.168.201.239 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 959 MBytes 804 Mbits/sec
You can test against with bi-directions from server to client then from client to server.
$ iperf -c 192.168.201.239 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.201.239, TCP port 5001
TCP window size: 162 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.201.153 port 56970 connected with 192.168.201.239 port 5001
[ 4] local 192.168.201.153 port 5001 connected with 192.168.201.239 port 59806
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 680 MBytes 570 Mbits/sec
[ 4] 0.0-10.0 sec 1023 MBytes 856 Mbits/sec
You can test against with UCP protocol as well, here is an example.
$ iperf -c 192.168.201.239 -u
------------------------------------------------------------
Client connecting to 192.168.201.239, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.201.153 port 40041 connected with 192.168.201.239 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.23 MBytes 1.03 Mbits/sec
[ 3] Sent 893 datagrams
read failed: Connection refused
[ 3] WARNING: did not receive ack of last datagram after 5 tries.
Here is the server end output example.
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.201.239 port 5001 connected with 192.168.201.153 port 56474
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 981 MBytes 822 Mbits/sec
[ 5] local 192.168.201.239 port 5001 connected with 192.168.201.153 port 56476
[ 5] 0.0-10.0 sec 959 MBytes 803 Mbits/sec
[ 4] local 192.168.201.239 port 5001 connected with 192.168.201.153 port 56970
------------------------------------------------------------
Client connecting to 192.168.201.153, TCP port 5001
TCP window size: 272 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.201.239 port 59806 connected with 192.168.201.153 port 5001
[ 6] 0.0-10.0 sec 1023 MBytes 858 Mbits/sec
[ 4] 0.0-10.0 sec 680 MBytes 568 Mbits/sec
iostat
iostat is most likely use for check drive io status. It's nothing with network. Howerver, I usually use it when I trouble shoot network issue on object storage.
iostat installation
It usually default install in linux however sometime you might still need to install by yourself, for this case happen, here is how to install it.
$ sudo apt install sysstat
iostat example 1
If you want to know how to run iostat, just try
iostat -h. Here is the most common one I am using now.$ iostat -dmx 2
Linux 4.4.0-89-generic (elk-swift) 08/10/2017 _x86_64_ (2 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 1.09 11.63 0.97 7.87 0.03 0.13 38.22 0.01 1.06 1.01 1.07 0.32 0.28
scd0 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.50 0.50 0.00 0.50 0.00
dm-0 0.00 0.00 0.75 14.31 0.02 0.12 19.62 0.01 0.73 1.24 0.71 0.18 0.27
dm-1 0.00 0.00 1.30 3.97 0.01 0.02 8.00 0.06 11.55 0.16 15.30 0.02 0.01
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.50 0.00 1.50 0.00 0.01 8.00 0.00 0.00 0.00 0.00 0.00 0.00
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 1.50 0.00 0.01 8.00 0.00 1.33 0.00 1.33 1.33 0.20
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
iostat example 2
Or you can try parameter as below and meantime to grep which drive e.g. sdd you would like to target on
# iostat -xtc 2 10
Linux 3.10.0-327.el7.x86_64 (ss02.ss.idv) 08/15/2017 _x86_64_ (2 CPU)
08/15/2017 07:28:44 PM
avg-cpu: %user %nice %system %iowait %steal %idle
9.38 0.00 6.35 0.32 0.00 83.94
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.35 0.83 1.88 8.36 9.56 13.19 0.00 1.15 0.72 1.35 0.80 0.22
sdd 0.00 0.32 0.73 0.70 10.33 6.39 23.51 0.00 1.63 0.84 2.45 0.97 0.14
sda 0.00 0.22 8.44 2.33 298.40 52.65 65.15 0.03 3.13 3.36 2.30 0.94 1.02
sdc 0.00 0.32 0.73 0.70 10.33 6.40 23.48 0.00 1.91 0.86 3.01 0.96 0.14
dm-0 0.00 0.00 7.31 1.80 262.74 50.22 68.69 0.03 3.06 2.98 3.37 1.09 0.99
dm-1 0.00 0.00 0.17 0.00 3.15 0.00 36.49 0.00 14.71 14.71 0.00 13.68 0.24
08/15/2017 07:28:46 PM
avg-cpu: %user %nice %system %iowait %steal %idle
3.14 0.00 1.31 0.00 0.00 95.55
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 1.50 0.00 5.00 0.00 36.25 14.50 0.02 4.30 0.00 4.30 0.70 0.35
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 6.00 0.00 36.25 12.08 0.03 5.08 0.00 5.08 0.58 0.35
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
# iostat -xtc 2 10 | grep sdd
sdd 0.00 0.32 0.73 0.70 10.34 6.38 23.37 0.00 1.63 0.84 2.44 0.96 0.14
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00