Tuesday, August 15, 2017

Analysis Object Storage Networking Tool Sets

Object Storage Networking Tools as Analyzer

ping

how to use

ping is default network tools in most of linux, you shouldn't yum or apt install it. Here is an ping cli example for ping with package size around MTU e.g. 1500 see how that happen
$ ping -s <package size; e.g. 1480 > -c <count; e.g. : 3> or -i <wait>
Here is the example
$ ping 192.168.202.10 -s 1480 -c 10
PING 192.168.202.10 (192.168.202.10): 1480 data bytes
1488 bytes from 192.168.202.10: icmp_seq=0 ttl=63 time=8.118 ms
1488 bytes from 192.168.202.10: icmp_seq=1 ttl=63 time=6.077 ms
1488 bytes from 192.168.202.10: icmp_seq=2 ttl=63 time=3.317 ms
1488 bytes from 192.168.202.10: icmp_seq=3 ttl=63 time=3.430 ms
1488 bytes from 192.168.202.10: icmp_seq=4 ttl=63 time=3.659 ms
1488 bytes from 192.168.202.10: icmp_seq=5 ttl=63 time=3.277 ms
1488 bytes from 192.168.202.10: icmp_seq=6 ttl=63 time=3.220 ms
1488 bytes from 192.168.202.10: icmp_seq=7 ttl=63 time=3.617 ms
1488 bytes from 192.168.202.10: icmp_seq=8 ttl=63 time=3.707 ms
1488 bytes from 192.168.202.10: icmp_seq=9 ttl=63 time=3.024 ms

--- 192.168.202.10 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.024/4.145/8.118/1.558 ms
you can see at the end of summary, it shows how many packages transmitted and how many packages received at remote end, and total package loss.


package loss

package loss is kinds of import, here is bad example
more info regarding package loss affect on your network.
Packet loss is almost always bad when it occurs at the final destination. Packet loss happens when a packet doesn't make it there and back again. Anything over 2% packet loss over a period of time is a strong indicator of problems. Most internet protocols can correct for some packet loss, so you really shouldn't expect to see a lot of impact from packet loss until that loss starts to approach 5% and higher. Anything less than this is showing a possible problem, but one that is probably not impacting your experience significantly at present (unless you're an online gamer or something similar that requires 'twitch' reflexes).

latency

latency estimation from ping result
e.g. 1488 bytes from 192.168.202.10: icmp_seq=9 ttl=63 time=20.024 ms
latency : 20.024 ms = 0.020024 seconds

If your network latency larger than 0.07, then we think this latency might be potential to hurt your object storage

trouble shoot when you can't run ping.

Please do make sure your firewall doesn't block icmp

dstat

dstat is not usually default install, you might need to install first, however installation is pretty straightforward.

dstat installation

# yum install dstat -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.hmc.edu
 * extras: mirror.sjc02.svwh.net
 * updates: mirror.linuxfix.com
Resolving Dependencies
--> Running transaction check
---> Package dstat.noarch 0:0.7.2-12.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================================================================
 Package                                                         Arch                                                             Version                                                                  Repository                                                      Size
================================================================================================================================================================================================================================================================================
Installing:
 dstat                                                           noarch                                                           0.7.2-12.el7                                                             base                                                           163 k

Transaction Summary
================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 163 k
Installed size: 752 k
Downloading packages:
dstat-0.7.2-12.el7.noarch.rpm                                                                                                                                                                                                                            | 163 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : dstat-0.7.2-12.el7.noarch                                                                                                                                                                                                                                    1/1
  Verifying  : dstat-0.7.2-12.el7.noarch                                                                                                                                                                                                                                    1/1

Installed:
  dstat.noarch 0:0.7.2-12.el7

Complete!

how to use dstat in general

I usually doesn't add too many parameters for dstat.
$ dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  4   1  95   0   0   0|  31k  138k|   0     0 |5335B   16k| 706  1480
  0   1  99   0   0   0|   0     0 |1774B  998B|   0     0 | 242   594
  1   0  99   0   0   0|   0     0 |1668B 1422B|   0     0 | 241   592
  2   0  98   0   0   0|   0    16k| 827B 1065B|   0     0 | 277   749
  1   1  99   0   0   0|   0     0 | 891B  350B|   0     0 | 242   616
  1   1  98   0   0   0|   0     0 | 666B  358B|   0     0 | 236   602
  1   0  99   0   0   0|   0     0 | 771B  350B|   0     0 | 217   581
  1   1  99   0   0   0|   0     0 |1355B 1320B|   0     0 | 265   612
  2   0  98   0   0   0|   0    12k| 831B 1065B|   0     0 | 267   755
  0   0  99   0   0   1|   0     0 | 666B  350B|   0     0 | 251   625
  1   1  98   0   0   0|   0     0 | 891B  358B|   0     0 | 249   638
  1   1  99   0   0   0|   0     0 |1601B  350B|   0     0 | 261   607
  1   0  99   0   0   0|   0     0 |1563B 1874B|   0     0 | 247   623
  2   0  98   0   0   0|   0    32k| 953B 1167B|   0     0 | 284   756

how to use dstat with more parameters

if you would like to specific drives or NICs, you can try as below. This dstat will list specific drives or NICs and collect info within 20 seconds as a batch.
# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   40G  0 disk
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 39.5G  0 part
  ├─centos-root 253:0    0 38.5G  0 lvm  /
  └─centos-swap 253:1    0    1G  0 lvm  [SWAP]
sdb               8:16   0    8G  0 disk /srv/node/d0
sdc               8:32   0    8G  0 disk /srv/node/d1
sdd               8:48   0    8G  0 disk /srv/node/d2

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:0c:4e:dc brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 86285sec preferred_lft 86285sec
    inet6 fe80::a00:27ff:fe0c:4edc/64 scope link
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:5c:d7:41 brd ff:ff:ff:ff:ff:ff
    inet 172.28.128.42/24 brd 172.28.128.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe5c:d741/64 scope link
       valid_lft forever preferred_lft forever
4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:be:6e:03 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a00:27ff:febe:6e03/64 scope link
       valid_lft forever preferred_lft forever
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none
    inet 10.123.0.4 peer 10.123.0.1/32 scope global tun0
       valid_lft forever preferred_lft forever

 # dstat -D sda,sdb,sdc,sdd -N enp0s3,enp0s8,enp0s9,tun0 5
 You did not select any stats, using -cdngy by default.
 ----total-cpu-usage---- --dsk/sda-----dsk/sdb-----dsk/sdc-----dsk/sdd-- -net/enp0s3--net/enp0s8--net/enp0s9---net/tun0- ---paging-- ---system--
 usr sys idl wai hiq siq| read  writ: read  writ: read  writ: read  writ| recv  send: recv  send: recv  send: recv  send|  in   out | int   csw
  10   7  82   1   0   0| 618k   64k:  14k   13k:  18k 9707B:  18k 9659B|   0     0 :   0     0 :   0     0 :   0     0 |   0     0 |1584  2081
  11   8  81   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |  60B  164B:  46B   57B:   0     0 :  10B   22B|   0     0 |1864  2192
   8   8  83   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |  60B  135B:1099B 4922B:   0     0 : 421B   11k|   0     0 |1836  2326
   5   1  93   0   0   0|   0   819B:  29k   12k:  34k 8192B:  34k 8397B| 129B  200B:  46B   38B:   0     0 :  10B   22B|   0     0 |1174  1822
  17  15  68   0   0   0|   0  9318B:5734B   29k:   0    19k:   0    19k|  96B  182B: 290B  296B:   0     0 : 110B  135B|   0     0 |2681  2983
   3   1  96   0   0   0|   0   289k:   0     0 :   0     0 :   0     0 |  60B  132B:  46B   57B:   0     0 :  10B   22B|   0     0 | 955  1417
   7   5  88   0   0   0|   0  2458B:   0     0 :   0     0 :   0     0 |  78B  140B: 237B  296B:   0     0 : 110B  135B|   0     0 |1491  1920
   9   7  84   0   0   0|   0  1638B:   0     0 :   0     0 :   0     0 |  60B  119B:  58B   69B:   0     0 :  10B   22B|   0     0 |1523  1847
   3   1  96   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |  60B  124B:1007B 4910B:   0     0 : 421B   11k|   0     0 |1041  1603
  12   7  80   0   0   0|   0     0 :   0    10k:   0  3789B:   0  5530B| 127B  185B:  46B   38B:   0     0 :  10B   22B|   0     0 |1815  2292
  19  13  68   0   0   0|   0    15k:   0    25k:   0    13k:   0    11k|  60B  122B: 283B  308B:   0     0 : 110B  135B|   0     0 |2219  2395
   6   1  93   0   0   0|   0  4915B:   0     0 :   0     0 :   0     0 |  60B  117B:  83B   57B:   0     0 :  10B   22B|   0     0 | 915  1267
   8   5  87   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |  60B  127B: 310B  370B:   0     0 : 150B  177B|   0     0 |1507  1936
   7   6  87   0   0   0|   0   287k:   0     0 :   0     0 :   0     0 |  60B  120B:  46B   57B:   0     0 :  10B   22B|   0     0 |1434  1801
   2   1  97   0   0   0|   0  3277B:   0     0 :   0     0 :   0     0 |  60B  114B:1019B 4997B:   0     0 : 421B   11k|   0     0 | 907  1489
  10   6  83   0   0   0|   0     0 :   0  9114B:  22k 4710B:   0  4710B|  93B  152B:  46B   38B:   0     0 :  10B   22B|   0     0 |1881  2385
  15  13  72   0   0   0|   0  3686B:  34k   25k:  13k   19k:  34k   17k|  96B  182B: 387B  323B:   0     0 : 131B  146B|   0     0 |2161  2406
   2   0  98   0   0   0|   0  1638B:   0   819B:   0     0 :   0     0 |  78B  138B:  46B   57B:   0     0 :  10B   22B|   0     0 | 753  1272
  10   9  81   0   0   0|   0  2458B:   0     0 :   0     0 :   0     0 |  60B  137B: 303B  335B:   0     0 : 131B  146B|   0     0 |1897  2255
   6   3  91   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |  60B  117B:  46B   57B:   0     0 :  10B   22B|   0     0 |1148  1533
   3   1  97   0   0   0|   0   288k:   0     0 :   0     0 :   0     0 |  78B  138B: 980B 4841B:   0     0 : 411B   11k|   0     0 | 909  1433
  13  11  75   0   0   0|   0     0 :   0    15k:   0  5222B:   0  8294B| 109B  186B:  95B   50B:   0     0 :  10B   22B|   0     0 |2012  2392

iperf

install iperf

iperf is usually not default install in the Linux, you might need to install it first before you start to use it.
$ yum install iperf3

trouble shoot for running iperf

From swift node to swift node, sometimes, iptables block iperf 5001 testing

Solution 1 , disable firewalld temporary for iperf

For CentOS

=== flush out iptables ===
$ iptables -F

=== bring it back once you finish

CentOS 6
$ service iptables restart

CentOS 7
$ service firewalld restart

Solution 2 , add rule in firewalld permeantly

$ iptables -I INPUT -p udp -m udp --dport 5001 -j ACCEPT

how to use iperf in general

In general how to run iperf is very straightforward. Server end for listening the packages from iperf clients.
$ iperf -s
Client end for generate the packages to iperf server.
$ iperf -c <server_ip>

how to use iperf example with more detail

The main purpose for me to use iperf is to measure bandwidth, here is an example to test against with TCP protocol.
$ iperf -c 192.168.201.239
------------------------------------------------------------
Client connecting to 192.168.201.239, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.201.153 port 56476 connected with 192.168.201.239 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   959 MBytes   804 Mbits/sec
You can test against with bi-directions from server to client then from client to server.
$ iperf -c 192.168.201.239 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.201.239, TCP port 5001
TCP window size:  162 KByte (default)
------------------------------------------------------------
[  5] local 192.168.201.153 port 56970 connected with 192.168.201.239 port 5001
[  4] local 192.168.201.153 port 5001 connected with 192.168.201.239 port 59806
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   680 MBytes   570 Mbits/sec
[  4]  0.0-10.0 sec  1023 MBytes   856 Mbits/sec
You can test against with UCP protocol as well, here is an example.
$ iperf -c 192.168.201.239 -u
------------------------------------------------------------
Client connecting to 192.168.201.239, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.201.153 port 40041 connected with 192.168.201.239 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.23 MBytes  1.03 Mbits/sec
[  3] Sent 893 datagrams
read failed: Connection refused
[  3] WARNING: did not receive ack of last datagram after 5 tries.
Here is the server end output example.
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.201.239 port 5001 connected with 192.168.201.153 port 56474
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   981 MBytes   822 Mbits/sec
[  5] local 192.168.201.239 port 5001 connected with 192.168.201.153 port 56476
[  5]  0.0-10.0 sec   959 MBytes   803 Mbits/sec
[  4] local 192.168.201.239 port 5001 connected with 192.168.201.153 port 56970
------------------------------------------------------------
Client connecting to 192.168.201.153, TCP port 5001
TCP window size:  272 KByte (default)
------------------------------------------------------------
[  6] local 192.168.201.239 port 59806 connected with 192.168.201.153 port 5001
[  6]  0.0-10.0 sec  1023 MBytes   858 Mbits/sec
[  4]  0.0-10.0 sec   680 MBytes   568 Mbits/sec

iostat

iostat is most likely use for check drive io status. It's nothing with network. Howerver, I usually use it when I trouble shoot network issue on object storage.

iostat installation

It usually default install in linux however sometime you might still need to install by yourself, for this case happen, here is how to install it.
$ sudo apt install sysstat

iostat example 1

If you want to know how to run iostat, just try iostat -h. Here is the most common one I am using now.
$ iostat -dmx 2
Linux 4.4.0-89-generic (elk-swift) 08/10/2017 _x86_64_ (2 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               1.09    11.63    0.97    7.87     0.03     0.13    38.22     0.01    1.06    1.01    1.07   0.32   0.28
scd0              0.00     0.00    0.00    0.00     0.00     0.00     8.00     0.00    0.50    0.50    0.00   0.50   0.00
dm-0              0.00     0.00    0.75   14.31     0.02     0.12    19.62     0.01    0.73    1.24    0.71   0.18   0.27
dm-1              0.00     0.00    1.30    3.97     0.01     0.02     8.00     0.06   11.55    0.16   15.30   0.02   0.01

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.50    0.00    1.50     0.00     0.01     8.00     0.00    0.00    0.00    0.00   0.00   0.00
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    1.50     0.00     0.01     8.00     0.00    1.33    0.00    1.33   1.33   0.20
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

iostat example 2

Or you can try parameter as below and meantime to grep which drive e.g. sdd you would like to target on
# iostat -xtc 2 10
Linux 3.10.0-327.el7.x86_64 (ss02.ss.idv)  08/15/2017  _x86_64_ (2 CPU)

08/15/2017 07:28:44 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.38    0.00    6.35    0.32    0.00   83.94

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     0.35    0.83    1.88     8.36     9.56    13.19     0.00    1.15    0.72    1.35   0.80   0.22
sdd               0.00     0.32    0.73    0.70    10.33     6.39    23.51     0.00    1.63    0.84    2.45   0.97   0.14
sda               0.00     0.22    8.44    2.33   298.40    52.65    65.15     0.03    3.13    3.36    2.30   0.94   1.02
sdc               0.00     0.32    0.73    0.70    10.33     6.40    23.48     0.00    1.91    0.86    3.01   0.96   0.14
dm-0              0.00     0.00    7.31    1.80   262.74    50.22    68.69     0.03    3.06    2.98    3.37   1.09   0.99
dm-1              0.00     0.00    0.17    0.00     3.15     0.00    36.49     0.00   14.71   14.71    0.00  13.68   0.24

08/15/2017 07:28:46 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.14    0.00    1.31    0.00    0.00   95.55

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     1.50    0.00    5.00     0.00    36.25    14.50     0.02    4.30    0.00    4.30   0.70   0.35
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    6.00     0.00    36.25    12.08     0.03    5.08    0.00    5.08   0.58   0.35
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00


# iostat -xtc 2 10 | grep sdd
sdd               0.00     0.32    0.73    0.70    10.34     6.38    23.37     0.00    1.63    0.84    2.44   0.96   0.14
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

No comments:

Post a Comment