Tuesday, December 16, 2014

Linux Networking

I was asked very often during the interview, what's your biggest concern when you were doing system administration. I always answer honestly which is networking. Here, I would like to summarized my Linux Networking note in my way, hope it helps.

Basic Network Interface Management: NIC Management

  •  /etc/sysconfig/network/ : configuration information for all network devices is stored in text files under this directory.
  •  /etc/init.d/network : init script will use the information in these files to setup static or dynamic IP
  •  /etc/sysconfig/networking-scripts/ifcfg-ethX : store the configuration for Ethernet device ethX
  •  /usr/share/doc/initscripts-*/sysconfig.txt listing all the valid options that can be set for a network interface.

Interface Management

  • ifconfig : most useful tool for diagnosing network problem
    • w/ no options will display the current networking settings include IP address, adapter MAC addresses, etc.
    • change w/ parameters of the network configuration as shown in following example:
      • e.g. : ifconfig eth0  192.168.x.x netmask 255.255.255.0
  • ip : it's more capable command is used to manage IP aliases, kernel routing, tunnels and policies.
    • following example shows using ip to list addresses assigned to all interfaces on system:
      • e.g. : ip addr show
      • e.g. : ip addr add 192.168.1.12/24 dev eth1
      • e.g. : ip addr del 192.168.1.12/24 dev eth1
      • e.g. : ip link show
    • running ip w/ the route argument will display the current routing
      • e.g. : ip route del default
      • e.g. : add default via 192.168.1.12
  • tcpdump : package capture
  • wireshark : package capture
Adapter Link
  • ethtool : to determine (and modify) the speed and duplex settings your Ethernet interface is operating at, or whether it has a link:
    • Display Setting for eth0:
      • ethtool eth0
      • PS: sometime old HW might try mii-tool 
  • route : to determine whether there is a default route.
    • Display default route:
      • route
      • PS : check default gateway address to make the network allows to access any other systems outside of its local network.
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.100.0   *               255.255.255.0   U     0      0        0 virbr1
192.168.122.0   *               255.255.255.0   U     0      0        0 virbr0
16.92.228.0     *               255.255.252.0   U     0      0        0 eth1
link-local      *               255.255.0.0     U     1003   0        0 eth1
default         16.92.228.1     0.0.0.0         UG    0      0        0 eth1


Kernel Module Problem
  • /etc/modprobe.d : All files underneath the /etc/modprobe.d directory which end with the .conf extension specify those options as required. Since the modprobe command can add or remove more than one module, due to modules having dependencies, we need a method of specifying what options what's options are to be used with those modules.
  • /var/lib/dhclient : The DHCP client file is named dhclient-$INTERFACE.lease
    • PS: Use the ifconfig command to find the MAC addresses assigned for Ethernet adapters. Look at the HWaddr label.
  • /etc/resolv.conf : if you find you can ping a numeric IP address but you can not ping its domain name equivalent, name resolution may be the culprit. Check the contents of the /etc/resolv.conf file for accurate information.
    • PS: NetworkManager daemon will overwrite changes to the /etc/resolv.conf. At least one, in successive order, of DNS1, DNS2, and DNS3 must be entered into the appropriate network interface configuration script in the /etc/sysconfig/network-scripts/ifcfg-ethX.
IP Aliases and Virtual Interfaces

Linux provided support for multiple IP addresses on a single physical interface by using virtual interfaces. Virtual interfaces can be configured by using the ip or ifconfig commands. 
  • eg: ethX:Y where ethX is the physical interface and Y is the virtual interface label.
    • ip addr add 10.100.0.247/24 dev eth0
    • ip -4 addr list eth0 or ifconfig eth0
The Xinetd Super Daemon

The extended internet service daemon (Xinetd) is a super daemon that manages several network services, which includes IMAP & POP servers, FTP servers, talk servers and time servers. Xinetd controls services and can apply access control list (ACL). 

ACLs can be used to determine which hosts can access the services from the network and what times of the day a service is available. Xinetd also leverages TCP wrappers to provide even more fine-grained control.

Xinetd uses individual configuration files for each service that it controls. If changes are made to a service's configuration file, then the xinetd daemon will need to be restarted. A common method for restarting the xinetd daemon is by using the SysV init script:
  • service xinetd restart
  • /etc/xinetd.conf : The main configuration file for the xinetd daemon location. It contains global parameters that affect the behavior of the xinetd daemon.
    • Each service managed by Xinetd can be enabled or disabled by editing its corresponding configuration file and modifying the "disabled = directive". If the disable=directive is missing from the file the service will be enabled.
  • /etc/xinetd.d : service specific configuration files are stored in this directory (this directory is specified in the /etc/xinetd.conf file, identified by the "includedir /etc/xinetd.d" )
  • /var/log/message (/var/log/secure): most Xinetd problems stem from "configuration errors". The Xinetd daemon does a good job of logging configuration parsing errors, so examining the /var/log/messages file is a good way to quickly identify most configuration related errors.
  • netstat, lsof or fuser : can display a list of in use (bound) ports and the processes and services that are using those ports:
    • fuser -v 22/tcp
    • [user@server /]$ sudo fuser -v 22/tcp
    • [sudo] password for wangjoh:
    •                      USER        PID ACCESS COMMAND
    • 22/tcp:              root       3214 F....  sshd
    •                      root      15542 f....  sshd
    •                      user      15546 F....  sshd
TCP Wrappers

TCP Wrappers provides a general security method for limiting access to services based on remote IP address. TCP Wrappers create a centralized administrative point where access to all services can be managed. It works by "wrapping around" existing applications.



  • /usr/sbin/tcpd or /lib/libwrap.so : binaries 
  • /etc/hosts.allow, /etc/host.deny : configuration files
  • /var/log/secure, /var/log/message : logs
  • daemon_list, client_list : syntax
Netfilter/iptables (firewall)

Netfilter is the standard Linux firewall solution. It has proven to be both robust and flexible. Because it is so capable, it can also be very complicated When a system has problems providing or connecting to network services, iptables should be one of the first suspects.

The iptables command is used to interact with Netfilter, which is inside the kernel.
  • related packages:
    • iptables
    • iptables-ipv6
  • binaires:
    • /sbin/iptables
    • /sbin/iptables-restore
    • /sbin/iptables-save
    • /sbin/ipt6tables
    • /sbin/ip6tables-restore
    • /sbin/ip6tables-save
  • configs:
    • /etc/sysconfig/iptables-config
    • /etc/sysconfig/ip6tables-config
  • logs:
    • /var/log/messages
  • data directory
    • /lib/xtables/
  • Init
    • /etc/init.d/iptables
    • /etc/init.d/ip6tables


iptables enable logging
  • iptables -L : By scanning the output of iptables to show Netfilter's contain rules
  • iptables -I INPUT 5 -m limit --limit 6/m -j LOG : enable logging before a suspect entry in the INPUT chain, with a limit of 6 messages per minute, the command.
  • /proc/net/ip_tables_names : A list of all tables that have been activated on system. 
    • PS: default "filter" table includes for Netfilter.
  • To disable Netfilter, use iptables to flush all the rules and ensure that policy allows traffic. Use the -F option to flush each table and then use the -P option to make sure each chain has a default policy of accept. It's also a good idea to remove any user-defined chains that may exist with the -X option, though this is not strictly necessary in order to disable Netfilter:
    • iptables -F : Flush each table
    • iptables -t nat -F
    • iptables -t mangle -F
    • iptables -X
    • iptables -t nat -X :-X --delete-chain
    • iptables -t mangle -X
    • iptables -P INPUT ACCEPT : -P --change policy on chain to target
    • iptables -P FORWARD ACCEPT
    • iptables -P OUTPUT ACCEPT
    • iptables -t nat -P PREROUTING ACCEPT : -t --table, table to manipulate (default: 'filter')
    • iptables -t nat -P OUTPUT ACCEPT
    • iptables -t nat -P POSTROUTING ACCEPT
    • iptables -t mangle -P PREROUTING ACCEPT
    • iptables -t mangle -P INPUT ACCEPT
    • iptables -t mangle -P FORWARD ACCEPT
    • iptables -t mangle -P OUTPUT ACCEPT
    • iptables -t mangle -P POSTROUTING ACCEPT

    Tuesday, December 2, 2014

    Linux System Boot Procedure

    I was awared the Linux boot process is a very fundamental question during the system infrastructure interview. I surf the internet but can't find the answer I feel reasonable for me. I know there has different version of Linux kernels that might be caused the differential. However, here I would like to share whatever I familiar boot process and it references RHEL6 most. I think it helps me to understand the OS fundamental concept and easier for me to follow the logic in my following research experiments.

    Linux System Main Actors - ref boot sequence:


    • System BIOS (basic I/O system)
    • MBR (master boot record, located in the 1st sector of the bootable disk /dev/hda or dev/sda)
    • GRUB (GRand Unified Bootloader, config file in /boot/grub/grub.conf)
    • initramfs Initial ramfs, initial ram base file system.
    It is a cpio archive of the initial file system that gets loaded into memory during the Linux startup processinitramfs needs to contain all of the device drivers and tools needed to mount the real root filesystem
    • initrd (initial ramdisk) is a scheme for loading a temporary root file system into memory in the boot process of the Linux kernel. 
    initrd and initramfs refer to two different methods of achieving this. Both are commonly used to make preparations before the real root file system can be mounted.
    • Linux Kernel
    The Linux kernel locate between bare metal (H/W) and O/S server ( User Apps ). It includes VFS(Virtual File System), IPC (Inter-Process Communication), Process Scheduler, Memory manager ( or Virtual Memory Manager), Dispatcher, Network Interface, Device Drivers ... etc.
    • /sbin/init lanuches bootup script from /etc/
    • Runlevel (/etc/rc.d/rc*.d/, you would see programs that start with S and K.
    a. Programs starts with S are used during startup. S for startup.
    b. Programs starts with K are used during shutdown. K for kill.
      When the Linux system is booting up, you might see various services getting started. For example, it might say “starting sendmail …. OK”. Those are the runlevel programs, executed from the run level directory as defined by your run level. )

      Linux System Boot Process in General - ref boot sequence:

      1. BIOS
      2. MBR located in the 1st sector of the bootable disk /dev/hda or dev/sda
      3. GRUB config file in /boot/grub/grub.conf
      4. Kernel:
              4.1 Init
              /etc/inittab file then execute runlevel program
              eg: grep initdefault /etc/inittab
              
              4.2 Runlevel: - showing message 
      When the Linux system is booting, see various services 
              getting started.
              e.g. it might say “starting sendmail …. OK”. 


      Linux System Boot Process in Detail - ref boot sequence:

      1. System BIOS performs three task
      • Power-On Self Test(POST)
      • Initial H/W setup and configuration
      • Loads option ROM from add-in card (SCSI, SAN HBA, RAID)
      • Selects boot device and executes MBR (master boot record, located in the 1st sector of the bootable disk /dev/hda or dev/sda)
      2. First stage GRUB in MBR(445 bytes); it loads:
      • stage1.5 GRUB using int13 BIOS calls, stage1.5 GRUB provides fs(filesystem) drivers and then loads stage2 GRUB
      3. Second stage GRUB 
      • Reads and uses configuration file or displays GRUB command prompt
      • Loads initial ram disk (usually specified)
      • Loads, decompresses, and executes selected Linux kernel from hard drive with command line arguments
      4. Linux Kernel
      • Initializes and configures H/W using drivers statistically compiled into the kernel
      • Decompresses the initramfs image and mount
      • Runs init script from initramfs image
      • init script loads kernel modules and performs task necessary to mount the real root fs(filesystem) including loading any required kernel modules stored in the initramfs image
      • Mounts the root partition passed to the kernel by the boot loader using the root=kernel command-line option(usually read only) as the root partition, replacing the initrd
      • Execut /sbin/init
      PS: for RHEL only:
      Upstart is async. replacement for the traditional init daemon. It is installed using sysvinit runlevel comptatibility mode so that it is mostly a drop-in replacement. Configuration is in the /etc/inittab and /etc/init/* files


      In sum, I know it looks complicated, let me try to see whether I can make the concept easier for understanding in the coming future.

      Share Movie Quote

      I may not know where I'm going, but I'm on my way. - (Movie: The Motorcycle Diaries), 2004