Tuesday, December 16, 2014

Linux Networking

I was asked very often during the interview, what's your biggest concern when you were doing system administration. I always answer honestly which is networking. Here, I would like to summarized my Linux Networking note in my way, hope it helps.

Basic Network Interface Management: NIC Management

  •  /etc/sysconfig/network/ : configuration information for all network devices is stored in text files under this directory.
  •  /etc/init.d/network : init script will use the information in these files to setup static or dynamic IP
  •  /etc/sysconfig/networking-scripts/ifcfg-ethX : store the configuration for Ethernet device ethX
  •  /usr/share/doc/initscripts-*/sysconfig.txt listing all the valid options that can be set for a network interface.

Interface Management

  • ifconfig : most useful tool for diagnosing network problem
    • w/ no options will display the current networking settings include IP address, adapter MAC addresses, etc.
    • change w/ parameters of the network configuration as shown in following example:
      • e.g. : ifconfig eth0  192.168.x.x netmask 255.255.255.0
  • ip : it's more capable command is used to manage IP aliases, kernel routing, tunnels and policies.
    • following example shows using ip to list addresses assigned to all interfaces on system:
      • e.g. : ip addr show
      • e.g. : ip addr add 192.168.1.12/24 dev eth1
      • e.g. : ip addr del 192.168.1.12/24 dev eth1
      • e.g. : ip link show
    • running ip w/ the route argument will display the current routing
      • e.g. : ip route del default
      • e.g. : add default via 192.168.1.12
  • tcpdump : package capture
  • wireshark : package capture
Adapter Link
  • ethtool : to determine (and modify) the speed and duplex settings your Ethernet interface is operating at, or whether it has a link:
    • Display Setting for eth0:
      • ethtool eth0
      • PS: sometime old HW might try mii-tool 
  • route : to determine whether there is a default route.
    • Display default route:
      • route
      • PS : check default gateway address to make the network allows to access any other systems outside of its local network.
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.100.0   *               255.255.255.0   U     0      0        0 virbr1
192.168.122.0   *               255.255.255.0   U     0      0        0 virbr0
16.92.228.0     *               255.255.252.0   U     0      0        0 eth1
link-local      *               255.255.0.0     U     1003   0        0 eth1
default         16.92.228.1     0.0.0.0         UG    0      0        0 eth1


Kernel Module Problem
  • /etc/modprobe.d : All files underneath the /etc/modprobe.d directory which end with the .conf extension specify those options as required. Since the modprobe command can add or remove more than one module, due to modules having dependencies, we need a method of specifying what options what's options are to be used with those modules.
  • /var/lib/dhclient : The DHCP client file is named dhclient-$INTERFACE.lease
    • PS: Use the ifconfig command to find the MAC addresses assigned for Ethernet adapters. Look at the HWaddr label.
  • /etc/resolv.conf : if you find you can ping a numeric IP address but you can not ping its domain name equivalent, name resolution may be the culprit. Check the contents of the /etc/resolv.conf file for accurate information.
    • PS: NetworkManager daemon will overwrite changes to the /etc/resolv.conf. At least one, in successive order, of DNS1, DNS2, and DNS3 must be entered into the appropriate network interface configuration script in the /etc/sysconfig/network-scripts/ifcfg-ethX.
IP Aliases and Virtual Interfaces

Linux provided support for multiple IP addresses on a single physical interface by using virtual interfaces. Virtual interfaces can be configured by using the ip or ifconfig commands. 
  • eg: ethX:Y where ethX is the physical interface and Y is the virtual interface label.
    • ip addr add 10.100.0.247/24 dev eth0
    • ip -4 addr list eth0 or ifconfig eth0
The Xinetd Super Daemon

The extended internet service daemon (Xinetd) is a super daemon that manages several network services, which includes IMAP & POP servers, FTP servers, talk servers and time servers. Xinetd controls services and can apply access control list (ACL). 

ACLs can be used to determine which hosts can access the services from the network and what times of the day a service is available. Xinetd also leverages TCP wrappers to provide even more fine-grained control.

Xinetd uses individual configuration files for each service that it controls. If changes are made to a service's configuration file, then the xinetd daemon will need to be restarted. A common method for restarting the xinetd daemon is by using the SysV init script:
  • service xinetd restart
  • /etc/xinetd.conf : The main configuration file for the xinetd daemon location. It contains global parameters that affect the behavior of the xinetd daemon.
    • Each service managed by Xinetd can be enabled or disabled by editing its corresponding configuration file and modifying the "disabled = directive". If the disable=directive is missing from the file the service will be enabled.
  • /etc/xinetd.d : service specific configuration files are stored in this directory (this directory is specified in the /etc/xinetd.conf file, identified by the "includedir /etc/xinetd.d" )
  • /var/log/message (/var/log/secure): most Xinetd problems stem from "configuration errors". The Xinetd daemon does a good job of logging configuration parsing errors, so examining the /var/log/messages file is a good way to quickly identify most configuration related errors.
  • netstat, lsof or fuser : can display a list of in use (bound) ports and the processes and services that are using those ports:
    • fuser -v 22/tcp
    • [user@server /]$ sudo fuser -v 22/tcp
    • [sudo] password for wangjoh:
    •                      USER        PID ACCESS COMMAND
    • 22/tcp:              root       3214 F....  sshd
    •                      root      15542 f....  sshd
    •                      user      15546 F....  sshd
TCP Wrappers

TCP Wrappers provides a general security method for limiting access to services based on remote IP address. TCP Wrappers create a centralized administrative point where access to all services can be managed. It works by "wrapping around" existing applications.



  • /usr/sbin/tcpd or /lib/libwrap.so : binaries 
  • /etc/hosts.allow, /etc/host.deny : configuration files
  • /var/log/secure, /var/log/message : logs
  • daemon_list, client_list : syntax
Netfilter/iptables (firewall)

Netfilter is the standard Linux firewall solution. It has proven to be both robust and flexible. Because it is so capable, it can also be very complicated When a system has problems providing or connecting to network services, iptables should be one of the first suspects.

The iptables command is used to interact with Netfilter, which is inside the kernel.
  • related packages:
    • iptables
    • iptables-ipv6
  • binaires:
    • /sbin/iptables
    • /sbin/iptables-restore
    • /sbin/iptables-save
    • /sbin/ipt6tables
    • /sbin/ip6tables-restore
    • /sbin/ip6tables-save
  • configs:
    • /etc/sysconfig/iptables-config
    • /etc/sysconfig/ip6tables-config
  • logs:
    • /var/log/messages
  • data directory
    • /lib/xtables/
  • Init
    • /etc/init.d/iptables
    • /etc/init.d/ip6tables


iptables enable logging
  • iptables -L : By scanning the output of iptables to show Netfilter's contain rules
  • iptables -I INPUT 5 -m limit --limit 6/m -j LOG : enable logging before a suspect entry in the INPUT chain, with a limit of 6 messages per minute, the command.
  • /proc/net/ip_tables_names : A list of all tables that have been activated on system. 
    • PS: default "filter" table includes for Netfilter.
  • To disable Netfilter, use iptables to flush all the rules and ensure that policy allows traffic. Use the -F option to flush each table and then use the -P option to make sure each chain has a default policy of accept. It's also a good idea to remove any user-defined chains that may exist with the -X option, though this is not strictly necessary in order to disable Netfilter:
    • iptables -F : Flush each table
    • iptables -t nat -F
    • iptables -t mangle -F
    • iptables -X
    • iptables -t nat -X :-X --delete-chain
    • iptables -t mangle -X
    • iptables -P INPUT ACCEPT : -P --change policy on chain to target
    • iptables -P FORWARD ACCEPT
    • iptables -P OUTPUT ACCEPT
    • iptables -t nat -P PREROUTING ACCEPT : -t --table, table to manipulate (default: 'filter')
    • iptables -t nat -P OUTPUT ACCEPT
    • iptables -t nat -P POSTROUTING ACCEPT
    • iptables -t mangle -P PREROUTING ACCEPT
    • iptables -t mangle -P INPUT ACCEPT
    • iptables -t mangle -P FORWARD ACCEPT
    • iptables -t mangle -P OUTPUT ACCEPT
    • iptables -t mangle -P POSTROUTING ACCEPT

    Tuesday, December 2, 2014

    Linux System Boot Procedure

    I was awared the Linux boot process is a very fundamental question during the system infrastructure interview. I surf the internet but can't find the answer I feel reasonable for me. I know there has different version of Linux kernels that might be caused the differential. However, here I would like to share whatever I familiar boot process and it references RHEL6 most. I think it helps me to understand the OS fundamental concept and easier for me to follow the logic in my following research experiments.

    Linux System Main Actors - ref boot sequence:


    • System BIOS (basic I/O system)
    • MBR (master boot record, located in the 1st sector of the bootable disk /dev/hda or dev/sda)
    • GRUB (GRand Unified Bootloader, config file in /boot/grub/grub.conf)
    • initramfs Initial ramfs, initial ram base file system.
    It is a cpio archive of the initial file system that gets loaded into memory during the Linux startup processinitramfs needs to contain all of the device drivers and tools needed to mount the real root filesystem
    • initrd (initial ramdisk) is a scheme for loading a temporary root file system into memory in the boot process of the Linux kernel. 
    initrd and initramfs refer to two different methods of achieving this. Both are commonly used to make preparations before the real root file system can be mounted.
    • Linux Kernel
    The Linux kernel locate between bare metal (H/W) and O/S server ( User Apps ). It includes VFS(Virtual File System), IPC (Inter-Process Communication), Process Scheduler, Memory manager ( or Virtual Memory Manager), Dispatcher, Network Interface, Device Drivers ... etc.
    • /sbin/init lanuches bootup script from /etc/
    • Runlevel (/etc/rc.d/rc*.d/, you would see programs that start with S and K.
    a. Programs starts with S are used during startup. S for startup.
    b. Programs starts with K are used during shutdown. K for kill.
      When the Linux system is booting up, you might see various services getting started. For example, it might say “starting sendmail …. OK”. Those are the runlevel programs, executed from the run level directory as defined by your run level. )

      Linux System Boot Process in General - ref boot sequence:

      1. BIOS
      2. MBR located in the 1st sector of the bootable disk /dev/hda or dev/sda
      3. GRUB config file in /boot/grub/grub.conf
      4. Kernel:
              4.1 Init
              /etc/inittab file then execute runlevel program
              eg: grep initdefault /etc/inittab
              
              4.2 Runlevel: - showing message 
      When the Linux system is booting, see various services 
              getting started.
              e.g. it might say “starting sendmail …. OK”. 


      Linux System Boot Process in Detail - ref boot sequence:

      1. System BIOS performs three task
      • Power-On Self Test(POST)
      • Initial H/W setup and configuration
      • Loads option ROM from add-in card (SCSI, SAN HBA, RAID)
      • Selects boot device and executes MBR (master boot record, located in the 1st sector of the bootable disk /dev/hda or dev/sda)
      2. First stage GRUB in MBR(445 bytes); it loads:
      • stage1.5 GRUB using int13 BIOS calls, stage1.5 GRUB provides fs(filesystem) drivers and then loads stage2 GRUB
      3. Second stage GRUB 
      • Reads and uses configuration file or displays GRUB command prompt
      • Loads initial ram disk (usually specified)
      • Loads, decompresses, and executes selected Linux kernel from hard drive with command line arguments
      4. Linux Kernel
      • Initializes and configures H/W using drivers statistically compiled into the kernel
      • Decompresses the initramfs image and mount
      • Runs init script from initramfs image
      • init script loads kernel modules and performs task necessary to mount the real root fs(filesystem) including loading any required kernel modules stored in the initramfs image
      • Mounts the root partition passed to the kernel by the boot loader using the root=kernel command-line option(usually read only) as the root partition, replacing the initrd
      • Execut /sbin/init
      PS: for RHEL only:
      Upstart is async. replacement for the traditional init daemon. It is installed using sysvinit runlevel comptatibility mode so that it is mostly a drop-in replacement. Configuration is in the /etc/inittab and /etc/init/* files


      In sum, I know it looks complicated, let me try to see whether I can make the concept easier for understanding in the coming future.

      Share Movie Quote

      I may not know where I'm going, but I'm on my way. - (Movie: The Motorcycle Diaries), 2004

      Monday, November 24, 2014

      How to Install OpenStack Juno on VirtualBox via DevStack

      In my previous post, I share how to set up your own openstack lab via devstack. After I played around the devstack lab, I think I can make whole set up easier. Here, I would like to share how to set up the newest openstack release - juno. The general steps list as below.


      1. Setup VirtualBox
      2. Install Unbuntu
      3. Clone Code from GitHub and Execute ./stack.sh
      4. Connect to Horizon ( web management console ) and command line.

      1. Setup VirtualBox
          a. CPU: 1 Processor
          b. Memory: 4GB
          c. HDD(Drive Space): 40GB
          d. Two Network Adaptors: 1 NAT, 2 Host-only

      cpu:


      Memory:


      Drive Space:


      NCI (Network Adaptor)1 :




















      NCI (Network Adaptor)2 :

      Update:
      if you would like to connect from another laptop in the local LAN, you can try to add 3rd NIC card (NCI (Network Adaptor)3 :) and set up with "Bridge Adapter" (Some people said port forwarding, but I used Bridge). When you check the ip addr / ifconfig you will see there has IP against the 3rd NIC card.

      2. Install Ubuntu
      I install unbuntu 14.04 desktop version (ubuntu-14.04.1-desktop-amd64.iso). I skip this since there has lots of online resources.

      3. Clone code and run ./Stack.sh
      After you setup your ubuntu box, login and launch command line ( terminal ) and execute the command as below


      # sudo apt-get update
      # sudo apt-get upgrade
      sudo apt-get dist-upgrade
      sudo reboot

      sudo apt-get install git

      git clone -b stable/juno https://github.com/openstack-dev/devstack.git
      cd devstack
      wget https://dl.dropboxusercontent.com/u/44260569/localrc
      ./stack.sh

      Installation Done ! username: admin / demo and password: devstack



























      ps: 
      • Download juno version from github is using -b ( branch ) to version juno.
      • I prepared the localrc for juno. You can download it via wget. localrc is old version configuration file, the new configuration file is local.conf but when setup program check there has localrc, it will reference it first, otherwise it will follow local.conf. 
      • If you want to change user account (admin or demo) to your password, you can edit localrc file. However, it's better to use vim (# sudo apt-get install vim ) since vi is painful when I use it.
      • ./stack.sh will take a while, but if your OS is forced to sleep via such as screen saver, you better to stop it or watch it when it running to avoid the interruption.


      4. Connect to horizon and ssh
      Since I setup 2nd host-only Network adapter. The 1st NCI card is eth0 and 2nd should be eth1. You just try the command(# ifconfig) to get the eth1 IP address and should be able to connect to your juno lab via it.

      Check 2nd NCI, eth1 via linux cli
      # ifconfig

      Use Host-only Adaptor's IP for horizon login




























      Connect via SSH


















      PS: SSH login via username demo and password devstack.
      You can run source openrc admin admin to grant admin privilege for your command line test after you login.

      here is the example.
      # cd devstack
      # source openrc admin admin

      This is easier way I found so far, please free to share if you found the better way to setup the newest release openstack lab.

      PS: the solution of the post has some issues, I listed as below fyi. For these issues, I will study and post the follow up solutions for it.

      • All OpenStack Modules works but SWIFT. 
      I actually fixed the issue. Here is the solution which updated the localrc (the dropbox download has been updated) and add parameter as below. 


      # SWIFT - Object Storage

      ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account



      Now, the swift is running ok , here is the example. I create on "Container1" and upload two files (objects).








      • After reboot, the virtualbox network configuration might corrupt. You might take a snapshot after your first installation and retrieve the status back.
      I read some blogs mentioned about the services didn't up after reboot. Some people said it's a known issue and some people said it's not a "issue" but is that way by design. No matter how, I post the update as below which may helps you restart the devstack services.

      Restart DevStack

      1. First make sure your volume up.
      Bring volume group online so that cinder-volume will start without errors

          ls demo@demo-juno:/opt/stack/data

      double check your cinder-volume name. and you losetup to set up and control loop devices.
      sudo losetup -f /opt/stack/data/stack-volumes-backing-file
      
      2. Use rejoin-stack.sh to restart the DevStack
      ./rejoin-stack.sh

      Sunday, November 9, 2014

      Voice Recognition with Java and C#

      In previous post, I mentioned AIML. It's an AI ChatBot to digest natural language chat. I am always think it's cool to combine with Voice Recognition. Thus, people can be not only asking the ChatBot quesion for the answer but also asking AI ChatBot to "do something". Here is an example that the Voice activation with system (US Pizza Papa John’s allows to order pizza via Voice System, same with their supply chain system :http://www.papajohns.com/ordering/mobile.shtm, http://www.mmh.com/article/papa_johns_fresh_take_on_wms_and_voice_technology )

      I spent my weekend and try a little bit voice recognition on demo with Java and C#, here are the details.
      (PS: It’s better you try this in Meeting Room by alone or at your home, otherwise when you test, you will look like idiot )    J

      Java Version
      ---------------

      Preparation: 
      ----------------
      JDK 8 ( J2EE )
      Eclipse SDK ( LUNA )
      JSAPI ( Included in Sphinx 4 : jsapi.jar)

      1.       Create a java project and add jars

      2.       Test HelloWorld.jar

      3.       Try Hello World Source Code

      a.       HelloWorld.java
      b.      hello.gram ( grammer file )
      c.       helloworld.config.xml ( config file )

      4.       I added new recognition grammer “Hey Johnny” in *.gram


      Modified *.java showing in command line.


      5.       I don’t wanna change config.xml now so I follow whatever in manifest/config first. ( add one more jar into UserLib ). Now it’s working


      Last: I can duplicate this voice recognition to trigger “Check Vagrant Version” with my Java ChatBot. Then it will trigger check vagrant version cli.
      -----------------------------------------------------------------------------------
      C# version, here has full example.

      add  System.Speech reference



      Here is form1.cs example:


      using System;
      using System.Collections.Generic;
      using System.ComponentModel;
      using System.Data;
      using System.Drawing;
      using System.Linq;
      using System.Text;
      using System.Threading.Tasks;
      using System.Windows.Forms;
      using System.Speech.Recognition;

      namespace CSharpVoiceRecognition
      {
          public partial class Form1 : Form
          {
              SpeechRecognitionEngine recEngine = new SpeechRecognitionEngine();
              public Form1()
              {
                  InitializeComponent();
              }

              private void Form1_Load(object sender, EventArgs e)
              {
                  Choices commands = new Choices();
                  commands.Add(new string[] { "Say Hello""Print My Name""Love You""Please Check My System" });
                  GrammarBuilder gBuilder = new GrammarBuilder();
                  gBuilder.Append(commands);
                  Grammar grammer = new Grammar(gBuilder);

                  recEngine.LoadGrammarAsync(grammer);
                  recEngine.SetInputToDefaultAudioDevice();
                  recEngine.SpeechRecognized +=recEngine_SpeechRecognized;
                 
              }

              private void recEngine_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
              {
                  switch (e.Result.Text)
                  {
                      case "Say Hello":
                          MessageBox.Show("Hello. How are you ?");
                          break;
                      case "Print My Name":
                          richTextBox1.Text+="\nJohnny";
                          break;
                      case "Love You":
                          richTextBox1.Text += "\nLove You Too!";
                          break;
                      case "Please Check My System":
                          richTextBox1.Text += "\nNo Problem, I'm checking vagrant.";
                          break;
                      default:
                          richTextBox1.Text = e.Result.Text;
                          break;
                  }
                  //throw new NotImplementedException();
              }

              private void btnEnable_Click(object sender, EventArgs e)
              {
                  recEngine.RecognizeAsync(RecognizeMode.Multiple);
                  richTextBox1.Text += "\nEnable Voice Recognition";
              }

              private void btnDisable_Click(object sender, EventArgs e)
              {
                  recEngine.RecognizeAsyncStop();
                  richTextBox1.Text += "\nDisable Voice Recognition";
              }
          }

      }