Sunday, July 18, 2021

How to install Kubeflow on Rancher K8s

 

Kubeflow Architecture



How to install Kubeflow

Preparation

Required Tools

  • Current only work for Kubeflow v1.0.0 (Newest is v1.3 on Portal and v1.7 ? in github ?)

  • Rancher 2.x

  • docker 19.03.13

    $ docker version
    Client: Docker Engine - Community
    Version:           19.03.13
    API version:       1.40
    Go version:        go1.13.15
    Git commit:        4484c46d9d
    Built:             Wed Sep 16 17:03:45 2020
    OS/Arch:           linux/amd64
    Experimental:      false
    
    Server: Docker Engine - Community
    Engine:
    Version:          19.03.13
    API version:      1.40 (minimum version 1.12)
    Go version:       go1.13.15
    Git commit:       4484c46d9d
    Built:            Wed Sep 16 17:02:21 2020
    OS/Arch:          linux/amd64
    Experimental:     false
    containerd:
    Version:          1.4.4
    GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
    nvidia:
    Version:          1.0.0-rc93
    GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
    docker-init:
    Version:          0.18.0
    GitCommit:        fec3683
    
    • If you are running NVidia GPU server than please install nvidia container toolkit https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#id2

      • Install NVidia Toolkit for Docker (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-centos-7-8)
        • Setting up NVIDIA Container Toolkit
          # Setup the stable repository and the GPG key:
          $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
          && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
          
          # install Nvidia Docker Toolkit
          $ sudo yum clean expire-cache
          $ sudo yum install -y nvidia-docker2
          $ sudo systemctl restart docker
          
    • setting docker daemons.json

      • setup docker damemon - for NVidia GPU Server
        sudo mkdir /etc/docker
        
        sudo cat > /etc/docker/daemon.json <<EOF
        {
            "default-runtime": "nvidia",
            "runtimes": {
                "nvidia": {
                    "path": "/usr/bin/nvidia-container-runtime",
                    "runtimeArgs": []
                }
            },
            "exec-opts": ["native.cgroupdriver=systemd"],
            "log-driver": "json-file",
            "log-opts": {
                "max-size": "100m"
            },
            "storage-driver": "overlay2",
            "storage-opts": ["overlay2.override_kernel_check=true"]
        }
        EOF
        
      • setup docker damemon - for NVidia GPU Server
        sudo systemctl --now enable docker
        or
        sudo systemctl enable docker
        sudo systemctl start docker
        systemctl status docker
        
  • K8s v1.16.15

    $ kubectl version
    Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:40:00Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:31:21Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}    
    
    • kubectl client can be update by as below
      $ which kubectl
      /usr/bin/kubectl
      $ mkdir kubectl
      $ cd kubectl
      $  curl -LO https://dl.k8s.io/release/v1.16.15/bin/linux/amd64/kubectl
      $ sudo cp kubectl /usr/bin/kubectl
      $ kubectl version --client
      Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:40:00Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
      
    • kubectl server version can be selected when you create rancher k8s cluster
      • Rememeber Copy config file to ~/.kube/config
        $ mkdir .kube
        $ cd .kube/
        $ vi config
        # copy and paste k8s config
        $ chmod go-r ~/.kube/config
        
      • Rememeber Copy config file to ~/.kube/config
        $ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
        $ chmod 700 get_helm.sh
        $ ./get_helm.sh
        $ helm version
        version.BuildInfo{Version:"v3.6.0", GitCommit:"7f2df6467771a75f5646b7f12afb408590ed1755", GitTreeState:"clean", GoVersion:"go1.16.3"}
        
      • if you are running NVidia GPU you will need Nvidia Plug-in$ kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.9.0/nvidia-device-plugin.yml
  • kfctl v1.2.0-0-gbc038f9

    $ mkdir kfctl
    $ cd kfctl
    $wget https://github.com/kubeflow/kfctl/releases/download/v1.2.0/kfctl_v1.2.0-0-gbc038f9_linux.tar.gz
    $ tar -xvf kfctl_v1.2.0-0-gbc038f9_linux.tar.gz
    $ chmod 755 kfctl
    $ cp kfctl /usr/bin
    $ kfctl version
    kfctl v1.2.0-0-gbc038f9
    
  • kustomize v4.1.3

    $ mkdir kustomize
    $ cd kustomize
    $ wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv4.1.3/kustomize_v4.1.3_linux_amd64.tar.gz
    $ tar -xzvf kustomize_v4.1.3_linux_amd64.tar.gz
    $ chmod 755 kustomize
    $ mv kustomize /use/bin/
    $ kustomize version
    {Version:kustomize/v4.1.3 GitCommit:0f614e92f72f1b938a9171b964d90b197ca8fb68 BuildDate:2021-05-20T20:52:40Z GoOs:linux GoArch:amd64}
    

Default Storage Class via Local Path

  • Add Extra Drive e.g. sdc for your nodes, you can skip if you don't need default storage class from local path

    • make partition fdisk /dev/sdc

    • make filesystem mkfs.xfs /dev/sdc1

    • make mount point mkdir /mnt/sdc1

    • change permission chmod 711 /mnt/sdc1

    • mount setting - check UUID

      $ sudo blkid /dev/sdb1
      /dev/sdb1: UUID="cae2b0cb-c45f-4f08-b698-8a6c49f80b76" TYPE="xfs" PARTUUID="f5183f01-c945-4a77-8aeb-a38296f67024"
      add /etc/fstab
      
    • edit /etc/fstab e.g. vi /etc/fstab

      #
      # /etc/fstab
      # Created by anaconda on Fri Mar  8 05:01:22 2019
      #
      # Accessible filesystems, by reference, are maintained under '/dev/disk'
      # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
      #
      /dev/mapper/centos-rootlv /                       xfs     defaults        0 0
      UUID=73a9046c-bd04-4021-a585-cd6ad1a9fe5f /boot                   xfs     defaults        0 0
      /dev/mapper/centos-swaplv swap                    swap    defaults        0 0
      /dev/mapper/centos-swaplv      none    swap    sw,comment=cloudconfig  0       0
      UUID=cae2b0cb-c45f-4f08-b698-8a6c49f80b76 /mnt/sdc1       xfs     defaults        0       0
      e
      
    • mount $ sudo mount -a

  • Setup local path storage class

    • login master node
      $ mkdir local-path-provisioner
      $ cd local-path-provisioner
      $ sudo yum install git -y
      $ git clone https://github.com/rancher/local-path-provisioner.git
      $ cd deploy
      $ vi local-path-storage.yaml
      # Search /opt and change it to e.g., </mnt/sdc1/local-path-provisioner>
      data:
          config.json: |-
              {
                      "nodePathMap":[
                      {
                              "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
                              "paths":["/mnt/sdc1/local-path-provisioner"]
                      }
                      ]
              }
      
      $ kubectl apply -f local-path-storage.yaml
      namespace/local-path-storage created
      serviceaccount/local-path-provisioner-service-account created
      clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
      clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
      deployment.apps/local-path-provisioner created
      storageclass.storage.k8s.io/local-path created
      configmap/local-path-config created
      
      # double check
      $ ls /mnt/sdc1/local-path-provisioner/ -larth
      
      # mark default
      $ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
      storageclass.storage.k8s.io/local-path patched
      
    • You can double check in Rancher

Installation

Setup Env variables Very Important

  • Setup Enviroment Variables
    $ mkdir kubeflow
    $ cd kubeflow
    
    $ export KF_NAME=kubeflow
    
    # your current directory e.g., pwd
    $ export BASE_DIR="/home/idps"
    
    $ export KF_DIR=${BASE_DIR}/${KF_NAME}
    
    # can skip
    $ mkdir -p ${KF_DIR}
    $ cd ${KF_DIR}
    
    $ wget https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.0.yaml
    $ export CONFIG_URI=${KF_DIR}/"kfctl_k8s_istio.v1.0.0.yaml"
    
    $ echo ${CONFIG_URI}
    /home/idps/kubeflow/kfctl_k8s_istio.v1.0.0.yaml
    $ ls /home/idps/kubeflow/kfctl_k8s_istio.v1.0.0.yaml
    /home/idps/kubeflow/kfctl_k8s_istio.v1.0.0.yaml    
    
  • Or you can install directly from tar.gz but this one it never works for me.
    $ mkdir kubeflow
    $ cd kubeflow
    $ curl -L -O https://github.com/kubeflow/kfctl/releases/download/v1.0/kfctl_v1.0-0-g94c35cf_linux.tar.gz
    $ tar -xvf kfctl_v1.0-0-g94c35cf_linux.tar.gz
    $ mkdir yaml
    $ cd yaml
    $ export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.0.yaml"
    

Install Kubeflow

  • Install (It will run a while since it will download images from external)

    $ kfctl apply -V -f ${CONFIG_URI}
    
    • if you are seeing this warn then it's fine just wait the cert-manager spin up

      WARN[0017] Encountered error applying application cert-manager:  (kubeflow.error): Code 500 with message: Apply.Run : error when creating "/tmp/kout072586623": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server could not find the requested resource  filename="kustomize/kustomize.go:284"
      
    • Double check

      $ kubectl get pods -n cert-manager
      NAME                                      READY   STATUS    RESTARTS   AGE
      cert-manager-cainjector-c578b68fc-d47zg   1/1     Running   0          100m
      cert-manager-fcc6cd946-h84bh              1/1     Running   0          100m
      cert-manager-webhook-657b94c676-v8mjv     1/1     Running   0          100m
      
      $ kubectl get pods -n istio-system
      NAME                                      READY   STATUS      RESTARTS   AGE
      cluster-local-gateway-78f6cbff8d-vtghm    1/1     Running     0          100m
      grafana-68bcfd88b6-2ndhj                  1/1     Running     0          100m
      istio-citadel-7dd6877d4d-blm58            1/1     Running     0          100m
      istio-cleanup-secrets-1.1.6-pzhkr         0/1     Completed   0          100m
      istio-egressgateway-7c888bd9b9-bgv5k      1/1     Running     0          100m
      istio-galley-5bc58d7c89-p2gbl             1/1     Running     0          100m
      istio-grafana-post-install-1.1.6-dbv25    0/1     Completed   0          100m
      istio-ingressgateway-866fb99878-rkhjs     1/1     Running     0          100m
      istio-pilot-67f9bd57b-fw5h5               2/2     Running     0          100m
      istio-policy-749ff546dd-g8brc             2/2     Running     2          100m
      istio-security-post-install-1.1.6-8fqzl   0/1     Completed   0          100m
      istio-sidecar-injector-cc5ddbc7-fpzxx     1/1     Running     0          100m
      istio-telemetry-6f6d8db656-tcgdx          2/2     Running     2          100m
      istio-tracing-84cbc6bc8-kdtqw             1/1     Running     0          100m
      kiali-7879b57b46-xbw6n                    1/1     Running     0          100m
      prometheus-744f885d74-4bpjp               1/1     Running     0          100m
      
      $ kubectl get pods -n knative-serving
      NAME                               READY   STATUS    RESTARTS   AGE
      activator-58595c998d-84jwb         2/2     Running   1          97m
      autoscaler-7ffb4cf7d7-jspgd        2/2     Running   2          97m
      autoscaler-hpa-686b99f459-f6v9c    1/1     Running   0          97m
      controller-c6d7f946-bs2n2          1/1     Running   0          97m
      networking-istio-ff8674ddf-l94d5   1/1     Running   0          97m
      webhook-6d99c5dbbf-bfdmw           1/1     Running   0          97m
      
      $ kubectl get pods -n kubeflow
      NAME                                                          READY   STATUS      RESTARTS   AGE
      admission-webhook-bootstrap-stateful-set-0                    1/1     Running     0          98m
      admission-webhook-deployment-59bc556b94-w2ghb                 1/1     Running     0          97m
      application-controller-stateful-set-0                         1/1     Running     0          100m
      argo-ui-5f845464d7-8cds6                                      1/1     Running     0          98m
      centraldashboard-d5c6d6bf-lfdsh                               1/1     Running     0          98m
      jupyter-web-app-deployment-544b7d5684-vqjn8                   1/1     Running     0          98m
      katib-controller-6b87947df8-c8jg4                             1/1     Running     0          97m
      katib-db-manager-54b64f99b-zsf7n                              1/1     Running     0          97m
      katib-mysql-74747879d7-vqjpp                                  1/1     Running     0          97m
      katib-ui-76f84754b6-ch8qq                                     1/1     Running     0          97m
      kfserving-controller-manager-0                                2/2     Running     1          98m
      metacontroller-0                                              1/1     Running     0          98m
      metadata-db-79d6cf9d94-blzgc                                  1/1     Running     0          98m
      metadata-deployment-5dd4c9d4cf-j7t89                          1/1     Running     0          98m
      metadata-envoy-deployment-5b9f9466d9-mb7mf                    1/1     Running     0          98m
      metadata-grpc-deployment-66cf7949ff-m4q6v                     1/1     Running     1          98m
      metadata-ui-8968fc7d9-559fz                                   1/1     Running     0          98m
      minio-5dc88dd55c-nf7rz                                        1/1     Running     0          97m
      ml-pipeline-55b669bf4d-fcjz7                                  1/1     Running     0          97m
      ml-pipeline-ml-pipeline-visualizationserver-c489f5dd8-7lcbh   1/1     Running     0          97m
      ml-pipeline-persistenceagent-f54b4dcf5-lc2f2                  1/1     Running     0          97m
      ml-pipeline-scheduledworkflow-7f5d9d967b-brw6d                1/1     Running     0          97m
      ml-pipeline-ui-7bb97bf8d8-dfnkd                               1/1     Running     0          97m
      ml-pipeline-viewer-controller-deployment-584cd7674b-ww575     1/1     Running     0          97m
      mysql-66c5c7bf56-62rr8                                        1/1     Running     0          97m
      notebook-controller-deployment-576589db9d-sl6dx               1/1     Running     0          98m
      profiles-deployment-769b65b76d-jpgs4                          2/2     Running     0          97m
      pytorch-operator-666dd4cd49-k5ph9                             1/1     Running     0          98m
      seldon-controller-manager-5d96986d47-h2sd4                    1/1     Running     0          97m
      spark-operatorcrd-cleanup-v6jbm                               0/2     Completed   0          98m
      spark-operatorsparkoperator-7c484c6859-tmgtg                  1/1     Running     0          98m
      spartakus-volunteer-7465bcbdc-q7vbd                           1/1     Running     0          97m
      tensorboard-6549cd78c9-bm57g                                  1/1     Running     0          97m
      tf-job-operator-7574b968b5-27xqt                              1/1     Running     0          97m
      workflow-controller-6db95548dd-s4mzx                          1/1     Running     0          98m
      
    • You can also create a project e.g., Kubeflow in Rancher and Move istio-systemknative-servingkubeflow and local-path-storage to your project e.g., Kubeflow

      • Then go to Prject's Resource -->Workloads and will display more detail for the status.
  • Delete Don't manual delete one by one if you want to restart

    $ kfctl Delete  -V -f ${CONFIG_URI}
    
  • Login Kubeflow

    • Under Rancher, it's pretty straightforward. Just go to Project's Resource --> Workloads --> Active istio-ingressgateway try those tcp ports since it's automatically mapping w/ Server Ports.

    • Check Port Mapping, e.g., 80:31380/TCP the port is 31388

      $ kubectl -n istio-system get svc istio-ingressgateway
      NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                                                                                                      AGE
      istio-ingressgateway   NodePort   10.43.97.164   <none>        15020:30228/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:30932/TCP,15030:30521/TCP,15031:30290/TCP,15032:31784/TCP,15443:30939/TCP   120m

Wednesday, March 10, 2021

k8s Monitoring via Image Processing Example

Image Processing K8s Monitoring

Preparation and Scoping

  • a server for hosting e.g. test-host
    • Monitoring Server Apps:
      • Prometheus
      • Grafana
    • Logging Server Apps:
      • Elasticsearch
      • Kibana
    • Tracing Server Apps:
      • Jaeger
  • 4 node K8s Cluster: Master x 1 and Slaves x 3 e.g. img-test-01~04
    • Monitoring Client Apps
      • Node Exporter
      • Kube-state-metrics
      • cAdvisor
    • Logging
      • Fluentd
  • Opentracing Module: It's Python Module, thus it can be your local and in the future, it can be your container image wrap your features/applications.

Default K8s Dashboard:

Original K8s Dashboard - https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

Steps on K8s Controller

install Default K8s Dashboard
```
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
```
Leverage Kubctl Proxy to forward port 8001
```
### $ kubectl proxy&
[idps@img-test-01 ~]$ kubectl proxy&
[1] 12911
[idps@img-test-01 ~]$ Starting to serve on 127.0.0.1:8001
```
Get Token
```
### $ Create Account and Get Token
[idps@img-test-01 ~]$ kubectl create serviceaccount dashboard-admin-sa
serviceaccount/dashboard-admin-sa created

[idps@img-test-01 ~]$ kubectl create clusterrolebinding dashboard-admin-sa --clusterrole=cluster-admin --serviceaccount=default:dashboard-admin-sa
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin-sa created

[idps@img-test-01 ~]$ kubectl get secrets
NAME                             TYPE                                  DATA   AGE
dashboard-admin-sa-token-rkq7z   kubernetes.io/service-account-token   3      41s
default-token-9kt7g              kubernetes.io/service-account-token   3      114d

[idps@img-test-01 ~]$ kubectl describe secret dashboard-admin-sa-token-rkq7z
Name:         dashboard-admin-sa-token-rkq7z
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin-sa
            kubernetes.io/service-account.uid: 94726478-8d7c-4dac-8c4c-7d8edadc5e20

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  7 bytes
token:      xxx
```
Steps on Local to access K8s Default Dashboard
```
### create ssh tunnel , you can use $ps -ef | grep ssh and if there is no ssh tunnel processing running please re-run it.
~ ssh -g -L 8001:localhost:8001 -f -N idps@172.30.30.106
idps@172.30.30.106's password:
~
```
put the url and login via token you found from the new account

Prometheus w/ Node Exporter

Node Exporter: https://github.com/prometheus/node_exporter

  • install node exporter can reference nttd_playbook repo

    Kube-state-metrics

  • Kube-state-metrics listens to the Kubernetes API server and generates metrics about the state of numerous Kubernetes objects, including cron jobs, config maps, pods, and nodes. These metrics are unmodified, unlike kubectl metrics that use the same Kubernetes API but apply some heuristics to display comprehensible and readable messages.
Install from github
```
git clone https://github.com/kubernetes/kube-state-metrics.git
cd kube-state-metrics
kubectl apply -f examples/standard
```
Install from helm chart
```
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update

$ kubectl create namespace my-monitor

# Helm 3
$ helm install [RELEASE_NAME] kube-state-metrics/kube-state-metrics --namespace my-monitor

# Helm 2
$ helm install --name [RELEASE_NAME] kube-state-metrics/kube-state-metrics --namespace my-monitor

$ helm install --namespace my-k8s-monitor kube-state-metrics/kube-state-metrics --name my-k8s-monitor --set service.type=NodePort --set service.nodePort=30001

# Helm 3
$ helm uninstall [RELEASE_NAME]

# Helm 2
# helm delete --purge [RELEASE_NAME] --purge


[idps@img-test-01 tmp]$ kubectl get all -n my-k8s-monitor
NAME                                                     READY   STATUS    RESTARTS   AGE
pod/my-k8s-monitor-kube-state-metrics-77c7b956c8-7tdbk   1/1     Running   0          51s

NAME                                        TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/my-k8s-monitor-kube-state-metrics   NodePort   10.97.115.127   <none>        8080:30001/TCP   51s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-k8s-monitor-kube-state-metrics   1/1     1            1           51s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/my-k8s-monitor-kube-state-metrics-77c7b956c8   1         1         1       51s
[idps@img-test-01 tmp]$ curl http://localhost:30001/metrics
```
Configure Prometheus
```
$ [idps@test-host ~]$ cat /etc/prometheus/prometheus.yml

- job_name: k8s-kube-state-metrics-img
honor_timestamps: true
metrics_path: /metrics
scheme: http
static_configs:
    - targets: ['img-test-01:30001']
# sudo systemctl restart prometheus
# sudo systemctl status prometheus

[idps@test-host ~]$ sudo systemctl status prometheus
● prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2021-03-06 00:00:15 PST; 7s ago
Main PID: 28166 (prometheus)
Tasks: 53 (limit: 104856)
Memory: 248.6M
CGroup: /system.slice/prometheus.service
    └─28166 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/local/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --storage.tsdb.retention.time=90>

Mar 06 00:00:18 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:18.084Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9171 maxSegment=9174
Mar 06 00:00:19 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:19.782Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9172 maxSegment=9174
Mar 06 00:00:20 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:20.058Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9173 maxSegment=9174
Mar 06 00:00:20 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:20.058Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9174 maxSegment=9174
Mar 06 00:00:20 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:20.058Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=4.354068139s
Mar 06 00:00:20 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:20.104Z caller=main.go:693 fs_type=XFS_SUPER_MAGIC
Mar 06 00:00:20 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:20.104Z caller=main.go:694 msg="TSDB started"
Mar 06 00:00:20 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:20.104Z caller=main.go:798 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Mar 06 00:00:20 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:20.105Z caller=main.go:826 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
Mar 06 00:00:20 test-host.test.internal prometheus[28166]: level=info ts=2021-03-06T08:00:20.105Z caller=main.go:645 msg="Server is ready to receive web requests."

# then you should be able to see the crape query in prometheus Web UI
http://test-host:9090/graph?g0.range_input=1h&g0.expr=container_cpu_cfs_throttled_periods_total&g0.tab=1

```

cAdvisor (In Progress)

cAdvisor: From Chianingwang Github Repo (Waiting for upload the helm chart)

```
[idps@img-test-01 cadvisor]$ helm install --namespace my-k8s-monitor --name my-k8s-cadvisor -f values.yaml ./
NAME:   my-k8s-cadvisor
LAST DEPLOYED: Sat Mar  6 16:31:32 2021
NAMESPACE: my-k8s-monitor
STATUS: DEPLOYED

RESOURCES:
==> v1/DaemonSet
NAME      AGE
cadvisor  0s

==> v1/Pod(related)
NAME            AGE
cadvisor-9qcm8  0s
cadvisor-hfr4h  0s
cadvisor-k459p  0s
cadvisor-tq4br  0s

==> v1/Service
NAME      AGE
cadvisor  0s

==> v1/ServiceAccount
NAME             AGE
my-k8s-cadvisor  0s

[idps@img-test-01 cadvisor]$ kubectl get all -n my-k8s-monitor
NAME                                                     READY   STATUS    RESTARTS   AGE
pod/cadvisor-9qcm8                                       1/1     Running   0          76s
pod/cadvisor-hfr4h                                       1/1     Running   0          76s
pod/cadvisor-k459p                                       1/1     Running   0          76s
pod/cadvisor-tq4br                                       1/1     Running   0          76s
pod/my-k8s-monitor-kube-state-metrics-77c7b956c8-7tdbk   1/1     Running   0          16h

NAME                                        TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/cadvisor                            NodePort   10.96.23.179    <none>        8080:30002/TCP   76s
service/my-k8s-monitor-kube-state-metrics   NodePort   10.97.115.127   <none>        8080:30001/TCP   16h

NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/cadvisor   4         4         4       4            4           <none>          76s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-k8s-monitor-kube-state-metrics   1/1     1            1           16h

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/my-k8s-monitor-kube-state-metrics-77c7b956c8   1         1         1       16h

[idps@test-host ~]$ cat /etc/prometheus/prometheus.yml
- job_name: k8s-kube-state-metrics-img
honor_timestamps: true
metrics_path: /metrics
scheme: http
static_configs:
- targets: ['img-test-01:30001']
- job_name: k8s-cadvisor-img
honor_timestamps: true
metrics_path: /metrics
scheme: http
static_configs:
- targets: ['img-test-01:30002']
[idps@test-host ~]$ sudo systemctl restart prometheus

[idps@test-host ~]$ sudo systemctl status prometheus
● prometheus.service - Prometheus
Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2021-03-06 16:36:14 PST; 12s ago
Main PID: 83754 (prometheus)
Tasks: 53 (limit: 104856)
Memory: 172.0M
CGroup: /system.slice/prometheus.service
    └─83754 /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/local/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --storage.tsdb.retention.time=90>

Mar 06 16:36:19 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:19.026Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9188 maxSegment=9191
Mar 06 16:36:21 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:21.926Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9189 maxSegment=9191
Mar 06 16:36:22 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:22.248Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9190 maxSegment=9191
Mar 06 16:36:22 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:22.249Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9191 maxSegment=9191
Mar 06 16:36:22 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:22.249Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=7.223842521s
Mar 06 16:36:22 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:22.272Z caller=main.go:693 fs_type=XFS_SUPER_MAGIC
Mar 06 16:36:22 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:22.272Z caller=main.go:694 msg="TSDB started"
Mar 06 16:36:22 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:22.272Z caller=main.go:798 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Mar 06 16:36:22 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:22.274Z caller=main.go:826 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
Mar 06 16:36:22 test-host.test.internal prometheus[83754]: level=info ts=2021-03-07T00:36:22.274Z caller=main.go:645 msg="Server is ready to receive web requests."
```

cAdvisor: https://github.com/ckotzbauer/helm-charts/tree/master/charts/cadvisor (from helm repo but there is no config in value.yaml for change to NodePort)

```
[idps@img-test-01 ~]$ helm repo add ckotzbauer https://ckotzbauer.github.io/helm-charts
"ckotzbauer" has been added to your repositories

[idps@img-test-01 ~]$ helm search ckotzbauer/
NAME                                       CHART VERSION    APP VERSION    DESCRIPTION
ckotzbauer/access-manager                  0.5.1            0.5.1          Kubernetes-Operator to simplify RBAC configurations
ckotzbauer/cadvisor                        1.2.2            0.38.6         A chart for a Cadvisor deployment
ckotzbauer/nfs-client-provisioner          1.0.2            3.1.0          nfs-client is an automatic provisioner that used your *al...
ckotzbauer/prometheus-blackbox-exporter    1.0.3            0.18.0         Prometheus Blackbox Exporter

[idps@img-test-01 ~]$ helm search ckotzbauer/ --versions | grep cadvisor
ckotzbauer/cadvisor                        1.2.2            0.38.6         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.2.1            0.38.4         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.2.0            0.37.0         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.1.4            0.36.0         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.1.3            0.36.0         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.1.2            0.36.0         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.1.1            0.36.0         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.1.0            0.36.0         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.0.1            0.36.0         A chart for a Cadvisor deployment
ckotzbauer/cadvisor                        1.0.0            0.35.0         A chart for a Cadvisor deployment

[idps@img-test-01 ~]$ helm install --namespace my-k8s-monitor ckotzbauer/cadvisor --name my-k8s-cadvisor
NAME:   my-k8s-cadvisor
LAST DEPLOYED: Sat Mar  6 16:06:57 2021
NAMESPACE: my-k8s-monitor
STATUS: DEPLOYED

RESOURCES:
==> v1/DaemonSet
NAME      AGE
cadvisor  0s

==> v1/Pod(related)
NAME            AGE
cadvisor-l5qb5  0s
cadvisor-phwxt  0s
cadvisor-tkvzm  0s
cadvisor-xzvmm  0s

==> v1/Service
NAME      AGE
cadvisor  0s

==> v1/ServiceAccount
NAME             AGE
my-k8s-cadvisor  0s

[idps@img-test-01 ~]$ helm list
NAME               REVISION    UPDATED                     STATUS      CHART                            APP VERSION            NAMESPACE
my-k8s-cadvisor    1           Sat Mar  6 16:06:57 2021    DEPLOYED    cadvisor-1.2.2                   0.38.6                 my-k8s-monitor
my-k8s-monitor     1           Fri Mar  5 23:50:19 2021    DEPLOYED    kube-state-metrics-2.13.0        1.9.8                  my-k8s-monitor
my-kafka-op        1           Mon Mar  1 20:54:26 2021    DEPLOYED    strimzi-kafka-operator-0.16.2    0.16.2                 my-kafka
spark-operator     1           Mon Mar  1 21:34:54 2021    DEPLOYED    sparkoperator-0.6.6              v1beta2-1.1.0-2.4.5    spark-operator

[idps@img-test-01 ~]$ kubectl get all -n my-k8s-monitor
NAME                                                     READY   STATUS    RESTARTS   AGE
pod/cadvisor-l5qb5                                       1/1     Running   0          24s
pod/cadvisor-phwxt                                       1/1     Running   0          24s
pod/cadvisor-tkvzm                                       1/1     Running   0          24s
pod/cadvisor-xzvmm                                       1/1     Running   0          24s
pod/my-k8s-monitor-kube-state-metrics-77c7b956c8-7tdbk   1/1     Running   0          16h

NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/cadvisor                            ClusterIP   10.96.124.167   <none>        8080/TCP         24s
service/my-k8s-monitor-kube-state-metrics   NodePort    10.97.115.127   <none>        8080:30001/TCP   16h

NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/cadvisor   4         4         4       4            4           <none>          24s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-k8s-monitor-kube-state-metrics   1/1     1            1           16h

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/my-k8s-monitor-kube-state-metrics-77c7b956c8   1         1         1       16h
```

Prometheus: https://prometheus.io/

  • configure Prometheus to collect metrics from node exporter can reference nttd_playbook repo

Grafana: https://grafana.com/

Grafana Dashboard (In Progress)

Logging

Fluentd: https://www.fluentd.org/

ElasticSearch: https://www.elastic.co/

Kibana: https://www.elastic.co/

Install Elasticsearch and Kibana via Single Node Docker

  • detach mode -d, without detach mode just remove -d

     [idps@test-host ~]$ docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.11.1
     aa3b4499e7f725ccb13f3601a6972b013e8301f91846861ddd604ef3c0d3bade
    
     [idps@test-host ~]$ docker run -d --name kibana --link elasticsearch:elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:7.11.1
     c9c19afd7755b8846c5c3b9784caae415110ee261856b5bae4b6e495cdf69d11
    
     [idps@test-host ~]$ docker ps -a
     CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS                      PORTS                                            NAMES
     c9c19afd7755        docker.elastic.co/kibana/kibana:7.11.1                 "/bin/tini -- /usr/l…"   3 minutes ago       Up 3 minutes                0.0.0.0:5601->5601/tcp                           kibana
     aa3b4499e7f7        docker.elastic.co/elasticsearch/elasticsearch:7.11.1   "/bin/tini -- /usr/l…"   3 minutes ago       Up 3 minutes                0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch
    

Install Fluentd helm chart and configuration forward the log to elasticsearch

```
# prepare a yaml file
$ vi fluentd-daemonset-values.yaml
elasticsearch:
hosts: ["test-host:9200"]

# add kiwigrid repo this is only for helm chart v1 (v2 will error out since the helm version we own)
$ helm repo add kiwigrid https://kiwigrid.github.io

# install fluentd helm chart
$ helm install --namespace my-k8s-monitor kiwigrid/fluentd-elasticsearch -f fluentd-daemonset-values.yaml --name my-k8s-logging

# display result
[idps@img-test-01 ~]$ helm list | grep logging
my-k8s-logging     1           Mon Mar  8 18:15:46 2021    DEPLOYED    fluentd-elasticsearch-9.6.2      3.0.4                  my-k8s-monitor

# Go to Kibana Web UI (7.11.1)
http://test-host:5601
```
  • go to Management --> stack management -> Index patterns --> create index "logstash-*" --> Time field: '@timestamp'

  • go to Analytics --> Discovery

Tracing

Jaeger: https://github.com/jaegertracing/jaeger

OpenTrace: https://opentracing.io/

Reference Tutorials

Install Jaeger

Install Jaeger all-in-one
```
$ docker run -d --name jaeger -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411 jaegertracing/all-in-one:latest

[idps@test-host ~]$ docker run -d --name jaeger -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411 jaegertracing/all-in-one:latest
Unable to find image 'jaegertracing/all-in-one:latest' locally
latest: Pulling from jaegertracing/all-in-one
f84cab65f19f: Pull complete
345ee7d1a790: Pull complete
324734746d2e: Pull complete
2e77c8b8b0bf: Pull complete
Digest: sha256:369d9a662ff6fccecfc6143ab86f0c70b1908e7874d156e4eeb63fe28ac96d5f
Status: Downloaded newer image for jaegertracing/all-in-one:latest
5572aa3cc12b1005090124a3084a8216b57fb86aa4fb93265d1655e52b7db938

[idps@test-host ~]$ docker ps -a
CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS                   PORTS                                                                                                                                                                     NAMES
5572aa3cc12b        jaegertracing/all-in-one:latest                        "/go/bin/all-in-one-…"   6 seconds ago       Up 5 seconds             0.0.0.0:5775->5775/udp, 0.0.0.0:5778->5778/tcp, 0.0.0.0:9411->9411/tcp, 0.0.0.0:14268->14268/tcp, 0.0.0.0:6831-6832->6831-6832/udp, 0.0.0.0:16686->16686/tcp, 14250/tcp   jaeger

# quick test udp port
[idps@test-host ~]$ nc -z -v -u test-host 5775
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 172.30.10.12:5775.
Ncat: UDP packet sent successfully
Ncat: 1 bytes sent, 0 bytes received in 2.01 seconds.
```
go to jaeger web gui (Port 16686)

Write OpenTracing Python Example

create python virtual env
  • Local Env Preparation

     $ mkdir opentracing
     $ cd opentracing
     $ virtualenv venv
     $ cd venv
     $ venv source ./bin/activate
     $ (venv)  venv pip install jaeger-client
     $ pip install ipython
    
prepare a config.py and copy it for configuring the Tracer and trace sample code
  • local_agent, 'reporting_host': '<your jaeger server name>' and 'reporting_port': '<your jaeger port , default udp 5775>'

     $ mkdir demo
     $ cd demo
     # cat config.py content and copy it into memory
     $ (venv)  demo cat config.py
     import logging
     from jaeger_client import Config
    
     def init_tracer(service):
         logging.getLogger('').handlers = []
         logging.basicConfig(format='%(message)s', level=logging.DEBUG)
    
         config = Config(
             config={
                 'sampler': {
                     'type': 'const',
                     'param': 1,
                 },
                 'local_agent': {
                     'reporting_host': 'test-host',
                     'reporting_port': '5775',
                 },
                 'logging': True,
             },
             service_name=service,
         )
    
         # this call also sets opentracing.tracer
         return config.initialize_tracer()
    
Step 1: jump into ipython command line and use %paste command for init_tracer python module
```
$ (venv)  demo ipython
Python 3.9.0 (default, Oct 27 2020, 14:15:17)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: %paste
import logging
from jaeger_client import Config

def init_tracer(service):
    logging.getLogger('').handlers = []
    logging.basicConfig(format='%(message)s', level=logging.DEBUG)

    config = Config(
        config={
            'sampler': {
                'type': 'const',
                'param': 1,
            },
            'local_agent': {
                'reporting_host': 'test-host',
                'reporting_port': '5775',
            },
            'logging': True,
        },
        service_name=service,
    )

    # this call also sets opentracing.tracer
    return config.initialize_tracer()

## -- End pasted text --
```
Step 2: run init_tracer module for initializing Jaeger Tracer with UDP reporter
```
In [2]: tracer = init_tracer('first-service')
Initializing Jaeger Tracer with UDP reporter
Using selector: KqueueSelector
Using sampler ConstSampler(True)
opentracing.tracer initialized to <jaeger_client.tracer.Tracer object at 0x10cb455b0>[app_name=first-service]
```
Step 3: create 1st span
```   
In [3]: with tracer.start_span('first-span') as span:
...:     span.set_tag('first-tag', '100')
...:
Reporting span bc1f64a0ee94e229:fa3903683ae96d36:0:1 first-service.first-span
```
Step 4: start a span within another span, create 2nd span and also 3rd span
```   
In [4]: with tracer.start_span('second-span') as span2:
...:     span2.set_tag('second-tag', '90')
...:     with tracer.start_span('third-span') as span3:
...:         span3.set_tag('third-tag', '80')
...:
Reporting span af9361a04f79a819:2824469d19e96aa1:0:1 first-service.third-span
Reporting span a87effad357e5e1b:fc511d3bbfdd5c4a:0:1 first-service.second-span
```
Step 5: create 4th span and child 5th span
```
In [5]: with tracer.start_span('fourth-span') as span4:
...:     span4.set_tag('fourth-tag', '60')
...:     with tracer.start_span('fifth-span', child_of=span4) as span5:
...:         span5.set_tag('fifth-tag', '80')
...:
Reporting span 530d8df44a6e7bc8:40aef4569ff20919:2910c125043277f6:1 first-service.fifth-span
Reporting span 530d8df44a6e7bc8:2910c125043277f6:0:1 first-service.fourth-span
```

Tracing an HTTP Requests Example

Step 6: for a run the next example we need to install the python-requests module
```
In [6]: !pip install requests
Collecting requests
Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
    |████████████████████████████████| 61 kB 6.1 MB/s
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
    |████████████████████████████████| 137 kB 12.0 MB/s
Collecting chardet<5,>=3.0.2
Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
    |████████████████████████████████| 178 kB 14.3 MB/s
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Installing collected packages: urllib3, idna, chardet, certifi, requests
Successfully installed certifi-2020.12.5 chardet-4.0.0 idna-2.10 requests-2.25.1 urllib3-1.26.3
```
Step 7: import requests module
```
In [7]: import requests
```
Step 8: create a homepages list
```
In [8]: homepages = []
```
Step 9: use requests to query the Github jobs API for Python jobs, and then get each Company’s listed website.
```
In [9]: res = requests.get('https://jobs.github.com/positions.json?description=python')
Starting new HTTPS connection (1): jobs.github.com:443
https://jobs.github.com:443 "GET /positions.json?description=python HTTP/1.1" 200 None
```
Step 10: without trace span, just for loop and try to connect company website
```
In [10]: for result in res.json():
    ...:     print('Getting website for %s' % result['company'])
    ...:     try:
    ...:         homepages.append(requests.get(result['company_url']))
    ...:     except:
    ...:         print('Unable to get site for %s' % result['company'])
    ...:
Getting website for Huey Magoo's LLC
Starting new HTTP connection (1): www.hueymagoos.com:80
http://www.hueymagoos.com:80 "GET / HTTP/1.1" 301 0
Starting new HTTPS connection (1): hueymagoos.com:443
https://hueymagoos.com:443 "GET / HTTP/1.1" 200 37578
Getting website for Brave New Software Project, Inc.
Unable to get site for Brave New Software Project, Inc.
Getting website for Teleport
Starting new HTTPS connection (1): goteleport.com:443
https://goteleport.com:443 "GET / HTTP/1.1" 200 None
Getting website for Finity
Starting new HTTP connection (1): finity.com:80
http://finity.com:80 "GET / HTTP/1.1" 301 231
Starting new HTTPS connection (1): www.finity.com:443
Unable to get site for Finity
Getting website for Saildrone
Unable to get site for Saildrone
Getting website for Saildrone
Unable to get site for Saildrone
Getting website for Frogslayer
Starting new HTTPS connection (1): frogslayer.com:443
https://frogslayer.com:443 "GET / HTTP/1.1" 200 None
Getting website for Yubico
Starting new HTTPS connection (1): www.yubico.com:443
https://www.yubico.com:443 "GET / HTTP/1.1" 200 184430
Getting website for Playco
Starting new HTTP connection (1): www.play.co:80
http://www.play.co:80 "GET / HTTP/1.1" 301 166
Starting new HTTPS connection (1): www.play.co:443
https://www.play.co:443 "GET / HTTP/1.1" 200 6201
Getting website for ACCURE Battery Intelligence GmbH
Starting new HTTPS connection (1): www.accure.net:443
https://www.accure.net:443 "GET / HTTP/1.1" 200 6995
Getting website for Commonwealth Bank
Starting new HTTPS connection (1): www.commbank.com.au:443
https://www.commbank.com.au:443 "GET /about-us/careers/engineering.html HTTP/1.1" 200 11130
Getting website for TeleClinic GmbH
Unable to get site for TeleClinic GmbH
Getting website for Explect
Starting new HTTPS connection (1): explect.nl:443
https://explect.nl:443 "GET / HTTP/1.1" 200 47755
Getting website for TeleClinic GmbH
Unable to get site for TeleClinic GmbH
Getting website for Countfire
Starting new HTTP connection (1): www.countfire.com:80
http://www.countfire.com:80 "GET / HTTP/1.1" 301 185
Starting new HTTPS connection (1): www.countfire.com:443
https://www.countfire.com:443 "GET / HTTP/1.1" 200 None
Getting website for Shell Business operations, Chennai
Starting new HTTPS connection (1): jobs.shell.com:443
https://jobs.shell.com:443 "GET / HTTP/1.1" 302 12354
https://jobs.shell.com:443 "GET /search-jobs HTTP/1.1" 200 25498
Getting website for Clark Germany GmbH
Starting new HTTPS connection (1): www.clark.de:443
https://www.clark.de:443 "GET / HTTP/1.1" 200 None
Getting website for Boston Red Sox
Unable to get site for Boston Red Sox
Getting website for Axios
Starting new HTTP connection (1): axios.com:80
http://axios.com:80 "GET / HTTP/1.1" 301 183
Starting new HTTPS connection (1): axios.com:443
https://axios.com:443 "GET / HTTP/1.1" 301 0
Starting new HTTPS connection (1): www.axios.com:443
https://www.axios.com:443 "GET / HTTP/1.1" 200 None
Getting website for Cornell University - Breeding Insight
Unable to get site for Cornell University - Breeding Insight
Getting website for Media Predict Inc.
Unable to get site for Media Predict Inc.
Getting website for McKinsey & Company
Unable to get site for McKinsey & Company
Getting website for McKinsey & Company
Unable to get site for McKinsey & Company
Getting website for Empower Associates
Starting new HTTPS connection (1): www.empower.associates:443
https://www.empower.associates:443 "GET / HTTP/1.1" 200 11837
Getting website for Build Change
Starting new HTTPS connection (1): buildchange.org:443
https://buildchange.org:443 "GET / HTTP/1.1" 200 None
Getting website for Lowers Risk Group LLC
Starting new HTTP connection (1): http:80
Unable to get site for Lowers Risk Group LLC
Getting website for Solv
Starting new HTTP connection (1): www.solvhealth.com:80
http://www.solvhealth.com:80 "GET / HTTP/1.1" 301 183
Starting new HTTPS connection (1): www.solvhealth.com:443
https://www.solvhealth.com:443 "GET / HTTP/1.1" 200 None
Getting website for SteepRock Inc.
Starting new HTTPS connection (1): www.steeprockinc.com:443
https://www.steeprockinc.com:443 "GET / HTTP/1.1" 200 6318
Getting website for clicks digital GmbH
Starting new HTTPS connection (1): clicks.digital:443
https://clicks.digital:443 "GET /karriere HTTP/1.1" 200 19375
Getting website for Pangaea Data
Starting new HTTPS connection (1): www.pangaeadata.ai:443
https://www.pangaeadata.ai:443 "GET / HTTP/1.1" 200 11459
Getting website for X24 Factory GmbH
Starting new HTTPS connection (1): x24-factory-gmbh.jobs.personio.de:443
https://x24-factory-gmbh.jobs.personio.de:443 "GET /job/318719 HTTP/1.1" 200 None
Getting website for Metas Solutions
Unable to get site for Metas Solutions
Getting website for brightwheel
Starting new HTTPS connection (1): mybrightwheel.com:443
https://mybrightwheel.com:443 "GET /careers/ HTTP/1.1" 403 146
Getting website for Cyphercor Inc.
Unable to get site for Cyphercor Inc.
Getting website for strongDM
Starting new HTTPS connection (1): jobs.lever.co:443
https://jobs.lever.co:443 "GET /strongdm HTTP/1.1" 200 None
Getting website for smartmicro
Starting new HTTPS connection (1): www.smartmicro.com:443
https://www.smartmicro.com:443 "GET / HTTP/1.1" 200 None
Getting website for LORENZ Life Sciences Group
Starting new HTTPS connection (1): www.lorenz.cc:443
https://www.lorenz.cc:443 "GET / HTTP/1.1" 200 None
Getting website for VIVIO Health
Starting new HTTPS connection (1): vonq.io:443
https://vonq.io:443 "GET /3uocF3l HTTP/1.1" 301 650
Starting new HTTPS connection (1): sp.vonq-collector.com:443
https://sp.vonq-collector.com:443 "GET /vq/r?u=https://jobs.smartrecruiters.com/VIVIOHealth/743999735729760-tech-lead-remote-us-%3Fvq_campaign%3Dc0889ab7-55b4-5095-bfd3-2ec219d0ee84%26vq_source%3D819&e=ue&ue_px=eyJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsICJkYXRhIjogeyJzY2hlbWEiOiAiaWdsdTpjb20udm9ucS9jdGEvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOiB7ImNhbXBhaWduIjoiYzA4ODlhYjctNTViNC01MDk1LWJmZDMtMmVjMjE5ZDBlZTg0Iiwic291cmNlIjoiODE5IiwiZW50aXR5IjoiYzM4ZWU2YWYtZTZhMi00ZDI1LWIyZDctNjg4YWEzNGQxYWI2In0gfX0%3D HTTP/1.1" 302 0
Starting new HTTPS connection (1): jobs.smartrecruiters.com:443
https://jobs.smartrecruiters.com:443 "GET /VIVIOHealth/743999735729760-tech-lead-remote-us-?vq_campaign=c0889ab7-55b4-5095-bfd3-2ec219d0ee84&vq_source=819 HTTP/1.1" 200 None
Getting website for BITGRIP GmbH
Starting new HTTPS connection (1): www.bitgrip.de:443
https://www.bitgrip.de:443 "GET / HTTP/1.1" 200 None
Getting website for Premiere Digital
Starting new HTTP connection (1): premieredigital.net:80
http://premieredigital.net:80 "GET / HTTP/1.1" 403 146
Getting website for Evernest GmbH
Unable to get site for Evernest GmbH
Getting website for Evernest GmbH
Unable to get site for Evernest GmbH
Getting website for enote GmbH
Starting new HTTPS connection (1): www.enote.com:443
https://www.enote.com:443 "GET / HTTP/1.1" 200 42092
Getting website for InnoGames GmbH
Starting new HTTPS connection (1): www.innogames.com:443
https://www.innogames.com:443 "GET / HTTP/1.1" 200 10886
Getting website for CompuGroup Medical SE & Co. KGaA
Starting new HTTPS connection (1): vonq.io:443
https://vonq.io:443 "GET /3prfoVJ HTTP/1.1" 301 736
Starting new HTTPS connection (1): sp.vonq-collector.com:443
https://sp.vonq-collector.com:443 "GET /vq/r?u=https://cgm-success.csod.com/ux/ats/careersite/5/requisition/5675/application%3Fc%3Dcgm-success%26c%3Dcgm-success%26jobboardid%3D0%26jobboardid%3D01%26source%3Dgit%26vq_campaign%3Dbf0763cf-27e1-53d2-b785-c5ff5844f78f%26vq_source%3D819%231&e=ue&ue_px=eyJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsICJkYXRhIjogeyJzY2hlbWEiOiAiaWdsdTpjb20udm9ucS9jdGEvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOiB7ImNhbXBhaWduIjoiYmYwNzYzY2YtMjdlMS01M2QyLWI3ODUtYzVmZjU4NDRmNzhmIiwic291cmNlIjoiODE5IiwiZW50aXR5IjoiNTk3ODJmYzItMTgyOC00MGU5LWEyN2UtNTJjMzMwOGYxN2ZmIn0gfX0%3D HTTP/1.1" 302 0
Starting new HTTPS connection (1): cgm-success.csod.com:443
https://cgm-success.csod.com:443 "GET /ux/ats/careersite/5/requisition/5675/application?c=cgm-success&c=cgm-success&jobboardid=0&jobboardid=01&source=git&vq_campaign=bf0763cf-27e1-53d2-b785-c5ff5844f78f&vq_source=819 HTTP/1.1" 200 1617
Getting website for 7digital Limited
Unable to get site for 7digital Limited
Getting website for Humanoo
Starting new HTTP connection (1): humanoo.com:80
http://humanoo.com:80 "GET / HTTP/1.1" 301 162
Starting new HTTPS connection (1): www.humanoo.com:443
https://www.humanoo.com:443 "GET / HTTP/1.1" 200 None
Getting website for fibrisTerre Systems GmbH
Starting new HTTP connection (1): www.fibristerre.de:80
http://www.fibristerre.de:80 "GET / HTTP/1.1" 302 211
Starting new HTTPS connection (1): www.fibristerre.de:443
https://www.fibristerre.de:443 "GET / HTTP/1.1" 200 None
Getting website for Nav Inc.
Unable to get site for Nav Inc.
Getting website for Agiloft Inc
Unable to get site for Agiloft Inc
```
Step 11: wrap both of these requests in spans, and make each of the child websites a child span of our main request:
```
In [11]: with tracer.start_span('get-python-jobs') as span:
    ...:     homepages = []
    ...:     res = requests.get('https://jobs.github.com/positions.json?description=python')
    ...:     span.set_tag('jobs-count', len(res.json()))
    ...:     for result in res.json():
    ...:         with tracer.start_span(result['company'], child_of=span) as site_span:
    ...:             print('Getting website for %s' % result['company'])
    ...:             try:
    ...:                 homepages.append(requests.get(result['company_url']))
    ...:             except:
    ...:                 print('Unable to get site for %s' % result['company'])
    ...:
Starting new HTTPS connection (1): jobs.github.com:443
https://jobs.github.com:443 "GET /positions.json?description=python HTTP/1.1" 200 None
Getting website for Huey Magoo's LLC
Starting new HTTP connection (1): www.hueymagoos.com:80
http://www.hueymagoos.com:80 "GET / HTTP/1.1" 301 0
Starting new HTTPS connection (1): hueymagoos.com:443
https://hueymagoos.com:443 "GET / HTTP/1.1" 200 37578
Reporting span e18207fdeb663a87:8ab706093479e628:84141cf69ffbc34:1 first-service.Huey Magoo's LLC
Getting website for Brave New Software Project, Inc.
Unable to get site for Brave New Software Project, Inc.
Reporting span e18207fdeb663a87:fd5b5849b1878fa2:84141cf69ffbc34:1 first-service.Brave New Software Project, Inc.
Getting website for Teleport
Starting new HTTPS connection (1): goteleport.com:443
https://goteleport.com:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:5599b2079e7607d6:84141cf69ffbc34:1 first-service.Teleport
Getting website for Finity
Starting new HTTP connection (1): finity.com:80
http://finity.com:80 "GET / HTTP/1.1" 301 231
Starting new HTTPS connection (1): www.finity.com:443
Unable to get site for Finity
Reporting span e18207fdeb663a87:81d25bfb7157d6ec:84141cf69ffbc34:1 first-service.Finity
Getting website for Saildrone
Unable to get site for Saildrone
Reporting span e18207fdeb663a87:324855c207007909:84141cf69ffbc34:1 first-service.Saildrone
Getting website for Saildrone
Unable to get site for Saildrone
Reporting span e18207fdeb663a87:a136b3d290cb8be0:84141cf69ffbc34:1 first-service.Saildrone
Getting website for Frogslayer
Starting new HTTPS connection (1): frogslayer.com:443
https://frogslayer.com:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:3fee2193ff21fcb4:84141cf69ffbc34:1 first-service.Frogslayer
Getting website for Yubico
Starting new HTTPS connection (1): www.yubico.com:443
https://www.yubico.com:443 "GET / HTTP/1.1" 200 184430
Reporting span e18207fdeb663a87:1f5750fcca3707b8:84141cf69ffbc34:1 first-service.Yubico
Getting website for Playco
Starting new HTTP connection (1): www.play.co:80
http://www.play.co:80 "GET / HTTP/1.1" 301 166
Starting new HTTPS connection (1): www.play.co:443
https://www.play.co:443 "GET / HTTP/1.1" 200 6201
Reporting span e18207fdeb663a87:fc8b3e0503a83be3:84141cf69ffbc34:1 first-service.Playco
Getting website for ACCURE Battery Intelligence GmbH
Starting new HTTPS connection (1): www.accure.net:443
https://www.accure.net:443 "GET / HTTP/1.1" 200 6995
Reporting span e18207fdeb663a87:920295bbe4541380:84141cf69ffbc34:1 first-service.ACCURE Battery Intelligence GmbH
Getting website for Commonwealth Bank
Starting new HTTPS connection (1): www.commbank.com.au:443
https://www.commbank.com.au:443 "GET /about-us/careers/engineering.html HTTP/1.1" 200 11130
Reporting span e18207fdeb663a87:a188ebc608602fb9:84141cf69ffbc34:1 first-service.Commonwealth Bank
Getting website for TeleClinic GmbH
Unable to get site for TeleClinic GmbH
Reporting span e18207fdeb663a87:12f769719c991b05:84141cf69ffbc34:1 first-service.TeleClinic GmbH
Getting website for Explect
Starting new HTTPS connection (1): explect.nl:443
https://explect.nl:443 "GET / HTTP/1.1" 200 47755
Reporting span e18207fdeb663a87:e37dccefff36c06d:84141cf69ffbc34:1 first-service.Explect
Getting website for TeleClinic GmbH
Unable to get site for TeleClinic GmbH
Reporting span e18207fdeb663a87:ca518500dbb9436d:84141cf69ffbc34:1 first-service.TeleClinic GmbH
Getting website for Countfire
Starting new HTTP connection (1): www.countfire.com:80
http://www.countfire.com:80 "GET / HTTP/1.1" 301 185
Starting new HTTPS connection (1): www.countfire.com:443
https://www.countfire.com:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:153f3fec0c08b16a:84141cf69ffbc34:1 first-service.Countfire
Getting website for Shell Business operations, Chennai
Starting new HTTPS connection (1): jobs.shell.com:443
https://jobs.shell.com:443 "GET / HTTP/1.1" 302 12354
https://jobs.shell.com:443 "GET /search-jobs HTTP/1.1" 200 25498
Reporting span e18207fdeb663a87:374f2611c9ead517:84141cf69ffbc34:1 first-service.Shell Business operations, Chennai
Getting website for Clark Germany GmbH
Starting new HTTPS connection (1): www.clark.de:443
https://www.clark.de:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:2eb86ef875399434:84141cf69ffbc34:1 first-service.Clark Germany GmbH
Getting website for Boston Red Sox
Unable to get site for Boston Red Sox
Reporting span e18207fdeb663a87:86f5a95335a7ce4b:84141cf69ffbc34:1 first-service.Boston Red Sox
Getting website for Axios
Starting new HTTP connection (1): axios.com:80
http://axios.com:80 "GET / HTTP/1.1" 301 183
Starting new HTTPS connection (1): axios.com:443
https://axios.com:443 "GET / HTTP/1.1" 301 0
Starting new HTTPS connection (1): www.axios.com:443
https://www.axios.com:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:e6bf65384721cd63:84141cf69ffbc34:1 first-service.Axios
Getting website for Cornell University - Breeding Insight
Unable to get site for Cornell University - Breeding Insight
Reporting span e18207fdeb663a87:caf1eef4aa37410d:84141cf69ffbc34:1 first-service.Cornell University - Breeding Insight
Getting website for Media Predict Inc.
Unable to get site for Media Predict Inc.
Reporting span e18207fdeb663a87:b6cd04fa28538e75:84141cf69ffbc34:1 first-service.Media Predict Inc.
Getting website for McKinsey & Company
Unable to get site for McKinsey & Company
Reporting span e18207fdeb663a87:6ecb8f24ba8ae17a:84141cf69ffbc34:1 first-service.McKinsey & Company
Getting website for McKinsey & Company
Unable to get site for McKinsey & Company
Reporting span e18207fdeb663a87:f3531cfefb4b46f2:84141cf69ffbc34:1 first-service.McKinsey & Company
Getting website for Empower Associates
Starting new HTTPS connection (1): www.empower.associates:443
https://www.empower.associates:443 "GET / HTTP/1.1" 200 11837
Reporting span e18207fdeb663a87:eee5d522a774d37c:84141cf69ffbc34:1 first-service.Empower Associates
Getting website for Build Change
Starting new HTTPS connection (1): buildchange.org:443
https://buildchange.org:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:a30f8b52b43d12d3:84141cf69ffbc34:1 first-service.Build Change
Getting website for Lowers Risk Group LLC
Starting new HTTP connection (1): http:80
Unable to get site for Lowers Risk Group LLC
Reporting span e18207fdeb663a87:c6b3a23975158b9d:84141cf69ffbc34:1 first-service.Lowers Risk Group LLC
Getting website for Solv
Starting new HTTP connection (1): www.solvhealth.com:80
http://www.solvhealth.com:80 "GET / HTTP/1.1" 301 183
Starting new HTTPS connection (1): www.solvhealth.com:443
https://www.solvhealth.com:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:3451a6e76b75d1a7:84141cf69ffbc34:1 first-service.Solv
Getting website for SteepRock Inc.
Starting new HTTPS connection (1): www.steeprockinc.com:443
https://www.steeprockinc.com:443 "GET / HTTP/1.1" 200 6318
Reporting span e18207fdeb663a87:d4736de810cbf857:84141cf69ffbc34:1 first-service.SteepRock Inc.
Getting website for clicks digital GmbH
Starting new HTTPS connection (1): clicks.digital:443
https://clicks.digital:443 "GET /karriere HTTP/1.1" 200 19375
Reporting span e18207fdeb663a87:fe75c12b136c0cee:84141cf69ffbc34:1 first-service.clicks digital GmbH
Getting website for Pangaea Data
Starting new HTTPS connection (1): www.pangaeadata.ai:443
https://www.pangaeadata.ai:443 "GET / HTTP/1.1" 200 11459
Reporting span e18207fdeb663a87:d52ba81bd7af7ae0:84141cf69ffbc34:1 first-service.Pangaea Data
Getting website for X24 Factory GmbH
Starting new HTTPS connection (1): x24-factory-gmbh.jobs.personio.de:443
https://x24-factory-gmbh.jobs.personio.de:443 "GET /job/318719 HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:5a29100da1480fd4:84141cf69ffbc34:1 first-service.X24 Factory GmbH
Getting website for Metas Solutions
Unable to get site for Metas Solutions
Reporting span e18207fdeb663a87:df1657bc4b977384:84141cf69ffbc34:1 first-service.Metas Solutions
Getting website for brightwheel
Starting new HTTPS connection (1): mybrightwheel.com:443
https://mybrightwheel.com:443 "GET /careers/ HTTP/1.1" 403 146
Reporting span e18207fdeb663a87:fa92025b8cab83a5:84141cf69ffbc34:1 first-service.brightwheel
Getting website for Cyphercor Inc.
Unable to get site for Cyphercor Inc.
Reporting span e18207fdeb663a87:e11524a60f009ba5:84141cf69ffbc34:1 first-service.Cyphercor Inc.
Getting website for strongDM
Starting new HTTPS connection (1): jobs.lever.co:443
https://jobs.lever.co:443 "GET /strongdm HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:c440ae7fa9a01354:84141cf69ffbc34:1 first-service.strongDM
Getting website for smartmicro
Starting new HTTPS connection (1): www.smartmicro.com:443
https://www.smartmicro.com:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:52d5f69ba7aeb065:84141cf69ffbc34:1 first-service.smartmicro
Getting website for LORENZ Life Sciences Group
Starting new HTTPS connection (1): www.lorenz.cc:443
https://www.lorenz.cc:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:2048d0e215b004d8:84141cf69ffbc34:1 first-service.LORENZ Life Sciences Group
Getting website for VIVIO Health
Starting new HTTPS connection (1): vonq.io:443
https://vonq.io:443 "GET /3uocF3l HTTP/1.1" 301 650
Starting new HTTPS connection (1): sp.vonq-collector.com:443
https://sp.vonq-collector.com:443 "GET /vq/r?u=https://jobs.smartrecruiters.com/VIVIOHealth/743999735729760-tech-lead-remote-us-%3Fvq_campaign%3Dc0889ab7-55b4-5095-bfd3-2ec219d0ee84%26vq_source%3D819&e=ue&ue_px=eyJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsICJkYXRhIjogeyJzY2hlbWEiOiAiaWdsdTpjb20udm9ucS9jdGEvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOiB7ImNhbXBhaWduIjoiYzA4ODlhYjctNTViNC01MDk1LWJmZDMtMmVjMjE5ZDBlZTg0Iiwic291cmNlIjoiODE5IiwiZW50aXR5IjoiYzM4ZWU2YWYtZTZhMi00ZDI1LWIyZDctNjg4YWEzNGQxYWI2In0gfX0%3D HTTP/1.1" 302 0
Starting new HTTPS connection (1): jobs.smartrecruiters.com:443
https://jobs.smartrecruiters.com:443 "GET /VIVIOHealth/743999735729760-tech-lead-remote-us-?vq_campaign=c0889ab7-55b4-5095-bfd3-2ec219d0ee84&vq_source=819 HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:e230240a435f6f03:84141cf69ffbc34:1 first-service.VIVIO Health
Getting website for BITGRIP GmbH
Starting new HTTPS connection (1): www.bitgrip.de:443
https://www.bitgrip.de:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:1f87bdf85142f17c:84141cf69ffbc34:1 first-service.BITGRIP GmbH
Getting website for Premiere Digital
Starting new HTTP connection (1): premieredigital.net:80
http://premieredigital.net:80 "GET / HTTP/1.1" 403 146
Reporting span e18207fdeb663a87:6aea9b6da233f7f4:84141cf69ffbc34:1 first-service.Premiere Digital
Getting website for Evernest GmbH
Unable to get site for Evernest GmbH
Reporting span e18207fdeb663a87:6103a954480e7ddb:84141cf69ffbc34:1 first-service.Evernest GmbH
Getting website for Evernest GmbH
Unable to get site for Evernest GmbH
Reporting span e18207fdeb663a87:705a2a5e260398a2:84141cf69ffbc34:1 first-service.Evernest GmbH
Getting website for enote GmbH
Starting new HTTPS connection (1): www.enote.com:443
https://www.enote.com:443 "GET / HTTP/1.1" 200 42092
Reporting span e18207fdeb663a87:cb05e25e5a961411:84141cf69ffbc34:1 first-service.enote GmbH
Getting website for InnoGames GmbH
Starting new HTTPS connection (1): www.innogames.com:443
https://www.innogames.com:443 "GET / HTTP/1.1" 200 10886
Reporting span e18207fdeb663a87:f15ce49ea8ac7b6c:84141cf69ffbc34:1 first-service.InnoGames GmbH
Getting website for CompuGroup Medical SE & Co. KGaA
Starting new HTTPS connection (1): vonq.io:443
https://vonq.io:443 "GET /3prfoVJ HTTP/1.1" 301 736
Starting new HTTPS connection (1): sp.vonq-collector.com:443
https://sp.vonq-collector.com:443 "GET /vq/r?u=https://cgm-success.csod.com/ux/ats/careersite/5/requisition/5675/application%3Fc%3Dcgm-success%26c%3Dcgm-success%26jobboardid%3D0%26jobboardid%3D01%26source%3Dgit%26vq_campaign%3Dbf0763cf-27e1-53d2-b785-c5ff5844f78f%26vq_source%3D819%231&e=ue&ue_px=eyJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsICJkYXRhIjogeyJzY2hlbWEiOiAiaWdsdTpjb20udm9ucS9jdGEvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOiB7ImNhbXBhaWduIjoiYmYwNzYzY2YtMjdlMS01M2QyLWI3ODUtYzVmZjU4NDRmNzhmIiwic291cmNlIjoiODE5IiwiZW50aXR5IjoiNTk3ODJmYzItMTgyOC00MGU5LWEyN2UtNTJjMzMwOGYxN2ZmIn0gfX0%3D HTTP/1.1" 302 0
Starting new HTTPS connection (1): cgm-success.csod.com:443
https://cgm-success.csod.com:443 "GET /ux/ats/careersite/5/requisition/5675/application?c=cgm-success&c=cgm-success&jobboardid=0&jobboardid=01&source=git&vq_campaign=bf0763cf-27e1-53d2-b785-c5ff5844f78f&vq_source=819 HTTP/1.1" 200 1616
Reporting span e18207fdeb663a87:b66a604703ce8260:84141cf69ffbc34:1 first-service.CompuGroup Medical SE & Co. KGaA
Getting website for 7digital Limited
Unable to get site for 7digital Limited
Reporting span e18207fdeb663a87:67d2af2ee247ed1c:84141cf69ffbc34:1 first-service.7digital Limited
Getting website for Humanoo
Starting new HTTP connection (1): humanoo.com:80
http://humanoo.com:80 "GET / HTTP/1.1" 301 162
Starting new HTTPS connection (1): www.humanoo.com:443
https://www.humanoo.com:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:2635f7bd18185aee:84141cf69ffbc34:1 first-service.Humanoo
Getting website for fibrisTerre Systems GmbH
Starting new HTTP connection (1): www.fibristerre.de:80
http://www.fibristerre.de:80 "GET / HTTP/1.1" 302 211
Starting new HTTPS connection (1): www.fibristerre.de:443
https://www.fibristerre.de:443 "GET / HTTP/1.1" 200 None
Reporting span e18207fdeb663a87:b05f7722fe916baf:84141cf69ffbc34:1 first-service.fibrisTerre Systems GmbH
Getting website for Nav Inc.
Unable to get site for Nav Inc.
Reporting span e18207fdeb663a87:93862d825e0a8f33:84141cf69ffbc34:1 first-service.Nav Inc.
Getting website for Agiloft Inc
Unable to get site for Agiloft Inc
Reporting span e18207fdeb663a87:c79456cec4cbdbf9:84141cf69ffbc34:1 first-service.Agiloft Inc
Reporting span e18207fdeb663a87:84141cf69ffbc34:0:1 first-service.get-python-jobs
```
Step 12: In order to see the failures, we’ll need to add tagging to our traces. Once they’re properly tagged, we’ll be able to see them in Jaeger.
```
In [12]: with tracer.start_span('get-python-jobs') as span:
    ...:     homepages = []
    ...:     res = requests.get('https://jobs.github.com/positions.json?description=python')
    ...:     span.set_tag('jobs-count', len(res.json()))
    ...:     for result in res.json():
    ...:         with tracer.start_span(result['company'], child_of=span) as site_span:
    ...:             print('Getting website for %s' % result['company'])
    ...:             try:
    ...:                 homepages.append(requests.get(result['company_url']))
    ...:                 site_span.set_tag('request-type', 'Success')
    ...:             except:
    ...:                 print('Unable to get site for %s' % result['company'])
    ...:                 site_span.set_tag('request-type', 'Failure')
    ...:
Starting new HTTPS connection (1): jobs.github.com:443
https://jobs.github.com:443 "GET /positions.json?description=python HTTP/1.1" 200 None
Getting website for Huey Magoo's LLC
Starting new HTTP connection (1): www.hueymagoos.com:80
http://www.hueymagoos.com:80 "GET / HTTP/1.1" 301 0
Starting new HTTPS connection (1): hueymagoos.com:443
https://hueymagoos.com:443 "GET / HTTP/1.1" 200 37578
Reporting span 92deb8546e42cf9b:7c741ba9f533332d:1b131a37563e495c:1 first-service.Huey Magoo's LLC
Getting website for Brave New Software Project, Inc.
Unable to get site for Brave New Software Project, Inc.
Reporting span 92deb8546e42cf9b:4089e54f0227e051:1b131a37563e495c:1 first-service.Brave New Software Project, Inc.
Getting website for Teleport
Starting new HTTPS connection (1): goteleport.com:443
https://goteleport.com:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:602dc05500bc83b9:1b131a37563e495c:1 first-service.Teleport
Getting website for Finity
Starting new HTTP connection (1): finity.com:80
http://finity.com:80 "GET / HTTP/1.1" 301 231
Starting new HTTPS connection (1): www.finity.com:443
Unable to get site for Finity
Reporting span 92deb8546e42cf9b:faab70f0d557dd9f:1b131a37563e495c:1 first-service.Finity
Getting website for Saildrone
Unable to get site for Saildrone
Reporting span 92deb8546e42cf9b:d23f4ea89fd95837:1b131a37563e495c:1 first-service.Saildrone
Getting website for Saildrone
Unable to get site for Saildrone
Reporting span 92deb8546e42cf9b:c2396b55e1713a39:1b131a37563e495c:1 first-service.Saildrone
Getting website for Frogslayer
Starting new HTTPS connection (1): frogslayer.com:443
https://frogslayer.com:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:6e32e27c64faa50d:1b131a37563e495c:1 first-service.Frogslayer
Getting website for Yubico
Starting new HTTPS connection (1): www.yubico.com:443
https://www.yubico.com:443 "GET / HTTP/1.1" 200 184430
Reporting span 92deb8546e42cf9b:4c49dcc8c8cae2ef:1b131a37563e495c:1 first-service.Yubico
Getting website for Playco
Starting new HTTP connection (1): www.play.co:80
http://www.play.co:80 "GET / HTTP/1.1" 301 166
Starting new HTTPS connection (1): www.play.co:443
https://www.play.co:443 "GET / HTTP/1.1" 200 6201
Reporting span 92deb8546e42cf9b:8fb708d7c565d01f:1b131a37563e495c:1 first-service.Playco
Getting website for ACCURE Battery Intelligence GmbH
Starting new HTTPS connection (1): www.accure.net:443
https://www.accure.net:443 "GET / HTTP/1.1" 200 6995
Reporting span 92deb8546e42cf9b:a1ce23006d0db2bb:1b131a37563e495c:1 first-service.ACCURE Battery Intelligence GmbH
Getting website for Commonwealth Bank
Starting new HTTPS connection (1): www.commbank.com.au:443
https://www.commbank.com.au:443 "GET /about-us/careers/engineering.html HTTP/1.1" 200 11130
Reporting span 92deb8546e42cf9b:4af7250acfda2ccd:1b131a37563e495c:1 first-service.Commonwealth Bank
Getting website for TeleClinic GmbH
Unable to get site for TeleClinic GmbH
Reporting span 92deb8546e42cf9b:3067650536352497:1b131a37563e495c:1 first-service.TeleClinic GmbH
Getting website for Explect
Starting new HTTPS connection (1): explect.nl:443
https://explect.nl:443 "GET / HTTP/1.1" 200 47755
Reporting span 92deb8546e42cf9b:ed0c142d7fc661dd:1b131a37563e495c:1 first-service.Explect
Getting website for TeleClinic GmbH
Unable to get site for TeleClinic GmbH
Reporting span 92deb8546e42cf9b:857d566276bd5912:1b131a37563e495c:1 first-service.TeleClinic GmbH
Getting website for Countfire
Starting new HTTP connection (1): www.countfire.com:80
http://www.countfire.com:80 "GET / HTTP/1.1" 301 185
Starting new HTTPS connection (1): www.countfire.com:443
https://www.countfire.com:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:b90a4f592cadbbc0:1b131a37563e495c:1 first-service.Countfire
Getting website for Shell Business operations, Chennai
Starting new HTTPS connection (1): jobs.shell.com:443
https://jobs.shell.com:443 "GET / HTTP/1.1" 302 12354
https://jobs.shell.com:443 "GET /search-jobs HTTP/1.1" 200 25498
Reporting span 92deb8546e42cf9b:7631a5cefa66563b:1b131a37563e495c:1 first-service.Shell Business operations, Chennai
Getting website for Clark Germany GmbH
Starting new HTTPS connection (1): www.clark.de:443
https://www.clark.de:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:f93293130c4b63c3:1b131a37563e495c:1 first-service.Clark Germany GmbH
Getting website for Boston Red Sox
Unable to get site for Boston Red Sox
Reporting span 92deb8546e42cf9b:31e5fb10830ceb2c:1b131a37563e495c:1 first-service.Boston Red Sox
Getting website for Axios
Starting new HTTP connection (1): axios.com:80
http://axios.com:80 "GET / HTTP/1.1" 301 183
Starting new HTTPS connection (1): axios.com:443
https://axios.com:443 "GET / HTTP/1.1" 301 0
Starting new HTTPS connection (1): www.axios.com:443
https://www.axios.com:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:4fa5871ccd610d0:1b131a37563e495c:1 first-service.Axios
Getting website for Cornell University - Breeding Insight
Unable to get site for Cornell University - Breeding Insight
Reporting span 92deb8546e42cf9b:ea1cb398ea2d6947:1b131a37563e495c:1 first-service.Cornell University - Breeding Insight
Getting website for Media Predict Inc.
Unable to get site for Media Predict Inc.
Reporting span 92deb8546e42cf9b:b733a3947caf83b4:1b131a37563e495c:1 first-service.Media Predict Inc.
Getting website for McKinsey & Company
Unable to get site for McKinsey & Company
Reporting span 92deb8546e42cf9b:c187274ac6fb0718:1b131a37563e495c:1 first-service.McKinsey & Company
Getting website for McKinsey & Company
Unable to get site for McKinsey & Company
Reporting span 92deb8546e42cf9b:524e1f9dfebc7d58:1b131a37563e495c:1 first-service.McKinsey & Company
Getting website for Empower Associates
Starting new HTTPS connection (1): www.empower.associates:443
https://www.empower.associates:443 "GET / HTTP/1.1" 200 11837
Reporting span 92deb8546e42cf9b:9239da49761ff09e:1b131a37563e495c:1 first-service.Empower Associates
Getting website for Build Change
Starting new HTTPS connection (1): buildchange.org:443
https://buildchange.org:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:da76f959eb5ba93f:1b131a37563e495c:1 first-service.Build Change
Getting website for Lowers Risk Group LLC
Starting new HTTP connection (1): http:80
Unable to get site for Lowers Risk Group LLC
Reporting span 92deb8546e42cf9b:58788935244046b0:1b131a37563e495c:1 first-service.Lowers Risk Group LLC
Getting website for Solv
Starting new HTTP connection (1): www.solvhealth.com:80
http://www.solvhealth.com:80 "GET / HTTP/1.1" 301 183
Starting new HTTPS connection (1): www.solvhealth.com:443
https://www.solvhealth.com:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:3b72c0fb58c8b60:1b131a37563e495c:1 first-service.Solv
Getting website for SteepRock Inc.
Starting new HTTPS connection (1): www.steeprockinc.com:443
https://www.steeprockinc.com:443 "GET / HTTP/1.1" 200 6318
Reporting span 92deb8546e42cf9b:42e14b0663b07512:1b131a37563e495c:1 first-service.SteepRock Inc.
Getting website for clicks digital GmbH
Starting new HTTPS connection (1): clicks.digital:443
https://clicks.digital:443 "GET /karriere HTTP/1.1" 200 19375
Reporting span 92deb8546e42cf9b:4c0fb1b74d3427fe:1b131a37563e495c:1 first-service.clicks digital GmbH
Getting website for Pangaea Data
Starting new HTTPS connection (1): www.pangaeadata.ai:443
https://www.pangaeadata.ai:443 "GET / HTTP/1.1" 200 11459
Reporting span 92deb8546e42cf9b:3a1a0276f369138d:1b131a37563e495c:1 first-service.Pangaea Data
Getting website for X24 Factory GmbH
Starting new HTTPS connection (1): x24-factory-gmbh.jobs.personio.de:443
https://x24-factory-gmbh.jobs.personio.de:443 "GET /job/318719 HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:134540d363ea4f16:1b131a37563e495c:1 first-service.X24 Factory GmbH
Getting website for Metas Solutions
Unable to get site for Metas Solutions
Reporting span 92deb8546e42cf9b:b5da2be3cc47029e:1b131a37563e495c:1 first-service.Metas Solutions
Getting website for brightwheel
Starting new HTTPS connection (1): mybrightwheel.com:443
https://mybrightwheel.com:443 "GET /careers/ HTTP/1.1" 403 146
Reporting span 92deb8546e42cf9b:5e6badb6d85f3bc7:1b131a37563e495c:1 first-service.brightwheel
Getting website for Cyphercor Inc.
Unable to get site for Cyphercor Inc.
Reporting span 92deb8546e42cf9b:9b38bf03ed80daab:1b131a37563e495c:1 first-service.Cyphercor Inc.
Getting website for strongDM
Starting new HTTPS connection (1): jobs.lever.co:443
https://jobs.lever.co:443 "GET /strongdm HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:f94aa6b28615c918:1b131a37563e495c:1 first-service.strongDM
Getting website for smartmicro
Starting new HTTPS connection (1): www.smartmicro.com:443
https://www.smartmicro.com:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:3bd22ae5368aa937:1b131a37563e495c:1 first-service.smartmicro
Getting website for LORENZ Life Sciences Group
Starting new HTTPS connection (1): www.lorenz.cc:443
https://www.lorenz.cc:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:218d44082629dcf4:1b131a37563e495c:1 first-service.LORENZ Life Sciences Group
Getting website for VIVIO Health
Starting new HTTPS connection (1): vonq.io:443
https://vonq.io:443 "GET /3uocF3l HTTP/1.1" 301 650
Starting new HTTPS connection (1): sp.vonq-collector.com:443
https://sp.vonq-collector.com:443 "GET /vq/r?u=https://jobs.smartrecruiters.com/VIVIOHealth/743999735729760-tech-lead-remote-us-%3Fvq_campaign%3Dc0889ab7-55b4-5095-bfd3-2ec219d0ee84%26vq_source%3D819&e=ue&ue_px=eyJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsICJkYXRhIjogeyJzY2hlbWEiOiAiaWdsdTpjb20udm9ucS9jdGEvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOiB7ImNhbXBhaWduIjoiYzA4ODlhYjctNTViNC01MDk1LWJmZDMtMmVjMjE5ZDBlZTg0Iiwic291cmNlIjoiODE5IiwiZW50aXR5IjoiYzM4ZWU2YWYtZTZhMi00ZDI1LWIyZDctNjg4YWEzNGQxYWI2In0gfX0%3D HTTP/1.1" 302 0
Starting new HTTPS connection (1): jobs.smartrecruiters.com:443
https://jobs.smartrecruiters.com:443 "GET /VIVIOHealth/743999735729760-tech-lead-remote-us-?vq_campaign=c0889ab7-55b4-5095-bfd3-2ec219d0ee84&vq_source=819 HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:918893e35b418675:1b131a37563e495c:1 first-service.VIVIO Health
Getting website for BITGRIP GmbH
Starting new HTTPS connection (1): www.bitgrip.de:443
https://www.bitgrip.de:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:4ee423385914e9b8:1b131a37563e495c:1 first-service.BITGRIP GmbH
Getting website for Premiere Digital
Starting new HTTP connection (1): premieredigital.net:80
http://premieredigital.net:80 "GET / HTTP/1.1" 403 146
Reporting span 92deb8546e42cf9b:53e3c741ba06f8f6:1b131a37563e495c:1 first-service.Premiere Digital
Getting website for Evernest GmbH
Unable to get site for Evernest GmbH
Reporting span 92deb8546e42cf9b:229d9b29a14b457c:1b131a37563e495c:1 first-service.Evernest GmbH
Getting website for Evernest GmbH
Unable to get site for Evernest GmbH
Reporting span 92deb8546e42cf9b:3784bde2b4bb0b10:1b131a37563e495c:1 first-service.Evernest GmbH
Getting website for enote GmbH
Starting new HTTPS connection (1): www.enote.com:443
https://www.enote.com:443 "GET / HTTP/1.1" 200 42092
Reporting span 92deb8546e42cf9b:4a8e1f4d6a5490e5:1b131a37563e495c:1 first-service.enote GmbH
Getting website for InnoGames GmbH
Starting new HTTPS connection (1): www.innogames.com:443
https://www.innogames.com:443 "GET / HTTP/1.1" 200 10886
Reporting span 92deb8546e42cf9b:55f2cd77fdca8c78:1b131a37563e495c:1 first-service.InnoGames GmbH
Getting website for CompuGroup Medical SE & Co. KGaA
Starting new HTTPS connection (1): vonq.io:443
https://vonq.io:443 "GET /3prfoVJ HTTP/1.1" 301 736
Starting new HTTPS connection (1): sp.vonq-collector.com:443
https://sp.vonq-collector.com:443 "GET /vq/r?u=https://cgm-success.csod.com/ux/ats/careersite/5/requisition/5675/application%3Fc%3Dcgm-success%26c%3Dcgm-success%26jobboardid%3D0%26jobboardid%3D01%26source%3Dgit%26vq_campaign%3Dbf0763cf-27e1-53d2-b785-c5ff5844f78f%26vq_source%3D819%231&e=ue&ue_px=eyJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsICJkYXRhIjogeyJzY2hlbWEiOiAiaWdsdTpjb20udm9ucS9jdGEvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOiB7ImNhbXBhaWduIjoiYmYwNzYzY2YtMjdlMS01M2QyLWI3ODUtYzVmZjU4NDRmNzhmIiwic291cmNlIjoiODE5IiwiZW50aXR5IjoiNTk3ODJmYzItMTgyOC00MGU5LWEyN2UtNTJjMzMwOGYxN2ZmIn0gfX0%3D HTTP/1.1" 302 0
Starting new HTTPS connection (1): cgm-success.csod.com:443
https://cgm-success.csod.com:443 "GET /ux/ats/careersite/5/requisition/5675/application?c=cgm-success&c=cgm-success&jobboardid=0&jobboardid=01&source=git&vq_campaign=bf0763cf-27e1-53d2-b785-c5ff5844f78f&vq_source=819 HTTP/1.1" 200 1613
Reporting span 92deb8546e42cf9b:1ac7548a02577b69:1b131a37563e495c:1 first-service.CompuGroup Medical SE & Co. KGaA
Getting website for 7digital Limited
Unable to get site for 7digital Limited
Reporting span 92deb8546e42cf9b:f4505cb7cbfc75e4:1b131a37563e495c:1 first-service.7digital Limited
Getting website for Humanoo
Starting new HTTP connection (1): humanoo.com:80
http://humanoo.com:80 "GET / HTTP/1.1" 301 162
Starting new HTTPS connection (1): www.humanoo.com:443
https://www.humanoo.com:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:ea46dca07d9c64ff:1b131a37563e495c:1 first-service.Humanoo
Getting website for fibrisTerre Systems GmbH
Starting new HTTP connection (1): www.fibristerre.de:80
http://www.fibristerre.de:80 "GET / HTTP/1.1" 302 211
Starting new HTTPS connection (1): www.fibristerre.de:443
https://www.fibristerre.de:443 "GET / HTTP/1.1" 200 None
Reporting span 92deb8546e42cf9b:abfd8f8915f97ae2:1b131a37563e495c:1 first-service.fibrisTerre Systems GmbH
Getting website for Nav Inc.
Unable to get site for Nav Inc.
Reporting span 92deb8546e42cf9b:6275269c42b3763:1b131a37563e495c:1 first-service.Nav Inc.
Getting website for Agiloft Inc
Unable to get site for Agiloft Inc
Reporting span 92deb8546e42cf9b:94076ea9c3994aac:1b131a37563e495c:1 first-service.Agiloft Inc
Reporting span 92deb8546e42cf9b:1b131a37563e495c:0:1 first-service.get-python-jobs

In [13]:
```