Zero to hero service mesh

11 min readJan 6, 2023

This blog will help you understand service mesh and how to setup it up on Kubernetes.

Afraid of service mesh?

I was checking articles on the internet to set up service mesh from scratch but none of them worked well for me. So I thought of writing this zero to hero blog/tutorial which will contain all steps and procedures to set up service mesh.

What is Service mesh?

In Simple words service mesh help to create a network between multiple microservices regardless of where they are running or if they are running in different clusters from with different network. A Service Mesh is a configurable infrastructure layer that makes communication between microservice applications possible, structured, and observable. A service Mesh also monitors and manages network traffic between microservices, redirecting and granting or limiting access as needed to optimize and protect the system.

Common use cases of a service mesh

Set up high availability of the services in multiple clusters.
Canary deployment to available new releases for services to a specific set of users.
Split traffic based on percentage.

In this blog, we are going to set up a service mesh between two clusters using Istio.

Istio extends Kubernetes to establish a programmable, application-aware network using the powerful Envoy service proxy.

Prerequisites -

For this tutorial, we will need two Kubernetes clusters running version 1.22 or the latest version. Clusters need to be run in two different networks. For this POC we have created the clusters using kops in different VPCs.

You can create minimal Kubernetes clusters with kops. Follow the steps given below:

# kops installation 
curl -Lo kops https://github.com/kubernetes/kops/releases/download/v1.23.3/kops-darwin-amd64
chmod +x kops
sudo mv kops /usr/local/bin/kops1233
alias kops="kops-1233"

# To Use with SSO profile
aws sso login --profile=profile_name# Export required environment variables
export AWS_PROFILE=profile_name
export NAME=test2.yourcluster.com
export KOPS_STATE_STORE=s3://my-kops-s3# Generate RSA Key pair for cluster
ssh-keygen -t rsa -b 4096# Create KOPS cluster
kops create cluster $NAME \
    --state=$KOPS_STATE_STORE \
    --zones "us-east-1a" \
    --master-zones "us-east-1a" \
    --master-size t2.medium \
    --master-count 1 \
    --networking calico \
    --topology private \
    --node-count 1 \
    --node-size t2.medium \
    --kubernetes-version v1.22.0 \
    --ssh-public-key id_rsa.pubkops update cluster --name $NAME --yes --adminkops export kubecfg --admin=87600h0m0s --name $NAME --admin

2. Istio CLI version — 1.16.1 Download it from here.

We are going to use istioctl to install istio components in Kubernetes clusters.

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.16.1 TARGET_ARCH=x86_64 sh -
cd istio-1.16.1
export PATH=$PWD/bin:$PATH

Step 1: Setup mTLS

We are going to setup service mesh between to clusters running in two different networks so we need a secure way of communicating between services.

Mutual TLS (mTLS) secures communication between microservices in a service mesh. It uses cryptographically secure techniques to mutually authenticate individual microservices and encrypt the traffic between them

We have explained the process of creating CA Certs in short down below. create the following files.

ca.json

{
    "CN": "Istio Service Mesh Root CA",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
    {
      "C": "GB",
      "L": "London",
      "O": "Istio Service Mesh",
      "OU": "Istio Service Mesh Root CA",
      "ST": "England"
    }
   ]
  }

intermediate-ca.json

{
    "CN": "Intermidiate CA Istio",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C":  "GB",
        "L":  "London",
        "O":  "Intermidiate CA",
        "OU": "Intermidiate CA Istio",
        "ST": "England"
      }
    ],
    "ca": {
      "expiry": "42720h"
    }
  }

cfssl.json

{
    "signing": {
      "default": {
        "expiry": "8760h"
      },
      "profiles": {
        "intermediate_ca": {
          "usages": [
              "signing",
              "digital signature",
              "key encipherment",
              "cert sign",
              "crl sign",
              "server auth",
              "client auth"
          ],
          "expiry": "8760h",
          "ca_constraint": {
              "is_ca": true,
              "max_path_len": 0, 
              "max_path_len_zero": true
          }
        },
        "peer": {
          "usages": [
              "signing",
              "digital signature",
              "key encipherment", 
              "client auth",
              "server auth"
          ],
          "expiry": "8760h"
        },
        "server": {
          "usages": [
            "signing",
            "digital signing",
            "key encipherment",
            "server auth"
          ],
          "expiry": "8760h"
        },
        "client": {
          "usages": [
            "signing",
            "digital signature",
            "key encipherment", 
            "client auth"
          ],
          "expiry": "8760h"
        }
      }
    }
  }

Run the following commands in the directory where we have created the above files.

cfssl gencert -initca ca.json | cfssljson -bare ca
cfssl gencert -initca intermediate-ca.json | cfssljson -bare intermediate_ca
cfssl sign -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile intermediate_ca intermediate_ca.csr | cfssljson -bare intermediate_ca

With the first command, you are creating root certs(At the end you can rename the cert as root-cert.pem), with the second command, you’ll create intermediate certs and with the third command, you are signing intermediate certs with root certs(At the end you can rename the intermediate cert files as: ca-cert.pem and ca-key.pem).

After this, you will have the following files -

root-cert.pem
ca-cert.pem
ca-key.pem

Now, rename generated certificates and key files to match kubernetes secret format.

mv ca.pem root-cert.pem
mv ca-key.pem root-key.pem
mv intermediate_ca-key.pem ca-key.pem
mv intermediate_ca.pem ca-cert.pem
cp ca-cert.pem cert-chain.pem
cat root-cert.pem >> cert-chain.pem

To verify that your certs are correct, you can use the following commands:

openssl verify -CAfile cert-chain.pem ca-cert.pem
openssl verify -CAfile root-cert.pem ca-cert.pem

Both the commands should give you the output like:

ca-cert.pem: OK
ca-cert.pem: OK

Now, apply these certs in cluster1 and cluster2 using the command:

kubectl create secret generic cacerts -n istio-system \
--from-file=ca-cert.pem \
--from-file=ca-key.pem \
--from-file=root-cert.pem \
--from-file=cert-chain.pem

Step 2: Setup Istio in Cluster1

Now we are going to install istio in cluster1 So Service workloads across cluster boundaries communicate indirectly, via dedicated gateways for east-west traffic. The gateway in each cluster must be reachable from the other cluster.

Set contexts environment variables for cluster1 and cluster2

export CTX_CLUSTER1="cluster1-context"
export CTX_CLUSTER2="cluster2-context"

Set the default network for cluster1

If the istio-system namespace is already created, we need to set the cluster’s network there:

kubectl --context="${CTX_CLUSTER1}" get namespace istio-system && \
kubectl --context="${CTX_CLUSTER1}" label namespace istio-system topology.istio.io/network=network1

Configure cluster1 as a primary

Create the Istio configuration for cluster1:

cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
EOF

Apply the configuration to cluster1:

istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml

Install the east-west gateway in cluster1

Install a gateway in cluster1 that is dedicated to east-west traffic. By default, this gateway will be public on the Internet. Production systems may require additional access restrictions (e.g. via firewall rules) to prevent external attacks. Check with your cloud vendor to see what options are available.

samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster cluster1 --network network1 | \
    istioctl --context="${CTX_CLUSTER1}" install -y -f -

Wait for the east-west gateway to be assigned an external IP address:

kubectl --context="${CTX_CLUSTER1}" get svc istio-eastwestgateway -n istio-system

Expose services in cluster1

Since the clusters are on separate networks, we need to expose all services (*.local) on the east-west gateway in both clusters. While this gateway is public on the Internet, services behind it can only be accessed by services with a trusted mTLS certificate and workload ID, just as if they were on the same network.

kubectl --context="${CTX_CLUSTER1}" apply -n istio-system -f \
    samples/multicluster/expose-services.yaml

Step 3: Setup Istio in Cluster2

Set the default network for cluster2

If the istio-system namespace is already created, we need to set the cluster’s network there:

kubectl --context="${CTX_CLUSTER2}" get namespace istio-system && \
  kubectl --context="${CTX_CLUSTER2}" label namespace istio-system topology.istio.io/network=network2

Configure cluster2 as a primary

Create the Istio configuration for cluster2:

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
EOF

Apply the configuration to cluster2:

istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml

Install the east-west gateway in cluster2

As we did with cluster1 above, install a gateway in cluster2 that is dedicated to east-west traffic.

samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster cluster2 --network network2 | \
    istioctl --context="${CTX_CLUSTER2}" install -y -f -

Wait for the east-west gateway to be assigned an external IP address:

kubectl --context="${CTX_CLUSTER2}" get svc istio-eastwestgateway -n istio-system

Expose services in cluster2

As we did with cluster1 above, expose services via the east-west gateway.

kubectl --context="${CTX_CLUSTER2}" apply -n istio-system -f \
    samples/multicluster/expose-services.yaml

Step 4: Enable Endpoint Discovery

To enable endpoint service discovery we have to grant both clusters to access each other Kubernetes api services. For this, we are going to create the secrets mentioned below.

Install a remote secret in cluster2 that provides access to cluster1’s API server.

istioctl x create-remote-secret \
  --context="${CTX_CLUSTER1}" \
  --name=cluster1 | \
  kubectl apply -f - --context="${CTX_CLUSTER2}"

Install a remote secret in cluster1 that provides access to cluster2’s API server.

istioctl x create-remote-secret \
  --context="${CTX_CLUSTER2}" \
  --name=cluster2 | \
  kubectl apply -f - --context="${CTX_CLUSTER1}"

Now we have completed istio installation and configuration in both clusters. Maker sure all pods in istio-system namespace are running without any errors.

Verify the installation

To verify the whole setup we will deploy the HelloWorld application V1 to cluster1 and V2 to cluster2. Upon receiving a request, HelloWorld will include its version in its response.

We will also deploy the Sleep container to both clusters. We will use these pods as the source of requests to the HelloWorld service, simulating in-mesh traffic. Finally, after generating traffic, we will observe which cluster received the requests.

Deploy the HelloWorld Service

In order to make the HelloWorld service callable from any cluster, the DNS lookup must succeed in each cluster (see deployment models for details). We will address this by deploying the HelloWorld Service to each cluster in the mesh.

To begin, create the sample namespace in each cluster:

kubectl create --context="${CTX_CLUSTER1}" namespace sample
kubectl create --context="${CTX_CLUSTER2}" namespace sample

Enable automatic sidecar injection for the sample namespace:

kubectl label --context="${CTX_CLUSTER1}" namespace sample \
    istio-injection=enabled
kubectl label --context="${CTX_CLUSTER2}" namespace sample \
    istio-injection=enabled

Create the HelloWorld service in both clusters:

kubectl apply --context="${CTX_CLUSTER1}" \
    -f samples/helloworld/helloworld.yaml \
    -l service=helloworld -n sample
kubectl apply --context="${CTX_CLUSTER2}" \
    -f samples/helloworld/helloworld.yaml \
    -l service=helloworld -n sample

Deploy HelloWorld V1

Deploy the helloworld-v1 application to cluster1:

kubectl apply --context="${CTX_CLUSTER1}" \
    -f samples/helloworld/helloworld.yaml \
    -l version=v1 -n sample

Confirm the helloworld-v1 pod status:

kubectl get pod --context="${CTX_CLUSTER1}" -n sample -l app=helloworld

Wait until the status of helloworld-v1 is Running.

Deploy HelloWorld V2

Deploy the helloworld-v2 application to cluster2:

kubectl apply --context="${CTX_CLUSTER2}" \
    -f samples/helloworld/helloworld.yaml \
    -l version=v2 -n sample

Confirm the status of the helloworld-v2 pod status:

kubectl get pod --context="${CTX_CLUSTER2}" -n sample -l app=helloworld

Wait until the status of helloworld-v2 is Running.

Deploy Sleep

Deploy the Sleep application to both clusters:

kubectl apply --context="${CTX_CLUSTER1}" \
    -f samples/sleep/sleep.yaml -n sample
kubectl apply --context="${CTX_CLUSTER2}" \
    -f samples/sleep/sleep.yaml -n sample

Confirm the status Sleep pod on cluster1:

kubectl get pod --context="${CTX_CLUSTER1}" -n sample -l app=sleep

Wait until the status of the Sleep pod is Running.

Confirm the status of the Sleep pod on cluster2:

kubectl get pod --context="${CTX_CLUSTER2}" -n sample -l app=sleep

Wait until the status of the Sleep pod is Running.

Verifying Cross-Cluster Traffic

To verify that cross-cluster load balancing works as expected, call the HelloWorld service several times using the Sleep pod. To ensure load balancing is working properly, call the HelloWorld service from all clusters in your deployment.

Send one request from the Sleep pod on cluster1 to the HelloWorld service:

kubectl exec --context="${CTX_CLUSTER1}" -n sample -c sleep \
    "$(kubectl get pod --context="${CTX_CLUSTER1}" -n sample -l \
    app=sleep -o jsonpath='{.items[0].metadata.name}')" \
    -- curl -sS helloworld.sample:5000/hello

Repeat this request several times and verify that the HelloWorld version should toggle between v1 and v2:

Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8
Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv
...

Now repeat this process from the Sleep pod on cluster2:

kubectl exec --context="${CTX_CLUSTER2}" -n sample -c sleep \
    "$(kubectl get pod --context="${CTX_CLUSTER2}" -n sample -l \
    app=sleep -o jsonpath='{.items[0].metadata.name}')" \
    -- curl -sS helloworld.sample:5000/hello