dapr 102 - State Management and Tracing

2020-07-08 dapr kubernetes dapr-series dapr kubernetes Comments Word Count: 2672words Read Count: 13minutes

Posts in this series

dapr 101 - Concepts and Setup
dapr 102 - State Management and Tracing (This Post)
dapr 103 - Pub/Sub and Observability Metrics

In a Galaxy far far away…

We took a look at the basic idea behind dapr and the building block that make up dapr. We also went through the steps of how to get the dapr Development environment up and running on a k3s cluster with the help of dapr/cli and setup a redis based state-management building block.

Goals

This post will aim at providing you an overview of how to write your first few microservices using dapr and run them as well as configure a tracing specification to generate some tracing. We will also look at some of the methods used by the dapr sidecar to invoke other services along the way.

Pre-Requisites

Kubernetes Cluster and `dapr` Setup

If, for some reason, this is the first post you landed in, please make sure you have the following setup configured so that you can follow along with rest of the post in this.

~ at ☸️  v1.18.4+k3s1 k3s-default
➜ kubectl get nodes -o wide
NAME                       STATUS   ROLES    AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE   KERNEL-VERSION     CONTAINER-RUNTIME
k3d-k3s-default-worker-1   Ready    <none>   18h   v1.18.4+k3s1   172.21.0.4    <none>        Unknown    4.19.76-linuxkit   containerd://1.3.3-k3s2
k3d-k3s-default-worker-2   Ready    <none>   18h   v1.18.4+k3s1   172.21.0.5    <none>        Unknown    4.19.76-linuxkit   containerd://1.3.3-k3s2
k3d-k3s-default-worker-0   Ready    <none>   18h   v1.18.4+k3s1   172.21.0.3    <none>        Unknown    4.19.76-linuxkit   containerd://1.3.3-k3s2
k3d-k3s-default-server     Ready    master   18h   v1.18.4+k3s1   172.21.0.2    <none>        Unknown    4.19.76-linuxkit   containerd://1.3.3-k3s2

~ at ☸️  v1.18.4+k3s1 k3s-default
➜ kubectl get pods -o wide | grep dapr-
dapr-sentry-58c576ff98-vl8n8            1/1     Running   0          18h   10.42.2.3   k3d-k3s-default-worker-1   <none>           <none>
dapr-operator-75b4f7986b-r25d4          1/1     Running   0          18h   10.42.3.3   k3d-k3s-default-worker-2   <none>           <none>
dapr-sidecar-injector-c898fb49b-v4pnf   1/1     Running   0          18h   10.42.3.2   k3d-k3s-default-worker-2   <none>           <none>
dapr-placement-84f9cd87b7-ckjrl         1/1     Running   1          18h   10.42.0.3   k3d-k3s-default-server     <none>           <none>

~ at ☸️  v1.18.4+k3s1 k3s-default
➜ kubectl get svc -o wide | grep dapr-
dapr-api                ClusterIP      10.43.39.14     <none>        80/TCP                                18h   app=dapr-operator
dapr-placement          ClusterIP      10.43.245.162   <none>        80/TCP                                18h   app=dapr-placement
apr-sentry             ClusterIP      10.43.84.125    <none>        80/TCP                           18h   app=dapr-sentry
dapr-sidecar-injector   ClusterIP      10.43.252.177   <none>        443/TCP                               18h   app=dapr-sidecar-injector

If your system doesn’t have this, please go back and take a look at the Post #1 of this series to find out how to get the setup created.

Programming Language Environments

For the purpose of this post, we will be using python and go based sdk provided by dapr and orchestrate some of the workflows. Please setup the following environments if you don’t have them created already

Python 3.8 or above
Golang 1.14.2 or above
Your favorite code editor. (Mine is vim)

Code Repo

Please clone the harshanarayana/dapr-servies and checkout the dapr-102 branch to follow along with this post to make it easier.

git clone git@github.com:harshanarayana/dapr-series.git
cd dapr-series
git checkout dapr-102

Once you checkout the repo, the structure would look like this.

Python Service

For all future references in this post, this service will be referred to as Service1 or Service 1 or service-1. This will be a basic sanic based API using the dapr/python-sdk and create a kubernetes deployment infrastructure to run them.

Design

sequenceDiagram User ->> Service 1: Invoke POST /state API Service 1 -->> Dapr Sidecar : Invoke POST /v1.0/state/{stateStore} API Dapr Sidecar -->> Redis: persist Dapr Sidecar ->> Service 1: Confirmation Service 1 -->> User: Confirmation User ->> Service 1: Invoke GET /state API Service 1 -->> Dapr Sidecar: Invoke GET /v1.0/state/{stateStore}/{key} API Dapr Sidecar -->> Redis: HGETALL "{serviceID} || {key}" Redis -->> Dapr Sidecar: Response Dapr Sidecar -->> Service 1: Forward Service 1 ->> User: Response

Install Dependencies

Setup Python new Virtual Env

pyenv install 3.8.3
pyenv virtualenv 3.8.3 dapr-series
pyenv activate dapr-series

Install `dapr` SDK

pip install dapr dapr-dev sanic

Create the base `dapr` interaction

For the example section, we are going to leverage the grpc client of the python-sdk.

import grpc

from dapr.proto import api_v1, common_v1, api_service_v1
from google.protobuf.any_pb2 import Any

"""
Following ports are configurable. These ports are used in the sidecar container 
that runs the `daprd` process. These Ports and `localhost` is how the service
interacts with the `daprd` service which will internally take care of persisting
the sate in the state store
"""
DAPR_GRPC_PORT = getenv("DAPR_GRPC_PORT", "50001")
DAPR_HTTP_PORT = getenv("DAPR_HTTP_PORT", "3500")
STATE_KEY = getenv("STATE_KEY", "dapr-series")

# You need to take extra care here to make sure that this `store name` matches the
# `component` name you have created which points to the redis during initial setup of
# `dapr` on your kubernetes cluster.
STORE_NAME = getenv("STORE_NAME", "statestore")

DAPR_CLIENT = api_service_v1.DaprStub(
    grpc.insecure_channel(f"localhost:{DAPR_GRPC_PORT}")
)


def _store_state(state_value):
    logger.info(
        f"CLIENT: {DAPR_CLIENT}, STORE_NAME: {STORE_NAME}, STATE_KEY: {STATE_KEY}"
    )
    request = common_v1.StateItem(key=STATE_KEY, value=state_value.encode("utf-8"))
    state = api_v1.SaveStateRequest(store_name=STORE_NAME, states=[request])
    return DAPR_CLIENT.SaveState(state)


def _get_state():
    logger.info(
        f"CLIENT: {DAPR_CLIENT}, STORE_NAME: {STORE_NAME}, STATE_KEY: {STATE_KEY}"
    )
    request = api_v1.GetStateRequest(store_name=STORE_NAME, key=STATE_KEY)
    state = DAPR_CLIENT.GetState(request=request)
    return state.data.decode("utf-8")


def _delete_state():
    logger.info(
        f"CLIENT: {DAPR_CLIENT}, STORE_NAME: {STORE_NAME}, STATE_KEY: {STATE_KEY}"
    )
    request = api_v1.DeleteStateRequest(store_name=STORE_NAME, key=STATE_KEY)
    return DAPR_CLIENT.DeleteState(request=request)

State Store Name

While creating the STORE_NAME variable you need to be extra careful. If you don’t setup this the right way, you will run into ERR_STATE_STORE_NOT_FOUND errors. The description of this error is not easy to root cause the issue. It also accompanies Error received from peer ipv6:[::1]:50001 message along with it making it twice as hard to find out what is wrong.

Here is how you decide what the STORE_NAME value is going to be.

➜ kubectl get component -o json | jq '.items[] | select(.spec.type=="state.redis") | .metadata.name'
"statestore"

API Wrapper

In order for you to be able to invoke these options, let us create a simple CRUD API service using sanic and add some basic loggers to support debugging if required. I am not going into the details of how to write API using sanic in this post. That will probably be a post for a different day on it’s own.

from sanic import Sanic
from sanic.log import logger
from sanic.request import Request
from sanic.response import json

app = Sanic(__name__)

@app.listener("after_server_start")
async def log_info(app: Sanic, loop):
    logger.info("=================================================")
    logger.info(f"DAPR_GRPC_PORT -> {DAPR_GRPC_PORT}")
    logger.info(f"DAPR_HTTP_PORT -> {DAPR_HTTP_PORT}")
    logger.info("=================================================")


@app.get("/ping")
async def ping(request: Request):
    return json({"message": "ping"})


@app.delete("/state")
async def delete(request: Request):
    _delete_state()
    return json({"message": "state deleted"})


@app.get("/state")
async def state(request: Request):
    return json({"state": _get_state()})


@app.post("/state")
async def save(request: Request):
    body = request.json
    _store_state(body.get("value", "TEST"))
    return json({"message": "State Stored"})


if __name__ == "__main__":
    app.run(port=6060, debug=True)

Kubernetes Deployment and Service

---
# Service specification
apiVersion: v1
kind: Service
metadata:
  name: service-1
  labels:
    app: service-1
spec:
  selector:
    app: service-1
  ports:
    - port: 80
      targetPort: 6060
      name: http
  type: LoadBalancer
---
# Deployment Specification
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: service-1
  name: service-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-1
  template:
    metadata:
      labels:
        app: service-1
      annotations:
        dapr.io/enabled: "true"
        dapr.io/id: "service-1"
        dapr.io/port: "6060"
    spec:
      containers:
        - name: service-1
          image: harshanarayana/dapr-series-service-1:latest
          ports:
            - containerPort: 6060
              name: http

Testing Things

Now that we have the basic infra setup, let us give this a spin and see how things work out. And in doing so, let us also figure out a few other things related to how the sidecar interfaces against the main service as well as how the sidecar APIs are invoked by the service itself.

kubectl apply -f service-1/kubernetes/service-1.yaml

Running the above command will get your services up and running. Now, you need to do a few things before you can take this API for a spin.

# forward the port so that you can invoke the API as localhost. If you don't want to do that, then you can use the
# load-balancer's external API to reach out to the API. Whichever works well for you.
kubectl port-forward deployment/service-1 6060 &

curl --location --request GET 'localhost:6060/state' --header 'Content-Type: application/json'

# {"state":""}

curl --location --request POST 'localhost:6060/state' --header 'Content-Type: application/json' --data-raw '{
        "value": "New State"
}'

# {"message":"State Stored"}

curl --location --request GET 'localhost:6060/state' --header 'Content-Type: application/json'

# {"state":"New State"}

curl --location --request DELETE 'localhost:6060/state' --header 'Content-Type: application/json'

# {"message":"state deleted"}

curl --location --request GET 'localhost:6060/state' --header 'Content-Type: application/json'

# {"state":""}

Golang Service

Design

sequenceDiagram User ->> Service 2: Invoke POST /state API Service 2 -->> Dapr Sidecar : Invoke POST /v1.0/state/{stateStore} API Dapr Sidecar -->> Redis: persist Dapr Sidecar -->> Service 2: Confirmation Service 2 ->> User: Confirmation User ->> Service 2: Invoke GET /state API Service 2 -->> Dapr Sidecar: Invoke GET /v1.0/state/{stateStore}/{key} API Dapr Sidecar -->> Redis: HGETALL "{serviceID} || {key}" Redis -->> Dapr Sidecar: Response Dapr Sidecar -->> Service 2: Forward Service 2 ->> User: Response

Install Dependencies

Let us create a golang Dev environment using gvm and setup the required dependencies to create a new project to build our example service. This service will be called as Service 2, Service2 or service-2 for the reset of this post.

If you are not comfortable using gvm you can use any other way of your choice to setup your environment. However, please keep in mind that you need a golang version v1.14.x to be running.

Please note that I am using gin-gonic/gin HTTP Framework to create the API. However, you are free to opt for any other one your liking.

gvm install go1.14.4
gvm use go1.14.4

Setup `gomod` Project

Since we are using go1.14.4 you are free to create your project anywhere so long as you have the right $GOPATH and $GOROOT configured.

mkdir -p $GOPATH/src/github.com/dapr-series
cd $GOPATH/src/github.com/dapr-series
go mod init

go get github.com/dapr/go-sdk/client
go get github.com/gin-gonic/gin
touch main.go

Create `dapr` Client

// file: main.go
package main
import (
  "github.com/dapr/go-sdk/client"
)

var daprClient client.Client

func init()  {
	if c, e := client.NewClient(); e != nil {
		panic(e)
	} else {
		daprClient = c
	}
}

The above snipper will setup a base dapr client in GRPC mode. There are other ways to create the client instead of using the client.NewClient() method in case if you are not using the default GRPC port of 50001.

In golang the init method is a special function that is invoked which will help initialize a client as soon as the API context starts up.

Create `dapr` Wrappers

Now, like we did in case of python service, let us create some wrapper methods to interact with the dapr to perform state persistence. In this example, I am wrapping the state management methods directly in the API handlers. However, you can move them into a standalone method for your usecases as required.

const (
	StateKey = "dapr-series-go"
	StoreName = "statestore-2"
)

type StateRequest struct {
	Message string `json:"message"`
}

func getState() gin.HandlerFunc {
	return func(context *gin.Context) {
		if data, eTag, err := daprClient.GetState(context2.Background(), StoreName, StateKey); err != nil {
			context.JSON(404, gin.H{
				"message": "Not found",
			})
		} else {
			context.JSON(200, gin.H{
				"state": string(data),
				"eTag": eTag,
			})
		}
	}
}

func saveState() gin.HandlerFunc {
	return func(context *gin.Context) {
		var data StateRequest
		if e := context.Bind(&data); e != nil {
			context.JSON(http.StatusBadRequest, gin.H{"error": e.Error()})
			return
		}
		request := &client.State{
			StoreName: StoreName,
			States: []*client.StateItem{
				{
					Key: StateKey,
					Value: []byte(data.Message),
				},
			},
		}
		if e := daprClient.SaveState(context2.Background(), request); e != nil {
			context.JSON(http.StatusBadRequest, gin.H{"error": e.Error()})
			return
		} else {
			context.JSON(200, gin.H{
				"message": "State Persisted",
			})
		}
	}
}

API Wrapper

In order for you to be able to invoke these options, let us create a simple CRUD API service using gin as follow.

func main() {
	r := gin.New()
	r.Use(gin.Logger())
	r.Use(gin.Recovery())

	r.GET("/ping", ping())
	r.GET("/state", getState())
	r.POST("/state", saveState())

	s := &http.Server{
		Addr: ":7070",
		Handler: r,
	}

  // The following section of the code is responsible for ensuring that the
  // grpc client connection is terminated cleanly when we kill the pod or
  // restart it as part of the pod life-cycle events.
	go func() {
		if err := s.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			panic(err)
		}
	}()

	done := make(chan os.Signal)
	signal.Notify(done, os.Interrupt)
	<-done

	daprClient.Close()
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	if err := s.Shutdown(ctx); err != nil {
		log.Fatal("Server Shutdown:", err)
	}
	log.Println("Server exiting")
}

Kubernetes Artifacts

In order for us to test the service-2, let us first create a new state store that we can use with the new service and create the necessary deployment and service objects.

Statestore

# statestore-2.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore-2
  namespace: default
spec:
  type: state.redis
  metadata:
    - name: redisHost
      value: redis-leader.default:6379
    - name: redisPassword
      value: ""

kubectl apply -f service-2/kubernetes/statestore-2.yaml

Kubernetes Objects

---
apiVersion: v1
kind: Service
metadata:
  name: service-2
  labels:
    app: service-2
spec:
  selector:
    app: service-2
  ports:
    - port: 81
      targetPort: 7070
      name: http
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: service-2
  name: service-2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-2
  template:
    metadata:
      labels:
        app: service-2
      annotations:
        dapr.io/enabled: "true"
        dapr.io/id: "service-2"
        dapr.io/port: "7070"
    spec:
      containers:
        - name: service-2
          image: harshanarayana/dapr-series-service-2:latest
          ports:
            - containerPort: 7070
              name: http

kubectl apply -f service-2/kubernetes/service-2.yaml

Testing Things

# forward the port so that you can invoke the API as localhost. If you don't want to do that, then you can use the
# load-balancer's external API to reach out to the API. Whichever works well for you.
kubectl port-forward deployment/service-2 7070 &

curl --location --request GET 'localhost:7070/state' --header 'Content-Type: application/json'

# {"state":""}

curl --location --request POST 'localhost:7070/state' --header 'Content-Type: application/json' --data-raw '{
        "value": "New State"
}'

# {"message":"State Stored"}

curl --location --request GET 'localhost:7070/state' --header 'Content-Type: application/json'

# {"state":"New State"}

Tracing

As part of it’s observability building block, dapr does the following for use out of the box.

dapr is responsible for generation and propagation of W3C trace context between services
dapr is responsible for generation of the trace context but you are going to propagate the context or vice-versa.

Design

sequenceDiagram User ->> Service 1: Invoke GET /golang/ping rect rgba(0, 0, 255, .1) Note right of Service 1: Tracing Middleware and Backend Service 1 -->> Dapr Sidecar 1: Invoke GET /invoke/service-2/ping Dapr Sidecar 1 -->> Dapr Sidecar 2: Invoke GET /ping Dapr Sidecar 2 -->> Service 2: Invoke /ping Service 2 -->> Dapr Sidecar 2: Responsd with {"message": "pong"} Dapr Sidecar 2 -->> Dapr Sidecar 1: Forward Dapr Sidecar 1 -->> Service 1: Forward end Service 1 ->> User: Forward

Configuration

In order for us to be able to visualize the tracing aspects of the service observability, we need to configure something that can be used to visualize the traces. Since dapr generates the traces in W3C standards, any utility that can leverage this standard can be used to visualize the traces.

For the purpose of this demo, we will be using jaeger as a way to visualize the traces generated by the dapr.

Visualization

# jaeger.yaml
apiVersion: v1
kind: Service
metadata:
  name: jaeger
  labels:
    app: jaeger
    jaeger-infra: query-service
spec:
  ports:
    - name: jaeger
      port: 16686
      targetPort: 16686
    - name: zipkin
      port: 9411
      targetPort: 9411
  selector:
    app: jaeger
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  labels:
    app: jaeger
    jaeger-infra: agent-daemonset
spec:
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
        - name: agent-instance
          image: jaegertracing/all-in-one:1.18
          env:
            - name: COLLECTOR_ZIPKIN_HTTP_PORT
              value: "9411"
          ports:
            - containerPort: 9411
              protocol: TCP
            - containerPort: 16686
              protocol: TCP
          resources:
            requests:
              memory: 200M
              cpu: 200m
            limits:
              memory: 200M
              cpu: 200m

kubectl apply -f deployment/jaeger.yaml

`dapr` Config

We need to create an new Component in Kubernetes with the type exporters.zipkin in order for us to tell dapr what to use for exporting tracing as part of observability components. In this case, we configure the zipkin exporter and enable the zipkin compatible endpoint in jaeger on port 9411. Currently this is persistence less in our examples.

# component.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: zipkin
  namespace: default
spec:
  type: exporters.zipkin
  metadata:
    - name: enabled
      value: "true"
    - name: exporterAddress
      value: "http://jaeger.default.svc.cluster.local:9411/api/v2/spans"

Once we have the exporters.zipkin component created, we need to create a Configuration CR indicating the tracing configuration to be used for exporting the traces to jaeger with sampling rate.

# configuration.yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: tracing
  namespace: default
spec:
  tracing:
    samplingRate: "1"

kubectl apply -f observability/component.yaml
kubectl apply -f observability/configuration.yaml

Enable Tracing

Pick one of the services we created above and try out the tracing just by checking how it looks and works in the single service usecase. i.e. User invokes the service and the service returns a response back.

In order for you to do that, you need to add a dapr.io/config: "tracing" annotation to your existing deployment of service-1.

annotation:
  dapr.io/configuration: "tracing"

Once you do this, do a few API calls to GET and POST and DELETE and you can check the traces getting generated on your jaeger front-end.

List All traces view

Detailed trace view

Service to Service Tracing

Now that we have the fundamental idea of how the tracing works, let us take a look at understanding how service to service tracing can be achieved.

In order for you to do that, we are going to add a new GET /s2/ping API to service-1 which will invoke the GET /ping endpoint of the service-2. We will also ensure we enable tracing on both service-1 and service-2 using the annotation specified above.

# service-1/python/app.py

# import python requests module to invoke the `dapr` sidecar API
import requests

# This is the base URL pointing to `dapr` sidecar HTTP url for Service to service
# request forwarding and invocation.

DAPR_FORWARDER = f"http://localhost:{DAPR_HTTP_PORT}/v1.0/invoke"

# Add a new route and invoke the `/method/ping` API on the sidecar.
@app.get("/s2/ping")
async def s2_ping(request: Request):
    d = requests.get(f"{DAPR_FORWARDER}/service-2/method/ping")
    return json({"message": d.json()})

The most important part of this entire invocation sequence is the /service-2/method/ping. This URL basically follows the following pattern. <service-name>/method/<url-to-invoke>

Testing

curl --location --request GET 'localhost:6060/s2/ping' --header 'Content-Type: application/json'

# {"message":{"message":"ping"}}

Now, if you open up your jaeger UI and filter the spans for service-1 or service-2 you will notice the following traces.

Simple Trace View

Detailed Trace View

What Next?

This is all for this post for now. In the next post of this series, we will talk about how to consume pub-sub components to build an event based service and enable metrics part of the observability using grafana dashboard.

Permalink: https://harshanarayana.dev/2020/07/dapr-102/

dapr 102 - State Management and Tracing

Posts in this series

In a Galaxy far far away…

Goals

Pre-Requisites

Kubernetes Cluster and dapr Setup

Programming Language Environments

Code Repo

Python Service

Design

Install Dependencies

Setup Python new Virtual Env

Install dapr SDK

Create the base dapr interaction

State Store Name

API Wrapper

Kubernetes Deployment and Service

Testing Things

Golang Service

Design

Install Dependencies

Setup gomod Project

Create dapr Client

Create dapr Wrappers

API Wrapper

Kubernetes Artifacts

Statestore

Kubernetes Objects

Testing Things

Tracing

Design

Configuration

Visualization

dapr Config

Enable Tracing

List All traces view

Detailed trace view

Service to Service Tracing

Testing

Simple Trace View

Detailed Trace View

What Next?

Kubernetes Cluster and `dapr` Setup

Install `dapr` SDK

Create the base `dapr` interaction

Setup `gomod` Project

Create `dapr` Client

Create `dapr` Wrappers

`dapr` Config