发送OpenTelemetry数据

本节我们将在PetAdoptionHistory服务中添加OpenTelemetry (OTEL) 相关代码，以收集trace和metric信息，我们将在X-ray中看到跟trace相关的信息，在CloudWatch/Prometheus中看到metric相关的信息。

主要流程如下：

  1. 修改应用代码，添加OpenTelemetry的SDK。 可参考 https://opentelemetry.io/docs/instrumentation/python/

  2. 应用的trace被收集到ADOT，然后发送到X-Ray

添加ADOT(AWS Distro for OpenTelemetry)

我们已经提前配置好了ADOT Collector，现在只需要更新应用来暴露相关的trace数据：

关于ADOT Collector的配置，可以参考 https://github.com/aws-samples/one-observability-demo/blob/main/PetAdoptions/petadoptionshistory-py/deployment.yaml ：

image-20230824220519963

更新应用的代码

原来PetAdoptionHistory API的代码很简单，用Flask创建两个接口负责get和delete请求：

image-20230824223120096

我们在此代码基础上添加跟OpenTelemetry相关的代码，将petadoptionshistory.py内容更新为：

import logging
import os
import psycopg2
import config
import repository
from flask import Flask, jsonify

# OTLP Tracing
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource

# Exporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Propagation
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.aws import AwsXRayPropagator

# AWS X-Ray ID Generator
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator

# Instrumentation
from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor

# Instrumentation
BotocoreInstrumentor().instrument()

# Setup flask app
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

logging.basicConfig(level=os.getenv('LOG_LEVEL', 20), format='%(message)s')
logger = logging.getLogger()
cfg = config.fetch_config()
conn_params = config.get_rds_connection_parameters(cfg['rds_secret_arn'], cfg['region'])
db = psycopg2.connect(**conn_params)

@app.route('/petadoptionshistory/api/home/transactions', methods=['GET'])
def transactions_get():
    transactions = repository.list_transaction_history(db)
    return jsonify(transactions)

@app.route('/petadoptionshistory/api/home/transactions', methods=['DELETE'])
def transactions_delete():
    repository.delete_transaction_history(db)
    return jsonify(success=True)

@app.route('/health/status')
def status_path():
    return jsonify(success=True)
    
# Setup AWX X-Ray Propagator
set_global_textmap(AwsXRayPropagator())

resource = Resource(attributes={
    SERVICE_NAME: "PetAdoptionsHistory"
})

provider = TracerProvider(resource=resource, id_generator=AwsXRayIdGenerator())
processor = BatchSpanProcessor(OTLPSpanExporter())
provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(provider)

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer(__name__)

相关依赖包的安装方式如下，不过由于后面在docker中构建镜像，所以不必在控制台安装：

pip3 install opentelemetry-api
pip3 install opentelemetry-sdk
pip3 install opentelemetry-distro
pip3 install opentelemetry-exporter-otlp-proto-grpc
pip3 install opentelemetry-sdk-extension-aws
pip3 install opentelemetry-propagator-aws-xray
pip3 install opentelemetry-instrumentation-botocore
pip3 install opentelemetry-instrumentation-flask

重新部署应用

上面petadoptionshistory.py的代码更新后，我们重新部署下应用。

获取ECR仓库的地址：

PETHISTORYECR=$(aws ssm get-parameter --name '/petstore/pethistoryrepositoryuri' | jq -r .Parameter.Value)

登录到ECR：

aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $PETHISTORYECR

重新构建镜像，并将镜像上传：

docker build -t pet-adoptions-history:latest .
docker tag pet-adoptions-history:latest $PETHISTORYECR:latest
docker push $PETHISTORYECR:latest

上传完成后，删除原来的pod，会重新拉取最新的镜像然后重新创建：

kubectl get pods # identify pethistory-deployment pod
kubectl delete "pod/$(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep pethistory-deployment)"
kubectl get pods # check AGE column of pod

检查新的应用是否正常运行：

curl $PETHISTORYURL/api/home/transactions

kubectl logs "pod/$(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep pethistory-deployment)" -c pethistory

image-20230824223808619

现在我们的应用已经往X-Ray发送OpenTelemetry数据，我们将在下一节查看这些数据