OpenTelemetry with ASP.NET Core

Introduction

I wrote a post not too long ago about the building blocks of telemetry, or distributed tracing, with ASP.NET Core. Now I'm going to talk about how we can see it working using an open source bundle, otel-lgtm. Mind you, this may not be 100% suitable for production, but for development purposes, it should be fine, all concepts apply, of course.

I won't cover everything about OpenTelemetry here, but should hopefully give you some insights as to what's there and how you can make good use of it. First, lets see what some of the concepts are.

OpenTelemetry

OpenTelemetry (OTel) provides a single, open-source standard, and a set of technologies to capture and export metrics, traces, and logs from your cloud-native applications and infrastructure. OTel is both a standard and a reference implementation that builds on other standards, such as W3C Distributed Tracing. OpenTelemetry is essentially a data collector that stores data that is sent seamlessly by web servers and clients, and there are implementations for .NET. OpenTelemetry collects logs, metrics, and traces - the three pillars of observability - from different hosts, and thus can be used for observability. OpenTelemetry for .NET can instrument both server-side (ASP.NET Core) as well as client code (HttpClient calls).

I will now introduce some related tools.

Grafana, Loki, Tempo, and Prometheus

There are plenty of tools that can be used to collect, gather, and visualise logs, traces, and metrics, the following ones are part of a package that I will use to demonstrate the concepts discussed in this post. They reflect just my preferences, and, as I said, there are many others.

Visualisation: Grafana

Grafana is, per the official documentation, the open source analytics & monitoring solution for every database. It can be used to visualise and build dashboards over the collected data. It is available as source from https://github.com/grafana/grafana.

Logging: Loki

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system. It can be obtained from https://github.com/grafana/loki.

Tracing: Grafana Tempo

Grafana Tempo is an open source, easy-to-use, and high-scale distributed tracing backend. Tempo is available from https://github.com/grafana/tempo.

Metrics: Prometheus

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach. You can get it from https://github.com/prometheus/prometheus.

Putting it all together

docker-otel-lgtm is an OpenTelemetry backend bundled as a Docker image. It contains Grafana, Loki, Tempo, and Prometheus, all fully configured and integrated. This bundle is described here, and you may want to have a look. Grafana is used for visualisations and dashboards, and is pre-connected to the Loki and Tempo databases, and to Prometheus, which is used for querying them. Full source is available here and you can learn more on how to use metrics, logs, traces and profiles in Grafana here.

The best way to get into it it is by cloning the repository and then using Docker to run it. You can run it directly using docker command or by using a Docker Compose file. Here is my docker-compose.yml:

services:
  otel-lgtm:
    image: grafana/otel-lgtm
    container_name: otel-lgtm
    ports:
      - 3000:3000
      - 4317:4317
      - 4318:4318

Ports 4317 and 4318 are for the GRPC and the HTTP Protobuf exporter, and 3000 is for Grafana and Prometheus. More on this in a moment. Nothing too fancy here.

After all the services are running, you can point your browser to http://localhost:3000.

Note: if you are using Google Chrome, because of HSTS, you may be redirected to https://localhost:3000, but because the site is not being served using HTTPS, the connection will fail. If this happens, navigate to chrome://net-internals/#hsts and then delete domain security policies for host localhost.

Here is a screenshot of what Grafana looks like:

The Grafana explorer
And in http://localhost:3000/connections/datasources we can see the configured data sources (Loki, Prometheus, and Tempo):

The datasources

Setting up OpenTelemetry in ASP.NET Core

As we said, OpenTelemetry collects different information:

  • Logs
  • Metrics
  • Traces
We need to setup OpenTelemetry in ASP.NET Core for all of these. First, we'll need a few Nuget packages:

After you setup OpenTelemetry in your ASP.NET Core project, the logs, metrics, and traces will be sent seamlessly to the OpenTelemetry. Let's have a look at this.

Creating Activity Sources, Meters, Gauges, and Counters

Let's create an activity source, an instance of ActivitySource, that will be used to create activities for tracing operations (like action method calls or other requests):

var forecastActivitySource = new ActivitySource("Forecast.ActivitySource");

We will create a meter too:

var forecastMeter = new Meter("forecast_meter", "1.0.0");

From this meter we will create a counter and a gauge (new in .NET 9):

var forecastCounter = forecastMeter.CreateCounter<int>("forecast_counter", description: "Counts the number of forecasts");
var temperatureGauge = forecastMeter.CreateGauge<int>("forecast_temperature", description: "Forecast temperature");

We'll se in a moment what they will be used for. For now, it is sufficient to say that a counter will represent a single monotonically increasing numeric value, whereas a gauge represents also a single numerical value that can arbitrarily go up and down.

Now, because we will need to access this activity source, we need to store it in some place where can easily reach for it. Two main options:

  • Store in a static field
  • Add to the Dependency Injector (DI) container as a Singleton

If we go down the static field option, it could be like this:

public static class Activities
{
    internal static readonly ActivitySource ForecastActivitySource = new ActivitySource("Forecast.ActivitySource");
}

And, for the DI version:

using var forecastActivitySource = new ActivitySource("Forecast.ActivitySource");
builder.Services.AddKeyedSingleton<ActivitySource>(forecastActivitySource, "Forecast.ActivitySource");

We're registering our ActivitySource instance under a specific key, because we have have more than one. In a while we'll see a better version of this.

Since ActivitySource is disposable, in this example, we are implicitly (using) disposing of the instance at the end of the current scope, which will likely be the shutdown of the web application (WebApplication.Run(), in the default code template). For the static field version, we should dispose of it ourselves explicitly.

Slightly better, we can create a class to store both our ActivitySource and our metrics (the counter and the gauge):

public sealed class Instrumentation : IDisposable
{
    public Instrumentation(ActivitySource forecastActivitySource, Counter<int> forecastCounter, Gauge<int> forecastTemperature)
    {
        ForecastActivitySource = forecastActivitySource;
        ForecastCounter = forecastCounter;
        ForecastTemperature = forecastTemperature;
    }

    public ActivitySource ForecastActivitySource { get; }
    public Counter<int> ForecastCounter { get; }
    public Gauge<int> ForecastTemperature { get; }

    public void Dispose()
    {
        ForecastActivitySource.Dispose();
    }
}

It is now just a matter of registering it with the DI container:

builder.Services.AddSingleton(new Instrumentation(forecastActivitySource, forecastCounter, forecastTemperature));

Likewise, we should not forget to dispose of it when the application is terminating.

Note: the inspiration for this class came from a Microsoft sample made available here.

Setting Up OpenTelemetry

Next, we need to register the OpenTelemetry services by calling the AddOpenTelemetry extension method:

var telemetry = builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(options => options
        .AddService(serviceName: Environment.MachineName)
        .AddAttributes(new Dictionary<string, object> { ["host.environment"] = builder.Environment.EnvironmentName }));

Notice here that we are setting the name of the machine the code is running as the OpenTelemetry service name; other options could be leaving it as blank, which means the current assembly name would be used, or some other meaningful name, we'll see later how this is used. We're also, for the sake of completion, passing a custom attribute "host.environment" that contains the current execution environment.

Logs

Let's start by making OpenTelemetry aware of our logs:

telemetry.WithLogging(static options =>
{
    options.AddConsoleExporter();
});

And if we need to control (filter out) what is logged to OpenTelemetry:

builder.Logging.AddFilter<OpenTelemetryLoggerProvider>("*", LogLevel.Error);
builder.Logging.AddFilter<OpenTelemetryLoggerProvider>("MyNamespace.MyClass", LogLevel.Warning);

Please refer to this page for more information on how to filter logs with the OpenTelemetryLoggerProvider.

If we want, we can also include activity-specific information in the logs automatically:

builder.Logging.Configure(static options =>
{
    options.ActivityTrackingOptions = ActivityTrackingOptions.SpanId |
                                      ActivityTrackingOptions.TraceId |
                                      ActivityTrackingOptions.ParentId |
                                      ActivityTrackingOptions.Baggage |
                                      ActivityTrackingOptions.Tags;
});

And here is the documentation for LoggerFactoryOptions and ActivityTrackingOptions.

Metrics

Because the AddOpenTelemetry and ConfigureResource methods allow composability, we can then add other services, such as metrics:

telemetry.WithMetrics(options => options
    .AddAspNetCoreInstrumentation()
    .AddMeter(forecastMeter.Name)
    .AddMeter("Microsoft.AspNetCore.Hosting")
    .AddMeter("Microsoft.AspNetCore.Server.Kestrel")
    .AddPrometheusExporter()
    .AddConsoleExporter());

Notice that we are adding some default metrics ("Microsoft.AspNetCore.Hosting" and "Microsoft.AspNetCore.Server.Kestrel"), besides our own custom metric ("forecast_meter"). The full list of out-of-the-box metrics is available here. With this, whenever we make changes to one of its meters - the counter and the gauge, in this example - they will be propagated transparently to the OpenTelemetry server.

It's the AddAspNetCoreInstrumentation method that actually makes all the magic occur for ASP.NET Core, without it, the metrics wouldn't be sent. There are many other libraries that offer other instrumentation capabilities:

Nuget Package Target
OpenTelemetry.Instrumentation.AspNetCore ASP.NET Core requests
OpenTelemetry.Instrumentation.Http HttpClient requests
OpenTelemetry.Instrumentation.SqlClient SQL Server
OpenTelemetry.Instrumentation.GrpcNetClient GRPC client
OpenTelemetry.Instrumentation.Process .NET processes
OpenTelemetry.Instrumentation.StackExchangeRedis Redis
OpenTelemetry.Instrumentation.EntityFrameworkCore Entity Framework Core
OpenTelemetry.Instrumentation.Quartz Quartz.NET
OpenTelemetry.Instrumentation.Hangfire Hangfire
OpenTelemetry.Instrumentation.EventCounters .NET event counters
OpenTelemetry.Instrumentation.Wcf WCF
OpenTelemetry.Instrumentation.ElasticsearchClient ElasticsearchClient
OpenTelemetry.Instrumentation.AWS AWS
OpenTelemetry.Instrumentation.AWSLambda AWS Lambda functions
OpenTelemetry.Instrumentation.GrpcCore GRPC core

Each of these packages requires its own setup, and will send metrics that are particular to its target.

We also adding the Prometheus exporter, which we will discuss in a moment, the other exporter is the console, which just means any metrics will be output to the console, besides being sent to OpenTelemetry.

For the Prometheus exporter, we also need to add some middleware to our pipeline, after it is built:

app.MapPrometheusScrapingEndpoint(
    path: "/metrics"
);

The path parameter is optional, and it defaults to "/metrics". The is the relative path, from your web application, where the Prometheus metrics will be exposed.

Tracing

Let's add tracing too:

telemetry.WithTracing(options => options
    .AddAspNetCoreInstrumentation()
    .AddSource(forecastActivitySource.Name)
    .AddConsoleExporter());

We added explicitly our "Forecast.ActivitySource" activity source, without this, any activities created from it would not be propagated.

Adding the console exporter means that all traces will be sent to the console.

After all the services are registered, we need to tell OpenTelemetry to actually use the OpenTelemetry (OTLP) protocol exporter:

telemetry.UseOtlpExporter(
    protocol: OtlpExportProtocol.HttpProtobuf,
    baseUrl: new("http://localhost:4318/")
);

The protocol and baseUrl parameters are optional and default, respectively, to OtlpExportProtocol.Grpc and "http://localhost:4317/". For OtlpExportProtocol.HttpProtobuf, the URL should be set to "http://localhost:4318/". This can be done on a service-by-service basis (metrics, logs, traces) or globally for all, as I just did.

We're now ready to go!

Applying OpenTelemetry

As an example, let's suppose we have an API controller such as WeatherForecastController, from the standard ASP.NET Core template. In it we have a Get method that returns dummy weather information (note: for a working example about returning actual weather information, please read this).

In this method, we'll inject the Instrumentation class that we registered a while back:

[HttpGet(Name = "GetWeatherForecast")]
public IEnumerable<WeatherForecast> Get([FromServices] Instrumentation instrumentation)
{
    //...
}

And inside of it, we'll create a new activity to track the usage of the method:

using var forecastActivity = instrumentation.ForecastActivitySource.StartActivity("GetWeatherForecast");

And after get the weather forecast dummy samples, we can add some tags to it, like the current temperature and summary:

forecastActivity.SetTag("Temperature", forecast[0].TemperatureC);
forecastActivity.SetTag("Summary", forecast[0].Summary);

Also, we add values to our metrics:

instrumentation.ForecastCounter.Add(1);
instrumentation.ForecastTemperature.Record(forecast[0].TemperatureC);

And, just so that we have a complete example, let's log the current values:

_logger.LogInformation("Date: {Date}, Summary: {Summary}, Temperature: {Temperature}", forecast[0].Date, forecast[0].Summary, forecast[0].TemperatureC);

Now, you may wonder why I'm not using string interpolation on the call to LogInformation, see here why that is.

If the OpenTelemetry is successfully set up, the server will start receiving our logs, metrics, and traces!

Monitoring OpenTelemetry

Now, it's time to watch it in action! Navigate to the weather forecast endpoint and make a few calls, so that we have data to watch.

Watching Logs

By navigating to http://localhost:3000/a/grafana-lokiexplore-app/explore, or choosing Explore - Logs on the left menu, we get this:

The logs visualiser

Expanding one of the log entries we get:

A log entry

You can see here the individual values (fields) for the log entry, the ones we didn't use interpolation for - each of the fields in the log call. You can also filter by time interval or by any of the fields that exist in any log entry, and, of course, by service name - remember that you can collect logs for many services.

Watching Metrics

Metrics are available from Explore - Metrics, or by navigating to http://localhost:3000/explore/metrics/trail:

The metrics

Again, you can filter by time interval or by any of the imported metrics. Remember, you will only see the metrics that you explicitly configured.

Prometheus

Remember when we talked about the Prometheus exporter? Well, if you enable it and go to http://localhost:3000/metrics, you can see a dump of all the metrics as they are being received:

The Prometheus exporter

Watching Traces

You can watch the traces on the main screen (Explore), in http://localhost:3000/explore. Select Tempo as the data source:

The traces

We can always filter or expand a trace and see its children and attributes, on the right panel:

Trace attributes
Also remember that you will only see traces from sources you explicitly configured.

Conclusion

As you can see, telemetry is a fascinating subject. The ability to have, in a centralised location, all of the logs, metrics, and traces, of many applications is very powerful and incredibly useful. Long gone are the days of looking at individual log files. This is based on standards and can be used in many different languages and APIs. I hope you find this post useful, looking forward for your comments!

Location: Coimbra, Portugal

Comments

Post a Comment

Popular posts from this blog

ASP.NET Core Middleware

Image
Introduction This post is another of those "back to basics", still, there was one thing or two that I feel were not properly discussed elsewhere, hence this post. The ASP.NET Core Pipeline and Middleware You may know that ASP.NET Core defines a pipeline. This is where the incoming requests are processed and the response submitted. Each component in the pipeline - a middleware - can do things with the message (the request), such as check if it is authenticated and have rights to access the requested resource, turn the request into an MVC-style controller and action invocation, log it, and what not. You can read more about it here . The ASP.NET Core pipeline, from Microsoft The order by which the middleware is added to the pipeline matters. For example, the authorisation middleware must come after the authentication one, static files must come before routing, and exception handling must come before everything. The middleware order, from Microsoft A middleware is just a piece of...
Read more

.NET Cancellation Tokens

Introduction Cancellation tokens in .NET were introduced as part of the  .NET 4 Cancellation Framework . Essentially, it is a standard, cooperative, way to request for the cancellation of the execution of an operation. By cooperative it means that the code that uses it must abide to some rules, as opposed to the cancellation just stopping (killing) the execution. One example of cancellation is, on a web app, when the client closes the connection while the server is performing some long operation. In cases such as this, we may want to abort the operation, as we are not sending the results anywhere (or not, as we shall see). Because of this, ASP.NET Core allows us to add a parameter of type  CancellationToken  to our asynchronous actions, it is automatically associated with the client and is therefore signalled when the client connection is closed. A  CancellationToken  is, essentially, behind the scenes, a ManualResetEvent . Throughout this article I will either ...
Read more

Audit Trails in EF Core

Introduction Audit trails is a common feature in database systems. Essentially, it's about knowing, for every table (or some selected ones), who created/updated each record, and when the creates/updates happened. It can also be useful to store the previous values too. Some regulations, such as SOX , require this for compliance. There are some libraries that can do this automatically in EF Core , and there's also the Change Data Capture in SQL Server (and possibly other databases too), of which I wrote about some time ago, but, hey, I wanted to write my own, so here is EFAuditable , my library for doing audit trails in EF Core. This one is designed with some extensibility in mind, so, please, read on. Scenarios EFAuditable  allows the following scenarios: Entities marked for auditing will have the audit columns added and persisted automatically to the database, on the same table Entities marked for history keeping will have their old values (in the case of updates or deletes) ...
Read more