.NET Metrics

Introduction

I recently posted about OpenTelemetry, and I mentioned metrics there, as it's an important part of observability. On this post I will dig a bit more about metrics, what they are and how they are used by .NET and ASP.NET Core. Very important, these definitions match the observability/OpenTelemetry standard, specifically, the Metrics API, and are designed to work together.

Creating Metrics

There are different kinds of metrics, and they are all usually created from the Meter class, part of the .NET System.Diagnostics.Metrics API.

A Meter has a name (Name), an optional scope (Scope), an optional version (Version), and some optional tags (Tags), all unchangeable and initialised at constructor time, except tags. Besides this, it has build methods for all supported meters/instrument types. It can be constructed directly:

var meter = new Meter("Some.Meter", version: "1.0.0");

The Meter class is disposable, so make sure you dispose of it when no longer needed.

Meters

The base class for all meters, or instruments, is Instrument, which has the base properties name (Name), unit (Unit), description (Description), tags (Tags), originating meter (Meter), whether or not it is observable (IsObservable) and enabled (Enabled), and then Instrument<T> inherits from it and adds the concept of a typed value. All meters are strongly typed and inherit from Instrument<T>, so they all require a generic type parameter. All of their properties can only be set at constructor time, except for the tags.

The scope is an optional logical unit of the application code with which the emitted telemetry can be associated. It is typically the developer’s choice to decide what denotes a reasonable instrumentation scope. See this.

Tags are sent with each metric, they are free-form string-object pairs. Metrics collectors store them.

If a metric is not enabled, it will not send its value.

Unit, description, version, are all optional and are not used in any way.

Important to know that the current value or values of a meter cannot be read at random, only when one is set, is it propagated. The value - the generic template parameter - has to be a value type, typically, an integral numeric or a floating point number.

The meter names should follow a "A.B" structure, sometimes, reverse domain is used, but have a look at the guidelines.

There are four kinds of instruments, which will be explained now. The version, unit, description, and tags parameters shown in the following examples all map to the corresponding same-name properties of Instrument<T> and are all optional. The name should be snake-cased and is the only required parameter.

Counter

A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart. For example, you can use a counter to represent the number of requests served, tasks completed, or errors. A counter starts at 0.

Do not use a counter to expose a value that can decrease. For example, do not use a counter for the number of currently running processes; instead use a gauge or an up down, read on.

In .NET, a counter is represented by the Counter<T> class, which only offers of interest an Add method.

To create a counter:

var counter = meter.CreateCounter<int>("test_counter", version: "1.0.0");

Gauge

A gauge is a metric that represents a single numerical value that can be set arbitrarily.

Gauges are typically used for measured values like temperature or current memory usage, but also values that can go up and down, like the number of concurrent requests.

In .NET we have the Gauge<T> to represent gauges, and the method to set the current value is Record.

To create a gauge we do this:

var gauge = meter.CreateGauge<int>("test_gauge", unit: "X");

Histogram

A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets (X for bucket 5-10, Y for bucket 25-50, Z for bucket > 100, etc). It also provides a sum of all observed values.

The Histogram<T> is used in .NET to represent an histogram. The method to set the current value is Record too. Histograms stores values in buckets, the default bucket configuration is: [ 0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000 ]. Obviously, you can change that.

To create an histogram:

var histogram = meter.CreateHistogram<int>("test_histogram", description: "Some histogram");

If you want to customise the buckets, you can pass on an advice (InstrumentAdvice<T>):

var histogram = meter.CreateHistogram<int>("test_histogram", description: "Some histogram", advice: new InstrumentAdvice<int> { HistogramBucketBoundaries = [1, 5, 10, 50, 100] });

An advice is just an instance of InstrumentAdvice<T> which contains additional information for collectors, it is actually only used by histograms, for the distribution. It's the HistogramBucketBoundaries property. Histograms are slightly more complex, and you can read more from here.

Up Down

Similar to counter, but this instrument tracks a value that may increase or decrease over time. The difference to gauges is that up down metrics can only get down to 0, while gauges don't have this limitation.

The class to use in .NET is UpDownCounter<T>. Like with Counter<T>, the method to increment or decrement is called Add, but, unlike Counter<T>, you can send negative numbers.

We create one like this:

var updown = meter.CreateUpDown<int>("test_updown", tags: new TagList(new KeyValuePair<string, object>("foo", "bar")));

Observable Metrics

Some metrics - counter, gauge, up down - have an observable version. The name observable is a bit misleading, what it really means is that the value comes from the outside, and thus can be changed outside the scope of the meter. The classes are ObservableCounter<T>, ObservableGauge<T>, and ObservableUpDownCounter<T>. All of them are usually built from a Meter instance, and receive as a parameter to its build method the source of their data. For observable instruments, their IsObservable property will return true.

For example, for an ObservableCounter<T>, we can have this:

var count = 0;

var obsCounter = meter.CreateObservableCounter<int>("test_obs_counter", () => count);

Now each time we make changes to the underlying variable, the observable counter gets notified (not instantly, mind you). Don't forget that if we want to be notified about observable metrics changes we need to enable it on the listener (more on this in a moment).

Meter Factory

In ASP.NET Core, there's a meter factory instance registered by default: it's the IMeterFactory that has a default class DefaultMeterFactory (internal). From it we can create Meter instances:

using var meterFactory = app.Services.GetRequiredService<IMeterFactory>();

var meter = meterFactory.CreateMeter("Some.Meter", version: "1.0.0");

The advantage of using a meter factory is, because IMeterFactory is disposed of at application shutdown, so are any meters created off it, so we don't have to care.

Source Generated Metrics

Using a source generator, we can have a method for updating a named meter with a set of tags and a delta. Let's start by defining some types:

public class RequestTags

{

    [TagName(nameof(ActionName))]

    public string ActionName { get; set; }

    [TagName(nameof(ControllerName))]

    public string ControllerName { get; set; }

}

public partial static class MeterExtensions

{

    [Counter<int>(typeof(RequestTags), Name = "request_count")]

    public static partial RequestCount GetRequestCounter(this Meter meter);

}

The MeterExtension.GetRequestCounter (you can give it any name) is declared as partial, inside an also partial class, and because it has a [Counter], [Counter<T>], [Gauge], [Histogram], or [Histogram<T>] attribute applied to it, it means that the body will be generated for us by .NET. The actual code depends on the attribute used. The RequestCount return type is also generated, don't worry about it, it's a simple class that essentially offers an Add method, similar to the Add from the Counter<T> and UpDownCounter<T> classes, or a Record one, as the one for Gauge<T> and Histogram<T>, of course, depending on the attribute being applied. The RequestTags class will merely hold tags as properties:

var requestCount = meter.GetRequestCounter();

requestCount.Add(1, new RequestTags { ControllerName = "Home", ActionName = "Index" });

The [Counter<T>] constructor has two overloads: one that takes a type (the one I've shown), with possibly some [TagName] attributes applied to its properties, and another which just takes a list of tags. Depending on which overload you use, you get different generated GetRequestCounter methods. So, you could also have this:

[Counter<int>("ControllerName", "ActionName", Name = "request_count")]

public static partial RequestCount GetRequestCounter(this Meter meter);

Results in this method instead (no RequestTags class):

var requestCount = meter.GetRequestCounter();

requestCount.Add(1, "Home", "Index");

Using this approach you can pass the RequestCount class around and you can set values to the meter conveniently.

Read more about this approach here.

.NET Metrics

.NET's APIs produces lots of metrics, you can find the full list of built-in metrics here. They include memory usage, threads, and other relevant information. Some metrics exposed by .NET, its extensions, or EF Core (in no means exhaustive) are:

Metric	Meter Type	Purpose
dotnet.process.cpu.time	counter	CPU time used by the process
dotnet.process.memory.working_set	up down	The number of bytes of physical memory mapped to the process context
dotnet.thread_pool.thread.count	up down	The number of thread pool threads that currently exist
dotnet.assembly.count	up down	The number of .NET assemblies that are currently loaded
dotnet.health_check.unhealthy_checks	counter	Number of times a health check reported the health status of an application as Degraded or Unhealthy
microsoft.entityframeworkcore.active_dbcontexts	observable up down	Number of currently active DbContext instances
microsoft.entityframeworkcore.savechanges	observable counter	Cumulative count of changes saved
microsoft.entityframeworkcore.optimistic_concurrency_failures	observable counter	Cumulative number of optimistic concurrency failures
http.client.open_connections	up down	Number of outbound HTTP connections that are currently active or idle on the client
http.client.request.duration	histogram	The duration of outbound HTTP requests

ASP.NET Core Metrics

ASP.NET Core too produces lots of metrics, the list is available here. These include number of requests, errors, request duration, etc. Some handy metrics include:

Metric	Meter Type	Purpose
http.server.request.duration	histogram	Measures the duration of inbound HTTP requests
http.server.active_requests	up down	Measures the number of concurrent HTTP requests that are currently in-flight
aspnetcore.diagnostics.exceptions	counter	Number of exceptions caught by exception handling middleware
aspnetcore.rate_limiting.queued_requests	up down	Number of requests that are currently queued waiting to acquire a rate limiting lease

Inside of an MVC action, we can add tags to the metrics that will be collected from that scope, through the IHttpMetricsTagsFeature ASP.NET Core request feature:

var feature = HttpContext.Features.Get<IHttpMetricsTagsFeature>();

feature.Tags.Add(new KeyValuePair<string, object>("Foo", "Bar"));

And also disable metrics collecting altogether:

feature.MetricsDisabled = true;

OpenTelemetry Collector

To gather metrics, we need to have a metrics collector. I introduced one on my previous post. We can also use the OpenTelemetry one, from OpenTelemetry.Instrumentation.AspNetCore Nuget package, and we set it up like this:

var meterProviderBuilder = OpenTelemetry.Sdk.CreateMeterProviderBuilder()

                .AddConsoleExporter()

                .AddAspNetCoreInstrumentation()

                .AddRuntimeInstrumentation()

                .AddProcessInstrumentation();

meterProviderBuilder.AddMeter(counter.Name);

using var meterProvider = meterProviderBuilder.Build();

This starts the collector and registers the counter with it, so that it can be monitored (AddMeter). It also registers the console exporter (AddConsoleExporter) and adds instrumentation for ASP.NET Core (AddAspNetCoreInstrumentation), for the .NET runtime (AddRuntimeInstrumentation) and for the current process (AddProcessInstrumentation). After this, metrics will start to show on the console:

Metrics being output to the console

Viewing Metrics

There are a few ways to monitor metrics as they are being produced, let's examine them here. I already showed the console exporter, but there are other ways.

Metrics Listeners

The easiest way to view metric's values as they are being set is by using a MeterListener. After you create it, you need to explicitly register the metrics you are interested in, and you can get a notification when a value is set on each of the registered metrics. An example:

using var meterListener = new MeterListener();

meterListener.EnableMeasurementEvents(counter);

meterListener.SetMeasurementEventCallback<int>((instrument, measurement, tags, state) =>
{
    //called when the instrument's value changes
});
meterListener.Start();

Notice that we are calling MeterListener.EnableMeasurementEvents to register interest for a particular metric, and also SetMeasurementEventCallback to be notified whenever a specific metric value changes. MeterListener must be explicitly started (Start) and should be disposed of when no longer needed.

It is also possible to inspect all instruments (metrics) as they are being created, so as to decide if we want to subscribe to them or not:

meterListener.InstrumentPublished = (instrument, listener) =>

{

    if (instrument.Name == "test_counter")

    {

        meterListener.EnableMeasurementEvents(instrument);

    }

});

By default MeterListener does not record values for observable metrics, so we need to call RecordObservableInstruments to explicitly enable it:

meterListener.RecordObservableInstruments();

Using dotnet-counters

The "old" way to receive notifications for metrics is using the dotnet-counters tool. This is a command line tool that can show all metrics being produced by a specific .NET process. Make sure you install it first:

dotnet tool install dotnet-counters -g:

Installing dotnet-counters tool globally

We then need to find out what processes are there that we can hook to: 

dotnet-counters ps

.NET process list

The process we're interested in has id 7068. Now, we select a single process, with:

dotnet-counters monitor --process-id 7068

We get the metrics in real-time:

.NET metrics for a process

Another option is to collect metrics for a period of time and export them to a CSV file (counter.csv), for that, we use:

dotnet-counters collect --process-id 7068

Metrics dump for a .NET process

So, we have two options:

• Monitoring in real-time (dotnet-counters monitor)
• Collecting metrics over a period of time, or until the process shuts down (dotnet-counters collect)

Choose the one that best serves you.

Using Prometheus

Prometheus is a free and open-source metrics collector. It can be used to collect metrics from many different applications and servers and expose them over HTTP.

We add the Prometheus exporter for the OpenTelemetry metrics provider through the AddPrometheusHttpListener:

var meterProviderBuilder = OpenTelemetry.Sdk.CreateMeterProviderBuilder()

                .AddConsoleExporter()

                .AddPrometheusHttpListener()

                .AddAspNetCoreInstrumentation()

                .AddRuntimeInstrumentation()

                .AddProcessInstrumentation();

You can tweak the port, but, now, you can head to http://localhost:9464/metrics and observe the metrics in real-time:

The Prometheus output

Putting It All Together

A full example:

using var meterListener = new MeterListener();

meterListener.InstrumentPublished = (instrument, listener) =>

{

    //decide whether or not to subscribe to this instrument...

};

meterListener.SetMeasurementEventCallback<int>((instrument, measurement, tags, state) =>

{

    //called when an instrument's value changes

});

meterListener.RecordObservableInstruments();

meterListener.Start();

//uncomment the next two lines if using ASP.NET Core

//var meterFactory = builder.Services.GetRequiredService<IMeterFactory>();

//meter = meterFactory.Create(new MeterOptions("Test.Meter") { Version = "1.0.0" });

using var meter = new Meter("Test.Meter", "1.0.0");

var counter = meter.CreateCounter<int>("test_counter");

meterListener.EnableMeasurementEvents(counter);

var counterValue = 0;

var observableCounter = meter.CreateObservableCounter<int>("test_obs_counter", () => counterValue);

meterListener.EnableMeasurementEvents(observableCounter);

//increment counters

counter.Add(1);

counterValue++;

Conclusion

Metrics are an important concept to understand what is happening in our applications without debugging them, and .NET offers good support for this. Hope you found this useful, and let me know your thoughts!