Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OpenTelemetry Sample #78

Merged
merged 7 commits into from
Sep 3, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -19,20 +19,20 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Temporalio" Version="1.1.2" />
<PackageReference Include="Temporalio.Extensions.DiagnosticSource" Version="1.1.2" />
<PackageReference Include="Temporalio.Extensions.Hosting" Version="1.1.2" />
<PackageReference Include="Temporalio.Extensions.OpenTelemetry" Version="1.1.2" />
<PackageReference Include="Temporalio" Version="1.2.0" />
<PackageReference Include="Temporalio.Extensions.DiagnosticSource" Version="1.2.0" />
<PackageReference Include="Temporalio.Extensions.Hosting" Version="1.2.0" />
<PackageReference Include="Temporalio.Extensions.OpenTelemetry" Version="1.2.0" />
<!--
Can also reference the SDK downloaded to a local directory:
<ProjectReference Include="$(MSBuildThisFileDirectory)..\temporal-sdk-dotnet\src\Temporalio\Temporalio.csproj" />
-->
</ItemGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Logging" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="7.0.0" />
<PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="17.4.33" PrivateAssets="all" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="8.0.0" />
robcao marked this conversation as resolved.
Show resolved Hide resolved
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="8.0.0" />
<PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="17.10.48" PrivateAssets="all" />
<PackageReference Include="StyleCop.Analyzers" Version="1.2.0-beta.435" PrivateAssets="all" />
</ItemGroup>
</Project>
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Prerequisites:
* [Encryption](src/Encryption) - End-to-end encryption with Temporal payload codecs.
* [Mutex](src/Mutex) - How to implement a mutex as a workflow. Demonstrates how to avoid race conditions or parallel mutually exclusive operations on the same resource.
* [Polling](src/Polling) - Recommended implementation of an activity that needs to periodically poll an external resource waiting its successful completion.
* [OpenTelemetry](src/OpenTelemetry) - Demonstrates how to set up OpenTelemetry tracing and metrics for both the client and worker.
robcao marked this conversation as resolved.
Show resolved Hide resolved
* [Saga](src/Saga) - Demonstrates how to implement a saga pattern.
* [Schedules](src/Schedules) - How to schedule workflows to be run at specific times in the future.
* [SignalsQueries](src/SignalsQueries) - A loyalty program using Signals and Queries.
Expand Down
15 changes: 13 additions & 2 deletions TemporalioSamples.sln
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,11 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TemporalioSamples.Mutex", "
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TemporalioSamples.Saga", "src\Saga\TemporalioSamples.Saga.csproj", "{B79F07F7-3429-4C58-84C3-08587F748B2D}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TemporalioSamples.WorkflowUpdate", "src\WorkflowUpdate\TemporalioSamples.WorkflowUpdate.csproj", "{B3DB7B8C-7BD3-4A53-A809-AB6279B1A630}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TemporalioSamples.WorkflowUpdate", "src\WorkflowUpdate\TemporalioSamples.WorkflowUpdate.csproj", "{B3DB7B8C-7BD3-4A53-A809-AB6279B1A630}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TemporalioSamples.ContextPropagation", "src\ContextPropagation\TemporalioSamples.ContextPropagation.csproj", "{7B797D20-485F-441D-8E71-AF7E315FA9CF}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TemporalioSamples.ContextPropagation", "src\ContextPropagation\TemporalioSamples.ContextPropagation.csproj", "{7B797D20-485F-441D-8E71-AF7E315FA9CF}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TemporalioSamples.OpenTelemetry", "src\OpenTelemetry\TemporalioSamples.OpenTelemetry.csproj", "{7F8C536E-7664-4A19-A153-C86209EC0BDF}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Expand Down Expand Up @@ -149,6 +151,14 @@ Global
{7B797D20-485F-441D-8E71-AF7E315FA9CF}.Debug|Any CPU.Build.0 = Debug|Any CPU
{7B797D20-485F-441D-8E71-AF7E315FA9CF}.Release|Any CPU.ActiveCfg = Release|Any CPU
{7B797D20-485F-441D-8E71-AF7E315FA9CF}.Release|Any CPU.Build.0 = Release|Any CPU
{7F8C536E-7664-4A19-A153-C86209EC0BDF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{7F8C536E-7664-4A19-A153-C86209EC0BDF}.Debug|Any CPU.Build.0 = Debug|Any CPU
{7F8C536E-7664-4A19-A153-C86209EC0BDF}.Release|Any CPU.ActiveCfg = Release|Any CPU
{7F8C536E-7664-4A19-A153-C86209EC0BDF}.Release|Any CPU.Build.0 = Release|Any CPU
{CDFAD7B0-FC43-4ECD-9E36-1023322294C9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{CDFAD7B0-FC43-4ECD-9E36-1023322294C9}.Debug|Any CPU.Build.0 = Debug|Any CPU
{CDFAD7B0-FC43-4ECD-9E36-1023322294C9}.Release|Any CPU.ActiveCfg = Release|Any CPU
{CDFAD7B0-FC43-4ECD-9E36-1023322294C9}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down Expand Up @@ -178,5 +188,6 @@ Global
{B79F07F7-3429-4C58-84C3-08587F748B2D} = {1A647B41-53D0-4638-AE5A-6630BAAE45FC}
{B3DB7B8C-7BD3-4A53-A809-AB6279B1A630} = {1A647B41-53D0-4638-AE5A-6630BAAE45FC}
{7B797D20-485F-441D-8E71-AF7E315FA9CF} = {1A647B41-53D0-4638-AE5A-6630BAAE45FC}
{7F8C536E-7664-4A19-A153-C86209EC0BDF} = {1A647B41-53D0-4638-AE5A-6630BAAE45FC}
EndGlobalSection
EndGlobal
14 changes: 14 additions & 0 deletions src/OpenTelemetry/Activities.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
using Microsoft.Extensions.Logging;
using Temporalio.Activities;

namespace TemporalioSamples.OpenTelemetry
robcao marked this conversation as resolved.
Show resolved Hide resolved
{
public static class Activities
{
[Activity]
public static void MyActivity(string input)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd be valuable here and in workflow to show a custom metric. E.g. ActivityExecutionContext.Current.MetricMeter.CreateCounter<int>("my-counter").Add(123) and Workflow.MetricMeter.CreateCounter<int>("my-counter").Add(123). Technically you could instantiate the activity class with the .NET mertic meter, but this way respects the existing abstraction (so you don't have to have custom meter), and for workflows you can't inject anything and it's a replay-safe metric meter (i.e. it doesn't record on replay).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

{
ActivityExecutionContext.Current.Logger.LogInformation("Executing activity for OpenTelemetry sample.");
}
}
}
23 changes: 23 additions & 0 deletions src/OpenTelemetry/MyWorkflow.workflow.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
namespace TemporalioSamples.OpenTelemetry;

using Microsoft.Extensions.Logging;
using Temporalio.Workflows;

[Workflow]
public class MyWorkflow
{
[WorkflowRun]
public async Task<string> RunAsync()
{
Workflow.Logger.LogInformation("Running workflow {WorkflowId}.", Temporalio.Workflows.Workflow.Info.WorkflowId);
robcao marked this conversation as resolved.
Show resolved Hide resolved

await Workflow.ExecuteActivityAsync(
() => Activities.MyActivity("input"),
new()
{
StartToCloseTimeout = TimeSpan.FromMinutes(5),
});

return "complete!";
}
}
104 changes: 104 additions & 0 deletions src/OpenTelemetry/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
using System.Diagnostics.Metrics;
using System.Reflection;
using Microsoft.Extensions.Logging;
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Temporalio.Client;
using Temporalio.Extensions.DiagnosticSource;
using Temporalio.Extensions.OpenTelemetry;
using Temporalio.Runtime;
using Temporalio.Worker;
using TemporalioSamples.OpenTelemetry;

AssemblyName assemblyName = typeof(TemporalClient).Assembly.GetName();
robcao marked this conversation as resolved.
Show resolved Hide resolved

using var meter = new Meter(assemblyName.Name!, assemblyName.Version!.ToString());

string instanceId = args.ElementAtOrDefault(0) ?? throw new ArgumentException("Must pass 'worker' or 'workflow' as the single argument");

ResourceBuilder resourceBuilder = ResourceBuilder.
CreateDefault()
.AddService("TemporalioSamples.OpenTelemetry", serviceInstanceId: instanceId);

using TracerProvider tracerProvider = Sdk.
CreateTracerProviderBuilder().
SetResourceBuilder(resourceBuilder).
AddSource(TracingInterceptor.ClientSource.Name, TracingInterceptor.WorkflowsSource.Name, TracingInterceptor.ActivitiesSource.Name).
AddOtlpExporter().
Build();

using MeterProvider meterProvider = Sdk.
CreateMeterProviderBuilder().
SetResourceBuilder(resourceBuilder).
AddMeter(assemblyName.Name!).
AddOtlpExporter().
Build();

// Create a client to localhost on default namespace
var client = await TemporalClient.ConnectAsync(new("localhost:7233")
{
LoggerFactory = LoggerFactory.Create(builder =>
builder.
AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] ").
SetMinimumLevel(LogLevel.Information)),
Interceptors = new[] { new TracingInterceptor() },
Runtime = new TemporalRuntime(new TemporalRuntimeOptions()
{
Telemetry = new TelemetryOptions()
{
Metrics = new MetricsOptions()
{
CustomMetricMeter = new CustomMetricMeter(meter),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrmm, so there are two ways to configure OTel telemetry metrics. One way is this, but the other way is simply to set the OTel options in this metric options so our internal code forwards. We often encourage the latter because it's less setup and it matches other SDKs.

I wonder if there is some way we can show both? Like maybe switched from one to the other via CLI flag?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I split this into two samples, OpenTelemetry/CoreSdkFowarding and OpenTelemetry/DotNetMetrics.

I thought this was cleanest, and follows a similar pattern that exists with the polling samples

Copy link
Member

@cretz cretz Aug 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will look. Alternatively we probably could have just had a different parameter accepted in the Program.cs to choose one approach or the other instead of completely separate projects.

},
},
}),
});

async Task RunWorkerAsync()
{
// Cancellation token cancelled on ctrl+c
using var tokenSource = new CancellationTokenSource();
Console.CancelKeyPress += (_, eventArgs) =>
{
tokenSource.Cancel();
eventArgs.Cancel = true;
};

// Run worker until cancelled
Console.WriteLine("Running worker");
using var worker = new TemporalWorker(
client,
new TemporalWorkerOptions(taskQueue: "opentelemetry-sample").
AddWorkflow<MyWorkflow>().
AddActivity(Activities.MyActivity));
try
{
await worker.ExecuteAsync(tokenSource.Token);
}
catch (OperationCanceledException)
{
Console.WriteLine("Worker cancelled");
}
}

async Task ExecuteWorkflowAsync()
{
Console.WriteLine("Executing workflow");
await client.ExecuteWorkflowAsync(
(MyWorkflow wf) => wf.RunAsync(),
new(id: "opentelemetry-sample-workflow-id", taskQueue: "opentelemetry-sample"));
}

switch (args.ElementAtOrDefault(0))
{
case "worker":
await RunWorkerAsync();
break;
case "workflow":
await ExecuteWorkflowAsync();
break;
default:
throw new ArgumentException("Must pass 'worker' or 'workflow' as the single argument");
}
12 changes: 12 additions & 0 deletions src/OpenTelemetry/Properties/launchSettings.json
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file needed? We don't usually add to the rest of our samples

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed, removed.

launchSettings are a convenience that popular .NET IDEs such as Rider, VS, and VSCode support that allow developers to pre-configure executables launched directly in the IDE with command line arguments, environment variables, etc

Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"profiles": {
"Worker": {
"commandName": "Project",
"commandLineArgs": "worker"
},
"Workflow": {
"commandName": "Project",
"commandLineArgs": "workflow"
}
}
}
41 changes: 41 additions & 0 deletions src/OpenTelemetry/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# OpenTelemetry

This sample shows how to configure OpenTelemetry to capture workflow traces and SDK metrics.

To run, first see [README.md](../../README.md) for prerequisites.

Then, run the following from this directory to start the .NET Aspire Dashboard which will collect telemetry. The dashboard UI is available at http://localhost:18888.

docker compose up

Then, run the following from this directory in a separate terminal to start the worker:

dotnet run worker

Then in another terminal, run the workflow from this directory:

dotnet run workflow

The workflow will complete.

## Traces

Traces can be viewed at http://localhost:18888/traces.

You can select either `worker` or `workflow` for traces; both should show the same trace. The workflow should appear and when clicked, may look something like:

![Tracing Screenshot](tracing-screenshot.png)

## Metrics

Metrics can be viewed by clicking the metrics tab on the dashboard.

Similar to traces, you can select either `worker` or `workflow`.

`worker` will show the metrics emitted by the worker. It may look something like:

![Worker Metrics Screenshot](worker-metrics-screenshot.png)

`workflow` will show the metrics emitted by the client. It may look something like:

![Client Metrics Screenshot](client-metrics-screenshot.png)
14 changes: 14 additions & 0 deletions src/OpenTelemetry/TemporalioSamples.OpenTelemetry.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.0" />
</ItemGroup>

</Project>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions src/OpenTelemetry/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
services:
aspire-dashboard:
environment:
Dashboard__Frontend__AuthMode: Unsecured
image: mcr.microsoft.com/dotnet/aspire-dashboard:8.0
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did see that the python samples are using the Jaeger image and not the Aspire Dashboard, which is developed by Microsoft: https://github.com/temporalio/samples-python/tree/main/open_telemetry

I can change to the Jaeger image if desired.

The main benefit of using the Aspire dashboard over Jaeger is that it provides visualizations for metrics, which iirc Jaeger does not. I think using a Microsoft image is probably okay specifically for .NET

Copy link
Member

@cretz cretz Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind this, I like the metrics, so I think this image is great. Alternatively, you can get rid of docker compose and just tell people how to start the this docker container directly in the README, but this is fine too.

Copy link
Contributor Author

@robcao robcao Jul 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo, docker compose is easier with a single command, since environment variables and port forwarding are involved

ports:
- 4317:18889
- 18888:18888
Binary file added src/OpenTelemetry/tracing-screenshot.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/OpenTelemetry/worker-metrics-screenshot.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading