Skip to main content

OpenTelemetry v2 integration

View Markdown

Temporal's OpenTelemetry integration lets you understand the internal state of Temporal applications across Clients, Workflows, Activities, and Nexus Operations by instrumenting them with OpenTelemetry.

Temporal provides durable execution. OpenTelemetry is the vendor-neutral framework for generating and exporting telemetry to your backend.

The OpenTelemetry plugin is what connects the two. It propagates OpenTelemetry context across Temporal boundaries. It can also create spans and emit metrics for Temporal SDK operations.

All code snippets in this guide are taken from the OpenTelemetry v2 sample. Refer to the sample for complete code.

Install​

Add the OpenTelemetry v2 integration to your Go module:

go get go.temporal.io/sdk/contrib/opentelemetry-v2@latest

Also add the OpenTelemetry SDK packages and the exporter or metric reader your backend requires.

Set up the tracer provider​

A Tracer Provider is the factory for Tracers. Create Temporal's replay-safe Tracer Provider and install it as the OpenTelemetry global before you create the plugin or call Tracer:

opentelemetry-v2/setup.go

// ...
	provider := temporalotel.NewReplaySafeTracerProvider(
		// WithBatcher performs exporter I/O outside the Workflow goroutine.
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		)),
	)
	otel.SetTracerProvider(provider)

NewReplaySafeTracerProvider keeps span IDs stable across retries and replay when instrumenting Workflows. A standard OpenTelemetry Tracer Provider is not safe for creating spans in Workflows.

Your application owns the Tracer Provider for the life of the process. Shut it down before exit so remaining spans can flush through the trace exporter.

Set up the meter provider​

A Meter Provider is the factory for Meters. If you enable MetricsHandlerOptions, install a configured Meter Provider with otel.SetMeterProvider before you create the plugin, or pass a Meter via MetricsHandlerOptions.Meter. OpenTelemetry's default global Meter Provider is a no-op.

Add the plugin​

Pass the plugin to your Temporal Client when you create it. Workers made from that Client get the plugin automatically.

opentelemetry-v2/workflow-activity-propagation/worker/main.go

plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

c, err := client.Dial(client.Options{Plugins: []client.Plugin{plugin}})
if err != nil {
	return fmt.Errorf("unable to create client: %w", err)
}
defer c.Close()

By default the plugin only performs context propagation so Span Context can cross Temporal boundaries.

Add custom spans​

In Workflows​

A Tracer creates spans. In Workflows, use Tracer instead of otel.Tracer. It keeps span IDs and start times accurate across retries and replay. A standard OpenTelemetry Tracer is not safe for creating spans in Workflows.

As in OpenTelemetry Go, Start returns a context that contains the active span. Pass that workflow.Context to downstream Temporal calls so later spans nest under it as children:

opentelemetry-v2/workflow-activity-propagation/opentelemetry.go

// ...
func Workflow(ctx workflow.Context, name string) (string, error) {
	tracer := temporalotel.Tracer(instrumentationName)
	ctx, span := tracer.Start(ctx, "workflow-operation")
	defer span.End()

	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	})

	var result string
	if err := workflow.ExecuteActivity(ctx, Activity, name).Get(ctx, &result); err != nil {
		return "", err
	}

	return result, nil
}

Outside Workflows​

In Clients, Activities, and other non-Workflow code, use an ordinary OpenTelemetry Tracer:

opentelemetry-v2/workflow-activity-propagation/opentelemetry.go

// ...
func Activity(ctx context.Context, name string) (string, error) {
	_, span := otel.Tracer(instrumentationName).Start(ctx, "activity-operation")
	defer span.End()

	return fmt.Sprintf("Hello, %s!", name), nil
}

Enable automatic instrumentation​

Set options on PluginOptions to create spans and emit metrics for Temporal SDK operations:

opentelemetry-v2/automatic-instrumentation/worker/main.go

plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{
	TracerOptions: tracing.TracerOptions{
		AddTemporalSpans: true,
	},
	MetricsHandlerOptions: &temporalotel.MetricsHandlerOptions{
		UseMonotonicCounters: true,
	},
})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

AddTemporalSpans​

Set AddTemporalSpans to true to create spans for Temporal SDK operations across Clients, Workflows, Activities, and Nexus Operations.

MetricsHandlerOptions​

Set MetricsHandlerOptions to a non-nil value to emit Temporal SDK metrics through OpenTelemetry.

Configure context propagation​

Context propagation is how OpenTelemetry moves context across process boundaries: inject on the way out, extract on the way in.

The plugin propagates Span Context, which keeps spans linked into one trace, and baggage: optional key-value data that travels with the context. Do not put credentials, tokens, or personal data in baggage. The plugin serializes baggage into Temporal headers that can be persisted in Workflow Event History.

TextMapPropagator​

The plugin injects and extracts both with a TextMapPropagator. By default that propagator supports W3C Trace Context and W3C Baggage. Set PluginOptions.TextMapPropagator to override it.

HeaderKey​

Propagated values are stored in the Temporal header under _tracer-data. Set TracerOptions.HeaderKey to use a different key.

DisableBaggage​

Set DisableBaggage to true to stop propagating baggage.

AllowInvalidParentSpans​

Set AllowInvalidParentSpans to true to ignore errors when extracting Span Context from Temporal headers. Use this when migrating between tracing libraries while Workflows or Activities are still in progress.

Resources​