Airflow¶

Airflow has a native OpenTelemetry integration for traces and metrics, which involves creating an exporter internally that sends data to the configured backend.

To configure Airflow to send data to Logfire, you'll need to:

Set the OTEL_EXPORTER_OTLP_HEADERS environment variable.
Configure the otel_* settings in the airflow.cfg file.

Warning

If your apache-airflow is older than 2.10.4, this section will not work for you.

In that case, go to the [Airflow with OpenTelemetry Collector] section.

export OTEL_EXPORTER_OTLP_HEADERS="Authorization=${LOGFIRE_TOKEN}"

Where ${LOGFIRE_TOKEN} is your Logfire write token.

airflow.cfg

[metrics]
otel_on = True
otel_host = logfire-api.pydantic.dev
otel_port = 443
otel_prefix = airflow
otel_interval_milliseconds = 30000  # The interval between exports, defaults to 60000
otel_ssl_active = True

[traces]
otel_on = True
otel_host = logfire-api.pydantic.dev
otel_port = 443
otel_prefix = airflow
otel_ssl_active = True
otel_task_log_event = True

For more details, check airflow traces and metrics documentation.

Airflow with OpenTelemetry Collector¶

If your apache-airflow is older than 2.10.4, it means that you'll not be able to set the OTEL_EXPORTER_OTLP_HEADERS environment variable.

Why can't I set the OTEL_EXPORTER_OTLP_HEADERS environment variable?

This was a bug that was fixed in the 2.10.4 version of apache-airflow.

The Logfire team fixed it in apache/airflow#44346.

In that case, you'll need to set up an OpenTelemetry Collector to send data to Logfire.

OpenTelemetry Collector¶

The OpenTelemetry Collector is a vendor-agnostic agent that can collect traces and metrics from your applications and send them to various backends.

In this case, we are interested in sending data to Logfire.

Note

Using a collector is an option when you are already sending data to a backend, but you want to migrate to Logfire.

Then you can configure the collector to send data to Logfire, as well as your current backend. This way you can compare the data and ensure that everything is working as expected. Cool, right?

You can check the OpenTelemetry Collector installation guide to set it up, but I'll help you with the configuration.

otel-collector-config.yaml

receivers:  # (1)!
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

exporters:  # (2)!
  debug:  # (3)!
  otlphttp:
    endpoint: https://logfire-api.pydantic.dev/
    compression: gzip
    headers:
      Authorization: "Bearer ${env:LOGFIRE_TOKEN}"  # (4)!

processors:
  batch:
    timeout: 1s
    send_batch_size: 32768

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"

service:  # (5)!
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlphttp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlphttp]

Define the receivers to collect data from your applications.

See more about it on the OpenTelemetry Collector Receiver section.
Define the exporters to send data to Logfire.

The otlphttp exporter is used to send data to Logfire.
The debug exporter is used to send data to the console, so you can see what's being sent.

This is useful for debugging purposes, but it can be removed in production.
Set the Authorization header to send data to Logfire.

The {env:LOGFIRE_TOKEN} will be replaced by the environment variable.
Define the service to configure the pipelines.

The traces pipeline is used to send trace data, and the metrics pipeline is used to send metrics data.

Airflow configuration¶

To configure Airflow to send data to the OpenTelemetry Collector, we'll need the following settings:

On your airflow.cfg file, add the following configuration:

airflow.cfg

[metrics]
otel_on = True
otel_host = localhost
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000  # The interval between exports, defaults to 60000
otel_ssl_active = False

[traces]
otel_on = True
otel_host = localhost
otel_port = 4318
otel_prefix = airflow
otel_ssl_active = False
otel_task_log_event = True