Monitoring Airbyte

Cloud Self-Managed Community (OSS)Self-Managed Enterprise

Airbyte offers extensive logging capabilities.

Connection logging

All Airbyte instances include extensive logging for each connector. These logs give you detailed reports on each data sync. Learn more about browsing logs.

Datadog Integration

Airbyte uses Datadog to monitor Airbyte Cloud performance on a number of metrics important to your experience. This integration only works on legacy Docker deployments of Airbyte. We're working on an improved version for abctl and Kubernetes. This could become available later as an enterprise feature to help you monitor your own deployment. If you're an enterprise customer and Datadog integration is important to you, let us know.

OpenTelemetry metrics monitoring (Self-Managed Enterprise only)

Airbyte Self-Managed Enterprise generates a number of crucial metrics about syncs and volumes of data moved. You can configure Airbyte to send telemetry data to an OpenTelemetry collector endpoint so you can consume these metrics in your downstream monitoring tool of choice. Airbyte does not send traces and logs.

Airbyte sends specific metrics to provide you with health insight in the following areas.

• Resource provisioning: Monitor API requests and sync attempts to ensure your deployment has adequate resources

• Sync performance: Track sync duration and data volume moved to understand performance

• System health: Monitor sync status and completion rates to ensure system stability

Configure OpenTelemetry metrics

1. Deploy an OpenTelemetry collector if you don't already have one. See the OpenTelemetry documentation for help doing this. If you use Datadog as your monitoring tool, they have an excellent guide to set up a collector and exporter.

2. Update your values.yaml file to enable OpenTelemetry.

   global:
       edition: enterprise # This is an enterprise-only feature
       metrics:
           enabled: true
           otlp:
               enabled: true
               collectorEndpoint: "YOUR_ENDPOINT" # The OTel collector endpoint Airbyte sends metrics to. You configure this endpoint outside of Airbyte as part of your OTel deployment.

3. Redeploy Airbyte with the updated values.

Airbyte sends metrics to the collector you specified in your configuration.

Available metrics

The following metrics are available. They're published every minute.

Metric	Tag	Example Value
airbyte.syncs	connection_id	653a067e-cd0b-4cab-96b5-5e5cb03f159b
	workspace_id	bed3b473-1518-4461-a37f-730ea3d3a848
	job_id	23642492
	status	success, failed
	attempt_count	3
	version	1.5.0
	source_connector_id	82c7fb2d-7de1-4d4e-b12e-510b0d61e374
	destination_connector_id	3cb42982-755b-4644-9ed4-19651b53ebdd
airbyte.gb_moved	connection_id	653a067e-cd0b-4cab-96b5-5e5cb03f159b
	workspace_id	bed3b473-1518-4461-a37f-730ea3d3a848
	job_id	23642492
	source_connector_id	82c7fb2d-7de1-4d4e-b12e-510b0d61e374
	destination_connector_id	3cb42982-755b-4644-9ed4-19651b53ebdd
	version	1.5.0
airbyte.sync_duration	connection_id	653a067e-cd0b-4cab-96b5-5e5cb03f159b
	workspace_id	bed3b473-1518-4461-a37f-730ea3d3a848
	job_id	23642492
	source_connector_id	82c7fb2d-7de1-4d4e-b12e-510b0d61e374
	destination_connector_id	3cb42982-755b-4644-9ed4-19651b53ebdd
	version	1.5.0
airbyte.api_requests	workspace_id	bed3b473-1518-4461-a37f-730ea3d3a848
	endpoint	/v1/connections/sync
	status	200