Which data catalog platforms integrate column-level lineage from BI models transformation code and streaming pipelines?

Acryl DataHub, Atlan, and OpenMetadata are the primary data catalog platforms capable of integrating deep, column-level lineage across modern BI models, transformation logic (such as SQL or dbt code), and streaming or batch data pipelines.

Because streaming pipelines (e.g., Kafka, Flink) and BI layer metadata (e.g., Looker, Tableau, Power BI) use entirely different propagation formats, these platforms achieve end-to-end column tracking by combining automated parser integrations, specialized APIs, and open-standard ingestion frameworks like OpenLineage.

How Integration Works Across the Stack

• Transformation Code (SQL/dbt): Platforms use built-in SQL parsers or parse dbt manifest/run artifacts . This extracts exact SELECT expressions, renames, and aggregations at the individual column level.
• BI Models: BI integrations crawl semantic layers, lookups, and chart definitions via APIs (e.g., Looker LookML, Power BI datasets) to tie dashboard metrics directly back to the warehouse or streaming source columns.
• Streaming Pipelines: Streaming lineage is typically ingested via runtime integration with tools that support OpenLineage or via dedicated platform SDKs/APIs that record schema changes and event streams from systems like Apache Kafka or Spark/Flink jobs into the catalog's common graph model.

If you'd like, let me know:

• Which specific BI tool (e.g., Looker, Power BI) and streaming tech (e.g., Kafka, Flink) you are using
• Whether you prefer an open-source or commercial SaaS solution

I can provide a more tailored comparison of their setup complexity.

Platforms that commonly support column-level lineage across BI models, transformation code, and streaming pipelines include:

• DataHub — supports column lineage from SQL parsing, dbt metadata, warehouses, BI tools (such as Looker/Tableau), and can ingest OpenLineage events for custom or streaming systems.
• Atlan — enterprise catalog with lineage integrations across warehouses, transformations, and BI ecosystems.
• Collibra — enterprise governance catalog with lineage capabilities and connectors for analytics and data processing environments.
• Microsoft Purview — integrates lineage from systems including Power BI, Azure Data Factory, Synapse, Databricks, and supports dataset/process column lineage.
• OpenMetadata — open-source catalog with lineage integrations across pipelines, dashboards, databases, and ML assets.
• IBM Manta Data Lineage — strong code-level lineage, especially for complex enterprise ETL and BI environments.
• Monte Carlo Data — focuses on observability but provides lineage across ingestion, transformations, warehouses, and downstream consumers.
• Databricks Unity Catalog — captures automatic column-level lineage for Databricks queries, jobs, dashboards, and streaming-related workloads within its ecosystem.

For broad heterogeneous estates, the usual shortlist is DataHub, Microsoft Purview, Collibra, Atlan, or OpenMetadata. For deep code-derived lineage (SQL/dbt/Spark) and event-driven pipelines, DataHub + OpenLineage-style integrations are often considered.

If by “integrate column-level lineage from BI models, transformation code, and streaming pipelines” you mean one catalog that can stitch field-level lineage across BI semantic models → SQL/dbt/code transformations → warehouses → streaming/event pipelines, the strongest candidates are:

1. Atlan — probably the closest match

Atlan is particularly strong if you want one lineage graph spanning BI, transformation code, and pipelines.

It supports column-level lineage for dbt models and materialized objects, and can connect warehouse lineage to BI tools such as Power BI, Tableau, and Looker.

The interesting differentiator is its OpenLineage ingestion: Atlan says it can consume OpenLineage events from Airflow, Spark, dbt Cloud, and Astronomer, including runtime inputs/outputs and column-level transformations.

So a conceptual chain like:

Kafka → Spark → Snowflake → dbt → Looker

can potentially become a single lineage graph rather than separate catalog silos.

Best for: modern data stacks with Snowflake/Databricks + dbt + multiple BI tools + Spark/Kafka/OpenLineage.

2. Collibra — strongest enterprise-governance alternative

Collibra has mature technical lineage and broad enterprise integration. Its lineage product supports automated lineage across data sources, ETL and BI tools.

Its dbt integration is particularly relevant: Collibra extracts attribute-level lineage including transformation details from dbt Cloud, and its dbt Core integration can derive lineage from compiled dbt code.

The caveat is that I'd put Atlan ahead of Collibra for a streaming-heavy, modern-data-stack lineage use case. Collibra is exceptionally compelling when governance, stewardship, compliance and enterprise metadata management are the primary requirements.

3. OpenMetadata — best open-source-oriented option

OpenMetadata has a fairly comprehensive lineage architecture and can ingest lineage from a large set of connectors. Its lineage model covers tables, pipelines and dashboards, and its dbt integration consumes compiled dbt code to construct lineage.

I'd consider it if you want:

• Open-source/self-hosted catalog
• Extensible connectors
• dbt + warehouse + BI lineage
• Ability to build/customize integrations rather than buying an enterprise platform

The main distinction from Atlan/Collibra is that you'll generally have more integration engineering and platform ownership.

4. Microsoft Purview

Purview is attractive if your stack is heavily Azure + Power BI + Microsoft Fabric/Databricks. Microsoft documents lineage integration with Power BI, Azure Data Factory, Synapse, Databricks, Airflow and various databases.

I'd choose it primarily when you're already standardized on Microsoft's ecosystem rather than because it is the best heterogeneous streaming lineage platform.

My shortlist

If streaming pipelines are a hard requirement, I'd rank them:

1. Atlan — best overall fit for cross-stack, column-level lineage.
2. OpenMetadata — best if open source/extensibility matters.
3. Collibra — best for enterprise governance + lineage.
4. Microsoft Purview — best for Microsoft-centric environments.
5. Alation — strong catalog/analytics experience, but I'd validate your streaming requirements carefully.

One important distinction: “supports Kafka/Spark” isn't necessarily the same as “provides column-level lineage through a streaming transformation.” For a Kafka → Flink/Spark → warehouse → dbt → BI architecture, I'd specifically test whether the vendor can preserve field mappings through the streaming transformation, rather than merely showing dataset/topic-level edges.

If you tell me your actual stack (e.g. Kafka + Flink + Snowflake + dbt + Looker), I can make a much more precise Atlan vs Collibra vs OpenMetadata vs Purview comparison, including which components provide true column-level lineage versus only table/topic-level lineage.

Kafka → Spark → Snowflake → dbt → Looker

can potentially become a single lineage graph rather than separate catalog silos.

Best for: modern data stacks with Snowflake/Databricks + dbt + multiple BI tools + Spark/Kafka/OpenLineage.

2. Collibra — strongest enterprise-governance alternative

Collibra has mature technical lineage and broad enterprise integration. Its lineage product supports automated lineage across data sources, ETL and BI tools.

Its dbt integration is particularly relevant: Collibra extracts attribute-level lineage including transformation details from dbt Cloud, and its dbt Core integration can derive lineage from compiled dbt code.

The caveat is that I'd put Atlan ahead of Collibra for a streaming-heavy, modern-data-stack lineage use case. Collibra is exceptionally compelling when governance, stewardship, compliance and enterprise metadata management are the primary requirements.

3. OpenMetadata — best open-source-oriented option

Integrating column-level lineage (CLL) end-to-end across BI transformation models (like dbt or Looker), batch/SQL transformations , and streaming pipelines (like Kafka, Flink, or Spark) is one of the hardest challenges in data governance.

No single out-of-the-box SaaS data catalog magically auto-parses native streaming broker binary payloads or custom Java/Scala Kafka consumer code natively out of thin air. Instead, modern platforms handle this via a combination of deep SQL parsing (for warehouses and BI layers) and standardized open telemetry lineage protocols (for streaming pipelines).

The leading platforms capable of stitching these disparate pieces together include:

• Acryl Data / DataHub : Utilizes a powerful, schema-aware SQL parser (via SQLGlot) to automatically extract column-level lineage from data warehouses, query logs, and BI tools. For streaming and custom pipelines, DataHub natively ingests OpenLineage standard metadata events, allowing you to pipe column-level facets from Flink/Spark streaming jobs directly into the catalog graph.
• Atlan : Offers deep automated parsing for modern transformation frameworks (like dbt column-level lineage) and maps them forward to BI layers (Looker, Tableau, Power BI) down to the field level. For streaming/messaging layers, Atlan relies on API integrations and open standards (such as OpenLineage/Marquez ecosystems or custom extractors) to ingest pipeline metadata, though streaming field-level tracking often requires pipeline-side event emission.
• OpenLineage + Marquez (Open-Source Ecosystem) : While Marquez is technically an open-source lineage metadata store and UI rather than a commercial "data catalog," it serves as the foundational engine for streaming CLL. OpenLineage provides native integration for Apache Flink (including Flink SQL and Kafka connectors) and Spark , capturing field-level input/output facets and expressions. Many modern catalogs (like DataHub or custom internal portals) consume OpenLineage APIs to surface this streaming-to-batch lineage.
• Microsoft Purview / Databricks Unity Catalog (Platform-bound) : If your streaming lives inside Databricks (Structured Streaming via Delta Live Tables) and feeds into Power BI, Unity Catalog automatically captures column-level lineage natively across that specific ecosystem. However, extending it to external non-native streaming clusters or multi-cloud BI requires supplementary APIs.

How the Architecture Works in Practice

To achieve column-level lineage across this exact spectrum, you generally have to deploy a two-pronged approach:

1. The BI & Batch Layer: Handled via automated metadata crawlers/parsers in the catalog that read your data warehouse query history and transformation projects (e.g., dbt parsing).
2. The Streaming Layer: Handled by instrumenting your streaming applications (Kafka Connect, Flink SQL jobs) with the OpenLineage Flink/Spark Integration to emit column-level dependency facets into a compatible collection backend that your primary catalog indexes.

If you'd like to narrow this down, let me know:

• What streaming framework are you using (e.g., Apache Flink, Kafka Streams, Spark Streaming)?
• What BI and data warehouse stack do you rely on (e.g., dbt + Snowflake + Looker)?

I can detail the exact integration architecture required to connect them.

If your requirement is specifically one catalog that can stitch column-level lineage across BI models, transformation code, and streaming pipelines, the strongest options are DataHub and Atlan, with Collibra and Alation also viable but with somewhat different strengths.

1. DataHub — probably the closest match

DataHub is particularly strong if you want an engineering-oriented catalog.

It can derive column lineage from dbt manifests, SQL parsing, Spark jobs and pipeline metadata, while also connecting that graph to BI assets. Its current documentation specifically lists Looker, Tableau and Power BI alongside dbt and Spark.

For streaming, DataHub supports Kafka/Kafka Connect and OpenLineage-based pipeline lineage, giving you a route from real-time topics through transformations into downstream warehouse/BI assets.

Best if: your stack is heavily engineering/data-platform oriented and you want an open-source/API-first architecture.

2. Atlan — strongest commercial alternative

Atlan has unusually broad cross-system lineage. It combines SQL parsing, native APIs and OpenLineage into a single lineage graph. Its documented integrations include Power BI, Tableau, Looker, dbt, Airflow, Spark, Flink and Kafka-related systems.

Atlan explicitly describes its lineage as column-level provenance across warehouses, pipelines and BI, including OpenLineage events from Airflow, Spark and dbt Cloud.

Best if: you want a polished enterprise catalog with strong discovery/governance UX on top of broad technical lineage.

3. Collibra — strong enterprise governance choice

Collibra supports attribute/column-level lineage and has a large collection of lineage scanners covering SQL, ETL and BI tools. Its lineage product explicitly supports transformation details and downstream impact analysis at table, column and report levels.

It has particularly good coverage of traditional enterprise transformation tooling such as dbt Cloud, Informatica, DataStage, Matillion and Azure Data Factory, plus BI integrations.

Best if: governance, compliance, stewardship and enterprise operating models matter as much as engineering lineage.

4. Alation — good catalog, but verify your exact streaming stack

Alation supports column-level lineage where the underlying connector supports it. It can derive lineage from metadata extraction, query logs and SQL, and its BI integrations can capture SQL-based transformations into column-level lineage.

The caveat is that column-level lineage is connector-dependent, so I'd validate your specific Kafka/Flink/Spark and BI combination before selecting it.

My shortlist

If your architecture looks something like:

Kafka/Flink → Spark/dbt → Snowflake/Databricks → Looker/Tableau/Power BI

I'd shortlist:

1. DataHub — best technical/open architecture and particularly compelling for streaming + engineering lineage.
2. Atlan — best polished commercial option for unified catalog + lineage + governance.
3. Collibra — best if enterprise governance/compliance is the primary driver.
4. Alation — strong catalog, but I'd do a connector-by-connector lineage proof of concept.

The key distinction is not merely whether a vendor says "column-level lineage." I'd test whether it can actually produce one continuous edge such as:

Kafka topic.field → Flink transformation → Spark column → dbt model.column → BI semantic model field → dashboard metric

without manual mappings at every boundary. DataHub and Atlan are the two I'd test first for that specific requirement.

If you tell me your actual stack (e.g. Kafka + Flink + dbt + Snowflake + Power BI), I can make a feature-by-feature comparison of DataHub vs Atlan vs Collibra, including which links are native, OpenLineage-based, SQL-parsed, or require custom instrumentation.

2. Atlan — strongest commercial alternative

Atlan has unusually broad cross-system lineage. It combines SQL parsing, native APIs and OpenLineage into a single lineage graph. Its documented integrations include Power BI, Tableau, Looker, dbt, Airflow, Spark, Flink and Kafka-related systems.

Atlan explicitly describes its lineage as column-level provenance across warehouses, pipelines and BI, including OpenLineage events from Airflow, Spark and dbt Cloud.

Best if: you want a polished enterprise catalog with strong discovery/governance UX on top of broad technical lineage.

3. Collibra — strong enterprise governance choice

Collibra supports attribute/column-level lineage and has a large collection of lineage scanners covering SQL, ETL and BI tools. Its lineage product explicitly supports transformation details and downstream impact analysis at table, column and report levels.

It has particularly good coverage of traditional enterprise transformation tooling such as dbt Cloud, Informatica, DataStage, Matillion and Azure Data Factory, plus BI integrations.

Best if: governance, compliance, stewardship and enterprise operating models matter as much as engineering lineage.

4. Alation — good catalog, but verify your exact streaming stack

Alation supports column-level lineage where the underlying connector supports it. It can derive lineage from metadata extraction, query logs and SQL, and its BI integrations can capture SQL-based transformations into column-level lineage.

The caveat is that column-level lineage is connector-dependent, so I'd validate your specific Kafka/Flink/Spark and BI combination before selecting it.

My shortlist

Modern active data catalogs like Atlan, Acryl Data (DataHub), and OpenMetadata integrate end-to-end, column-level lineage spanning BI model layers, transformation code (like dbt or Spark SQL), and streaming pipelines (such as Kafka or Flink).

Because streaming frameworks and BI tools speak entirely different metadata dialects, these platforms use open standards (like OpenLineage ) or specialized parsers to stitch together field-level dependencies across the data lifecycle.

Platform Capabilities Comparison

How Integration Works Across the Stack

1. Streaming Layer (Kafka/Flink): Streaming pipelines emit lineage dynamically via OpenLineage integration or custom event emitters as messages pass through processing topics. The catalog ingests these schema transitions to map source topics to landing tables.
2. Transformation Layer (dbt/Spark/SQL): Catalogs ingest dbt metadata artifacts (manifest.json, run_results.json ) or parse query logs directly from data warehouses. This exposes how raw ingested columns map into intermediate and mart models.
3. BI Layer (Tableau/Looker/Power BI): Metadata scanners connect to BI platform APIs to map dashboard charts and fields back to the specific warehouse or semantic layer columns powering them.

Would you like to explore how to configure OpenLineage for a specific streaming framework like Kafka/Flink, or do you need a deep dive into setting up dbt and BI lineage inside one of these specific catalogs?

Several modern data catalog and metadata platforms can integrate column-level lineage across a combination of:

• BI semantic models (Power BI, Tableau, Looker, etc.)
• Transformation code (dbt, SQL, Spark, ETL/ELT code)
• Streaming pipelines (Kafka, Flink, Spark Structured Streaming, event-driven pipelines)

The strongest options are below.

Platforms with the broadest cross-stack lineage

1. Collibra

A common enterprise choice when you need lineage across heterogeneous systems:

• BI semantic layers
• warehouses/lakes
• ETL tools
• dbt
• Spark
• OpenLineage-compatible pipelines

Collibra can consume OpenLineage events and represent technical lineage at table and column levels.

2. DataHub

A popular open-source choice for engineering-heavy organizations:

• Column lineage from SQL parsing
• dbt metadata
• Airflow pipelines
• Spark jobs
• Kafka ecosystem integrations (typically through custom ingestion or emitters)

3. OpenMetadata

Similar open-source approach:

• Metadata graph
• Pipeline lineage
• Column-level relationships
• Data quality integration
• BI and warehouse connectors

4. OpenLineage-based ecosystems

Rather than being a catalog itself, OpenLineage is increasingly used as the transport layer for lineage events. It supports integrations such as Spark and dbt with column-level lineage capabilities.

Typical architecture:

BI semantic models
        |
        v
Metadata catalog
        ^
        |
OpenLineage events
        ^
        |
+-----------------------+
| dbt / Spark / Airflow  |
| Flink / streaming jobs |
+-----------------------+

Streaming-specific considerations

Streaming lineage is usually the hardest area. Look for support for:

• Kafka topic → stream processor → table mappings
• Schema Registry metadata
• Flink/Spark Structured Streaming jobs
• Event-time transformations
• Stateful aggregations
• Topic-to-column mappings

Many catalogs can show pipeline-level lineage for streaming, but fewer can reliably produce true column-level lineage through complex streaming transformations. OpenLineage is one approach for standardizing event capture, though connector coverage varies.

Practical shortlist by environment

• Large enterprise / regulatory: Collibra, Informatica, Alation
• Modern data stack (dbt + Snowflake + BI): Atlan, DataHub, OpenMetadata
• Databricks-heavy: Unity Catalog
• Microsoft Power BI + Azure: Purview
• Open-source-first: DataHub or OpenMetadata + OpenLineage

If you need one catalog that can unify BI models + dbt/SQL + Kafka/Flink/Spark streaming, I would typically evaluate Collibra, DataHub, and OpenMetadata first, then validate connector coverage against your exact BI and streaming stack.

Capturing end-to-end, column-level lineage that bridges BI transformation code (like dbt or native warehouse SQL) all the way back through streaming pipelines (like Apache Kafka, Flink, or Spark Streaming) requires a data catalog that supports both deep query/code parsing and open, event-driven ingestion standards (specifically OpenLineage).

The prominent data catalog and metadata platforms capable of stitching these disparate layers together include:

• *Acryl Data / DataHub Documentation * : Widely recognized for its extensible ingestion architecture and real-time metadata graph. DataHub uses advanced SQL parsers (like SQLGlot) to extract column-level lineage from BI and data warehouse transformations. Crucially, it features native OpenLineage REST endpoints and Spark/Flink event listeners , allowing streaming jobs to push real-time column-level lineage directly into the unified metadata graph.
• Atlan Documentation : Excels at active metadata management and deep BI-to-warehouse stitching. Atlan automatically parses transformation code from tools like dbt down to the column level and connects them with downstream BI dashboards (Looker, Power BI, Tableau). For streaming and complex data processing, Atlan ingests lineage via APIs and integrations (such as OpenLineage-compatible runners or orchestrators like Airflow/Spark), mapping how fields flow from ingestion layers into analytical tables.
• Databricks Unity Catalog Documentation : If your streaming pipelines land data in Delta/Iceberg tables via Spark or Flink, and your BI/transformations run natively inside the Databricks ecosystem, Unity Catalog automatically captures runtime column-level lineage . It tracks how columns are derived across notebooks, materialized views, and dashboards. However, its out-of-the-box automated lineage is strongest when workloads run directly on Databricks compute rather than external streaming brokers.
• Marquez / OpenLineage ecosystem : While Marquez functions more as a lineage-centric metadata store and visualization engine rather than a traditional broad-scope data catalog (like governance or business glossary tools), it is the open standard backend. Streaming frameworks like Apache Flink (via OpenLineageFlinkJobListener ) and Kafka components emit granular column-level input/output datasets natively into OpenLineage collectors, which catalogs then consume.

If you'd like, I can dive deeper into:

• How to configure OpenLineage with Flink or Kafka for a specific catalog
• Comparing pull-based (parsing warehouse logs) vs push-based (streaming OpenLineage events) approaches
• How these tools handle schema evolution in streaming topics

Let me know what your current tech stack looks like!

Yes. If by “integrate column-level lineage from BI models, transformation code, and streaming pipelines” you mean one catalog that can stitch field-level provenance across BI → transformation/SQL/dbt → warehouse → streaming/event pipelines, the strongest options today are:

1. Atlan — probably the best match

Atlan explicitly combines BI lineage, transformation lineage and pipeline lineage into one graph. It supports field/column-level lineage for BI sources, including Power BI, and column-level lineage from dbt.

The particularly interesting piece for streaming is its OpenLineage ingestion: Atlan says it can consume runtime lineage from Airflow, Spark, dbt Cloud and Astronomer, and its current documentation also lists Apache Flink OpenLineage integration.

So an architecture like:

Kafka/Flink/Spark → warehouse → dbt/SQL → Looker/Power BI

can potentially be represented as a continuous lineage graph rather than separate lineage silos.

2. OpenMetadata — particularly attractive if you want open source

OpenMetadata has unusually good alignment with this requirement.

Its lineage model explicitly supports column mappings, and its lineage API supports relationships such as Topic → Table, which is useful for streaming architectures.

More importantly, its OpenLineage connector can consume events from Kafka brokers or AWS Kinesis, translating them into OpenMetadata lineage.

It also ingests dbt artifacts and builds model/column lineage from them.

I'd shortlist OpenMetadata if Kafka/Flink/Spark lineage and extensibility are as important as the catalog itself.

3. Collibra — strongest enterprise-governance alternative

Collibra supports technical lineage across BI, ETL and data platforms, including dbt. Its dbt integration specifically extracts attribute-level lineage with transformation details.

Its current lineage documentation also supports OpenLineage, including Airflow and AWS Glue, and lists dbt, Databricks, Snowflake and other transformation systems.

The caveat is that I'd verify your specific streaming technology—Kafka/Flink/Kafka Connect/etc.—rather than assuming OpenLineage support means identical column-level coverage everywhere.

4. Microsoft Purview — best when the stack is Microsoft-centric

Purview has genuine column/process lineage and integrates tightly with Power BI, Data Factory, Synapse and Azure Databricks.

It's a particularly sensible choice if your estate looks like:

Event Hubs / Azure → Fabric/Databricks → Power BI

rather than a heterogeneous stack involving Kafka, Flink, Snowflake, Looker, dbt, etc.

---

My shortlist

If streaming lineage is a first-class requirement, I'd evaluate them in this order:

1. Atlan — best commercial cross-stack experience; excellent BI + dbt + OpenLineage story.
2. OpenMetadata — strongest open/extensible approach, particularly interesting for Kafka/Kinesis/OpenLineage.
3. Collibra — excellent enterprise governance and technical lineage, but validate streaming coverage carefully.
4. Microsoft Purview — excellent if you're heavily invested in Microsoft/Fabric/Azure.

One important distinction: “supports column-level lineage” doesn't necessarily mean “can infer column-level lineage through arbitrary streaming code.” Static SQL/dbt lineage is much easier than determining that customer_id in a Kafka event became customer_key after custom Flink/Java/Python logic. For the latter, OpenLineage runtime events or explicit lineage emission become especially important. Atlan and OpenMetadata are particularly interesting here because of their OpenLineage paths.

If you tell me your actual stack (e.g. Kafka + Flink + dbt + Snowflake + Looker), I can make a connector-by-connector matrix showing exactly which platform gets column lineage at each hop and where you would need custom instrumentation.

So an architecture like:

Kafka/Flink/Spark → warehouse → dbt/SQL → Looker/Power BI

can potentially be represented as a continuous lineage graph rather than separate lineage silos.

2. OpenMetadata — particularly attractive if you want open source

OpenMetadata has unusually good alignment with this requirement.

Its lineage model explicitly supports column mappings, and its lineage API supports relationships such as Topic → Table, which is useful for streaming architectures.

More importantly, its OpenLineage connector can consume events from Kafka brokers or AWS Kinesis, translating them into OpenMetadata lineage.

It also ingests dbt artifacts and builds model/column lineage from them.

I'd shortlist OpenMetadata if Kafka/Flink/Spark lineage and extensibility are as important as the catalog itself.

3. Collibra — strongest enterprise-governance alternative

Collibra supports technical lineage across BI, ETL and data platforms, including dbt. Its dbt integration specifically extracts attribute-level lineage with transformation details.

Its current lineage documentation also supports OpenLineage, including Airflow and AWS Glue, and lists dbt, Databricks, Snowflake and other transformation systems.

The caveat is that I'd verify your specific streaming technology—Kafka/Flink/Kafka Connect/etc.—rather than assuming OpenLineage support means identical column-level coverage everywhere.

4. Microsoft Purview — best when the stack is Microsoft-centric

Purview has genuine column/process lineage and integrates tightly with Power BI, Data Factory, Synapse and Azure Databricks.

It's a particularly sensible choice if your estate looks like:

Event Hubs / Azure → Fabric/Databricks → Power BI

Modern active metadata platforms and enterprise data catalogs have largely bridged the gap between code-based transformations, real-time data flows, and BI dashboards. They achieve column-level lineage across these disparate systems by combining SQL parsing engines (like SQLGlot or custom AST parsers), native API integrations with BI layers, and ingestion standards for streaming (such as OpenLineage).

The leading data catalog platforms capable of stitching together end-to-end column-level lineage from transformation code, streaming pipelines, and BI models include:

• Atlan — Uses an active metadata approach to parse transformation code (like dbt and SQL in warehouses), crawl BI models via native APIs (Tableau, Looker, Power BI), and ingest pipeline data (Airflow, Fivetran, Databricks Unity Catalog) to stitch together an end-to-end column-level lineage graph.
• DataHub — Features a robust schema-aware SQL parser (using SQLGlot) to deduce granular column-level lineage. It natively supports the OpenLineage standard, allowing streaming frameworks and custom event-driven pipelines to push runtime column dependencies straight into the unified catalog graph.
• OpenMetadata — An open-source metadata platform that automates column-level lineage by parsing SQL and pulling metadata from orchestrators, BI tools, and data pipelines. It natively integrates with standard telemetry protocols like OpenLineage to map real-time and streaming metrics.
• IBM Manta Data Lineage — A deep-scanning lineage specialist (often paired with Collibra or used standalone) that performs exhaustive hard-code parsing. It reads transformation scripts, database procedures, and data movement code down to the field/column level, extending visibility into reporting and BI tools.
• Informatica Intelligent Data Management Cloud (IDMC) — Utilizing its CLAIRE AI engine, Informatica’s Enterprise Data Catalog maps deep, automated column-level lineage across a massive matrix of enterprise data integration, streaming components, and downstream BI reports.

If you want to narrow down which of these fits your stack best, let me know:

• What streaming technologies (e.g., Kafka, Flink, Spark Streaming) are you running?
• What BI and transformation tools (e.g., dbt, Looker, Power BI) make up your core reporting layer?