Data Engineering & Analytics

I build production data platforms for healthcare organizations. Databricks lakehouses with 500+ dbt models, automated data contracts, and incremental CDC pipelines. Value-based care analytics: HEDIS and Stars quality reporting, CMS-HCC risk adjustment feeds, and population health dashboards.

Most of my work in this area has been for a Medicare Advantage primary care organization with five clinic locations. The platform I built there serves 117 users across clinical operations, quality, risk adjustment, and finance.

The problem

Healthcare organizations hit a wall when their reporting outgrows their data infrastructure. The pattern is always the same. Someone builds a Tableau dashboard. Someone else builds a different dashboard from a different data source. A metric like "average HbA1c for diabetic members" has three different answers depending on which workbook you open. No data contracts. No lineage. No way to trace a number on a dashboard back to the SQL that produced it.

In Medicare Advantage that is not just messy. It is a compliance risk. CMS-HCC risk adjustment scores determine revenue. HEDIS quality measures drive Star ratings, which drive bonus payments. RADV audits require you to reproduce every number you submitted. When the auditor asks "how did you calculate that RAF score," the answer has to be a deterministic query, not "the Tableau workbook Carol built in 2023."

The specific problems I solve:

• Ungoverned metrics. Multiple definitions for the same measure, scattered across different tools and spreadsheets. No single source of truth.
• Scattered reporting. Dashboards built from ad-hoc queries with no version control. Changes break things silently. Nobody knows which source is authoritative.
• No data contracts. Columns get renamed or dropped without warning. Downstream dashboards break. Nobody catches it until a clinical director sees wrong numbers in a meeting.
• Manual quality reporting. HEDIS and Stars measures calculated by hand in spreadsheets. Months of staff time, error-prone, not auditable.
• No CDC pipeline. Full table reloads instead of incremental change processing. Slow, expensive, and impossible to maintain at scale.

My approach

I build this in layers. Each layer solves a specific problem and the stack works together as a governed system.

Metric registry

Everything starts with a metric registry. A dbt seed file (CSV) that binds every metric name to its mart model, column, aggregation function, grain, and owner. This CSV is the single source of truth. The BI layer reads from it. Dashboards do not define their own SQL. They reference the registry and the registry points to the mart column.

Adding a new metric is a pull request that adds a row to the CSV and the corresponding mart model. Removing a metric flags it in the registry and the CI gate catches downstream references before they break.

dbt transformation layer

The data flow: raw sources land in Databricks via CDC pipelines. dbt staging models clean and rename them. Intermediate models join and transform. Mart models produce the final aggregates at the grain the registry specifies. Every model has YAML documentation, tests, and enforced data contracts.

When contract: enforced is set on a model, dbt fails the run if a column changes type or gets dropped. Nobody can silently rename a column without the pipeline catching it. That protection is critical at 500+ models where a change in one staging table can cascade through dozens of downstream marts.

Databricks lakehouse

Databricks serves as the compute and storage layer. Raw data arrives through Change Data Feed pipelines. Incremental dbt models process only the partitions that changed since the last run, using partition_by and unique_key for upserts. The full_refresh=false config is set at the model level to prevent accidental full reprocessing of CDC data.

Data contracts and CI/CD

Every model has a YAML contract that locks its column names and types. A consolidation ledger tracks every model in the platform: its grain, owner, materialization, contract status, consumers, and migration status. The CI gate validates that no model is removed or renamed without updating all consuming models and the ledger.

Every PR against main runs through: dbt compile for syntax errors, dbt test for data quality checks, and the ledger validator for contract integrity. The gate does not care who opened the PR.

Governed BI serving layer

Mart output lands in a curated schema. A BFF layer exposes registered metrics through a typed API. Dashboard authors reference metrics by registry name and get data at the correct grain with the correct aggregation. This is how you get governed metrics without trusting dashboard authors to write correct SQL.

Deliverables

What you get when I build this out:

• Production data platform. dbt project with staging, intermediate, and mart layers running on Databricks. Incremental CDC pipelines for all source data. Full lineage from raw source to dashboard.
• Governed dashboards. BI dashboards that reference the metric registry instead of defining their own SQL. Every number traces back to a single, versioned, tested transformation.
• Metric registry. A living CSV that documents every metric the organization tracks: name, mart model, column, aggregation, grain, owner, domain. The most important file in the platform.
• Automated tests. dbt tests on every model: not_null, unique, accepted_range, and custom business logic tests. CI gate that blocks PRs with failing tests.
• Data contracts. YAML contracts on every mart model. Column names and types are locked. Breaking changes fail the pipeline before they reach production.
• Consolidation ledger. A governance artifact tracking every model: who owns it, what consumes it, what its grain is, and its migration status.
• CI/CD pipeline. Automated compile, test, contract validation, and ledger checks on every PR. Works for human and autonomous agent PRs.
• Quality reporting. HEDIS and Stars measures calculated from the metric registry. CMS-HCC risk adjustment feeds derived from governed models. Auditable, reproducible, version-controlled.

Proof points

These numbers come from the platform I built and maintain in production.

I migrated the entire data stack from Tableau, Snowflake, and Cube to dbt, Databricks, and a governed BI layer. The old stack is fully retired. The consolidation eliminated the "which source is right" problem and reduced licensing costs.

Both human developers and autonomous AI agents submit pull requests on this platform. Both go through the same CI gate. No autonomous merge to main. A human reviews and approves every change.

Related reading

I wrote up the technical details in two blog posts if you want the architecture deep-dive.