Databricks
Power BI
Scheduled API RefreshThe Problem
The Consumer Financial Protection Bureau publishes millions of consumer complaint records, but a raw dataset of this size — over 8.3 million rows spanning January 2025 through July 2026 — isn't something a business leader can read directly. Without a reporting layer, systemic issues like which companies generate the most complaints, or which regions see the slowest response times, stay buried in the data.
How the Pipeline Works
The defining feature of this project is that it's not a static report — it's a live system. Data flows from source to dashboard automatically, on a schedule, without anyone touching a spreadsheet.
Scheduled Ingestion
A scheduled job calls the Databricks API to pull the latest complaint records directly from the cloud workspace — no manual exports or file transfers.
Transform & Clean
Records are deduplicated, standardized, and structured into a reporting-ready model, isolating the fields that matter most for trend and root-cause analysis.
Load
Cleaned data lands in a reporting layer optimized for fast queries, so the dashboard stays responsive even at millions of rows.
Publish to Power BI
Power BI connects directly to the refreshed dataset. Each cycle, the dashboard updates itself — leadership always sees current numbers, not a stale snapshot.
What This Demonstrates
- Cloud Data Platform Integration: Connecting to and pulling from Databricks via API, rather than relying on manual file exports.
- Automation & Scheduling: Building a refresh cycle that keeps a large-scale dataset current with no ongoing manual effort.
- Data Modeling at Scale: Structuring millions of records into a model that stays fast and query-ready inside Power BI.
- Executive Communication: Translating a massive raw dataset into a handful of decision-ready insights.
Key Findings
- Credit Bureaus Dominate the Data: TransUnion, Equifax, and Experian together account for over 7.1 million of the total complaints analyzed.
- Root Cause Identification: Decomposition tree analysis shows "Credit reporting" as the primary product of concern, with "Incorrect information" as the leading sub-issue at over 4.3 million complaints.
- Volume Trends & Geography: A 3-month rolling average shows a steady upward trend in complaint volume from early 2025 through mid-2026, with geospatial tracking highlighting regions where company responses regularly exceed 3 days.