Case study ยท Group project
Translytics: Databricks medallion pipeline
I worked with a team on a big data project for a fictional company, National Logistics Co. The pipeline runs on Databricks with PySpark, in a medallion layout. The data is synthetic.
Bronze
Raw shipment, driver, warehouse, and customer data is loaded into Bronze tables. Bronze keeps the files as they arrived, before cleanup.
Silver
Silver is the cleaned layer. The team prepares the Bronze rows so later totals can be trusted.
- Provinces and statuses are normalized.
- Type casting puts each field in the right type.
- Deduplication removes repeated rows.
- Invalid records are removed.
- Orphan records are removed.
- Days delayed and on-time flags are added.
Gold and the dashboard
Gold tables aggregate the Silver data. Those tables power a Databricks operations dashboard. The dashboard covers revenue, shipments, failed deliveries, on-time rate, a driver leaderboard, and a warehouse comparison.