Consultant

Senior Data Engineer • Developed and maintained 30+ production Azure Databricks jobs for enterprise-scale ETL/ELT pipelines using PySpark, Scala, SQL, Delta Lake, and Azure Data Factory. • Optimized Databricks workloads, reducing cluster costs by 20% through Spark tuning, partitioning, caching, broadcast joins, and cluster right-sizing. • Worked on streaming IoT data from retail fuel stations using Azure Event Hubs, enabling near real-time data processing and monitoring. • Built scalable data pipelines integrating SQL Server, ADLS Gen2, Azure Blob Storage, Delta Tables, CSV/Excel files, JDBC, and REST APIs. • Processed and managed 20+ million transactions daily, ensuring high performance, scalability, and data reliability. • Implemented complex Spark transformations, aggregations, window functions, and performance optimization for large-scale datasets. • Designed data quality, reconciliation, duplicate detection, and validation frameworks to improve data accuracy. • Developed parameterized Azure Data Factory pipelines and Databricks Workflows for automated production scheduling and monitoring. • Integrated Azure Key Vault and Databricks Secret Scopes for secure credential management. • Delivered curated Delta Lake datasets for Power BI dashboards and collaborated with cross-functional teams to resolve production issues and deliver scalable data solutions.