Tushar Kumar

Senior Data Engineer • Developed and maintained 30+ production Azure Databricks jobs for enterprise-scale ETL/ELT pipelines using PySpark, Scala, SQL, Delta Lake, and Azure Data Factory. • Optimized Databricks workloads, reducing cluster costs by 20% through Spark tuning, partitioning, caching, broadcast joins, and cluster right-sizing. • Worked on streaming IoT data from retail fuel stations using Azure Event Hubs, enabling near real-time data processing and monitoring. • Built scalable data pipelines integrating SQL Server, ADLS Gen2, Azure Blob Storage, Delta Tables, CSV/Excel files, JDBC, and REST APIs. • Processed and managed 20+ million transactions daily, ensuring high performance, scalability, and data reliability. • Implemented complex Spark transformations, aggregations, window functions, and performance optimization for large-scale datasets. • Designed data quality, reconciliation, duplicate detection, and validation frameworks to improve data accuracy. • Developed parameterized Azure Data Factory pipelines and Databricks Workflows for automated production scheduling and monitoring. • Integrated Azure Key Vault and Databricks Secret Scopes for secure credential management. • Delivered curated Delta Lake datasets for Power BI dashboards and collaborated with cross-functional teams to resolve production issues and deliver scalable data solutions.

Designed and developed scalable data pipelines using Azure Data Factory (ADF) for ingesting, transforming, and loading data from on-premises and cloud-based sources into Azure Synapse Analytics and Azure SQL Database. Built and optimized ETL workflows using ADF pipelines, parameterized datasets, and data flows to support various business intelligence and reporting requirements. Leveraged Azure Databricks and PySpark for big data processing and transformation of semi-structured data from Azure Data Lake Gen2. Implemented Delta Lake architecture for incremental data loads and ensured data quality through validation scripts and partitioning strategies. Deployed and maintained enterprise-wide data lake architecture, integrating structured and unstructured data for real-time and batch analytics. Collaborated with data analysts and Power BI teams to create data models, ensuring accurate and timely dashboards and reports. Used Azure Monitor and Log Analytics to track pipeline performance, diagnose failures, and implement proactive alerts for data quality and SLA compliance. Worked with stakeholders and cross-functional teams to gather requirements, deliver PoCs, and implement scalable solutions based on agile methodologies. Documented architecture, pipeline logic, and operational procedures for seamless knowledge transfer and audit readiness.