2+ years of hands-on experience with Databricks in designing, developing, and maintaining scalable data engineering solutions.
Strong expertise in Apache Spark / PySpark, SQL, and Python for large-scale data processing and transformation.
Strong understanding of Delta Lake and Lakehouse architecture, including data modeling, optimization, and reliable data storage patterns.
Experience with workflow orchestration, including designing and managing data pipelines, job scheduling, dependencies, and monitoring.
Strong knowledge of data quality, validation, reconciliation, and error-handling practices to ensure data accuracy and reliability.
Experience working with heterogeneous enterprise data sources, including structured, semi-structured, and unstructured data.
Azure data-platform experience is strongly preferred, particularly with Azure-based data engineering and analytics services.
EDI knowledge is not mandatory and will be considered an added advantage.
Qualifications:
Bachelor's degree in computer science, Information Technology, Engineering, or equivalent.
Job Description:
Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python to support enterprise data-processing requirements.
Build and optimize Lakehouse solutions using Databricks and Delta Lake, ensuring scalability, performance, reliability, and maintainability.
Ingest and process data from diverse enterprise sources, including structured, semi-structured, and unstructured data, while handling different formats and data complexities.
Develop and manage workflow orchestration for data pipelines, including scheduling, dependency management, monitoring, failure handling, and operational support.
Implement data-quality and validation frameworks to perform data profiling, reconciliation, completeness checks, accuracy validation, and anomaly detection.
Optimize Spark/PySpark and SQL workloads by applying appropriate partitioning, caching, query optimization, Delta Lake optimization, and other performance-tuning techniques.
Collaborate with data engineers, architects, analysts, and business stakeholders to understand data requirements and deliver secure, reliable, and scalable Azure-based data solutions.