ETL Developer

Job Summary

• As a PySpark & Azure Databricks Data Engineer, we need to design, implement, and maintain data engineering solutions.
• Building robust data pipelines, optimizing data processing, and ensuring data availability and reliability.
• Provide technical support and troubleshooting for applications developed using Databricks Unified Data Analytics Platform.

Goals and deliverables

Minimum Requirements

• Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
•  Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
• Provide technical support and troubleshooting for applications developed using Databricks Unified Data Analytics Platform.
• Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
• Collaborate with data engineering and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.
• Proven experience in data engineering, with a focus on PySpark on Azure Databricks.
• Proficiency in programming languages like Python and SQL.
• Should have knowledge of ADLS storing data objects, such as files, blobs, queues, and tables as delta lake in Azure.
• Experience with cloud platforms, particularly Microsoft Azure.
• Resolve urgent production issues and incidents by troubleshooting and identifying root cause analysis, including documentation and preventative measures.
• Perform routine maintenance (Vacuum & Optimize run), updates and troubleshooting of the Azure Environment.
• Monitor cluster performance, resource utilization, and troubleshoot issues to ensure optimal performance.
• Create the azure databricks workflows and schedule the jobs.
• Document data engineering processes, workflows, and best practices.
• Excellent Communication and Team Coordination Skills
• ETL jobs maintenance and scheduling.
• unity catalog on databricks knowledge is added advantage
• Must be available during normal business hours and be able to provide support if needed in case of Production or infrastructure related issues off-hours.

Education qualifications

• Graduate degree (or equivalent experience) in a systems or software engineering discipline