Manager Data Engineer
- LatentView
- 9 - 14 years
- Chennai
- 1 month ago
- Email to a friend
- Report this job
Job Description
Job Description:
We are looking for a highly motivated and skilled L4 Data Engineer Manager, particularly with Databricks experience, is generally responsible for leading data engineering projects, overseeing a team of data engineers, and designing data architectures on platforms like Databricks.
Responsibilities:
Leadership and Team Management:
- Lead, mentor, and develop a team of data engineers, ensuring high performance and career growth.
- Collaborate with stakeholders across data science, analytics, and engineering teams to deliver high-impact data solutions.
- Oversee the data engineering project lifecycle, from requirements gathering to deployment, ensuring quality and timeliness.
Data Architecture and Strategy:
- Develop scalable data architectures and solutions on Databricks for ETL processes, data warehousing, and big data processing.
- Define and enforce data governance policies, best practices, and standards for data processing and management.
- Design data flow pipelines for efficient data ingestion, storage, and processing in cloud environments (e.g., AWS, Azure, GCP).
Databricks Management and Optimization:
- Optimize Databricks clusters for cost and performance efficiency, leveraging cluster scaling and resource management best practices.
- Implement advanced data transformations and data models within Databricks using Spark and Delta Lake.
- Ensure integration between Databricks and other data tools, such as data lakes, SQL databases, and BI tools.
Data Quality and Security:
- Monitor and ensure data quality, reliability, and security within the Databricks environment.
- Implement data validation checks, data profiling, and error-handling mechanisms.
- Collaborate with security teams to ensure compliance with data privacy regulations and internal security standards.
Technical Development and Innovation:
- Stay updated with the latest Databricks capabilities and cloud technology trends to introduce innovative data engineering practices.
- Develop reusable and efficient code, libraries, and tools to automate and streamline data workflows.
- Troubleshoot and resolve complex data pipeline issues and provide continuous improvements in performance and reliability.
Skills:
- Advanced experience with Databricks, PySpark and data pipeline frameworks.
- Proficiency in Python, SQL, and/or Scala.
- Familiarity with cloud platforms like AWS, Azure, or GCP.
- 8+ years in data engineering, with 2+ years in a leadership or managerial role.
- Strong experience in data architecture, data pipeline optimization, and data modeling.
- Proven experience in managing large-scale data processing systems and ETL pipelines.
- Familiarity with Airflow for workflow orchestration and experience with Linux administration and shell scripting.
- Excellent communication skills to collaborate effectively with cross-functional teams.
- Strong problem-solving abilities and attention to detail.
Job Classification
Industry: Analytics / KPO / ResearchFunctional Area / Department: OtherRole Category: OtherRole: OtherEmployement Type: Full timeContact Details:
Company: LatentViewLocation(s): Chennai+ View Contact
Keyskills: Pyspark Data Bricks SQL
Fraud Alert to job seekers!
₹ 0-45 Lacs P.A
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in