Consultant - Sr. Data Engineer
- Affine Analytics
- 5 - 9 years
- Bengaluru
- 2 months ago
- Email to a friend
- Report this job
Job Description
Roles and Responsibilities:
- Design and develop data ingestion frameworks for batch and real-time data processing using distributed systems.
- Create, implement, and maintain streaming data solutions leveraging technologies like Kafka, Apache Spark, and Hadoop.
- Develop data transformation pipelines, ensuring high performance, scalability, and reliability of data systems.
- Work closely with business analysts and stakeholders to understand requirements and translate them into technical ETL solutions.
- Build, deploy, and support application code and analytical models across various environments.
- Integrate open-source data tools to enhance analytics capabilities and operationalize them for enterprise usage.
- Use tools like Python, Java, Hive, Presto, and Scala to prototype and build data engineering solutions.
- Ensure adherence to data governance policies by implementing or validating data lineage, data quality checks, and data classification.
- Participate in on-call support rotation, ensuring high availability and incident resolution.
- Align solutions with company-wide software development lifecycle (SDLC) and DevOps best practices.
- Provide recommendations for process improvements and develop innovative solutions for complex data engineering challenges.
- Collaborate with cross-functional teams to support delivery of critical data projects and enterprise data platforms.
Job Requirement
Must-Have Skills & Qualifications:
- Bachelors degree in computer science, Engineering, or a related field; masters degree preferred.
- 5+ years of experience in data engineering or a related role.
- Proficiency in Apache Airflow for workflow scheduling and management.
- Strong experience with Hadoop ecosystems, including HDFS, MapReduce, Hive, Presto.
- Expertise in Apache Spark/Scala, Kafka for large-scale and real time data processing.
- Proficient in Python
- Advanced SQL skills for data analysis and reporting.
- Experience with cloud platforms (e.g., AWS, Google Cloud, Azure) preferably GCP is a plus.
- Strong analytical and problem-solving skills.
- Excellent communication and collaboration abilities.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Affine AnalyticsLocation(s): Bengaluru+ View Contact
Keyskills: Data Engineering Apache Airflow GCP Scala Hadoop Kafka HDFS AWS Apache Spark Python MapReduce
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in