Lead Software Engineer/Sr Software Engineer- Data Engineer - Python
- CGI
- 6 - 8 years
- Hyderabad
- 17 days ago
- Email to a friend
- Report this job
Job Description
Lead Software Engineer/Sr Software Engineer- Data Engineer -Python & Hadloop Job Overview:
CGI is looking for a talented and motivated Data Engineer with strong expertise in Python, Apache Spark, HDFS, and MongoDB to build and manage scalable, efficient, and reliable data pipelines and infrastructure
Youll play a key role in transforming raw data into actionable insights, working closely with data scientists, analysts, and business teams.Key Responsibilities:
- Design, develop, and maintain scalable data pipelines using Python and Spark.
- Ingest, process, and transform large datasets from various sources into usable formats.
- Manage and optimize data storage using HDFS and MongoDB.
- Ensure high availability and performance of data infrastructure.
- Implement data quality checks, validations, and monitoring processes.
- Collaborate with cross-functional teams to understand data needs and deliver solutions.
- Write reusable and maintainable code with strong documentation practices.
- Optimize performance of data workflows and troubleshoot bottlenecks.
- Maintain data governance, privacy, and security best practices.
Required qualifications to be successful in this role: - Minimum 6 years of experience as a Data Engineer or similar role.
- Strong proficiency in Python for data manipulation and pipeline development.
- Hands-on experience with Apache Spark for large-scale data processing.
- Experience with HDFS and distributed data storage systems.
- Strong understanding of data architecture, data modeling, and performance tuning.
- Familiarity with version control tools like Git.
- Experience with workflow orchestration tools (e.g., Airflow, Luigi) is a plus.
- Knowledge of cloud services (AWS, GCP, or Azure) is preferred.
- Bachelors or Masters degree in Computer Science, Information Systems, or a related field.Preferred Skills:
- Experience with containerization (Docker, Kubernetes).
- Knowledge of real-time data streaming tools like Kafka.
- Familiarity with data visualization tools (e.g., Power BI, Tableau).
- Exposure to Agile/Scrum methodologies.
Skills: - Hadoop Hive
- Python
- SQL
- English
Note
- This role will require- 8 weeks of in-office work after joining, after which we will transition to a hybrid working model, with 2 days per week in the office.
- Mode of interview F2F
- Time : Registration Window -9am to 12.30 pm. Candidates who are shortlisted will be required to stay throughout the day for subsequent rounds of interviews
Notice Period: 0-45 Days
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: CGILocation(s): Hyderabad
Keyskills: Data Engineering Azure Docker GCP HDFS Python Spark MongoDB HFDS AWS Python Apache Spark Kubernetes
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in