Senior Data Engineer
- Accion Labs
- 6 - 10 years
- Hyderabad
- 1 month ago
- Email to a friend
- Report this job
Job Description
Senior Data Engineer Real-Time Analytics (Retail Domain Preferred)
Experience:
6+ years of overall data engineering experience
Job Summary:
We are looking for a skilled Senior Data Engineer with hands-on experience in Python, Spark, and AWS technologies to join our growing data engineering team. The ideal candidate will have a strong understanding of real-time data processing frameworks, be comfortable working with large-scale distributed systems, and preferably have exposure to the retail domain. This role will focus on building scalable and efficient data pipelines that support real-time analytics and decision-making systems.
Key Responsibilities:
- Design, develop, and maintain real-time and batch data pipelines using PySpark/Spark and Python.
- Implement data ingestion and transformation logic for streaming and batch processing using tools like AWS Glue, EMR, and Lambda.
- Build and optimize ETL/ELT processes that move and transform data from various sources (S3, RDS, Redshift, etc.).
- Leverage AWS Step Functions, EventBridge, and SQS to orchestrate complex workflows and handle event-driven architectures.
- Work closely with data scientists, analysts, and product teams to support data needs for analytics and reporting.
- Develop and manage real-time data streaming solutions for ingesting, processing, storing, and serving data using modern architectures.
- Ensure data quality, integrity, and governance across all pipelines.
- Optimize queries and performance for Redshift and SQL-based data stores.
- Build monitoring and alerting for data pipeline health and system performance.
- Participate in code reviews and promote best practices in software engineering.
Required Skills & Experience:
- Strong programming skills in Python, especially for data manipulation using Pandas.
- Proficient in PySpark/Spark for large-scale data processing.
- Strong SQL skills with experience in working with Redshift, RDS, or similar databases.
- Experience with AWS cloud ecosystem:
- EMR, Glue, S3, RDS, Redshift, Lambda, SQS, Step Functions, EventBridge
- Experience in building real-time data analytics pipelines and streaming systems (Kafka, Kinesis, or similar tools is a plus).
- Solid understanding of data modeling, data warehousing, and distributed computing.
- Familiarity with data visualization or action layers is a plus.
- Exposure to retail domain use cases (e.g., customer analytics, inventory, or POS systems) is preferred.
- Comfortable working in Agile environments and collaborating with cross-functional teams.
Preferred Qualifications:
- Bachelor's or Masters degree in Computer Science, Engineering, or a related field.
- Experience with CI/CD for data pipelines, infrastructure as code (e.g., Terraform or CloudFormation).
- Knowledge of data security, privacy, and compliance in cloud-based environments.
- Experience with containerization (Docker, ECS/EKS) is a plus.
Nice to Have:
- Familiarity with data visualization tools like QuickSight, Tableau, or Power BI.
- Exposure to machine learning pipelines or integration with data science platforms.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Accion LabsLocation(s): Hyderabad
Keyskills: AWS EMR/Glue/S3/RDS/Redshift/Lambda/SQS/AWS Step Function/EventBridge data ingestion/ stream processing/ real-time storage/serving layer/ visualization/action layer Data Engineer Python/Spark/PySpark/Pandas SQL
Similar positions
Accion Labs
Accion Labs is a Product Engineering Company Helping to Transform Businesses Through Emerging Technologies. This includes Web 2.0, Open Source, SaaS/Cloud, Mobility, IT Operations Management/ITSM, Big Data, and traditional BI/DW. Through nine global offices and a rapid-response delivery model, Accio...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in