Senior Data Engineer
- Healthcare
- 4 - 9 years
- Hyderabad
- 2 months ago
- Email to a friend
- Report this job
Job Description
Key skills: Python, SQL, Pyspark, Databricks, AWS (Manadate)
Added advantage: Life sciences/Pharma
Roles and Responsibilities
1.Data Pipeline Development: Design, build, and maintain scalable data pipelines for ingesting,
processing, and transforming large datasets from diverse sources into usable formats.
2.Data Integration and Transformation: Integrate data from multiple sources, ensuring data is
accurately transformed and stored in optimal formats (e.g., Delta Lake, Redshift, S3).
3.Performance Optimization: Optimize data processing and storage systems for cost efficiency and high performance, including managing compute resources and cluster configurations.
4.Automation and Workflow Management: Automate data workflows using tools like Airflow,
Data bricks APIs, and other orchestration technologies to streamline data ingestion, processing, and reporting tasks.
5.Data Quality and Validation: Implement data quality checks, validation rules, and transformation logic to ensure the accuracy, consistency, and reliability of data.
6.Cloud Platform Management: Manage and optimize cloud infrastructure (AWS, Databricks) for data storage, processing, and compute resources, ensuring seamless data operations.
7.Migration and Upgrades: Lead migrations from legacy data systems to modern cloud-based
platforms, ensuring smooth transitions and enhanced scalability.
8.Cost Optimization: Implement strategies for reducing cloud infrastructure costs, such as optimizing resource usage, setting up lifecycle policies, and automating cost alerts.
9.Data Security and Compliance: Ensure secure access to data by implementing IAM roles and
policies, adhering to data security best practices, and enforcing compliance with organizational
standards.
10.Collaboration and Support: Work closely with data scientists, analysts, and business teams to understand data requirements and provide support for data-related tasks.
Job Classification
Industry: Pharmaceutical & Life SciencesFunctional Area / Department: IT & Information SecurityRole Category: IT Infrastructure ServicesRole: IT Infrastructure Services - OtherEmployement Type: Full timeContact Details:
Company: healthcare insuranceLocation(s): Hyderabad
Keyskills: AWS Data Bricks Python SQL Airflow Pyspark Data Lake Spark
Similar positions
Healthcare
About Agilisium https://www.agilisium.com/ • Agilisium, is an AWS technology Advanced Consulting Partner that enables companies to accelerate their Data-to-Insights-Leap. • With $25+ million in annual revenue and over 40% year-over-year growth, Agilisi...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in