Utilizes software engineering principles to deploy and maintain fully automated data transformation pipelines that combine a large variety of storage and computation technologies to handle a distribution of data types and volumes in support of data architecture design. A Senior Data Engineer designs and oversees the entire data infrastructure, data products and data pipelines that are resilient to change, modular, flexible, scalable, reusable, and cost effective.
Bachelor s degree in computer science engineering or related field
Technical Leadership
Consulting and managing business needs.
Strong experience in Python is preferred but experience in other languages such as Scala Java, C#, etc is accepted.
Experience building spark applications utilizing PySpark.
Experience with file formats such as Parquet, Delta, Avro.
Experience efficiently querying API endpoints as a data source.
Understanding of Git workflows in software development.
Using Azure DevOps pipeline and repositories to deploy and maintain solutions. Understanding of Ansible and how to use it in Azure DevOps pipelines.
Job Classification
Industry: BankingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full time