Data Engineer
- Uplers
- 4 - 6 years
- Bengaluru
- 7 days ago
- Email to a friend
- Report this job
Job Description
Reference Code: HR1175440031573725
Designation: Data Engineer Azure & DatabricksJob Description for Data Engineer Azure & Databricks
Location: Bengaluru [Hybrid]
Experience Required: 4 to 6 years
Employment Type: Full-time
Department: Data Engineering / Analytics
(*Note: This is a requirement for one of Uplers Clients)About the Client: Our client is building scalable data platforms leveraging modern cloud-native and open-source technologies to drive decision intelligence across the organization. Our data lake is designed on the Microsoft Medallion Architecture, powered by Databricks and Azure-native services. We are looking for a passionate Data Engineer to join our team and help in expanding, optimizing, and maintaining our enterprise-grade data pipelines and architectures.Key Responsibilities
- Design, develop, and maintain robust ETL/ELT pipelines using Azure Data Factory (ADF), Databricks (PySpark/Scala), and SQL Server.
- Work on the Bronze-Silver-Gold (Medallion) architecture to organize data ingestion, cleansing, and transformation layers efficiently.
- Ingest and process both structured and unstructured data from diverse data sources such as APIs, cloud storages, transactional systems, and logs.
- Collaborate with data scientists, analysts, and business teams to ensure data availability and reliability for downstream use cases.
- Implement CI/CD for data pipelines using DevOps best practices (Azure DevOps / GitHub Actions).
- Optimize data processing performance on Databricks using Delta Lake, caching, Z-Ordering, and cluster tuning.
- Ensure data quality, observability, and lineage through tools like Great Expectations, OpenLineage, and dbt.
- Support metadata management and documentation for governance and compliance using tools like Apache Atlas or Unity Catalog.
- Explore and integrate open-source data tooling to supplement cloud-native workflows.
Technical Skills Required (Area-wise breakdown of tools and technologies)
- Cloud Platform - Azure (Data Factory, Data Lake Gen2, Blob Storage, Azure SQL DB, Key Vault, Azure DevOps)
- Compute & Processing - Databricks (Spark, PySpark, Delta Lake), SQL Server
- Orchestration - Azure Data Factory, dbt, Airflow (nice to have)
- Data Modeling - Dimensional Modeling, Data Vault (plus), Star/Snowflake schemas
- Programming Languages - Python, SQL, Spark (PySpark/Scala)
- CI/CD & DevOps - Azure DevOps, Git, GitHub Actions
- Data Observability & Quality - Great Expectations, Soda SQL, Monte Carlo (plus)
- Metadata Management - Unity Catalog, Apache Atlas
- Monitoring & Logging - Datadog, Azure Monitor, Log Analytics
- Open-Source Tools (Plus) - dbt, Apache Kafka, Apache Airflow, OpenLineage
Qualifications
- Bachelor's or Masters degree in Computer Science, Information Systems, or a related field.
- 46 years of hands-on experience in data engineering with strong exposure to Azure and Databricks ecosystems.
- Experience working in a modern data stack and familiarity with open-source best practices.
- Strong understanding of data warehouse concepts, ETL frameworks, and cloud-native data pipelines.
- Familiarity with data governance, lineage, cataloging, and access control.
- Strong problem-solving skills, attention to detail, and ability to work in agile, fast-paced environments.
Good to Have
- Certification in Azure Data Engineer Associate or Databricks Data Engineer Associate/Professional.
- Exposure to Data Mesh, Lakehouse architectures, and streaming systems like Kafka or Azure Event Hubs.
- Familiarity with machine learning pipeline orchestration (e.g., MLflow, Feature Store).
Job Classification
Industry: Software ProductFunctional Area / Department: Data Science & AnalyticsRole Category: Data Science & Machine LearningRole: Data EngineerEmployement Type: Full timeContact Details:
Company: UplersLocation(s): Bengaluru
Keyskills: Data Engineering Azure Databricks Pyspark Open Source Oracle ADF SQL Server Data Bricks SQL Azure Data Factory Ci / Cd Tools Delta Lake Azure Data Lake Data Lake Data Modeling Spark Python
Similar positions
Uplers
Uplers is a one-stop digital services company delivering end-to-end web, design, digital marketing, and email production services to businesses and agencies across 52+ Nations. Backed by a team of 550+ digital expert.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in