Databricks & Lakehouse Engineer
- Intelliswift software
- 8 - 12 years
- Delhi, NCR
- 16 hours ago
- Email to a friend
- Report this job
Job Description
Databricks & Lakehouse Engineering
- Manage and optimize Databricks clusters, jobs, notebooks, and libraries.
- Architect Lakehouse solutions using Delta Lake, Unity Catalog, and AWS-native services.
- Develop and maintain scalable ETL/ELT pipelines (batch and streaming) using Spark, Python, Auto Loader, Glue, and Kinesis.
Workflow Orchestration & Runtime Management
- Build and manage workflows using Databricks Workflows or Airflow.
- Define job dependencies, retries, alerts, and execution SLAs.
- Plan and manage Databricks Runtime upgrades with proper validation and performance checks.
Security, Compliance & Governance
- Implement data access controls using Unity Catalog (row- and column-level).
- Enforce encryption, private network access (VPC/PrivateLink), and audit logging.
- Support data compliance requirements such as GDPR and HIPAA.
Observability & Monitoring
- Monitor workloads using Databricks native tools .
- Integrate with Datadog for real-time alerts, system health metrics, and performance dashboards.
Disaster Recovery & Backup
- Design and implement disaster recovery strategies across Availability Zones and regions.
- Implement backup and restore workflows for Databricks workspaces, metastore, and data lake storage.
Cost Governance & FinOps
- Apply cluster policies and tagging for usage tracking.
- Monitor cost KPIs such as $/TB processed, idle-time burn, autoscaling metrics.
- Integrate Datadog for real-time cost and performance observability.
CI/CD & Automation
- Automate deployments using Terraform, CloudFormation, and Git-based CI/CD.
- Integrate data quality checks (e.g., Great Expectations) into release pipelines.
Additional Responsibilities
- Refactor and optimize existing codebase for maintainability and performance.
- Perform root cause analysis (RCA) and resolve bugs in production workloads.
- Build prototypes for new requirements and estimate engineering effort.
- Collaborate with analytics, DevOps, business teams, and document architecture and processes.
Required Skills & Experience
- 710 years of data engineering experience, with 5+ years on Databricks and Apache Spark.
- Expert-level hands-on experience with Databricks and AWS (S3, Glue, EMR, Kinesis, Lambda, IAM, CloudWatch).
- Primary language: Python; strong skills in Spark SQL.
- Deep understanding of Lakehouse architecture, Delta Lake, Parquet, Iceberg.
- Strong experience with Databricks Workflows, Unity Catalog, Runtime upgrades, and cost optimization.
- Experience with Databricks native monitoring tools and Datadog integration.
- Security and compliance expertise across data governance and infrastructure layers.
- Experience with CI/CD automation using Terraform, CloudFormation, and Git.
- Hands-on experience with disaster recovery and multi-region architecture.
- Strong problem-solving, debugging, and documentation skills.
Nice to Have
- Experience with GCP (BigQuery, GCS) or Azure (ADLS Gen2, ADF, Synapse).
- Certifications: Databricks Certified Data Engineer Pro, AWS Data Analytics, AWS Solutions Architect.
- Familiarity with Kafka, Spark Structured Streaming, or EventBridge.
- Experience working with BI tools like Tableau, Power BI, or Looker.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Engineering - Software & QARole Category: Software DevelopmentRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Intelliswift softwareLocation(s): Delhi, NCR
Keyskills: Spark Data Bricks Python SQL Lakehouse
Similar positions
Intelliswift software
Also, here is a brief description of our company. For more info, do check our website https://www.intelliswift.com Intelliswift, an LTTS company, is a global leader in Digital Product Engineering, Data & AI, and Digital Enterprise Solutions.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in