Senior Azure Databricks Infrastructure Engineer
- ATT Communication
- 8 - 12 years
- Hyderabad
- 6 days ago
- Email to a friend
- Report this job
Job Description
Job Title: Databricks Infrastructure Engineer
Location: Hyderabad/Bengaluru
Job Summary:
We are looking for a skilled Databricks Infrastructure Engineer to design, build, and manage the cloud infrastructure that supports Databricks development efforts. This role will focus on creating and maintaining scalable, secure, and automated infrastructure environments using Terraform and other Infrastructure-as-Code (IaC) tools. The infrastructure will enable data engineers and developers to efficiently create notebooks, pipelines, and ingest data following the Medallion architecture (Bronze, Silver, Gold layers). The ideal candidate will have strong cloud engineering skills, deep knowledge of Terraform, and hands-on experience with Databricks platform provisioning.
Key Responsibilities:
Infrastructure Design & Provisioning:
- Design and implement scalable and secure infrastructure environments to support Databricks workloads aligned with the Medallion architecture.
- Develop and maintain Infrastructure-as-Code (IaC) scripts and templates using Terraform and/or ARM templates for provisioning Databricks workspaces, clusters, storage accounts, networking, and related Azure resources.
- Automate the setup of data ingestion pipelines, storage layers (Bronze, Silver, Gold), and access controls necessary for smooth data operations.
Platform Automation & Optimization:
- Create automated deployment pipelines integrated with CI/CD tools (e.g., Azure DevOps, Jenkins) to ensure repeatable and consistent infrastructure provisioning.
- Optimize infrastructure configurations to balance performance, scalability, security, and cost-effectiveness.
- Monitor infrastructure health and perform capacity planning to support evolving data workloads.
- Implement and maintain backup, recovery, and disaster recovery strategies for Databricks environments.
- Optimize performance of Databricks clusters, jobs, and SQL endpoints.
- Automate routine administration tasks using scripting and orchestration tools.
- Troubleshoot platform issues, identify root causes, and implement solutions.
Security & Governance:
- Implement security best practices including network isolation, encryption, identity and access management (IAM), and role-based access control (RBAC) within the infrastructure.
- Collaborate with governance teams to embed compliance and audit requirements into infrastructure automation.
Collaboration & Support:
- Work closely with data engineers, data scientists, and platform administrators to understand infrastructure requirements and deliver solutions that enable efficient data engineering workflows.
- Provide documentation and training on infrastructure setup, usage, and best practices.
- Troubleshoot infrastructure issues and coordinate with cloud and platform support teams for resolution.
- Stay up to date with Databricks features, releases, and best practices
Required Qualifications:
- 8-12 years of experience in Databricks and cloud infrastructure engineering, preferably with Azure
- Strong hands-on experience writing Infrastructure-as-Code using Terraform; experience with ARM templates or CloudFormation is a plus.
- Practical knowledge of provisioning and managing Databricks environments and associated cloud resources.
- Familiarity with Medallion architecture and data lake house concepts.
- Experience with CI/CD pipeline creation and automation tools such as Azure DevOps, Jenkins, or GitHub Actions.
- Solid understanding of cloud networking, storage, security, and identity management.
- Proficiency in scripting languages such as Python, Bash, or PowerShell.
- Strong collaboration and communication skills to work across cross-functional teams.
Preferred Skills:
- Prior experience working with Databricks platform, including workspace and cluster management.
- Knowledge of data governance tools and practices.
- Experience with monitoring and logging tools (e.g., Azure Monitor, CloudWatch).
- Exposure to containerization and orchestration technologies such as Docker and Kubernetes.
- Understanding of data ingestion frameworks and pipeline orchestration tools like Apache Airflow or Azure Data Factory.
Location:
IND:AP:Hyderabad / Argus Bldg 4f & 5f, Sattva, Knowledge City- Adm: Argus Building, Sattva, Knowledge City Job Classification
Industry: Telecom / ISPFunctional Area / Department: Engineering - Software & QARole Category: DevOpsRole: Site Reliability EngineerEmployement Type: Full timeContact Details:
Company: ATT CommunicationLocation(s): Hyderabad+ View Contact
Keyskills: orchestration Access management Powershell Disaster recovery Apache SQL Python Auditing Identity management Capacity planning
Fraud Alert to job seekers!
₹ Not Disclosed
Similar positions
ATT Communication
ATT Inc. (originally the American Telephone and Telegraph Company) is an American multinational telecommunications. It is the worlds largest telecommunications company and the largest provider of mobile telephone services in the U.S.
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in