Lead Data Engineer
- Smartavya Analytica
- 8 - 10 years
- Pune
- 25 days ago
- Email to a friend
- Report this job
Job Description
We are seeking a hands-on Lead Data Engineer to drive the design and delivery of scalable, secure data platforms on Google Cloud Platform (GCP). In this role you will own architectural decisions, guide service selection, and embed best practices across data engineering, security, and performance disciplines. You will partner with data modelers, analysts, security teams, and product owners to ensure our pipelines and datasets serve analytical, operational, and AI/ML workloads with reliability and cost efficiency. Familiarity with Microsoft Azure data services (Data Factory, Databricks, Synapse, Fabric) is valuable, as many existing workloads will transition from Azure to GCP.
Key Responsibilities
- Lead end-to-end development of high-throughput, low-latency data pipelines and lake-house solutions on GCP (BigQuery, Dataflow, Pub/Sub, Dataproc, Cloud Composer, Dataplex, etc.).
- Define reference architectures, technology standards for data ingestion, transformation, and storage.
- Drive service-selection trade-offscost, performance, scalability, and securityacross streaming and batch workloads.
- Conduct design reviews and performance tuning sessions; ensure adherence to partitioning, clustering, and query-optimization standards in BigQuery.
- Contribute to long-term cloud data strategy, evaluating emerging GCP features and multi-cloud patterns (Azure Synapse, Data Factory, Purview, etc.) for future adoption.
- Lead the code reviews and oversee the development activities delegated to Data engineers.
- Implement best practices recommended by Google Cloud
- Provide effort estimates for the data engineering activities
- Participate in discussions to migrate existing Azure workloads to GCP, provide solutions to migrate the work loads for selected data pipelines
Must-Have Skills
- 810 years in data engineering, with 3+ years leading teams or projects on GCP.
- Expert in GCP data services (BigQuery, Dataflow/Apache Beam, Dataproc/Spark, Pub/Sub, Cloud Storage) and orchestration with Cloud Composer or Airflow.
- Proven track record designing and optimizing large-scale ETL/ELT pipelines (streaming + batch).
- Strong fluency in SQL and one major programming language (Python, Java, or Scala).
- Deep understanding of data lake / lakehouse, dimensional & data-vault modeling, and data governance frameworks.
- Excellent communication and stakeholder-management skills; able to translate complex technical topics to non-technical audiences.
Nice-to-Have Skills
- Hands-on experience with Microsoft Azure data services (Azure Synapse Analytics, Data Factory, Event Hub, Purview).
- Experience integrating ML pipelines (Vertex AI, Dataproc ML) or real-time analytics (BigQuery BI Engine, Looker).
- Familiarity with open-source observability stacks (Prometheus, Grafana) and FinOps tooling for cloud cost optimization.
Preferred Certifications
- Google Professional Data Engineer (strongly preferred) or Google Professional Cloud Architect
- Microsoft Certified: Azure Data Engineer Associate (nice to have)
Education
- Bachelors or Masters degree in Computer Science, Information Systems, Engineering, or a related technical field. Equivalent professional experience will be considered.
Job Classification
Industry: IT Services & ConsultingFunctional Area / Department: Data Science & AnalyticsRole Category: Data Science & Machine LearningRole: Data EngineerEmployement Type: Full timeContact Details:
Company: Smartavya AnalyticaLocation(s): Pune
Keyskills: Microsoft Azure Java Data Factory Purview Data engineering Prometheus Dataproc ML Looker Grafana SQL Azure Synapse Analytics Event Hub Vertex AI BigQuery BI Engine Python
Similar positions
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in