ETL Developer Data Ingestion & Extraction
- Techment Technology
- 8 to 12 Yrs
- Indore
- 5 months ago
- Email to a friend
- Report this job
Job Description
- Job Description We are seeking a highly skilled and experienced ETL Developer specializing in data ingestion and extraction to join our team. The ideal candidate will have 812 years of experience in building and managing scalable ETL pipelines, integrating diverse data sources, and optimizing data workflows for Snowflake. You will collaborate with cross-functional teams to extract, transform, and load large-scale datasets, ensuring data quality, consistency, and performance in a cloud-based data ecosystem. Roles & Responsibilities Data Extraction and Ingestion: Design and implement processes to extract data from various sources such as on-premise databases, cloud storage (S3, GCS), APIs, and third-party applications. Ensure seamless data ingestion into Snowflake, utilizing tools and techniques like SnowSQL, COPY INTO commands, Snowpipe, and third-party ETL tools (Matillion, Talend, Fivetran). Develop robust solutions for handling data ingestion challenges, including connectivity issues, schema mismatches, and data format inconsistencies. Data Transformation and Integration: Perform complex data transformations within Snowflake using SQL-based ELT methodologies. Implement incremental loading strategies and track data changes using Change Data Capture (CDC) techniques. Optimize transformation processes for performance and scalability, leveraging Snowflakes native capabilities such as clustering, materialized views, and UDFs. Pipeline Development and Maintenance: Design and maintain ETL pipelines capable of processing terabytes of data efficiently. Optimize ETL jobs for performance, parallelism, and data compression. Ensure error logging, retry mechanisms, and real-time monitoring for robust pipeline operation. Data Quality and Governance: Implement mechanisms for data validation, integrity checks, duplicate handling, and consistency verification. Collaborate with stakeholders to ensure adherence to data governance standards and compliance requirements. Collaboration and Documentation: Work closely with data engineers, analysts, and business stakeholders to define requirements and deliver high-quality solutions. Document data workflows, technical designs, and operational procedures Skills Experience: 812 years of experience in ETL development and data engineering, with significant experience in Snowflake. Expertise in Tools and Technologies: Snowflake: SnowSQL, COPY INTO, Snowpipe, external tables. ETL Tools: Matillion, Talend, Fivetran, or equivalent. Cloud Storage: S3, GCS, Azure Blob Storage. Databases: Oracle, SQL Server, PostgreSQL, MySQL. APIs: REST, SOAP for data extraction. Strong SQL Skills: Proficient in writing and optimizing complex SQL queries. Performance Optimization: Bulk data loading best practices: partitioning, parallel processing, compression, and small batch processing. ETL pipeline optimization for scalability and speed. Data Transformation: Proficient in ELT processes and understanding of traditional ETL differences. Experience with CDC techniques for incremental loading. Data Quality: Expertise in data validation, integrity checks, and error handling. Soft Skills: Strong analytical thinking, problem-solving abilities, and excellent communication skills. Experience 8-12 Years Location: Bhilai, Indore,
Employement Category:
Employement Type: Full timeIndustry: IT Services & ConsultingRole Category: Not SpecifiedFunctional Area: Not SpecifiedRole/Responsibilies: ETL Developer Data Ingestion & ExtractionContact Details:
Company: Techment TechnologyLocation(s): Indore+ View Contact
Keyskills: data engineering Snowflake Talend GCS Oracle SQL Server PostgreSQL MySQL REST SOAP SQL
Fraud Alert to job seekers!
₹ Not Specified
Similar positions
Techment Technology
About Techment We are an IT product development and services company with proven competency in Mobile and Web Application development. We are helping clients strategize their business roadmap through building technology agnostic solutions that fulfill their business needs. Headquartered in Bhil...
Job Listings
We are in open beta version of website, please let us know if any issues, at: info(at)indigojobs.in