We are seeking a skilled and motivated Web Crawler to join our team. The ideal candidate will have 3 to 5 years of experience in developing and maintaining robust web scraping solutions. You will be responsible for designing, implementing, and optimizing web crawlers to extract valuable data from diverse online sources. This role requires a strong understanding of web technologies, data handling, and problem-solving skills.
Design, develop, and maintain efficient and scalable web crawlers using Python, Mozenda,
etc.
Utilize web scraping frameworks such as Scrapy, Beautiful Soup, or Selenium to extract
data from websites.
Implement and optimize data extraction logic using XPath, CSS selectors, and JSONPath.
Understand and effectively navigate website structures and implement strategies to
bypass anti-scraping measures.
Test, maintain, and troubleshoot web scraping processes to identify and resolve any issues
or errors.
Ensure data integrity and quality through rigorous testing and validation.
Monitor and troubleshoot crawler performance, identifying and resolving complex
technical issues.
Work with SQL and NoSQL databases to store and manage extracted data.
Collaborate with cross-functional teams to define data requirements and deliver
actionable insights.
Maintain comprehensive documentation for all crawler development and maintenance
activities.
Demonstrate a strong understanding of the HTTP protocol and web technologies.