Crawler Engineer

We're hiring

At Cognizant, we have an ideal opportunity for you to be part of one of the largest companies in the digital sector worldwide. A Great Place To Work where we look for people who contribute new ideas, experiencing a dynamic and growing environment. At Cognizant, we promote an inclusive culture, where we value different perspectives providing career growth and development opportunities. #WelcomeToCognizant

We have an exciting opportunity for an exceptional individual to work supporting one of our clients as a Crawler Engineer.

This role focuses on enhancing and mastering Lex Machina's crawling infrastructure to support our customer-facing products. You will be responsible for designing and maintaining robust automated data collection systems in a challenging and large-scale environment. Your work will play a critical role in driving the success of our products, making this an impactful and rewarding opportunity.

1. Build, maintain, and optimize automated data collection and storage systems, ensuring reliability and scalability.
2. Master and enhance Lex Machina's crawling infrastructure to support customer-facing applications.
3. Extract and process large datasets from industries such as retail pricing, neobanks, and airlines or OTAs.
4. Debug, troubleshoot, and optimize crawlers for maximum performance and accuracy.
5. Collaborate with cross-functional teams to align data collection processes with business goals.
6. Create and maintain comprehensive documentation of crawling infrastructure and processes.
7. Communicate effectively with stakeholders by asking and answering questions, addressing issues, and providing detailed updates.

1. Bachelor's degree in computer science, Software Engineering, or a related field (or equivalent experience).
2. 3-5+ years of experience in backend software engineering, with a focus on Python in challenging environments.
3. Proven expertise in large-scale data collection, processing, and storage systems.
4. Strong problem-solving skills with an enthusiasm for detail and mastery of technical systems.
5. Excellent communication skills, including thorough documentation and the ability to ask/answer questions diligently.
6. Self-motivated, with a passion for delivering high-quality work that supports customer-facing products.

1. Experience with distributed systems and scalable architecture.
2. Knowledge of modern software development practices, such as CI/CD and Agile methodologies.
3. Familiarity with cloud environments such as AWS or Azure.

Improve your career in one of the largest and fastest-growing IT services providers worldwide. Receive ongoing support and funding with training and development plans. Have a highly competitive benefits and salary package. Get the opportunity to work for leading global companies. We are committed to respecting human rights and building a better future by helping your minds and the environment. We invest in people and their wellbeing.

We create conditions for everyone to thrive. We do not discriminate based on race, religion, color, sex, age, disability, nationality, sexual orientation, gender identity or expression, or for any other reason covered. At Cognizant, we believe that our culture makes us stronger Join us now