Research Engineer - Web Crawlers

📋 Description

• Build and operate large-scale, distributed web crawlers
• Fetch and extract data across billions of pages reliably
• Address content extraction, deduplication, freshness and recrawl
• Design pipelines for audio/video/multilingual data
• Create tooling to request, monitor, explore crawled data

🎯 Requirements

• Hands-on experience building/scaling web crawlers
• Distributed systems expertise (Kubernetes, queues, pipelines)
• Ability to evaluate data quality, coverage, and compliance

🎁 Benefits

• Innovative culture and impact-focused work
• Growth opportunities and learning stipends
• Annual offsite and co-working stipend