Tech Associate

Job Description

We are looking for a highly skilled Backend Engineer with expertise in web crawling, web scraping, and data extraction to join our team. The ideal candidate will design, develop, and maintain large-scale web crawlers and automated data collection systems while ensuring efficiency, scalability, and reliability. Responsibilities: - Design and implement scalable web crawlers to extract data from various websites and APIs. - Develop robust data parsing and cleaning pipelines. - Implement mechanisms to handle anti-scraping measures (e. g., CAPTCHA, IP blocking, User-Agent rotation, proxies, etc. ). - Optimize crawling efficiency and data storage for large-scale data extraction. - Ensure compliance with legal and ethical guidelines related to web scraping. - Work with databases (SQL/NoSQL) to store and retrieve extracted data efficiently. - Implement distributed crawling techniques using cloud services (AWS, GCP, Azure). - Maintain documentation and improve code quality through best practices and code reviews. Requirements: - 3+ years of experience in backend development, with a focus on web crawling and web scraping. - Strong proficiency in Python (Scrapy, BeautifulSoup, Requests, Selenium, Playwright, etc. ). - Experience with Java-based crawling frameworks (e. g., Apache Nutch, Jsoup, StormCrawler) is a plus. - Solid understanding of HTTP protocols, web technologies, and browser behavior. - Experience handling headless browsers and automated scraping mechanisms. - Familiarity with distributed systems and large-scale data extraction. - Strong knowledge of database technologies (SQL, NoSQL, Elasticsearch, MongoDB, Redis, etc. ). - Experience working with asynchronous programming (asyncio, aiohttp, Celery, etc. ). - Knowledge of cloud platforms (AWS Lambda, S3 EC2 GCP, etc. ) for scalable crawling solutions. - Experience with Machine Learning/NLP for data processing and text extraction. - Knowledge of containerization and orchestration tools like Docker & Kubernetes. - Exposure to data engineering tools like Apache Kafka, Spark, or Hadoop. Company Summary Bessemer Venture Partners was born from innovations in steel that literally forged modern building and manufacturing. Today, we work with people who want to create revolutions of their own. Our partners help founders lay enduring foundations to create companies that matter, starting with seed and Series A investments and sticking with our companies at every stage of their growth. We've been part of over 120 IPOs in the last 50 years, including Shopify, Yelp, LinkedIn, Skype, LifeLock, Twilio, SendGrid, DocuSign, Wix, and MindBody, and we're not stopping anytime soon. Bessemer's 15 investing partners operate from offices in Silicon Valley, San Francisco, New York City, Boston, Israel, and India.

Required Skills

[Django, Python, FastAPI, Flask, Java, Next.js]

Additional Information

NA