BENGALURU · FULLTIME
Scraper Developer
Avolta Global Business Services
Bengaluru · onsite · Posted 13d ago
Sourced from
Undisclosed1–3 yrsfulltimeBengaluru
Your match
?
Sign in to see your match score, skill gaps & tailored resume.
Section · 01
About this role
The Scraper Developer role is focused on a single mission: implementing data extraction pipelines for new competitor websites at the rate required to meet Avoltas competitive intelligence coverage targets. Working within the framework designed by the Lead Scraping Engineer, you will onboard new competitor sites one to three per week at steady state writing the parsers, selectors and extraction logic to turn raw HTML and API responses into clean, structured product and pricing data.
This is a coding-heavy, execution-focused role. You will spend most of your day writing Python, reading browser DevTools network traces, and debugging extraction issues. The right candidate is a developer who genuinely enjoys the puzzle of getting data out of websites that dont want to give it to you and who takes pride in writing clean, well-documented, maintainable code.
Key Responsibilities
- Competitor Onboarding (Primary 70% of time): Receive a list of assigned competitor URLs and implement scrapers that extract: product name, brand, category, price (regular and promotional), currency, availability status, product URL and any other fields specified in the data schema.
- Analyse each target site using Chrome DevTools before writing any code: identify whether data is served as static HTML, AJAX-loaded JSON, or via a private API endpoint and choose the extraction approach accordingly.
- Write HTML parsers using BeautifulSoup4 and lxml: construct robust CSS and XPath selectors, handle nested elements, extract text with normalisation (strip whitespace, handle encoding, remove special characters).
- Use Scrapy as the primary scraping framework: implement Spider classes, define item schemas, configure per-spider settings (download delay, concurrent requests, retry policy).
- For JavaScript-rendered sites, use Playwright or Selenium to automate browser interaction: wait for network idle, click through tabs or load-more buttons, extract from the rendered DOM.
- Handle pagination systematically: detect pagination type (URL parameter, next-button link, infinite scroll, cursor-based API), implement accordingly, and verify total page count is extracted correctly.
- Implement basic evasion measures within team guidelines: set appropriate User-Agent headers, add randomised delays, use proxy rotation via team middleware.
- Data Quality Validation (20% of time): Write pytest unit tests for all parser functions: test against saved HTML fixtures, cover edge cases (missing field, null price, non-standard currency format).
- Run scrapers in development and verify output against a visual check of the source website before handing off to QA.
- Complete scraper documentation for every new implementation: target URL pattern, fields extracted, extraction method (HTML/API/headless), known issues, estimated run time and record count.
- Respond to QA-flagged data quality issues: investigate root cause, fix parser logic, rerun to confirm resolution.
- Maintenance Support (10% of time): Fix scrapers assigned to you when they break due to site changes: update selectors, handle new page structures, adapt to changed API endpoints.
- Escalate to Lead Scraping Engineer when breakage is caused by new anti-bot measures rather than structural changes.
- Python 3.7+ (solid working knowledge)
- BeautifulSoup4 (HTML parsing primary tool)
- lxml (XPath-based parsing, element tree traversal)
- Scrapy (spider development, items, pipelines)
- Selenium WebDriver (dynamic page interaction)
- Playwright for Python (headless browser automation)
- CSS selectors (authoring, specificity, debugging)
- Experience with anti-bot evasion basics: setting realistic headers, cookie handling, referrer management.
- Rotating proxy usage: how to configure a scraper to route through a proxy pool.
- Basic JavaScript: reading page source JS to find embedded JSON data objects (e.g. window.__INITIAL_DATA__).
- CAPTCHA solving basics: integrating a third-party solver API.
- Experience scraping e-commerce, travel, retail or comparison shopping sites.
- SQL (basic SELECT queries for verifying extracted data).
- Docker basics: running a scraper inside a container.
- 1-3 years of professional software engineering or data engineering experience.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Let the agent run this one for you.
Tailored resume, auto-apply, and referral lookup — in under 2 minutes.
Section · 02
Skills
data qualitydata extractionAutomationDockerGithubCompetitive intelligencecursorAuthoringEndpointfixturesHTMLExpress.jsEmbedded SystemsContainerdebuggingconcurrentData EngineeringCSSadvertisingInformation TechnologyAjax
Section · Company
About Avolta Global Business Services
Avolta Global Business Services
Financial Services
350
employees
1865
161 years old
Bengaluru, Karnataka
India
About
We are the Global Financial Shared Services (FSSC) team, responsible for managing and streamlining all financial operations for Avolta worldwide.
Industries
Financial ServicesAccounting & Auditing
Employees rate it well for
Senior Management9Salary8Work Culture6Work Environment6Learning6
Find them on