Data Engineer AWS Data Stack, APIs & Agentic AI

Data Engineer

AWS Data Stack, APIs & Agentic AI

Location : Bangalore

Experience : 7 years above

Level : SAL2

Key Responsibilities

• Build and maintain batch and incremental pipelines in Python/PySpark on AWS Glue. You'll own their freshness, completeness and cost.
• Fetch data from APIs: build reliable ingestion from internal and third-party REST and GraphQL APIs. Handle authentication (API keys, OAuth 2.0), pagination, rate limits, retries with backoff, incremental extraction and schema changes.
• Integrate APIs: connect the data platform with source systems, SaaS platforms and partner endpoints using APIs, webhooks and event-driven flows (EventBridge, SQS, Lambda).
• Build APIs: design, build and deploy data APIs that serve curated datasets to applications and other teams, using Amazon API Gateway, Lambda and FastAPI. Own versioning, authentication, documentation (OpenAPI) and monitoring.
• Bring data from internal systems, APIs and files into the S3 data lake. Manage partitioning, schema changes and Apache Iceberg tables.
• Turn raw data into curated, documented layers (raw → staging → mart) that analysts can query directly in Athena.
• Use SageMaker Unified Studio to develop, catalogue and publish data assets: projects, SageMaker Lakehouse, SageMaker Catalog, notebooks and query editors.
• Make governed data available to AI agents. Build MCP servers or tool endpoints on top of curated datasets, and support Amazon Bedrock agents, knowledge bases and RAG use cases.
• Use AI coding agents (Amazon Q Developer, Claude, Kiro, Cursor or similar) to develop, refactor, test and document code. Code review and testing are still required before anything goes live.
• Orchestrate workflows with Airflow (MWAA), Step Functions or Glue Workflows, including retries, backfills and alerting.
• Set up data quality checks for schema, freshness, row counts and reconciliation with source systems. Trace issues to their root cause.
• Apply security and governance: least-privilege IAM, Lake Formation / SageMaker Catalog permissions, encryption, secrets management for API credentials, PII handling and retention.
• Tune Athena and Glue for performance and cost through partition design, file compaction, columnar formats and workgroup limits.
• Keep all code in Git with CI/CD, and deploy infrastructure as code.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering or a related field.
• 3+ years of hands-on data engineering on production pipelines.
• Python (strong): production-quality code, pytest, PySpark, pandas, boto3.
• SQL (strong): window functions, CTEs, incremental logic, tuning queries on large partitioned tables.
• APIs:
• Building APIs: has designed and built REST APIs with FastAPI or Flask, and deployed them on AWS (API Gateway + Lambda, or containers)
• Fetching data from APIs: has built production ingestion from REST/GraphQL APIs with requests/httpx. Handles pagination, rate limits, retries and incremental loads
• API integration: OAuth 2.0 and API-key authentication, JSON/XML payloads, webhooks, and OpenAPI/Swagger specs
• AWS data stack:
• AWS Glue: ETL jobs, crawlers, Data Catalog, Glue Studio
• Amazon Athena: query authoring, workgroups, federated queries, cost controls
• Amazon SageMaker Unified Studio: projects, Lakehouse, Catalog, unified notebooks and query editor
• Amazon S3: data lake layout, partitioning, lifecycle policies
• Supporting services: Lambda, API Gateway, Step Functions, EventBridge, IAM, CloudWatch, Secrets Manager, Lake Formation
• Agentic and AI tools: has used AI coding agents in real work, and understands Amazon Bedrock (agents, knowledge bases), Model Context Protocol (MCP), tool and prompt design, and RAG.
• Workflow orchestration with Airflow, Step Functions or an equivalent tool.
• Data modelling (dimensional and event-based), plus Git and CI/CD practice.

Preferred Qualifications

• Experience running Apache Iceberg in production (compaction, time travel).
• Streaming with Kinesis, MSK (Kafka) or Spark Structured Streaming.
• Amazon AppFlow or managed connectors for SaaS ingestion (e.g. Salesforce, Google Analytics).
• GraphQL API development; async Python (asyncio, aiohttp) for high-volume API extraction.
• Redshift / Redshift Serverless or EMR.
• dbt; Terraform or CDK.
• Has built AI agents with Bedrock AgentCore, Strands Agents SDK, LangGraph or custom MCP servers.
• Vector stores and embedding pipelines (OpenSearch Serverless, pgvector).
• AWS certification: Data Engineer Associate or Machine Learning Engineer Associate.
• Experience with a BI tool such as QuickSight or Power BI.

Competencies

• Delivery focus: ships working pipelines and sees operational issues through to resolution.
• Rigour: checks numbers before publishing them.
• Judgment with AI tools: uses agents to move faster, but reviews, tests and owns everything that goes to production.
• Curiosity: keeps up with the fast-changing AWS data and AI stack.
• Collaboration: communicates clearly with analysts and the business, and documents work so others can reuse it.