Senior Data Engineer (GCP)

About the Role: Pixeldust Technologies is looking for a highly skilled

Senior Data Engineer to join our growing Data & AI team. You will be responsible for designing, building, and optimizing scalable data platforms, real-time data pipelines, and cloud-native data engineering solutions on

Google Cloud Platform (GCP) . The ideal candidate should have strong expertise in

Python, PySpark, SQL, GCP Data Engineering services , and modern microservices architecture. Experience with

streaming pipelines, Graph Databases, and Agentic AI frameworks will be an added advantage. This role offers the opportunity to work on cutting-edge AI, Analytics, and Data Engineering projects involving large-scale datasets and real-time processing. Key Responsibilities: Data Engineering & Architecture:

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build robust ETL/ELT pipelines using Python, PySpark, and SQL.
• Develop high-performance data processing frameworks for structured and unstructured datasets.
• Design scalable cloud-native data architectures on Google Cloud Platform. Google Cloud Platform: Develop and manage solutions using:
• BigQuery
• Cloud Storage (GCS)
• Cloud Run
• Pub/Sub
• Dataflow
• Cloud Functions
• Cloud Scheduler
• Secret Manager
• IAM
• Cloud Monitoring
• Logging
• Artifact Registry Optimize cloud infrastructure for scalability, reliability, and cost efficiency. Streaming & Data Pipelines:
• Build streaming data pipelines using Pub/Sub, Kafka, and Dataflow.
• Develop batch ingestion pipelines from multiple enterprise data sources.
• Optimize pipeline performance and reduce execution time.
• Implement monitoring, retry mechanisms, and error handling. Microservices Development: Develop REST APIs and data services using:
• FastAPI
• Flask Build scalable microservices supporting AI and Data Engineering workflows. Data Modeling & Warehousing:
• Design dimensional and normalized data models.
• Build enterprise-grade data warehouses in BigQuery.
• Optimize partitioning, clustering, indexing, and query performance.
• Implement metadata management and data governance practices. Performance Optimization:
• Optimize SQL queries and PySpark jobs.
• Improve data pipeline performance and throughput.
• Drive cloud cost optimization initiatives.
• Implement monitoring and alerting for production workloads. AI & Emerging Technologies (Preferred) Exposure to:
• Neo4j Graph Database
• Knowledge Graphs
• Model Context Protocol (MCP)
• Agentic AI
• Retrieval-Augmented Generation (RAG)
• Vector Databases Experience working on AI-powered data engineering solutions is a plus. DevOps & Deployment:
• Build CI/CD pipelines using GitHub Actions.
• Implement Infrastructure as Code (Terraform preferred).
• Deploy containerized applications using Docker and Cloud Run.
• Follow DevOps best practices for automated testing and deployments. Leadership & Collaboration:
• Mentor junior engineers and conduct code reviews.
• Collaborate with Product Managers, Data Scientists, AI Engineers, and Architects.
• Participate in Agile ceremonies including sprint planning, stand-ups, retrospectives, and backlog refinement.
• Drive engineering best practices and technical excellence. Required Skills: Programming:
• Python (Expert)
• SQL (Expert)
• PySpark (Expert) Google Cloud Platform Strong hands-on experience with:
• BigQuery
• Cloud Run
• Cloud Storage
• Pub/Sub
• Dataflow
• Cloud Functions
• Cloud Scheduler
• IAM
• Monitoring & Logging Frameworks:
• FastAPI
• Flask Streaming Technologies:
• Kafka
• Pub/Sub
• Dataflow Data Engineering:
• ETL/ELT Pipelines
• Batch Processing
• Streaming Pipelines
• Data Warehousing
• Data Modeling
• Data Integration

DevOps:

• Git
• GitHub Actions
• Docker
• CI/CD
• Deployment Strategies Preferred Skills:
• Neo4j
• Knowledge Graphs
• MCP (Model Context Protocol)
• Agentic AI
• Vector Databases
• RAG Architecture
• Terraform
• Kubernetes
• Cloud Run
• AI/ML Pipelines Qualifications:
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 6+ years of experience in Data Engineering.
• Experience delivering enterprise-scale data platforms on GCP. What We're Looking For: We're looking for someone who:
• Thinks like a software engineer and an architect.
• Enjoys solving complex data engineering problems.
• Has experience building scalable cloud-native systems.
• Takes ownership and drives projects independently.
• Mentors teammates and promotes engineering excellence.
• Is passionate about modern AI, Data Engineering, and Cloud technologies. Why Join Pixeldust Technologies?
• 🚀 Work on cutting-edge AI, Data Engineering, and GenAI projects.
• ☁️ Build cloud-native solutions using the latest GCP technologies.
• 📈 Opportunity to work on enterprise-scale data platforms.
• 🤝 Collaborative and innovation-driven work culture.
• 📚 Continuous learning and exposure to emerging technologies like Agentic AI, Knowledge Graphs, and MCP.
• 💡 High-impact role with opportunities for technical leadership and career growth.