Data Engineer

We are looking for a highly skilled Data Engineer with strong expertise in modern data platforms, scalable data pipelines, and cloud ecosystems, along with exposure to generative AI technologies. The ideal candidate should have hands-on experience in building large-scale ETL/ELT pipelines, distributed data processing, and cloud-native architectures while also understanding GenAI/LLM-based systems such as RAG pipelines, vector databases, embeddings, and AI-powered analytics workflows. You will work closely with data engineers, AI/ML teams, product teams, and architects to build intelligent, scalable, and high-performance data solutions.

Responsibilities:

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build and optimize ETL/ELT workflows using PySpark, SQL, and cloud-native services.
• Work with structured and semi-structured datasets across lakehouse and warehouse environments.
• Develop data ingestion frameworks for multiple enterprise data sources.
• Optimize large-scale data processing and pipeline performance.
• Build reusable and scalable data engineering components and frameworks.
• Collaborate with analytics, AI/ML, and product teams for data-driven solutions.
• Ensure data quality, governance, monitoring, and observability standards.
• Work with cloud ecosystems such as AWS, Azure, Databricks, or GCP.
• Support deployment and orchestration workflows using CI/CD and automation tools.
• Contribute to AI-powered applications and GenAI-enabled analytics systems.
• Work with vector databases, embeddings, and document-processing pipelines for AI use cases.

Requirements:

• Strong hands-on experience in Python and SQL.
• Strong experience with PySpark and Databricks.
• Experience building scalable ETL/ELT and data pipelines.
• Good understanding of data warehousing and lakehouse concepts.
• Experience with AWS / Azure / GCP cloud platforms.
• Hands-on experience with distributed data processing systems.
• Experience with orchestration tools such as Airflow or similar schedulers.
• Strong understanding of data modeling, performance tuning, and optimization.
• Experience working with APIs, ingestion frameworks, and large datasets.
• Strong problem-solving and debugging skills.
• Good understanding of CI/CD, Git, and deployment workflows.

Good to Have Skills:

• Experience with generative AI and LLM-based applications.
• Knowledge of RAG architectures and AI-powered search systems.
• Experience with LangChain, LlamaIndex, OpenAI APIs, or Hugging Face.
• Understanding of vector databases such as FAISS, Pinecone, ChromaDB, or Weaviate.
• Exposure to prompt engineering and AI orchestration workflows.
• Experience integrating AI/ML models into enterprise data platforms.
• Knowledge of Docker, Kubernetes, and containerized deployments.
• Experience with streaming technologies such as Kafka or Event Hub.

Preferred Experience:

• 5+ years of experience in data engineering.
• Experience working on enterprise-scale analytics or AI platforms.
• Exposure to GenAI-enabled enterprise applications is a strong plus.