Data Engineer

Job Description :

Data Engineering :

• Design, develop, and maintain scalable batch and real-time data pipelines using Python, SQL, PySpark, and dbt.
• Build, optimize, and maintain ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Develop scalable and reusable data ingestion frameworks to ingest data from APIs, databases, cloud storage, streaming platforms, and third-party systems.
• Design, develop, and optimize enterprise-grade Snowflake Data Cloud solutions, including databases, schemas, tables, views, tasks, streams, dynamic tables, Snowpark, and stored procedures.
• Develop reusable dbt models, macros, snapshots, seeds, and tests following software engineering best practices.
• Implement data quality checks, data lineage, documentation, and automated testing using dbt.
• Design and implement dimensional data models, including Star and Snowflake schemas, for analytics and reporting.
• Optimize SQL queries and data-processing workloads to support high-performance analytical applications.
• Implement data governance, security, data masking, and role-based access control (RBAC).
• Develop reusable Python libraries, utilities, and frameworks for data processing and engineering.

AI / Machine Learning:

• Design, develop, and deploy machine learning models for predictive analytics, NLP, recommendation systems, computer vision, and Generative AI applications.
• Build scalable feature engineering and data preprocessing pipelines for machine learning workloads.
• Train, evaluate, tune, and optimize machine learning models to improve performance and business outcomes.
• Design and develop Retrieval-Augmented Generation (RAG) pipelines using vector databases and Large Language Models (LLMs).
• Integrate OpenAI, Azure OpenAI, Anthropic, and AWS Bedrock APIs into enterprise applications and AI solutions.
• Build AI agents and workflow automation solutions using frameworks such as LangChain, LangGraph, and LlamaIndex.
• Deploy machine learning and Generative AI solutions following MLOps best practices.
• Implement model monitoring, performance tracking, data/model drift detection, and automated retraining processes.

Cloud & DevOps

• Design and build cloud-native data and AI solutions on AWS and Azure.
• Develop scalable data platforms using Snowflake, Databricks, and Apache Spark.
• Design and implement data orchestration workflows using Apache Airflow, Azure Data Factory, AWS Glue, or similar orchestration tools.
• Build and maintain automated CI/CD and deployment pipelines using Azure DevOps and GitLab CI/CD.
• Monitor application, data-platform, and pipeline performance and optimize cloud resource utilization, scalability, reliability, and cost.
• Implement cloud security, monitoring, logging, and operational best practices across data and AI platforms.