Senior Data Engineer

We are seeking a talented AI and Data Engineer to join our team. In this role, you will design, build, and maintain robust data pipelines while developing and deploying cutting-edge AI solutions. You will work at the intersection of data engineering and artificial intelligence, leveraging modern cloud platforms and AI frameworks to drive business value.

The candidate will have responsibilities across the following functions:

Data Infrastructure and Pipeline Development:

• Design, develop, and maintain scalable ETL/ELT pipelines to process large volumes of user interaction data, content metadata, and streaming analytics.
• Build and optimise data warehouses and data lakes to support both real-time and batch processing requirements.
• Implement data quality monitoring and validation frameworks to ensure data accuracy and reliability.
• Develop automated data ingestion systems from various sources, including mobile apps, web platforms, and third-party integrations.

Analytics and Reporting Infrastructure:

• Create and maintain data models that support business intelligence, user analytics, and content performance metrics.
• Build self-service analytics platforms enabling stakeholders to access insights independently.
• Implement real-time dashboards and alerting systems for key business metrics.
• Support A/B testing frameworks and experimental data analysis requirements.

Data Architecture and Optimisation:

• Collaborate with software engineers to optimise database performance and query efficiency.
• Design data storage solutions that balance cost, performance, and accessibility requirements.
• Implement data governance practices including data cataloguing, lineage tracking, and access controls.
• Ensure GDPR and data privacy compliance across all data systems.

Collaboration and Support:

• Work closely with data scientists, product managers, and analysts to understand data requirements.
• Participate in code reviews and maintain high standards of code quality and documentation.
• Mentor junior team members and contribute to knowledge-sharing initiatives.

Requirements:

• Programming Languages: Proficiency in Python and SQL.
• Big Data Technologies: Hands-on experience with Apache Spark, Kafka, Airflow, and distributed computing frameworks.
• Cloud Platforms: Strong experience with AWS(S3 EMR, etc. )
• Database Systems: Expertise in both SQL (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra, Redis)databases.
• Data Warehousing: Experience with modern data warehouse solutions like Databricks, Snowflake or BigQuery.
• Containerization: Proficiency with Docker and Kubernetes for deploying data applications.
• 4 to 6 years of experience in data engineering or related roles.
• Proven track record of building and maintaining production data pipelines at scale.
• Experience with streaming data processing and real-time analytics systems.
• Strong understanding of data modelling, schema design, and data architecture principles.
• Experience with version control systems (Git) and CI/CD pipelines.

Advanced Technical Skills:

• Understanding of data mesh architecture and domain-driven data design.
• Experience with data privacy and security implementations.
• Agentic AI automation approach for Data and Analytics.