AI Data Engineering and Sr Cloud Specialist

Job Description

Responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines and infrastructure to support autonomous driving research and development.
• Extract, transform, and load (ETL) data from various sources, including sensors, simulations, and databases.Collaborate with data scientists, researchers, and engineers to understand data requirements and translate them into technical solutions.
• Develop and optimize data storage and retrieval strategies, ensuring data quality and integrity.
• Implement data governance and security measures to protect sensitive data.
• Monitor and troubleshoot data pipelines and systems to identify and resolve issues promptly.
• Continuously explore and evaluate new data technologies and tools to improve data processing efficiency and scalability.
• Mentor and guide junior data engineers in their professional development.

Mandatory Core Skills & Competencies:

• Strong proficiency in Python or another programming language commonly used in data engineering.
• Expertise in data engineering tools and frameworks, such as Apache Spark, Hadoop, Kafka, and Airflow.
• Deep understanding of data warehousing and data lake concepts.
• Experience with cloud platform (Azure) and cloud-based data services.Knowledge of SQL and NoSQL databases.
• Site Reliability Engineering.
• Excellent problem-solving and analytical skills.
• Ability to work independently and as part of a cross-functional team.Strong communication and documentation skills.

Nice to Have Skills:

• Experience with machine learning and deep learning frameworks (TensorFlow, PyTorch).
• Familiarity with autonomous driving technologies and concepts.Knowledge of data visualization tools (e.g., Tableau, Matplotlib).
• Experience with real-time data processing and streaming technologies.
• Understanding of data quality and validation techniques.Experience with containerization technologies (Docker, Kubernetes).
• Familiarity with CI/CD pipelines and DevOps practices.
• Experience with data governance and compliance frameworks (e.g., GDPR, CCPA).Ph.D. or Master's degree in Computer Science, Data Engineering, or a related field.Publications in relevant conferences or journals.

Qualifications

BE M Tech PhD in Computer Science

Additional Information

6 years