Data Engineer

We seek a skilled Data Engineer to build and optimize our data infrastructure. As a key contributor, you will collaborate closely with cross-functional teams to design and implement robust data pipelines that efficiently extract, transform, and load data into our AWS-based data lake and data warehouse. Your expertise will be instrumental in empowering data-driven decision making through advanced analytics and predictive modeling.

Roles & Responsibilities:

• Building and optimizing data pipelines, data warehouses, and data lakes on the AWS and Databricks platforms.
• Managing and maintaining the AWS and Databricks environments.
• Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring.
• Maintain system uptime and optimal performance.
• Working closely with cross-functional teams to understand business requirements and translate them into technical solutions.
• Exploring and implementing new tools and technologies to enhance ETL platform performance.

Functional Skills:

Must-Have Skills:

• Proficient in SQL for extracting, transforming, and analyzing complex datasets from both relational and columnar data stores. Proven ability to optimize query performance on big data platforms.
• Proficient in leveraging Python, PySpark, and Airflow to build scalable and efficient data ingestion, transformation, and loading processes.
• Ability to learn new technologies quickly.
• Strong problem-solving and analytical skills.
• Excellent communication and teamwork skills.

Good-to-Have Skills:

• Experienced with SQL/NOSQL database, vector database for large language models.
• Experienced with data modeling and performance tuning for both OLAP and OLTP databases.
• Experienced with Apache Spark, Apache Airflow.
• Experienced with software engineering best practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven, etc.), automated unit testing, and DevOps.
• Experienced with AWS, GCP, or Azure cloud services.

Education and Professional Certifications:

5 to 9 years of experience and Bachelor's degree in computer science and engineering preferred; other engineering fields are considered.

AWS Certified Data Engineer preferred.

Databricks Certificate preferred.

Soft Skills:

• Excellent analytical and troubleshooting skills.
• Strong verbal and written communication skills.
• Ability to work effectively with global, virtual teams.
• High degree of initiative and self-motivation.
• Ability to manage multiple priorities successfully.
• Team-oriented, with a focus on achieving team goals.
• Strong presentation and public speaking skills.