Data Engineer

Role & responsibilities

Data Engineer

• Strong hands-on experience with PySpark (Spark SQL, Dataframes and RDD).
• Experience with Hive, HDFS and Distributed Storage systems.
• Hands-on experience with Hive (HiveQL, table optimization, partition management).
• Strong knowledge of MySQL (indexes, query tuning, schema design, stored procedures).
• Strong hands-on experience with PySpark and Spark optimization techniques (RDD, DataFrame APIs, Spark SQL, partitioning, caching).
• Proficiency in Python (Experience on Pandas and Numpy) •Proficiency in SQL (MySQL, Oracle, SQL Server, PostgreSQL, etc.).
• Hands on experience Building end to end Scalable Data Pipelines.
• Optimize Spark jobs for performance, memory management and resource utilization.
• Implement data validation, quality checks and error handling mechanism.
• Experience with Data Warehousing concepts (Star schema, SCD, facts, dimensions).
• Knowledge of ETL best practices, data modeling, and metadata management.
• Experience working with:
• Flat files (CSV, TXT)
• Excel/XML/JSON
• APIs / REST services
• Cloud platforms (AWS, Azure, GCP) optional but preferred Experience with version control (Git, SVN).
• Knowledge of Airflow/Oozie or other orchestration tools.