Custom Software Engineering Lead

Project Role : Custom Software Engineering Lead
Project Role Description : Own the technical direction and architecture of custom software solutions, leading teams through design and delivery. Set development standards and ensure code quality, scalability, and performance aligned to business objectives.
Must have skills : Microsoft Azure Databricks
Good to have skills : NA
Minimum 7.5 year(s) of experience is required
Educational Qualification : 15 years full time education

Summary:
As a Custom Software Engineering Lead, a typical day involves overseeing the technical direction and architectural design of custom software solutions. This role requires guiding and supporting teams throughout the design and delivery phases, ensuring that development standards are maintained. The position focuses on aligning software performance and scalability with business goals, while fostering collaboration and innovation within the team to meet project objectives effectively.

Roles Responsibilities:
- Expected to be an SME, collaborate and manage the team to perform.
- Responsible for team decisions.
- Engage with multiple teams and contribute on key decisions.
- Provide solutions to problems for their immediate team and across multiple teams.
- Drive continuous improvement initiatives to enhance software development processes and team productivity.
- Mentor and support junior team members to foster professional growth and skill development.

a bachelor s degree or higher in Computer Science/ Engineering/ Science
data engineer with 8 to 10 years of experience, having 5+ experience in Azure cloud native services.
Experience in implementing and Designing Big Data solutions in Azure Databricks for Batch and Real Time Analytics uses cases.
Experience in core concepts like Lakehouse, Delta Lake, Data Mesh, Data Virtualization, Dimensional Modelling techniques, Data Aggregation techniques.
Experience in programming skills in languages such as Python/Pyspark and frameworks and libraries based on them with a focus on data engineering.
Extensive experience designing, building, and operating robust distributed data platforms (e.g., Spark, Kafka, Flink, HBase) and handling data at the petabyte scale.
Experience in data engineering on Apache Spark and populate downstream batch data stores (such as data mart) for BI use cases to generate downstream feeds (ie., flat wide tables or compressed for Data Science use cases
Experience in Real-time service integration to process business events off Kafka and persist in ADLS Gen 2 using Databricks.
Experience in Near real-time stream processing to derive features for ML model inference.
Experience in Hadoop based implementations, Python, Spark, SQL, Unix, and Hive
Extensive experience and knowledge of a variety of data technologies and frameworks such as Delta Lake, Datahub, Apache Spark, Azure Databricks, Databricks Genie
Strong familiarity with the concepts of RDBMS, Data Lake, Data Warehouse, Data Lakehouse, and Medallion Architecture.
Knowledge of at least one workflow orchestration tools such as Apache Airflow, Luigi, Oozie, AWS Glue or similar frameworks
Experience in Azure data bricks performance optimization techniques (Liquid clustering, vacuums, Z-ordering etc)
Experience in Azure Services like Azure Data Factory, Azure Data Bricks (Scala/Python), ADLS Gen2, Azure SQL Database, SQL/T-SQL
Experience working with the DevOps teams to drive operationally excellent infrastructure.
Knowledge on Microsoft Fabric and its framework
Basic understanding of data quality dimensions like Consistency, Completeness, Accuracy, Lineage etc.
Understanding on different monitoring and logging solutions on Azure
Good to have experience on Data Visualization Tools like Power BI and Tableau
Good to have experience of Apache Flink/Apache Kafka and the ELK stack (Elasticsearch, Logstash Kibana)
great communicator, you know how to speak with people at all levels.
team player, working with a globally distributed team.


Professional Technical Skills:
- Must To Have Skills: Proficiency in Microsoft Azure Databricks.
- Experience with cloud-based data engineering and analytics platforms.
- Strong knowledge of distributed computing and big data processing frameworks.
- Ability to design scalable and high-performance data pipelines.
- Familiarity with data security and governance best practices.
- Experience in troubleshooting and optimizing complex data workflows.
- The candidate should have minimum 5 years of experience in Microsoft Azure Databricks.
- This position is based at our Pune office.
- A 15 years full time education is required.




Qualification
15 years full time education