Senior Data Engineer

We are seeking a skilled Data Specialist (Senior Data Engineer) to join our Financial Systems Tech Support - CAT IT Division.

We are seeking a highly motivated and detail-oriented Junior Data Engineer to join our dynamic team. The ideal candidate will have a good foundation in data engineering principles, a keen analytical mindset, and the ability to work collaboratively in a fast-paced environment. As a Junior Data Engineer, you will play a critical role in designing, developing, and maintaining data pipelines and infrastructure to support our organization's data needs.

What you will do:

• Design, develop, and maintain scalable and reliable data pipelines to ingest, process, and store large volumes of data from various sources.
• Implement data workflows using ETL (Extract, Transform, Load) processes to ensure data quality and integrity.
• Collaborate with data architects, product owners, business analysts, and other stakeholders to understand data requirements and deliver data solutions that meet their needs.
• Integrate data from multiple sources, including databases, APIs, and external data providers, into a unified data platform.
• Manage and optimize data storage solutions, ensuring efficient data retrieval and processing.
• Implement data quality checks and validation processes to ensure the accuracy and consistency of data.
• Adhere to data governance policies and best practices to maintain data security and compliance.
• Monitor and optimize data pipeline performance to ensure timely and efficient data processing.
• Identify and resolve performance bottlenecks and system issues to maintain data pipeline reliability.
• Document data engineering processes, workflows, and solutions for future reference and knowledge sharing.
• Collaborate with cross-functional teams to support data-driven decision-making and project initiatives.

What you will have:

• Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field.
• 5-8 years of experience in data engineering, data analysis, or software development.
• Strong experience in advanced SQL and databases such as Oracle, Teradata, and Snowflake.
• Experience in data integration and hands-on experience in ETL tools such as Datastage, Informatica, and SnapLogic.
• Ability to transform technical requirements into data collection queries.
• Should be capable of working with business and other IT teams and converting requirements into queries.
• Good understanding of ETL architecture and design.
• Good knowledge of databases, SQL, or PL/SQL.
• Good experience with CI/CD tools such as GitHub Actions and AzDO pipelines.
• Good analytical and problem-solving skills with attention to detail.
• Excellent communication and interpersonal skills to collaborate effectively with cross-functional teams.
• Ability to manage multiple tasks and prioritize workload in a fast-paced environment.
• Eagerness to learn new technologies and stay updated with industry trends.

Good to have:

• Knowledge of Unix commands.
• Experience with AWS Glue.
• Knowledge of Qlik Replication Tool.

Additional role details:

• This position requires candidates to work a 5-day-a-week schedule in the office.
• Shift timing: 01:00-10:00 PM IST.

Skills desired:

• Value Realization - Level: Basic Understanding
  • Describes the importance of value realization milestones for long-term projects.
  • Assists in measuring and attaining post-project implementation value.
  • Identifies process changes or business value integration in change initiative plans and activities.
  • Contributes to project team communication regarding the linkage of a change initiative to its associated business goals and case.
• Communicating Complex Concepts - Level: Working Knowledge
  • Uses comparisons to familiar ideas when introducing a technical or novel feature.
  • Separates fundamental concepts from supporting details in explaining a product or service.
  • Explains products and services with little or no use of technical jargon or advanced vocabulary.
  • Emphasizes the most important facts or features of a product or service.
  • Perceives lack of audience comprehension and further simplifies explanation when needed.
• Agile Development - Level: Working Knowledge
  • Explains a specific agile process and its associated checkpoints and deliverables.
  • Works with a specific agile development methodology.
  • Interprets metrics used to measure progress and effectiveness.
  • Defines key selection and qualifying criteria for projects suitable for agile development.
  • Applies major tools and techniques associated with the specific methodology.
• Cloud Computing - Level: Working Knowledge
  • Designs and implements a cloud-based solution to meet a specific business need or requirement.
  • Configures and deploys cloud-based infrastructure, such as virtual machines, storage, and networks.
  • Migrates applications and data to a cloud-based environment, including data migration and integration.
  • Implements cloud-based security measures, such as access controls, encryption, and firewalls.
  • Troubleshoots and resolves common cloud-based issues, such as connectivity problems and performance degradation.
• Database Design (Physical) - Level: Working Knowledge
  • Researches relationships between different business data sets.
  • Participates in the establishment of data structures based on their relationships.
  • Adheres to an organization's database design policies.
  • Utilizes database design tools and techniques.
  • Tests the effectiveness of the database before its release for business use.
• ETL Process - Level: Working Knowledge
  • Ensures the accuracy and effectiveness of data provided for the warehouse.
  • Loads selected data into the warehouse on a regular basis.
  • Derives necessary business or technical information from original data.
  • Acquires a large amount of data from different sources/systems.
  • Solves technical and administrative problems during the ETL process.
• Information Management - Level: Extensive Experience
  • Directs the implementation of information processing and operation standards to protect and manage information assets.
  • Develops capabilities, features, and uses for major IM platforms.
  • Coaches others on issues and considerations for managing structured vs. unstructured data.
  • Advises on improvement plans for existing and forecasted information architecture.
  • Consults on all major information management platforms and practices.
  • Consults on information architecture concepts, disciplines, and practices.
• Modeling: Data, Process, Events, Objects - Level: Working Knowledge
  • Defines common types of data, processes, or event models.
  • Participates in model reviews and walkthroughs.
  • Explains existing models to clients and describes their associated business processes.
  • Focuses on modifying data structure, access, and entity relationship diagrams.
  • Prepares reports to recap current findings and issues in data, processes, or event models.