NOIDA · FULLTIME
Senior Data Engineer
FCI CCM
Noida · onsite · Posted 37d ago
Sourced from
Undisclosed5–9 yrsfulltimeNoida
About a minute. You'll make an account on the way through.
agent · working
- Read this posting0.3s
- Found the form on Naukri · 11 questions to answer1.4s
- Decided the answers for all of them2.9s
- Tailor your résumé to this role
- Waiting for you
Section · 01
About this role
Job Summary
Responsibilities
Requirements
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Broad function: The Senior Data Engineer - VARTA SENSE will be responsible for designing, developing, and managing scalable and reliable data pipelines that form the core data foundation of the VARTA SENSE platform. The role will work closely with the ML Lead, Solution Architect, Backend Engineering team and other technology stakeholders to transform complex banking and transaction data into standardized, reusable and production-ready datasets and features. The position requires strong hands-on expertise in Python, SQL, Spark/PySpark, data pipelines, orchestration, data quality and distributed data processing, with particular emphasis on reliability, performance, scalability and secure deployment within restricted banking environments.
Responsibilities
- Data Pipeline Development & Engineering: Design and develop scalable batch, incremental and production-grade data pipelines for large volumes of transaction and customer data. Build standardized and reusable data models and canonical schemas for transactions, customer attributes, offers, exposure and outcome events. Develop configurable ingestion and mapping frameworks to integrate data received from different banks and source systems. Manage pipeline orchestration including dependencies, checkpointing, retries, partial failures, backfills and safe replay mechanisms. Implement deduplication, late-arriving data handling, incremental loads, CDC, watermarking and merge/upsert processing.
- Data Quality, Reconciliation & Governance: Establish strong data-quality controls, automated testing, source-to-target reconciliation and schema validation. Ensure data lineage and schema evolution are properly managed across pipelines. Proactively identify and prevent issues such as missing data, duplicate records, incorrect aggregations, silent data loss and double counting. Maintain reliable and auditable data-processing standards across production environments.
- ML Feature Engineering Support: Work closely with the ML Lead to develop and maintain analytical and ML feature pipelines. Ensure consistency between training and production/inference datasets. Maintain point-in-time correctness and prevent data leakage within feature pipelines. Support versioned analytical features and reusable feature definitions.
- Performance & Scalability: Optimize large-scale data-processing workloads through effective partitioning, query optimization, join strategies, memory management and I/O optimization. Benchmark workloads against expected transaction volumes and available infrastructure. Identify bottlenecks and continuously improve processing speed, stability and infrastructure utilization. Ensure data pipelines remain scalable as transaction volumes and customer deployments increase.
- Deployment, Security & Production Reliability: Develop and support pipelines for deployment within bank-controlled on-premise and private-cloud environments. Ensure pipelines can operate in restricted or offline environments without public-internet dependency at runtime. Implement appropriate controls for PII handling, encryption, tokenization, masking, access management and secure connectivity. Establish monitoring and alerting for pipeline failures, data-quality issues, infrastructure bottlenecks and processing delays. Define and maintain backfill, replay, recovery and disaster-recovery procedures for critical pipelines.
- Cross-functional Collaboration: Work closely with ML, Architecture, Backend Engineering, DevOps and Infrastructure teams to design and deliver reliable data solutions. Collaborate with development partners to reproduce, transition and operationalize data pipelines within FCI. Support integration between data platforms and downstream application / ML services. Participate in technical discussions, architecture reviews and production troubleshooting.
- Reusability, Documentation & Knowledge Transfer: Build reusable adapters and configuration layers so that onboarding a new bank can be managed through configuration and mapping rather than changes to the core product code. Maintain clear technical documentation, runbooks and operational procedures. Document architecture decisions, troubleshooting steps and recovery procedures. Enable alternate engineering resources to independently operate and troubleshoot critical pipelines. Participate in code reviews, Git-based development, CI/CD practices and continuous improvement of engineering standards.
Requirements
- 5-9 years of relevant experience in Data Engineering, with hands-on ownership of production-grade data pipelines.
- Strong hands-on expertise in Python and Advanced SQL.
- Strong understanding of SQL concepts including window functions, query optimization and query-plan analysis.
- Practical experience with distributed data-processing platforms such as Apache Spark / PySpark or equivalent technologies.
- Experience building and managing production pipelines using Apache Airflow or an equivalent orchestration framework.
- Strong understanding of data modelling, ETL/ELT, batch processing and incremental data pipelines.
- Experience working with Parquet and other columnar data formats, partitioning, schema evolution and reconciliation.
- Practical experience with CDC / incremental ingestion technologies and concepts.
- Exposure to technologies such as Kafka, Debezium or equivalent event / CDC platforms would be preferred.
- Experience implementing automated Data Quality frameworks using tools such as Great Expectations, dbt tests, Soda or equivalent.
- Understanding of Data Lineage and Metadata Management using OpenLineage, Marquez or similar self-hosted solutions would be an advantage.
- Experience with lakehouse technologies such as Delta Lake, Apache Iceberg or Apache Hudi would be preferred.
- Exposure to Feature Store platforms such as Feast or equivalent would be beneficial.
- Working knowledge of Git, CI/CD, Docker and Linux environments.
- Experience with monitoring tools such as Prometheus, Grafana or equivalent platforms.
- Strong understanding of production support, troubleshooting, performance tuning and failure recovery.
- Ability to clearly explain technical design decisions, performance trade-offs and production engineering challenges.
- Strong collaboration skills with Data Science, ML, Backend Engineering, Architecture and Infrastructure teams.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Section · 02
Skills
data qualityAccess ManagementPerformance TuningDisaster recoverymetadataTechnical DocumentationPythonSQLProduction SupportLinux
Section · Company
About FCI CCM
FCI CCM
Software Product
550
employees
1959
67 years old
Noida, Uttar Pradesh
India
About
Today, we lie at the forefront of the Customer Communications Management industry that is helping organizations deliver exceptional customer experiences with our products and services. Backed by 6 dec
Industries
Software ProductIT Services & Consulting
Employee ratings
304 reviews
4.4/ 5
Culture
4.4
Career growth
4.1
Work-life
4.3
Employees rate it well for
Work Environment46Work Culture43Senior Management42Learning25Team Members19
Find them on