PUNE · FULLTIME
Senior Databricks and AI Engineer
Visual Technologies
Pune · onsite · Posted 3d ago
Your match
Sign in to see your match score, skill gaps & tailored resume.
Section · 01
About this role
Senior Databricks and AI Engineer
Job Summary:
Key Responsibilities:
1. Databricks Platform Expertise
- Design, develop, and maintain robust, scalable ETL/ELT pipelines on Databricks using Spark, Python, Scala, and SQL.
- Optimize Databricks cluster performance, job execution, and cost efficiency.
- Implement data governance, security, and access control best practices on the Databricks Lakehouse.
- Work with Delta Lake, Delta Live Tables, and other Databricks data management features.
2. SQL Server to Databricks Migration
- Lead and execute the migration of data and workloads from on-premises SQL Server databases to the Databricks Lakehouse.
- Develop strategies for schema conversion, data ingestion, and transformation from SQL Server to Delta Lake format.
- Identify and implement performance tuning strategies for migrated SQL workloads on Databricks.
- Ensure data accuracy, consistency, and completeness throughout the migration process.
- Develop and execute comprehensive testing plans to validate migrated data and functionality.
3. AI/ML Development
- Collaborate with data scientists and stakeholders to operationalize AI/ML models on Databricks.
- Develop and deploy machine learning pipelines, including data preprocessing, feature engineering, model training, and inference.
- Utilize Databricks MLflow for experiment tracking, model management, and deployment.
- Explore and implement advanced AI/ML techniques relevant to business needs.
4. Collaboration and Leadership
- Mentor junior engineers and contribute to the teams technical growth.
- Work closely with data architects, data scientists, analysts, and business stakeholders to understand requirements and deliver solutions.
- Provide technical guidance and documentation for data engineering processes and solutions.
- Troubleshoot and resolve complex data-related issues.
5. Qualifications
- Bachelors or Masters degree in Computer Science, Engineering, Data Science, or a related quantitative field.
- 5+ years of experience in data engineering, with a strong focus on cloud-based data platforms.
- Mandatory: Proven, hands-on experience migrating complex SQL Server environments to cloud-based data warehouses or lake houses, specifically Databricks.
- Mandatory: Deep expertise in Databricks platform, including Spark, Delta Lake, SQL Analytics, and Databricks Runtime.
- Proficiency in programming languages such as Python and SQL. Experience with Scala is a plus.
- Strong understanding of data warehousing concepts, data modeling, ETL/ELT best practices, and database performance tuning.
- Experience with AI/ML workflows, MLOps, and tools like MLflow.
- Familiarity with cloud platforms (AWS, Azure, or GCP) and their data services.
- Excellent problem-solving, analytical, and communication skills.
- Ability to work independently and collaboratively in a fast-paced environment.
6. Preferred Skills
- Databricks certifications.
- Experience with containerization technologies (Docker, Kubernetes).
- Knowledge of other database technologies.
- Experience with CI/CD practices for data pipelines.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Let the agent run this one for you.
Tailored resume, auto-apply, and referral lookup — in under 2 minutes.
Section · 02
Skills
Section · Company
About Visual Technologies
Visual Technologies
175
employees
1996
30 years old
Noida, Uttar Pradesh
India
₹3.8L PA avg
Avg at Visual Technologies
About
Established in 1996, VTI made its foray into the broadcasting industry as an Exclusive Distributor for the global giant Panasonic to market and distribute their Professional Audio/Video Broadcast Equipment.
Over the years, VTI has diversified its activities into Design, System Integration & Configuration and commenced undertaking Turnkey Projects. Representing more than 40 International Broadcasting brands, VTI has become a one stop shop for all professional Audio / Video requirements. VTI has also system designed, assembled, commissioned and supplied DSNG and OB Vans to both private and public sectors. VTI is ISO 9001-2000 Certified company in the Broadcast Industry in India, dedicated to provide its customers with an uncompromising level of service and long lasting support.
Built around the core commitment to excel, VTI is driving marketing innovation in the Professional Audio-Video Broadcast Equipment, across India.
In everything we do, we weave in the triad of Precision, Perfection, and Passion.
Industries
Employee ratings
10 reviews
Culture
4.5
Career growth
3.9
Work-life
4.1
Employees rate it well for
Find them on