Spark Developer

Role & responsibilities

Job Title Spark Developer

Relevant Experience (in Yrs) 5+

Technical/Functional Skills • Strong Knowledge on Spark, Scala, Python, PySpark • Strong in SQL concepts • Good to have Knowledge on Bigdata Technologies like Hive ,Sqoop,Pig,Kafka,Flume. • Experience in Version Control tools like SVN, GITHUB as well as experiences in Production deployment process. • Basic knowledge in below skill sets: a. SQL Server b. Linux Shell Scripting c. Data Warehousing concepts

Experience Required 7+

Roles & Responsibilities • Interacting with IT Business Analysts and / or our business partners to acquire the information necessary for product design. • Following written requirements to help design databases and develop dashboards to support our customers. • Translating raw data from multiple sources (ETL) into a single dataset at aggregate levels necessary to meet project needs. • Suggesting alternative ways to group, filter and sort aggregated data for optimal presentation and performance possibilities. • Providing front-line production support by analyzing customer questions, evaluating data sets to explain end results or quickly adjust processes based on consumer demands. • Working independently and with groups while engaging in multiple initiatives simultaneously in a fast paced IT environment. • Resolving business process conflicts as they relate to project specifications. • Reporting project statuses to IT Management staff as required. • Review current environment that is used for product launch. Design and recommend the go forward architecture using Spark, Python and Teradata.

• Migrate to and implement the new rules and jobs to support Operational Intelligence • Set up and configure alerting and monitoring • Create STTMs, ERDs, data dictionaries and other knowledge artifacts for the NDW Base environment • The person should have the ability to leverage BTEQ scripts, familiar with Informatica, UC4/Automic and obviously strong SQL skills