West London, London
Up to £600 per day
8 months ago
Sr Big Data Engineer AWS, Spark (Scala Stack) will Lead the design, implementation, and continuous delivery of pipelines using AWS services and Spark (with Scala Stack) big data technologies supporting data processing initiatives across batch and streaming datasets. Providing Global Data Analytics for this exciting Media company based in West London.
Responsible for development using Scala, Python languages and Big Data Frameworks such as Spark, EMR, Presto, AWS Athena, Kafka, Zepplin , and Kinesis . The data engineer would design, develop, automate, maintain and test big data solutions within this well-known Global Media organisation based in West London.
· 4+ years of development experience with AWS services Must have EC2, EMR, RedShift, Airflow (Must), S3, and CLI (must to have ) and Jenkins
· 3+ years of development experience with Apache Spark (with Scala Stack), Presto, SQL, notebook and NoSQL Implementation
· 4+ years of extensive working knowledge in different programming Scala (Must), Shell and Python .
· Proficiency working with structured, semi-structured and unstructured data sets including social, web logs and real time streaming data feeds
· Able to tune Big Data solutions to improve performance and end-user experience
· Expert level usage with Jenkins, GitHub is preferred for Spark built
· Comfortable to work in Agile environment
· 5-6 years Software Industry experience and a Bachelor's or Master's degree in computer science or software engineering
· Big Data Developer Certification is a plus such as Spark Developer / AWS Developer
· Experience in the media industry is a plus
· Must have the legal right to work in the United Kingdom and work on site in West London