Data Engineer

  • Omise
  • Bangkok, Thailand
  • May 11, 2018
Full time Data Developer

Job Description

We are looking for a savvy Data Engineer to join Machine Learning team in our Engineering department. This position will be responsible for building a scalable data architecture to support real time stream processing.  The ideal candidate is a coder with a good experienced in using a real time aggregation framework such as Flink,  Spark or Storm). The Data Engineer will support our software developers, database architects, data analysts and data scientists by ensuring that optimal data delivery architecture is consistent throughout ongoing projects. The right candidate will be excited by the prospect of optimizing or even re-designing our company’s data architecture to support our next generation of products and data initiatives.

Responsibilities

  • Create and maintain stream and batch processing data pipeline

  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources into SQL based database on a cloud service (AWS or Google Cloud).

  • Keep our data separated and secure across national boundaries through multiple data centers and AWS regions.

  • Work with machine learning engineer / data scientist to create machine learning service infrastructure.

Required Skills

  • Knowledge of stream processing framework (Flink /  Spark / Storm)

  • Experience in developing batch processing pipeline (MapReduce / Spark)

  • Competence in Scala and Python.

  • A successful history of manipulating, processing and extracting value from large disconnected datasets.

  • Strong project management and organizational skills.

  • Experience supporting and working with cross-functional teams in a dynamic environment.

  • We are looking for a candidate with 3+ years of experience in a Data Engineer role, with bachelor’s degree or higher in a relevant technical field.

  • Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.

  • Experience with cloud services: AWS or Google Cloud