Real Estate

Airflow Data Engineer

Remote

The Airflow Data Engineer is responsible for designing, implementing, deploying, and supporting various data management technologies and architectures.  In partnership with business leaders, key stakeholders and cross-functional project teams, the Data Engineer will be an active contributor in a collaborative team structure and will have the opportunity to accelerate the delivery of and improve the quality of products providing increased operational excellence, a greater client experience and other strategic objectives.

•           Experience with data pipeline and workflow management tools: Airflow.

•           Experience with developing guidelines for Airflow clusters and DAG's/Task's etc.

•           Experience with Performance tuning of the DAG and task implementation

•           Develop DAG - data pipeline to on-board and change management of datasets

•           Experience installing Apache Airflow, configuring, and monitoring Airflow cluster

•           Understanding of airflow rest services and integration of airflow platform eco-system

•           Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with databases.

•           Knowledge about big data eco systems like Cloudera/Hortonworks components hadoop, spark, Hive etc..

•           Experience in building and optimizing - big data- data pipelines, architectures, and data sets.

•           Build processes supporting data transformation, data structures, metadata, dependency and workload management.

•           Orchestrating the Airflow/workflow in hybrid cloud platform AWS/Azure/GCP setup and administration is a plus.

•           Proficient in modern programming languages (Python, Java) with experience and open-source technologies.

•           Possess professional software engineering practices and best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations.

•           Knowledge in DEVOPS models with experience in containerization platform via building the docker, assembly, deployment, automation etc.. and best practices for managing the Applications.

•           Experience in managing a shared services is a plus and knowledge about Infrastructure management capacity assessment, forecast, planning etc..

•           Ability to maintain clean and secure data environments

•           Fast learner and a team player

•           Apache Airflow Fundamentals

•           Agile/Scrum principles