Senior Data Engineer

Purpose of the role:

The Senior Data Developer develops software, helps build technical solutions and effectively and efficiently manages the performance, integrity, and security of data. This role includes planning, developing, and troubleshooting of databases. The DD is responsible for developing, testing, improving, and maintaining new and existing databases to help users retrieve data effectively.

The primary functions are

• Putting the Data Scientist's models into production and building the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
• Choosing optimal solutions to use for collection, storing, processing, and analysing of huge sets of data, then maintaining, implementing, and monitoring them. The DD is also responsible for integrating them with the architecture used across the company

Reports to:

• Line Manager BI & Data Management

Responsibilities and duties:

• Works with stakeholder teams to assist with data-related technical issues and support their data infrastructure needs.
• Evaluates business needs and objectives.
• Builds the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
• Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Works on legacy data migration.
• Selects and integrates Data tools and frameworks required to provide requested capabilities.
• Monitors performance and advises any necessary infrastructure changes.
• Ensures database optimization, integrity, consistency, security, and privacy.
• Participates in SQL code reviews, writes, and deploys SQL patches.
• Continually evaluates software components and tools for software applications.
• Recommends ways to improve data reliability, efficiency, and quality.
• Remains up to date with industry standards and technological advancements that will improve the quality of outputs.
• Works with the engineering teams to resolve issues, evaluate software modules and develops specifications

Knowledge and skills required:

• Bachelor’s Degree in Computer Science, Informatics and Computer Engineering or related field

Experience with:

• AWS solutions development and implement.
• Hands-on in software
• Big data tools: Hadoop, Spark, Kafka, etc. is a
• Working with relational databases, query authoring (SQL) as well as working familiarity with a variety of
• Investigating data sources, understanding, and documenting data

Knowledge of:

• Advance working
• Stored procedure writing, query optimization, and
• Query troubleshooting and query execution
• One ETL tool, at
• Modelling and designing
• Redshift based DWH using SQL to develop complex and stored procedures is a
• Designing and developing different kind of reports such as analysis reports, ad hoc and standard reports.