Job Title:

Data Engineer

Function:

Technical

Name of Job Holder:

Location:

BIPP, ISB, Mohali

Reports to position:

Project Manager

Band:

Reportees to Position:

None

Job Purpose

Design, build and manage the data pipelines and databases (both Relational as well as NoSQL) for a large national-scale spatio-temporal data warehousing, visualization, and analytics project.

Job  Outline

·         Develop the database architecture keeping in view the broader project requirements.

·         Design the database schemas and tables based on system requirements, keeping in view data storage, access, and management needs.

·         Use programming skills in Python or other open-source tools to build robust data pipelines.

·         Integrate data from a variety of sources, ensuring adherence to quality and accessibility standards.

·         Design and deploy database monitoring systems to ensure high availability.

·         Write and maintain documentation that includes database structures, data standards, procedures and definitions for the data dictionary (metadata).

·         Manage the security and disaster recovery aspects of the databases.

Job Specification

Knowledge / Education

Specific Skills

Desirable Experience/Skills

BE/BTech in
Computer Science/Computer Engineering/Information technology/Electronics
OR
Master of Computer Applications (MCA)

·         2-5 years of progressively complex industry experience in the design and management of opensource database systems, namely Postgre SQL (preferred) or My SQL

·         Hands-on programming experience in Python, R and/or Java along with BASH shell scripts.

·         Comfortable working with Linux systems and experience with Linux utilities and commands.

·         Demonstrated experience with sound software engineering practice, in particular software asset management – preferably using a Distributed Version Control System like Git or Mercurial.

·         Fresh graduates with demonstrable database skills may also be considered at the intern level.

·         Ability to leverage multiple tools and programming languages to analyze and manipulate data sets from disparate data sources.

·         Basic Linux administration knowhow

·         Knowledge of Big Data platforms like Hadoop and Spark.

·         Knowledge of No SQL databases like MongoDB.

·         Knowledge of Amazon Elastic MapReduce (EMR) and Amazon Redshift

Job Interface/Relationships:

Internal

External

• Dedicated project personnel
• Other BIPP personnel
• Other ISB teams and projects

§  External partners

§  Consultants

S.N

Key Responsibilities

% Time Spent

1

Design and implement a dynamic database architecture in consultation with broader project team and users. This will include both the hardware as well as the software aspects.

30

2

Designing the data provenance and metadata framework

15

3

Design and management of the database schemas and tables

25

4

Developing and maintaining scripts and tools for database monitoring and ensuring database uptime

25

5

Keeping abreast of latest technology and process developments

5

Total Time Spent on All Responsibilities

100%

SN

Key Result Area

Key Performance Indicator

Measure

Weightage

1

Establish Database Architecture

§ Development of database architecture as per the project requirements

§ Appropriate usage of latest hardware and software technologies

25%

2

Design Database Systems

§ Design of data provenance system and metadata framework

§ Design and management of database schemas and tables

§ Design, development and management of database systems

50%

3

Maintain Database Systems

§ Develop and maintain scripts and tools for database monitoring, and ensure database uptime

§ Keep abreast of latest technology and process developments

§ Maintenance, and smooth functioning of database systems with latest technologies incorporated

25%