- $83,690.00 -125,470.00/year*
2 E 2nd St
The Data Engineer role provides the technical expertise for the design, modeling and implementation of organization's data. The individual will design the data model to store in Enterprise Data Warehouse and Hadoop systems. Implements logical design into physical database design and possesses end-to-end vision on the enterprise data. Manage data across Enterprise data warehouse, Hadoop and other analytical databases.
- Responsible for the architecture and design of the data for Hadoop ecosystem and SAP BW on HANA.
- Responsible for both structured and unstructured data design into SAP BW on HANA, Hadoop and Vertica (any relational DB).
- Perform data analysis, data modeling and data design tasks on complicated datasets with potentially complex data integration scenario for Hadoop and SAP BW on HANA.
- Design Data Lake in Hadoop ecosystem.
- Ensure the data architecture can be extendable for big data solutions. Determine overall data modeling standards, guidelines, best practices and approved modeling techniques and approaches.
- Manage data movements across these platforms to minimize cost.
- Design and implement data ingestion techniques for real time and batch processes for structured and unstructured data sources into Hadoop ecosystems and HDFS clusters.
- Understand complex business requirements and propose end to end and simplified enterprise information architecture solutions.
- Develop and implement data design methods, data structures, and modeling standards which work with multiple business intelligence tools.
- Work closely with Analytics team and implement their self-service and analytics requirements.
- Suggest, design and plan improvements, maintenance, and upgrades for Wawas Enterprise Data Warehouse (EDW) and big data platforms
- Architect sensitive data storage and participate in annual Security audit of the EDW, big data lake and data mart environments, with recommendations and plans to fix issues and publication of Security standards.
- Assist the Manager of Big Data Analytics in creating and implementing data governance and standards. Drive Enterprise Information Management disciplines into both structured and unstructured data.
- Design and engineer ETL solutions from various internal and social media data sources to load EDW and Hadoop data lake using tools like informatica
- Design big data analytics solutions for Wawas current toolsets including SAP, MicroStrategy, SAS and Tableau.
- Collaborate with Analytics team to build solutions that enable business analytics. Develop quality scalable, tested, and reliable data services using industry best practices. Develop and configure Hadoop / Big Data components.
- Manage all activities centered on obtaining data and loading into a data lake environment in Hadoop. Assess the suitability and quality of candidate data sets for the Data Lake.
- Balance business requirements with technical feasibility and set expectations on new projects. Recommend changes in development, maintenance and system standards.
- Design and build integration components and interfaces in collaboration with Architects and Infrastructure Engineers as necessary. Perform unit, component, integration testing of software components including the design, implementation, evaluation, and execution of unit and assembly test scripts.
- Determine if the data received from the upstream systems are of good quality based on the rules and data quality validations defined and in case of any issues with the data quality analyze and come up with a preliminary summary of the root cause/issue.
- Assist the Analytics team by leveraging the Hadoop ecosystem knowledge to design, and develop capabilities to deliver our solutions using Spark, Scala, Python, Hive, Oozie, Kafka, and other things in the Hadoop ecosystem.
- Follow security standards for all data and tools that are being introduced in the team.
- Handle multiple priorities simultaneously
- Work collaboratively with cross-functional teams
- Establish and maintain a working environment conducive to positive morale, individual style, quality, creativity, and teamwork
- Ability to build strong trusting relationships with business partners
- Ability to work in a fast-paced, team environment
- Excellent communication skills
- Basic project management skills required
- Work with the team to lead and maintain data strategy standards in all the data team is responsible for
- Bachelors degree in Computer Science/Engineering preferred
- 8+ years database, data integration experience
- 3+ years experience with Hadoop, SQL and Big Data solutions
- Preferred experience with SAP HANA
- 5+ years experience in designing and implementing the data architecture (conceptual, logical, physical & dimensional models).
- Developing Enterprise Business Intelligence solutions on one or more of the following EDW platforms: Teradata, Oracle, HANA, and BW on HANA
- Experience implementing Big Data solutions using open source technologies within the Hadoop ecosystem such as: Impala, Hive, Spark, Pig, etc.
- Strong knowledge of key scripting and programming languages such as Python, Java. Experience with data integration tools such as Informatica
- Experience designing and implementing ETL strategies
- Hands-on experience with dimensional modeling techniques and creation of logical and physical data models (entity relationship modeling, Erwin diagrams, etc.)
- Strong knowledge of data security principles
- Proven track record working with complex, interrelated systems and bringing that data together on Big Data platforms.
Associated topics: data administrator, data center, data integration, data integrity, data manager, data management, database administrator, etl, mongo database, mongo database administrator
* The salary listed in the header is an estimate based on salary data for similar jobs in the same area. Salary or compensation data found in the job description is accurate.
Loading some great jobs for you...