We are looking for a fast, enthusiastic, and accountable Data Engineer. As part of our in-house Data Technology team, you’ll be handling the design and construction of scalable management systems, ensuring that all data systems meet company requirements. You should also know the ins and outs of the industry such as data mining practices, algorithms, and how data can be used.
Your responsibilities:
- Develop, maintain, and monitor data pipelines.
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Apache, SQL, and AWS ‘big data’ technologies.
- Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs.
- Work with data and analytics experts to strive for greater functionality in our data systems.
- Responsible for POCs on any tech stacks introduced, or possible version upgrades.
- Perform root cause analysis on internal and external data to answer specific business questions and identify opportunities for improvement.
- Performing tasks such as web scraping, writing Unix/Shell scripts, invoking APIs, write amp; analyze SQL queries.
Requirements
About you:
- At least 5 to 6 years of experience as a data engineer.
- Big data technology knowledge in Apache Hadoop and its eco system, Batch and Real-time processing. Hive, Spark and Spark streaming.
- Must have experience in object-oriented languages like Spark (Scala/Python), Kafka, or Java.
- Experience building on AWS using S3, EC2, EMR, and Lambda.
- Experience with NoSQL databases, such as HBase, Cassandra, MongoDB.
- Good analytical skills with excellent knowledge of SQL.
- Experience in agile development methodologies is a must.
Good to Have: - Working experience on Apache Nifi/Airflow/informatica/SAP BODS.
- Experience in big data Cloud offerings like Redshift, Snowflake.
- AWS certifications or other related professional technical certifications.
- Experience with any other Cloud Platform.