Job Summary:
Cloud Data Engineer will be responsible for developing on the data lake platform and all applications on Azure cloud. Good data Engineering, data modeling, SQL knowledge is a must with Python programming background. The Data Engineer will be responsible for providing design and development solutions for applications in the cloud.
Minimum Qualifications and Job Requirements:
* Bachelor's degree in CS.
· 7 years of hands-on experience in designing and developing distributed data pipelines.
· 5 years of hands-on experience in Azure data service technologies.
· 5 years of hands-on experience in Python, SQL, Object oriented programming, ETL and unit testing
· Experience with data integration with APIs, Web services, Queues
· Experience with Azure DevOps and CI/CD as well as agile tools and processes including JIRA, confluence.
Essential Job Functions:
- Understand requirements and engage with team to design and deliver projects.
- Design and implement data lake house projects within azure.
- Design and develop application lifecycle utilizing Microsoft Azure technologies
- Participate in design and planning and necessary documentation
- Participate in Agile ceremonies including daily standups, scrum, retrospectives, demos, code reviews.
- Hands on with Python/SQL development and Azure data pipelines
- Engage with team to develop and deliver cross functional products
- · Key Skills
- a. Data Engineering and SQL
- b. Python
- c. PySpark
- d. Azure Data lake and ADF
- e. Databricks
- f. CICD
- g. Strong communication
* Document and maintain project artifacts.
* Maintain comprehensive knowledge of industry standards, methodologies, processes, and best practices.
* Complete training as required for Privacy, Code of Conduct etc.
* Promptly report any known or suspected loss, theft or unauthorized disclosure or use of PI to the General Counsel/Chief Compliance Officer or Chief Information Officer.
* Adhere to the company's compliance program.
* Safeguard the company's intellectual property, information, and assets.
* Other duties as assigned.