Data Engineer

ClearScale is a leading cloud system integration company and AWS Premier Consulting Partner providing a wide range of cloud services including cloud consulting, architecture design, migration, automation, application development, and managed services.

We help Fortune 500 enterprises, mid-sized businesses, and startups in verticals like Healthcare, Education, Financial Services, Security, Media, and Technology succeed with ambitious, challenging, and unique cloud projects. We architect, develop, and launch innovative and sophisticated solutions using the best cutting-edge cloud technologies.

ClearScale is growing quickly and there is high demand for the services we provide. Clients come to us for our deep experience with Big Data, Containerization, Serverless Infrastructure, Microservices, IoT, Machine Learning, DevOps, and more.

ClearScale is looking for an experienced Data Engineer to participate in a custom data pipeline development project.

Responsibilities:

  • Migrate data located in a multitude of data stores, into the Data Lake
  • Orchestrate processes to ETL that data, slice it into the various data marts
  • Manage access to the data through Lake Formation
  • Build a data delivery pipeline to ingest a high volume of the real-time streams, detect anomalies, slice into the window analytics, put those results in the Elastic search system for the further dashboard consumption
  • Analyze, scope, and estimate tasks, identify technology stack and tools
  • Design and implement optimal architecture and migration plan
  • Develop new and re-architecture solution modules, re-design and re-factor program code
  • Specify the infrastructure and assist DevOps engineers with provisioning
  • Examine performance and advise necessary infrastructure changes
  • Communicate with the client on project-related issues
  • Collaborate with in-house and external development and analytical team

Basic Qualifications:

  • Hands-on experience designing efficient architectures for high-load enterprise-scale applications or ‘big data’ pipelines
  • 5+ years of AWS experience
  • 5+ years NoSQL DB experience
  • 5+ years experience working with time series data and windowing functions
  • Development experience building large scale data solutions
  • Experience with AWS Timestream
  • Proven ability to work closely with business and product teams to ensure data solutions are aligned with business initiatives and are of high quality
  • Ability to communicate technical hurdles and challenges clearly and succinctly
  • Take problems from inception to completion - own the building, testing, deployment, and maintenance of the code that you work on
  • Experience with reporting, ETL and data modeling
  • Excellent skills in Java
  • Production experience building data pipelines
  • AWS Certifications

Preferred Qualifications:

  • We are looking for a candidate with 5+ years of experience in Data, Cloud, or Software Engineer role, who has attained a degree in Computer Science, Statistics, Informatics, Information Systems, or another quantitative field
  • Experience with AWS Kinesis, AWS Kinesis Data Analytics for Apache Flink, AWS Grafana