First Street Foundation Optimizes Application Performance with AWS Containerization Services

Executive Summary

First Street Foundation is a nonprofit research and technology group whose mission is to quantify flood risk facing private and commercial properties across the contiguous U.S. The organization uses cutting-edge modeling techniques to calculate a Flood Factor for individual properties that represents future flood risk based on ever-changing environmental factors.

First Street Foundation distributes its data and findings through an API to businesses, government entities, and researchers who want to pursue further analyses. The Foundation also formed Flood Lab, a cohort of experts from leading academic institutions, which aims to determine the effects of flooding on the national economy.

Recently, First Street Foundation sought Amazon Web Services (AWS) expertise to enhance its computing and data management infrastructure. As an Premier Consulting Partner with extensive cloud experience, ClearScale had the technical knowledge First Street Foundation needed to accomplish its goals.

The Challenge

First Street Foundation’s web application relied on Kubernetes, an open-source tool that helps users manage containerized applications. The development team needed assistance in updating their legacy Amazon Elastic Kubernetes Service (EKS) setup into an automated and optimized workflow with modern tooling.

First Street Foundation wanted to incorporate monitoring, logs, metrics, and incident reporting into its IT infrastructure using cloud-native technologies. They also wanted advice on whether it should stick with its original EKS setup that was managed with Amazon EC2 instances or switch over to AWS Fargate, a serverless computing launch type designed for containers. To ensure it was making the right decision, First Street Foundation decided to bring in an AWS cloud expert to illuminate the path forward.

First Street Foundation found ClearScale through the AWS Partner Network (APN) portal. Once the two organizations were connected, First Street Foundation shared that it wanted assistance with the following tasks:

• Updating EKS clusters by either upgrading the current setup or integrating AWS Fargate
• Developing rollback procedures for the cluster upgrade process
• Implementing environment (infrastructure and applications) monitoring and metrics reporting with New Relic
• Updating continuous integration / continuous delivery (CI/CD) for EKS deployments
• Configuring PagerDuty to send relevant alerts and create incident responses
• Confirming auto-scaling rules for the Kubernetes cluster
• Adding log management functionality at the cluster level, using Fluent Bit to pipe logs to Elasticsearch
• Creating documentation for the IT infrastructure update procedures, autoscaling and logging

With its team of AWS specialists, ClearScale developed a proposal to solve these technical requirements.

The ClearScale Solution

ClearScale began by gathering First Street Foundation’s success metrics and evaluating the organization’s existing IT infrastructure. Rather than update the nonprofit’s existing clusters, ClearScale decided it was best to create new EKS clusters on the latest version of the Kubernetes engine.

ClearScale designed the clusters and developed a roadmap for migrating all of First Street Foundation’s applications to the new clusters. As part of the process, ClearScale’s experts also set up a CI/CD pipeline to streamline the application development and deployment processes going forward.

To add the monitoring and alerting capabilities that First Street Foundation wanted, ClearScale implemented Amazon CloudWatch, a service built for DevOps teams and engineers that provides data related around application performance, resource utilization, and overall operational health. ClearScale used the Container Insights feature within CloudWatch, which gathers and aggregates metrics for containerized applications and microservices.

ClearScale also installed New Relic, a product that enables users to track AWS spend and performance across its existing AWS environments. Finally, the ClearScale team implemented AWS Systems Manager Parameter Store so that First Street Foundation could store data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values. It allows users to store values as plain text or encrypted data. It can also reference Systems Manager parameters in scripts, commands, SSM documents, and configuration and automation workflows by using the unique name that was specified when the parameter was created.

Architecture Diagram

The Benefits

Thanks to ClearScale’s help, First Street Foundation gained the clarity it needed around its Kubernetes utilization in EKS and overall cloud environment design. ClearScale helped the nonprofit refresh its infrastructure and optimize workflow around unique business needs.

Now, First Street Foundation can easily track key metrics for its applications and trust that all of its data is stored properly in the cloud. On top of that, the organization has full confidence that its services will auto-scale with demand. First Street Foundation users can look up unique Flood Factors or access the nonprofit’s comprehensive database without experiencing any performance issues.

Going forward, First Street Foundation has a highly available and reliable platform to continue building its products. The organization can closely monitor its infrastructure to identify wasteful resource utilization or spending, as well as maintain the integrity of its containerized applications.