Data Pipeline Solution Accelerates Pharmaceutical Research

June 17 2019

Pharmaceutical research is data-intensive. It takes tremendous computational power to parse and analyze the requisite giant chunks of data. The cloud delivers that power and helps accelerate research processes and, ultimately, research findings. That’s why one pharmaceutical company was using some “off the shelf” CloudFormation templates to create its own powerful informatics environment on the AWS cloud.

However, the company determined that it needed assistance in optimizing that environment. A key driver was its use of a set of analysis pipelines that process output to generate analysis.

The Challenge

The company’s IT team had experimented with deploying the specialized software for drug discovery and development process on Amazon EC2 instances. The problem was that some of processes could take many hours. In many cases, they had to be executed several times. To accelerate data analyses, the company needed to automate as many steps as possible, from initial data ingestion through storing the output in an Aurora database.

Because the data could include electronic protected health information (ePHI), whatever solution was implemented also had to meet specific privacy and security requirements of the Health Insurance Portability and Accountability Act (HIPAA).

ClearScale, an AWS Premier Consulting Partner with extensive experience in the medical industry, was tasked with the job. Specifically, ClearScale would work with the pharmaceutical research company to develop two new data processing pipelines and refine the previously developed one with the goal of speeding the data processing side of the company’s research.

The ClearScale Solution

ClearScale’s multi-tiered solution features a combination of AWS services and best practices to meet the pharmaceutical research company’s specific needs. While not a comprehensive list of all the solution components, key elements include Amazon Elastic Container Service (Amazon ECS) at the application tier. The highly scalable, high-performance container orchestration service supports Docker containers and allows for easily running and scaling containerized applications on a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances.

It’s used in conjunction with Amazon CloudWatch, a monitoring and management service. A CloudWatch EC2 agent is installed on each EC2 instance using HashiCorp’s Terraform, an open source tool for creating, changing and improving infrastructure. The CloudWatch agent provides an automated way to send log data to CloudWatch logs from the Amazon EC2 instances.

Also included is AWS Lambda, a compute service that allows for running code without provisioning or managing servers. It automatically scales applications by running code in response to each trigger.

At the database tier is Amazon Aurora (Aurora), a relational database engine. It’s fully managed by Amazon Relational Database Service (RDS), which automates time-intensive administration tasks such as hardware provisioning, database setup and backups. The storage tier employs Amazon Elastic File System (Amazon EFS), an elastic file system that scales on demand without disrupting applications.

At the integration tier is Amazon Simple Queue Service (SQS), a fully managed message queuing service that decouples and scales the fleet of worker nodes. It’s complemented by Amazon Simple Notification Service (SNS), a highly available, secure, fully managed publish/subscribe messaging service for decoupling microservices, distributed systems, and serverless applications.

Architecture Diagram alt

The HIPAA Component

The solution also includes a number of AWS services to help satisfy HIPAA requirements for security and privacy. Among them is AWS Identity and Access Management (IAM). The service allows for centrally managing users, security credentials and permissions that control which AWS resources users and applications can access.

AWS Key Management Service (KMS) is used to create and control the encryption keys used to encrypt data. Amazon EC2 Systems Manager Parameter Store provides secure, hierarchical storage for configuration data management and secrets management. AWS CloudTrail enables governance, compliance, operational auditing, and risk auditing of the neuroscience company’s AWS account.

The Solution in Action

In very simple terms, the ClearScale solution works like this: Amazon CloudWatch triggers an ingestion task to look for new data on their corporate network. If new data is found, it’s fetched and copied to Amazon EFS storage. SQS, used as a messaging queue, triggers a lambda function. The lambda function starts the processing task. Amazon ECS scales out the resources to meet the needs of the processing task. After the processing is complete, results are written to an Aurora database. An SNS notification is sent, and the ECS resources are scaled in.

The Results

With ClearScale’s solution deployed, tested and successfully operating, the pharmaceutical research company now can conduct the analysis portion of its research much faster. Its researchers, data analysts and IT team no longer have to devote time to tedious processes and can instead focus on more strategic endeavors. That all translates to quicker research findings — and faster time-to-market for potential treatments.

The secure, reliable infrastructure and the ability to scale compute resources as needed also provide greater flexibility and potential cost savings for future research projects.

What Can ClearScale Do for Your Company

If you’re interesting in learning more about ClearScale’s work with data pipelines, analytics and similar topics, you’ll find more case studies here.

Interested in ClearScale’s other areas of expertise or what we can do for your company? Let us know.

Get in touch today to speak with a Cloud expert and discuss how we can help:

Call us at 1-800-591-0442
Send us an email: sales@clearscale.net
Fill out a Contact Form
Read our Customer Case Studies

San Francisco

Headquarters

71 Stevenson St.

Suite 400

San Francisco, CA 94105

O: 1-800-591-0442

F: 1-415-655-6601

San Jose

5450 Thornwood Dr

Suite #L

San Jose, CA 95123

Denver

1400 16th Street,

Suite 400

Denver, CO 80202

O: 1-720-932-8028

Phoenix

2942 N 24th Street,

Suite 114

Phoenix, AZ 85016

O: 1-602-560-1198

New York

165 Broadway, 23rd Floor

New York City, NY 10006

O: 1-646-759-3656

Houston

11757 Katy Freeway

Suite 1300

Houston, Texas 77079

O: 1-281-854-2088

Toronto

100 King Street West

Suite 5600

Toronto, Ontario, M5X 1C9

O: 1-416-479-5447

About Us  |  Careers  |  Privacy Policy
@ Subscribe
Share