Technology challenges can come from literally any sector or vertical in place in the world today, even from out of this world. This was true with a client of ours that works in the shipping vessel telemetry space. Using their own satellites and ground stations to capture Automatic Identification System (AIS) data from all ships, this company uses the data to provide precise tracking information for sea-going shipping vessels around the world. Keeping track of thousands of vessels over millions of square miles of ocean down to the exact latitude and longitude on a second-by-second basis is no small feat, especially from a data perspective.
Spire came to us to see if we could help them build out a proof-of-concept for a high performing AIS Data Streaming system that could ingest high volumes of data and then transform it for delivery to client applications.
As a Premier Consulting Partner, ClearScale has developed a deep knowledge of AWS services that can meet the needs for any of our clients, regardless of the complexity of the issue or task. Having successfully proven in previous client engagements that the AWS Lambda and API Gateway services were robust, scalable, and able to handle numerous requests with minimal latency on response time, we began building our proof-of-concept around using these services as part of our cloud solution for Spire.
In this case, we loaded GIS data that was being sent from the satellite and ground station into the S3 bucket we set up for our proof-of-concept. We then leveraged AWS Lambda to first parse the data and then load it into tables we had created in DynamoDB. Once loaded there, we used DynamoDB-Geo to manage the geohash box indexes to streamline the query response time. Finally, we used yet another AWS Lambda function to essentially create a REST API that we would use to query and retrieve geographic data for use in the external client application and it was placed in the AWS API Gateway service.
Using this design, we were able to query the geographic data loaded in the S3 instance to return results based around specific requests, which in turn were filtered based on a given geographical area bounded by a series of latitudinal and longitudinal coordinates, also known as a polygon. We were able to conclusively demonstrate that our proof-of-concept could easily query the data and return the results within the given parameters submitted via the REST API we had set up in the AWS API Gateway, thus allowing the API to eventually be leveraged by the external client applications.
“Spire joined the AWS Startup Program and worked with ClearScale to implement our proof-of-concept. ClearScale team was professional, knowledgeable and responsive. They delivered to the requirements with quality and on budget.”
Russell Muzzolini | Spire CTO
Being able to query and get results was only the first step, however. The proof-of-concept showed the success of being able to do this with test data we had loaded into the S3 instance, but it did not account for being able to perform this function efficiently given the large volumes of data that would be sent in real-time from the satellite and ground tracking stations. We also needed to demonstrate that the DynamoDB we had set up would be up to the task of not only ingesting large volumes of data but concurrently demonstratinge its ability to respond quickly to client queries through the API with minimal latency.
For our proof-of-concept, we implemented instances of DynamoDB, PostgreSQL, and MongoDB and set them up for testing of high production data loads and near real-time queries to external clients. To accomplish this, our teams at ClearScale developed custom scripts for a JMeter performance testing tool to simulate the large volume of GIS data the systems would expect to receive, as well as a second set of custom scripts to emulate the client queries for given geographical polygons. The scripts were designed to mimic the design specifications for the application proof-of-concept which included a low number of threads ingesting a high volume of data using the Lambda parse and insert functions we had developed previously.
After initial testing of all three setups, ClearScale identified a number of configuration issues and worked to identify ways to improve overall performance through configuration parameter settings. ClearScale also recognized the need to invoke additional Lambda instances that would work in parallel for the GIS data ingestion process and a modification of the REST API to use a different call pattern than the one originally implemented to reduce round-trip request latency.
Using the newly tuned databases, along with the changes to AWS Lambda and the REST API, ClearScale once again performed the same testing scenario against all three databases’ instances. With a low price point, and high read and write capabilities and response time, DynamoDB showed that it was indeed the best solution for the given needs of the client.
ClearScale presented the proof-of-concept and findings to our client who ultimately implemented the solution. Faced with the prospect of large amounts of geographically relevant data and no easy way to parse and understand the results, Spire’s decision to engage with ClearScale proved to be strategically sound. Our ability to understand the unique nuances of the market the client operates in and, in turn, provide a robust and cost effective cloud solution that meets the client’s needs is what makes ClearScale the company to partner with.
Get in touch today to speak with a cloud expert and discuss how we can help: