The effective and efficient trading and transporting of energy at a global scale is a critical business activity for our client. This Fortune 100 organization provides trusted energy solutions to its customers world wide, 365 days a year.
High value data transactions from a business partner system must be synchronized in an automated fashion with an internal system of our client. This process has been performed manually by highly skilled and experienced employees up until now. The work must be performed by this level of employee due to the nature of the data and applications being used.
The external business partner owns and operates a fuel logistics portal which is used to create and update transactions representing the purchase, sale and movement of fuel. This system is operated by the business partner at a site which is external to our client. The transactions entered into this system by employees must be reflected in a commodities management application and database which is internal to our client and operates in a company data center.
The decision was made early on to develop this system as a cloud native solution to be deployed on AWS. Our client is in the process of migrating many of its systems from their company data centers to the cloud and AWS is the company’s cloud solution provider of choice. Scalable Tech, a cloud technology services provider, was selected to architect and develop the AWS based solution for our client.
Decoupling Systems with Microservices, AWS ECS and AWS SQS
One of the primary upfront directives from leadership was to develop a system with a decoupled architecture that would allow it to be extended for use with other partner companies. To achieve this, Scalable Tech decided to employ a microservices style architecture with loosely coupled services that communicate through messaging systems.
Our client did not have full control over how the data would flow from the external partner to them. In order to maintain the exact data processing sequence, Scalable Tech opted to operate the system as a set of container services in AWS ECS rather than as serverless functions running in AWS Lambda.
Key microservices were developed for two main areas: partner data processing and commodities data management. Java and Spring Boot were employed for developing the microservices for the following reasons:
- The company has a significant existing Java code base and skilled Java developers. Scalable Tech will develop the system along with the Java developer employees of our client. Eventually the developer employees will take over and maintain the system.
- The service logic is very complex, requiring many classes and many thousands of lines of code, making strongly typed Java and its mature ecosystem a good fit.
- Spring Boot allows for the quick development of production ready services.
The partner data processing services were designed such that the overall architecture and key services could be reused for processing data from additional partners. The commodities data management services were designed as a domain API allowing for its use throughout the enterprise.
The transactions to be processed by the system are of high dollar value but transaction volume is not at Big Data scale, so Scalable Tech decided it was most appropriate and cost effective to use AWS SQS as a messaging system to decouple the services instead of AWS Kinesis or Apache Kafka. SQS has proven to be very effective and the AWS SDKs make it quite simple to use from within the Spring Boot, Java services.
Our client has reduced costs significantly by eliminating repetitive work through an automated process. The highly skilled and experienced employees that performed this tedious, error prone data entry can now spend their time on work that provides much greater value to the company.
The data which previously was manually updated with lag times of days or weeks is now synchronized in near real time. This provides the users of this data at the company which much more timely information which allows them to make better decisions in setting up their commodities trading and transport deals.
No additional footprint was added to the on prem data centers. Furthermore, the decoupled architecture of the Partner Integration service put in place by Scalable Tech fully supports the company’s future plans to migrate their commodities management application and database to the AWS cloud.
About Scalable Tech
Scalable Tech is a cloud technology service provider that specializes in delivering custom cloud based solutions that are secure, reliable, maintainable, scalable and cost optimized. We can help your business maximize its investment in cloud technology.