Use case
Cost-Optimized and Scalable Deployment of SQL Flow Application
- Dhruv Bardolia
About the Customer
Illuminate Tech is a leading provider of advanced data analytics solutions. They help businesses process and analyze large volumes of data to derive actionable insights. Their robust platform supports high availability, scalability, and performance to meet the needs of their diverse clientele.
Customer Challenge
Illuminate Tech needed a cost-optimized, efficient, and scalable solution to manage the deployment of Gudu SQLFlow, an automated data lineage tool, across their infrastructure. The previous setup was manually configured and deployed on an On-Demand EC2 instance, lacking autoscaling capabilities and cost efficiency.
Solution
To address these challenges, IAMOPS deployed SQLFlow using Application Load Balancers (ALB) and Auto Scaling Groups (ASG) across multiple private subnets to ensure high availability. The deployment leveraged EC2 Spot instances to reduce costs, with instances managed by ASG using a custom AMI through a launch template, enabling the system to scale as needed. This infrastructure also included a canary release strategy to ensure smooth updates with minimal disruption.
The following diagram illustrates the solution architecture implemented:
Semantic Release Workflow
Results & Benefits
The deployment of SQLFlow resulted in several key benefits:
- Cost Optimization: The use of EC2 Spot instances led to approximately 60% cost savings compared to On-Demand instances.
- Scalability: The Auto Scaling Group (ASG) allowed SQLFlow to automatically scale up or down based on real-time demand.
- High Availability: By distributing SQLFlow instances across multiple availability zones, the system achieved high availability and resilience.
- Improved Load Management: The ALB efficiently balanced traffic between instances, ensuring consistent performance and minimizing downtime.
About IAMOPS
IAMOPS is a full DevOps suite company that supports technology companies to achieve intense production readiness.
Our mission is to ensure that our clients’ infrastructure and CI/CD pipelines are scalable, mitigate failure points, optimize performance, ensure uptime, and minimize costs.
Our DevOps suite includes DevOps Core, NOC 24/7, FinOps, QA Automation, and DevSecOps to accelerate overall exponential growth.
As an AWS Advanced Tier Partner and Reseller, we focus on two key pillars: Professionalism by adhering to best practices and utilizing advanced technologies, and Customer Experience with responsiveness, availability, clear project management, and transparency to provide an exceptional experience for our clients.