Use case
Dynamic Workflow Scaling with KEDA and Apache Airflow
- Lavish Gupta
About the Customer
CyvexTech is a platform dedicated to securing the software supply chain. It provides features such as code assurance, tamper-proof evidence documentation, visibility into anomalies, and seamless operation. Leveraging advanced concepts and frameworks, the platform ensures security of code artifacts throughout the software lifecycle, from production to delivery.
Customer Challenge
The existing workflow management system faced challenges in efficiently handling variable traffic loads. During peak periods, task queues grew, leading to delays, while resources were often underutilized during idle times. The primary objective was to enhance the scalability and stability of this system, ensuring it could dynamically adjust to changing task loads without over-provisioning.
Solution
IAMOPS implemented a scalable solution using Apache Airflow integrated with KEDA (Kubernetes Event-Driven Autoscaler) and the Celery Executor. This architecture introduced an event-driven scaling mechanism that dynamically adjusted the number of Celery workers based on real-time workload conditions.
Results & Benefits
The integration of Apache Airflow with KEDA provided notable performance improvements and resource efficiency:
- Scalability: Celery workers scaled automatically according to real-time task load, allowing efficient handling of high traffic periods.
- Resource Optimization: Worker scaling down to zero during idle times led to significant cost savings.
- Performance Improvement: Task execution times decreased, accelerating processing of queued tasks.
- System Stability: Dynamic scaling ensured stable performance, reducing risk of system overload.
About IAMOPS
IAMOPS is a full suite DevOps company that supports technology companies to achieve intense production readiness.
Our mission is to ensure that our clients’ infrastructure and CI/CD pipelines are scalable, mitigate failure points, optimize performance, ensure uptime, and minimize costs.
Our DevOps suite includes DevOps Core, NOC 24/7, FinOps, QA Automation, and DevSecOps to accelerate overall exponential growth.
As an AWS Advanced Tier Partner and Reseller, we focus on two key pillars: Professionalism by adhering to best practices and utilizing advanced technologies, Customer Experience with responsiveness, availability, clear project management, and transparency to provide an exceptional experience for our clients.
