Use case
Optimizing Ticket Data Storage and Querying with AWS Kinesis and S3
- Bhautik Gajera
About the Customer
CasicoreX is a leading iGaming supplier known for its extensive portfolio of over 300 high-quality games. The company holds an MGA license and is certified in multiple jurisdictions, including but not limited to, Denmark, Sweden, Portugal, Spain, Italy, Latvia, Lithuania, and Switzerland. CasicoreX’s rapid growth and commitment to compliance make it a trusted name in the gaming industry.
Customer Challenge
CasicoreX needed a scalable, cost-effective solution for storing and managing ticket history data generated from player spins. On average, the platform generates around 600 tickets per second, with each ticket weighing approximately 10KB, amounting to about 21GB per hour of data. Initially, this data was stored in an MSSQL database hosted on AWS EC2, with the data storage on AWS EBS volumes.
This architecture resulted in high operational costs and scalability issues due to the massive amount of data being processed and stored. CasicoreX required a more efficient, cost-optimized solution without compromising data integrity or query performance.
Solution
IAMOPS proposed leveraging AWS Kinesis, AWS S3, and Amazon Athena to address CasicoreX’s data challenges. This solution focused on real-time data streaming, efficient storage, and cost-effective querying using cloud-native services.
1. Data Streaming with AWS Kinesis:
AWS Kinesis was used to stream ticket data in real-time, dynamically partitioning the data based on keys derived from the ticket’s JSON structure. The stream processes approximately 21GB of ticket data every hour.
2. Efficient Batching and Storage:
The data was batched into 30MB segments, which were then converted into Apache Parquet format. Parquet, being a highly efficient columnar storage format, reduced storage costs and optimized query performance.
The processed data was stored in AWS S3 for highly durable, scalable storage.
3. Querying and Analysis with Amazon Athena:
Amazon Athena was utilized to query the data stored in S3. This approach allowed for serverless querying, where CasicoreX could run SQL queries directly on the data in S3, significantly reducing costs and improving response times.
4. Automated Data Management:
AWS Lambda functions were used to automate the update of data partitions, ensuring that data was always organized and ready for fast querying.
Error handling mechanisms routed any issues in data ingestion to a dedicated S3 “/error” bucket, ensuring no data was lost and that issues could be traced easily.
Results & Benefits
The AWS-based solution provided CasicoreX with a highly efficient, scalable, and cost-effective infrastructure for managing its ticket history data:
- Cost Savings: By shifting from MSSQL on AWS EBS volumes to AWS S3 for data storage, the operational cost was reduced by 25%, bringing the monthly storage cost to approximately $815.
- Improved Query Performance: The use of the Parquet format in conjunction with Athena reduced query time, leading to faster data retrieval and lower query costs.
- Scalability: The use of AWS Kinesis for dynamic partitioning and AWS S3 for storage ensured that the infrastructure could seamlessly scale to accommodate the high data volume.
About IAMOPS
IAMOPS is a full DevOps suite company that supports technology companies to achieve intense production readiness.
Our mission is to ensure that our clients’ infrastructure and CI/CD pipelines are scalable, mitigate failure points, optimize performance, ensure uptime, and minimize costs.
Our DevOps suite includes DevOps Core, NOC 24/7, FinOps, QA Automation, and DevSecOps to accelerate overall exponential growth.
As an AWS Advanced Tier Partner and Reseller, we focus on two key pillars: Professionalism by adhering to best practices and utilizing advanced technologies, and Customer Experience with responsiveness, availability, clear project management, and transparency to provide an exceptional experience for our clients.