In today's financial ecosystem, where every second of inactivity or error can translate into millions of dollars in losses and customer distrust, real-time anomaly detection has become a strategic pillar. Companies that process hundreds of millions of transactions a month, such as Razorpay in India, have had to evolve from monitoring systems based on static thresholds to adaptive platforms, capable of identifying anomalous patterns in less than 30 seconds. This article discusses how the combination of Amazon MSK, Apache Flink, and a declarative architecture can achieve this goal, and how enterprises can benefit from similar approaches with the support of specialized technology partners.
The main challenge faced by large-scale payment platforms is the high cardinality of metrics. Individually monitoring thousands of traders with fixed rules generates an unsustainable number of false positives or, worse, silences real incidents. When thresholds don't adapt to seasonal behavior—such as sales spikes on weekends or during promotional campaigns—the system ends up being useless. The solution is to abandon static thresholds and adopt dynamic baselines that consider the day of the week, the time of day and adjustments for holidays. Not only does this reduce false alarms by more than 90%, but it allows operations teams to focus on what really matters.
The architecture behind this detection capability is supported by a reliable streaming backbone. Amazon MSK (Managed Streaming for Apache Kafka) acts as the core of integration, absorbing millions of events per second from transactions, settlements, and payments. By partitioning topics by business unit—payments, payroll, banking—you achieve logical isolation over a shared infrastructure, significantly reducing operational costs. This transport layer not only ensures durability with replication across Availability Zones, but also enables hot-distribution of detection rules. When an analyst updates a definition in the declarative language AdaDSL, that new rule is published in a Kafka topic and is consumed by Flink pipelines as a broadcast state, without the need to restart the process.
Apache Flink handles stateful processing, applying time windows and asynchronous joins to reference databases such as ClickHouse. This hybrid combination exploits the strengths of each engine: Flink for subminute detection with exactly-once semantics, and ClickHouse for deterministic calculation of historical baselines and analytical queries. The result is a system capable of detecting everything from systemic degradations in the success rate to fraud patterns such as card-testing (several rejections followed by a success) in less than 30 seconds. In addition, the separation between rule definition and execution allows domain experts—without being platform engineers—to create their own alerts using a simple DSL.
From a business point of view, the benefits go beyond technical efficiency. The cost reduction compared to previous architectures (such as Pinot + ThirdEye) reaches approximately 80%, while the detection time drops from minutes to seconds. For a company that processes more than 5 billion events daily, this means direct savings in licensing and infrastructure, and a substantial improvement in merchant confidence. The ability to maintain a 99.99% availability SLA even during peak loads is a testament to the robustness of the solution.
What lessons can other organizations draw from this experience? First, that investing in a robust streaming layer from the start is critical. Amazon MSK not only transports events, it becomes the central nervous system that connects producers, detection engines, and alert consumers. Second, that multi-tenure should be designed from day one, not added later. Sharing infrastructure while maintaining logical isolation through partitioned topics and row-level security policies is the key to scaling without skyrocketing costs. Third, that extensibility is vital: a system of plugins allows you to incorporate machine learning models (such as ETS or Prophet for forecasting) or complex event processing patterns without modifying the base platform.
These principles are perfectly applicable to any company that needs AWS and Azure cloud services to build real-time monitoring solutions. At Q2BSTUDIO, we understand that every business has unique particularities, which is why we offer AWS and Azure cloud services tailored to your needs, whether it's migrating legacy infrastructures or designing scalable data pipelines from scratch. In addition, our experience in custom software allows us to develop artificial intelligence for companies that enhances the detection of anomalies, combining machine learning techniques with declarative business rules. All backed by a focus on cybersecurity and business intelligence services that ensures sensitive data is protected and information reaches decision-makers at the right time.
Razorpay's evolution proves that anomaly detection is no longer a luxury, but a requirement to operate in the fintech sector. However, not every company has the volume or resources to build a platform from scratch. This is where technology partners come into play who offer custom applications and AI agents ready to integrate into existing environments. For example, our power bi solution allows you to visualize performance metrics and anomalies in real time, facilitating data-driven decision-making. In addition, incorporating AI for business using predictive models can anticipate failures before they occur, reducing downtime and improving the customer experience.
In short, the architecture based on Amazon MSK, Apache Flink, and ClickHouse represents a paradigm shift in transaction monitoring. By separating detection logic from execution infrastructure, organizations achieve agility, scalability, and cost reduction. The key is to adopt a declarative and multi-tenant approach by design, and to rely on specialized providers that provide knowledge in AWS and Azure cloud services, artificial intelligence and cybersecurity. If your company is ready to make the leap towards real-time anomaly detection, at Q2BSTUDIO we can help you design and implement the solution that best suits your context, ensuring that every second counts.
