Scalable personalized recommendation system on AWS: batch to real-time

In today's digital commerce landscape, delivering personalized and relevant recommendations in real time has become a key differentiator for user retention and satisfaction. However, building a recommendation system capable of processing millions of requests per second, maintaining data freshness, and controlling infrastructure costs poses both an orchestration and a machine learning challenge. At Q2BSTUDIO, as a company specialized in custom software development and cloud solutions, we have designed and implemented architectures that combine batch processing with real-time capabilities, leveraging the AWS ecosystem to deliver personalized experiences at scale.

This article explores how to build a scalable recommendation system following a 'batch-first' approach and progressively extending it with real-time components. The architecture relies on services such as AWS Lake Formation, Amazon Managed Workflows for Apache Airflow (MWAA), Amazon Athena, AWS Glue, Amazon SageMaker, Amazon DynamoDB, and Amazon MemoryDB, all orchestrated to balance operational efficiency and minimal latency. Additionally, it addresses how to integrate artificial intelligence signals and AI agents to enrich user profiles without inflating costs.

Most recommendation systems start with a batch pipeline that processes historical data — catalogs, transactions, embeddings — and generates candidate lists stored in a low-latency database. This approach is reliable, cost-effective, and easy to debug. In our experience, the first step is to build a centralized data lake that serves as the single source of truth. With AWS Lake Formation and Amazon S3, we create curated, validated 'golden' datasets accessible securely from multiple AWS accounts. This eliminates data duplication and ensures that all pipelines (model training, batch inference, BI dashboards) work on the same information.

Orchestration is handled by Amazon MWAA, which manages complex workflows defined in Python code. Each pipeline follows a consistent pattern: data extraction via Amazon Athena, intensive transformations with AWS Glue (PySpark), model training and vector search with Amazon SageMaker, and publishing results to DynamoDB. To accelerate development, we create reusable custom operators that encapsulate common logic (IAM role management, writing to convention-based S3 paths, automatic retries). At Q2BSTUDIO we apply the same philosophy when developing custom applications: invest in reusable components so that teams can ship new features in days, not weeks.

A key aspect is marketplace isolation. Each marketplace runs in its own task group within the DAG, so a failure in one does not affect the others. Pipelines run sequentially to avoid resource contention on heavy SageMaker and Glue jobs. All data exchange between steps uses implicit S3 paths, without hard-coded addresses.

The batch model covers most use cases (recommendations based on purchase history, catalog relationships), but there are situations where information changes in seconds: recent searches, cart additions, in-session activity. For these scenarios, we extended the system with Amazon MemoryDB (Valkey-compatible) for real-time vector search and SageMaker endpoints for on-demand embedding generation. The same batch pipeline that publishes to DynamoDB also weekly refreshes the product index in MemoryDB. At serving time, when a user performs a recent action, the system generates an embedding on the fly, queries the nearest neighbors in MemoryDB (sub-millisecond latency), and combines those results with batch candidates through re-ranking in the presentation layer.

This hybrid architecture maintains the efficiency of batch processing for slow signals — like historical preferences — and the agility of real time for fast signals — like the current session. All while sharing the same models, the same data lake, and the same serving service. At Q2BSTUDIO we help our clients design such gradual transitions, avoiding the temptation to build a fully real-time system from the start. As we've learned, 'batch first, real-time later' is a strategy that reduces risk and cost.

Furthermore, integrating AI agents allows further personalization of results. For example, an agent can analyze session context and apply dynamic business rules before returning the final recommendation. These agents are deployed as serverless functions or SageMaker endpoints, and orchestrated within the same Airflow flow or from the serving service.

Security is cross-cutting. With Lake Formation we control table-level access across accounts, each compute engine (MWAA, Glue, SageMaker) uses least-privilege IAM roles, and environments are deployed in VPCs isolated by region. Reliability relies on regional isolation: each region runs its own independent copy of the system, so a failure in one does not affect others. If a batch pipeline fails mid-execution, the previous DynamoDB data continues serving until the next update. Automatic retries from Airflow and DynamoDB TTL expiration ensure that stale data does not persist indefinitely.

Key lessons learned include:

Separate compute and storage: Using S3 as an intermediate layer and ephemeral Glue/SageMaker jobs lets you pay only for active compute, with no idle clusters between weekly runs.

Invest in a governed data lake: Golden datasets accelerate onboarding of new pipelines and make model comparisons trustworthy. The initial investment in Lake Formation pays off quickly.

Not all signals need real time: Classify signals by speed of change and route them appropriately (batch for slow, real-time for fast, re-ranking for in-between) to optimize cost and performance.

At Q2BSTUDIO we apply these principles in every digital transformation project. Whether developing custom software with cloud components, integrating artificial intelligence, or implementing cybersecurity solutions, our goal is to deliver scalable, secure systems tailored to real business needs. If you'd like to explore how to implement a personalized recommendation system in your organization, we invite you to learn about our services in custom software development and cloud AWS/Azure.

In summary, building a scalable recommendation system on AWS is not just about choosing the right services, but about orchestrating them following an iterative approach that prioritizes batch efficiency before adding real-time complexity. The combination of Amazon MWAA, Lake Formation, SageMaker, DynamoDB, and MemoryDB, together with AI agents and a governed data lake, provides a solid foundation for delivering personalized experiences to millions of users without skyrocketing costs. As always, the best system is one that evolves with the business.