This analysis summarizes the findings of 162 A/B tests conducted at Vinted, Europe's largest second-hand clothing marketplace, and compares statistical methods commonly used in experimentation: safe t-tests, classic t-tests, mSPRT, and safe proportion tests. The goal is to translate empirical evidence into practical recommendations for product and data teams operating in fast-moving marketplaces.
Key results: safe t-tests often agree with classic t-tests in many cases, but show an advantage in short-term metrics such as searches and sessions by detecting effects faster and with lower risk of premature erroneous decisions. However, in long-term metrics such as transactions and retention, safe t-tests perform worse than more conventional methods or experimental designs with a longer time window, suggesting that the choice of test should depend on the horizon and volatility of the metric.
Regarding proportion tests, the safe proportion test proved to outperform the chi-square test in certain SRM detection scenarios, i.e., sample proportion mismatches, especially when sample sizes or conversion rates are low or when exposure is monitored continuously. mSPRT offers advantages for sequential testing in environments where fast decision-making is needed, but requires careful design and control of type I error over time.
Practical implications: 1) choose the test based on the metric and its natural effect window; 2) use safe t-tests or mSPRT for immediate interaction metrics and when rapid iteration is required; 3) prefer classic tests or longer designs for transactional and business metrics with weak signals; 4) incorporate safe proportion tests and automated SRM checks to avoid deviations in experimental assignment; 5) instrument pipelines with continuous monitoring and dashboards to detect biases and changes in real time.
At Q2BSTUDIO we help implement these learnings in production. We are a custom software and application development company specialized in artificial intelligence, cybersecurity, and AWS and Azure cloud services. We design robust experimentation pipelines, custom software, and advanced analytics solutions that include business intelligence services and Power BI for visualization and monitoring of experimental KPIs.
Our services include building experimentation platforms, integration with data tools, deployment on AWS and Azure cloud, artificial intelligence models for funnel optimization, AI agents that automate alerts and decisions, and cybersecurity audits to protect the integrity of experiments. We offer complete AI solutions for businesses and custom software architecture that scales with the business.
If your marketplace needs to accelerate iterations without sacrificing statistical validity, consider a hybrid approach: safe tests for short-term tactical metrics, classic designs or extended windows for strategic objectives, and automated SRM detection mechanisms with safe proportion tests. At Q2BSTUDIO we can advise on technique selection, implement reproducible experiments, and deploy Power BI dashboards and reporting systems so teams can make fast and reliable decisions.
Integrated keywords for SEO: custom applications, custom software, artificial intelligence, cybersecurity, AWS and Azure cloud services, business intelligence services, AI for businesses, AI agents, Power BI.
