KOEN
Read Original
#AI

Academic publications & Airbnb Tech: 2025 year in review

Academic publications & Airbnb Tech: 2025 year in review
01

Summary

Airbnb's 2025 AI Roadmap: From Multimodal Search to Agentic Support

A deep dive into 15 groundbreaking papers that redefined travel tech and data science at scale.

Airbnb recaps a pivotal year of research across ML, NLP, and data systems, showcasing how they bridge the gap between academic theory and production reality. These studies highlight innovative ways to optimize two-sided marketplaces and leverage LLMs for continuous operational improvement.

  • 01Achieved tens of millions in revenue growth via BiListing multimodal embeddings.
  • 02Revolutionized map-based discovery with a specialized NDCG metric.
  • 03Launched the Agent-in-the-Loop flywheel for real-time LLM support refinement.
  • 04Introduced SQL:Trek for automated database index design across MySQL/PostgreSQL.
  • 05Developed a causal framework to estimate long-term ranking impacts beyond experiment windows.

RECOMMENDATION

Engineers scaling ML-driven products should study Airbnb's approach to counterfactual evaluation and adaptive experimentation to accelerate their production cycles.

The Problem

Airbnb faced challenges including long A/B testing cycles due to travel booking patterns, the failure of list-based metrics on map-centric UIs, and the need for LLMs to stay updated with evolving support policies.

The Solution

They implemented interleaving and counterfactual evaluations for rapid ranking assessment, BiListing embeddings for multimodal data, and an Agent-in-the-Loop (AITL) framework for continuous LLM refinement via agent feedback.

The Result

The BiListing implementation drove tens of millions in incremental revenue with a 0.425% NDCB gain, while comparison-shopping models yielded a 1.7% NDCG increase and 0.6% booking conversion boost.

Trade-off

While rapid pre-A/B assessments like interleaving reduce experiment cycles, they may lack the ability to capture long-term seasonal dynamics, necessitating supplemental causal inference frameworks.

03

Key Concepts

Concept · 01

BiListing (Bimodal Listing)

A technique that integrates unstructured text and photo data into a single unified embedding vector using LLMs and pretrained vision-language models.

  • Used as ranking signals in search, leading to a 0.425% increase in Normalized Discounted Cumulative Booking (NDCB).
Concept · 02

Agent-in-the-Loop (AITL)

A data flywheel framework that leverages human agent interaction data to continuously improve the performance of LLM-based customer support systems.

  • Piloted in the US to maintain system accuracy against shifting user preferences and support policies.
Concept · 03

SQL:Trek

An automated index design approach that utilizes query compiler cost models to identify effective indexes across various relational databases.

  • Implemented to automate index optimization for MySQL and PostgreSQL derivatives within Airbnb's data systems.