KOEN
Read Original
#Backend

Smarter URL Normalization at Scale: How MIQPS Powers Content Deduplication at Pinterest

Smarter URL Normalization at Scale: How MIQPS Powers Content Deduplication at Pinterest
01

Summary

Stop Wasting Resources on Duplicate URLs: Pinterest’s Smart Normalization Engine

How MIQPS uses visual content signals to master URL deduplication at a global scale.

This article explores how Pinterest solves the challenge of URL duplication using the MIQPS algorithm. It details a data-driven approach that identifies essential query parameters by analyzing visual content changes, combined with a robust architecture for safe deployment.

  • 01Automated classification of parameter importance using visual content fingerprints
  • 02Dynamic generation of per-domain MIQPS maps for scalable deduplication
  • 03Hybrid normalization strategy combining static curated rules with dynamic learning
  • 04Regression prevention through an automated anomaly detection layer
  • 05Offline computation model that optimizes runtime latency and infrastructure costs

RECOMMENDATION

Highly recommended for backend and infrastructure engineers building large-scale data ingestion or crawling systems aimed at optimizing compute and storage resources.

The Problem

URLs ingested from millions of domains often include redundant tracking parameters like UTM or session tokens, causing identical pages to be independently fetched and rendered, leading to massive computational waste.

The Solution

Pinterest developed the Minimal Important Query Param Set (MIQPS) algorithm, which learns parameter importance by comparing visual Content IDs of pages with and without specific parameters, integrated into a multi-layer normalization pipeline.

The Result

The system achieved dynamic URL normalization across a vast number of domains, significantly reducing redundant processing and ensuring high catalog consistency and search/recommendation quality.

Trade-off

An offline batch processing approach was chosen to minimize latency, resulting in minor staleness of normalization rules, and a conservative default is used to prevent accidental data loss.

03

Key Concepts

Concept · 01

MIQPS

Short for Minimal Important Query Param Set, it refers to the smallest collection of query parameters required to uniquely identify content on a specific domain.

  • Used to create independent parameter importance maps for each merchant domain.
  • Evaluates whether a parameter is neutral or non-neutral based on content change thresholds.
Concept · 02

Content ID

A digital fingerprint or hash derived from the rendered visual representation of a web page.

  • Serves as the ground truth for content identity, regardless of HTML variations.
  • Enables empirical testing of how specific URL parameters affect the actual page content.
Concept · 03

Multi-Layer Normalization

A defense-in-depth strategy that combines multiple layers of logic to normalize URLs.

  • Integrates static allowlists, regex patterns, and MIQPS-computed dynamic maps.
  • Ensures broad coverage by keeping a parameter if it matches any of the defined security layers.