Efficient Deduplication

Deduplication is one of the most resource-intensive operations in analytical workloads. Apache Doris provides two dedicated data types as alternatives to COUNT DISTINCT, completing deduplication with lower memory and latency cost: choose BITMAP when you need exact results, and choose HLL when you can accept a 1%–2% error in exchange for smaller storage.

Exact Deduplication

BITMAP Exact Deduplication

Use the BITMAP type as a replacement for COUNT DISTINCT to achieve exact deduplication while improving query speed and reducing memory and disk usage.

Approximate Deduplication

HLL Approximate Deduplication

HyperLogLog-based approximate deduplication with error controlled within 1%–2% and small storage footprint, suitable for large-scale UV and cardinality statistics scenarios.