Remove Duplicates Node

Remove duplicate items based on field values. V1 only — V2 cross-execution deduplication unavailable.

Written By pvdyck

Last updated 21 minutes ago

The Remove Duplicates node removes items that have the same value for one or more specified fields within a single execution.

Parameters

Parameter	Description
Compare	What to compare: All Fields (entire item must match) or Selected Fields (match on specific fields only)
Fields to Compare	When using Selected Fields, the field name(s) to check for duplicates (e.g., `email`, `id`)

Option	Description
Disable Dot Notation	Treat dots in field names as literal characters

Items are compared in order. The first occurrence is kept; subsequent duplicates are removed.
When comparing all fields, every field value must match for items to be considered duplicates.
When comparing selected fields, only the specified fields are checked.

V1 only -- Only the V1 remove-duplicates logic is available
V2 features unavailable -- Cross-execution deduplication (storing seen values between workflow runs) is not supported. Each execution starts fresh.

For cross-execution deduplication, use an external key-value store via the HTTP Request node (e.g., a Redis API endpoint or KV store).