In-Run Event Stream Deduplication and Cumulative Routing

How it works

1. 1.Trigger workflow manually using Manual Trigger.
2. 2.Emit sample batch with known duplicate event IDs via Code node.
3. 3.Process stream using Map-based deduplication and attach occurrence metrics.
4. 4.Branch items using If node based on boolean is_unique flag.
5. 5.Tag unique items for production and aggregate into final reconciliation report.
6. 6.Tag duplicate items for audit archive and route to NoOp sink.

Use cases

• Preventing duplicate webhook processing from systems with aggressive retry policies.
• Cleaning transaction streams prior to ERP ledger insertion.
• Deduplicating batched IoT sensor telemetry payloads in memory.

Requirements

• n8n version 1.0.0 or higher.
• No third-party credentials or API integrations required.

Setup and testing in n8n

1. Import the workflow and click 'Test Step' on the Manual Trigger node.
2. Inspect 'Generate Event Batch' to verify 7 sample records are emitted.
3. Verify 4 records take the true branch and 3 duplicate records take the false branch.
4. Check 'Build Reconciliation Report' for the single summary object with matching figures.

Service Value

Ready-made workflow template for automation delivery and service execution.