Free · no signup · browser-local

JSONL Dataset Deduplicator by Row or Text Field

Deduplicate JSONL by raw row or a named text field. Configure case, Unicode and whitespace normalization, keep first or last, and review conflicting records before export.

Browse all tools →

Browser-local preparation · no model calls · no input storage

Choose exact comparison

Default matching is exact and includes raw line spaces. Optional transformations affect comparison keys only, in order: NFKC → Unicode lowercase (not full Unicode case folding) → whitespace. They can merge distinct records. No fuzzy or semantic matching.

Surviving records remain in original source order. All physical lines must be valid JSONL; blank lines, BOM, duplicate keys and invalid fields block the entire export. Original strings and numeric spellings are preserved; output record separators are LF.

Your output

kept.jsonl

Process the input to preview, copy and download a result.

This tool finds exact duplicates after chosen normalization, not semantic duplicates or train/test leakage. The report contains counts and source positions, without original comparison values. Editing input or options invalidates results and pending exports. Inputs are never saved or placed in URLs.

Getting started

Choose a raw-line or text-field key, optional normalization and first/last retention. Review conflicts before downloading kept, removed and report files.

Before you use the result

This is exact or normalized text matching, not semantic deduplication. Matching prompts can have different answers; inspect the conflict groups.

Try an example

Compare question values with case and whitespace differences, then inspect how enabling normalization changes the groups.

How it works

Local bounded JSONL validation and deterministic grouping; source numeric spellings and row text remain unchanged in kept/removed exports.

Frequently asked questions

Find duplicate raw rows or matching text fields with explicit normalization, conflict review and preserved originals.

How do I use AI JSONL Dataset Deduplicator?

Choose a raw-line or text-field key, optional normalization and first/last retention. Review conflicts before downloading kept, removed and report files.

Does this tool upload my data?

Inputs and files are processed in your browser. Page-view analytics do not contain your tool input.

What should I check before using the result?

This is exact or normalized text matching, not semantic deduplication. Matching prompts can have different answers; inspect the conflict groups.

Related tools

AI Training Dataset ConverterMap single-turn CSV or JSON question and answer fields into messages or Alpaca JSONL, with an explicit row report.AI Dataset Train / Validation / Test SplitterSplit JSONL reproducibly with a seed, explicit ratios, optional duplicate removal and a field that keeps groups together.AI Response JSON ExtractorFind complete JSON objects and arrays inside mixed response text, select a candidate and export without rounding numeric values.