Data cleaning

Real-world files arrive messy — inconsistent labels, duplicate rows, cryptic columns. Data cleaning adds a Standardisation step to the upload wizard that fixes those problems before the dataset exists, with AI doing the drafting and you approving every change.

Turn it on#

Cleaning is a per-project choice: switch on Enable Data Cleaning when creating a project, or later in the project's settings. The toggle notes the trade-off — "Enables data cleaning and standardisation during dataset upload. Maximum upload size: 800 MB."

With cleaning on, the wizard's steps become Upload Dataset → Standardisation → AI Readiness → Confirmation.

Review the inferred schema#

When your file lands, GoPie infers a schema and streams its progress: classifying columns, searching dictionaries, inferring enum values, assigning semantic groups. The result is an editable cleaning schema table — each column with its database type, a semantic type and role, and any matched dictionary — that you review and can edit inline before continuing.

Two ways to shape it:

• Edit in place — adjust what the inference got wrong.
• Refine with an instruction — describe changes in plain language: "treat state_name as geographical", "mark population as a measure", "remove the id column". Select Apply edit and the AI produces a new schema version.

When it looks right, select Approve & clean — that starts the interactive cleaning waves.

Note

Dictionaries — reference tables of coded values and their canonical labels, managed on the project's Dictionaries tab — are what the inference matches columns against. A matched dictionary standardises that column's values and can add enrichment columns.

Answer the quick questionnaire#

Before planning, two yes/no questions:

• Remove duplicate columns? — columns with identical names are deduplicated, keeping the first.
• Remove rows where all values are null? — entirely-empty rows are dropped.

Work through the cleaning waves#

GoPie then works the dataset in stages, surfacing one wave of decisions at a time — grouped by column when there are many:

• Confirm — yes/no calls, like applying a suggested fix.
• Choose — value standardisations the AI is less sure about ("Karntaka" → "Karnataka"), picked from suggested options or entered as free input.
• Order — a suggested column ordering you can drag to adjust.

Answer a wave's decisions and select Apply & continue; GoPie validates them and advances to the next wave. Repeat until cleaning completes, at which point a report shows what changed. The cleaned data then carries forward automatically.

From there the wizard continues as normal — AI Readiness, then Confirmation. The dataset is created from the cleaned data; your original file isn't modified.

Get the cleaning report#

On the success screen, Download Metadata produces a report (PDF or JSON) of the finished dataset — overview (rows, columns, inferred time period, units, granularity), per-column details, and the dictionaries used during cleaning. It doubles as a data-dictionary you can hand to consumers of the dataset.

Mind the limits#

• 800 MB per file in cleaning-enabled projects (the standard cap applies elsewhere).
• A cleaning session stays live for 72 hours — leave the wizard for longer and you'll need to start the upload again.
• Your accepted schema and answered decisions are saved in the session, so a recoverable wave failure can be retried from the wizard without re-uploading.