UnicodeNormalizer

The UnicodeNormalizer is an input processor that normalizes Unicode text to ensure consistent formatting and remove potentially problematic characters before messages are sent to the language model. This processor handles Unicode representations and removes control characters. It also standardizes whitespace formatting.

Usage example

import { UnicodeNormalizer } from '@mastra/core/processors'

const processor = new UnicodeNormalizer({
  stripControlChars: true,
  collapseWhitespace: true,
})

Constructor parameters

options?:

Options

Configuration options for Unicode text normalization

Options

stripControlChars?:

boolean

Whether to strip control characters. When enabled, removes control characters except , ,

preserveEmojis?:

boolean

Whether to preserve emojis. When disabled, emojis may be removed if they contain control characters

collapseWhitespace?:

boolean

Whether to collapse consecutive whitespace. When enabled, multiple spaces/tabs/newlines are collapsed to single instances

trim?:

boolean

Whether to trim leading and trailing whitespace

Returns

id:

string

Processor identifier set to 'unicode-normalizer'

name:

string

Processor display name. Always Unicode Normalizer.

processInput:

(args: { messages: MastraDBMessage[]; abort: (reason?: string) => never }) => MastraDBMessage[]

Processes input messages to normalize Unicode text

Extended usage example

src/mastra/agents/normalized-agent.ts

import { Agent } from '@mastra/core/agent'
import { UnicodeNormalizer } from '@mastra/core/processors'

export const agent = new Agent({
  id: 'normalized-agent',
  name: 'normalized-agent',
  instructions: 'You are a helpful assistant',
  model: 'openai/gpt-5.6-sol',
  inputProcessors: [
    new UnicodeNormalizer({
      stripControlChars: true,
      preserveEmojis: true,
      collapseWhitespace: true,
      trim: true,
    }),
  ],
})

Related

• Guardrails