Syntax for the pattern tokenizer

When creating a new token profile you can select Pattern as Tokenizer. In this case you need to define a custom pattern for the tokens to be generated.

The pattern syntax is similar to regular expressions and has the following notation:

Table 1. Pattern tokenizer syntax
Syntax element Explanation Example
<?> The characters between the parentheses are output directly.

Allowed characters: valid chars except ">" which has to be escaped: \>
  • Input in the Pattern field: <test>
  • Token input value: Homer Simpson
  • Token output value: test
[?] One character is chosen randomly from:
  • a-z - one letter from a-z
  • A-Z - one letter from A-Z
  • 0-9 - one digit from 0-9
  • 1-9 - one digit from 1-9
  • User specified chars:
    • Example: abcdefg.€$%& - one character from the noted is chosen
    • Exception: "]" has to be escaped: \]
  • Also possible are combinations like: [a-zA-Z0-9!$%&]
  • Input in the Pattern field: [a-zA-Z0-9!$%&]
  • Token input value: Homer Simpson
  • Token output value: o
[](?) In the round parentheses you enter how many characters should be generated.
  • Input in the Pattern field: [a-zA-Z0-9](8)
  • Token input value: Homer Simpson
  • Token output value: 2jdfEFJX
[](#) Generate a token with the same length as the input value (# = length of input value).
  • Input in the Pattern field: [a-zA-Z0-9](#)
  • Token input value: Homer Simpson
  • Token output value: v9CvSZ5bCQ1rO
[](#-?) Generate a token with the same length as the input value minus the noted number.
  • Input in the Pattern field: [a-zA-Z0-9](#-3)
  • Token input value: Homer Simpson
  • Token output value: GE7zGtcUQL
{?} Take from the input value the char from the noted position.
Note: Regarding the position of chars, One-based numbering is used (the initial element of a sequence is assigned the index 1).
  • Input in the Pattern field: [a-zA-Z0-9](#){7}
  • Token input value: Homer Simpson
  • Token output value: In0x5lVkFtol1S
{?...?} Take from the input value the chars from the noted start to end.
  • Input in the Pattern field: [a-zA-Z0-9](#){1...5}
  • Token input value: Homer Simpson
  • Token output value: MhnSwmxnhKvnnHomer
{?...#} Take from the input value the chars from the noted start to the end of the input value.
  • Input in the Pattern field: [a-zA-Z0-9](#){7...#}
  • Token input value: Homer Simpson
  • Token output value: j7xonYWdFiVzNSimpson
{#-x} Select a single character from the input value with the index # (end of input) minus x.
  • Input in the Pattern field: [a-zA-Z0-9](#){#-1}
  • Token input value: Homer Simpson
  • Token output value: j7xonYWdFiVzNo

Some more examples:

  • Input in the Pattern field: <XYZ>{4...#}[0-9](3)
  • Token input value: qwertz
  • Token output value: XYZrtz771

  • Input in the Pattern field: [a-z](#)
  • Token input value: qwertz
  • Token output value: stlgmu

  • Input in the Pattern field: [1-9](1)[0-9](7)<9>[0-9](7)
  • Token input value: 1234567890123456
  • Token output value: 5794786197851005

Note: Any parts of the pattern that go out of bounds are ignored / tokenized to "". For example: With the input value 12345 the pattern <outofrange>{#-10}[A-Z](3) produces the following token: outofrangeKFV or something similar. Whereas with the input value 1234567890abcdef a token like outofrange6BEM is produced.
Note: Pattern Tokenizers can be generic under the condition of having a prefix with a minimum length of three characters (e.g., <PRE>[a-zA-Z0-9](8)).