Syntax
SourceData=<string>
[Introducers=<string>]
[Terminators=<string>]
[Separators=<string>]
[Escapes=<string>]
[DiscardEscapes]
Result=<result-var>Description
Creates a list of tokens from an input string according to the supplied specifiers.Parameters
SourceData
- The string to be processed.
Introducers
- Optional. Characters that mark the beginning of a token in the SourceData. The character is included in the token extracted.
Terminators
- Optional. Characters that mark the end of a token in the SourceData. The character is included in the token extracted.
Separators
- Optional. Characters that separate tokens in the SourceData. The characters are not included in the extracted tokens.
Escapes
- Optional. Characters that delimit a section of the SourceData that is not to be tokenized, but rather passed through without processing.
DiscardEscapes
- Optional. If this parameter is present, the escape characters are discarded (i.e. they are not included in the output tokens of which they were a part)
Result
- The list of tokens extracted from the SourceData.
Remarks
Sequences of multiple separator characters in the string are treated as a single separator.The StringAsList statement provides more limited capabilities but is easier to use.
Example
str = "This is the P1Systems <A HREF="http://www.p1.com/index.html">home page</A>."
Tokenize
SourceData=str
Introducers="<"
Terminators=">"
Escapes=""""
Result=tokens
MessageBox Text=tokensThe above code fragment displays the string "This is the P1Systems ,<A HREF="http://www.p1.com/index.html">,home page,</A>,." in the message box.