§ LaTeXML::Token
Representation of a token, and LaTeXML::Tokens, representing lists of tokens.
§ Description
This module defines Tokens (LaTeXML::Token, LaTeXML::Tokens) that get created during tokenization and expansion.
A LaTeXML::Token represents a TeX token which is a pair of a character or string and a category code. A LaTeXML::Tokens is a list of tokens (and also implements the API of a LaTeXML::Mouth so that tokens can be read from a list).
§ Common methods
The following methods apply to all objects.
- @tokens = $object->unlist;
-
Return a list of the tokens making up this $object.
- $string = $object->toString;
-
Return a string representing $object.
§ Token methods
The following methods are specific to LaTeXML::Token.
- $string = $token->getCSName;
-
Return the string or character part of the $token; for the special category codes, returns the standard string (eg. T_BEGIN-getCSName>returns "{").
- $string = $token->getString;
-
Return the string or character part of the $token.
- $code = $token->getCharcode;
-
Return the character code of the character part of the $token, or 256 if it is a control sequence.
- $code = $token->getCatcode;
-
Return the catcode of the $token.
- $defn = $token->getDefinition;
-
Return the current definition associated with $token in $STATE, or undef if none.
§ Tokens methods
The following methods are specific to LaTeXML::Tokens.
- $tokenscopy = $tokens->clone;
-
Return a shallow copy of the $tokens. This is useful before reading from a LaTeXML::Tokens.
- $token = $tokens->readToken;
-
Returns (and remove) the next token from $tokens. This is part of the public API of LaTeXML::Mouth so that a LaTeXML::Tokens can serve as a LaTeXML::Mouth.