Lexer (xonsh.lexer)

Lexer for xonsh code.

Written using a hybrid of tokenize and PLY.

class xonsh.lexer.Lexer[source]

Implements a lexer for the xonsh language.

Attributes

fname (str) Filename
last (token) The last token seen.
lineno (int) The last line number seen.
build(**kwargs)[source]

Part of the PLY lexer API.

input(s)[source]

Calls the lexer on the string s.

reset()[source]
token()[source]

Retrieves the next token.

tokens = ('NUMBER', 'STRING', 'NEWLINE', 'MOD', 'DEDENT', 'TILDE', 'EQUALS', 'RSHIFTEQUAL', 'ATDOLLAR', 'NE', 'PLUS', 'SEMI', 'LE', 'INDENT', 'AMPERSAND', 'PLUSEQUAL', 'POWEQUAL', 'TIMESEQUAL', 'LSHIFT', 'GE', 'AMPERSANDEQUAL', 'ATEQUAL', 'DIVEQUAL', 'MINUSEQUAL', 'AWAIT', 'ASYNC', 'COMMA', 'REGEXPATH', 'IOREDIRECT', 'DOLLAR_NAME', 'COLON', 'TIMES', 'PIPEEQUAL', 'EQ', 'DOUBLEDIVEQUAL', 'AT', 'DOUBLE_QUESTION', 'GT', 'LSHIFTEQUAL', 'POW', 'PIPE', 'RARROW', 'RSHIFT', 'MODEQUAL', 'XOREQUAL', 'LT', 'DIVIDE', 'MINUS', 'DOUBLEDIV', 'QUESTION', 'ELLIPSIS', 'XOR', 'PERIOD', 'NAME', 'WS', 'LPAREN', 'RPAREN', 'LBRACKET', 'RBRACKET', 'LBRACE', 'RBRACE', 'AT_LPAREN', 'BANG_LPAREN', 'BANG_LBRACKET', 'DOLLAR_LPAREN', 'DOLLAR_LBRACE', 'DOLLAR_LBRACKET', 'ATDOLLAR_LPAREN', 'FALSE', 'NONE', 'TRUE', 'AND', 'AS', 'ASSERT', 'BREAK', 'CLASS', 'CONTINUE', 'DEF', 'DEL', 'ELIF', 'ELSE', 'EXCEPT', 'FINALLY', 'FOR', 'FROM', 'GLOBAL', 'IF', 'IMPORT', 'IN', 'IS', 'LAMBDA', 'NONLOCAL', 'NOT', 'OR', 'PASS', 'RAISE', 'RETURN', 'TRY', 'WHILE', 'WITH', 'YIELD')
xonsh.lexer.get_tokens(s)[source]

Given a string containing xonsh code, generates a stream of relevant PLY tokens using handle_token.

xonsh.lexer.handle_double_amps(state, token)[source]
xonsh.lexer.handle_double_pipe(state, token)[source]
xonsh.lexer.handle_error_space(state, token)[source]

Function for handling special whitespace characters in subprocess mode

xonsh.lexer.handle_error_token(state, token)[source]

Function for handling error tokens

xonsh.lexer.handle_ignore(state, token)[source]

Function for handling tokens that should be ignored

xonsh.lexer.handle_name(state, token)[source]

Function for handling name tokens

xonsh.lexer.handle_rbrace(state, token)[source]

Function for handling }

xonsh.lexer.handle_rbracket(state, token)[source]

Function for handling ]

xonsh.lexer.handle_rparen(state, token)[source]

Function for handling )

xonsh.lexer.handle_token(state, token)[source]

General-purpose token handler. Makes use of token_map or special_map to yield one or more PLY tokens from the given input.

Parameters:

state :

The current state of the lexer, including information about whether we are in Python mode or subprocess mode, which changes the lexer’s behavior. Also includes the stream of tokens yet to be considered.

token :

The token (from tokenize) currently under consideration

xonsh.lexer.special_handlers = {0: <function handle_ignore at 0x7fea90fc3730>, 1: <function handle_name at 0x7fea90fc3378>, (53, '$['): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc3ae8>, (53, ')'): <function handle_rparen at 0x7fea90fc3488>, (53, '}'): <function handle_rbrace at 0x7fea90fc3510>, (53, '['): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc3950>, (53, '${'): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc3b70>, (53, '@('): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc3d08>, (53, '!('): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc3bf8>, (53, '@$('): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc3d90>, (53, ']'): <function handle_rbracket at 0x7fea90fc3598>, (53, '{'): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc39d8>, (53, '||'): <function handle_double_pipe at 0x7fea90fc3840>, (53, '('): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc38c8>, (56, ' '): <function handle_error_space at 0x7fea90fc3620>, (53, '!['): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc3c80>, (53, '$('): <function _make_matcher_handler.<locals>._inner_handler at 0x7fea90fc3a60>, (53, '&&'): <function handle_double_amps at 0x7fea90fc37b8>, 56: <function handle_error_token at 0x7fea90fc36a8>, 57: <function handle_ignore at 0x7fea90fc3730>, 58: <function handle_ignore at 0x7fea90fc3730>, 59: <function handle_ignore at 0x7fea90fc3730>}

Mapping from tokenize tokens (or token types) to the proper function for generating PLY tokens from them. In addition to yielding PLY tokens, these functions may manipulate the Lexer’s state.

xonsh.lexer.token_map = {2: 'NUMBER', 3: 'STRING', 4: 'NEWLINE', (53, '%'): 'MOD', 6: 'DEDENT', (53, '~'): 'TILDE', (53, '='): 'EQUALS', (53, '>>='): 'RSHIFTEQUAL', (53, '@$'): 'ATDOLLAR', (53, '!='): 'NE', (53, '+'): 'PLUS', (53, ';'): 'SEMI', (53, '<='): 'LE', 5: 'INDENT', (53, '&'): 'AMPERSAND', (53, '+='): 'PLUSEQUAL', (53, '**='): 'POWEQUAL', (53, '*='): 'TIMESEQUAL', (53, '<<'): 'LSHIFT', (53, '>='): 'GE', (53, '&='): 'AMPERSANDEQUAL', (53, '@='): 'ATEQUAL', (53, '/='): 'DIVEQUAL', (53, '-='): 'MINUSEQUAL', 54: 'AWAIT', 55: 'ASYNC', (53, ','): 'COMMA', 60: 'REGEXPATH', 61: 'IOREDIRECT', 62: 'DOLLAR_NAME', (53, ':'): 'COLON', (53, '*'): 'TIMES', (53, '|='): 'PIPEEQUAL', (53, '=='): 'EQ', (53, '//='): 'DOUBLEDIVEQUAL', (53, '@'): 'AT', (53, '??'): 'DOUBLE_QUESTION', (53, '>'): 'GT', (53, '<<='): 'LSHIFTEQUAL', (53, '**'): 'POW', (53, '|'): 'PIPE', (53, '->'): 'RARROW', (53, '>>'): 'RSHIFT', (53, '%='): 'MODEQUAL', (53, '^='): 'XOREQUAL', (53, '<'): 'LT', (53, '/'): 'DIVIDE', (53, '-'): 'MINUS', (53, '//'): 'DOUBLEDIV', (53, '?'): 'QUESTION', (53, '...'): 'ELLIPSIS', (53, '^'): 'XOR', (53, '.'): 'PERIOD'}

Mapping from tokenize tokens (or token types) to PLY token types. If a simple one-to-one mapping from tokenize to PLY exists, the lexer will look it up here and generate a single PLY token of the given type. Otherwise, it will fall back to handling that token using one of the handlers in special_handlers.