Lexer (xonsh.lexer)

Lexer for xonsh code.

Written using a hybrid of tokenize and PLY.

class xonsh.lexer.Lexer[source]

Implements a lexer for the xonsh language.

Attributes

fname (str) Filename
last (token) The last token seen.
lineno (int) The last line number seen.
build(**kwargs)[source]

Part of the PLY lexer API.

input(s)[source]

Calls the lexer on the string s.

reset()[source]
token()[source]

Retrieves the next token.

tokens = ('EQ', 'STRING', 'DIVIDE', 'GT', 'DEDENT', 'QUESTION', 'POW', 'RSHIFTEQUAL', 'COMMA', 'LE', 'MINUSEQUAL', 'PLUSEQUAL', 'DIVEQUAL', 'INDENT', 'AT', 'LT', 'AMPERSANDEQUAL', 'LSHIFT', 'NE', 'PIPEEQUAL', 'TIMES', 'ATEQUAL', 'PIPE', 'AWAIT', 'ASYNC', 'TILDE', 'PLUS', 'REGEXPATH', 'ATDOLLAR', 'GE', 'MODEQUAL', 'NUMBER', 'RSHIFT', 'DOLLAR_NAME', 'DOUBLE_QUESTION', 'PERIOD', 'COLON', 'DOUBLEDIV', 'SEMI', 'MOD', 'DOUBLEDIVEQUAL', 'POWEQUAL', 'RARROW', 'TIMESEQUAL', 'XOR', 'IOREDIRECT', 'ELLIPSIS', 'LSHIFTEQUAL', 'EQUALS', 'AMPERSAND', 'XOREQUAL', 'NEWLINE', 'MINUS', 'NAME', 'WS', 'LPAREN', 'RPAREN', 'LBRACKET', 'RBRACKET', 'LBRACE', 'RBRACE', 'AT_LPAREN', 'BANG_LPAREN', 'BANG_LBRACKET', 'DOLLAR_LPAREN', 'DOLLAR_LBRACE', 'DOLLAR_LBRACKET', 'ATDOLLAR_LPAREN', 'FALSE', 'NONE', 'TRUE', 'AND', 'AS', 'ASSERT', 'BREAK', 'CLASS', 'CONTINUE', 'DEF', 'DEL', 'ELIF', 'ELSE', 'EXCEPT', 'FINALLY', 'FOR', 'FROM', 'GLOBAL', 'IF', 'IMPORT', 'IN', 'IS', 'LAMBDA', 'NONLOCAL', 'NOT', 'OR', 'PASS', 'RAISE', 'RETURN', 'TRY', 'WHILE', 'WITH', 'YIELD')
xonsh.lexer.get_tokens(s)[source]

Given a string containing xonsh code, generates a stream of relevant PLY tokens using handle_token.

xonsh.lexer.handle_double_amps(state, token)[source]
xonsh.lexer.handle_double_pipe(state, token)[source]
xonsh.lexer.handle_error_space(state, token)[source]

Function for handling special whitespace characters in subprocess mode

xonsh.lexer.handle_error_token(state, token)[source]

Function for handling error tokens

xonsh.lexer.handle_ignore(state, token)[source]

Function for handling tokens that should be ignored

xonsh.lexer.handle_name(state, token)[source]

Function for handling name tokens

xonsh.lexer.handle_rbrace(state, token)[source]

Function for handling }

xonsh.lexer.handle_rbracket(state, token)[source]

Function for handling ]

xonsh.lexer.handle_rparen(state, token)[source]

Function for handling )

xonsh.lexer.handle_token(state, token)[source]

General-purpose token handler. Makes use of token_map or special_map to yield one or more PLY tokens from the given input.

Parameters:

state :

The current state of the lexer, including information about whether we are in Python mode or subprocess mode, which changes the lexer’s behavior. Also includes the stream of tokens yet to be considered.

token :

The token (from tokenize) currently under consideration

xonsh.lexer.special_handlers = {0: <function handle_ignore at 0x7f4c365339d8>, 1: <function handle_name at 0x7f4c36533620>, (53, '$('): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36533d08>, (53, '&&'): <function handle_double_amps at 0x7f4c36533a60>, (53, '$['): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36533d90>, (53, '||'): <function handle_double_pipe at 0x7f4c36533ae8>, (56, ' '): <function handle_error_space at 0x7f4c365338c8>, (53, '@('): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36531048>, (53, '@$('): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c365310d0>, (53, '['): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36533bf8>, (53, '!['): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36533f28>, (53, '('): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36533b70>, 59: <function handle_ignore at 0x7f4c365339d8>, (53, '${'): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36533e18>, (53, ']'): <function handle_rbracket at 0x7f4c36533840>, (53, '!('): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36533ea0>, (53, ')'): <function handle_rparen at 0x7f4c36533730>, 56: <function handle_error_token at 0x7f4c36533950>, 57: <function handle_ignore at 0x7f4c365339d8>, 58: <function handle_ignore at 0x7f4c365339d8>, (53, '{'): <function _make_matcher_handler.<locals>._inner_handler at 0x7f4c36533c80>, (53, '}'): <function handle_rbrace at 0x7f4c365337b8>}

Mapping from tokenize tokens (or token types) to the proper function for generating PLY tokens from them. In addition to yielding PLY tokens, these functions may manipulate the Lexer’s state.

xonsh.lexer.token_map = {(53, '=='): 'EQ', 3: 'STRING', (53, '/'): 'DIVIDE', (53, '>'): 'GT', 6: 'DEDENT', (53, '?'): 'QUESTION', (53, '**'): 'POW', (53, '>>='): 'RSHIFTEQUAL', (53, ','): 'COMMA', (53, '<='): 'LE', (53, '-='): 'MINUSEQUAL', (53, '+='): 'PLUSEQUAL', (53, '/='): 'DIVEQUAL', 5: 'INDENT', (53, '@'): 'AT', (53, '<'): 'LT', (53, '&='): 'AMPERSANDEQUAL', (53, '<<'): 'LSHIFT', (53, '!='): 'NE', (53, '|='): 'PIPEEQUAL', (53, '*'): 'TIMES', (53, '@='): 'ATEQUAL', (53, '|'): 'PIPE', 54: 'AWAIT', 55: 'ASYNC', (53, '~'): 'TILDE', (53, '+'): 'PLUS', 60: 'REGEXPATH', (53, '@$'): 'ATDOLLAR', (53, '>='): 'GE', (53, '%='): 'MODEQUAL', 2: 'NUMBER', (53, '>>'): 'RSHIFT', 62: 'DOLLAR_NAME', (53, '??'): 'DOUBLE_QUESTION', (53, '.'): 'PERIOD', (53, ':'): 'COLON', (53, '//'): 'DOUBLEDIV', (53, ';'): 'SEMI', (53, '%'): 'MOD', (53, '//='): 'DOUBLEDIVEQUAL', (53, '**='): 'POWEQUAL', (53, '->'): 'RARROW', (53, '*='): 'TIMESEQUAL', (53, '^'): 'XOR', 61: 'IOREDIRECT', (53, '...'): 'ELLIPSIS', (53, '<<='): 'LSHIFTEQUAL', (53, '='): 'EQUALS', (53, '&'): 'AMPERSAND', (53, '^='): 'XOREQUAL', 4: 'NEWLINE', (53, '-'): 'MINUS'}

Mapping from tokenize tokens (or token types) to PLY token types. If a simple one-to-one mapping from tokenize to PLY exists, the lexer will look it up here and generate a single PLY token of the given type. Otherwise, it will fall back to handling that token using one of the handlers in special_handlers.