graphtage.expressions¶

A safer-than-Python-eval expression evaluation module.

It is extensible, supporting infix operators with adjustable precedence. It supports function calls, member lookup (getitem()), and provides a modicum of safety by only allowing access to a programmer-defined set of variables. By default, it also disallows access to protected and private member variables.

Example

Here is an example of its usage:

>>> parsed = parse('foo[(bar + 10) * 2]')
>>> parsed
Expression(rpn=(IdentifierToken('foo'), IdentifierToken('bar'), IntegerToken(raw_str='10', value=10), OperatorToken(op=<Operator.ADDITION: ('+', 5, <function Operator.<lambda> at 0x135057f80>)>), IntegerToken(raw_str='2', value=2), OperatorToken(op=<Operator.MULTIPLICATION: ('*', 4, <function Operator.<lambda> at 0x135057d40>)>), OpenBracket()))
>>> parsed.eval(locals={
...     'foo': {
...         40: 1234
...     },
...     'bar': 10
... })
1234
>>> parse('parsed.__dict__', locals=locals())
# Elided stack trace
graphtage.expressions.ParseError: Cannot read protected and private member variables: Expression(rpn=(IdentifierToken('foo'), IdentifierToken('bar'), IntegerToken(raw_str='10', value=10), OperatorToken(op=<Operator.ADDITION: ('+', 5, <function Operator.<lambda> at 0x127808170>)>), IntegerToken(raw_str='2', value=2), OperatorToken(op=<Operator.MULTIPLICATION: ('*', 4, <function Operator.<lambda> at 0x127805ef0>)>), OpenBracket())).__dict__ at offset 15

graphtage.expressions.DEFAULT_GLOBALS¶

The default set of globals available to expressions that will be provided if the globals argument to Expression.eval() is not provided. This includes the following functions and types:

abs(), all(), any(), ascii(), bin(), bool, bytearray, bytes, chr(), complex, dict, enumerate(), filter(), float, frozenset, hash(), hex(), id(), int, iter(), len(), list, map(), max(), min(), oct(), ord(), round(), set, slice, sorted(), str, sum(), tuple, zip()

Type: Dict[str, Any]

graphtage.expressions.OPERATORS_BY_NAME¶

A mapping of operator names to Operator objects, used in parsing.

Type: Dict[str, Operator]

graphtage.expressions.IDENTIFIER_BYTES¶

The set of valid bytes that can be used in a IdentifierToken string. This is currently the set of all letters and numbers plus “_” and “-“.

Type: Set[str]

expressions classes¶

CloseBracket¶

class graphtage.expressions.CloseBracket(offset: int)¶

Bases: graphtage.expressions.Token, graphtage.expressions.PairedEndToken

A closing bracket token.

__init__(offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

name = 'brackets'¶

property raw_token¶: Returns the original string parsed from input.

start_token_type¶: alias of OpenBracket

CloseParen¶

class graphtage.expressions.CloseParen(offset: int)¶

Bases: graphtage.expressions.Parenthesis, graphtage.expressions.PairedEndToken

An closing parenthesis token.

__init__(offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

name = 'parenthesis'¶

property raw_token¶: Returns the original string parsed from input.

start_token_type¶: alias of OpenParen

CollectionInfo¶

class graphtage.expressions.CollectionInfo(collection_type: Union[Type[tuple], Type[list]])¶

Bases: object

A datastructure used by the infix_to_rpn() function to keep track of list and tuple semantics.

__init__(collection_type: Union[Type[tuple], Type[list]])¶: Initialize self. See help(type(self)) for accurate signature.

Comma¶

class graphtage.expressions.Comma(offset: int)¶

Bases: graphtage.expressions.Token

A comma token.

__init__(offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

property raw_token¶: Returns the original string parsed from input.

Expression¶

class graphtage.expressions.Expression(rpn: Iterable[graphtage.expressions.Token])¶

Bases: object

An expression is a sequence of tuples in Reverse Polish Notation that can be evaluated.

__init__(rpn: Iterable[graphtage.expressions.Token])¶: Initialize self. See help(type(self)) for accurate signature.

eval(locals: Optional[Dict[str, Any]] = None, globals: Optional[Dict[str, Any]] = None)¶

Evaluates this expression given the provided state.

Parameters

locals – An optional mapping of local variables, by name. If omitted, it will be equivalent to an empty dict.
globals – An optional mapping of global variables, by name. If omitted, it will default to graphtage.expressions.DEFAULT_GLOBALS.

Returns

The result of evaluating the expression.

Return type

Any

static get_value(token: graphtage.expressions.Token, locals: Dict[str, Any], globals: Dict[str, Any])¶

Determines the value of a token given the provided state.

Literal tokens like NumericToken and StringToken will return their values, and identifiers will be resolved using locals and globals.

Parameters

token – The token to resolve.
locals – A mapping of local variables, by name.
globals – A mapping of global variables, by name.

Raises

KeyError – If token is an IdentifierToken and its name was not found in either locals or globals.
ValueError – If the token is not numeric, a string, or an identifier.

Returns

The resolved value of the token.

Return type

Any

FixedSizeCollection¶

class graphtage.expressions.FixedSizeCollection(size: int, container_type: Type[Collection], offset: int)¶

Bases: graphtage.expressions.Token

A meta-token injected by the tokenizer specifying a fixed-size collection of items on the stack.

This is used for parsing and evaluating argument lists of functions of unknown arity.

__init__(size: int, container_type: Type[Collection], offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

container_type: Type[Collection] = None¶: The type of collection in which to store the items.

property raw_token¶: Returns the original string parsed from input.

size: int = None¶: The number of items on the stack to include.

FloatToken¶

class graphtage.expressions.FloatToken(raw_str: str, value: N, offset: int)¶

Bases: graphtage.expressions.NumericToken

A numeric token for floats.

__init__(raw_str: str, value: N, offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

property raw_token¶: Returns the original string parsed from input.

FunctionCall¶

class graphtage.expressions.FunctionCall(offset: int)¶

Bases: graphtage.expressions.OperatorToken

A meta-token for when parenthesis are being used to indicate a function call.

This is automatically inserted by the tokenizer in contexts where parenthesis are being used to represent a function call.

__init__(offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

property raw_token¶: Returns the original string parsed from input.

IdentifierToken¶

class graphtage.expressions.IdentifierToken(name: str, offset: int)¶

Bases: graphtage.expressions.Token

An identifier, such as a variable name or attribute name.

__init__(name: str, offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

name: str = None¶: The name of this identifier

property raw_token¶: Returns the original string parsed from input.

IntegerToken¶

class graphtage.expressions.IntegerToken(raw_str: str, value: N, offset: int)¶

Bases: graphtage.expressions.NumericToken

A numeric token for integers.

__init__(raw_str: str, value: N, offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

property raw_token¶: Returns the original string parsed from input.

NumericToken¶

class graphtage.expressions.NumericToken(raw_str: str, value: N, offset: int)¶

Bases: graphtage.expressions.Token, typing.Generic

An abstract base class for numeric tokens.

__init__(raw_str: str, value: N, offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

property raw_token¶: Returns the original string parsed from input.

value: N = None¶: The numeric value of this token.

OpenBracket¶

class graphtage.expressions.OpenBracket(offset: int, is_list: bool)¶

Bases: graphtage.expressions.OperatorToken, graphtage.expressions.PairedStartToken

An opening bracket token.

__init__(offset: int, is_list: bool)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

discard: bool = False¶

is_list: bool = None¶: If True, this pair of brackets delimits a list. Otherwise it is a Operator.GETITEM access.

name = 'brackets'¶

property raw_token¶: Returns the original string parsed from input.

OpenParen¶

class graphtage.expressions.OpenParen(offset: int, is_function_call: bool)¶

Bases: graphtage.expressions.Parenthesis, graphtage.expressions.PairedStartToken

An opening parenthesis token.

__init__(offset: int, is_function_call: bool)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

discard: bool = True¶

name = 'parenthesis'¶

property raw_token¶: Returns the original string parsed from input.

Operator¶

class graphtage.expressions.Operator(token: str, priority: int, execute: Callable[[Any, Any], Any], is_left_associative: bool = True, arity: int = 2, include_in_global_operator_table: bool = False, expand: Optional[Tuple[bool, …]] = None)¶

Bases: enum.Enum

An enumeration of operators.

ADDITION = ('+', 5, <function Operator.<lambda>>)¶

BITWISE_AND = ('&', 9, <function Operator.<lambda>>)¶

BITWISE_LEFT_SHIFT = ('<<', 6, <function Operator.<lambda>>)¶

BITWISE_NOT = ('~', 3, <function Operator.<lambda>>, False, 1)¶

BITWISE_OR = ('|', 11, <function Operator.<lambda>>)¶

BITWISE_RIGHT_SHIFT = ('>>', 6, <function Operator.<lambda>>)¶

BITWISE_XOR = ('^', 10, <function Operator.<lambda>>)¶

DIVISION = ('/', 4, <function Operator.<lambda>>)¶

EQUALS = ('==', 8, <function Operator.<lambda>>)¶

FUNCTION_CALL = ('→', 2, <function Operator.<lambda>>, True, 2, True)¶

GETITEM = ('[', 1, <function Operator.<lambda>>)¶

GREATER_THAN = ('>', 7, <function Operator.<lambda>>)¶

GREATER_THAN_EQUAL = ('>=', 7, <function Operator.<lambda>>)¶

IN = ('in', 7, <function Operator.<lambda>>)¶

INT_DIVISION = ('//', 4, <function Operator.<lambda>>)¶

LESS_THAN = ('<', 7, <function Operator.<lambda>>)¶

LESS_THAN_EQUAL = ('<=', 7, <function Operator.<lambda>>)¶

LOGICAL_AND = ('and', 12, <function Operator.<lambda>>)¶

LOGICAL_NOT = ('not', 3, <function Operator.<lambda>>, False, 1)¶

LOGICAL_OR = ('or', 13, <function Operator.<lambda>>)¶

MEMBER_ACCESS = ('.', 1, <function Operator.<lambda>>, True, 2, False, (True, False))¶

MULTIPLICATION = ('*', 4, <function Operator.<lambda>>)¶

NOT_EQUAL = ('!=', 8, <function Operator.<lambda>>)¶

REMAINDER = ('%', 4, <function Operator.<lambda>>)¶

SUBTRACTION = ('-', 5, <function Operator.<lambda>>)¶

TERNARY_CONDITIONAL = ('?', 15, <function Operator.<lambda>>, False)¶

TERNARY_ELSE = (':', 14, <function Operator.<lambda>>, False)¶

UNARY_MINUS = ('-', 3, <function Operator.<lambda>>, False, 1, True)¶

UNARY_PLUS = ('+', 3, <function Operator.<lambda>>, False, 1, True)¶

__init__(token: str, priority: int, execute: Callable[[Any, Any], Any], is_left_associative: bool = True, arity: int = 2, include_in_global_operator_table: bool = False, expand: Optional[Tuple[bool, …]] = None)¶

Initializes an operator enum.

Raises: ValueError – If the length of token is greater than three. The token will be used to automatically parse the operator, and the tokenizer currently supports tokens of at most three characters.

arity: int = None¶: The number of arguments consumed by the operator.

execute: Callable[[Any, Any], Any] = None¶: A function to call when the operator is being executed.

expand: Tuple[bool, ...] = None¶: Whether each of the operator’s arguments should be auto-expanded before execution.

left_associative: bool = None¶: Whether the operator is left-associative.

priority: int = None¶: The operator’s precedence priority.

token: str = None¶

The token string associated with this operator. It is used for automatically parsing the operators.

Tokens must be unique. There is no programmatic check to ensure this.

OperatorToken¶

class graphtage.expressions.OperatorToken(op: Union[str, graphtage.expressions.Operator], offset: int)¶

Bases: graphtage.expressions.Token

A token associated with an Operator.

__init__(op: Union[str, graphtage.expressions.Operator], offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

op: Operator = None¶: The operator associated with this token.

property raw_token¶: Returns the original string parsed from input.

PairedEndToken¶

class graphtage.expressions.PairedEndToken¶

Bases: graphtage.expressions.PairedToken

The ending token of a pair.

Examples include “]” and “)”.

__init__()¶: Initialize self. See help(type(self)) for accurate signature.

name = None¶

start_token_type: Type[graphtage.expressions.PairedStartToken] = None¶

PairedStartToken¶

class graphtage.expressions.PairedStartToken¶

Bases: graphtage.expressions.PairedToken

The starting token of a pair.

Examples include “[” and “(“.

__init__()¶: Initialize self. See help(type(self)) for accurate signature.

discard: bool = True¶: Whether this token should be discarded after it is parsed. (e.g., if it is solely used for operator precedence, like parenthesis).

name = None¶

PairedToken¶

class graphtage.expressions.PairedToken¶

Bases: object

Abstract base class for a token that is always paried with another.

Examples include brackets and parenthesis.

__init__()¶: Initialize self. See help(type(self)) for accurate signature.

name: str = None¶: The name of this type of pair. For example, “parenthesis”.

Parenthesis¶

class graphtage.expressions.Parenthesis(*args, **kwargs)¶

Bases: graphtage.expressions.Token

Abstract base class for parenthesis.

__init__(*args, **kwargs)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

property raw_token¶: Returns the original string parsed from input.

ParseError¶

class graphtage.expressions.ParseError(message, offset)¶

Bases: RuntimeError

Base error type of the expressions module.

__init__(message, offset)¶: Initialize self. See help(type(self)) for accurate signature.

args¶

with_traceback()¶: Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

StringToken¶

class graphtage.expressions.StringToken(raw_text: str, offset: int)¶

Bases: graphtage.expressions.Token

A token representing a string constant

__init__(raw_text: str, offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

property raw_token¶: Returns the original string parsed from input.

Token¶

class graphtage.expressions.Token(raw_text: str, offset: int)¶

Bases: object

Base class for an expression token.

__init__(raw_text: str, offset: int)¶

Initializes a token.

Parameters

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset = None¶: Offset of the token in the input.

property raw_token¶: Returns the original string parsed from input.

Tokenizer¶

class graphtage.expressions.Tokenizer(stream: Union[str, IO])¶

Bases: object

The expression tokenizer.

__init__(stream: Union[str, IO])¶

Initializes a tokenizer, but does not commence any tokenization.

Parameters: stream – The input stream from which to tokenize.

__iter__() → Iterator[graphtage.expressions.Token]¶: Iterates over all of the tokens in the stream.

has_next() → bool ¶

Returns whether another token is available.

This is equivalent to:

return self.peek() is not None

next() → Optional[graphtage.expressions.Token]¶

Returns the next token in the stream.

Returns: The next token, or None if there are no more tokens.
Return type: Optional[Token]

peek() → Optional[graphtage.expressions.Token]¶

Returns the next token that would be returned from a call to Tokenizer.next().

This function actually computes and caches the next token if it has not already been cached.

Returns: The next token that would be returned from a call to Tokenizer.next(), or None if there are no more tokens.
Return type: Optional[Token]

prev_token: Optional[Token] = None¶: The previous token yielded by this tokenizer.

expressions functions¶

get_member¶

graphtage.expressions.get_member(obj, member: graphtage.expressions.IdentifierToken)¶

Gets a member of an object by the member’s identifier.

Parameters

obj – Any Python object.
member – An identifier token representing the name of the member.

This is equivalent to:

getattr(obj, member.name)

However, this implementation will not permit retrieving members that start with an underscore:

if member.name.startswith('_'):
    raise ParseError(...)

Raises

ParseError – If member is not an IdentifierToken.
ParseError – If member.name starts with “_”.

Returns

The requested attribute of obj.

Return type

Any

Todo

Provide an API to programmatically override and customize the behavior of this function.

infix_to_rpn¶

graphtage.expressions.infix_to_rpn(tokens: Iterable[graphtage.expressions.Token]) → Iterator[graphtage.expressions.Token]¶: Converts an infix expression to reverse Polish notation using the Shunting Yard algorithm.

parse¶

graphtage.expressions.parse(expression_str: str) → graphtage.expressions.Expression ¶

Convenience function for parsing an expression string.

This is equivalent to:

Expression(infix_to_rpn(tokenize(expression_str)))

tokenize¶

graphtage.expressions.tokenize(stream_or_str: Union[IO, str]) → Iterator[graphtage.expressions.Token]¶

Convenience function for tokenizing a string.

This is equivalent to:

yield from Tokenizer(stream_or_str)