graphtage.expressions

A safer-than-Python-eval expression evaluation module.

It is extensible, supporting infix operators with adjustable precedence. It supports function calls, member lookup (getitem()), and provides a modicum of safety by only allowing access to a programmer-defined set of variables. By default, it also disallows access to protected and private member variables.

Example

Here is an example of its usage:

>>> parsed = parse('foo[(bar + 10) * 2]')
>>> parsed
Expression(rpn=(IdentifierToken('foo'), IdentifierToken('bar'), IntegerToken(raw_str='10', value=10), OperatorToken(op=<Operator.ADDITION: ('+', 5, <function Operator.<lambda> at 0x135057f80>)>), IntegerToken(raw_str='2', value=2), OperatorToken(op=<Operator.MULTIPLICATION: ('*', 4, <function Operator.<lambda> at 0x135057d40>)>), OpenBracket()))
>>> parsed.eval(locals={
...     'foo': {
...         40: 1234
...     },
...     'bar': 10
... })
1234
>>> parse('parsed.__dict__', locals=locals())
# Elided stack trace
graphtage.expressions.ParseError: Cannot read protected and private member variables: Expression(rpn=(IdentifierToken('foo'), IdentifierToken('bar'), IntegerToken(raw_str='10', value=10), OperatorToken(op=<Operator.ADDITION: ('+', 5, <function Operator.<lambda> at 0x127808170>)>), IntegerToken(raw_str='2', value=2), OperatorToken(op=<Operator.MULTIPLICATION: ('*', 4, <function Operator.<lambda> at 0x127805ef0>)>), OpenBracket())).__dict__ at offset 15
graphtage.expressions.DEFAULT_GLOBALS

The default set of globals available to expressions that will be provided if the globals argument to Expression.eval() is not provided. This includes the following functions and types:

Type

Dict[str, Any]

graphtage.expressions.OPERATORS_BY_NAME

A mapping of operator names to Operator objects, used in parsing.

Type

Dict[str, Operator]

graphtage.expressions.IDENTIFIER_BYTES

The set of valid bytes that can be used in a IdentifierToken string. This is currently the set of all letters and numbers plus “_” and “-“.

Type

Set[str]

expressions classes

CloseBracket

class graphtage.expressions.CloseBracket(offset: int)

Bases: graphtage.expressions.Token, graphtage.expressions.PairedEndToken

A closing bracket token.

__init__(offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

name: str = 'brackets'
property raw_token

Returns the original string parsed from input.

start_token_type

alias of OpenBracket

CloseParen

class graphtage.expressions.CloseParen(offset: int)

Bases: graphtage.expressions.Parenthesis, graphtage.expressions.PairedEndToken

An closing parenthesis token.

__init__(offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

name: str = 'parenthesis'
property raw_token

Returns the original string parsed from input.

start_token_type

alias of OpenParen

CollectionInfo

class graphtage.expressions.CollectionInfo(collection_type: Union[Type[tuple], Type[list]])

Bases: object

A datastructure used by the infix_to_rpn() function to keep track of list and tuple semantics.

__init__(collection_type: Union[Type[tuple], Type[list]])

Initialize self. See help(type(self)) for accurate signature.

Comma

class graphtage.expressions.Comma(offset: int)

Bases: graphtage.expressions.Token

A comma token.

__init__(offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

property raw_token

Returns the original string parsed from input.

Expression

class graphtage.expressions.Expression(rpn: Iterable[graphtage.expressions.Token])

Bases: object

An expression is a sequence of tuples in Reverse Polish Notation that can be evaluated.

__init__(rpn: Iterable[graphtage.expressions.Token])

Initialize self. See help(type(self)) for accurate signature.

eval(locals: Optional[Dict[str, Any]] = None, globals: Optional[Dict[str, Any]] = None)

Evaluates this expression given the provided state.

Parameters
  • locals – An optional mapping of local variables, by name. If omitted, it will be equivalent to an empty dict.

  • globals – An optional mapping of global variables, by name. If omitted, it will default to graphtage.expressions.DEFAULT_GLOBALS.

Returns

The result of evaluating the expression.

Return type

Any

static get_value(token: graphtage.expressions.Token, locals: Dict[str, Any], globals: Dict[str, Any])

Determines the value of a token given the provided state.

Literal tokens like NumericToken and StringToken will return their values, and identifiers will be resolved using locals and globals.

Parameters
  • token – The token to resolve.

  • locals – A mapping of local variables, by name.

  • globals – A mapping of global variables, by name.

Raises
Returns

The resolved value of the token.

Return type

Any

FixedSizeCollection

class graphtage.expressions.FixedSizeCollection(size: int, container_type: Type[Collection], offset: int)

Bases: graphtage.expressions.Token

A meta-token injected by the tokenizer specifying a fixed-size collection of items on the stack.

This is used for parsing and evaluating argument lists of functions of unknown arity.

__init__(size: int, container_type: Type[Collection], offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

container_type: Type[Collection]

The type of collection in which to store the items.

property raw_token

Returns the original string parsed from input.

size: int

The number of items on the stack to include.

FloatToken

class graphtage.expressions.FloatToken(raw_str: str, value: N, offset: int)

Bases: graphtage.expressions.NumericToken

A numeric token for floats.

__init__(raw_str: str, value: N, offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

property raw_token

Returns the original string parsed from input.

FunctionCall

class graphtage.expressions.FunctionCall(offset: int)

Bases: graphtage.expressions.OperatorToken

A meta-token for when parenthesis are being used to indicate a function call.

This is automatically inserted by the tokenizer in contexts where parenthesis are being used to represent a function call.

__init__(offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

property raw_token

Returns the original string parsed from input.

IdentifierToken

class graphtage.expressions.IdentifierToken(name: str, offset: int)

Bases: graphtage.expressions.Token

An identifier, such as a variable name or attribute name.

__init__(name: str, offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

name: str

The name of this identifier

property raw_token

Returns the original string parsed from input.

IntegerToken

class graphtage.expressions.IntegerToken(raw_str: str, value: N, offset: int)

Bases: graphtage.expressions.NumericToken

A numeric token for integers.

__init__(raw_str: str, value: N, offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

property raw_token

Returns the original string parsed from input.

NumericToken

class graphtage.expressions.NumericToken(raw_str: str, value: N, offset: int)

Bases: graphtage.expressions.Token, typing.Generic

An abstract base class for numeric tokens.

__init__(raw_str: str, value: N, offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

property raw_token

Returns the original string parsed from input.

value: N

The numeric value of this token.

OpenBracket

class graphtage.expressions.OpenBracket(offset: int, is_list: bool)

Bases: graphtage.expressions.OperatorToken, graphtage.expressions.PairedStartToken

An opening bracket token.

__init__(offset: int, is_list: bool)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

discard: bool = False
is_list: bool

If True, this pair of brackets delimits a list. Otherwise it is a Operator.GETITEM access.

name: str = 'brackets'
property raw_token

Returns the original string parsed from input.

OpenParen

class graphtage.expressions.OpenParen(offset: int, is_function_call: bool)

Bases: graphtage.expressions.Parenthesis, graphtage.expressions.PairedStartToken

An opening parenthesis token.

__init__(offset: int, is_function_call: bool)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

discard: bool = True
name: str = 'parenthesis'
property raw_token

Returns the original string parsed from input.

Operator

class graphtage.expressions.Operator(value)

Bases: enum.Enum

An enumeration of operators.

ADDITION = ('+', 5, <function Operator.<lambda>>)
BITWISE_AND = ('&', 9, <function Operator.<lambda>>)
BITWISE_LEFT_SHIFT = ('<<', 6, <function Operator.<lambda>>)
BITWISE_NOT = ('~', 3, <function Operator.<lambda>>, False, 1)
BITWISE_OR = ('|', 11, <function Operator.<lambda>>)
BITWISE_RIGHT_SHIFT = ('>>', 6, <function Operator.<lambda>>)
BITWISE_XOR = ('^', 10, <function Operator.<lambda>>)
DIVISION = ('/', 4, <function Operator.<lambda>>)
EQUALS = ('==', 8, <function Operator.<lambda>>)
FUNCTION_CALL = ('→', 2, <function Operator.<lambda>>, True, 2, True)
GETITEM = ('[', 1, <function Operator.<lambda>>)
GREATER_THAN = ('>', 7, <function Operator.<lambda>>)
GREATER_THAN_EQUAL = ('>=', 7, <function Operator.<lambda>>)
IN = ('in', 7, <function Operator.<lambda>>)
INT_DIVISION = ('//', 4, <function Operator.<lambda>>)
LESS_THAN = ('<', 7, <function Operator.<lambda>>)
LESS_THAN_EQUAL = ('<=', 7, <function Operator.<lambda>>)
LOGICAL_AND = ('and', 12, <function Operator.<lambda>>)
LOGICAL_NOT = ('not', 3, <function Operator.<lambda>>, False, 1)
LOGICAL_OR = ('or', 13, <function Operator.<lambda>>)
MEMBER_ACCESS = ('.', 1, <function Operator.<lambda>>, True, 2, False, (True, False))
MULTIPLICATION = ('*', 4, <function Operator.<lambda>>)
NOT_EQUAL = ('!=', 8, <function Operator.<lambda>>)
REMAINDER = ('%', 4, <function Operator.<lambda>>)
SUBTRACTION = ('-', 5, <function Operator.<lambda>>)
TERNARY_CONDITIONAL = ('?', 15, <function Operator.<lambda>>, False)
TERNARY_ELSE = (':', 14, <function Operator.<lambda>>, False)
UNARY_MINUS = ('-', 3, <function Operator.<lambda>>, False, 1, True)
UNARY_PLUS = ('+', 3, <function Operator.<lambda>>, False, 1, True)
__init__(token: str, priority: int, execute: Callable[[Any, Any], Any], is_left_associative: bool = True, arity: int = 2, include_in_global_operator_table: bool = False, expand: Optional[Tuple[bool, ]] = None)

Initializes an operator enum.

Raises

ValueError – If the length of token is greater than three. The token will be used to automatically parse the operator, and the tokenizer currently supports tokens of at most three characters.

arity: int

The number of arguments consumed by the operator.

execute: Callable[[Any, Any], Any]

A function to call when the operator is being executed.

expand: Tuple[bool, ]

Whether each of the operator’s arguments should be auto-expanded before execution.

left_associative: bool

Whether the operator is left-associative.

priority: int

The operator’s precedence priority.

token: str

The token string associated with this operator. It is used for automatically parsing the operators.

Tokens must be unique. There is no programmatic check to ensure this.

OperatorToken

class graphtage.expressions.OperatorToken(op: Union[str, graphtage.expressions.Operator], offset: int)

Bases: graphtage.expressions.Token

A token associated with an Operator.

__init__(op: Union[str, graphtage.expressions.Operator], offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

op: Operator

The operator associated with this token.

property raw_token

Returns the original string parsed from input.

PairedEndToken

class graphtage.expressions.PairedEndToken

Bases: graphtage.expressions.PairedToken

The ending token of a pair.

Examples include “]” and “)”.

__init__()

Initialize self. See help(type(self)) for accurate signature.

name: str = None
start_token_type: Type[graphtage.expressions.PairedStartToken] = None

PairedStartToken

class graphtage.expressions.PairedStartToken

Bases: graphtage.expressions.PairedToken

The starting token of a pair.

Examples include “[” and “(“.

__init__()

Initialize self. See help(type(self)) for accurate signature.

discard: bool = True

Whether this token should be discarded after it is parsed. (e.g., if it is solely used for operator precedence, like parenthesis).

name: str = None

PairedToken

class graphtage.expressions.PairedToken

Bases: object

Abstract base class for a token that is always paried with another.

Examples include brackets and parenthesis.

__init__()

Initialize self. See help(type(self)) for accurate signature.

name: str = None

The name of this type of pair. For example, “parenthesis”.

Parenthesis

class graphtage.expressions.Parenthesis(*args, **kwargs)

Bases: graphtage.expressions.Token

Abstract base class for parenthesis.

__init__(*args, **kwargs)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

property raw_token

Returns the original string parsed from input.

ParseError

class graphtage.expressions.ParseError(message, offset)

Bases: RuntimeError

Base error type of the expressions module.

__init__(message, offset)

Initialize self. See help(type(self)) for accurate signature.

args
with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

StringToken

class graphtage.expressions.StringToken(raw_text: str, offset: int)

Bases: graphtage.expressions.Token

A token representing a string constant

__init__(raw_text: str, offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

property raw_token

Returns the original string parsed from input.

Token

class graphtage.expressions.Token(raw_text: str, offset: int)

Bases: object

Base class for an expression token.

__init__(raw_text: str, offset: int)

Initializes a token.

Parameters
  • raw_text – The raw parsed text of the token.

  • offset – The offset of the token within the input.

offset

Offset of the token in the input.

property raw_token

Returns the original string parsed from input.

Tokenizer

class graphtage.expressions.Tokenizer(stream: Union[str, IO])

Bases: object

The expression tokenizer.

__init__(stream: Union[str, IO])

Initializes a tokenizer, but does not commence any tokenization.

Parameters

stream – The input stream from which to tokenize.

__iter__() → Iterator[graphtage.expressions.Token]

Iterates over all of the tokens in the stream.

has_next()bool

Returns whether another token is available.

This is equivalent to:

return self.peek() is not None
next() → Optional[graphtage.expressions.Token]

Returns the next token in the stream.

Returns

The next token, or None if there are no more tokens.

Return type

Optional[Token]

peek() → Optional[graphtage.expressions.Token]

Returns the next token that would be returned from a call to Tokenizer.next().

This function actually computes and caches the next token if it has not already been cached.

Returns

The next token that would be returned from a call to Tokenizer.next(), or None if there are no more tokens.

Return type

Optional[Token]

prev_token: Optional[Token]

The previous token yielded by this tokenizer.

expressions functions

get_member

graphtage.expressions.get_member(obj, member: graphtage.expressions.IdentifierToken)

Gets a member of an object by the member’s identifier.

Parameters
  • obj – Any Python object.

  • member – An identifier token representing the name of the member.

This is equivalent to:

getattr(obj, member.name)

However, this implementation will not permit retrieving members that start with an underscore:

if member.name.startswith('_'):
    raise ParseError(...)
Raises
Returns

The requested attribute of obj.

Return type

Any

Todo

  • Provide an API to programmatically override and customize the behavior of this function.

infix_to_rpn

graphtage.expressions.infix_to_rpn(tokens: Iterable[graphtage.expressions.Token]) → Iterator[graphtage.expressions.Token]

Converts an infix expression to reverse Polish notation using the Shunting Yard algorithm.

parse

graphtage.expressions.parse(expression_str: str)graphtage.expressions.Expression

Convenience function for parsing an expression string.

This is equivalent to:

Expression(infix_to_rpn(tokenize(expression_str)))

tokenize

graphtage.expressions.tokenize(stream_or_str: Union[IO, str]) → Iterator[graphtage.expressions.Token]

Convenience function for tokenizing a string.

This is equivalent to:

yield from Tokenizer(stream_or_str)