graphtage.expressions

A safer-than-Python-eval expression evaluation module.

It is extensible, supporting infix operators with adjustable precedence. It supports function calls, member lookup (getitem()), and provides a modicum of safety by only allowing access to a programmer-defined set of variables. By default, it also disallows access to protected and private member variables.

Example

Here is an example of its usage:

>>> parsed = parse('foo[(bar + 10) * 2]')
>>> parsed
Expression(rpn=(IdentifierToken('foo'), IdentifierToken('bar'), IntegerToken(raw_str='10', value=10), OperatorToken(op=<Operator.ADDITION: ('+', 5, <function Operator.<lambda> at 0x135057f80>)>), IntegerToken(raw_str='2', value=2), OperatorToken(op=<Operator.MULTIPLICATION: ('*', 4, <function Operator.<lambda> at 0x135057d40>)>), OpenBracket()))
>>> parsed.eval(locals={
...     'foo': {
...         40: 1234
...     },
...     'bar': 10
... })
1234
>>> parse('parsed.__dict__', locals=locals())
# Elided stack trace
graphtage.expressions.ParseError: Cannot read protected and private member variables: Expression(rpn=(IdentifierToken('foo'), IdentifierToken('bar'), IntegerToken(raw_str='10', value=10), OperatorToken(op=<Operator.ADDITION: ('+', 5, <function Operator.<lambda> at 0x127808170>)>), IntegerToken(raw_str='2', value=2), OperatorToken(op=<Operator.MULTIPLICATION: ('*', 4, <function Operator.<lambda> at 0x127805ef0>)>), OpenBracket())).__dict__ at offset 15

graphtage.expressions.DEFAULT_GLOBALS

The default set of globals available to expressions that will be provided if the globals argument to Expression.eval() is not provided. This includes the following functions and types:

abs(), all(), any(), ascii(), bin(), bool, bytearray, bytes, chr(), complex, dict, enumerate(), filter(), float, frozenset, hash(), hex(), id(), int, iter(), len(), list, map(), max(), min(), oct(), ord(), round(), set, slice, sorted(), str, sum(), tuple, zip()

Type:: Dict[str, Any]

graphtage.expressions.OPERATORS_BY_NAME

A mapping of operator names to Operator objects, used in parsing.

Type:: Dict[str, Operator]

graphtage.expressions.IDENTIFIER_BYTES

The set of valid bytes that can be used in a IdentifierToken string. This is currently the set of all letters and numbers plus “_” and “-“.

Type:: Set[str]

expressions classes

CloseBracket

class graphtage.expressions.CloseBracket(offset: int)

Bases: Token, PairedEndToken

A closing bracket token.

__init__(offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

name: str = 'brackets': The name of this type of pair. For example, “parenthesis”.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

start_token_type: alias of OpenBracket

CloseParen

class graphtage.expressions.CloseParen(offset: int)

Bases: Parenthesis, PairedEndToken

An closing parenthesis token.

__init__(offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

name: str = 'parenthesis': The name of this type of pair. For example, “parenthesis”.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

start_token_type: alias of OpenParen

CollectionInfo

class graphtage.expressions.CollectionInfo(collection_type: Type[tuple] | Type[list])

Bases: object

A datastructure used by the infix_to_rpn() function to keep track of list and tuple semantics.

__init__(collection_type: Type[tuple] | Type[list])

Comma

class graphtage.expressions.Comma(offset: int)

Bases: Token

A comma token.

__init__(offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

Expression

class graphtage.expressions.Expression(rpn: Iterable[Token])

Bases: object

An expression is a sequence of tuples in Reverse Polish Notation that can be evaluated.

__init__(rpn: Iterable[Token])

eval(locals: Dict[str, Any] | None = None, globals: Dict[str, Any] | None = None)

Evaluates this expression given the provided state.

Parameters:

locals – An optional mapping of local variables, by name. If omitted, it will be equivalent to an empty dict.
globals – An optional mapping of global variables, by name. If omitted, it will default to graphtage.expressions.DEFAULT_GLOBALS.

Returns:

The result of evaluating the expression.

Return type:

Any

static get_value(token: Token, locals: Dict[str, Any], globals: Dict[str, Any])

Determines the value of a token given the provided state.

Literal tokens like NumericToken and StringToken will return their values, and identifiers will be resolved using locals and globals.

Parameters:

token – The token to resolve.
locals – A mapping of local variables, by name.
globals – A mapping of global variables, by name.

Raises:

KeyError – If token is an IdentifierToken and its name was not found in either locals or globals.
ValueError – If the token is not numeric, a string, or an identifier.

Returns:

The resolved value of the token.

Return type:

Any

FixedSizeCollection

class graphtage.expressions.FixedSizeCollection(size: int, container_type: Type[Collection], offset: int)

Bases: Token

A meta-token injected by the tokenizer specifying a fixed-size collection of items on the stack.

This is used for parsing and evaluating argument lists of functions of unknown arity.

__init__(size: int, container_type: Type[Collection], offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

container_type: Type[Collection]: The type of collection in which to store the items.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

size: int: The number of items on the stack to include.

FloatToken

class graphtage.expressions.FloatToken(raw_str: str, value: N, offset: int)

Bases: NumericToken[float]

A numeric token for floats.

__init__(raw_str: str, value: N, offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

value: N: The numeric value of this token.

FunctionCall

class graphtage.expressions.FunctionCall(offset: int)

Bases: OperatorToken

A meta-token for when parenthesis are being used to indicate a function call.

This is automatically inserted by the tokenizer in contexts where parenthesis are being used to represent a function call.

__init__(offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

op: Operator: The operator associated with this token.

property raw_token: str: Returns the original string parsed from input.

IdentifierToken

class graphtage.expressions.IdentifierToken(name: str, offset: int)

Bases: Token

An identifier, such as a variable name or attribute name.

__init__(name: str, offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

name: str: The name of this identifier

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

IntegerToken

class graphtage.expressions.IntegerToken(raw_str: str, value: N, offset: int)

Bases: NumericToken[int]

A numeric token for integers.

__init__(raw_str: str, value: N, offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

value: N: The numeric value of this token.

NumericToken

class graphtage.expressions.NumericToken(raw_str: str, value: N, offset: int)

Bases: Token, Generic[N]

An abstract base class for numeric tokens.

__init__(raw_str: str, value: N, offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

value: N: The numeric value of this token.

OpenBracket

class graphtage.expressions.OpenBracket(offset: int, is_list: bool)

Bases: OperatorToken, PairedStartToken

An opening bracket token.

__init__(offset: int, is_list: bool)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

discard: bool = False: Whether this token should be discarded after it is parsed. (e.g., if it is solely used for operator precedence, like parenthesis).

is_list: bool: If True, this pair of brackets delimits a list. Otherwise it is a Operator.GETITEM access.

name: str = 'brackets': The name of this type of pair. For example, “parenthesis”.

offset: Offset of the token in the input.

op: Operator: The operator associated with this token.

property raw_token: str: Returns the original string parsed from input.

OpenParen

class graphtage.expressions.OpenParen(offset: int, is_function_call: bool)

Bases: Parenthesis, PairedStartToken

An opening parenthesis token.

__init__(offset: int, is_function_call: bool)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

discard: bool = True: Whether this token should be discarded after it is parsed. (e.g., if it is solely used for operator precedence, like parenthesis).

name: str = 'parenthesis': The name of this type of pair. For example, “parenthesis”.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

Operator

class graphtage.expressions.Operator(value)

Bases: Enum

An enumeration of operators.

ADDITION = ('+', 5, <function Operator.<lambda>>)

BITWISE_AND = ('&', 9, <function Operator.<lambda>>)

BITWISE_LEFT_SHIFT = ('<<', 6, <function Operator.<lambda>>)

BITWISE_NOT = ('~', 3, <function Operator.<lambda>>, False, 1)

BITWISE_OR = ('|', 11, <function Operator.<lambda>>)

BITWISE_RIGHT_SHIFT = ('>>', 6, <function Operator.<lambda>>)

BITWISE_XOR = ('^', 10, <function Operator.<lambda>>)

DIVISION = ('/', 4, <function Operator.<lambda>>)

EQUALS = ('==', 8, <function Operator.<lambda>>)

FUNCTION_CALL = ('→', 2, <function Operator.<lambda>>, True, 2, True)

GETITEM = ('[', 1, <function Operator.<lambda>>)

GREATER_THAN = ('>', 7, <function Operator.<lambda>>)

GREATER_THAN_EQUAL = ('>=', 7, <function Operator.<lambda>>)

IN = ('in', 7, <function Operator.<lambda>>)

INT_DIVISION = ('//', 4, <function Operator.<lambda>>)

LESS_THAN = ('<', 7, <function Operator.<lambda>>)

LESS_THAN_EQUAL = ('<=', 7, <function Operator.<lambda>>)

LOGICAL_AND = ('and', 12, <function Operator.<lambda>>)

LOGICAL_NOT = ('not', 3, <function Operator.<lambda>>, False, 1)

LOGICAL_OR = ('or', 13, <function Operator.<lambda>>)

MEMBER_ACCESS = ('.', 1, <function Operator.<lambda>>, True, 2, False, (True, False))

MULTIPLICATION = ('*', 4, <function Operator.<lambda>>)

NOT_EQUAL = ('!=', 8, <function Operator.<lambda>>)

REMAINDER = ('%', 4, <function Operator.<lambda>>)

SUBTRACTION = ('-', 5, <function Operator.<lambda>>)

TERNARY_CONDITIONAL = ('?', 15, <function Operator.<lambda>>, False)

TERNARY_ELSE = (':', 14, <function Operator.<lambda>>, False)

UNARY_MINUS = ('-', 3, <function Operator.<lambda>>, False, 1, True)

UNARY_PLUS = ('+', 3, <function Operator.<lambda>>, False, 1, True)

__init__(token: str, priority: int, execute: Callable[[Any, Any], Any], is_left_associative: bool = True, arity: int = 2, include_in_global_operator_table: bool = False, expand: Tuple[bool, ...] | None = None)

Initializes an operator enum.

Raises:: ValueError – If the length of token is greater than three. The token will be used to automatically parse the operator, and the tokenizer currently supports tokens of at most three characters.

arity: int: The number of arguments consumed by the operator.

execute: Callable[[Any, Any], Any]: A function to call when the operator is being executed.

expand: Tuple[bool, ...]: Whether each of the operator’s arguments should be auto-expanded before execution.

left_associative: bool: Whether the operator is left-associative.

priority: int: The operator’s precedence priority.

token: str

The token string associated with this operator. It is used for automatically parsing the operators.

Tokens must be unique. There is no programmatic check to ensure this.

OperatorToken

class graphtage.expressions.OperatorToken(op: str | Operator, offset: int)

Bases: Token

A token associated with an Operator.

__init__(op: str | Operator, offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

op: Operator: The operator associated with this token.

property raw_token: str: Returns the original string parsed from input.

PairedEndToken

class graphtage.expressions.PairedEndToken

Bases: PairedToken

The ending token of a pair.

Examples include “]” and “)”.

__init__()

name: str = None: The name of this type of pair. For example, “parenthesis”.

start_token_type: Type[PairedStartToken] = None

PairedStartToken

class graphtage.expressions.PairedStartToken

Bases: PairedToken

The starting token of a pair.

Examples include “[” and “(“.

__init__()

discard: bool = True: Whether this token should be discarded after it is parsed. (e.g., if it is solely used for operator precedence, like parenthesis).

name: str = None: The name of this type of pair. For example, “parenthesis”.

PairedToken

class graphtage.expressions.PairedToken

Bases: object

Abstract base class for a token that is always paried with another.

Examples include brackets and parenthesis.

__init__()

name: str = None: The name of this type of pair. For example, “parenthesis”.

Parenthesis

class graphtage.expressions.Parenthesis(*args, **kwargs)

Bases: Token

Abstract base class for parenthesis.

__init__(*args, **kwargs)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

ParseError

class graphtage.expressions.ParseError(message, offset)

Bases: RuntimeError

Base error type of the expressions module.

__init__(message, offset)

args

with_traceback(): Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

StringToken

class graphtage.expressions.StringToken(raw_text: str, offset: int)

Bases: Token

A token representing a string constant

__init__(raw_text: str, offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

Token

class graphtage.expressions.Token(raw_text: str, offset: int)

Bases: object

Base class for an expression token.

__init__(raw_text: str, offset: int)

Initializes a token.

Parameters:

raw_text – The raw parsed text of the token.
offset – The offset of the token within the input.

offset: Offset of the token in the input.

property raw_token: str: Returns the original string parsed from input.

Tokenizer

class graphtage.expressions.Tokenizer(stream: str | IO)

Bases: object

The expression tokenizer.

__init__(stream: str | IO)

Initializes a tokenizer, but does not commence any tokenization.

Parameters:: stream – The input stream from which to tokenize.

__iter__() → Iterator[Token]: Iterates over all of the tokens in the stream.

has_next() → bool

Returns whether another token is available.

This is equivalent to:

return self.peek() is not None

next() → Token | None

Returns the next token in the stream.

Returns:: The next token, or None if there are no more tokens.
Return type:: Optional[Token]

peek() → Token | None

Returns the next token that would be returned from a call to Tokenizer.next().

This function actually computes and caches the next token if it has not already been cached.

Returns:: The next token that would be returned from a call to Tokenizer.next(), or None if there are no more tokens.
Return type:: Optional[Token]

prev_token: Token | None: The previous token yielded by this tokenizer.

expressions functions

get_member

graphtage.expressions.get_member(obj, member: IdentifierToken)

Gets a member of an object by the member’s identifier.

Parameters:

obj – Any Python object.
member – An identifier token representing the name of the member.

This is equivalent to:

getattr(obj, member.name)

However, this implementation will not permit retrieving members that start with an underscore:

if member.name.startswith('_'):
    raise ParseError(...)

Raises:

ParseError – If member is not an IdentifierToken.
ParseError – If member.name starts with “_”.

Returns:

The requested attribute of obj.

Return type:

Any

Todo

Provide an API to programmatically override and customize the behavior of this function.

infix_to_rpn

graphtage.expressions.infix_to_rpn(tokens: Iterable[Token]) → Iterator[Token]: Converts an infix expression to reverse Polish notation using the Shunting Yard algorithm.

parse

graphtage.expressions.parse(expression_str: str) → Expression

Convenience function for parsing an expression string.

This is equivalent to:

Expression(infix_to_rpn(tokenize(expression_str)))

tokenize

graphtage.expressions.tokenize(stream_or_str: IO | str) → Iterator[Token]

Convenience function for tokenizing a string.

This is equivalent to:

yield from Tokenizer(stream_or_str)