graphtage.utils¶

Generic utility functions and classes.

utils classes¶

HashableCounter¶

class graphtage.utils.HashableCounter(*args, **kwargs)¶

Bases: typing.Generic, collections.Counter

A Counter that supports being hashed even though it is mutable.

__add__(other)¶

Add counts from two counters.

>>> Counter('abbb') + Counter('bcc')
Counter({'b': 4, 'c': 2, 'a': 1})

__and__(other)¶

Intersection is the minimum of corresponding counts.

>>> Counter('abbb') & Counter('bcc')
Counter({'b': 1})

__delitem__(elem)¶: Like dict.__delitem__() but does not raise KeyError for missing values.

__iadd__(other)¶

Inplace add from another counter, keeping only positive counts.

>>> c = Counter('abbb')
>>> c += Counter('bcc')
>>> c
Counter({'b': 4, 'c': 2, 'a': 1})

__iand__(other)¶

Inplace intersection is the minimum of corresponding counts.

>>> c = Counter('abbb')
>>> c &= Counter('bcc')
>>> c
Counter({'b': 1})

__init__(*args, **kwargs)¶: Initialize self. See help(type(self)) for accurate signature.

__ior__(other)¶

Inplace union is the maximum of value from either counter.

>>> c = Counter('abbb')
>>> c |= Counter('bcc')
>>> c
Counter({'b': 3, 'c': 2, 'a': 1})

__isub__(other)¶

Inplace subtract counter, but keep only results with positive counts.

>>> c = Counter('abbbc')
>>> c -= Counter('bccd')
>>> c
Counter({'b': 2, 'a': 1})

__missing__(key)¶: The count of elements not in the Counter is zero.

__neg__()¶: Subtracts from an empty counter. Strips positive and zero counts, and flips the sign on negative counts.

__or__(other)¶

Union is the maximum of value in either of the input counters.

>>> Counter('abbb') | Counter('bcc')
Counter({'b': 3, 'c': 2, 'a': 1})

__pos__()¶: Adds an empty counter, effectively stripping negative and zero counts

__sub__(other)¶

Subtract count, but keep only results with positive counts.

>>> Counter('abbbc') - Counter('bccd')
Counter({'b': 2, 'a': 1})

_keep_positive()¶: Internal method to strip elements with a negative or zero count

clear() → None. Remove all items from D.¶

copy()¶: Return a shallow copy.

elements() → Iterator¶

Iterator over elements repeating each as many times as its count.

Examples:

>>> c = Counter('ABCABC')
>>> sorted(c.elements())
['A', 'A', 'B', 'B', 'C', 'C']

# Knuth's example for prime factors of 1836:  2**2 * 3**3 * 17**1
>>> prime_factors = Counter({2: 2, 3: 3, 17: 1})
>>> product = 1
>>> for factor in prime_factors.elements():     # loop over factors
...     product *= factor                       # and multiply them
>>> product
1836

Note

If an element’s count has been set to zero or is a negative number, elements() will ignore it.

classmethod fromkeys(iterable, v=None)¶: Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)¶: Return the value for key if key is in the dictionary, else default.

items() → a set-like object providing a view on D’s items¶

keys() → a set-like object providing a view on D’s keys¶

most_common(n=None)¶

List the n most common elements and their counts from the most common to the least. If n is None, then list all element counts.

>>> Counter('abracadabra').most_common(3)
[('a', 5), ('b', 2), ('r', 2)]

pop(k[, d]) → v, remove specified key and return the corresponding value.¶: If key is not found, d is returned if given, otherwise KeyError is raised

popitem()¶

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)¶

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

subtract(iterable=None, /, **kwds)¶

Like dict.update() but subtracts counts instead of replacing them. Counts can be reduced below zero. Both the inputs and outputs are allowed to contain zero and negative counts.

Source can be an iterable, a dictionary, or another Counter instance.

>>> c = Counter('which')
>>> c.subtract('witch')             # subtract elements from another iterable
>>> c.subtract(Counter('watch'))    # subtract elements from another counter
>>> c['h']                          # 2 in which, minus 1 in witch, minus 1 in watch
0
>>> c['w']                          # 1 in which, minus 1 in witch, minus 1 in watch
-1

update(iterable=None, /, **kwds)¶

Like dict.update() but add counts instead of replacing them.

Source can be an iterable, a dictionary, or another Counter instance.

>>> c = Counter('which')
>>> c.update('witch')           # add elements from another iterable
>>> d = Counter('watch')
>>> c.update(d)                 # add elements from another counter
>>> c['h']                      # four 'h' in which, witch, and watch
4

values() → an object providing a view on D’s values¶

OrderedCounter¶

class graphtage.utils.OrderedCounter(iterable=None, /, **kwds)¶

Bases: collections.Counter, collections.OrderedDict

A counter that remembers the order elements are first encountered

__add__(other)¶

Add counts from two counters.

>>> Counter('abbb') + Counter('bcc')
Counter({'b': 4, 'c': 2, 'a': 1})

__and__(other)¶

Intersection is the minimum of corresponding counts.

>>> Counter('abbb') & Counter('bcc')
Counter({'b': 1})

__delitem__(elem)¶: Like dict.__delitem__() but does not raise KeyError for missing values.

__iadd__(other)¶

Inplace add from another counter, keeping only positive counts.

>>> c = Counter('abbb')
>>> c += Counter('bcc')
>>> c
Counter({'b': 4, 'c': 2, 'a': 1})

__iand__(other)¶

Inplace intersection is the minimum of corresponding counts.

>>> c = Counter('abbb')
>>> c &= Counter('bcc')
>>> c
Counter({'b': 1})

__init__(iterable=None, /, **kwds)¶

Create a new, empty Counter object. And if given, count elements from an input iterable. Or, initialize the count from another mapping of elements to their counts.

>>> c = Counter()                           # a new, empty counter
>>> c = Counter('gallahad')                 # a new counter from an iterable
>>> c = Counter({'a': 4, 'b': 2})           # a new counter from a mapping
>>> c = Counter(a=4, b=2)                   # a new counter from keyword args

__ior__(other)¶

Inplace union is the maximum of value from either counter.

>>> c = Counter('abbb')
>>> c |= Counter('bcc')
>>> c
Counter({'b': 3, 'c': 2, 'a': 1})

__isub__(other)¶

Inplace subtract counter, but keep only results with positive counts.

>>> c = Counter('abbbc')
>>> c -= Counter('bccd')
>>> c
Counter({'b': 2, 'a': 1})

__missing__(key)¶: The count of elements not in the Counter is zero.

__neg__()¶: Subtracts from an empty counter. Strips positive and zero counts, and flips the sign on negative counts.

__or__(other)¶

Union is the maximum of value in either of the input counters.

>>> Counter('abbb') | Counter('bcc')
Counter({'b': 3, 'c': 2, 'a': 1})

__pos__()¶: Adds an empty counter, effectively stripping negative and zero counts

__sub__(other)¶

Subtract count, but keep only results with positive counts.

>>> Counter('abbbc') - Counter('bccd')
Counter({'b': 2, 'a': 1})

_keep_positive()¶: Internal method to strip elements with a negative or zero count

clear() → None. Remove all items from od.¶

copy()¶: Return a shallow copy.

elements() → Iterator¶

Iterator over elements repeating each as many times as its count.

Examples:

>>> c = Counter('ABCABC')
>>> sorted(c.elements())
['A', 'A', 'B', 'B', 'C', 'C']

# Knuth's example for prime factors of 1836:  2**2 * 3**3 * 17**1
>>> prime_factors = Counter({2: 2, 3: 3, 17: 1})
>>> product = 1
>>> for factor in prime_factors.elements():     # loop over factors
...     product *= factor                       # and multiply them
>>> product
1836

Note

If an element’s count has been set to zero or is a negative number, elements() will ignore it.

classmethod fromkeys(iterable, v=None)¶: Create a new ordered dictionary with keys from iterable and values set to value.

get(key, default=None, /)¶: Return the value for key if key is in the dictionary, else default.

items() → a set-like object providing a view on D’s items¶

keys() → a set-like object providing a view on D’s keys¶

most_common(n=None)¶

List the n most common elements and their counts from the most common to the least. If n is None, then list all element counts.

>>> Counter('abracadabra').most_common(3)
[('a', 5), ('b', 2), ('r', 2)]

move_to_end(key, last=True)¶

Move an existing element to the end (or beginning if last is false).

Raise KeyError if the element does not exist.

pop(k[, d]) → v, remove specified key and return the corresponding¶: value. If key is not found, d is returned if given, otherwise KeyError is raised.

popitem(last=True)¶

Remove and return a (key, value) pair from the dictionary.

Pairs are returned in LIFO order if last is true or FIFO order if false.

setdefault(key, default=None)¶

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

subtract(iterable=None, /, **kwds)¶

Like dict.update() but subtracts counts instead of replacing them. Counts can be reduced below zero. Both the inputs and outputs are allowed to contain zero and negative counts.

Source can be an iterable, a dictionary, or another Counter instance.

>>> c = Counter('which')
>>> c.subtract('witch')             # subtract elements from another iterable
>>> c.subtract(Counter('watch'))    # subtract elements from another counter
>>> c['h']                          # 2 in which, minus 1 in witch, minus 1 in watch
0
>>> c['w']                          # 1 in which, minus 1 in witch, minus 1 in watch
-1

update(iterable=None, /, **kwds)¶

Like dict.update() but add counts instead of replacing them.

Source can be an iterable, a dictionary, or another Counter instance.

>>> c = Counter('which')
>>> c.update('witch')           # add elements from another iterable
>>> d = Counter('watch')
>>> c.update(d)                 # add elements from another counter
>>> c['h']                      # four 'h' in which, witch, and watch
4

values() → an object providing a view on D’s values¶

Sized¶

class graphtage.utils.Sized(*args, **kwargs)¶

Bases: typing.Protocol

A protocol for objects that have a getsizeof function.

__init__(*args, **kwargs)¶: Initialize self. See help(type(self)) for accurate signature.

getsizeof: Callable[], int]¶: Returns the size of this object.

SparseMatrix¶

class graphtage.utils.SparseMatrix(num_rows: Optional[int] = None, num_cols: Optional[int] = None, default_value: Optional[T] = None)¶

Bases: graphtage.utils.Sized, collections.abc.Mapping, typing.Generic

A sparse matrix that can store arbitrary Python objects.

For sparse matrices storing homogeneous items and/or native types, it is more efficient to use an implementation like a scipy sparse matrix.

class SparseMatrixRow(row_num: int, num_cols: Optional[int] = None, default_value: Optional[T] = None)¶

Bases: graphtage.utils.Sized, collections.abc.MutableMapping, typing.Generic

A row of a sparse matrix.

__getitem__(col: int) → Optional[T]¶

Returns the value of the given column of this row.

Parameters: col – The index of the column to get.
Returns: The value of the column, or the default value if it has not yet been set.
Return type: Optional[T]
Raises: IndexError – If self.num_cols is not None and col is greater than or equal to it.

__init__(row_num: int, num_cols: Optional[int] = None, default_value: Optional[T] = None)¶

Initializes a sparse matrix row.

Parameters

row_num – The index of this row.
num_cols – An optional number of columns in this row. If omitted, this row will be unbounded and allow insertion and access at any column index.
default_value – An optional default value to use if a column is accessed before it is assigned.

__iter__() → Iterator[int]¶: Iterates over the indexes of columns that have been set in this row.

__len__() → int ¶: Returns the number of cells defined this row.

clear()¶: Clears the contents of this row.

default_value: Optional[T]¶: The default value for this row.

get(k[, d]) → D[k] if k in D, else d. d defaults to None.¶

getsizeof() → int ¶

Returns an approximation of the number of bytes used by this row in memory.

This is equivalent to:

return sys.getsizeof(self) + getsizeof(self.row)

items() → a set-like object providing a view on D’s items¶

keys() → a set-like object providing a view on D’s keys¶

num_cols: Optional[int]¶: The number of columns in this row.

pop(k[, d]) → v, remove specified key and return the corresponding value.¶: If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair¶: as a 2-tuple; but raise KeyError if D is empty.

row: Dict[int, Optional[T]]¶: Data structure holding the contents of this row.

row_num: int¶: The index of this row.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D¶

shape() → int ¶

The number of columns in this row.

This is equivalent to:

if self.num_cols is None:
    if self.row:
        return max(self.row.keys()) + 1
    else:
        return 0
else:
    return self.num_cols

update([E, ]**F) → None. Update D from mapping/iterable E and F.¶: If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values() → an object providing a view on D’s values¶

__getitem__(row: int) → MutableMapping[int, Optional[T]]¶

Returns the value of the given row of this matrix.

Parameters: row – The index of the row to get.
Returns: The contents of the row.
Return type: MutableMapping[int, Optional[T]]
Raises: IndexError – If self.num_rows is not None and row is greater than or equal to it.

__init__(num_rows: Optional[int] = None, num_cols: Optional[int] = None, default_value: Optional[T] = None)¶

Initializes a sparse matrix.

Parameters

num_rows – An optional bound on the number of rows.
num_cols – An optional bound on the number of columns.
default_value – An optional default value to return if cells are accessed before they are set.

__len__() → int ¶

Returns the number of rows in this matrix.

If self.num_rows is None, this will return the maximum row index of a cell that has been set, plus one.

clear()¶: Clears the contents of this matrix.

default_value: Optional[T]¶: The default value to return if cells are accessed before they are set.

get(k[, d]) → D[k] if k in D, else d. d defaults to None.¶

getsizeof() → int ¶: Calculates the approximate memory footprint of this matrix in bytes.

items() → a set-like object providing a view on D’s items¶

keys() → a set-like object providing a view on D’s keys¶

num_cols: Optional[int]¶: The number of columns in this matrox, or None if there is no bound.

num_filled_elements() → int ¶: Counts the number of elements in this matrix that have been explicitly set.

num_rows: Optional[int]¶: The number of rows in this matrix, or None if there is no bound.

rows: Dict[int, SparseMatrix.SparseMatrixRow]¶: The rows of this matrix.

shape() → Tuple[int, int]¶: Returns the (number of rows, number of columns) of this matrix.

values() → an object providing a view on D’s values¶

Tempfile¶

class graphtage.utils.Tempfile(contents: bytes, prefix: Optional[str] = None, suffix: Optional[str] = None)¶

Bases: object

Creates a temporary file containing a given byte sequence.

The file will automatically be cleaned up after its use. This is useful for interacting with functions and libraries that only accept a path and not a stream object.

Examples

>>> from graphtage.utils import Tempfile
>>> with Tempfile(b"foo") as tmp:
...     print(tmp)
/var/folders/bs/hrvzrctx6tg2_j17gb6wckph0000gn/T/tmpkza5fvr_
>>> with Tempfile(b"foo") as tmp:
...     with open(tmp, 'r') as f:
...         print(f.read())
foo

__enter__() → str ¶: Constructs a tempfile, returning the path to the file.

__exit__(type, value, traceback)¶: Automatically cleans up the tempfile.

__init__(contents: bytes, prefix: Optional[str] = None, suffix: Optional[str] = None)¶

Initializes a Tempfile

Parameters

contents – The contents to be populated in the file.
prefix – An optional prefix for the filename.
suffix – An optional suffix for the filename.

utils functions¶

getsizeof¶

graphtage.utils.getsizeof(obj) → int ¶

A function to calculate the memory footprint of an object.

If the object implements the Sized protocol (i.e., if it implements a getsizeof method), then this is used:

if hasattr(obj, 'getsizeof'):
    return obj.getsizeof()

If the object is a list or tuple, then:

return sys.getsizeof(obj) + sum(getsizeof(i) for i in obj)

If the object is a dict, then:

return sys.getsizeof(obj) + sum(getsizeof(key) + getsizeof(value) for key, value in obj.items())

Otherwise:

return sys.getsizeof(obj)

Parameters: obj – The object to measure.
Returns: An approximation of the memory footprint of the object, in bytes.
Return type: int

largest¶

graphtage.utils.largest(*sequence: Union[T, Iterable[T]], n: int = 1, key: Optional[Callable[[T], Any]] = None) → Iterator[T]¶

smallest¶

graphtage.utils.smallest(*sequence: Union[T, Iterable[T]], n: int = 1, key: Optional[Callable[[T], Any]] = None) → Iterator[T]¶