Surveyors: Traverse Structures¶

Surveyors chart nested data, using chained Compass objects to build (Course, value) pairs

Surveyor Class¶

class gemma.Surveyor(compasses=None, compasses_extra=None, end_points=None, end_points_extra=None, course_type=<class 'gemma._course.Course'>)[source]¶

__init__(compasses=None, compasses_extra=None, end_points=None, end_points_extra=None, course_type=<class 'gemma._course.Course'>)[source]¶

Charts courses through data structure

Parameters

compasses (Optional[List[Compass]]) – available compasses to navigate bearings
compasses_extra (Optional[List[Compass]]) – compasses to use in addition to defaults
end_points (Union[Tuple[Type, …], Type, None]) – types where courses should terminate
end_points_extra (Union[Tuple[Type, …], Type, None]) – end points to use in addition to defaults
course_type (Type[Course]) – course class to use for course creation

The main functionality for this course is through Surveyor.chart_iter() and Surveyor.chart()

See below documentation for usage.

chart(target, exceptions=True)[source]¶

As Surveyor.chart_iter(), but returns all result pairs as list.

Parameters

target (Any) – data structure to chart
exceptions (bool) –
- True: Exceptions will be raised during process
- False: Exceptions will be suppressed and a SuppressedErrors Error will be raised at the end.

Return type

List[Tuple[Course, Any]]

Returns

list of (Course, value) pairs

Raises

NonNavigableError – If a target is found that can’t be navigated. This error will be suppressed if exceptions is set to False
SuppressedErrors – Raised at end if NonNavigableError occurs and exceptions is set to False`. Partial chart can be recovered from ``SuppressedErrors.chart_partial

See examples below.

chart_iter(target, exceptions=True)[source]¶

Charts data structure, yielding each (Course, value) pair one at a time

Parameters

target (Any) – data structure to chart
exceptions (bool) –
- True: Exceptions will be raised during process
- False: Exceptions will be suppressed and a SuppressedErrors Error will be raised at the end.

Return type

Generator[Tuple[Course, Any], None, None]

Returns

yields Course, value pairs

Raises

NonNavigableError – If a target is found that can’t be navigated. This error will be suppressed if exceptions is set to False
SuppressedErrors – Raised at end if NonNavigableError occurs and exceptions is set to False

See examples below.

Get all Courses in Object¶

>>> from gemma import Surveyor
>>> from gemma.test_objects import test_objects
...
... simple, data_dict, data_list, structured, target = test_objects()
>>> default = Surveyor()
>>>
>>> for course, value in default.chart_iter(data_dict):
...     print(repr(course), '|', value)
...
<Course: <Item: 'a'>> | a dict
<Course: <Item: 'b'>> | b dict
<Course: <Item: 1>> | one dict
<Course: <Item: 2>> | two dict
<Course: <Item: 'nested'>> | {'one key': 1, 'two key': 2}
<Course: <Item: 'nested'> / <Item: 'one key'>> | 1
<Course: <Item: 'nested'> / <Item: 'two key'>> | 2
<Course: <Item: 'simple'>> | DataSimple(text='string value', number=40)
<Course: <Item: 'simple'> / <Attr: 'text'>> | string value
<Course: <Item: 'simple'> / <Attr: 'number'>> | 40

example object refs: data_dict

Charting with Custom Compasses¶

>>> keys = ['a', 'b', 'nested']
>>> dict_compass = Compass(target_types=dict, items=keys)
>>>
>>> custom = Surveyor(compasses=[dict_compass])
>>> for course, value in custom.chart_iter(data_dict):
...     print(repr(course), value)
...
<Course: <Item: 'a'>> a dict
<Course: <Item: 'b'>> b dict
<Course: <Item: 'nested'>> {'one key': 1, 'two key': 2}

Only "a", "b", and "nested" are returned. Why did the :class:’Surveyor’ not recurse into "nested", instead returning only the first layer?

Because we have supplied only one Compass -- *tied to all ``dict``objects* -- that returns keys for `”a”, ``"b", and "nested" only. Lets override Compass and set a better filter for when it should be used, something more granular than type.

>>> class DictDataCompass(Compass):
...     def __init__(self):
...         keys = ['a', 'b', 'nested']
...         super().__init__(target_types=dict, items=keys)
...
...     def is_navigable(self, target):
...         if not super().is_navigable(other):
...             return False
...         if 'nested' in other:
...             return True
...         else:
...             return False
...

We override Compass.is_navigable(), checking that target is a dict through the super() call, then confirm the "nested" key exists – the defining characteristic of our target dict.

We also override __init__ to restrict our return keys rather than doing it every time this compass is initialized.

Lets load it into a Surveyor and try to chart dict_data again.

>>> dict_data_compass = DictDataCompass()
>>> custom = Surveyor(compasses=[dict_data_compass])
>>>
>>> for course, value in custom.chart_iter(data_dict):
...     print(repr(course), value)
...
<Course: <Item: 'a'>> a dict
<Course: <Item: 'b'>> b dict
<Course: <Item: 'nested'>> {'one key': 1, 'two key': 2}
Traceback (most recent call last):
    ...
gemma._exceptions.NonNavigableError: could not find compass for f{'one ...

A NonNavigableError once the surveyor hits 'nested'. Why?

Because we have only supplied a Compass that can handle dict_data, but no Compass that can handle a generic dict.

Lets use the default Compass. We list it after DictDataCompass, so the default is not chosen first for every object. Surveyor uses the first compatible Compass it finds.

>>> default_compass = Compass()
>>> custom = Surveyor(compasses=[dict_data_compass, default_compass])
>>> for course, value in custom.chart_iter(data_dict):
...     print(repr(course), value)
...
<Course: <Item: 'a'>> a dict
<Course: <Item: 'b'>> b dict
<Course: <Item: 'nested'>> {'one key': 1, 'two key': 2}
<Course: <Item: 'nested'> / <Item: 'one key'>> 1
<Course: <Item: 'nested'> / <Item: 'two key'>> 2

Now we get the full chart.

We can use compasses_extra to add our special Compass before the default automatically.

>>> custom = Surveyor(compasses_extra=[dict_data_compass])
>>>
>>> simple, data_dict, data_list, structured, target = test_objects()
>>> for course, value in custom.chart_iter(data_dict):
...     print(repr(course), value)
...
<Course: <Item: 'a'>> a dict
<Course: <Item: 'b'>> b dict
<Course: <Item: 'nested'>> {'one key': 1, 'two key': 2}
<Course: <Item: 'nested'> / <Item: 'one key'>> 1
<Course: <Item: 'nested'> / <Item: 'two key'>> 2

The default Compass is used for any objects compasses_extra doesn’t catch.

example object refs: data_dict

Adding Additional End Points¶

The surveyor needs to know how it recognizes where a course should terminate, and has a tuple of types it will not traverse into.

The default types are: (str, int, float, type). Additional types can be anything capable of an isinstance() check.

In the last example, data_dict contains a Dataclass, “DataSimple”. Lets say we don’t wish to recurse into DataSimple, instead treating it as primary data type.

We can add it as an additional end points type as so:

>>> from gemma.test_objects import DataSimple
>>> more_end_points = Surveyor(end_points_extra=(DataSimple,))
>>> for course, value in more_end_points.chart_iter(data_dict):
...     print(repr(course), value)
...
<Course: <Item: 'a'>> a dict
<Course: <Item: 'b'>> b dict
<Course: <Item: 1>> one dict
<Course: <Item: 2>> two dict
<Course: <Item: 'nested'>> {'one key': 1, 'two key': 2}
<Course: <Item: 'nested'> / <Item: 'one key'>> 1
<Course: <Item: 'nested'> / <Item: 'two key'>> 2
<Course: <Item: 'simple'>> DataSimple(text='string value', number=40)

The resulting courses do not dig further then the “simple” key.

Like compasses and compasses_extra, you can change or remove the default end points using the end_points keyword argument.

To see why endpoints are important, try removing str as an endpoint and chart a dict with string values.

Suppressing NonNavigableError¶

It may be useful in certain situations to get a partial map, suppressing NonNavigableError exceptions instead of throwing them and not returning.

The exceptions keyword will suppress errors until the end of the operation. Lets take a look at the normal way:

>>> from gemma import Compass, Surveyor
>>>
>>> dict_compass = Compass(target_types=dict)
>>> data = {
...     'a': 'a value',
...     'list': [1, 2, 3],
...     'dict': {'b': 'b value'}
... }
...
>>> for x in raises_non_navigable.chart_iter(data):
...     print(repr(x))
...
(<Course: <Item: 'a'>>, 'a value')
(<Course: <Item: 'list'>>, [1, 2, 3])
Traceback (most recent call last):
    ...
gemma._exceptions.NonNavigableError: could not find compass for f[1, 2, 3]

At the "list" key, we throw an error – our surveyor cannot traverse lists with the provided compasses.

Lets set exceptions to False to suppress the exception and continue charting.

>>> for x in raises_non_navigable.chart_iter(data, exceptions=False):
...     print(repr(x))
...
(<Course: <Item: 'a'>>, 'a value')
(<Course: <Item: 'list'>>, [1, 2, 3])
(<Course: <Item: 'dict'>>, {'b': 'b value'})
(<Course: <Item: 'dict'> / <Item: 'b'>>, 'b value')
Traceback (most recent call last):
    ...
gemma._exceptions.SuppressedErrors: some objects could not be charted

The surveyor finished before raising a SuppressedErrors exception. It can’t traverse into the list, and moves past it.

When using Surveyor.chart() instead of Surveyor.chart_iter(), you can recover a partial list from SuppressedErrors.chart_partial.

>>> from gemma import SuppressedErrors
>>>
>>> try:
...     chart = raises_non_navigable.chart(data, exceptions=False)
... except SuppressedErrors as error:
...     print(error.chart_partial)
...
[(<Course: <Item: 'a'>>, 'a value'), (<Course: <Item: 'list'>>, [1, 2, 3]), ...

Get a list of the Suppressed errors:

>>> try:
...     chart = raises_non_navigable.chart(data, exceptions=False)
... except SuppressedErrors as error:
...     print(error.errors)
...
[NonNavigableError('could not find compass for f[1, 2, 3]')]