EyeDataDict#

class pypillometry.EyeDataDict(*args, **kwargs)[source]#

Bases: MutableMapping, Stateful

A dictionary that contains 1-dimensional ndarrays of equal length and with the same datatype (float). Drops empty entries (None or length-0 ndarrays).

Keys stored in this dictionary have the following shape: (eye)_(variable) where eye can be any string identifier (e.g., “left”, “right”, “mean”, “median”, “regress”, or any other custom identifier) and variable is any string identifier (e.g., “x”, “y”, “pupil”, “baseline”, “response”, or any other custom identifier).

Values are stored as given (converted to float); NaN values are masked. No other values are treated as missing (e.g., a pupil size of 0 is kept); converting missing-value codes to NaN is up to the code that reads the data.

Each key also has mask layers: a bit-field (layers, dtype LAYER_DTYPE) with one bit per named layer (e.g., "blink"). The layer names are registered in layer_names (the same for all keys); a layer gets the next free bit when it is first set. Layers that apply to all keys (e.g., "excluded") are stored once, in rec_layers (recording-wide layers, same registry). mask is the mask under the default mask policy as one 0/1 array per key (see MaskPolicy).

The dictionary can be indexed by a string “eye_variable” or by a tuple (“eye”, “variable”) like data[“left”,”x”] or data[“left_x”].

Methods

`add_mask`	Mask the samples where of key (`"user"` layer; existing masks are kept).
`clear`
`clear_layer`	Clear layer name of key everywhere or at the samples where (an index, e.g. a boolean array or `np.s_[10:20]`).
`clear_rec_layer`	Clear the recording-wide layer name.
`copy`	Create a deep copy of the dictionary (data and masks).
`count_masked`	Count the number of missing values based on masks.
`get`
`get_available_eyes`	Return a list of available eyes.
`get_available_variables`	Return a list of available variables.
`get_eye`	Return a subset EyeDataDict with all variables for a given eye.
`get_layer`	Boolean array that is True where layer name is set for key.
`get_mask`	Get mask for a specific key or joint mask across all keys.
`get_policy_mask`	Boolean mask of key under policy (True: unusable sample).
`get_rec_layer`	Boolean array that is True where the recording-wide layer name is set.
`get_size`	Return the size of the dictionary in bytes.
`get_variable`	Return a subset EyeDataDict with all eyes for a given variable.
`items`
`keys`
`layer_bits`	Bit mask of the layers names (0 for layers that are not registered).
`pop`	value.
`popitem`	as a 2-tuple; but raise KeyError if D is empty.
`select`	Return the existing (eye, variable) pairs for the given eyes and variables.
`set_layer`	Set layer name of key where mask is True.
`set_mask`	Set mask for a specific key.
`set_rec_layer`	Set the recording-wide layer name (all keys) where mask is True.
`set_with_mask`	Set data array with explicit mask control.
`setdefault`
`update`	If E present and has a .keys() method, does:
`values`

Details

add_mask(key, where)[source]#

Mask the samples where of key ("user" layer; existing masks are kept).

Parameters:

key (str) – Key of the data (e.g., “left_pupil”)
where (array-like of bool, slice, int or array of int) – the samples to mask: a boolean array (one value per sample, True: mask the sample) or any index, e.g. np.s_[10:20]

Raises:

KeyError – if there is no data for key

Return type:

None

clear() → None. Remove all items from D.#

clear_layer(key, name, where=None)[source]#

Clear layer name of key everywhere or at the samples where (an index, e.g. a boolean array or np.s_[10:20]).

Return type:: None

clear_rec_layer(name)[source]#

Clear the recording-wide layer name.

Return type:: None

copy()[source]#

Create a deep copy of the dictionary (data and masks).

Return type:: EyeDataDict

count_masked(per_key=False)[source]#

Count the number of missing values based on masks.

Parameters:: per_key (bool) – If True, return a dictionary with counts per key. If False, return maximum count across all keys. Default is False.
Return type:: Union[int, Dict[str, int]]
Returns:: If per_key is False, returns maximum number of missing values across all keys. If per_key is True, returns a dictionary mapping keys to their missing value counts.

Examples

>>> d = EyeDataDict(left_x=[1,2,3], left_y=[4,5,6])
>>> d.set_mask("left_x", [0,1,0])  # second value is missing
>>> d.set_mask("left_y", [1,0,1])  # first and last values are missing
>>> d.count_masked()  # max missing values
2
>>> d.count_masked(per_key=True)  # missing values per key
{'left_x': 1, 'left_y': 2}

get(k[, d]) → D[k] if k in D, else d. d defaults to None.#

get_available_eyes(variable=None)[source]#

Return a list of available eyes.

Parameters:: variable (str, optional) – If specified, return only eyes for this variable.
Return type:: list of str
Returns:: names of the available eyes (e.g., ["left", "right"]), in the order in which they were added

get_available_variables()[source]#

Return a list of available variables.

Return type:: list of str
Returns:: names of the available variables (e.g., ["x", "y", "pupil"]), in the order in which they were added

get_eye(eye)[source]#

Return a subset EyeDataDict with all variables for a given eye.

Parameters:: eye (str) – The eye to get data for (‘left’, ‘right’, ‘mean’, etc.)
Return type:: EyeDataDict
Returns:: A new EyeDataDict containing only data for the specified eye (copies, with their masks and layers)

Examples

>>> d = EyeDataDict(left_x=[1,2], left_y=[3,4], right_x=[5,6])
>>> left_data = d.get_eye('left')
>>> print(left_data.data.keys())
dict_keys(['left_x', 'left_y'])

get_layer(key, name)[source]#

Boolean array that is True where layer name is set for key.

Return type:: NDArray

get_mask(key=None)[source]#

Get mask for a specific key or joint mask across all keys.

Parameters:: key (str) – Key to get mask for. If None, returns logical OR of all masks.
Return type:: NDArray
Returns:: Mask array for specified key, or joint mask if no key given.

get_policy_mask(key, policy='default')[source]#

Boolean mask of key under policy (True: unusable sample).

Parameters:

key (str) – key of the data (e.g., “left_pupil”)
policy (str, MaskPolicy or list of str) – a predefined policy (see POLICIES), a MaskPolicy, or a list of layers (each of them masks a sample; NaN as well)

Raises:

ValueError – if the policy names a layer that is neither a standard layer nor registered

Return type:

NDArray

get_rec_layer(name)[source]#

Boolean array that is True where the recording-wide layer name is set.

Return type:: NDArray

get_size()[source]#

Return the size of the dictionary in bytes.

Return type:: ByteSize
Returns:: Total size in bytes.

get_variable(variable)[source]#

Return a subset EyeDataDict with all eyes for a given variable.

Parameters:: variable (str) – The variable to get data for (‘x’, ‘y’, ‘pupil’, etc.)
Return type:: EyeDataDict
Returns:: A new EyeDataDict containing only data for the specified variable (copies, with their masks and layers)

items() → a set-like object providing a view on D's items#

keys() → a set-like object providing a view on D's keys#

layer_bits(names, create=False)[source]#

Bit mask of the layers names (0 for layers that are not registered).

Parameters:

names (str or list of str) – layer name(s)
create (bool) – register layers that are not registered yet (next free bit)

Return type:

int

Returns:

the bits of the layers OR-ed together

Raises:

ValueError – if a new layer is needed but all MAX_LAYERS bits are used

pop(k[, d]) → v, remove specified key and return the corresponding#: value. If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair#: as a 2-tuple; but raise KeyError if D is empty.

select(eyes=None, variables=None)[source]#

Return the existing (eye, variable) pairs for the given eyes and variables.

Only pairs for which data is stored are returned, so data where not every eye has every variable (e.g., left_pupil, left_x, right_pupil) can be processed pair by pair.

Parameters:

eyes (str or list of str, optional) – eye(s) to select; if None or empty, all available eyes
variables (str or list of str, optional) – variable(s) to select; if None or empty, all available variables

Return type:

List[Tuple[str, str]]

Returns:

(eye, variable) pairs, ordered by eye, then variable (in the order given or, for all eyes/variables, the order of the data)

Raises:

KeyError – if something that was explicitly requested does not exist: a pair (if both eyes and variables are given), or an eye or variable without any data among the selected pairs (if only one of them is given). The message lists the available channels.
ValueError – if a requested eye name contains _ (eye names may not contain _, see _validate_key())

Examples

>>> d = EyeDataDict(left_pupil=[1,2], left_x=[3,4], right_pupil=[5,6])
>>> d.select()
[('left', 'pupil'), ('left', 'x'), ('right', 'pupil')]
>>> d.select(variables="x")
[('left', 'x')]

set_layer(key, name, mask, replace=False)[source]#

Set layer name of key where mask is True.

Parameters:

key (str) – key of the data (e.g., “left_pupil”)
name (str) – layer name (registered if new)
mask (NDArray) – one value per sample
replace (bool) – clear the layer first (True) or add to it (False, default)

Return type:

None

set_mask(key, mask, layer='user')[source]#

Set mask for a specific key.

Afterwards, self.mask[key] (the mask under the default policy) equals mask: samples that are not masked in mask are removed from the layers of the key that mask them, samples that were not masked yet are added to layer (default "user"; where they are interpolated, the "interpolated" layer is cleared, as it would make them usable). NaN values and the recording-wide layers (e.g., excluded periods) are not changed, so these samples stay masked.

Parameters:

key (str) – Key of the data (e.g., “left_pupil”)
mask (NDArray) – Mask array (1 for masked, 0 for valid values) with the same shape as the data; converted to int
layer (str) – layer for the newly masked samples (default "user")

Raises:

KeyError – if there is no data for key
ValueError – if mask does not have the same shape as the data

Return type:

None

set_rec_layer(name, mask, replace=False)[source]#

Set the recording-wide layer name (all keys) where mask is True.

Parameters:

name (str) – layer name (registered if new)
mask (NDArray) – one value per sample
replace (bool) – clear the layer first (True) or add to it (False, default)

Return type:

None

set_with_mask(key, value, mask=None, preserve_mask=False)[source]#

Set data array with explicit mask control.

This method provides more control over mask handling than __setitem__, which always resets the mask to zeros (except for NaN values) and clears the layers.

Parameters:

key (Union[str, Tuple[str, str]]) – Dictionary key for the data (e.g., “left_pupil” or (“left”, “pupil”))
value (NDArray) – Data array to store
mask (Optional[NDArray]) – Mask array to use (see set_mask). If None and preserve_mask=True, keeps existing mask and layers. If None and preserve_mask=False, creates new zero mask (with NaNs masked) and clears the layers.
preserve_mask (bool) – If True and mask is None, preserve the existing mask and layers for this key. Ignored if mask is explicitly provided.

Return type:

None

Examples

>>> d = EyeDataDict(left_pupil=[1., 2., 3.])
>>> d.set_mask("left_pupil", [0, 1, 0])
>>> d.set_with_mask("left_pupil", [4., 5., 6.], preserve_mask=True)
>>> d.mask["left_pupil"]
array([0, 1, 0])
>>> d.set_with_mask("left_pupil", [4., 5., 6.], mask=[1, 0, 0])
>>> d.mask["left_pupil"]
array([1, 0, 0])
>>> d.set_with_mask("left_pupil", [4., np.nan, 6.])  # fresh mask
>>> d.mask["left_pupil"]
array([0, 1, 0])

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D#

update([E, ]**F) → None. Update D from mapping/iterable E and F.#

If E present and has a .keys() method, does:: for k in E.keys(): D[k] = E[k]
If E present and lacks .keys() method, does:: for (k, v) in E: D[k] = v
In either case, this is followed by:: for k, v in F.items(): D[k] = v

values() → an object providing a view on D's values#

property eyes: List[str]#: List of all available eyes.

property mask: Mapping#

Mask of each key under the default mask policy (read-only mapping key -> int array, 1: masked; see get_policy_mask).

The arrays are read-only.

property variables: List[str]#: List of all available variables.