Pipeline-based processing in pypillometry#
[1]:
import pypillometry as pp
from pypillometry import u
pypillometry processes data in pipelines: processing steps (e.g., blink detection, a filter, downsampling) are combined with the | operator and applied to an EyeData object:
[2]:
# load a small example dataset
d = pp.get_example_data("rlmw_002_short")
d = d | pp.pupil.blinks_detect() \
| pp.pupil.blinks_merge_close(distance=100 * u.ms) \
| pp.pupil.lowpass_filter(3) \
| pp.downsample(50)
This command loads a data file (here we use an example dataset included with pypillometry), detects blinks in the signal and masks them, merges short, successive blinks, applies a 3 Hz low-pass filter, and downsamples the signal to 50 Hz. The final result of this processing pipeline is stored in object d.
The steps for pupil data are pp.pupil.<method>(...), those for gaze data pp.gaze.<method>(...), and the generic steps are available at the top level of the package (pp.downsample, pp.crop, pp.scale, …). They take the same arguments as the methods of the same name (d.pupil.blinks_detect(), d.downsample(), …), which run the same code.
Here, for better visibility, we put each operation on a separate line. For that to work, we need to tell Python that the statement has not yet ended, which we achieve by putting a backslash \ at the end of each (non-final) line (or by putting the whole expression in parentheses).
We can get a useful summary of the dataset and the operations applied to it by simply printing it:
[3]:
d
[3]:
EyeData — rlmw_002_short
Signal overview
Blink statistics
| Eye | Count | Mean ± SD (ms) | Range (ms) |
|---|---|---|---|
| left_pupil | 13 | 236.9 ± 256.9 | [60.0, 980.0] |
| right_pupil | 13 | 136.9 ± 84.8 | [40.0, 400.0] |
Data glimpse
EyeDataDict(vars=6,n=2047,shape=(2047,)): left_x (float64): 656.55, 656.14, 654.04, 652.02, 650.4... left_y (float64): 597.54, 591.5, 588.39, 593.78, 598.32... left_pupil (float64): 1128.8869, 1133.8058, 1139.7069, 1146.4462, 1153.8028... right_x (float64): 775.56, 777.14, 777.08, 776.99, 778.72... right_y (float64): 597.31, 601.3, 600.48, 595.34, 595.1... right_pupil (float64): 954.2415, 958.9101, 963.6321, 968.2405, 972.675...
Recent history
- reset_time()
- pupil.blinks_detect()
- pupil.blinks_merge_close(distance=100 ms)
- pupil.lowpass_filter(3)
- downsample(50)
We see that sampling rate, number of datapoints and more is printed along with the history of all operations applied to the dataset. The same information can be retrieved in a form useful for further processing with the function summary(), which returns a dict:
[4]:
d.summary()
[4]:
{'name': 'rlmw_002_short',
'n': 2047,
'sampling_rate': 50.0,
'eyes': ['left', 'right'],
'data': ['left_x',
'left_y',
'left_pupil',
'right_x',
'right_y',
'right_pupil'],
'nevents': 40,
'experimental_setup': {'screen_resolution': (1280, 1024),
'physical_screen_size_mm': (300.0, 200.0),
'eye_to_screen_distance_mm': 600.0,
'camera_position': 'not set'},
'calibration': 'not available',
'duration_minutes': 0.6823333333333333,
'start_min': 0.00015,
'end_min': 0.6820666666666667,
'params': {},
'nblinks': {'left_pupil': 13, 'right_pupil': 13},
'blinks': {'left_pupil': {'n': 13,
'mean': 236.92307692307693,
'sd': 256.94610675226807,
'min': 60.0,
'max': 980.0,
'total_duration': 3080.0,
'units': 'ms'},
'right_pupil': {'n': 13,
'mean': 136.92307692307693,
'sd': 84.79700787397336,
'min': 40.0,
'max': 400.0,
'total_duration': 1780.0,
'units': 'ms'}}}
The history alone can be shown with print_history():
[5]:
d.print_history()
* rlmw_002_short
└ reset_time()
└ pupil.blinks_detect()
└ pupil.blinks_merge_close(distance=100 ms)
└ pupil.lowpass_filter(3)
└ downsample(50)
Pipelines are values#
A combination of steps is a Pipeline. It can be stored in a variable, printed, and applied to as many datasets as needed (e.g., to all participants of a study):
[6]:
preproc = pp.pupil.blinks_detect() \
| pp.pupil.blinks_merge_close(distance=100 * u.ms) \
| pp.pupil.lowpass_filter(3)
print(preproc)
pp.pupil.blinks_detect() | pp.pupil.blinks_merge_close(distance=100 ms) | pp.pupil.lowpass_filter(3)
[7]:
d = pp.get_example_data("rlmw_002_short")
d2 = d | preproc | pp.downsample(50)
print(f"d has {len(d)} samples and {len(d.get_blinks()['left_pupil'])} blinks")
print(f"d2 has {len(d2)} samples and {len(d2.get_blinks()['left_pupil'])} blinks")
d has 20465 samples and 0 blinks
d2 has 2047 samples and 13 blinks
d | pipeline does not change d: it copies d once and applies all steps to the copy, which is returned. (Python reads d | a | b as (d | a) | b, which copies at each |; for large datasets, apply the whole pipeline at once, as in d | preproc or d | (a | b).)
Parameters that differ between participants can be passed with a small function that returns the pipeline:
[8]:
def preproc_subj(vel_onset):
return pp.pupil.blinks_detect(vel_onset=vel_onset) | pp.pupil.lowpass_filter(3)
d2 = d | preproc_subj(vel_onset=-10)
If a step fails, the error message says which step it was.
Calling the methods directly#
The same operations are available as methods of the data object: the pupil methods in d.pupil, the gaze methods in d.gaze and the generic ones directly on d. This is convenient for single steps:
[9]:
d = pp.get_example_data("rlmw_002_short")
d.pupil.blinks_detect()
d.downsample(50)
[9]:
EyeData — rlmw_002_short
Signal overview
Blink statistics
| Eye | Count | Mean ± SD (ms) | Range (ms) |
|---|---|---|---|
| left_pupil | 14 | 218.6 ± 232.9 | [60.0, 980.0] |
| right_pupil | 13 | 136.9 ± 84.8 | [40.0, 400.0] |
Data glimpse
EyeDataDict(vars=6,n=2047,shape=(2047,)): left_x (float64): 656.55, 656.14, 654.04, 652.02, 650.4... left_y (float64): 597.54, 591.5, 588.39, 593.78, 598.32... left_pupil (float64): 1123.2, 1132.5, 1135.1, 1142.6, 1152.3... right_x (float64): 775.56, 777.14, 777.08, 776.99, 778.72... right_y (float64): 597.31, 601.3, 600.48, 595.34, 595.1... right_pupil (float64): 956.7, 964.2, 970.6, 970.8, 970.3...
Recent history
- reset_time()
- pupil.blinks_detect()
- downsample(50)
In-place vs. copying data#
d | step never changes d. The methods, however, work in-place by default. That means, after running an operation (e.g., d.downsample()), the dataset will have changed and the returned object is the same object as d.
For example, the next code downsamples the data to 50 Hz and returns dataset d2. However, since the operation was in-place, the original dataset d has also changed:
[10]:
d = pp.get_example_data("rlmw_002_short")
d2 = d.downsample(50)
print(f"d has {len(d)} samples")
print(f"d2 has {len(d2)} samples")
print(f"d and d2 are the same: {d is d2}")
d has 2047 samples
d2 has 2047 samples
d and d2 are the same: True
This can be tricky when re-running a cell of a notebook as the operations will be applied again to the already processed data (i.e., blink detection, downsampling, smoothing etc. will be run multiple times on the same data).
This can be prevented by adding inplace=False to any operation or by using d.copy() explicitly:
[11]:
# using inplace=False
d = pp.get_example_data("rlmw_002_short")
d2 = d.downsample(50, inplace=False)
print(f"d has {len(d)} samples")
print(f"d2 has {len(d2)} samples")
print(f"d and d2 are the same: {d is d2}")
d has 20465 samples
d2 has 2047 samples
d and d2 are the same: False
[12]:
# using copy()
d = pp.get_example_data("rlmw_002_short")
d2 = d.copy()
d2.downsample(50)
print(f"d has {len(d)} samples")
print(f"d2 has {len(d2)} samples")
print(f"d and d2 are the same: {d is d2}")
d has 20465 samples
d2 has 2047 samples
d and d2 are the same: False
If you prefer that operations never change the object they are called on, you can change the default for a single object (inplace=False when creating it) or for the whole package with pp.set_default_inplace():
[13]:
pp.set_default_inplace(False)
d = pp.get_example_data("rlmw_002_short")
d2 = d.downsample(50)
print(f"d has {len(d)} samples")
print(f"d2 has {len(d2)} samples")
pp.set_default_inplace(True) # back to the default
d has 20465 samples
d2 has 2047 samples