Loading and saving data#

Eyelink EDF files#

EDF files from SR Research Eyelink eye trackers are read with from_eyelink(), which uses the eyelinkio package:

import pypillometry as pp

d = pp.EyeData.from_eyelink("recording.edf")
p = d.drop_data(variables=["x", "y"], inplace=False)   # pupil only

The result contains all samples, all messages (as events), the screen resolution and the calibration/validation results (as SpatialCalibration objects in d.calibration). Eyelink codes missing pupil samples as 0; these are converted to NaN and masked. Eyelink’s own messages (calibration, drift correction, …) are removed from the events unless you pass remove_eyelink_triggers=False.

Instead of a local file name, you can also pass an HTTP(S) URL or an SSH URL (ssh://server/folder/file.edf). The file is then downloaded and cached (see fetch_url()).

Files from public figshare items are listed with get_figshare_files() and downloaded (cached, checksum-verified) with fetch_figshare_file(), using the item’s ID, URL or DOI:

pp.get_figshare_files(28776617)    # file name -> id, url, size, md5
fname = pp.fetch_figshare_file(28776617, "001_rlmw_samples.asc",
                               cache_dir="data")

Reading large EDF files is slow (about a minute for a 90 MB file). Save the result (see below), which is read in well under a second, or convert the file once with the command-line tool edf2pypil.

See Importing data from Eyelink EDF files.

Text files and your own arrays#

Data from other eye trackers, or exported as text (e.g., Eyelink’s ASC format), can be read with numpy or pandas and passed to the constructors as arrays:

d = pp.EyeData(
    time=df["time"],                      # in ms
    left_x=df["lx"], left_y=df["ly"], left_pupil=df["lp"],
    right_x=df["rx"], right_y=df["ry"], right_pupil=df["rp"],
    event_onsets=events["time"],          # in ms
    event_labels=events["label"],
    sampling_rate=500,
    name="subject01",
)

Any of the eyes or variables can be left out (e.g., only left_pupil for pupil data without gaze); the x and y of an eye go together. Gaps in the time vector (e.g., when recording was paused) are filled with masked samples, so the samples are evenly spaced.

See Importing data from Eyelink ASC files for a complete example.

Events that are only in the behaviour log of the experiment (on its own clock) can be added after loading; see Events from a behaviour log.

Example datasets#

A few datasets are available for trying out the package:

pp.get_available_datasets()                # names and descriptions
d = pp.get_example_data("rlmw_002_short")  # 40 s, included in the package
d = pp.get_example_data("rlmw_002")        # 20 min, downloaded from OSF

See Example datasets included in pypillometry.

Saving and loading objects#

Processed objects, including their history, are saved with write_file() and loaded with from_file():

d.write_file("subject01.pypil")
d = pp.EyeData.from_file("subject01.pypil")

Files are pickle files compressed with zstd (pass compress=False to turn this off). write_file() and read_file() do the same for any Python object, e.g., a list of objects or a SegmentedEyeData. read_file() also reads files from a URL.

Warning

Pickle files can contain arbitrary code. Only load files from sources you trust.

If the filename ends in .h5 or .hdf5, the object is saved in an HDF5 file instead:

d.write_file("subject01.h5")
d = pp.EyeData.from_file("subject01.h5")

HDF5 files do not run code when they are read, and other programs (e.g., h5py, R or MATLAB) can read them: the data are datasets (tx, data/values/left_pupil, data/layers/left_pupil, …; the mask layers are bits, bit i is the layer data/layer_names[i], e.g. "blink"), the other information (name, sampling rate, events, history, …) is stored in groups and attributes. They work for eyedata objects and SegmentedEyeData (not for arbitrary Python objects). Values in info or params that HDF5 cannot store are saved as text (with a warning). The loading functions detect the format from the file content.

Both formats store a versioned description of the object. Files written by older versions of pypillometry are upgraded when they are loaded; pypil-upgrade converts them once (see Command-line tools).

Saved files can be inspected with pypilinfo and opened in the viewer with pypilview (see Command-line tools).

Whole studies#

For studies with many participants, a small configuration file describes where the raw data are and how to read them. load_study_local() and load_study_osf() then load all participants at once (see Studies with many participants).