{ "cells": [ { "cell_type": "markdown", "id": "d0e5f760", "metadata": {}, "source": [ "# Missing data\n", "\n", "Pupil and gaze data have gaps and bad samples for many reasons: the eye\n", "tracker loses the pupil during a blink, the gaze leaves the screen, the\n", "two eyes point to different places, or a part of the recording is simply\n", "not part of the analysis. `pypillometry` keeps track of *why* a sample is\n", "bad, and a *mask policy* decides which samples the analyses leave out.\n", "\n", "This tutorial goes through these steps with a real recording. The\n", "concepts are explained in the user guide ([Missing data](../user_guide/missing_data.rst))." ] }, { "cell_type": "code", "execution_count": 1, "id": "76a76342", "metadata": { "execution": { "iopub.execute_input": "2026-11-09T20:27:35.084963Z", "iopub.status.busy": "2026-11-09T20:27:35.084769Z", "iopub.status.idle": "2026-11-09T20:27:36.189982Z", "shell.execute_reply": "2026-11-09T20:27:36.188736Z" } }, "outputs": [], "source": [ "import pypillometry as pp\n", "from pypillometry import u\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "pp.logging_set_level(\"WARNING\")" ] }, { "cell_type": "markdown", "id": "7771a5b3", "metadata": {}, "source": [ "We use the example dataset `rlmw_002`: a 20-minute recording of both eyes\n", "(pupil and gaze, 500 Hz) from a reinforcement-learning task. It is\n", "downloaded from OSF on first use (see [Example datasets](example_datasets.ipynb))." ] }, { "cell_type": "code", "execution_count": 2, "id": "44cb66d8", "metadata": { "execution": { "iopub.execute_input": "2026-11-09T20:27:36.194270Z", "iopub.status.busy": "2026-11-09T20:27:36.193763Z", "iopub.status.idle": "2026-11-09T20:27:38.291215Z", "shell.execute_reply": "2026-11-09T20:27:38.289626Z" } }, "outputs": [ { "data": { "text/html": [ "
EyeDataDict(vars=6,n=597301,shape=(597301,)): \n",
" left_x (float64): 655.6, 655.7, 655.0, 654.5, 655.0...\n",
" left_y (float64): 599.9, 598.9, 597.6, 597.8, 597.8...\n",
" left_pupil (float64): 1121.0, 1122.0, 1124.0, 1126.0, 1126.0...\n",
" right_x (float64): 773.1, 773.8, 774.8, 776.3, 776.7...\n",
" right_y (float64): 594.2, 593.9, 596.5, 597.4, 597.3...\n",
" right_pupil (float64): 949.0, 951.0, 956.0, 959.0, 960.0...\n",
"\n",
" | \n", " | NaN | \n", "blink | \n", "offscreen | \n", "divergence | \n", "masked (default) | \n", "masked (strict) | \n", "
|---|---|---|---|---|---|---|
| left_x | \n", "3.13 | \n", "0.00 | \n", "0.97 | \n", "0.97 | \n", "4.86 | \n", "4.86 | \n", "
| left_y | \n", "3.13 | \n", "0.00 | \n", "0.97 | \n", "0.97 | \n", "4.86 | \n", "4.86 | \n", "
| left_pupil | \n", "3.13 | \n", "5.84 | \n", "0.00 | \n", "0.00 | \n", "5.84 | \n", "5.84 | \n", "
| right_x | \n", "2.07 | \n", "0.00 | \n", "1.04 | \n", "0.97 | \n", "4.06 | \n", "4.06 | \n", "
| right_y | \n", "2.07 | \n", "0.00 | \n", "1.04 | \n", "0.97 | \n", "4.06 | \n", "4.06 | \n", "
| right_pupil | \n", "2.07 | \n", "4.08 | \n", "0.00 | \n", "0.00 | \n", "4.09 | \n", "4.09 | \n", "
| \n", " | NaN | \n", "blink | \n", "offscreen | \n", "divergence | \n", "excluded | \n", "masked (default) | \n", "masked (strict) | \n", "
|---|---|---|---|---|---|---|---|
| left_x | \n", "3.13 | \n", "0.00 | \n", "0.97 | \n", "0.97 | \n", "0.35 | \n", "5.13 | \n", "5.13 | \n", "
| left_y | \n", "3.13 | \n", "0.00 | \n", "0.97 | \n", "0.97 | \n", "0.35 | \n", "5.13 | \n", "5.13 | \n", "
| left_pupil | \n", "3.13 | \n", "5.84 | \n", "0.00 | \n", "0.00 | \n", "0.35 | \n", "6.19 | \n", "6.19 | \n", "
| right_x | \n", "2.07 | \n", "0.00 | \n", "1.04 | \n", "0.97 | \n", "0.35 | \n", "4.33 | \n", "4.33 | \n", "
| right_y | \n", "2.07 | \n", "0.00 | \n", "1.04 | \n", "0.97 | \n", "0.35 | \n", "4.33 | \n", "4.33 | \n", "
| right_pupil | \n", "2.07 | \n", "4.08 | \n", "0.00 | \n", "0.00 | \n", "0.35 | \n", "4.44 | \n", "4.44 | \n", "
| \n", " | NaN | \n", "blink | \n", "offscreen | \n", "divergence | \n", "excluded | \n", "interpolated | \n", "masked (default) | \n", "masked (strict) | \n", "
|---|---|---|---|---|---|---|---|---|
| left_x | \n", "3.13 | \n", "0.00 | \n", "0.97 | \n", "0.97 | \n", "0.35 | \n", "0.00 | \n", "5.13 | \n", "5.13 | \n", "
| left_y | \n", "3.13 | \n", "0.00 | \n", "0.97 | \n", "0.97 | \n", "0.35 | \n", "0.00 | \n", "5.13 | \n", "5.13 | \n", "
| left_pupil | \n", "0.00 | \n", "5.84 | \n", "0.00 | \n", "0.00 | \n", "0.35 | \n", "7.49 | \n", "0.35 | \n", "7.84 | \n", "
| right_x | \n", "2.07 | \n", "0.00 | \n", "1.04 | \n", "0.97 | \n", "0.35 | \n", "0.00 | \n", "4.33 | \n", "4.33 | \n", "
| right_y | \n", "2.07 | \n", "0.00 | \n", "1.04 | \n", "0.97 | \n", "0.35 | \n", "0.00 | \n", "4.33 | \n", "4.33 | \n", "
| right_pupil | \n", "0.00 | \n", "4.08 | \n", "0.00 | \n", "0.00 | \n", "0.35 | \n", "5.66 | \n", "0.35 | \n", "6.01 | \n", "
| \n", " | NaN | \n", "blink | \n", "offscreen | \n", "divergence | \n", "excluded | \n", "interpolated | \n", "user | \n", "masked (default) | \n", "masked (strict) | \n", "
|---|---|---|---|---|---|---|---|---|---|
| left_pupil | \n", "0.0 | \n", "5.84 | \n", "0.0 | \n", "0.0 | \n", "0.35 | \n", "7.49 | \n", "0.97 | \n", "1.04 | \n", "8.53 | \n", "
| right_pupil | \n", "0.0 | \n", "4.08 | \n", "0.0 | \n", "0.0 | \n", "0.35 | \n", "5.66 | \n", "1.04 | \n", "1.04 | \n", "6.70 | \n", "
Epochs 'S_' | 300 epochs, window (-500, 2500) ms\n", "
| \n", " | 500 Hz | \n", "50 Hz | \n", "
|---|---|---|
| NaN | \n", "0.00 | \n", "0.00 | \n", "
| blink | \n", "5.84 | \n", "5.78 | \n", "
| offscreen | \n", "0.00 | \n", "0.00 | \n", "
| divergence | \n", "0.00 | \n", "0.00 | \n", "
| excluded | \n", "0.35 | \n", "0.35 | \n", "
| interpolated | \n", "7.49 | \n", "7.45 | \n", "
| masked (default) | \n", "0.35 | \n", "0.35 | \n", "
| masked (strict) | \n", "7.84 | \n", "7.80 | \n", "
| \n", " | NaN | \n", "blink | \n", "offscreen | \n", "divergence | \n", "excluded | \n", "interpolated | \n", "masked (default) | \n", "masked (strict) | \n", "
|---|---|---|---|---|---|---|---|---|
| left_pupil | \n", "0.0 | \n", "5.84 | \n", "0.0 | \n", "0.0 | \n", "0.35 | \n", "7.49 | \n", "0.35 | \n", "7.84 | \n", "
| right_pupil | \n", "0.0 | \n", "4.08 | \n", "0.0 | \n", "0.0 | \n", "0.35 | \n", "5.66 | \n", "0.35 | \n", "6.01 | \n", "
| mean_pupil | \n", "0.0 | \n", "3.33 | \n", "0.0 | \n", "0.0 | \n", "0.35 | \n", "4.84 | \n", "0.35 | \n", "5.19 | \n", "
| \n", " | NaN | \n", "blink | \n", "offscreen | \n", "divergence | \n", "excluded | \n", "interpolated | \n", "masked (default) | \n", "masked (strict) | \n", "
|---|---|---|---|---|---|---|---|---|
| mean_pupil | \n", "0.0 | \n", "3.33 | \n", "0.0 | \n", "0.0 | \n", "0.35 | \n", "4.84 | \n", "0.35 | \n", "5.19 | \n", "
| \n", " | raw | \n", "preprocessed | \n", "
|---|---|---|
| pupil.left.missing_pct | \n", "3.138882 | \n", "0.0 | \n", "
| pupil.left.dropout_rate | \n", "15.423076 | \n", "0.0 | \n", "
| gaze.left.missing_pct | \n", "3.138882 | \n", "3.138882 | \n", "
| artifacts.left.pupil_masked_pct | \n", "NaN | \n", "0.0 | \n", "
| artifacts.left.gaze_masked_pct | \n", "NaN | \n", "4.795098 | \n", "
| duration | \n", "19.9 min | \n", "
| included | \n", "19.8 min | \n", "
| sampling rate | \n", "500 Hz | \n", "
| samples | \n", "597301 | \n", "
| events | \n", "1320 | \n", "
| time gaps | \n", "0 | \n", "
| \n", " | left | \n", "right | \n", "
|---|---|---|
| missing data | \n", "0.0% good | \n", "0.0% good | \n", "
| dropouts | \n", "0 /min good | \n", "0 /min good | \n", "
| longest dropout | \n", "0 s good | \n", "0 s good | \n", "
| noise | \n", "0.027% good | \n", "0.032% good | \n", "
| noise (SD) | \n", "0.283 | \n", "0.286 | \n", "
| jumps | \n", "0 /min good | \n", "0 /min good | \n", "
| \n", " | left | \n", "right | \n", "
|---|---|---|
| missing data | \n", "3.1% good | \n", "2.1% good | \n", "
| off-screen | \n", "1.0% good | \n", "1.1% good | \n", "
| precision | \n", "0.0185 deg good | \n", "0.0186 deg good | \n", "
| \n", " | left/right | \n", "
|---|---|
| pupil correlation | \n", "0.943 good | \n", "
| gaze disparity | \n", "0.418 deg good | \n", "
| \n", " | left | \n", "right | \n", "
|---|---|---|
| number | \n", "321 | \n", "307 | \n", "
| blink rate | \n", "16.2 /min good | \n", "15.5 /min good | \n", "
| mean duration | \n", "217 ms | \n", "159 ms | \n", "
| time in blinks | \n", "5.9% good | \n", "4.1% good | \n", "
| long (>500 ms) | \n", "14 | \n", "10 | \n", "
| \n", " | left | \n", "right | \n", "
|---|---|---|
| pupil masked | \n", "0.0% good | \n", "0.0% good | \n", "
| interpolated | \n", "7.5% good | \n", "5.7% good | \n", "
| gaze masked | \n", "4.8% good | \n", "4.0% good | \n", "
| masked off-screen | \n", "0.97% | \n", "1.0% | \n", "
| masked divergence | \n", "0.89% | \n", "0.89% | \n", "
| other artifacts | \n", "0.0% | \n", "0.0% | \n", "
| pupil.missing_pct | good <= 10, ok <= 20, poor <= 40, else bad |
| pupil.dropout_rate | good <= 30, ok <= 60, poor <= 150, else bad |
| pupil.longest_dropout | good <= 2, ok <= 5, poor <= 30, else bad |
| pupil.noise | good <= 0.5, ok <= 1, poor <= 2, else bad |
| pupil.jump_rate | good <= 1, ok <= 10, poor <= 100, else bad |
| gaze.missing_pct | good <= 10, ok <= 20, poor <= 40, else bad |
| gaze.dropout_rate | good <= 30, ok <= 60, poor <= 150, else bad |
| gaze.longest_dropout | good <= 2, ok <= 5, poor <= 30, else bad |
| gaze.offscreen_pct | good <= 5, ok <= 15, poor <= 30, else bad |
| gaze.precision_deg | good <= 0.05, ok <= 0.1, poor <= 0.2, else bad |
| binocular.pupil_correlation | good >= 0.9, ok >= 0.8, poor >= 0.6, else bad |
| binocular.gaze_disparity_deg | good <= 1, ok <= 2, poor <= 3, else bad |
| blinks.rate | good <= 20, ok <= 35, poor <= 70, else bad |
| blinks.time_pct | good <= 10, ok <= 20, poor <= 35, else bad |
| artifacts.pupil_masked_pct | good <= 10, ok <= 20, poor <= 35, else bad |
| artifacts.gaze_masked_pct | good <= 10, ok <= 20, poor <= 35, else bad |
| artifacts.interpolated_pct | good <= 10, ok <= 20, poor <= 30, else bad |
| calibration.mean_error | good <= 0.5, ok <= 1, poor <= 1.5, else bad |
| calibration.max_error | good <= 1, ok <= 1.5, poor <= 2, else bad |