{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Importing Data Example\n", "\n", "In order to import data into `pypillometry`, we have to load the data from the source using other packages and then wrap it into `PupilData` objects. \n", "\n", "Here we will show and example where we translate a file recorded in Eyelinks EDF-format to a file readable by `pandas.read_table()`.\n", "\n", "First, we import the needed modules." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import sys, os\n", "sys.path.insert(0,\"..\") # this is not needed if you have installed pypillometry\n", "import pypillometry as pp\n", "import pandas as pd\n", "import numpy as np\n", "import pylab as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, we use data recorded with an Eyelink-eyetracker. These eyetrackers store the files in binary files with the extension `.edf`. Some information about this file-format is [here](http://download.sr-support.com/dispdoc/page25.html). We use a command-line utility released by Eyelink to convert this proprietory format into a more easily read `.asc` file that is a whitespace-separated plain-text format. The converter, `edf2asc` is a program that can be downloaded for different platforms from the [Eyelink support forum](https://www.sr-research.com/support/). There is a GUI-based program for windows and command-line programs for linux and mac. Binaries of the command-line tools for linux and mac are included in `pypillometry` under [this link](https://github.com/ihrke/pypillometry/tree/master/external). \n", "\n", "On linux, we would call these programs on an example edf-file twice as follows. " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "EDF2ASC: EyeLink EDF file -> ASCII (text) file translator\n", "EDF2ASC version 3.1 MacOS X Jul 13 2010 \n", "(c)1995-2009 by SR Research, last modified Jul 13 2010\n", "\n", "processing file ../data/test.edf \n", "=======================Preamble of file ../data/test.edf=======================\n", "| DATE: Fri Feb 14 08:48:33 2020 |\n", "| TYPE: EDF_FILE BINARY EVENT SAMPLE TAGGED |\n", "| VERSION: EYELINK II 1 |\n", "| SOURCE: EYELINK CL |\n", "| EYELINK II CL v6.12 Feb 1 2018 (EyeLink Portable Duo) |\n", "| CAMERA: EyeLink USBCAM Version 1.01 |\n", "| SERIAL NUMBER: CLU-DAC49 |\n", "| CAMERA_CONFIG: DAC49200.SCD |\n", "| Psychopy GC demo |\n", "===============================================================================\n", "\n", "Converted successfully: 0 events, 1245363 samples, 6 blocks.\n", "\n", "EDF2ASC: EyeLink EDF file -> ASCII (text) file translator\n", "EDF2ASC version 3.1 MacOS X Jul 13 2010 \n", "(c)1995-2009 by SR Research, last modified Jul 13 2010\n", "\n", "processing file ../data/test.edf \n", "=======================Preamble of file ../data/test.edf=======================\n", "| DATE: Fri Feb 14 08:48:33 2020 |\n", "| TYPE: EDF_FILE BINARY EVENT SAMPLE TAGGED |\n", "| VERSION: EYELINK II 1 |\n", "| SOURCE: EYELINK CL |\n", "| EYELINK II CL v6.12 Feb 1 2018 (EyeLink Portable Duo) |\n", "| CAMERA: EyeLink USBCAM Version 1.01 |\n", "| SERIAL NUMBER: CLU-DAC49 |\n", "| CAMERA_CONFIG: DAC49200.SCD |\n", "| Psychopy GC demo |\n", "===============================================================================\n", "\n", "Converted successfully: 36371 events, 0 samples, 6 blocks.\n" ] } ], "source": [ "!../external/edf2asc-mac -y -s ../data/test.edf ../data/test_samples.asc\n", "!../external/edf2asc-mac -y -e ../data/test.edf ../data/test_events.asc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This results in two files, one containing all the samples and one all the recorded events." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "fname_samples=\"../data/test_samples.asc\"\n", "fname_events=\"../data/test_events.asc\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The samples-files contains a large table containing the timestamp, x/y-coordinates for the eyeposition and pupil-area for both the left and the right eye. Here are the first few rows of this file:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3385900\t 817.3\t 345.2\t 1707.0\t 860.6\t 375.2\t 1738.0\t.....\n", "3385902\t 817.0\t 343.5\t 1706.0\t 860.7\t 375.9\t 1739.0\t.....\n", "3385904\t 816.7\t 341.6\t 1705.0\t 861.2\t 376.6\t 1739.0\t.....\n", "3385906\t 816.7\t 340.4\t 1706.0\t 861.7\t 376.8\t 1740.0\t.....\n", "3385908\t 816.7\t 340.2\t 1707.0\t 861.6\t 376.9\t 1742.0\t.....\n", "3385910\t 816.8\t 340.2\t 1708.0\t 861.1\t 377.1\t 1743.0\t.....\n", "3385912\t 816.9\t 340.9\t 1708.0\t 860.7\t 377.5\t 1744.0\t.....\n", "3385914\t 816.1\t 342.1\t 1710.0\t 861.1\t 378.7\t 1745.0\t.....\n", "3385916\t 815.2\t 343.2\t 1712.0\t 862.5\t 380.0\t 1746.0\t.....\n", "3385918\t 814.4\t 343.6\t 1713.0\t 863.9\t 380.7\t 1747.0\t.....\n" ] } ], "source": [ "!head $fname_samples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can easily read this file using `pandas.read_csv()`." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | time | \n", "left_x | \n", "left_y | \n", "left_p | \n", "right_x | \n", "right_y | \n", "right_p | \n", "
---|---|---|---|---|---|---|---|
0 | \n", "3385900 | \n", "817.3 | \n", "345.2 | \n", "1707.0 | \n", "860.6 | \n", "375.2 | \n", "1738.0 | \n", "
1 | \n", "3385902 | \n", "817.0 | \n", "343.5 | \n", "1706.0 | \n", "860.7 | \n", "375.9 | \n", "1739.0 | \n", "
2 | \n", "3385904 | \n", "816.7 | \n", "341.6 | \n", "1705.0 | \n", "861.2 | \n", "376.6 | \n", "1739.0 | \n", "
3 | \n", "3385906 | \n", "816.7 | \n", "340.4 | \n", "1706.0 | \n", "861.7 | \n", "376.8 | \n", "1740.0 | \n", "
4 | \n", "3385908 | \n", "816.7 | \n", "340.2 | \n", "1707.0 | \n", "861.6 | \n", "376.9 | \n", "1742.0 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
1245358 | \n", "5923060 | \n", ". | \n", ". | \n", "0.0 | \n", ". | \n", ". | \n", "0.0 | \n", "
1245359 | \n", "5923062 | \n", ". | \n", ". | \n", "0.0 | \n", ". | \n", ". | \n", "0.0 | \n", "
1245360 | \n", "5923064 | \n", ". | \n", ". | \n", "0.0 | \n", ". | \n", ". | \n", "0.0 | \n", "
1245361 | \n", "5923066 | \n", ". | \n", ". | \n", "0.0 | \n", ". | \n", ". | \n", "0.0 | \n", "
1245362 | \n", "5923068 | \n", ". | \n", ". | \n", "0.0 | \n", ". | \n", ". | \n", "0.0 | \n", "
1245363 rows × 7 columns
\n", "\n", " | 0 | \n", "1 | \n", "2 | \n", "3 | \n", "4 | \n", "5 | \n", "6 | \n", "7 | \n", "8 | \n", "
---|---|---|---|---|---|---|---|---|---|
0 | \n", "MSG | \n", "3387245 | \n", "C_GW_1_1_UD_UD | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
1 | \n", "MSG | \n", "3390421 | \n", "F_GW_1_1_10_0 | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
2 | \n", "MSG | \n", "3392759 | \n", "C_NW_1_2_UD_UD | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
3 | \n", "MSG | \n", "3394293 | \n", "R_NW_1_2_UD_UD | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
4 | \n", "MSG | \n", "3395952 | \n", "F_NW_1_2_-1_0 | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
1065 | \n", "MSG | \n", "5893078 | \n", "V_UD_UD_16_UD_UD | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
1066 | \n", "MSG | \n", "5899076 | \n", "V_UD_UD_17_UD_UD | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
1067 | \n", "MSG | \n", "5905073 | \n", "V_UD_UD_18_UD_UD | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
1068 | \n", "MSG | \n", "5911072 | \n", "V_UD_UD_19_UD_UD | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
1069 | \n", "MSG | \n", "5917071 | \n", "V_UD_UD_20_UD_UD | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
1070 rows × 9 columns
\n", "\n", " | 0 | \n", "1 | \n", "2 | \n", "3 | \n", "4 | \n", "5 | \n", "6 | \n", "7 | \n", "8 | \n", "
---|---|---|---|---|---|---|---|---|---|
209 | \n", "MSG | \n", "3900393 | \n", "RECCFG | \n", "CR | \n", "500 | \n", "2 | \n", "1 | \n", "LR | \n", "None | \n", "
211 | \n", "MSG | \n", "3900393 | \n", "GAZE_COORDS | \n", "0.00 | \n", "0.00 | \n", "1919.00 | \n", "1079.00 | \n", "None | \n", "None | \n", "
212 | \n", "MSG | \n", "3900393 | \n", "THRESHOLDS | \n", "L | \n", "56 | \n", "231 | \n", "R | \n", "66 | \n", "239 | \n", "
213 | \n", "MSG | \n", "3900393 | \n", "ELCL_WINDOW_SIZES | \n", "176 | \n", "188 | \n", "0 | \n", "0 | \n", "None | \n", "None | \n", "
215 | \n", "MSG | \n", "3900393 | \n", "ELCL_PROC | \n", "CENTROID | \n", "(3) | \n", "None | \n", "None | \n", "None | \n", "None | \n", "
\n", " | time | \n", "event | \n", "
---|---|---|
0 | \n", "3387245 | \n", "C_GW_1_1_UD_UD | \n", "
1 | \n", "3390421 | \n", "F_GW_1_1_10_0 | \n", "
2 | \n", "3392759 | \n", "C_NW_1_2_UD_UD | \n", "
3 | \n", "3394293 | \n", "R_NW_1_2_UD_UD | \n", "
4 | \n", "3395952 | \n", "F_NW_1_2_-1_0 | \n", "
... | \n", "... | \n", "... | \n", "
1065 | \n", "5893078 | \n", "V_UD_UD_16_UD_UD | \n", "
1066 | \n", "5899076 | \n", "V_UD_UD_17_UD_UD | \n", "
1067 | \n", "5905073 | \n", "V_UD_UD_18_UD_UD | \n", "
1068 | \n", "5911072 | \n", "V_UD_UD_19_UD_UD | \n", "
1069 | \n", "5917071 | \n", "V_UD_UD_20_UD_UD | \n", "
1035 rows × 2 columns
\n", "