External Application Data Recorder

External Application Data Recorder 

The External Application Data Recorder allows you to record, save and synchronize sensor data while running external VR applications from any source (SteamVR game, Oculus, Web application, etc.). This can use dekstop video recorders, such as the Nvidia Shadow Play, or Windows Game Bar recording feature to also have the option of recording and saving a video that can then be viewed back and matched up with the data recording.

List of some of the data that can be recorded (will be saved to the data folder, unless a different directory is chosen):

• Facial Expression values (Meta Quest Pro, HTC Face Tracker)

• Eye gaze rotation

• Fixations  and Saccades

• Head orientation (needs to be manually aligned if not at same origin point)

• Pupil Diameter (Omnicept, Vive Pro Eye, Pupil Labs)

• Eye Openness Value (Omnicept, Vive Pro Eye)

• Heart Rate (Omnicept)

• Cognitive Load (Omnicept)

• Interactions using the chosen event trigger 

• TIme Stamp

• Gaze mat of eye rotation

• Video Recording

• All Biopac Acqknowledge physiological data if connecting to Biopac 

  • Video recording can be imported to Acqknowledge to see how it matches up with physiological data

• Gaze point overlay on video recording (currently may take some manual alignment depending on the field of view of the headset)

Some limitations:

• Only will work with PC based applications (android applications running on stand alone not supported)

• Gaze point position is not going to be accurate in  the Z axis, as it's only hitting a collide box but not objects in the scene a user is actually looking at

• Head orientation (Meta Quest Pro) would need to be manually aligned if not using the same origin starting point

• Not able to tag regions or objects of interest

• For headsets that go through SteamVR (Omnicept, Vive Focus 3, Vive Pro Eye) you will not get head orientation

• To better synchronize head position if using a controller for navigation, will have to modify the SightLab vizconnect to match the external application's navigation controls

Note to record videos you need to install the pyautogui python library using the Package Manager, as well as install Tkinter from the Vizard installer

If getting error " A supported game is required" message, make sure you are using your dedicated graphics card and not the onboard one

This can also be modified to use other screen recording options

To Run:

1. Start Data_Recorder_External_Application script

2. Start External Application

3. Make Data Recorder screen active and press spacebar

4. After timer value that is set, the video recording  and data recording will start. Alt-tab and go back to the external application within this time

5. Press spacebar to end recording

6. Quit External Application

7. Video recording will be saved in your UserName- Videos folder and data will be in 'data'

8. Can also leverage the Plotly visualization example to view a walk path and gaze path (requires the Plotly Python library)

9. Face Tracking Data can be viewed in utils/data and visualized using the facial_expressions_over_time script

10. If using Biopac Acqknowledge can open the video to see the synchronization with the physiological data. See here for how to do that

11. To view a replay run DataRecorderReplay.py that will show the gaze point and visualizations on a virtual screen that is tied to the user's head

To Configure for a specific recorder (such as Windows Game Bar), set the shortcuts for starting the application and recording in the software (default for the Data Recorder is Alt+Z to open the recorder and Alt+F9 to start/stop recording. This can also be changed in the Data Application code. 

List of configurable options using Data_Recorder_Config.py:

https://www.youtube.com/watch?v=xQZlbbrDUM8