Solution for Multimodular Naturalistic Data Analysis

Hasson Lab, in collaboration with Psychology Professor Casey Lew-William and Princeton’s Baby Lab, has started the First 1000 Days project, uniting specialists in life, engineering & computer science. It focuses on studying what affects children’s brain development in their first three years. The Hasson Lab team created a method that optimizes the research and streamlines the analysis of 100 years’ worth of video data from the first thousand days of newborns in 15 different families.

One of the Client’s challenges was creating a cloud infrastructure capable of processing the naturalistic data at scale and supporting further studies. It was decided to re-architect the existing algorithms to:

• Facilitate integration of diverse datasets for faster data ingestion
• Streamline data collection and safeguard efficient management & secure storage of third-party service data
• Optimize data processing and analysis with the new framework for parsing, labeling, and extracting valuable insights from collected data
• Elevate the analytical capabilities of the system to extract valuable insights from the data

The first version of the Data Platform was originally built using the AWS technology stack. We made sure to keep the advantages of the existing framework and added a few important updates to ensure the system meets the Client’s current and future needs while remaining optimal in costs.

The improvements yielded great results – historical processing speed for one algorithm surged by 300%. Moreover, all the data is now stored in a single source of truth, which on the one hand enhances analytical accuracy, and on the other hand, results in a nearly 50% reduction in storage costs.

• Implemented AWS Glue to improve the efficiency of the core processing framework responsible for large video data volume handling
• Used CloudWatch, CloudTrail, and EventBridge to amplify the system’s monitoring capabilities with advanced alert mechanisms and reduce error response time within the data flow
• Leveraged AWS services to configure data security, integrity, and consistency according to the best data practices and data security standards

Since the Hasson Lab team operates in the sensitive healthcare sector and works with vulnerable naturalistic data, the main priority was to protect the user data they work with. This meant safeguarding their data storage complies with the data security standards, including the CCPA (California Consumer Privacy Act) and U.S. Privacy Act of 1974.

Apart from safeguarding compliance and mitigating data security risks, we were also working on building three main pillars of sound and efficient data operations: data observability, reliability, and durability. To achieve those, we performed the following steps:

• Extended data monitoring framework with additional metrics, notifications, and alerts to prevent data loss and minimize latency
• Implemented retry policies for components facing failures/errors and set up the rollback process for critical components to eliminate data inconsistencies, duplication, etc.
• Configured cross-region data replication to safeguard data integrity in cases of regional data deletion or unavailability
• Integrated the Object Lock to every file to prevent accidental or deliberate file deletions