The Hasson Lab has significantly progressed with their studies on naturalistic data but ran into a technical issue. The volumes of data were massive, and their existing data processing system built using dedicated data center capabilities couldn’t process it at scale, which wasn’t acceptable for the kind of science the Client wanted to pursue.
Sigma Software’s data engineers helped the Client cope with their technology challenge by re-architecting a data processing system in AWS Cloud. As a result, we created a solution that not only addresses Hasson Lab’s needs but also serves as a blueprint for building a larger number of datasets in the social behavioral and cognitive sciences.
Hasson Lab, in collaboration with Psychology Professor Casey Lew-William and Princeton’s Baby Lab, has started the First 1000 Days project, uniting specialists in life, engineering & computer science. It focuses on studying what affects children’s brain development in their first three years. The Hasson Lab team created a method that optimizes the research and streamlines the analysis of 100 years’ worth of video data from the first thousand days of newborns in 15 different families.
One of the Client’s challenges was creating an efficient storing and processing data system capable of processing the naturalistic data at scale and supporting further studies. The team therefore decided to re-architect the existing algorithms to:
Sigma Software’s data engineers took a cloud migration and re-architecture approach rather than rebuilding from scratch, aligning with the AWS Migration Acceleration Program’s 7R practices and Well-Architectured Framework.
The first version of the Data Platform was originally built using Python scripts, orchestrated by Bash and scheduled by cron jobs. The team preserved the core framework functionality and business rules, and re-architected the core algorithms onto modern, managed AWS services to meet the Client’s current and future needs while remaining cost-optimal.
The improvements yielded great results – historical processing speed for one algorithm surged by 300%. Moreover, all the data is now stored in a single source of truth, which on the one hand enhances analytical accuracy, and on the other hand, results in a nearly 50% reduction in storage costs.
Based on the agreed roadmap, the solution was delivered incrementally, with each phase completed, validated, and handed over before proceeding to the next, ensuring controlled modernization and continuous business value:
Pre-migration
Before migration timeline was established, we assessed the existing architecture, data assets, and processing algorithms. Based on that, our experts defined the phased migration plan, designed the target AWS architecture for scalability and cost efficiency, as well as established the security and compliance readiness baseline required for handling sensitive research data to ensure the re-platform could proceed with minimal disruption to ongoing research.
Phased migration
The migration followed a phased delivery approach to minimize risk while aligning with the Client’s objectives and the selected Re-architect strategy. The migration scope was thoroughly assessed and documented, with Sigma Software and the Client conducting a series of workshops to define implementation phases, technical dependencies, and measurable outcomes.
Post-migration
Following implementation, we provided continuous managed operations and support, with a dedicated team of five full-time engineers supporting the platform since March 2023. This included operating and extending the monitoring, retry/rollback & cross-region replication safeguards, maintaining security and compliance controls, and continuously optimizing cost and performance as the dataset and research workloads grow.
Since the Hasson Lab team operates in the sensitive healthcare sector and works with vulnerable naturalistic data, the main priority was to protect the user data they work with. This meant safeguarding their data storage complies with the data security standards, including the CCPA (California Consumer Privacy Act) and U.S. Privacy Act of 1974.
Apart from safeguarding compliance and mitigating data security risks, we were also working on building three main pillars of sound and efficient data operations: data observability, reliability, and durability. To achieve those, we performed the following steps:
Liat Hasenfratz
Co-director
Hasson Lab
Sigma has provided us with the machinery to access the 1kD dataset and unlock its value. We expect it will serve us and generations of researchers - from various fields such as computer science, psychology, linguistics, sociology, economics, and neuroscience. We also believe that the batch system solutions provided by Sigma Software will serve as roadmaps for the creation of more naturalistic datasets in the social behavioral and cognitive sciences.
Stanislav Samko
Account and Program Manager
Sigma Software
This work is of exceptional scientific importance, addressing challenges of a magnitude yet to be resolved. It's a privilege to collaborate with such amazing people.
