The Heller School for Social Policy and Management
The Center for Youth and Communities

FIRST Longitudinal Study

Background

FIRST and Brandeis University’s Center for Youth and Communities are joining together to undertake a multi-year longitudinal study of the impacts on middle and high school-aged youth of participation in three of FIRST’s national after-school robotics programs: the FIRST® LEGO League, FIRST® Tech Challenge, and the FIRST® Robotics Competition.  The goals of the study are threefold:

  1. To examine the impacts of the three robotics programs on a core set of participant outcomes, including interest in STEM and STEM-related careers, college-going and completion, pursuit of STEM-related college majors and careers, and development of 21st century personal and workplace-related skills;
  2. To identify those elements of the program experience in FIRST, such as time in program, role on the team, or access to mentors,  that are critical to promoting positive academic, career, and developmental outcomes among participating youth; and
  3. To examine, to the extent possible, differences in experiences and impacts among key subpopulations, in particular young women and youth from low-income communities.

The study is being undertaken by the Center for Youth and Communities at Brandeis University’s Heller School for Social Policy and Management. Founded in 1983, the Center for Youth and Communities is one of the nation’s premier evaluation, research, policy, and capacity-building organizations dedicated to preparing young people for education, work and life.  The Principal Investigators for the study are Alan Melchior and Dr. Cathy Burack -- both are Associate Directors and Senior Fellows at the Center for Youth and Communities.  

Overview of the Study Design

The evaluation of the impacts of the FIRST robotics programs is designed as a multi-year longitudinal study that tracks a sample of participants from each of three FIRST programs (the FIRST® LEGO League (FLL®), FIRST® Tech Challenge (FTC®), and the FIRST® Robotics Competition (FRC®) over the five years following their entry into the program. The study will assess the impacts of the FIRST experience by comparing the experiences and outcomes of FIRST program participants with those of students in a matched comparison group of non-participating youth.  Approximately 1000 FIRST team members and 1500 comparison group members will be recruited for the study.

FIRST participants in the study will be new entrants into the FIRST program (“Rookie” participants) recruited from a representative sample of “veteran” FIRST teams (i.e., teams that have at least one year of prior FIRST participation). Comparison group students will be drawn from a pool of interested students at the schools where FIRST teams whose interests and involvement in STEM (Science, Technology, Engineering and Mathematics) are similar to those of program participants.

The major data collection for the longitudinal study will be a series of participant and comparison group surveys, administered on paper and online.  These include a baseline survey conducted at the beginning of the study, a follow-up survey at the end of the 2012-2013 program year (Spring 2013), and annual follow-up surveys thereafter (Spring 2014, 2015, 2016, 2017). Additional baseline information on study participants and their families will be collected through a parent survey. Information on the FIRST teams involved in the study will be collected through a end-of-season team leader survey. The study will also collect additional information on program experience and impacts through focus groups and interviews with selected team members and team leaders.

Key outcomes for the study include increased interest in science and technology; STEM-related course-taking in secondary school and college; pursuit of STEM-related majors in college; and interest in STEM-related careers. The study will also examine changes in life and workplace-related skills, such as teamwork, problem-solving, and communications.

The design for the study was developed by Brandeis in collaboration with FIRST staff, key FIRST stakeholders and a technical advisory group of outside researchers. The initial design and draft surveys for the study were developed with a core group of FIRST staff. They were then reviewed with a broader FIRST Advisory Group, which included regional directors, FIRST program partners, program alumni, and sponsors. Finally, the initial study design and materials were reviewed with the Technical Advisory Group. The final study design and surveys reflect the input from all the groups. The final study design and instruments were also reviewed and approved by the Brandeis University Institutional Review Board (IRB), which is responsible for approving all research at Brandeis involving human subjects.

For more information on the study, please feel free to email the evaluation team at Brandeis at FIRSTstudy@brandeis.edu or call the study office at 781-736-4890 (toll-free at 800-343-4705, extension 64890).