We are happy to share with our students that a detailed report about the SJSU Plus Spring Pilot is now available.
In late 2012, our CEO, Sebastian Thrun, visited the National Science Foundation, and in a conversation with Don Millard the idea was born to create an independent study for our upcoming SJSU Spring Pilot. Shortly thereafter, the NSF funded SJSU for such a study, which hired the RP Group, an independent group of researchers. The report has now been made public by the SJSU Principal Investigator, Elaine Collins, and is available here.
The report is worth reading. It contains a rich and insightful analysis of the student body, the outcomes, what went well, and what didn’t. As expressed in a previous blog post from Sebastian, many of the issues that didn’t work were fixed for the Summer Pilot; and we recognize that more work remains.
Udacity does take issue with one paragraph from the report:
“Another limitation resulted from the difficulty of obtaining Udacity instructional resources use and support service access data in a format that could be analyzed by statistical software. It took several months to receive this data in a flat-file format.”
A first data planning meeting with the RP Group took place on March 28. Between April 3 and May 7, all parties worked to define data formatting using test data in preparation for when actual data would be available. The RP Group’s first data request (with details of the type of data required) was issued to Udacity on May 31, 2013, and later modified on June 3 as the final raw data request. That same day, SJSU started setting up a secure file transfer server. Udacity was notified on June 13 that the server was ready. We then uploaded a first data set on the same day. Udacity then spent six days compiling tutoring data and uploaded that data to the server on June 19. On June 28, we submitted phone log data that we had placed to individual students. This completed the data compilation phase. According to our records, on July 25, the RP Group issued a request to reformat previously submitted data. Final data submission was on July 26.
Throughout this process, the time spent between data requests and delivery was due to tools that had to be built to extract the data from our servers into a format that RP Group could analyze. Udacity had never before worked with an outside group to assess its data. To us, the process of working with the RP Group and SJSU was constructive, professional, and expedient. In fact, Udacity prioritized these data requests over most other activities in the company.
The report moves on to say:
“While most of the data were provided by the end of the Spring 2013 semester, clarifications, corrections and data transformations had to be made for many weeks thereafter, including resolving accuracy questions that arose once the analysis of the Udacity platform data began.”
During the time of collaboration, a number of conversations took place in which researchers from the RP Group inquired about details of the pilot and the interpretation of the raw data. Udacity has no record of any inquiry or discussion pertaining to the accuracy of the data. All data we sent was consistent and factual throughout the entire process: we shared our raw data without any alteration.
Udacity aspires to discuss all findings — good and bad. We are grateful to the NSF, the RP Group, and SJSU for conducting this study. The learnings from the data are immense, and our team has been working long nights and weekends to learn from the pilot, improve our platform, and provide the students with a great learning experience.
