Chapter 5 — Letting the Insights Guide Us
Overview
For the past two weeks, Team Super Cool Scholars has mostly been working on conducting contextual inquiries with reviewers and continuing to synthesize our research data. We have also been discussing the insights we are starting to generate and using them to inform our next steps moving forward. In addition, we are testing our conceptual prototype with admissions reviewers and preparing for our upcoming spring presentation.
Conceptual Prototype Testing With Reviewers
Our team has been working on contacting and scheduling conceptual prototype testing sessions with reviewers and has currently conducted three interviews. As described in Chapter 4, as reviewers go through application 1, they take notes in the spreadsheet that we had designed to mimic the ApplyGrad system’s note-taking area, while another team member highlights information they deem as important in the third application. For application 2, the reviewer uses Google Docs features to highlight and comment on the document, while the notetaker continues to take notes and highlight important information. The third application will be the application that has been marked up by the notetaker, which the reviewer can then use to make additional comments.
So far, we are seeing that reviewers value having key information being drawn out from application materials, as it helps them quickly find important information about an applicant. The information highlighted also helps reviewers identify areas applicants mention that tie into what their program is looking for (areas like focus statements, vision for their future, etc.).
As we wrap up conceptual prototype testing with reviewers, we plan to synthesize our findings in order to discover the answers to our research questions and gauge whether the direction we are currently going will be beneficial to their needs.
Contextual Inquiry With Reviewers
We recently finished conducting contextual inquiry interviews with four reviewers from four different departments of varying sizes. Afterward, we held an interpretation session to synthesize our qualitative data and identified the following findings:
- Reviewers value taking notes during the review process because it allows them to remember each applicant’s story.
- Reviewers turn to external tools for additional functionality that the ApplyGrad system does not have.
- Reviewers look for certain keywords or phrases specific to their program in application materials to gauge if the applicant is a fit for their program.
- Reviewers want to maintain consistency throughout the review process to ensure fairness.
These findings support our conceptual prototype of a highlighting and note-taking system that tests the perceived user need for a better review system. As we continue to test our prototype with users, we will be able to have a better understanding of what would be beneficial to them in the review process.
Participant Recruitment
One of the biggest challenges that we have commonly come across during the course of this project is recruiting participants for our research. We understand that administrators and reviewers are extremely busy and can only spare a little time. Thus, we are coming up with workarounds to address this issue. One of the ways that we are tackling this is by contacting Ph.D. students who review master's applications for their respective programs. Some departments have Ph.D. students on the admissions committee, and so we are contacting them to get their input on our conceptual prototype.
Deepening Our Understanding Of ML
This week, we focused on exploring existing tools that we can use to integrate machine learning solutions into our design solutions. Since we are only designing but not implementing our solutions, instead of actually building a working machine learning solution, we tried to understand the capabilities and limitations of different ML tools. The goal is to deepen our understanding of state-of-the-art ML technology to make more informed design decisions in our project.
Specifically, we investigated the Watson Studio by IBM. We tested out their Natural Language Understanding service to find out possibilities of using natural language processing in parsing some application materials for reviewers, including statement of purposes and recommendation letters. We learned about the process of uploading datasets, labeling datasets, and training and deploying the ML model using Watson Studio. As we become more familiar with different ML technology, we are more capable of designing better ML solutions in the future.
Consolidating Our Research
As the semester slowly comes to an end, our team has been working to gather all the data we have been collecting and patterns we have been observing in order to connect our research and findings to one another. To do this, we are developing documents that detail all the insights we have generated for each research method, accompanied by evidence in the form of findings and quotes. This will allow us to be able to easily identify support for the direction that we recommend in our research report and spring presentation.
Next Steps
Walk the Wall
One activity we plan to do within the next week is ‘Walking the Wall’. In order to connect all our research to one another and develop deeper insights, we plan to post our models and research materials on a board and discuss all that we have done so far in order to come to a common understanding. This process will allow us to engage with and immerse ourselves in our findings, generate stronger insights, and continue thinking of the different design opportunities that we can use to address our end-users’ needs.
As we finish up the spring semester, our spring presentation is quickly approaching and Team Super Cool Scholars is diligently preparing to present our research and insights. Our faculty mentors, as well as other faculty within the MHCI program, have been extremely helpful in guiding us and providing us with numerous resources to ensure that we deliver an engaging and informative presentation, and we are excited to share our journey.
Until next time,
Team Super Cool Scholars
