Chapter 1 — Preparing to Apply
As the fields of machine learning and artificial intelligence continue to grow, so does the popularity of Computer Science graduate programs. Carnegie Mellon’s School of Computer Science is no exception. But with this popularity comes an increasing number of applicants; in the 2019–2020 application cycle, over 10,000 applications were submitted to SCS alone. While this is an exciting opportunity for the department, how can the graduate application process be improved to help address the growing workload for admissions reviewers as well as envision the next generation of graduate applications systems?
Team Super Cool Scholars is excited to team up with CMU’s School of Computer Science to propose both short- and long-term innovations to the graduate application system, ApplyGrad, in order to address the evolving needs of the graduate admissions process.
Meet the Super Cool Scholars:
Anna Yuan — Design Lead
Hello! I am a 2020 CMU grad and a current MHCI student. I have an interdisciplinary background in software engineering & UX design and am passionate about learning & exploring how to leverage advanced technologies to create better user experiences. In my free time, I do arts & baking for fun and love playing board games.
Emily Zou — Project Manager
Hi everyone! I’m a 2020 college grad from UVA who majored in Computer Science and technology. I have some experience working in startups and from directing a hackathon before. I’m pumped to work with a great team to make the admissions process, which we have all been through, better than it has ever been.
Lia Slaton — Research Lead/Client Communicator Lead
I majored in Philosophy-Neuroscience-Psychology with minors in English and computer science at Washington University in St. Louis. For the past two years, I served as a college adviser to help low-income, first-generation, and underrepresented high school students navigate the college-going process.
Ugochi Osita-Ogbonnaya — Documentation/Quality Assurance Lead
Hello! I recently graduated from UCR and have a background in biological sciences and cognitive science research. I am interested in contributing to inclusive and accessible user experiences that address people’s needs. In my free time, I like to read, write stories, and watch dramas.
Yuwen Lu — Technical Lead
Hey, I’m Yuwen! I recently graduated with a bachelor's in Software Engineering. I’m interested in understanding user intentions and creating value for users and societies through the design of technical solutions.
Meet Our Faculty Advisors
Geoff Kauffman
Professor Kauffman works in the Human-Computer Interaction and Language and Technology Institute at Carnegie Mellon. He studies the psychological and social impact of narratives, games, and computer-mediated interactions.
Rita Lee
Rita Lee teaches design at the college level for nearly a decade. She holds a master’s degree in public management and a bachelor’s degree in graphic design, both of which are from Carnegie Mellon.
Context of Our Research
Unlike past years of MHCI work, our team started out our research in the midst of a global pandemic. This unique context requires adapting research strategies to the remote environment. We start off our journey with background research, meeting clients, and working with our team through Zoom. We are also using industry-standard tools such as Google Doc, Figma, and Mural to collaborate in real-time with teammates and others to make designs and aggregate research. Most of our team management is done through Slack and Trello, too.
Background Research
While working to understand the scope of the project, we started conducting background research on the different graduate programs housed in the School of Computer Science. We looked through websites and past research from past capstone teams. We also asked many questions to our faculty advisors and started preparing materials for our clients to help us with idea generation and defining the scope of the project.
We wanted to gain a better understanding of the admissions process and requirements for each department in order to identify areas of overlap and divergence in how they interact with ApplyGrad. Even just after a week, we realized there are a lot of variations in the different programs that we should account for. After preliminary research, we divided our teams to conduct research into each of the 23 programs across 9 departments. Each member was paired with a partner to support as either a moderator or facilitator. This allows us to figure out how to address each program’s unique needs in a short amount of time.
Kickoff
While preparing plans for research, our team also prepared activities for our clients so we can answer the new questions that we generated from background research. We used a remote research guide to come up with kick-off ideas that would help align our vision with our clients and generate ideas.
Our initial kickoff meeting was a blast! The student team and representatives from SCS sat down over Zoom to get to know each other before participating in the ‘What’s On Your Radar’ activity using Mural. Our objective with this activity was to discover each individual’s understanding of the scope of our project, as well as how we each prioritized different areas of concern (primary, secondary, tertiary).
We divided the board into four quadrants, each one covering a different subject within our project summary; each participant then wrote down and categorized 2 comments/questions under the relevant section. After this, we discussed these comments and concerns. This allowed us to discover each individual’s focus and what issues had the highest priority as we began to explore the project space.
For our second meeting, we used Mural to develop a Creative Matrix activity. We wanted to start thinking about and discussing the different subject areas that relate to ApplyGrad and their various areas of opportunities given the information we are currently gathering on the admissions process.
We also sent our two clients in two separate breakout rooms because from the first session, we realized that the two different departments that they came from both have valuable insights that could help us identify the differences in pain points between departments.
Our kickoff meetings with representatives from SCS provided us with numerous insights regarding the scope of the project and our main stakeholders. We learned that we needed to narrow our focus to a specific area and discussed focusing on professional master’s programs since they make up the majority of master’s programs that SCS offers. One of the biggest areas we discussed was working on how to improve the efficiency of the admissions review process. This discussion aligned with a piece of advice that was given to us during our last meeting with our faculty advisors: to focus on the general process and types of challenges that the various departmental admissions committees have to address.
Primary Research Plan
While each member of the team has interacted with ApplyGrad at least once, it has been from an applicant’s point of view rather than a reviewer’s. Thus, we are currently planning to conduct interviews with the administrators of different master’s programs to learn about the admissions process and the structure of the ApplyGrad system. Our team is also currently working to identify how we can use surveys to gather relevant data.
One of the biggest obstacles for our team during this sprint is to get access to the ApplyGrad system due to FERPA concerns, which prevents us from looking at sensitive data from applicants. Currently, work is being done on getting us access to the system through either a sandbox account or the real one but with just dummy data entered. Still, access might not come in time, and we must move on.
We have decided to utilize the contextual inquiry method for our interviews. Administrators will be asked to walk through an ‘application’ in ApplyGrad and share their thoughts aloud with us, choosing whether they want to share their screen or not. This method will allow us to see how the administrator interacts with the system and help us learn how they evaluate applications. Our Capstone faculty advisors also recommended exploring non-invasive research methods such as diary studies, so we have plans to use this method if necessary.
With this method, we seek to better understand how they use ApplyGrad, what issues and frustrations they have with the current system, and how we can work to address their needs. Using the insights gathered from our primary research, we plan to create visual models of each program’s review process to see the similarities and differences between the structure of different programs.
Currently, we are scheduling interviews and constructing detailed plans for conducting research. We have a long list of contacts to get through and a lot to decide on how to refine our approach as we get more information. We are using a breadth-first approach, where we go for the top-level administrators of departments first before going down to interview the directors and then the admissions committee. We decided to do this because we learned from our clients that administrators generally know the most information about their department. They could help us pinpoint areas to look into to gain an accurate understanding of the department as a whole.
Next Steps
Pretotype
After conducting research with administrators, directors, and admission committee members of different SCS graduate programs, we plan to develop pretotypes to help us test out our early ideas with our target users. The purpose of the pretotypes is to guide our designs to meet both the admissions reviewers’ needs from the start, as well as generate possible solution paths to explore as we move forward.
Competitive Analysis
Another idea we have that we want to investigate for our secondary research is doing a competitive analysis. We are working on conducting a competitive analysis of other admissions platforms and perhaps other recruitment management platforms in order to analyze the structure of their system and to identify their strengths and weaknesses. We will specifically be looking at application sites like Slate, which have been used before in some departments for admissions processes, in order to figure out what ideas we can take for our own future designs.
Machine Learning and Automation
Most importantly, we are going to do a deep dive into looking at how machine learning can be utilized in the admissions process. We are planning to conduct a literature review to better familiarize ourselves with and inform our understanding of the machine learning space. We already have preliminary information on some past projects that did not go through. In addition, we plan to consult with machine learning and automation experts. Advancing our knowledge of its possibilities and limitations will help us identify areas where automation could be useful.
Some resources we will be consulting include:
A visual introduction to machine learning
Inherent Trade-Offs in the Fair Determination of Risk Scores
Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices
While we will be focusing on the reviewer’s side of the system, we understand that any improvements made to the system will improve the experience for all stakeholders. Our team is excited to explore the different areas and possibilities of this project and we can’t wait to see where it will take us.
