Two Semester Design Project

For students using their HCI 523 research project as their “M2” phase for their capstone (see more context about why this option was created here), their milestones will be as detailed below. 

Everything is due on Monday by midnight Central Time, of the week listed. This timeline can be negotiated, and I’m happy to react if you turn in milestones early. The more you plan ahead, the better. The oral presentation date is determined by the Grad College deadlines for MS thesis oral presentations, which means it needs to happen by around Week 13.

Week 1 – 

Week 2 – M2 mini-report

Week 3 – 

Week 4 – 

Week 5 – M3 Design explorations

Week 6 – 

Week 7 – M3.5 Concept testing

Week 8 – 

Week 9 – M4 Prototype

Week 10 – 

Week 11 – M5 Usability Testing

Week 12 – 

Week 13 – Oral Presentation (if graduating this semester)

Week 14 – You’re usually done by now because of graduation deadlines.

Week 15 – 

Below are full details for each milestone:

M1: Skipped

M2: Done during the semester they take HCI 5220 or 5230. Their final report for HCI 5220/5230 is akin to the M2 report described on the main Design Project page, and must include:

• findings on users’ current needs and practices (for which a future technology might be designed), including how users see and describe the problem to be solved
• at least three tasks that users will want to accomplish, with a solution they will be designing
• what are the key steps of the tasks? (You might consider a narrative like a user scenario or a structured task analysis.)
• What is the task environment where users will perform these tasks in the real world? In the car? In an elevator? Only sitting down, only standing up? In a hurry?  (Mention physical environment as well as time constraints and culture if appropriate.)
• What breakdowns occur while users perform their tasks currently (that your system will address)?
• What do they see as their needs for better tools?
• What system(s) are currently available to accomplish these tasks?
• What are their strong and weak characteristics?
• Overall, what are the other design implications for your system, based on what you learned? (e.g. what aspects or details of your design will be most important to get right? Are there particular ways they should be designed? Or, what constraints will you need to consider?)
• Cite as least 3 academic references as part of the lit review, to demonstrate your ability to cite research on what is known already in this space.

M2: MINI-REPORT

This will be the first milestone due, during the semester the student is doing HCI 599. This report should draw on the insights from their HCI 5220/5230 research, but will need additional analytical thinking and product sense to answer (some additional online research may also be needed). It should be about 2-3 pages total.

PROBLEM DEFINITION (2 points)

• Describe in 1-2 sentences the user problem or need that exists, that you’ll be addressing in your design solution.
• Tell me your early thoughts on strategy for addressing the problem. Make sure this includes:
  • what form will your system take: app? website? VR application? robot? smartwatch? etc
  • what high-level mechanics are involved: e.g. a LLM? chatbot? is it a two-sided marketplace? does it make use of gamification? etc

See the main Design Project page’s M1 section for several examples of problem statements and strategy for addressing the problem.

USERS (1 point)

• Besides the user group you will focus on, what other stakeholders are there for your system (both users, as well as non-users who are impacted by your system)?

ANALYSIS (3 point)

Think about what you learned in your research, about what current systems are available, their strong and weak characteristics, and users’ desires for better tools. Then answer:

• What functionality should your system include to accommodate users’ tasks, but in a way that is differentiated (and better) than other alternatives?

MEASURES (2 points)

• Imagine that in the future, your system idea is actually built and being used by users. What measurable criteria would you want to use to evaluate the success of your system (e.g. number of users? usage metrics? self-reported satisfaction metrics, etc)?

BUSINESS PERSPECTIVE (2 points)

• If built by a for-profit company, how would your system make money? What requirements does this add, to the design? (Paid apps, subscriptions, or ads are some ways to make money. Selling the data collected by your app could be another. There could also be other ways.)
• What will be necessary for your system to be feasible, and how would you ensure this happens? (For instance, if your system idea depends on a certain stakeholder group to provide data or to agree to be part of it, how would you incentivize their participation?)

Total points: 10

M3: THE DESIGN

Now that you understand your users, it’s time for brainstorming to ideate on concepts that address the problem you identified.

Come up with 2-3 concepts, that you will get feedback on during M3.5: Concept Testing. These concepts should be created in low fidelity: either paper sketches of the interface, a simplified digital interface, or storyboard scenarios. The concepts should be articulated in just enough detail that a concept test participant can understand the key details of the solution. See a helpful article with more guidance on this, here.

It will be helpful if you can brainstorm with someone.

M3 Grading – 25 points

• Write a report or presentation with your concepts. A total of 25 points will be assigned to this portion of the project as detailed in the grading matrix below. The outline of your document should look something like this:

SUMMARY (2 points)

• Brief summary of the system, its users’ needs, and the tasks it will support. (Feel free to re-use content from before.)

DESIGN SPACE (4 points)

• What key dimensions did you explore as you did your ideation? (e.g. for a robotic lawn mower system, one dimension might be how automated the system is by default. Another dimension might be: how much to explain to users about what is happening.)

THE CONCEPTS (10 points)

• Describe the 2-3 solution concepts you came up with, and what user feedback/research insights they were based on.
• Describe what each concept optimizes for, in terms of the dimensions you explored (e.g. maybe one solution optimizes for explainability but at the expense of interpretability, or convenience vs. privacy). Describe at least one, per solution concept.
• Include the sketched and scanned or digitally created images of the solution concepts here.
• If you created a paper or digital sketches rather than storyboards, also give a usage scenario for each solution concept, that is written from the user’s perspective. This helps you avoid a presentation that’s just “Here’s my concepts!” and instead tells a story of how a real user would use them.

FUTURE TECHNOLOGIES & SOCIAL IMPLICATIONS (6 points)

• Comment on the ways that emerging technologies affect or inspire your design. This is the part of the course where you demonstrate your knowledge of emerging technologies and HCI-related technologies, e.g. that you might have learned about from Alex Stoytchev or Tony Townsend (approximately 1 page).
• Discuss any ethical issues that may arise from your design, as well as the social implications of your design if it were successful (approximately 1 page). E.g. with smart phones, some people worry about members of families not communicating as well at family dinners because they’re distracted, or you could talk about txt-msging undermining political regimes. If you were designing software for sending instant videos across the network, you could worry about the privacy of people in public who might get filmed.

RESEARCH-BASED (3 points)

Use academic references where appropriate in the report (perhaps ones you read in the HCI core courses) to justify your approach and to demonstrate your ability to cite research as the basis of your decisions.

M3.5: CONCEPT TESTING 

It’s now time to get feedback on your concepts, to see if your solutions are headed in the right direction, and which concept (or combination of concepts) you should narrow down to focus on for further refinement in the next phase.

The point of concept testing is to see whether users think your design solutions are good solutions for addressing the problem you are trying to solve, what changes are needed to address it better, and which concept users like best. Concept testing is not usability testing! Usability testing is about how usable a prototype is to use, and comes later in the process. One useful guide to concept testing can be found here; an example HCI paper from a concept testing study is here.

Test with at least 5 users. The users should meet the same minimum criteria you established in M2 to ensure they were part of your target user group (e.g. environmentalists). They ideally should also strongly identify with the problem you are trying to solve, though for this milestone it’s fine if some don’t identify with it as strongly (e.g. suburban homeowners who care a bit about borrowing tools, though it isn’t a huge need). You may recruit the same users you talked to during M2 for the evaluation if you wish. This is helpful, as they will already have context about the problem space you are aiming to address, and can give good detailed feedback about your solution.

During the test session

Remember that it can be intimidating to be a participant in a user study, so be extra nice. The user will be trying to please you, which could distort your data (i.e., prevent you from hearing critical feedback). Therefore, be very open and receptive to anything and everything the user may say or do in order to gain trust.

Present each concept, one by one, to the user. For the first one, start with giving a brief overview. Do not show how to do tasks with it, but give a general idea of the system to set some context. “This is an interface for a robotic lawn mower. You would see this touch screen inside your house and use it to control the mowing of your lawn. This image shows your lawn currently, and this map shows the position of your mower.” 

After the introduction, stop to get their initial impressions of the concept (in terms of the high-level solution, not the aesthetics or design details!). Ask for elaboration to understand if their sentiment is positive/negative/neutral, and why. Then, walk through the rest of the concept to show any further key details of the solution (“Let’s say you are going to be on vacation, and want to set the lawnmower to mow your yard while you are gone. This is the screen you would see.”). [For concept testing, it’s alright to show and explain what’s happening to users, rather than gauge their understanding, since we are not testing for usability yet.] If you want, you could ask them to describe what they see on the screen and what they would do next, to get a bit of usability feedback (e.g. on design elements that are confusing). Just remember that usability is not the focus of this round of testing.

After showing the details of each concept, interview the user to get their overall feedback. Before asking these questions, make sure they no longer have any confusion about what your concept is or how it works – you might have to explain how it is intended to work, if it was still unclear. This will allow you to understand whether your solution, once understood, is a good one (you can always improve the clarity of how it is communicated). Ask questions such as: 

• Now that you have seen the full concept for this system, how compelling is it to you? (Followup: explain why it is compelling or not compelling) 
• How would you describe this to a friend or family member?
• How well do you feel the system solves the problem of <problem you are aiming to address>? Why? 
• How would this concept improve (and/or not improve) the experience of [activity related to concept]?
• How would you use this concept in your daily life?
• What stands out to you as great about this concept?
• What stands out to you as concerning about this concept?
• What parts, if any, were confusing?
• If the system was built and ready to use, how eager would you be to use it? Why?
• What changes would you make, to make the system more useful for you?

After showing all 2-3 solution concepts, also make sure to ask:

• Which of the solutions did you like best? Why?

Think about privacy: assign your users fake IDs to use in your write up to protect their identity. Ask for their consent to take pictures and if you want to post any identifying photos of them on your website. 

Report

Write a document presenting your method and results. Email it to me as a PDF.  Aim for approximately 10 pages.

The outline of your document should look something like this:

• Brief summary of the system, its users, and the tasks it will support. (Feel free to re-use content from before.)
• Evaluation Details
  • Your users (demographics, background, etc) – work with at least 5 users
  • The questions you asked each participant during the session
  • Your testing environment (At a bookstore café? At the user’s home?)
• Results: Present your data
  • No need to discuss the data here; that’s the next section
  • Be sure to include how many users, some data from each user
  • Give quotes, summary of findings, and a summary of overarching themes
•  Discussion
  • What do you conclude based on your data? What do the data mean?
  • Was anything surprising?
  • What are the main implications for your design? Which one concept (or combination of parts of the concepts) do you intend to move forward with?
  • Reflect on your evaluation process: what was the hardest part of this milestone? What would you do differently when doing concept testing for a future system design project?

Grading for M5 – Evaluation Report

Summary: 2 points

Evaluation: 6 points

Results: 6 points

Discussion: 6 points

TOTAL: 20 points

M4: PROTOTYPE

Use the insights from concept testing to narrow down to one design solution. Then, create a high-fidelity digital prototype of it, including fleshing out the details of how at least one task would be done in it, end-to-end. Your choice of prototyping tool is up to you: e.g. Figma, HTML, Powerpoint. In M5, you’ll be showing this prototype to five (5) users.

Prototyping

When prototyping, remember that they key is developing the interface, not the back-end functionality. You can use a “Wizard of Oz” approach during testing to simulate how the system would respond.

M4 Grading – 15 points

Create a document or website or video, etc (anything digital) presenting your prototype.

Make sure you document:

• Functional Requirements (3 points)
  • This is where you’re telling us specifically what the system must do. You can frame it from the user perspective, e.g. “The user must be able to: 1) …. 2) ….” or in the traditional “The system shall 1) and shall 2)…” We’re not as concerned about the specific format, as long as it shows a more mature set of requirements that are driving your design.
• Description of prototype (include images or link to prototype) – 10 points
  • Its features and screens
  • The tasks it supports
  • The tasks / features it doesn’t support, and why that’s ok (just mention briefly)
• Reflection – 2 points
  • Reflect on the prototyping process: what was the hardest part of this milestone? What, if anything, would you do differently when prototyping for a future system design project?
  • What did you learn about your design, and about prototyping generally?
  • Write about a page.

Two Semester Design Project

M5: USABILITY TESTING

There are many ways to conduct evaluative testing. Think about what makes the most sense, for your project.

The point of the evaluation, however, should be to understand what’s working about your design, and what changes are needed to make it better.

Test with at least 5 users. Again, the users should meet the same minimum criteria you established in M2 to ensure they were part of your target user group, and you may recruit the same users you talked to during M2 and M3.5 if you wish.

During the test session

When you meet the user, explain what you’re doing. Explain that you’re testing the interface, not him or her. If there’s confusion, it’s bad design, not the user’s fault.

As in the concept testing, present the system to the user and give a brief overview. Do not show how to do the tasks, but give a general idea of the system to set some context. Then, introduce the first task scenario and show the starting point in the prototype for that scenario. Ask them what they would do next, and listen and observe what they do. If needed, advance the prototype through the steps for that task, to respond as the real system would (you might use a Wizard of Oz technique to simulate the system’s behavior). Also take notes, describing why and when the user made comments like, “This is cool!” and “What in the world is that?”

After all your task scenarios, interview the user to get their overall feedback. Ask questions such as:

• What did you like the most about using this system?
• What did you like the least?
• What changes would you make, to make it work better for you?

For data gathering, it will be most valuable if you present users with the exact same tasks and exact same prototype. Even if user #1 gets really dislikes some aspect of your system, resist the urge to change it before user #2, because you don’t know whether the other users will have the same feedback. Don’t change things unless the issue significantly disrupted the evaluation process or prevented you from getting important feedback on some other aspect of the system.

Think about privacy: assign your users fake IDs to use in your write up to protect their identity. Ask for their consent to take pictures and if you want to post any identifying photos of them on your website.

Report

Write a document presenting your method and results. Email it to me as a PDF. Aim for approximately 10 pages.

The outline of your document should look something like this:

• Brief summary of the system, its users, and the tasks it will support. (Feel free to re-use content from before.)
• Evaluation Details
  • Your users (demographics, background, etc) – work with at least 5 users
  • The task scenarios you gave to users
  • Your interview questions
  • Your testing environment (At a bookstore café? At the user’s home?)
• Results: Present your data
  • No need to discuss the data here; that’s the next section
  • Be sure to include how many users, some data from each user
  • For qualitative data: Include quotes and a summary of any overarching themes. Also summarize what was learned in terms of usability issues, categorizing them this way:
    • Severe; must fix
    • Major problem
    • Minor problem
    • Cosmetic problem (more related to prototype itself than system)
    • New feature request
  • For any quantitative data, give raw data since stats like average don’t make sense with a small sample size, e.g. “3 of 6 users said…” or a chart showing a bar for each of 6 users with their ratings.
• Discussion
  • What do you conclude based on your data? What do the data mean?
  • Was anything surprising?
  • What are the main implications for your design?
  • Reflect on your evaluation process: what was the hardest part of this milestone? What would you do differently when doing usability testing for a future system design project?

Grading for M5 – Evaluation Report

Summary: 2 points

Evaluation: 6 points

Results: 6 points

Discussion: 6 points

TOTAL: 20 points

ORAL PRESENTATION

Same as for one-semester capstone option. See details on main Design Project page. Just make sure to flag that you did the “two-semester capstone option” so the faculty committee has that context.