@iamjessklein I wonder if this moderated user test might from 2014 [1] might be helpful as we think about the design of the user test we will be conducting for this intervention.

The above comes from these test findings [2], which I think will also be useful as we think about the design and priority of other features.

1. https://www.youtube.com/watch?v=NI_QLlh01ZY&list=UUQBrlvKHv6UjLL83BNOSdgw
2. https://www.mediawiki.org/wiki/Flow/Moderated_Testing,_November,_2014:_talk_pages_and_Flow

I started to draft out the test protocol.

On Friday @ppelberg and I discussed creating a clickable prototype to get the work out faster.

We should be preparing to test this (and any other prototype) on our target wikis. This is of course dependent on identifying the wikis and engaging with their communities.

In T236921#5631418, @iamjessklein wrote:

I started to draft out the test protocol.

Looking good, @iamjessklein. I've added some comments and questions to the script here: Usability test of Reply v1.0.

Note: for the time being, I focused my feedback on the testing goals and demographics. I figure once we have those in a good place, we can talk about contents of the tests.

I launched the tests on usertesting.com. The idea is that these tests will be identical to the control test conditions T239175 except that they are using the new link with the v1 prototype on it.

We will run tests on usertesting.com and on-wiki as well (in the near future) which will hopefully provide us with participants equipped with a range of experience levels to examine.

cc/ @ppelberg @Whatamidoing-WMF

Updating task description

• ADDED: "Confirm prototype is working as it should on beta labs"
• ADDED: Finalize on-wiki user testing script
• ADDED: Create test talk page on beta labs

I ran a test on usertesting.com on December 11, 2019 - [PT 1-5] The test recruited 5 random, technically - advanced web users. Participants were directed to an article page set up on the prototype server . The detailed findings can be found on limited access test log.

• 5 tests were conducted
• 2 participants were male ;3 participants were female
• 5 participants were desktop web users
• None of the participants had edited a Discussion page before, but all had familiarity with Wikipedia and some participants had previously edited an article.

Overall, the test indicated that more junior/new contributors had a positive experience adding a reply to a thread on the Discussion page (compared to T239175). The time that it took for participants to identify how to reply and then, in turn, write and publish the reply was cut in half. That said, there is still a lot more that we can do to help contributors parse comments, make sense of the "revision history" page and understand what the various pieces of displayed personal information mean (IP address, time stamp etc.) Testers used words such as "straight-forward," "no-problem whatsoever," and "really easy" to describe their experience replying to a discussion thread, which indicates their confidence in their actions. Testers used words such as "frustrated," "confused," and " daunting" to describe their experience on the Revision History page, which indicates that they were challenged by this experience.

Many testers were unpleasantly surprised to see their IP address revealed.

Here are the findings:

Identifying section heading
✅5/5 participants identified the section heading

Replying to comment
✅ 5/5 participants identified how to reply to and write a comment
✏️ 5/5 participants used proper indentation or signing practices due to the automated features
✏️The amount of time that testers took to identify where to click to reply was greatly reduced.

Publishing reply
✅ 5/5 participants published their reply
✏️"that was not hard at all"- PT-5

Locating published reply on article page
✅ 5/5 participants located their reply
✏️ Several participants took several minutes to find their reply on the page
✏️"i appreciate this. Without nesting, comments would be a mess." - PT-1
✏️"I do think that I published it because it has my date and time and IP address" - PT- 3
✏️""I don't like that my IP address is showing." - PT - 3
✏️"I wish there was more of a visual distinction so you can kind of see that *someone* is saying something." PT - 2

Cancelling out of task mid-edit
✅ 5/5 participants cancelled out of task mid-edit
✏️ 4/5 participants cancelled out using the "cancel" button (1/5 just tapped out and it still worked)
✏️ 1 participant had difficulty orienting themselves after canceling out of an edit. "The only thing painful about this experience was trying to find the comment I cancelled. I got really confused there and it sort of threw off my train of thought. If this were to happen to me outside of the test, I would have probably left the website! Other than that I enjoyed being able to have a discussion." - PT-3

Viewing page history tab
✅ 5/5 participants found the page history tab successfully
✏️ "The view history tab was a little confusing, I understood it a little bit (Enough to know what was what)." - PT-4
✏️"so every time someone comments, it's a revision?! crazy. What am I looking at right now?" PT-2
✏️"On the main page it was really easy to find. Like any normal, standard form or website it was at the end and timestamped. When I went to the Revision History tab it was not as obvious. It was weird." - PT - 4

Locating reply/summary in page history
✅ 5/5 participants were able to locate their reply in the page history
❌ 1/5 participants who located their reply did it with ease
✏️ "The section doesn't say anything about what I wrote there and there's nothing to indicate it's me." - PT-1
✏️""I don't know what's going on" re: revision history" - PT - 3
✏️"I assume this is an IP, timestamp and date" -PT - 4
✏️"it's a little frustrating - I have to go with the timestamp and IP address" PT-1

These findings are helpful – thank you for synthesizing them, @iamjessklein.

Two questions for you:

• 1. Are there improvements to this workflow, and the test itself, you think we ought to consider making? A few that came to mind after watching the tests and reviewing your analysis below.
• 2. "3/5 participants wrote an edit summary." <-- what is meant here considering the prototype does not give contributors the opportunity to customize/affect the edit summary?

Improvement to workflow

• Make it easier for contributors to differentiate between distinct comments.
• Make the "Reply" call to action easier to discover.

Improvement to test

• I wonder if there would be value in asking test participants what they understand the purpose of the [talk] page to be upon first viewing it
  • Reason: I think it would be helpful for us to know what attributes/components people expect to see in a discussion space, such as a talk page
• Have test participants log in
  • Reason: I am curious to know whether testers are able to more easily locate their comments in the "History" page if a clear piece of personally identifiable information accompanies their edits.

Other observations

• One person did not seem to be aware that talk pages already exist on Wikipedia, remarking, "I think it would be nice to have discussions in Wikipedia." | Source
• One person found the design of the "History" page discouraging, saying, "With the view history tab it's a little hard to read...it looks like, 'Oh, I'm not supposed to be here.'" | Source
  • The "...not supposed to be here." remark is interesting. I wonder if this is a pervasive feeling among newer contributors and intensified their reluctance to contribute.

Task description update(s):

• Updating "Done" section

Two questions for you:

• 1. Are there improvements to this workflow, and the test itself, you think we ought to consider making? A few that came to mind after watching the tests and reviewing your analysis below.
• 2. "3/5 participants wrote an edit summary." <-- what is meant here considering the prototype does not give contributors the opportunity to customize/affect the edit summary?

1. • I agree with you @ppelberg, I would like to do the test completely logged in
   • I'm also interested in running tests on different namespaces
   • It feels like the test should be done on different platforms
   • I agree with you that there's value in seeing if people can make sense of the page without support from us.

2. I noted this here anecdotally, I just thought that it was interesting to see if people actually read the instructions and did it, which would give some context to their contribution (this will hopefully be a better experience in comparison with automated summaries).

Re: your other observation - yes I'd say almost all testers had a similar general impression of the history page. I tried to communicate that in the high level summary section

"Testers used words such as "straight-forward," "no-problem whatsoever," and "really easy" to describe their experience replying to a discussion thread, which indicates their confidence in their actions. Testers used words such as "frustrated," "confused," and " daunting" to describe their experience on the Revision History page, which indicates that they were challenged by this experience."

• I just re-read and edited out that comment about the edit summary comment, that was left over from a prior phab ticket copy that I had incorrectly pasted over.

@ppelberg I would add to our on-wiki report a recommendation for investigate options for rethinking the design of the revision history page at some point in the future as this testing round (and the 2 control tests) showed that people struggled to make sense of that space. It's important because this is the last touchpoint in the process of adding a contribution for many experienced contributors (and possibly the first touch point for experienced content consumers).