It seems sort of inevitable that if a tool is made easier to use there will be more unproductive things done with it (simply because it will be easier). Perhaps it would be better to, e.g.,

• measure if such edits are being successfully reverted;
• consider methods of actively reducing the rate of such edits.

Additionally, reversions alone might not be sufficient as a metric, since some talk page comments might just end up going unnoticed (and thus would not be included in the measurement). Perhaps a sample of all edits by new users could be taken and then manually categorized.

It might also be interesting to analyze this issue for some wikis which don't regulate the use of talk pages as strictly, since other wiki communities might prefer to allow or prioritize "unproductive" discussion (for whatever reason; e.g. they want to increase traffic/readers, they installed MediaWiki just for the discussion interface).

I don't have any objection per-se to watching the suggested guardrail metrics, however I don't think they will be particularly useful.

The metric I am interested in is Talking-with-zero-article-edits. It's is a negative metric, and it shouldn't be included within the "contributions" metric.

I guess you could also look at the number of blocks for talk-only users. I would expect at least part of any problems to manifest there.

Task description update

• Updating the task description to make it more explicit that it is important to monitor the activity of contributors, across experience levels, as part of this work.

We could check the number of unreverted and non-reverting (non-bot?) edits to non-talk pages. We hope those would be stable or increase. (That is, fewer good-faith bad edits made, because it was easier to ask on the talk page than to screw up the article, and fewer reversions of those edits, because they weren't made in the first place.)

More activity on talk pages is not necessarily bad. This is a good exchange:

== Needs update ==
Hi, I just registered my account, but it still won't let me edit.  Can someone post that Michael Jackson just died?  I saw it in Google News.  [[User:Newbie]]

:Hi, Newbie.  I found three newspapers that said the same thing, so I added it.  You can't edit the page because we semi-protected the article to keep vandals away – nothing about you, of course.  [[User:Mid-century mod]]
::Thanks!  [[User:Newbie]]

Getting that "closed loop" in the newcomer's reply might be more likely if there's a quick [reply] button for the newcomer to use than if the newcomer has to edit the document.

We experienced editors are totally habituated to how this works, but newcomers find it quite strange and off-putting to add a comment in document-editing mode, with the ability to change anyone else's comments. They seem to expect "moderators" to exist, and for the moderators to be able to remove or adjust other people's comments, but they don't expect any random newbie like themselves to be able to edit anyone else's comments. So with current tools, we might get the talk-page request from the newbie, and we would get the experienced editor's reply, but we might not get that last reply. The new tools might increase the number of thanks/acknowledgement edits to close the communication loop.

I don't think that will be limited to newbies, either. If I have a choice between a quick [reply] and letting everyone know that I was grateful for the help, or hunting through the history of a talk page and clicking the thank button (so that only the one person knows about my gratitude), then I may use Special:Thanks less and [reply] more. This could have positive effects overall, if people feel like their work is appreciated more and that more people around them are friendly. So the volume of Special:Thanks use is another thing that could be measured, although I'm not sure whether increased/stable/decreased results should be considered good or bad.