Public Policy Initiative Learning Points

Introduction

Campus Ambassadors in Washington, D.C. (January 2011).

The Public Policy Initiative has been one of the most successful outreach projects ever undertaken by the Wikimedia Foundation. We offer our sincere thanks to the Stanton Foundation for funding for a bold idea: to improve Wikipedia's public policy articles on a large scale and to transform the way Wikipedia is thought of in academia.

Today, we see the seeds of our success. Forty professors at 24 universities taught 47 classes affiliated with the Public Policy Initiative. All told, more than 800 students contributed 8.8 million characters of content to the Wikipedia article namespace – the equivalent of 5,800 printed pages. And even better, the content students added was of high quality: Public policy articles that students worked on improved an average of 64 percent. Most important, because of the Stanton Foundation's commitment to establishing a support system for professors who want to use Wikipedia assignments in their classroom, professors in public policy and other disciplines around the world will continue to assign students to edit Wikipedia articles, improving article quality for years to come.

Up until this point, many of our projects to recruit subject-matter experts like academics produced only short bursts of activity on Wikipedia. The Public Policy Initiative has changed that. Not only are we improving the quality of Wikipedia every term, we are addressing some of the big questions facing the Wikimedia movement: recruiting new editors, closing the gender gap, and welcoming new users are all key facets of our strategic plan for the next five years. The Public Policy Initiative started making inroads on these movement-wide problems.

This document shares an outline of our learning points from the four major roles in this project (instructor, Campus Ambassador, Online Ambassador, and Regional Ambassador), course design findings, research on student improvements to Wikipedia, and other assorted learnings. We offer this report for the greater Wikipedia community to learn from the activities of the Public Policy Initiative over the last 18 months.

The Model

At its most basic, the model is such: Professors assign their students to contribute to Wikipedia in place of a traditional course assignment. Trained Campus Ambassadors teach Wikipedia basics, such as how to create a user account, and Online Ambassadors offer support for students on-wiki. Midway through the Public Policy Initiative, we developed a group of Regional Ambassadors, who work on recruiting professors and Campus Ambassadors in their regions within the United States. This section describes the learnings about each of these groups.

Instructors

Participating instructors are professors and other teachers who assign their students contribute to Wikipedia as part of their classwork. With the assistance of Ambassadors, they design Wikipedia assignments that are appropriate for their classes and also mesh with the Wikipedia Education Program's system for organizing and keeping track of participants. An instructor typically leads a pod of Campus and Online Ambassadors, who work together to help students contribute successfully to Wikipedia.

In some cases, instructors are also trained as Campus Ambassadors and provide Wikipedia training to students themselves.

Instructors who meet more stringent requirements for their assignment designs — major assignments where every student in the class participates and substantial improvements of Wikipedia content are expected — are designated as "Wikipedia Teaching Fellows."

Typical activities

Creating a Wikipedia account and trying out the basics of editing
Reading and signing the Memorandum of Understanding to join the Wikipedia Education Program
Recommending university staff, faculty, students, or others who might want to be Campus Ambassadors
Creating a course page (with help from Wikipedia Ambassadors, if requested)
Designing a detailed assignment plan and timeline (with a Sample Syllabus and other assignment design advice as a starting point, if desired)
Coordinating communication within their "pod" of Campus and Online Ambassadors
Explaining the Wikipedia assignment to students
Explaining to students the role of Campus Ambassadors and Online Ambassadors, and how to contact them for help
Keeping track of students' Wikipedia contributions as the assignment progresses
Providing advice and feedback to students on how to research and write effectively in the subject area of the class (as they would in a traditional assignment)
Evaluating and grading students' contributions
Communicating with the Regional Ambassador for their region about course progress and problems that arise
Providing feedback to the Wikipedia Education Program about how it can be improved

Motivations

Instructors come to the Wikipedia Education Program for a pretty wide range of reasons. The ones who fit best with the program tend to have a few prime motivations in common:

to provide a better learning experience for students
to explore new teaching methods
to improve Wikipedia as a useful public resource

In particular, the chance for students to write for a live audience and to get community feedback on their work are powerful motivators for many participating instructors.

Professional development is another common (although not universal) motivating factor for instructors: the designation as a Wikipedia Teaching Fellow, the chance to try a cutting edge way of teaching (and in some cases to conduct research about its effectiveness), and the chance (through students) to serve their scholarly community by improving Wikipedia coverage in their area of expertise. Each of these can help build an instructor's CV and professional reputation, although such benefits are usually only secondary motivations for instructors who thrive in the Wikipedia Education Program.

Recruitment

Some effective methods for recruiting instructors include:

Attending conferences where many instructors are attending — especially by setting up a high-profile vendor table and/or giving a presentation about the Wikipedia Education Program and its relevance to the attendees
Personal and professional connections from other program participants
Press — especially university newspapers, education-focused news venues and high-profile coverage by prominent news organizations

It's important to be selective when recruiting instructors; there are many professors who can be convinced of the benefits of trying out Wikipedia assignments, but those that end up the most successful with it tend to need little convincing beyond hearing about the program. A strong focus on the benefits for students is a key indicator of an instructor's fit with the Wikipedia Education Program.

Training

As of September 2011, the Wikipedia Education Program has not yet implemented a training program for instructors. However, a training program to get instructors started and teach the basics of Wikipedia editing and Wikipedia's policy and culture will be a key addition in the future. Many participating instructors have specifically suggested a training program, and others have had initial difficulty due to lack of knowledge of editing basics and Wikipedia policy.

Training for instructors should cover:

Account creation, basic editing and talk page communication
Wikipedia's policy and culture, especially Neutral Point of View, No Original Research, Verifiability, and Summary Style
The roles of Regional Ambassadors, Campus Ambassadors and Online Ambassadors
How to get help
Course page creation

Issues, Resolutions, Recommendations

Level of commitment

The Wikipedia Education Program is, so far, designed to support instructors doing major Wikipedia assignments, and therefore requires a high level of commitment from instructors to justify the volunteer and other resources used to support them. However, successful Wikipedia assignments can be very small as well; finding better ways to support instructors with smaller or more casual plans in one potential area for improvement.

Opportunities for program research

Most instructors are pre-tenure and need to have peer-reviewed publications to achieve career goals. Some may be interested in doing program-related research while participating in the Wikipedia Education Program. Encouraging and collaborating on such research can be a great opportunity to simultaneously further publicize the program, test alternative approaches and find ways the program can be improved, and provide benefit to the instructors.

Insufficient community feedback

Many instructors find that the editing community provides less feedback on students' work than they expected. The likelihood of getting adequate community feedback can be improved by teaching instructors and students about how to explicit seek feedback from Wikipedia editors, and by coordinating closely with Online Ambassadors to figure out when feedback is most helpful and how to obtain it. Students can also peer review each others' work, which is a strongly recommended component for Wikipedia assignments.

It's also important not to build up too high expectations for feedback from the Wikipedia community (which will vary widely based on the topics students choose and how much they interact with the community).

Campus Ambassadors

The primary role of a Campus Ambassador is to be the friendly face of Wikipedia. Campus Ambassadors provide in-person help to students and advice to instructors, and they come to classrooms and/or lab sessions to teach the basics of Wikipedia. They ease the Wikipedia learning curve for both students and instructors.

Campus Ambassadors may also help to grow the Wikipedia Education Program by recruiting new professors and Ambassadors, and by organizing Wikipedia Student Clubs.

Typical activities

Going through Campus Ambassador training
Meeting with instructors and Online Ambassadors before class begins to plan roles
Delivering in-class presentations on Wikipedia basics as well as specific aspects of Wikipedia relevant to students' assignments
Running lab sessions to walk students through early editing activities and key milestones
Holding office hours to answer questions and help students who are having trouble with Wikipedia
Posting updates to a Regional Ambassador about how their class is going

Motivations

Campus Ambassadors join primarily because they like and want to support Wikipedia. In particular, most are interested in:

Increasing Wikipedia's credibility within the education system
Developing Wikipedia as a teaching tool

Other motivations, such as improving content on Wikipedia and helping new editors also play a role — especially for experienced Wikipedians — but the connections to academia are the most widely shared motivations.

Campus Ambassadors are motivated to continue primarily because they enjoy teaching and working with students. The more they get to work with students, the more they tend to enjoy the Campus Ambassador role.

Recruitment

Anyone who supports Wikipedia and has good people skills can make an effective Campus Ambassador — whether an experienced Wikipedian, an occasional or new contributor, someone who has so far only been a reader. In fact, a Campus Ambassador pair that combines an experienced Wikipedian with a relative newcomer is often highly effective; the experienced Wikipedian will have a deeper knowledge of how Wikipedia works, and a newcomer will more easily relate to the perspective of students who are new to editing. We found no difference in the quality of Campus Ambassadors among experienced Wikipedians and new recruits; teaching skills and time to give to students were better indicators of success in the Campus Ambassador role. Recruiting experienced Wikipedians from within the community can be done through typical wiki-related channels and geonotices, while newcomers can be recruited through a combination of seeking recommendations from instructors and other Ambassadors, and reaching out to individuals and groups by email and other means.

Recruiting from the Wikipedia community

Geonotices, which show up on the watchlist of users within a specified geographical region, are a good way to start. In some cases, especially in dense population centers, a geonotice will attract multiple potential Campus Ambassadors.

Many Wikipedians indicate where they live or which universities they are associated with on their userpages. Searching userpages and user categories for potential Campus Ambassadors and leaving messages on their talk pages is a good followup to Geonotices.

Posting messages to central discussion pages (such as a Village Pump), news pages (such as The Signpost) and Wikimedia mailing lists can also be used to find additional interested Wikipedians. General messages may be less effective than targeted messages like geonotices and talk page messages, but they can reach users who would not otherwise know about the opportunity to join the Wikipedia Education Program.

Recruiting through recommendations

Seeking individual recommendations from Campus Ambassadors and participating instructors is a highly effective way to recruit Campus Ambassadors who are not already Wikipedia editors. Instructors can often suggest students, fellow instructors, and/or other university staff who may be interested in becoming Campus Ambassadors. Students who were assigned to edit Wikipedia in a previous course (and enjoyed doing so) are an especially good source of new Campus Ambassadors. In some cases, university staff such as librarians or 'teaching and learning' staff may be able to incorporate the Campus Ambassador role into their jobs.

Recruiting through cold emailing

When other methods aren't enough to secure Campus Ambassadors to support classes, "cold emailing" can be useful. Sending recruitment messages to individual librarians, faculty, and other university staff whose work involves online resources will often succeed, especially if messages are well-tailored to the recipients.

For recruiting students, reaching out to student groups and organizations that connect somehow to Wikipedia or volunteerism can be useful. If Students for Free Culture or a similar organization is active at a university, reaching out individually to leaders of that organization is a good idea.

Training

New Campus Ambassadors — both experienced editors and newcomers — go through a training program that covers how the Wikipedia Education Program works, what is expected of all ambassadors including the ambassador principles, what ambassadors can expect, an overview of Wikipedia culture and community, (for newcomers) the basics of editing Wikipedia, methods and advice for working with instructors and students, how to present specific aspects of Wikipedia to students, where to get help, and successful Wikipedia assignments.

For academic terms in 2010 and 2011, most Campus Ambassadors attended multi-day in-person training events. Those who could not attend the in-person training events, or who joined the program after the training events were over, were trained informally over video calls individually or in small groups. Beginning in 2012, online training events for Campus Ambassadors are expected to become the standard as the program expands beyond public policy classes into new fields of study, requiring a more scalable solution to train Ambassadors.

Contents of Campus Ambassador training

The training program was designed around a modular approach with four key modules that may be presented in the classroom. Campus Ambassadors may teach all or part of each module based on their consultation with the professor they are supporting. Module content includes:

"Introduction to Wikipedia", which covers the basics of what Wikipedia is, how popular it is, and how it works
"Wikipedia Literacy", which covers the Wikipedia user interface, the structure articles, advice for choosing articles to work on, and the basics of working with the community
"Wikipedia Essentials", which covers account creation and the basic mechanics of editing, using talk pages, adding references, and uploading images
"Working on Wikipedia", which covers the principles of verifiability and reliable sources, how to start articles, how to use the "Did You Know" process, and how to get feedback on articles

The overall Campus Ambassadors training experience so far is "learn by doing." With plenty of hands-on activities, Campus Ambassador trainees have the opportunity to work together to learn the content, present various topic areas, and get feedback. These activities also help Campus Ambassadors get to know one another and form a community. Through panel and informal discussions, non-Wikipedian ambassadors learn from Wikipedians about the community, cultural norms and "rules of the road." Additionally, ambassadors learn how to support professors in creating appropriate Wikipedia assignments.

Issues, Resolutions and Recommendations

Communication overload

Especially for Campus Ambassadors who are new to the Wikipedia community, it's easy to become overwhelmed with the volume of program-related email and other communication, especially if discussion go into arcane or technical Wikipedia topics. To avoid turning Campus Ambassadors off from key communication channels, it's important to separate out low-volume high-importance announcement from discussion, classroom reports, and other information. For the United States Education Program, there is an announcement list that all ambassadors are expected to be on, as well as a Google Group for discussion that is optional.

Reluctance to participate in on-wiki discussions

Campus Ambassadors are much less likely to participate in on-wiki discussions (or polls and votes) about the program or specific issues related to it than Online Ambassadors, because many of them prefer other forms of communication. Providing alternative ways to have their voice heard one way to resolve this, for example, by providing off-wiki poll forms (and invitations to participate) and discussion venues. Helping those Campus Ambassadors who are interested in becoming editors themselves also helps.

Not enough time in class

Many Campus Ambassadors report that they don't have enough time in class to teach the students as much about Wikipedia as they would like to, or work as much directly with students as they would like to. One way for Campus Ambassadors to have more time with students is to hold "office hours" and/or lab sessions outside of class where students can get help and instruction from Campus Ambassadors.

Online Ambassadors

Motivation of Online and Campus Ambassadors (Source: Wikimedia Foundation survey data, September 2011).

The primary role of an Online Ambassador is to support instructors and students online — primarily through communication on Wikipedia itself, but also by email, IRC or other means. Online Ambassadors provide expertise on how Wikipedia works, and in many cases, interest and experience with contributing in the particular subject area students are focused on. They answer questions from instructors, students and Campus Ambassadors, model good editor behavior and communication practices, provide advice to students as they complete their assignments, and help students to get feedback on their contributions.

Typical activities

Answering students' questions posted to course talk pages or their own talk page
Coordinating by email and other means with the instructor and Campus Ambassadors for the class(es) they are working with
Answering questions from students and instructors in a Wikipedia Education Program-linked IRC channel
Providing feedback on students' contributions
Requesting feedback on students' contributions from other editors' with relevant expertise
Helping students submit their contributions to review processes such as "Did You Know" and "Good Article Nominations"
Evaluating new Online Ambassador applicants
Maintaining Wikipedia project pages related to the Wikipedia Education Program

Motivations

Online Ambassadors are motivated to join primarily by opportunities for:

helping newcomers get started
improving the quality of Wikipedia
increasing Wikipedia's credibility in academia.

Some already focus on helping newcomers in other ways, and see the Wikipedia Education Program as another effective way to do that. Many are concerned about the dearth of new editors that has been highlighted by recent research on 'editor decline'. Others join because of the opportunity to work specifically with students contributing in a subject area they care about.

Other motivations that play a role include developing Wikipedia as a teaching tool, and simply the chance to try something new.

Online Ambassadors typically continue in the role if they find it a good use of their time relative to other wiki tasks they might otherwise take on.

Recruitment

Qualifications

Online Ambassadors should be experienced editors, familiar with all the basics of editing and able to find help from other editors when faced with technical or other questions beyond their ken. They should also be friendly and deal well with newcomers in a consistently civil and helpful way. Desirable qualities include:

experience contributing high quality content
experience reviewing articles
technical knowledge of templates, wiki formatting, and software tools for editors
subject matter expertise that is relevant to an individual class
experience welcoming and helping newcomers

However, anyone with a solid basic understanding of Wikipedia's core policies and the mechanics of editing can make an effective Online Ambassador.

Finding applicants

By far, the most effective way to recruit Online Ambassadors is by inviting potentially qualified editors individually with messages (on their talk pages, for example) — the more specific to the recipient, the better. Some ways of finding potentially qualified editors include:

Searching through lists of Wikipedians, such as the most active Wikipedians or Wikipedians with the most "Did You Know" credits
Searching through the recent contributors to key locations where experienced editors help newcomers, such as the Help Desk
Finding the participants lists of WikiProjects focused on the same topic areas as participating classes
Identifying active helpers in the help channels on IRC

Additional methods that can also be useful for attracting Online Ambassadors include:

Watchlist notices that appear on every user's watchlist
Calls for applicants at central news and discussion venues on Wikipedia, such as a Village Pump or The Signpost.

Application process

New would-be Online Ambassadors are vetted by the community, either through an on-wiki process with an open discussion of whether the applicant has the experience and disposition to make a good ambassador, or by sumbitting their application by email for private discussion and evaluation by a small group of trusted ambassadors.

In either case, applicants answer an application questionnaire to begin to orient applicants towards the expectations for the Online Ambassador role, and help those evaluating the application make sense of the applicant's editing history.

With either application method (on-wiki or email), the decision of whether to accept the applicant as an Online Ambassador is based on consensus among the discussants.

Orientation

Being experienced Wikipedians, new Online Ambassadors are capable of helping students and instructors with most of the questions and help requests they have. We opted not to provide any sort of training for Online Ambassadors during the Public Policy Initiative, but confusion arose about what specifically they were supposed to do, so we recommend a short orientation in the future. We expect this orientation will concisely introduce the structure of the Wikipedia Education Program to new Online Ambassadors, and give a clear picture of how they fit in, how they will be expected to work with Campus Ambassadors and instructors within "pods", and what resources are available for them to use in their role.

Key information for new Online Ambassadors includes:

Instructions for adding oneself to the official list of active ambassadors and creating an ambassador profile
The basic structure of the Wikipedia Education Program and the "pod" for an individual class
The role of the Online Ambassador within a pod
The kinds of activities expected of an Online Ambassador
How to find the Memorandum of Understanding and sign it to join a pod
The Wikipedia Ambassador Principles that ambassadors are expected to follow
The key communication channels for ambassadors, including the requirement to sign up for the ambassadors announcement email list

Issues, Resolutions, Recommendations

Burnout

One significant issue for the Online Ambassador role is burnout: ambassadors become overwhelmed or too stressed out by the role, especially if demands come all at once (as students scramble to meet deadlines) rather than spaced out over many weeks. High volume of ambassador-related communication also contributes to burnout and demotivation of Online Ambassadors. And as with other active contributors, within a large group of Online Ambassadors, some will take a break or retire from Wikipedia because of wiki conflicts outside of the ambassador role.

Key ways to minimize burnout include:

Making sure that directly helping newcomers and/or working on article content make up a high proportion of Online Ambassador activities. Helping newcomers and improving Wikipedia are strong motivating factors for Online Ambassadors.
Showing appreciation for good work by Online Ambassadors, and encouraging the instructors and Campus Ambassadors with whom they work to do the same
Limiting the amount of traffic in all communication channels through which important ambassador-related announcements are sent (including email lists and mass-delivered talk page messages)
Having Online Ambassadors work with classes in subject areas they are interested in

Non-responsiveness from students and instructors

One of the biggest demotivating factors for Online Ambassadors is non-responsiveness or lack of communication from students, or instructors, whom they are assigned to help. In particular, for most of the courses for the January - May 2011 term of the Public Policy Initiative, each student was assigned an individual Online Ambassador as a mentor (with ambassadors typically mentoring 2–5 students each); many ambassadors saw only limited activity from the students they were assigned to mentor, with talk page messages and other offers help left ignored. To minimize this (and increasing the amount of newcomer-helping each Online Ambassador gets to do), as of the September – December 2011 term, Online Ambassadors are assigned (or assign themselves) to classes rather than individual students, with the expectation that they will primarily work with those who need and ask for help, without feeling directly responsible for those students who are less inclinded to communicate with other Wikipedia editors or ask for help.

Other ways to improve communication between Online Ambassadors and the students and instructors who would benefit from their help include:

Informing students in class of the identity and contact information for the specific Online Ambassadors who are working with them
Notifying Online Ambassadors ahead of time about when help is likely to be needed (for example, when a new assignment milestone is approaching, or during a class or lab session where students will be editing)
Using early editing exercises as an opportunity to practice wiki communication by leaving a message on the talk page of an Online Ambassador, to establish personal relationships before help is needed

Inconsistent ambassador availability on IRC

Live help from Online Ambassadors on IRC can be very effective for students and instructors who avail themselves of it. However, making sure active helpers are always around in a program-specific IRC help channel has been difficult, since most Online Ambassadors do not use IRC regularly. In some cases, students who come to IRC looking for help end up leaving before anyone responds to them.

Ways of mitigating this include:

Pointing students and instructors to general Wikipedia help channels (which are busier, with helpers who may not be familiar with the typical needs of students or the structure of the Wikipedia Education Program, but which almost always have active helpers)
Recruiting more Online Ambassadors who are regulars on IRC
Directly encouraging more Online Ambassadors (and Campus Ambassadors as well) to hang out on IRC

Regional Ambassadors

The main roles of a Regional Ambassador are:

to grow the Wikipedia Education Program by finding new instructors and new Campus Ambassadors candidates who want to participate, within a particular geographical region
to organize the training of new instructors and Campus Ambassadors in the region
to help instructors and Campus and Online Ambassadors organize into "pods" for individual classes, and to check in with pods in their region over the course of the academic term

As of September 2011, Regional Ambassadors are recruiting participants and coordinating pods in 10 regions of the United States. As the Wikipedia Education Program grows within the United States (or another country that adopts the Regional Ambassador role) and individual regions have too much activity for their Regional Ambassadors to manage, more Regional Ambassadors will be recruited and trained and regions will be split apart. Ultimately, when enough pods are active in one university, an individual Regional Ambassador may be responsible for a region consisting of only that university.

Typical Activities

Recruiting new instructors to run Wikipedia assignments and work with the Wikipedia Education Program, based on recommendations from other Wikimedians and program participants as well as their own outreach efforts
Discussing the Wikipedia Education Program and the support and resources it can provide with interested instructors
Recruiting new Campus Ambassadors to support participating instructors, based on recommendations from others as well as their own on-wiki and off-wiki recruitment efforts
Organizing, and often conducting, the training of new Campus Ambassadors and instructors
Instituting new "pods" for participating classes
Checking in periodically with the members of active pods over the course of the academic term
Reporting the status of their activities, and of the Wikipedia Education Program overall in their regions, to the Wikimedia Foundation

Motivations

While the Regional Ambassador role is new and no surveys have been conducted yet about what motivates them, the Regional Ambassador role is designed to appeal to several key motivating factors:

a desire to support Wikipedia and/or the Wikimedia movement; genuine belief in the mission of Wikipedia Education Program
professional development of leadership, organizing and management skills
the chance to make connections in the academic world

Recruitment

The two main methods for recruiting Regional Ambassadors focused on the pool of people already active in Campus or Online Ambassador roles, and the broad swath of Wikipedia users in geographies where Regional Ambassadors were needed. Campus Ambassadors, in particular, often have the skills and motivation to take on new duties as Regional Ambassadors. Geonotices for individual geographies also proved effective for attracting experienced Wikipedians who were not previously serving as Campus or Online Ambassadors, as well as occasional users without extensive editing histories. As with Campus Ambassadors, experience as a Wikipedia editor is not a prerequisite to being an effective Regional Ambassador, so other outreach methods may be useful as well to recruit new Regional Ambassadors. The primary qualifications sought for include good leadership, management, and communication skills.

Training

The initial one-day training program for Regional Ambassadors focused on explaining the role of Regional Ambassadors and their place within the overall program, on how to recruit new participants, and on leadership and management skills: communication, understanding and evaluating individuals, and motivating teams. At the Wikimedia in Higher Education Summit immediately after this training, Regional Ambassadors also facilitated broad discussions with other Ambassadors in their respective regions to start putting their training into action and start building a regional community.

Issues, Resolutions, Recommendations

It's a big job.

Recruiting new participants, onboarding them and coordinating their training, and getting them started in time for the next academic term is a lot of high-level responsibility, and will be too much for some otherwise well-qualified and enthusiastic people. Therefore, coordinating amongst the Regional Ambassadors and making their jobs as easy as possible is critical to prevent them being overwhelmed. Splitting up the responsibilities of the Regional Ambassadors - by bringing on more people and/or by designing new roles - would also be beneficial.

Clear and timely instructions, and the maintenance of a positive attitude amongst the Regional Ambassador group, are very important, even more so than with other ambassador roles.

Coordinating the Regional Ambassadors

As the role is still being defined and fine-tuned, the initial cohort of Regional Ambassadors will need some higher-level coordination, which so far has been provided by Wikimedia Foundation staff. As the program — and the number of Regional Ambassadors — grows, coordinating the Regional Ambassadors will be even more challenging, and for the long-term growth of the Regional Ambassador-driven model of program growth, it will also be necessary to reduce or eliminate the need for Wikimedia Foundation staff to coordinate ambassador activities. How to resolve this is an open question that the Wikipedia Education Program will try to address in the coming year.

Strong potential for expanding existing programs

The Regional Ambassador role is proving effective for expanding the established program in the United States, and will likely be a good model for other countries once they have established an initial education program through a pilot project for one or more academic terms. The added complexity of recruiting, training and coordinating Regional Ambassadors may make the role inappropriate for programs that are just getting started.

Program Operations

Course Design

One thing we learned early on in the Public Policy Initiative is that each professor has a very specific idea of what they want their students to get out of the Wikipedia assignment. And it was important to work with professors to ensure that their assignment would work with Wikipedia's core policies. As part of the Public Policy Initiative, we collected a series of recommendations for assignment design and a sample syllabus that explains our best recommendations for using Wikipedia as a teaching tool. A Bookshelf brochure, Wikipedia as a Teaching Tool, offers professors more assistance in creating assignments.

We had one professor who assigned his students to develop images to accompany Wikipedia articles, whereas the more typical assignment was to write content. Some specific recommendations for what works well follows.

Issues, Resolutions, Recommendations

We recommend that a Wikipedia writing assignment should replace a traditional research paper in the syllabus. Professors whose courses required students to focus on original research or analysis of existing material only had a harder time assigning students to write an encyclopedia article, since it was at odds with the course goals.

We recommend that the Wikipedia assignment forms the traditional literature review section of a longer paper. We found that professors who chose this method of having their students contribute to Wikipedia were providing students the opportunity to do the typical analytical writing assignment in the second half of the assignment (not on Wikipedia), while also giving their students a chance to use what they'd written for Wikipedia in their final papers.

We encourage milestones to be spread throughout the term. Students procrastinate, and many underestimate how long it will take to write for Wikipedia. Establishing benchmarks (create a user account, make a preliminary edit, choose your topic, have a short outline, create a first draft, etc.) for students is key to their success.

Non-writing contributions are also valuable. One professor chose to ask his students to create infographics for Wikipedia articles. We also anticipate that tackling a project like recategorizing Wikipedia articles could be a good assignment for a course as well.

Clarify with the professor exactly how much they are expecting students to contribute before investing resources into finding Ambassadors for that course. Campus Ambassadors take time and energy (and in our case, money, although moving the training online should fix this), and several times, we encountered professors who were only interested in having their students add a reference or fix a sentence or two on Wikipedia. While all contributions are valuable, students don't need the help of an Ambassador for this short of an assignment. Our Teaching Fellowship Guidelines document encourages students to contribute 10-20 paragraphs of text.

Support Materials and Tools

A wide variety of tools and support materials were created as part of the Public Policy Initiative that are now available for professors and Ambassadors in other education programs. All of the materials are available using a free license, so professors unconnected to our program are welcome to use the materials. Materials are generally available in three places:

Education Portal – a resource for educators who want to use Wikipedia in their classrooms.
Bookshelf Project – a series of brochures and welcome videos that teaches students (or anyone else!) how to contribute to Wikipedia. Students found the Welcome to Wikipedia brochure to be particularly useful.
Ambassadors Resources – a series of handouts, videos, and interactive lessons that have been created by Ambassadors for use in their classes.

Some tools are available only to professors participating in the Wikipedia Education Program:

Leaderboard (example is for U.S. Education Program) – professors are able to see the ranking of courses and students based on characters added to Wikipedia's article namespace.

CourseInfo – this tool lists all the students in the course, which articles they have worked on, and when they made their last edit in the article namespace.

HeatMap – this tool lists the percentage and number of students who have been active in the article namespace within the last 30 days.

Issues, Resolutions, Recommendations

Survey results were mixed on what type of materials the students most appreciated when consulting help resources, leading to the belief that each student learns differently and it is thus important to provide printed materials, online materials, and videos so that each student's learning needs are met.
More resources are always needed, and many of the existing resources need to be updated due to change of functionality on Wikipedia or to make them more user-friendly or of higher quality.

Student Contributions: Article Quality Improvement

At the start of the Public Policy Initiative, Wikimedia Foundation staff worked with Wikipedia community members to develop a quantitative article quality metric. This metric assigns weights and values to Wikipedia policies regarding article quality; the assessor applies the metric and generates a numeric quality score which is translatable to the Wikipedia 1.0 rating system (Featured Article, Good Article, A, B, C, Start, and Stub ratings). (Only 0.1% of rated articles in Wikipedia earned a Featured Article status.) Public Policy Initiative Research Analyst Amy Roth made a scientific comparison of public policy expert scores and Wikipedian scores of article quality on this quantitative metric. Using a linear regression model (Total score = A + Wikipedia Expert + article + error), Wikipedians on average scored article quality 4 points lower than public policy experts with a p-value much less than 0.01 and an R^2 value of .67 measuring model fit.

The assessment team for student article quality was comprised of both policy experts and experienced Wikipedians. The policy experts all have graduate degrees in public policy or public affairs; they are graduates of universities including: Berkeley, Boise, Brandeis, Georgetown, Pennsylvania, Pepperdine, Sacramento, and Syracuse. The subject matter expert group included policy experts who have worked in policy for more than 30 years, emeritus professors of public policy, several recent graduates, and some currently active in the public policy field. Policy experts were recruited through alumni email lists of many universities. Wikipedians were recruited through direct invitation on their user talk pages and were selected for quality assessment experience, a non-confrontational style, and interest in public policy topics. In fall 2010 there were six policy experts and 15 Wikipedian assessors (1:2.5, PE:WP). In spring 2011 there were 25 policy experts and 11 Wikipedians (2.3:1, PE:WP). Over the course of both terms, there have been a total of 28 policy experts and 17 Wikipedians on the assessment team.

The assessment team performed multiple assessments on a randomly selected set of articles from each term. In most cases, three individual Wikipedians and three individual public policy experts assessed each article. For clarity, the group of Wikipedian and expert assessments will be called a review. For pre-existing articles, article reviews were performed on one version of the article before the student work and then another version after the student work. For the new articles that students created, the articles only received one review, which was conducted after the student contributions.

Results

Over the course of the project a total of 140 articles were randomly selected from the WikiProject United States Public Policy Course tab and assessed. 84 were pre-existing and 56 were new.
There was 64% increase in all articles, the average score went up 5.8 points (9.0 before, 14.8 after)

There was 50% increase in pre-existing articles, the average score went up 4.8 points (9.7 before, 14.5 after)

New articles had an average score of 15.4 which translates to an average B rating in the Wikipedia scale.

Over the course of two university terms, approximately 800 students participating in the Initiative contributed over 8.8 million bytes to the Wikipedia namespace.

Other Learning Points

Program organization on-wiki: We created Wikipedia:WikiProject United States Public Policy to be our central organizing place for course pages, lists of Ambassadors, news about the program, and discussion. WikiProject U.S. Public Policy worked well for us, as there was not previously a WikiProject of editors organizing around the topic of U.S. public policy. Having a central organizing place for all courses on Wikipedia is critical.

Subject matter: The fact that very few editors were working on U.S. public policy topics had mixed results for our program. In one way, students were able to learn how to edit Wikipedia and make newbie mistakes in an area of Wikipedia that had little attention, affording them the opportunity to work with their Online Ambassadors to correct any Wikipedia faux pas before their work reached a larger audience. It also meant that there were many articles which needed dramatic improvement, offering plenty of topics for students to work on. The downside to working on such a neglected subject matter on Wikipedia, however, was that students did not get the peer review and other interactions with the editing community that students working in other areas may have had.

Google Group: We tried creating a Google Group that included both Campus and Online Ambassadors, to facilitate communications between these two groups. But the group quickly became noisy for a lot of the participants, since many were unused to listservs, and discussion often revolved around minutia that was not applicable to everyone. The useful bits of the group often got lost in the noise of the other messages. For a while, we put the group on moderation, but by that point, most of the recipients had unsubscribed or were just deleting messages to the group unread. Since this was the primary channel that staff used to communicate to all Ambassadors, this was a problem. We created a second "Ambassadors-Announce-L" list, which we fully moderate and require everyone to subscribe to, and use the Google Group for less important things, making it an optional discussion list.

Student-Online Ambassador pairing: In the first term, we asked students to select an Online Ambassador mentor; ones who did had a significantly better experience than ones who didn't, but few students actually followed through. We made it a more strict requirement in the second term, but students found talk page communication difficult, and both students and Online Ambassadors felt like they didn't get responded to through these relationships. We hope having one Online Ambassador per class who is part of the pod and knows when assignments will be due, etc., will help this.

IRC: Many Campus and Online Ambassadors hung out in the #wikipedia-en-ambassadors channel on Freenode, and we directed all students to a #wikipedia-en-classroom channel to ask questions. Several students took advantage of this, and we hope to encourage the use of IRC in the future.

Advisory Board: At the beginning of the Public Policy Initiative, we recruited experts from public policy, open education, and academia to serve on an Advisory Board. We wanted to ensure that our efforts would be legitimized in the circles in which we were operating. The members of the Advisory Board gave us some very valuable feedback on our initial plans, but we found that professors interested enough in working with us simply accepted our legitimacy without question, given the strength of the Wikimedia and Wikipedia names. Therefore, we do not believe forming an Advisory Board is a necessary step for others looking to start university programs in their countries.

Promotional materials: In the first term, we used our promotional materials to offer a few items to students participating: Wikipedia laptop stickers, stress balls that had the words "Wikipedia Contributor" on them, and lanyards. We found that students didn't need many promotional materials to write articles for class, and professors found them distracting. Instead, we've focused our promotional efforts on providing T-shirts for Wikipedia Ambassadors and having stickers and stress balls available for Ambassadors who host a table on campus to recruit people to join a student club. Stress balls have been found to be the most effective way of bringing people to a table across both academic conferences and student tables.

Institutional partnerships: Members of one state university's Communication Across the Curriculum Department approached us and asked if we would train their entire staff as Campus Ambassadors. We eagerly accepted, and our first institutional partnership was born. We've found that the more we were able to find people on campus whose job it is to assist professors with new technologies to make their courses more interesting, the easier it is to have those people be the outreach folks on campus. Finding people within institutions to partner with was a key facet of our growth within the United States, and it's something we suggest replicating around the world.

Gender balance: We strove to ensure that we had about a 50/50 ratio of male to female Campus Ambassadors. We heard from several female students that having a female face in the classroom being the one to teach them how to use Wikipedia in their course was inspiring them to contribute. The more we can do to increase women as other women's first point of contact with Wikipedia editing, the more we can work to solve the gender gap in Wikipedia contributors.

Academic conferences: We attended two teaching and learning conferences where we had a booth, and we found in both cases that the combination of a booth and a presentation was a great way to attract professors who cared about student learning and were interested in trying new pedagogical methods in their courses. We also attended a couple of larger subject matter conferences (where booth space was much more expensive) and determined that they were less worth the effort, as the number of people we reached at those conferences who have joined our program is minimal. If the curriculum is controlled by the professor in your country, teaching and learning conferences are a great way to find professors who are interested in doing experiments like a Wikipedia assignment in class. Our conference attendance also seemed to raise Wikipedia's legitimacy in academia; many attendees told us how excited they were to see Wikimedians doing outreach to academia.

Outside educational evaluator: We hired an outside educational evaluator originally to provide credibility for Initiative work and results. However, the outside evaluator was not closely linked to project, geographically or philosophically, and their evaluation did not encompass our primary areas of interest. It was a drain on staff time to educate the evaluator about the project and to administer the evaluator's metrics (i.e. student surveys and providing contacts for interviews). Resources would have been better spent developing, administering, and analyzing metrics that were more tailored to project needs and questions.

Conclusion

The Public Policy Initiative's goal was to improve the quality of Wikipedia articles in the area of U.S. public policy. A secondary goal was to see if we could systematically integrate Wikipedia assignments into university classrooms, providing support for a continued outreach program at universities around the world.

Our high-level learning points from the Public Policy Initiative includes:

The model of professors assigning students to edit actually improves the quality of Wikipedia articles.
Basic face-to-face Wikipedia introduction is very valuable to students, and can be taught by Campus Ambassadors who are not Wikipedians themselves; teaching skills, a willingness to learn Wikipedia basics and enthusiasm for Wikipedia are more important than edit counts for the Campus Ambassador role. We saw no correlation between someone's performance as a Campus Ambassador and their level of prior Wikipedia-editing experience.
Online mentorship and support from experienced Wikipedians like Online Ambassadors is crucial for students' success.
Even with support, students will make mistakes, but it's important to have a safety net of Ambassadors so students can learn from their mistakes.

As we expand the model, we see the following big challenges moving forward:

Scaling support from Ambassadors, especially Online Ambassadors as their current definition limits them to being experienced Wikipedians.
Differences between U.S. educational system and other countries' systems as we expand globally.

We look forward to expanding Wikipedia's use as a teaching tool in higher education through the Global Education Program.