Wikipedia talk:India Education Program/Analysis

Outstanding Questions

A big question for me is: what is the best way to measure the impact of this program on the community? We certainly don't want to put more work on the community to estimate this impact, but we also believe measuring the program's impact on ordinary editors is critical to having a full picture of what happened in the Pune pilot. Any ideas of how to overcome this challenge? -- LiAnna Davis (WMF) (talk) 22:14, 1 December 2011 (UTC)

I think the best way to go about this would be to estimate some sort of "crap fraction" reflecting problems that need to be cleaned up in a similar manner to the PPP metric:

$k_{1}(a+b)+k_{2}c+k_{3}d+k_{4}e \over m$

with

• a = copyvio edits in any namespace
• b = copyvio uploads (local + commons)
• c = mainspace edits deleted or reverted for other reasons. Redirecting an article counts as reversion.
• d = mainspace edits with orange {{ambox}} problems (e.g. lack of RS, POV). If an edit has multiple problems, add the number of problems instead (e.g. unreferenced POV edit increases d by 2).
• e = mainspace edits with yellow {{ambox}} problems (e.g. wikify, copyedit). If an edit has multiple problems, add the number of problems instead.
• m = mainspace edits

Edits should be assigned to the category with the highest weight (e.g. G12 deletion => 10 points). The weights are arbitrary but reflect the severity of the problem: k₁ > k₂ > k₃ > k₄. I suggest something like k₁ = 10, k₂ = 4, k₃ = 2, k₄ = 1. MER-C 03:36, 2 December 2011 (UTC)

CCI

Attempting to assess the impact on the community of the CCI is virtually impossible at this stage because very little of the work has been done. The investigation is less than a week old and isn't going to be finished for a long time yet (we still have open two-year-old CCIs). I suppose you could investigate what percentage of contributions have been removed as copyright violations, but that's going to produce a severe underestimate because most of the edits haven't been systematically surveyed for copyright problems. Hut 8.5 23:56, 1 December 2011 (UTC)

Would there be data points like the backlog increase or anything like that we could use? We do have a couple of months to do this analysis, so we want to make it as thorough as possible, and if that takes waiting until the CCI process has had a chance to start so there is some time estimate able to be extrapolated, that's fine. -- LiAnna Davis (WMF) (talk) 00:03, 2 December 2011 (UTC)

You could count the number of edits, users or pages which have been reviewed, I suppose, but such a statistic might not be very meaningful because some users are much easier to check than others (some project participants have no edits apart from adding themselves to the project, for instance). It's possible that in a few months the first page of the CCI investigation might be complete, in which case you could get an idea of the number of copyright violations produced by a significant sample of users (196). Hut 8.5 00:17, 2 December 2011 (UTC)

Even that would be an underestimate due to copyright violations that were deleted before the CCI was run. To get a wholesome picture, you must consider both live and deleted edits. The CCI is also not complete: I noticed one user who posted a copyvio in the user talk namespace. The CCI backlog is so ridiculously large that the proportional impact of the IEP is rather small. The best way to assess impact on the community would be a crap edit (all namespaces+Commons)/potentially useful edit fraction weighted heavily towards copyvios. MER-C 02:13, 2 December 2011 (UTC)

Is analysis needed?

"We want to do a thorough analysis of the Pune pilot program to derive learnings and trends"

Or you could spend a few minutes asking some of the people on the (virtual) ground. This doesn't need statistics - a qualitative analysis (which half-a-dozen names could deliver in less than half-an-hour) would tell WMF more than I still suspect they want to hear. A snazzy Powerpoint with figures on is no excuse for A Clue. Some of the basics of what went wrong are very obvious and they just need fixing. We don't care whether they went a lot wrong or a little, they went too wrong, and they need to be avoided completely in the future. Analysing trends will not explain any of this.

I'd note also that some of the really deep questions rely on the so-far invisible relationship between the WMF and the Indian colleges. Who expected to gain what from this entire process? Without knowing what the intention ever was, it's hard to know just where it went wrong. Andy Dingley (talk) 00:18, 2 December 2011 (UTC)

Hi Andy, Please note that both quantitative and qualitative analyses are happening! As is mentioned, Tory Read (our outside evaluator) is interviewing a few Wikipedians for her report (I see from Hisham's talk page that you were one of the ones asked). We've also been taking note of all the information that's been previously mentioned on the talk pages. If you think we need more editor interviews, suggest that! We want to do such a thorough analysis because everyone we talk to has a different answer of what's wrong with our program, and I think that's because our program design was flawed on multiple points. Fixing one or two of the problems will just result in another frustrating pilot; we want to take the time to plan the next one right, and that involves a very thorough analysis to identify all the reasons why our first pilot failed, and what we can do to fix those problems.

I'm glad you said you think the relationship between WMF and the colleges is invisible, as I wasn't aware that was an issue. The Wikipedia Education Program (this is true for our program in India as well as our current programs in the U.S. and Canada) is designed to get more people contributing to Wikipedia and to improve the quality of the content of Wikipedia (you'll recognize these as goals from the Strategic Plan). Obviously the quality part failed miserably in our Pune pilot, but data from our U.S. pilot showed that article quality improved 64%, which led us to expand the pilot to other countries, including India. The India program is designed to specifically address the Global South parts of the Strategic Plan. You can see the benefits for instructors on the Education Portal's Reasons to Use Wikipedia page. If you're looking for more details on the communication between WMF and various other parties in India, I encourage you to check out the Wikipedia:India_Education_Program/Documentation page. I'd be happy to clarify any other questions you have about this -- I'm really sorry it wasn't clearer before! -- LiAnna Davis (WMF) (talk) 00:55, 2 December 2011 (UTC)

Extolling the virtues of some relatively minor successes is only to cloud the real issues. The Pune experiment was a major success in that it has clearly demonstrated how not to plan, organise, and execute what are nevertheless extremely important initiatives in the effort to expand the encyclopedias and reach out to other cultures. The endemic gap between the salaried operatives and the volunteers needs to be closed, and there should be less unilateral action on the part of the organisers without listening to the community amongst whom are also some experts (who are not paid). The organisers have been aware for a long time that the relationship between WMF and the colleges is invisible, as well as the issues concerning the general planning and management of the projects from the top down and the absence of communication and cooperation with the online community.--Kudpung กุดผึ้ง (talk) 03:59, 2 December 2011 (UTC)

"important initiatives in the effort to expand the encyclopedias "

That's my main point. Was this an effort to benefit the encyclopedia my adding content, by adding new editors, or to benefit the students/college by giving them an exercise? I don't want to care if this stuff is invisible or not, I don't want to even have to think about it, but when it's time for analysis we have to know what the original hope was, before we can judge whether it met it. All of these goals have some value to them and appear to be a simple goal to achieve. In actuality, none of them would be anything like so easy to meet.

• Expanding the encyclopedia is done by having competent people add competent content to it - as a minimum. Usually this arises through some personal passion for a topic and prior knowledge. Students who are arbitrarily assigned a topic are unlikely to have this. It was a huge problem for IEP engineering topics. Students obviously didn't know the topics, didn't stop and learn them before starting to write and had no inclination to even try to - most seemed to think that blindly pasting was enough, without any intervening comprehension. Engineering topics were also poorly worded as titles with no explanation as to what was meant.
• We're regularly told that we need new editors (despite the loss of good, undervalued editors being a far more serious problem). Recruiting them from students sounds like a good idea, but getting past WP:BITE needs a fair personal commitment, not just a course assignment. We not only asked students to edit, we asked them to deliver an acceptable article from cold, on their first article. That's hard to achieve on a first edit and it can only be done by people who know something well enough to write it, and who have enough editing skill to produce it. How do we train people to this level? As general quality standards are pushed higher, this is going to become a bigger and bigger problem - how do we get new editors past this skill gap without driving them away first by instant reversions and warnings?
• As a benefit to students, it's hard to see how it can possibly work. What's it for? To teach a topic, or to teach the technique of writing for a public audience? The second has some attraction, but where was the teaching? Students were dumped into the end of course assesment exercise without being taught anything beforehand. Just how was that expected to work?
• Any benefit to tutors is probably a bad idea from the outset. "Here's a chance for an easy self-marking exercise where a pre-existing community can be borrowed as tutors and assessors" is bad teaching, and sheer exploitation of the WP user community. I suspect this was one of the real goals behind this project.

The ones I feel sorry for in the midst of all this were the students. They were given a poorly thought out and broadly impossible task, then criticised from both sides afterwards. Andy Dingley (talk) 11:22, 2 December 2011 (UTC)

Analysis IS needed. First, at least if they are analyzing, then they are not implementing. Second, yeah...if we are going to do GEP/IEP at all, we need to understand costs and benefits better. Tingley's post has some great points on goals. I would just list all the different possible benefits and then quantify how we did. Look at retained new editors and created new content (amount, value, page views). Then at student learning, teacher satisfaction. I think the former two are what Wiki gets out of it. The latter two what schools get out of it. Actually my bigger worry is not that we are doing unneeded analysis, but that the analysis will be incomplete or poorly done (not ask the right questions).

Unless people are advocating closing GEP, re-assigning or firing the employees, and giving up on school-based outreach...then I think analysis is needed. I guess if someone wants to write off school based outreach forever, no...analysis is not needed. But if we do want to do it in the future...we need to understand Pune. Not just that it sucked. But how bad it sucked. In what aspects it sucked. Etc.  ;-)

TCO (talk) 03:34, 3 December 2011 (UTC)

P.s. I disagree a bit that school projects can not produce good content. The JBMurray and JimmyButler groups have done fine with grades and assignments being the motivator. Heck, if we had paid editors, I'm sure they would do fine, too. (And I'm not advocating that...just saying that the idea that people can only produce fine craftsmanship for free...is wrong.) -TCO

Programmatic results

To be certain we are measuring what needs to be measured, I'd like to see more detail regarding the parameters/dimensions of the program. I suggest starting with a list of the program goals and the planned program components and add a list of the unintended/unexpected outcomes. Then quantify those items and analyze the relationships among them. Jojalozzo 00:51, 2 December 2011 (UTC)

Great idea. I'll start working on this list. -- LiAnna Davis (WMF) (talk) 01:11, 2 December 2011 (UTC)

Is something that will be shared and added to the plan? What is the time line for conducting the analysis? Jojalozzo 19:29, 2 December 2011 (UTC)

Core issues

The answers do not lie in added bureaucracy: proposed solutions and metrics are all available already in the many and various comments by community members on the increasing maze of talk pages connected with the IEP and USEP projects. The impact on the community is blatantly obvious - why keep creating yet more pages to add to the confusion, and carry out further costly analysis when the answers are already staring us in the face complete with charts and graphs already provided, and have been discussed in depth on the mailing lists and other obscure lines of discussion? The main answer lies in rectifying the continuing lack of communication, transparency, and admission of errors. The main concern raised from the Pune pilot from the community angle is not being addressed, and LDavis and/or AnnieLin have made it clear elsewhere that they do not consider it part of their remit to take the community resources - the very impact that is being mentioned here - into consideration when planning their education projects; ignoring the known problems and trying to find solutions to new new ones that apparently still need to be identified is a redundant exercise. Ultimately, this will simply foster more ire and drive yet more volunteers and OAs away from wanting to be helpful, rather than solicit their aid which in any case can only be to repeat what they have already said time and time again. Solutions and suggestions have been tossed around by some extremely competent and knowledgeable members of the volunteer force only to land repeatedly in some kind of no man's land between the WMF and the community. It is imperative to understand that all education programmes will generate more articles - which is of course the goal of the initiatives, and which is recognised and supported in principle by everyone - that will still need to be policed by experienced regular editors, and that these programmes cannot be implemented before the online volunteer community is forewarned, and forearmed with the required tools and personnel. Perhaps Tory's independent analysis will come up with some answers (and I'm confident it will), and it may be best to wait for her report. . Kudpung กุดผึ้ง (talk) 03:33, 2 December 2011 (UTC)

Kudpung: Would saying that for the next India pilot, students are required to expand stub/start class articles rather than creating new articles address your NPP tools concerns? I do hear you that the NPP people need better tools to do your jobs, but I'm afraid WMF isn't going to put all outreach projects that might generate new pages on hold until those tools are fixed. As I said before, I (as a staff person in the Global Development department) have *much* less influence on the tech team's roadmap than you (as an active community member) do. In an ideal world, would WMF work in concert to ensure that all activities across all departments had adequate tech support to assist in all areas of their projects? Absolutely, but I have yet to find a place that works like that, and WMF certainly doesn't. So let's try to figure out a way around that issue for the time being by discussing ideas like requiring students to work on stub/starts class articles only. What other ideas could address this problem?

I don't know how much more we can take "community resources...into planning" than ask on this analysis than to ask the question: "What was the impact of this program on the community?" If the analysis says the Pune students caused the NPP's 4X the work that 800 normal newbies would, we obviously need to do something to make sure students aren't creating new pages and thus putting undue work on NPP. So how to we determine that impact? I really value your input, Kudpung, and I would like you to participate in this analysis. I completely agree that we made mistakes in our lack of communication, transparency, and admission of errors. But I feel like that's what this analysis is trying to rectify, and I'm sad to hear you think it's "continuing". I posted talk page messages to talk pages alerting you this page was here. We're being transparent about how we're doing the analysis, and we've posted all the emails that should have been posted earlier in the Wikipedia:India_Education_Program/Documentation page. We have admitted making a lot of mistakes. I honestly can't think of any piece of information we haven't released publicly at this point. You know what I know. I talked with Tory as well, and I look forward to her report on January 15, just like you do. I don't know how more transparent we can be, but please do tell me if I'm missing something. In the mean time, I do hope you'll help us figure out how to measure the impact of these programs on the community so we can brainstorm ideas of how to lessen the load on volunteers like you. -- LiAnna Davis (WMF) (talk) 16:21, 2 December 2011 (UTC)

I don't care about the precise size of the impact. It's clear that it was negative and appreciable, that's as much detail as we need. The useful question is about its cause, and how to avoid that happpening again.

As to expansion vs new pages, then this should be chosen for their effect (a complex issue, discussed previously) rather than for whether they go through NPP analysis or not. Whether NPP has adequate tools or not, or whether NPP applies closer scrutiny, should never be allowed to become an issue here. Would we allow low-quality expansion of articles to go ahead, just because our only quality focus was at NPP? Hopefully not! Andy Dingley (talk) 19:42, 2 December 2011 (UTC)

To be honest even if the average project participant did cause less damage than the average enthusiastic newbie that doesn't mean much. For all we know our infrastructure may not be equipped to handle a huge influx of industrious new editors. We should in fact expect the project particpants to be significantly better at editing than the average newbie. Otherwise all the support and advice given to project particpants would have been less effective than pointing the students to pre-existing help pages and the program would have wasted a lot of effort.

It's pretty obvious from reading the contributions of people involved in the project that the overall standard if very low. Even if you ignore the copyright violations many or most of them violate core principles or are on subjects which are inappropriate. I don't see that putting an exact figure on how bad they are is going to be much help. Hut 8.5 21:10, 2 December 2011 (UTC)

Anna, I'm saying that due to the IEP experience, and the thousands of words of advice that have already been provided by experienced Wikipedians, and their clear descriptions of the impact it had on the community, you don't really need my help right now, and I feel that there is little more I can contribute at this stage. I'll jump in again willingly when clear lines of responsibility have been established and announced - but I will not be enthused by the aspect of spending another 200 hours of work for it on the factory floor if I am to be accused of of bad faith again by some of those involved, and if the WMF can't provide us with the support we need. On the other hand, you could help us in the meantime by accepting that you may wish to look at the bigger picture after all - WMF job descriptions apart, their work overlaps all the time like the bubbles in a Lava Lamp, and rather than polarising and leaving the advice in the middle of the office floor, the essential practical solutions can only be implemented by collaborating on these issues, and including the volunteers in them (who in actual fact practically have no influence whatsoever) rather than just harvesting more comments from them that can only be repeats of what has already been said. Tory is in India at the moment, and I think we should wait for her independent report, based on which there could conceivably be some recommendations for shifts in infrastructure, and the details of what kind of articles should be produced by the next phase of the IEP could probably wait until then. Kudpung กุดผึ้ง (talk) 07:14, 3 December 2011 (UTC)

I fully agree with what AD, K and others have said above. Surely if WMF are serious about about copyvio (to take just one issue) then it must be many years before any "next phase of the IEP" (a phrase that must send shivers down the spine of anyone who's seen the damage already caused) should be considered. WMF should concentrate on stopping the still ongoing damage (e.g. [1]) and understanding what mistakes _they_ made. I don't know how many of the clear-up volunteers would be willing to help out should something like this ever happen again; if it appears that WP:EN is being changed from an encyclopedia (with the aim of quality copvio-free content) into an educational tool then I'd certainly reconsider my involvement. DexDor (talk) 10:22, 3 December 2011 (UTC)

There's a big difference between "normal" WP well-intentioned (i.e. not vandals/spammers) newbies and the Pune students. Normal newbies learn from the changes others make to their edits and any comments they receive ([2] shows an example of what I mean). The students have generally ignored such things and even copyvio warnings etc. A normal newbie probably wouldn't try to make large edits to a reasonable article, in a subject that they don't understand and in a language they are not fluent in. I'm happy to help voluntary newbies who could well become very useful editors, but IMO few of these students have the inclination or competence to become useful content-adding editors on WP:EN for some years. Another difference is that normal newbie edits are generally beneficial even if needing a bit of tidying; in contrast the changes to engineering articles have almost(?) all been damaging. I can only spend a certain amount of time on WP - and every hour spent clearing up bits of the IEP mess (or reading pages like this) is an hour less doing things like helping real newbies.DexDor (talk) 10:22, 3 December 2011 (UTC)

The Global Education Program was born out the paranoia of a slightly dwindling number of new articles and contributors. What the people involved with IEP who have '(WMF)' in their signatures don't realise, is that some of our most prolific and experienced editors are now jumping ship due to the IEP fiasco and looking for new hobbies (I've had three emails of intent to retire over the IEP issue in the last two weeks). By insisting on ploughing ahead regardless with these gung ho schemes, the support of the already depleted volunteer clean-up brigade is now at low ebb, and the paid personnel are throwing the baby out with the bathwater. --Kudpung กุดผึ้ง (talk) 12:13, 3 December 2011 (UTC)

The Wikimedia Foundation has strategic goals; you can review them at strategy:Wikimedia_Movement_Strategic_Plan_Summary. The Education Program is one of many programs the Foundation will be running in the next few years in an effort to enact that strategic plan. There is a focus on the Global South in particular, of which India is a part. Are you specifically objecting to the India Education Program, or are you objecting to the strategic goals? Please don't take this as a flip comment, I'm genuinely trying to understand what is causing people to be frustrated enough to seek out other hobbies.

I'm also surprised that you think we are "ploughing ahead" -- we are not. We canceled the program for the upcoming term so we could focus on fixing the problems. In previous talk page messages, you and other editors (rightfully) accused us of starting the Pune Pilot without involving the community in the planning process. Now we're trying to involve the community in planning for the next India pilot. We're trying to do it right this time. I'm glad you think Tory's report will be useful; I do also, but Tory is unable to talk with every single editor, and I don't feel that we can say we've involved the community in determining why the Pune Pilot failed and how to fix it next time if our only interaction with the community is through interviews with a handful of editors in Tory's report. I am in full good faith trying to fix what went wrong last time: that there was no community involvement. If those of you who talked with Tory feel like she will capture everything you have to say, then feel free to uncheck this page from your watchlist, but I really firmly believe that community members have great ideas, and I wouldn't want to miss something because we only involved a handful of people Tory talked to.

The point of this analysis is to analyze why the program failed, so we can address those problems in the planning stage. If I'm reading what's written above correctly, it seems like people believe students create more work than ordinary newbies because many of them don't want to contribute to Wikipedia. I think saying the majority of the Pune Pilot students weren't motivated enough to ensure their material was of high quality and copyvio-free, which caused strain on Wikipedia cleanup systems, is a fair assessment of one thing that went wrong with the Pune Pilot. Once we have this full analysis of what went wrong, we can tackle how to fix it in the next pilot. Scale is one obvious problem; the next pilot will be much smaller. To tackle the motivation issue, we've discussed making the assignment optional, so that only students who genuinely want to have their articles appear on Wikipedia and are willing to work for it would participate in the assignment. But I'd much rather have a full picture -- with community involvement -- on what went wrong before we start finding solutions. -- LiAnna Davis (WMF) (talk) 18:40, 5 December 2011 (UTC)

Anna, I am neither objecting to IEP, USEP, nor the strategic goal of which I am a firm believer and supporter and ready to be of assistance where I can. I've mentioned before however, that my concern at the moment is less for what the students should do in the future, than for the way the project's continuation is prepared for now. Tory cannot talk to every editor, but she can talk to a balanced number of people who represent the organisers, to those who speak for the community, and to those on the receiving end of the programme. By reading all the talk pages and mailing lists, she will also have been able to gauge the impact on the community and its stakeholders even before conducting her interviews, or waiting for numerical statistics of copyvios. The causes need more analysis than the effects, and this may require a closer examination of the infrastructure. This means that not only is input required from the community, but also involvement from it volunteers among whom there may also be some experts.

The main solutions to be addressed therefore, are ones of planning and executive coordination and not strictly those that lie in the curricula of the colleges. Discussing at this stage whether assignments should be optional or how the students should edit the encyclopedia are tactical issues, and are ones that can come later when the strategies have been properly worked out. Right now, although the community is providing excellent input for USEP, it remains fractious over the IEP and any dialogue risks to continue simply as a back-and-forth of criticisms from the community (like mine), and statements of apology from the organisers - and that is why Tory's report will be crucial to these issues.

It would be great if the volunteers could get together and draft their own 'official' report - but that is unlikely to happen. Kudpung กุดผึ้ง (talk) 06:10, 10 December 2011 (UTC)

High level thoughts

0. The community is seriously torqued and rightfully so. People have a feel good reaction to students and want this stuff to work. At least they did in the beginning. This may be changing, unfortunately to resistance, even shadenfreude. If you lose the community, none of your programs will go anywhere. We do need to try new things and GEP could really be something special. Something we can't execute without Foundation help. So I hope we can get to success. But...I'm worried.

1. There seems to be a lack of savvy about how Wikipedia itself works (what our content is, how articles get created, what the trials and tribulations are) from the WMF staff. Well, especially in GEP. I advise those of you who have not experienced the factory floor (uploaded an image, been reverted, had an edit war, used the citation template, earned a barnstar, etc.) to write an article or two. Get a DYK or take some article up to B or so. Get some feel for this place. It will really help you as you do external work if you understand the internal side of things. We are all trying to figure out Wiki as a system and how to make it work and grow and get better. Make it committed work for everyone in the department to get this done. A little time spent on training yourselves...will pay off in more effective work in the future. Don't be scared of messing up and looking bad. Be bold and learn by trying.

2. Going after Indian engineering college students was just daffy. They are not good English speakers (I don't care that they try to work in English...the issue is from our perspective they are not good.) Is Pune even a good school? IIT is the one I hear of as tops in India. Let's aim high. Let's go for people who can help Wiki. For the easy fits. Not the hard ones.

3. The small programs in the North America that worked better (Sage Ross, Jimmy Butler, JBMurray) did so because of small scale (allows more help) and committed teachers. I'm not saying we can never scale. But the whole approach of going for mass when we don't really understand how to do this stuff well, seems flawed. Cherry picking seems better.

The GEP unit should be judged a bit more on what it learns, not what it delivers for a while, not on these mass experiments. If you do it right and figure out the right model, it will scale then. But I'm not sure it is the current ambassador system (don't know). Perhaps it is writing something up with Jimmy Butler and then getting together with the organization or association of AP Bio/Chem teachers. Perhaps there is some way to spread more like an inkblot, more virally (have successful teachers pitch others). Also, are we monitoring how teachers do in returning to using Wiki? Or is it new prospects every year?

And I do give you credit for trying things, for your initiative. I mean people fly around and set up these programs and make things happen. Wiki community can be very conservative and study too much and not act. So I do give credit for trying things. But...we need to think through this puzzle more, now.

I guess the PPI/USEP also stands out as a success (maybe) and it was "big", but I would like to get a little more info on details of cost and benefit (include the volunteer time, especially if it is people like Christie diverted from content work) and what classes or approaches worked and didn't. Still feels fuzzy to me. I don't understand it the way I understand FA/VA where I cut the onion in several directions. (And you'll never understand anything perfectly or have time to do every analysis imaginable. But...just what I've seen so far on PPI has lacked detail. I just can't get my arms wrapped around it.)

4. I don't understand the participation objective. Frank seems to say it's not an objective--just giving the schools exercises is our mission? Or he thinks this is well spent money for the content recieved--doubt it. Also it doesn't seem like we are measuring retention of the new editors or had a goal in mind or a hypothesis to disprove (in terms of percent retention).

5. (small insight) One of the cool things about working with Jimmy Butler is that his high school kids have the whole year. There is just more of a feeling of ongoingness than with semester college kids that are really there for 16 weeks or so (less on trimesters). Also, the teachers have more continuity in terms of teaching the same classes year to year (vice uni where the profs vary every year what they teach). Maybe consider attacking the magnet schools? Or Thomas Jefferson and The Bronx High School of Science? Donno, just some wild ideas. (But TJ or Bronx would also be nice "names" for a press release, hint, hint.

6. Consider trying to go more "high end" instead of mass. Try for better schools. Try for committed teachers.

7. Think about subject a little more. Animal articles are very contained and we have a lot of them to be upgraded. They sort of "just work". The topics may be somewhat obscure, but there are a lot of them and they just can be used as training grounds. The engineering students trying to write articles struggled though. There is not the same diversity of topics. And they were messing up fundamental topics like "Lever" or "Welding".

A Shakespeare class might have the same problem (unless perhaps a graduate class...) with finding it hard to readily contribute...as there are really just a few plays there and they may have good articles. Donno, scared to check in case it depresses me!

Chemistry is probably another "good topic" where you can probably just find a lot of compounds to write about and they have a reasonably strong structure and can just be used a bit more for training. (But keep them away from elements please...off in random compound land. Elements are too important and vital. Doubt an individual unless Ph.D. grad student would do a good job.) One concern with chem is that it is rather an intense class, more mathematical and exercise-problem solving (and more labs) than descriptive biology. So I don't know if the kids have time to add in a research project. But...it's an idea. Oh...and before you roll it out in mass, prove it with one select class at least.

What are some other topics that are well suited or not well suited? Perhaps geography is well suited because of the diversity of topics and the structured nature of the articles? That is not really tought as a subject much in the United States any more, but perhaps in the United Kingdom. Or perhaps it is close enough to satisfy history teachers (it's sort of similar and the kids will still get the basic learnings of researching information and writing endnote citations).

---

I suspect GEP has been hocking the Wiki program to profs with a slick presentation and a willingness to take any customers and saying the offering fits all customers. (Guessing here, though.) Perhaps instead if we really think about who are the best prospects (and this post is not enough, it's more sitting back and really wrestling with it) and then targeting the "good customers", then we will do better.

Oh...and India should be on the deep freeze. Let them earn their way back with some cherrypicked 30 person class at IIT with the one superstar teacher able to make it work.

TCO (talk) 02:02, 3 December 2011 (UTC)

There seems to be a lack of savvy about how Wikipedia itself works (what our content is, how articles get created, what the trials and tribulations are) from the WMF staff.

Amen to that. Now the odd part - I was about to cite as evidence here the recent "Dabblers and Star Collectors" report that so comprehensively trashed the efforts of some of the best and most valuable (yet, it appears, least valued) editors. But hang on a minute... TCO? Andy Dingley (talk) 02:18, 3 December 2011 (UTC)

To one of your points; Pune is considered one of the best in India. What people don't seem to realize is that "limiting" the study to Pune would be roughly the equivalent of "limiting" something like this to Indiana University; it is the most difficult place to start because of its size. And I also agree with another point of yours; to shamelessly recycle something I've said a couple of times, I wouldn't mistake most Indians (of whom I know many) for Jawaharlal Nehru. The Blade of the Northern Lights (話して下さい) 06:31, 3 December 2011 (UTC)

Pune may be the Massachusetts or the Oxbridge of India, but the population is 4x that of the USA, and I have great qualms of what will happen when the project reaches out to Mumbai, Chennai, Kolkata, Delhi, Hyderabad, and Bangalore. And it will. WP:India has 452 members - where were they while all this was going on? Kudpung กุดผึ้ง (talk) 12:35, 3 December 2011 (UTC)

Right here. MER-C 12:46, 3 December 2011 (UTC)

(I actually meant the WP:India project back here and their availability to help with the cleaning up.) When I first read it on its arrival in my mailbox, it came as no surprise whatsoever, and now you'll understand the reasons for the many oblique references I have been making in my numerous postings; it summarised what I had assumed all along. I've often wondered if it should not be simply posted here on Wikipedia for all to see who are not subscribed and who may not even be aware of the existence of the mailing lists. Perhaps it should be, it's publicly accessible knowledge after all. It grieved me to think that I have been sitting here only 3 hours away for the past months and unable to do anything about it other than join the force of the cleaneruppers. Kudpung กุดผึ้ง (talk) 16:24, 3 December 2011 (UTC)

Thanks for this thoughtful post, TCO. I really appreciate the comments, and they will definitely come into play for planning the next phase of the IEP. -- LiAnna Davis (WMF) (talk) 18:45, 5 December 2011 (UTC)

Project design docs?

I expect that the pilot design includes a discussion of the analysis of results that would proceed from it. Where can I read about that? Jojalozzo 15:25, 3 December 2011 (UTC)

Quality problems extend to uploads as well

It's quite obvious from looking at some IEP uploads, e.g. File:Southasia111.jpg File:SOUTHASIADEV.jpg File:MU and TU of taxation.jpg File:Bowen's diagram 2.jpg File:Loanable funds market.JPG (see old version) File:Pds diagram 2.png File:BusRoute Use case diagram.jpg etc, that students should be using Inkscape to create diagrams and not M$ Word, Paint, a scanner or some other MacGyver-style method. Students should also be required to document their uploads -- most of them I've observed have poor descriptions.

It goes without saying that some uploads are copyvios. MER-C 04:23, 5 December 2011 (UTC)

I don't think requiring the world in general to use Inkscape makes sense. I have never used it. Do you think the average American office worker has?TCO (talk) 04:27, 5 December 2011 (UTC)

It takes as much effort to create a diagram in Inkscape (or some other competent SVG editor, I mentioned Inkscape because it is free) as the other methods. They might as well do it properly the first time and not leave the community to muck around with SVG conversion, cropping out huge amounts of whitespace or trying to comprehend awful "camcorder" diagrams. MER-C 04:35, 5 December 2011 (UTC)

Graphics Lab can fix it if there is strong need. I'm not learning SVG. This site already wants me to be too much of a programmer. Just got done doing some docs in PowerPoint and Word and it was refreshing how much more powerful and intuitive those programs were. Besides we have a lot of crappy photos. Are you off telling everyone to get properly done professional quality photos? Or are the snaps we have so much of OK.TCO (talk) 04:40, 5 December 2011 (UTC)

I spent a couple of years working featured pictures so my standards are high. You can't have a respectable encyclopedia with crappy photos and drawings, though this is a lesser priority than improving the quality of textual content. MER-C 05:59, 5 December 2011 (UTC)

Is there any solution to the copyvio problem?

Far more important than all this data gathering and analysis is the copyvio question.

By early October, this was already a hot issue, and Nitika and Hisham were reporting on heroic efforts that had already been made to explain to students, and to instructors and CAs, that copyvios were unacceptable; but they continued on a large scale well into November. Now, if:

• everything possible was done to explain to students that copying was not allowed, but
• they nevertheless continued copying,

then the depressing but unavoidable conclusion is that, for reasons presumably of a different academic culture,

• Indian students are not a suitable target for a Wikipedia outreach effort.

I hope this is not so, but Wikipedia:India Education Program/Analysis/WMF interviews suggests it may be.

The absolute highest priority before any new phase of IEP must be to work out, and test, a way by which the copyright message can really and effectively be got across. Presentations to classes, briefings to professors and CAs, talk page explanations and warning messages, deletion (sometimes repeated deletion) of copied material, short blocks of repeat offenders have all been tried and were not effective, or not effective enough.

The whole IEP program stands or falls by this: there will be strong resistance to any new IEP unless a convincing case can be made, based on more than wishful thinking, that the problems we have seen will not recur. We cannot contemplate continuing a program to systematically bring in large numbers of new editors every one of whose contributions has to be checked for copyright violations.

JohnCD (talk) 17:01, 13 December 2011 (UTC)

I have to agree with this. The interviews indicate that the copyright violations come from systemic problems in the Indian education system and in Indian society more generally. It's not at all clear that it is even possible to address these fundamental problems in the limited time available for training and instruction.

One thing that is clear is that any solution cannot be based around editors checking participants' contributions for copyright issues. We don't have enough editors doing copyright work and the ones who do aren't going to be happy cleaning up the program's mess. We can't delegate the task to the CAs because few of them have much editing experience at all and a few have even been caught adding copyright violations themselves. Hut 8.5 17:57, 13 December 2011 (UTC)

Each time theses two points have been raised, the the community was informed by the organisers that they were unimportant: the east-West cultural dichotomy as not being particularly relevant (perhaps because it is not fully understood by those who have not lived and worked in Asian education), and the capacity for policing new pages as being no concern of theirs. A better approach, as has been stated many times, would have been involvement of the community, more transparency, and better communications during the earliest planning stages. This would have preempted many of the issues that arose. It is necessary now that these fundamental aspects are given full consideration (not necessarily statistical analysis) before any extension of the programme can be considered. And until they are, the community probably won't want to assist in cleaning up another mess at all, or offer any advice on how it can be done. Kudpung กุดผึ้ง (talk) 07:15, 14 December 2011 (UTC)

As I said on my talk page, if you want a vision of the future, imagine an Indian student copy-pasting content onto Wikipedia -- forever. What Kudpung says is true -- I'd rather file for arbitration than open another IEP related CCI. MER-C 01:50, 24 December 2011 (UTC)

According to the Five Year Plan, diversity now trumps narrow specialism for quality. If "diversity" isn't just to be a vacuous PC term that the WMF hadn't stopped to think through (as if), then this presumably means that we are to embrace the diverse cultural practices around the world in terms of copyright, originality and article quality as well. Andy Dingley (talk) 09:54, 24 December 2011 (UTC)

Tory's report available on January 20

I heard from Tory today that she is putting the finishing touches on her report, and that we will receive it and post it here on January 20 (UTC). -- LiAnna Davis (WMF) (talk) 22:55, 13 January 2012 (UTC)

Quantitative Analysis now available

Spreading the word: Ayush Khanna, a data analyst for the Foundation, has completed his quantitative analysis of the Pune Pilot. His numbers and conclusions are available at Wikipedia:India_Education_Program/Analysis/Quantitative_Analysis. -- LiAnna Davis (WMF) (talk) 18:56, 19 January 2012 (UTC)

Tory's report now available

Hi all, just wanted to alert you that Tory Read has published her analysis of the Pune Pilot. We want to thank Tory for her generous time -- she went above and beyond what we had asked her to do and interviewed many more people than we'd originally planned so she could get a fuller picture of what happened during the Pune Pilot. Thanks to everyone who took the time to answer Tory's questions as well. We'll be using her report to plan the next phase of the pilot program. -- LiAnna Davis (WMF) (talk) 23:43, 20 January 2012 (UTC)