# How to stem the tide of duplicate Qs

Please note that I searched on the keywords "duplicate" and "exact-duplicate" before posting this :-)

It seems to be a point of pride for some that this site has hosted so many thousand Qs. But this counter produces rather empty numbers since a great many of these Qs are duplicates. Almost daily we receive new questions that are variants of "How can I analyze data from survey rating items?" or "How should I choose predictors for my model?" To deal with this, should we...

...find a way to make the FAQ more prominent? ...prominently display links to keyword searches for the most popular terms, such as Likert and Variable-selection? ...ask people to affirm that they have read the FAQ and/or conducted a keyword search before allowing them to pose a Q?

I realize that some people will object to anything that acts a disincentive to participation by new people. But think about what this site is becoming when we allow these duplicates or near-duplicates to populate so much of it. It makes it harder for the answers to stand out.

-
Community members like you have been very helpful all along in (a) voting to close apparent duplicates and (b) flagging questions as possible duplicates. As a moderator I am constantly grateful for that effort and welcome your suggestions to help us avoid duplication and redundancy. Note, however, that many of the suggestions here appear to apply to the entire SE model itself and perhaps should be made in the MetaSE forum. – whuber May 7 '12 at 14:48
I really like seeing this post! I have searched through just the FAQs looking for interesting questions and find that they are so cluttered that it is hard to find anything. i think I would have to rely on a moderator to point me to what I want. – Michael Chernick May 9 '12 at 16:26
I also find it irritating that regardless of how good a question is if a moderator or a member sees a different place the question somewhere else. Bill Huber's comment above may be an example unless he means that it would nice to post this at MetaSE in addition to here rather than in place of here. – Michael Chernick May 9 '12 at 16:29

Some of our "sister sites" in the SE network currently receive much more traffic and there may be lessons learned that we can transfer here.

Here are some that come to mind:

1. Judicious use of the faq tag. We don't appear to have a faq tag. But, the math website has taken to marking particularly common questions with this tag in order to facilitate quick identification of the most common duplicates. All such tagged questions also show up on a special tab to the right of the "featured" tab on their site. We could potentially use such an approach here.
2. Meta list to track major topics and the most common questions within each topic. Again, this comes from the math site and I find it to be a pretty useful feature, as long as you know it exists. The thread List of Generalizations of Common Questions contains the generalized versions of the most common questions by topic. (They even have a "statistics" topic, though the questions there are closer to elementary probability.) I think if we wanted to do something similar on this site, we'd probably want to adapt it a bit, but we get lots of repeat questions on linear regression, AN(C)OVA, Likert scales, basic time-series, etc. It would take some work on the front end to put together, but once started would be fairly easily updated. Note that each such question is also supposed to be marked faq.
3. Searching on tags and keywords. The tags can at least help narrow down the topic when searching for duplicates, but it's not easy, especially, e.g., if you're looking for something under r or a similarly popular tag. The search facilities are pretty poor, in my opinion, but whuber and others have some nice tricks for squeezing as much out of them as possible. (Helpful hint: Doing an empty search via the search box reveals lots of handy options on how to narrow your search.)
4. Google. The built-in search facilities being what they are, often a good Google search can uncover a duplicate pretty quickly. One particularly useful feature is to enter site:stats.stackexchange.com at the end of your search to limit the search to our domain. I've used this pretty often to find a duplicate for which I vaguely recall some feature (like an interesting phrase), but not the authors or title.

I do not see an easy solution to more rapidly acclimating new users. They all have different incentives and different experiences with technology. Some come seeking immediate help and wouldn't be bothered to look at the FAQ anyway. I fear that is a problem that will persist. Asking politely whether they've searched for duplicates can help, but ultimately I suspect the burden will be on the active members to identify duplicates and vote to close.

Thus, I believe our most productive angle is to try to have some simple tools and procedures that accelerate our work.

-
(+1) The meta list seems like a very fruitful path forward for everyone, as it should become a repository of useful references. It seems a bit unwieldy though, any thoughts on whether the format should be copied (or modified) here? – Andy W May 10 '12 at 12:22
@AndyW: I don't have any really strong opinions on the format (yet). I think they've done alright under the basic constraints imposed by the site. I like the idea of having everything in a single question or answer, rather than split across multiple answers, if for no other reason that it keeps things in a consistent order that can't be "manipulated" by voting things up or down. I'll give that some more thought. – cardinal May 10 '12 at 12:32

The answer is simple, however it is also an appeal for everyone -- rather giving the same answer again, use close as an exact duplicate option or (if you can't) flag.

-
The only problem with that is, to use that option one has to go and find the duplicate. I've done a fair amount of this over the last year and a half and I find it tedious to sort through similar threads to find the best one to designate as the one being duplicated. Call me ungenerous, but I think the burden of searching through prior threads should be on the person asking the question. – rolando2 May 8 '12 at 0:02
@rolando2 Agreed; I didn't want to be harsh, rather to promote closing as a duplicate. Improving automatic anti-duplicate shielding is of course a fair topic, but as whuber mentioned it fits better on Meta SO. – mbq May 8 '12 at 18:43
Huber said MetaSE. Are there two sites, MetaSE and MetaSO. If so it seem that you moderators can't even agree on where to move something. Why don't you cut down on so many of these suggestions! I think rolando2 is hitting on something that has bothered me and I have only been here 8 days. Much of what people suggest that we do to solve a problem is tedious and time consuming. Asking us to work harder is not the way to solve a problem. – Michael Chernick May 9 '12 at 16:34
@Michael: mbq and whuber were referring to the same site, i.e., the "global" meta one. – cardinal May 9 '12 at 16:42
@Michael, concerning moderator consistency please see the last part of my post at meta.stats.stackexchange.com/a/1098. – whuber May 9 '12 at 16:52
A great suggestion @mbq but not a realistic one. rolando2's reason is the main one - it's a pain to search sometimes, especially when the thing it duplicates used non-standard terminology. Also, there are some posters who've amassed an incredible number of reputation points by essentially answering the same few questions over and over and over again - so, there's an anti-incentive for some posters to flag as a duplicate. – Macro May 9 '12 at 17:50

Comments on questions are so regular I think it is reasonable to suggest the OP to search some of the prior questions on the site (of course in addition to closing the question if a duplicate is known).

This might include re-tagging with more appropriate tags, which should change the list of related questions on the right hand side I believe, or being even more explicit (e.g. you should peruse the tag for similar questions).

Of course such comments can be taken in a negative light depending on inferred tone (whether malice was intended or not). Perhaps we can add to our library of helpful responses a response for such a situation. My first stab at a response is;

Your question on ?X? is fairly broad, and has likely been covered in previous questions on the site. Have you perused the related questions on the site? See for instance [tag:your-tag-here] and/or [link to some search term](your link here).

I agree it can be annoying, but it just seems to be another part of the community moderation in an attempt to up-keep the site. This isn't terribly helpful in deciding when such prodding should be undertaken, but I guess I will leave that to the better judgement of regular contributors to particular topics.

-

I do see your point, but I guess it doesn't bother me as much as it does you. I also suspect that it's pretty easy to overestimate the number of questioners who are 'lazy' and not scanning through enough of the suggested 'related questions', although there's no doubt that happens. Moreover, I'm disinclined to vote to close questions that aren't egregious (ironically, I have done so several times over the last few days, though). Among other things, I suspect you're right that it would become a disincentive.

I think that part of the issue here is that there is an unavoidable tension between the simple question-and-answer nature of the site, and the intention to have the site build up a repository of information over time. Some of this is indicated by the discussion of how CV sees itself; there's a lot of overlap between this site and other venues on the internet. We want to build a lasting knowledge base, but a lot of new and one-time users will necessarily see this as a place to ask a simple question without thinking of those questions as something that gets stored for posterity.

In my opinion, two strategies are appropriate. I agree with @AndyW and others who suggest using comments that suggest other questions. Setting aside the effect this has on the questioner, it's been tremendously helpful for me. I can't count how many times I've gone to a question, and seen the comment '[this] question is pretty similar and has some really good answers', and then navigated over to the other question, and learned a lot more about the topic than whatever I would have said. Again, maybe that questioner never goes there, but that doesn't bother me; maybe a subsequent person ends up stumbling across the link and does end up reading the good question with the comprehensive answers. If nothing else, having lots of links pointing towards the best of CV helps Google to find the most helpful discussions.

It is definitely true that certain types of questions / issues come up over and over. Since I only know 3 or 4 things about statistics, I often end up over time with lists of questions on just a few topics where I've given similar answers. Eventually, I put together what I think of as my clearest and fullest statement on the issue. Afterwards, when those questions still arise, I can give a perfunctory answer and link to where I've discussed the topic more thoroughly. (I have also linked to other contributors' answers, of course, but I'm most familiar with what I've written.) For me at least, having tried to answer a question several times helps the ideas 'gel' in my head--I may well not have been able to give my best answer the first time. Furthermore, I think this process does tend to elevate better questions and answers over time, and is part of the resolution of the tension I mentioned. I guess what I'm saying is that I think the process is actually fairly healthy and organic, even though I can appreciate the frustration in the interim.

-
I feel gratified by the way so many of you have taken this seriously and given such thoughtful answers. – rolando2 May 11 '12 at 19:32

I noticed that when I start writing a question there is an automatic list of "similar questions" based on the words I use in my question. This can be helpful in not duplicating questions when the association of my word to words in other posts is good but it doesn't always work.

-
This is useful for statisticians, since we are more likely to use similar terminology but, as I'm sure you've seen, some of the laymen on this site use very odd ways of describing very standard problems and the filter has a tough time catching the duplicates in that case. Also, unfortunately, I'm not so sure that some of the more transient users of the site even look at the list. Especially the "lazy" ones you referred to in the other thread, who want their answer now and don't care to search through the existing list of questions... – Macro May 9 '12 at 17:47