Greetings from Stack Exchange HQ! We have been assigned to evaluate Cross Validated and recommend some methods for making it even better. Please note that this is a more hands-on evaluation, and is not related to the Community Quality Evaluations already done on sites. This is a distinct evaluation project designed to quantify the community and health of the site and recommend ways for you, the core users, to improve it! Yay quantifying and evaluating!

Let’s get right to it--your site is doing really well in a lot of key metrics! Your sites gets good traffic and has a good base of users. It also appears that Cross Validated has a lot of solid content. You have engaged users who care about the site and the quality of content on it. That is awesome. These are qualities and behaviors common to good sites, and we’re pleased to see your site doing well at them.

However, according to our metrics, these areas of the site need improvement.

  1. Answer ratio: Cross Validated is hovering at a 78% answer rate, which is the lowest of the sites we evaluated. We understand that ‘statistics’ is an incredibly broad topic covering many fields of study, but we also believe that in order for this site to flourish on our model we need to find a way to increase the answer ratio.
  2. Voting Ratio: this is tied to the previous item, but it is still an issue and one we have addressed before.
  3. Chat: Activity in your chat rooms is almost non-existent. Chat is a feature we believe strengthens and reinforces community and fosters positive and helpful communication between users.
  4. Lots of viewers: Unusually, perhaps, Cross Validated isn’t suffering from lack of views, but we believe it is suffering from lack of "quality of viewers". There is probably a better way to phrase that, but the upshot is that we need to get some more experienced eyes on the site.

Here are resources and ideas for improving the site and community:

  1. Reach out to your real life networks. Share the site with your co-workers, students, TAs, professors, etc. Share the site and invite your smartest, most knowledgeable stats-crazed friends. Referral Badges are awesome (just saying!).
  2. Chat Events! Whether they are weekly topical events or daily check-in times, increased chat usage is an easy way to increase the appeal of your site to new and experienced users.
  3. Social Media Networking: Cross Validated doesn’t really suffer from lack of views, but it seems like most of those views are primarily from the ‘homework’ crowd and aren’t going to do much to improve that answer rate or improve the breadth of knowledge in our sites. Use your personal social networks (twitter, tumblr, FB, stats messageboards, etc.) to promote the heck out of this site to the people you know would enjoy CV and be helpful resources to the community. We are not advocating spamming everyone you know. However, you, the current passionate community of Cross Validated probably have some contact in some network with the next generations of users, if you know anyone else in the field.

I want to make it clear that none of these suggestions are required. They are merely suggestions for actions that we believe help to grow and sustain your community--things we think will work and help.

We are relying on you, the core user community to implement and spearhead these efforts. Our position and expertise allows us to evaluate our sites, but YOU are the users. You know the site and we want to enable you to improve it.

If you, or other core users of Cross Validated, have any other ideas of what can be tried to improve on these metrics, or any other element of the site you see as problematic, please leave an answer to this meta post. Our team will be working with a selection of sites for the next several weeks and you can feel free to reach out to us directly via email for support.

This improvement push has grown out of a larger effort to create a model for judging the health and potential of each of our sites taking into account all the relevant data we have about your site. Please let us know if there is a metric you think we are missing.

Finally, we know you are excited about the potential for our sites but please keep this meta post focused on things you, as the collective community, can do as opposed to feature requests or larger branding campaigns. We want your ideas, so please make them separate meta posts so we can keep all these discussions focused!

I like this post a lot because (a) it gives really good, concrete suggestions of what individual users can do to help make the site 40% more awesome, and (b) none of these suggestions are particularly painful nor a massive undertaking by any single person. They all promote organic growth...just with a bit of an extra push. :) – Aarthi Nov 30 '12 at 21:25
Could you clarify point 4 ("quality of viewers"), please? Is this about referrals, or what metrics is used to quantify "experienced eyes"? This is important for us because the fact that ASA members or well-known statisticians visit this site (even irregularly) may seem important to some users, including myself. – chl Dec 2 '12 at 12:05
As a frequent questioner but infrequent answerer, my impression is that stats gets a lot of "suggestion" type answers given as comments. They are useful to me as a questioner, but aren't complete "solutions". So my guess is more than 78% of questioners were helped by the site. – Xodarap Dec 16 '12 at 18:49

4 Answers 4

Regarding the "answer ratio" - I think that is partially a function of the nature of statistics and the questions we get. Not so much that statistics is incredibly broad - because the people who answer the questions here have a remarkably broad range of expertise - but some questions either have no answer or are so poorly worded that they cannot be answered (although, if they were better worded, they could be answered).

As for "chat" - well, what is it for? I've been very active here over the past year, but I'm not sure what chat is for, except for the case where a conversation in one of the threads gets a bit astray.

(+1) It is important to emphasize that several chat options are available: our general Ten fold chat room, which is probably what Katey is referring to, and local chat rooms that are created on demand (when one-to-one discussion exceeds 20 comments on a thread). – chl Dec 2 '12 at 12:00
On the answering ratio, I agree with Peter Flom (although I fully support the original post). Compared to - for example - the homebrew stackexchange site (not that I've checked, but I'm guessing), we get a lot of people who have a question relevant for the site but relatively little background and even interest in the techniques available. This lends itself to questions that are difficult to answer well. But having said that, it would need some analysis to determine if those are the sorts of questions that remain unanswered. – Peter Ellis Dec 3 '12 at 19:35
@Peter, & if someone doesn't seem to have background or interest - but just wants an answer to "How should I analyse these data?" - it might seem a little unprofessional to make general assumptions about what they ought to be doing (without the dialogue that usually accompanies consultancy) & instruct them accordingly. Rather as if there were a medical Stack Exchange & some people described their symptoms & asked for a diagnosis. – Scortchi Dec 4 '12 at 0:47
+1 (to this post and all comments so far). Although the medical analogy posed by @Scortchi may seem a little self-serving coming from a statistician, it is apt: if we do not truly understand the circumstances and objectives of a question, we risk giving advice that is worse than bad. Our mistakes are not likely to kill anyone, but that does not relieve us of the responsibility to check that our answers are at once appropriate and will be correctly understood by everyone, including the OP and any future interested readers. – whuber Dec 5 '12 at 17:09
@whuber I sometimes say that answering a statistical question without context is like boxing while blindfold: You might knock your opponent out but you might bash your hand on the ring post. – Peter Flom Dec 5 '12 at 17:27
@whuber, I didn't mean to imply that only statis – Scortchi Dec 5 '12 at 20:41
-ticians should be using statistics, but rather (stretching the medical analogy) that we have questions from doctors, nurses, & first-aiders, who know what they don't know, & from patients, who don't know what they don't know & just want a quick cure. And I think that's why, often, people give a hesitant answer as a comment - they don't want to give a hasty diagnosis based on little information - & the question goes technically unanswered. Also it's perhaps why a lot of answers aren't accepted - the patient's cured & no longer bothered about coming back to the surgery. – Scortchi Dec 5 '12 at 20:50
  1. There are some topics with a lot of overlap with stackoverflow (e.g. data mining questions) and I guess also math.stackexchange and mathoverflow (yes, there are even two sites for math!)

    It's not always clear which site is the more appropriate place, and when to suggest people to use the other site. This of course also leads to cross-posts; these tend to get answered more quickly on the larger site (i.e. main SO).

    IMHO it would be more helpful if this were more clearly defined, and if the "migrate" function was easier to use. The "close" vote does not include migrate to stats. IMHO, it should offer at least all sites the user is also active on, not just some random 5 subset.

  2. My impression is that the easy questions are more likely to end up on main SO, while all the really tough statistics questions do end up here. To me this explains the lower answer and upvote ratio.

    This makes me wonder: should we actually care about these metrics?

  3. I tried chat on main SO, and found it pretty much worthless. Why would I use it here, then? You don't need "live" math, you need some time to think and write down your ideas in a well-formulated way.

This makes me wonder: should we actually care about these metrics? Regarding the upvotes, at least, I certainly don't (anymore). After answering about 200 questions, I find that my own assessment of the quality of my answers is only weakly correlated with the number of votes it receives. I also find that to be true for other users' answers. I think we should just continue to provide good answers and, through comments, etc., weed out bad/misleading questions/answers and if this continues to result in low votes totals/large unanswered %s, then that may just be a "fact of life" on CV. – Macro Jan 9 '13 at 15:28
I agree @macro . In particular, the number of upvotes an answer gets is highly dependent on how many people read the question and answer. This has nothing to do with how good an answer is. – Peter Flom Jan 12 '13 at 22:35
+1. About the third paragraph under #1, I asked once on MSO to include CV on the SO's off-topic list, but they denied arguing there was not enough room. And the subset on their list is not randomly chosen (at least nowadays), but according to the higher migration fluxes (according to them). – Andre Silva Sep 3 '13 at 11:54

This answer focuses entirely on the repeated encouragement to use chat.

I have glanced at chat and (more often) been encouraged by automatic prompts to move a conversation to chat, but I've always resisted.

I naturally have no objection to anyone using chat if they find it useful (or even entertaining) but chat to my mind has various disadvantages, at least for users like myself:

  1. What can be most valuable about threads is the diversity of suggestions, including frank disagreements about how to see or to tackle the problem. The comments system is already excellent and well suited to making smaller points, including those expressed a little more informally.

  2. I have noticed that although chat is very visible if you seek it out, many people lose inhibitions they have in the forum, become digressive, personal, trivial and even offensive. (A recent poster made obnoxious remarks in chat about people who had tried to help him, but gave up frustrated.) They also often become more inarticulate and write more poorly. The value of chat is thus very limited except for the people concerned at the time, so I really don't see that anyone should be encouraged to chat if they don't feel so inclined. (If you want to read this in reverse, and say "But that is what chat is for!" I agree, but it is not to my taste.)

  3. Invitations to chat often come from inexperienced users, intent on one or two things only, getting an individual answer to a specific question (and often getting blessing for something they have already decided they want to do, usually in the form "Is this OK?"). Chat just distracts and detracts from the bigger goal of CV, building up an archive of good answers of use to a large number of people.

(Using comments heavily does carry an obligation to clean up once a conversation is done, removing stuff made redundant by later edits, and so forth.)

I totally agree with 1 (especially compared to email list-serves), but I would note 2 and 3 are more about the people using chat than the actual design of the system. Normative behavior on the site can only be encouraged through experience, so newcomers will always have similar digressions (regardless of the medium). – Andy W Aug 29 '13 at 14:00
Very fair points. Chat is not discredited by those who misuse it. In principle, the tone of chat might be raised by providing better role models; in practice, that is not something that appeals to me. – Nick Cox Aug 29 '13 at 14:17
Two years on, I find I use chat more, to chat, very simply because I now feel I "know" some people, or at least know much more about their work and style and admire both. But the purpose of chat is not well understood: there are frequent posting of technical questions that should be on CV itself and there are frequent prompts to "Look at my question on CV". Those aren't out of order or obnoxious, and almost always they are perfectly polite, but they aren't what chat is for at all, in my view. – Nick Cox Oct 16 at 7:57

I disagree with one of your metrics - I hope that I am allowed to do that if I have a good reason. I think the zeroth metric to return is value. Value can hard to measure because it can mean different things to different people. One person's cost is another person's benefit, sometimes. Metrics should be built to measure and allow value to be maximized.

First off, CV is one my favorite sites. It really is. I can learn. I can be challenged. There is a community of excellent and admirable folks who treat each other well and if you treat them well and with respect you can ask good questions and get good (and sometimes truly amazing) answers. I am on here at least daily.

I'm going to tell you something personal, but brief. I had a neighbor that I tried to build any sort of connection with for years. One day I was making cookies, and had a thought. Even though I had sugar, I went and asked if I could borrow a cup of sugar. They loaned it to me, and in return I brought them some cookies. The next day there was a knock asking if they could borrow some butter. One day! By giving my neighbor a chance to give, I got the chance to have a good friend. It was a good win, and a win-win.

There are a number of different demographics that use the various Stack Exchange sites, and I think that CV as-it-is offers things that other, adjacent technical ones (Stack Overflow, mathematics, computational science) do not - an entrance to the community for "entry level folks" or even "highly capable but new-to-statistics" folks. For me that "entrance" or "accessible participation" has been through the older unanswered questions. I have earned "necromancer" by looking through unanswered questions under my favorite tags and seeing a "gem" that I know just the right answer for. There is a lot (whole honking lot) that I don't know and so having the right (non-zero) questions down my (atypical) alley has been good. If all the questions were answered then I would not have had that opportunity. This community would have been a lot more closed to me. Being able to answer questions, even if they are sometimes less technical, is like CV giving me the chance to share a cup of sugar.

One thing that I have oblique (pun intended) view into here is that this site, I think, returns very great value to the technical community. I will see someone who acts "newbish" asking several questions on a topic, and a few months later will see a published article on the topic. The person who published ins't always the "newb", but I suspect they were encouraged to go down the road they did because of the newb's question. I would bet that between 10% and 25% of tough questions here result in things like masters theses, PhD dissertations, and other peer-reviewed/published work and software packages that result in substantial improvement in the field. Seriously - I think you should apply for an NSF grant on the topic. I think that CV could be responsible for several hundreds to thousands of meaningful steps forward in statistics, the field that is most applied outside of directly engineering a product.

The "Nobel Prize" syndrome causes folks who found buried treasure in their backyard to go try to climb mount impossible. I think that all treasure is buried in backyards. I think that one of the things that CV does is help folks figure out how to look around where they are and find the unknown and fun, and that is where buried treasure is always found.

I have never successfully used chat here. I think that it missed its window. Around 10 years ago it would have been the only way to go. Now, unless it is a longer-term conversation, perhaps like on the back-side of Wikipedia pages, I don't know if it is ever going to gain traction. Is it really such a bad thing if it doesn't catch on?


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .