What you test vs how you test it.

I stated recently that WHAT YOU TEST is way more important than how you test it. I stand by that statement. But I've been thinking about the choice recently. I've also been thinking about how much it sucks to tell founders that they are "going the wrong way".

I am normally reluctant to make this sort of deal with the devil... but the more I think about it, the more I believe that it is better for teams at the very early stages to focus on testing the wrong thing well than the right thing poorly. Since they are more likely to actually do it. (curios to hear what +Justin Wilcox +Thor Ernstsson +Melissa Perri  think about that notion and if the means justify the ends).

1. Lean is a skill. If you focus on getting better and better at the actual science behind leanstartup, it will push you to start testing the right things anyway
2. Selfish reason. It is easier for mentors and coaches. When you tell founders that their experiment sucks, it is ok. They are not offended bc they don't think of themselves as scientists they think of themselves as entrepreneurs. They are usually quite receptive to the notion of improving their experiment. If you tell them they are testing the wrong thing, you are actually hinting more at the quality of the "idea" (vs the experiment) which they are way less likely to embrace. 
3. TESTING THE WRONG THING THE RIGHT WAY IS LIKELY TO LEAD TO TESTING THE RIGHT THING THE RIGHT WAY. 
4. TESTING THE RIGHT THING THE WRONG WAY is likely to lead to testing the wrong thing the wrong way (which is fatal). 

To my friends testing predictions in a private group vs public group, I HEAR YOU:) I spent YEARS testing this same issue at neighborrow. I should have been testing more macro hypotheses. Do people want to borrow shit in the first place. Which people. Supply or demand? Is trust as big of an issue as people think? I think you are testing the wrong thing. But I think it will be valuable for you to test the wrong thing the right way. 

As much as I hate for every experiment design to be a web based landing page, in this case it is the best way to test it. 

• Variant A should have a UVP (UNIQUE VALUE PROPOSITION) like "Make predictions to the public and ... be rewarded? be known as an expert? have pride? and 
• Variant B should say the exact same thing but insinuate that the prediction is going to be shared only with a private group (or their friends). 

MAKE THEM MAKE THE PREDICTION - simulate this as well as you can... don't ask them about the circumstances under which they think they will make the prediction. MAKE THEM MAKE THE PREDICTION. 

TRY TO FORGET THE FACT THAT YOU HAVE INFRASTRUCTURE ALREADY. 

All you are trying to do right now is control for the privacy factor using as few resources as possible. You are not trying to scale, you are not trying to get them to create a second prediction. You are not trying to get them to tell their friends. You are not even trying to learn WHY they would make predictions (that is probably what you should test first, but that's not the point of the post).

Don't be afraid to hack it at first. Don't be afraid to use infrastructure that is not your own. Maybe create a small FB group as the variable group (private) and use twitter as the control (public). Then measure the differences in posting