Sleeping Robot wrote: The question for Sal is whether the mule program is to be a serious, methodologically sound, statistically meaningful trial or not.
I agree--except for the "statistically meaningful" part. Statistically meaningful is icing on the cake, but basic validity is fundamental to a sound study. See the distinction that I'm drawing? The whole "statistical" thing may be of no interest to Spyderco, but you can bet a valid study is. I agree with you that it's up to Sal to determine what he wants out of the study. If he's OK with a non-valid study, so am I. You can still draw some
tentative conclusions from non-valid studies. But then the question is, why do 9/10ths of the work for a valid study and then have confounded, uninterpretable data?
Sleeping Robot wrote: Let Sal provide the knives, and let people who want to run a study take it from there.
Oh, I see what you're saying. Normally the subjects
don't run the study. The experimenter does. My impression is that Spyderco is contemplating an experiment, with us forumites as subjects in that study. You're suggesting the equivalent of a meta-analysis, where each subject is in fact an autonomous experimenter running his own study. That approach puts a lot of faith in the methodological ability of each independent experimenter, relies on each of them being blinded to condition, and represents an exponential increase in complexity--but if that's what Sal's after, then so be it.
Sleeping Robot wrote: My final point is that I think simple things should be simple, and people who want to do elaborate things should bear the cost of doing the elaborate thing. I just want to play around with some different steels, and I'd like the steel type to be stamped on the blade. I'd love to see the results of a real study, but I don't want to participate in it...
As Einstein said: "As simple as possible but no simpler." But what is this cost that you are bearing if a code is used? I thought Sal was providing the mules at cost, which is a pretty nice deal! Wouldn't we all like to get all our Spydies at cost--until the company went bankrupt, that is. But it seems to me that Spyderco is bearing the cost (meaning, no profit) in return for valid data from us. That's Spyderco's payback for their effort providing us with these at-cost, one-of-a-kind, exotic steel mules. I think where we disagree is what the study is "about." I think it's about Spyderco getting unbiased user preference information regarding different steels. You think it's about us, the forumites, having fun.
Heck, you might be right. I honestly don't know what Sal has in mind.
Interesting discussion. Hope you and other readers will forgive my long-windedness and understand I'm simply trying to explain what you can & can't get, in terms of knowledge gained, based on how you design a study.