Mule Study Methodology (Continued)

[Bolster]

Sleeping Robot and I were having a long-winded discussion about Spyderco's Mule project, particularly the methodology being used for the study, and we became worried that we were dominating the thread and boring other readers. So we've transplanted the thread here, where we can continue to (1) waste time and (2) insist we are right and the other person is wrong, without irritating readers on the main forum. Of course you are more than welcome to join in. And now let's look at the last posting from the previous thread, where Robot said:

--------------------------------------------------------------------

Quote:
Originally Posted by Bolstermanic
Sal could just leave a few mules un-stamped for the people committed to blind tests--those could be marked in code with paint, sharpie, engraver, etc.
Works for me. Not sure if Sal will want the complication of getting the right number of stamped vs. unstamped blades, but that's one possibility.

Quote:
Originally Posted by Bolstermanic
That was the only reason I could fathom, for the desire that the steel be identified by its name. To impress your friends! "This is real, actual, ZZP-950 steel! Check it out man!" And I was balking because I didn't think that was a good enough reason to confound a study that's going to cost Spyderco a lot of effort.
That's certainly part of it, but it is also customary for Spyderco to identify the steel. My Native says S30V and my Kopa says VG-10. I'd like my mules to tell me what they are as well. I'm kind of traditional that way I guess.

Quote:
Originally Posted by Bolstermanic
The lookup solution could be really simple, too. Let's say a blade is engraved with "W3R4X9" and the "solution" is that if there's a 9 anywhere, it's CMP S99V.
But that's exactly what won't work. Any simple scheme that people can memorize breaks the blinding. If you buy some blades and lend them to a forumite to try out, they shouldn't be able to look at the blades and know what kind of steel they are, even if they have been keeping up on all the discussions. That means no simple mnemonics like the one above, no single code for a single steel, no sequentially assigned numbers across the whole series, etc.

But the problem is even worse than that. People have been suggesting that the blades be stamped with blinding codes so that they won't know what kind of steel they are getting. But this doesn't work. When someone puts in the order for the mule, they're going to know what kind of steel it is. So no matter what is stamped on the blades, they're going to know what kind of steel it is when they get it, even before they open the package! When they open the package, knowing what kind of steel has to be inside, they then see whatever is stamped on the blade, and know what it must mean. The blinding code doesn't accomplish its intended application, but impedes other applications (bragging, tradition, long-term identification of steel type). That is what I'm whining about, and why I want people who care about blinding to take on the job and not impose it on everyone.

The blinding is less about what is on the blade than how the packages are opened. Here's an example. Since for blinding purposes it doesn't matter what is stamped on the blades, why not stamp them with the real steel name? Then, if you want blinded blades, you leave the first blade unopened until the second steel arrives. Then you get a friend to open the packages, cover the steel codes with tape, write A on one steel and B on the other, and give them back without telling you which is which.

What is fun here is that those of us who start out knowing the steel type start discussing what we do and don't like about the steels, which is strongly colored by what we expect to see based on the preexisting knowledge of steel type. Then, in a few months, more valid reports on knife performance come in from people like you. This provides Sal and crew (and us on the forum) with additional information on how the market perceives certain things - true or not - and how that is supported or contradicted by more valid information. Red faces and hilarity sure to follow.

Again, I think this is really Sal's call. The impression I had at the start was that this wasn't intended to be a rigorous study. If he wants to tweak things so that it is, that's cool. But let's be sure that is what is really wanted, and that the tweaks will actually accomplish what they are supposed to.

Best regards,
Ron

-----------------------------------------------------------------------------------

[Th232]

Mmm... I guess that if none of the blades have the steels marked on them, you could leave them alone until you get several blades, mix them up, and then use them at random. Of course, you won't have a hope of figuring out which one is which after, either :p .

Or once you've got several, get a friend to mark them and not tell you which mark means what. Make it a code you won't be able to understand, and I see that as having a real possibility of working.

[druid]

if you wanted to blind yourself just collect a couple blades place a small square of electrical tape over the acidifying markings. Preferably somebody else should do this, to eliminate any possibility of recognition, than shuffle the blades and use them. or even better send them to someone to have them cord wrapped. with the court or material underneath it obscuring the markings, the courts would be different colors and the person wrapping them would have a color code for later reference.

[Bolster]

Robot: "That's certainly part of it, but it is also customary for Spyderco to identify the steel. My Native says S30V and my Kopa says VG-10. I'd like my mules to tell me what they are as well. I'm kind of traditional that way I guess."

Robot, you're changing your reasons for going to war. I don't think tradition is Spyderco's goal with the mule project. Traditionally, their knives come with handles, too. Again, you're making the study about what's in it for you...and I suspect what you want is expensive, exotic steel at the lowest possible prices, preferably with a Spyderco tang stamp to keep value high, which would translate into a rare, desirable, and potentially lucrative item for you. I'm starting to repeat, but IF some sort of "give away" is Spyderco's goal, then by all means stamp the steel with its name. If the goal is for Spyderco to understand what steels are most highly rated--in the absence of marketing hype, prejudice, and tradition--then blinded conditions is the order of the day.

Robot: "But that's exactly what won't work. Any simple scheme that people can memorize breaks the blinding...but the problem is even worse than that...they're going to know what kind of steel it is when they get it, even before they open the package! The blinding code doesn't accomplish its intended application, but impedes other applications (bragging, tradition, long-term identification of steel type)..."

I see your point. If Spyderco decides to send one type of steel to users one month, and then another type of steel two months later, that would certainly cause difficulties with the study. It would be like running one condition of an experiment one month, and another condition two months later. That methodology is sometimes employed in quasi-experimental designs, but it certainly complicates things, because it allows lots of other variables to vary, besides the one you're interested in. IOW, it weakens the control conditions.

The fix is once again simple (you will say it's complex), and that is to distribute more than one type of steel at a time. You will object that (1) this means Spyderco now has to warehouse (big deal! They put a couple mules in a corner for a month) and (2) this won't allow you to ONLY purchase the hot steels you want, while ignoring the steels you've already decided you don't want (making the study, once again, revolve around your preferences).

Let's use an analogy you offered earlier. Imagine a patient who has been asked to join a randomized study to determine the effectiveness of a drug. He says, "OK, I'll join the study, but only if you promise I won't be in the placebo condition. And on top of that, I want to be able to choose which free drug I get to receive. I want to receive the best, most expensive drug. It will be a terrible, horrible, unbearable, unending burden if I am asked to take any other drug than the one I specify." If I were an academic researcher, I would say to that subject, "Thank you and good-bye. Please purchase exactly the drug you want from your pharmacy." If I were a consultant, I would say, "Sure," upcharge the subject so I made good money from his demands, and would not include his data in the study.

Let's further imagine two blow-hard forumites arguing the above case, one taking the side of the researcher ("Valid answers are needed to determine the superior drug for the good of future generations!") and the other, the side of the subject ("How dare you subjugate the needs and desires of this poor victim to your heartless and evil pursuit of science!") In a nutshell there, but for the grace of God, go we.

Robot: "What is fun here is that those of us who start out knowing the steel type start discussing what we do and don't like about the steels, which is strongly colored by what we expect to see based on the preexisting knowledge of steel type. Then, in a few months, more valid reports on knife performance come in from people like you. This provides Sal and crew (and us on the forum) with additional information on how the market perceives certain things - true or not - and how that is supported or contradicted by more valid information. Red faces and hilarity sure to follow."

OK, now you're sticking in the knife and twisting it, and causing me some serious pain. I don't know how much more I can take! Where to start. So many problems with that, but I'll try to keep it short.

Of what use is it to reconfirm one's biases about steels? Marked steels will fall prey to a well-researched human foible called "Confirmation Bias." It goes like this: when humans have a hunch (such as "I'll bet CPM-S99V is better than all the other steels I own") then the information search is conducted so that the hunch is confirmed rather than disconfirmed. Humans have a strong inclination to see data that fits, and to ignore or reinterpret data that doesn't fit. So from the perspective of someone who is interested in "truth," this exercise of reconfirming one's biases doesn't answer the important questions. It's just an exercise in self-pleasuring. Which can be gratified by just buying the steel you want, and caressing it.

Now some people will say, "I'm not biased." But that's like saying "I'm not human." To be human is to be biased. The scientific method is designed to reduce some of that bias. You try to *disconfirm* your hypothesis. You remain blinded to condition. You do what you can to obstruct those many human biases that are interfering with seeing reality as it exists.

Sal and co already knows how people perceive different steels. So do you and I, to a lesser degree. (Well, lesser degree for me; you may be very well informed.) There's a great deal of opinion put out there on different steels. So I would argue the only really imporant part of the project is the part where people are blinded to conditions and keeping biases in check, rather than wallowing in them.

When Doc Snubnose publishes his blinded and valid results, and forumite Mr. Bias has tested those same steels intuitively, wallowing in his biases, and finds he disagrees with Snubnose's results, do you think Mr. Bias is going to say, "Gosh, red face here, my bad!" No--he'll tell Snubnose and all the other methodologists that they're full of it. Then Mr. Bias will switch over to a case history (the Case History Bias, to be exact), and show how his one experience to the contrary nullifies Snub's 300 blinded trials. Then Mr Bias will criticize the studies, probably erroneously. A recipe to increase darkness and confusion, rather than reality & light.

Although it would make for some good entertainment, I agree.

Robot, you are obviously an intelligent dude and I have greatly enjoyed crossing swords with you. You rock. :D

[Sleeping Robot]

Robot: "That's certainly part of it, but it is also customary for Spyderco to identify the steel. My Native says S30V and my Kopa says VG-10. I'd like my mules to tell me what they are as well. I'm kind of traditional that way I guess."

Robot, you're changing your reasons for going to war.

To be picky, "also" means I'm providing additional reasons, not changing the original reason. But that's a side-track.

Again, you're making the study about what's in it for you...and I suspect what you want is expensive, exotic steel at the lowest possible prices

Yes.

, preferably with a Spyderco tang stamp to keep value high, which would translate into a rare, desirable, and potentially lucrative item for you.

No.

I have zero interest in the resale value of the mules. I have no intention to resell any that I buy. It's really about getting the exotic steels, and learning more about them. (I'd love it if Sal was able to provide heat-treat information about them as well - for example, with the 51200, what is the goal for martensite vs. bainite formation and what time/temperature profile did they shoot for to get it? I don't expect to get that info, although I would love it if I could.)

Because I'm in it for the steels, I'd like the steels to be marked according to their name.

I'm starting to repeat, but IF some sort of "give away" is Spyderco's goal, then by all means stamp the steel with its name. If the goal is for Spyderco to understand what steels are most highly rated--in the absence of marketing hype, prejudice, and tradition--then blinded conditions is the order of the day.

I don't know how the goals may have changed over time, but we can see the original goals by looking at the thread Sal started on bladeforums:

http://www.bladeforums.com/forums/showt ... p?t=445613

It's addressed to "Steel Junky's", and among other things it says:

...We coud probably do 1 - 4 runs per year. Each "Mule Team" knife would be engraved with the type of steel.

We would determine steel to test and Rc on the forum before making each run. We would cut them out, heat treat, tumble (cheap), grind, sharpen and ship. Some steels have to be ground before heat treat. No handle, no sheath. All would have to be pre-sold (I don't want to inventory.) which means some of the group would purchase more than one to be able to have some available for those not signed up. We would keep the cost as low as is possible. Costs would vary depending on mfg costs.

A few posts later in the thread, someone explicity asked what was the goal and Sal said:

...I guess the goal would be to be able to provide a "piece of the stuff" in a fairly consistent format, at an affordable cost. Same thickness, same grind, same tumbling, same edge angle.

We would shoot for an Rc on which the afi's agreed. CATRA and/or Charpy might give some basis for comparison from lab testing. Real world use would be a personal experience. A topic of conversation.

I think your ambition "for Spyderco to understand what steels are most highly rated--in the absence of marketing hype, prejudice, and tradition" is beyond Sal's original goals, which I saw as being to get exotic steels into the hands of the afis and steel junkies as cheaply and easily as possible.

I'm not saying your ambition is a bad goal. I think it would be wonderful. But I think it represents a change in goals, and contradicts other goals such as rock-bottom cost and "a topic of conversation" for the steel junkies. For me, those original goals are more important. But if Sal wants to change the goals around, that would be cool. It's his party. I know I can't sign up for a whole set of different steels in advance, but hey, there are lots of things I'm not going to get done before I die.

Robot: "But that's exactly what won't work. Any simple scheme that people can memorize breaks the blinding...but the problem is even worse than that...they're going to know what kind of steel it is when they get it, even before they open the package! The blinding code doesn't accomplish its intended application, but impedes other applications (bragging, tradition, long-term identification of steel type)..."

I see your point. If Spyderco decides to send one type of steel to users one month, and then another type of steel two months later, that would certainly cause difficulties with the study... it weakens the control conditions.

Yes.

The fix is once again simple (you will say it's complex), and that is to distribute more than one type of steel at a time. You will object that (1) this means Spyderco now has to warehouse (big deal! They put a couple mules in a corner for a month) and (2) this won't allow you to ONLY purchase the hot steels you want, while ignoring the steels you've already decided you don't want (making the study, once again, revolve around your preferences).

Now, now. No fair putting words into your opponent's mouth. (But congratulations on your future-telling abilities, you were mostly on-target.) ]
...(2) this won't allow you to ONLY purchase the hot steels you want, while ignoring the steels you've already decided you don't want (making the study, once again, revolve around your preferences)
[/QUOTE]
Pretty close. How about we change "preferences" to "preferences, budget, and what you can get your wife to agree to"?

Many of the forumites said they would be happy to sign up for the full series, but many of us did not. I'm sure budget and spousal acceptance entered into a few of those decisions. They certainly entered into mine.

I think Sal's original plan to agree on a steel beforehand and then let people buy or not buy was motivated by the desire to let people still have access to some exotic steels even if they don't have the cash to buy into the whole series. Sounds pretty good to me. I'd hate to see a bunch of people excluded.

Let's further imagine two blow-hard forumites arguing the above case, one taking the side of the researcher ("Valid answers are needed to determine the superior drug for the good of future generations!") and the other, the side of the subject ("How dare you subjugate the needs and desires of this poor victim to your heartless and evil pursuit of science!") In a nutshell there, but for the grace of God, go we.

Oh boy.
A more accuracte caricature of my position would be "Fine, knock yourself out, but don't use my budget, don't delay my schedule, and oh, by the way, your methodology needs some work".

...Confirmation Bias...
... Case History Bias ...

Thanks for the very clear explanations of what is going to happen. I learned a lot there.

It may, or may not, surprise you to hear that I completely agree with your prediction. So why am I arguing? My points are:

1) The mule program itself is not a controlled trial, although people who want a controlled trial could run one with the blades they get from it, and I'd love to see them do so. I'd even help a bit with design and critique of the trial.

2) Some of the original goals of the mule program, such as letting people buy only some of the blades in order to keep it cheap for them, mean its going to take some extra work to run a controlled trial.

3) The people who want a real controlled trial should recognize that the extra work is a burden they are taking on voluntarily. They should not try to foist part of the cost and effort onto others who are not participating in the controlled trial.

4) The suggestion of stamping a simple "code" instead of a steel type onto the blades won't accomplish its stated goal, but will annoy many of us outside the trial. (If the coding actually worked, that would be one thing. But stamping "A" instead of "51200" is just ... inadequate).

5) We can't stop people from being people. Everyone participating in the mule program is an adult. They have been warned about the mistakes they are likely to make. More reliable data will be forthcoming in the future. Your moral obligation has been discharged. That's good enough for me. It's not perfect, but re-read point 5.

Bolster, I've enjoyed the thread and learned a lot from your professional experience.

Later,
Ron

[Sleeping Robot]

What I would really like to talk about in this thread is the question: If there is going to be a study to compare the blade steels, what should be be measurements for comparison?

I was thinking that some of the things mentioned in http://www.cutleryscience.com/reviews/b ... sting.html
would be good to bring in - measures for testing sharpness, edge holding, sharpening ease, etc. Are others thinking along the same lines?

Best regards,
Ron

[Bolster]

We can't stop people from being people. Everyone participating in the mule program is an adult. They have been warned about the mistakes they are likely to make. More reliable data will be forthcoming in the future. Your moral obligation has been discharged. That's good enough for me. It's not perfect, but re-read point 5.

Whew, thank goodness my obligation has been discharged. I'm not accustomed to giving this much free advice on study design.

Unfortunately there's a lot of research that indicates forwarning people of their own biases and thinking errors doesn't reliably reduce them. That's why modern studies aren't run with a good warning followed by open disclosure of how the study is run. Valid studies tend to keep experimenters blinded and subjects (oops, PC police: now they're called "participants"...eye roll :rolleyes: ) in the dark until the study is run, for the sake of validity. As you correctly point out: less fun, more work.

But I think we've given the mule idea a good shaking, which IMHO it needed. Often, in the course of these types of discussions, the original goals of the study do change. In the many studies I've been part of, discussions like this usually lead the researcher to become *more* rigorous, not less. That's one of the benefits of these types of arguments. If it hadn't been for you making it a fun and challenging thread, Robot, I would have gone silent long ago. (I can hear distant cheering of the idea that Robot and I go silent.) I should point out that neither Sal nor Spyderco have asked me for advice, so I'm simply whistling into the wind. I wish them the best with the Mule project, whatever their goals may be. I hope Robot and I have helped sharpen those goals a little...if anybody's listening up there.

For the few spectators who have tuned in throught this conversation, what Robot and I are doing is a pretty close duplicate of what happens whenever a serious study is proposed in the sciences or in academia. Usually the study design is presented to a "murder board" staffed with smart people who try to tear the study to pieces. So this sort of wordy, adversarial mental combat is merely standard operating procedure in the sciences. Whatever bloody remnants of the original idea that remain standing, become the basis of the next revision of the study. Eventually you get to a point where the lines are hardened, people start repeating what they've already said, and the study is either aborted, or goes forward with modifications.

So now let's see what happens. Thanks Robot for you insights and good points.

[Th232]

For the few spectators who have tuned in throught this conversation, what Robot and I are doing is a pretty close duplicate of what happens whenever a serious study is proposed in the sciences or in academia. Usually the study design is presented to a "murder board" staffed with smart people who try to tear the study to pieces. So this sort of wordy, adversarial mental combat is merely standard operating procedure in the sciences. Whatever bloody remnants of the original idea that remain standing, become the basis of the next revision of the study. Eventually you get to a point where the lines are hardened, people start repeating what they've already said, and the study is either aborted, or goes forward with modifications.

I was considering a career in research. Now I'm reconsidering...

[zenheretic]

I quit reading at "please send the mules unstamped"...which is pretty early in this thread. Frankly, I doubt most people are NOT interested in mules without the name of the steel on them. If feel the need to conduct some sort of double blind inspired torture test simply have an uninterested party wrap the markings with tape. When you have several tape covered mules, have at your tests. You'll need to subject your mules to your tests on the same day anyway, in the interests of temperature,humidity, star alignment etc.

No offense, but unless you have machines to duplicate swings, force, etc. getting to "Scientific" is rather pointless anyways. If Sal wants scientific, he'll do it at Spyderco's Super Secret Area 51 Testing facility.

Send me your ground, your polished,
your tempered steel yearning to cut free.
The shiny mules of your foundry store.
Send these, the stainless, rockwell hardened to me,
I lift my knife arm aside the mules more.

[Slick]

I agree with zenheretic. Sal is making this offer to get user feedback. It will be biased, unscientific, and real world. Like most all of our feedback here. I am sure Spyderco will also have propper testing done.

This needs to be as simple as possible for Spyderco.

[jaislandboy]

...
Send me your ground, your polished,
your tempered steel yearning to cut free.
The shiny mules of your foundry store.
Send these, the stainless, rockwell hardened to me,
I lift my knife arm aside the mules more...

wow Zen....*choke*.....that was beautiful..... :p :D

[zenheretic]

wow Zen....*choke*.....that was beautiful..... :p :D

I may have drawn inspiration from the efforts of previous wordsmiths... :p

[Bolster]

Hey, guess what. The Spyderco Yang knife (made of VG-10) is not marked regarding steel type anywhere on the blade. Now that I noticed it's unmarked, it probably won't cut as well.