Mule Study Methodology (Continued)
Posted: Fri May 18, 2007 8:57 am
Sleeping Robot and I were having a long-winded discussion about Spyderco's Mule project, particularly the methodology being used for the study, and we became worried that we were dominating the thread and boring other readers. So we've transplanted the thread here, where we can continue to (1) waste time and (2) insist we are right and the other person is wrong, without irritating readers on the main forum. Of course you are more than welcome to join in. And now let's look at the last posting from the previous thread, where Robot said:
--------------------------------------------------------------------
Quote:
Originally Posted by Bolstermanic
Sal could just leave a few mules un-stamped for the people committed to blind tests--those could be marked in code with paint, sharpie, engraver, etc.
Works for me. Not sure if Sal will want the complication of getting the right number of stamped vs. unstamped blades, but that's one possibility.
Quote:
Originally Posted by Bolstermanic
That was the only reason I could fathom, for the desire that the steel be identified by its name. To impress your friends! "This is real, actual, ZZP-950 steel! Check it out man!" And I was balking because I didn't think that was a good enough reason to confound a study that's going to cost Spyderco a lot of effort.
That's certainly part of it, but it is also customary for Spyderco to identify the steel. My Native says S30V and my Kopa says VG-10. I'd like my mules to tell me what they are as well. I'm kind of traditional that way I guess.
Quote:
Originally Posted by Bolstermanic
The lookup solution could be really simple, too. Let's say a blade is engraved with "W3R4X9" and the "solution" is that if there's a 9 anywhere, it's CMP S99V.
But that's exactly what won't work. Any simple scheme that people can memorize breaks the blinding. If you buy some blades and lend them to a forumite to try out, they shouldn't be able to look at the blades and know what kind of steel they are, even if they have been keeping up on all the discussions. That means no simple mnemonics like the one above, no single code for a single steel, no sequentially assigned numbers across the whole series, etc.
But the problem is even worse than that. People have been suggesting that the blades be stamped with blinding codes so that they won't know what kind of steel they are getting. But this doesn't work. When someone puts in the order for the mule, they're going to know what kind of steel it is. So no matter what is stamped on the blades, they're going to know what kind of steel it is when they get it, even before they open the package! When they open the package, knowing what kind of steel has to be inside, they then see whatever is stamped on the blade, and know what it must mean. The blinding code doesn't accomplish its intended application, but impedes other applications (bragging, tradition, long-term identification of steel type). That is what I'm whining about, and why I want people who care about blinding to take on the job and not impose it on everyone.
The blinding is less about what is on the blade than how the packages are opened. Here's an example. Since for blinding purposes it doesn't matter what is stamped on the blades, why not stamp them with the real steel name? Then, if you want blinded blades, you leave the first blade unopened until the second steel arrives. Then you get a friend to open the packages, cover the steel codes with tape, write A on one steel and B on the other, and give them back without telling you which is which.
What is fun here is that those of us who start out knowing the steel type start discussing what we do and don't like about the steels, which is strongly colored by what we expect to see based on the preexisting knowledge of steel type. Then, in a few months, more valid reports on knife performance come in from people like you. This provides Sal and crew (and us on the forum) with additional information on how the market perceives certain things - true or not - and how that is supported or contradicted by more valid information. Red faces and hilarity sure to follow.
Again, I think this is really Sal's call. The impression I had at the start was that this wasn't intended to be a rigorous study. If he wants to tweak things so that it is, that's cool. But let's be sure that is what is really wanted, and that the tweaks will actually accomplish what they are supposed to.
Best regards,
Ron
-----------------------------------------------------------------------------------
--------------------------------------------------------------------
Quote:
Originally Posted by Bolstermanic
Sal could just leave a few mules un-stamped for the people committed to blind tests--those could be marked in code with paint, sharpie, engraver, etc.
Works for me. Not sure if Sal will want the complication of getting the right number of stamped vs. unstamped blades, but that's one possibility.
Quote:
Originally Posted by Bolstermanic
That was the only reason I could fathom, for the desire that the steel be identified by its name. To impress your friends! "This is real, actual, ZZP-950 steel! Check it out man!" And I was balking because I didn't think that was a good enough reason to confound a study that's going to cost Spyderco a lot of effort.
That's certainly part of it, but it is also customary for Spyderco to identify the steel. My Native says S30V and my Kopa says VG-10. I'd like my mules to tell me what they are as well. I'm kind of traditional that way I guess.
Quote:
Originally Posted by Bolstermanic
The lookup solution could be really simple, too. Let's say a blade is engraved with "W3R4X9" and the "solution" is that if there's a 9 anywhere, it's CMP S99V.
But that's exactly what won't work. Any simple scheme that people can memorize breaks the blinding. If you buy some blades and lend them to a forumite to try out, they shouldn't be able to look at the blades and know what kind of steel they are, even if they have been keeping up on all the discussions. That means no simple mnemonics like the one above, no single code for a single steel, no sequentially assigned numbers across the whole series, etc.
But the problem is even worse than that. People have been suggesting that the blades be stamped with blinding codes so that they won't know what kind of steel they are getting. But this doesn't work. When someone puts in the order for the mule, they're going to know what kind of steel it is. So no matter what is stamped on the blades, they're going to know what kind of steel it is when they get it, even before they open the package! When they open the package, knowing what kind of steel has to be inside, they then see whatever is stamped on the blade, and know what it must mean. The blinding code doesn't accomplish its intended application, but impedes other applications (bragging, tradition, long-term identification of steel type). That is what I'm whining about, and why I want people who care about blinding to take on the job and not impose it on everyone.
The blinding is less about what is on the blade than how the packages are opened. Here's an example. Since for blinding purposes it doesn't matter what is stamped on the blades, why not stamp them with the real steel name? Then, if you want blinded blades, you leave the first blade unopened until the second steel arrives. Then you get a friend to open the packages, cover the steel codes with tape, write A on one steel and B on the other, and give them back without telling you which is which.
What is fun here is that those of us who start out knowing the steel type start discussing what we do and don't like about the steels, which is strongly colored by what we expect to see based on the preexisting knowledge of steel type. Then, in a few months, more valid reports on knife performance come in from people like you. This provides Sal and crew (and us on the forum) with additional information on how the market perceives certain things - true or not - and how that is supported or contradicted by more valid information. Red faces and hilarity sure to follow.
Again, I think this is really Sal's call. The impression I had at the start was that this wasn't intended to be a rigorous study. If he wants to tweak things so that it is, that's cool. But let's be sure that is what is really wanted, and that the tweaks will actually accomplish what they are supposed to.
Best regards,
Ron
-----------------------------------------------------------------------------------