Fans of X-Guard (the fight gear company, not the guard) have been clamoring to know why it’s missing from the results. I owe it to them and everyone else to tell the story of how X-Guard (unfortunately) got dropped.
When the survey had just been closed and we took our first look at the rankings, here’s what we found:
Wow, X-Guard at #1! They weren’t even on my radar, so I was excited to see an unexpected company at the top. They’re just the kind of underdog I wrote about before:
We’d like to see these underdogs get the attention they deserve (as well as the fly by night companies), but as the survey’s administrator, I’m in a peculiar spot when it comes to spotlighting certain brands. Basically, I can’t.
In promoting this survey, I have to be careful not to skew the results or ruin its impartiality. For example, I try not to mention any particular brands, and I haven’t asked any companies to promote it to their customers, fearing a flood of overly positive reviews (though we can account for this bias on our end in the analysis.)
That said, I do want to see more reviews for certain smaller brands so we can see how they rank against the big dogs. In fact, I have a whole list of them, but I’m not sure I should share it.
So what can I do? The strategy is simply to push the overall number of responses up and hope that with enough people giving reviews, the little companies will get the reviews they need.
If X-Guard’s rank was legit, I’d be very excited. But at the same time, I needed to figure out how this happened. Since there isn’t a significant difference in ranking between the other top brands, how did X-Guard pull so far ahead? If I couldn’t explain this, no one was going to take the results seriously.
Travel back in time to the 2010 survey. Break Point took first place (with a very wide margin of error.) Like most, my response to this was “Who the hell is Break Point?” They edged out Shoyoroll for the top spot, and this was before Shoyoroll was having problems hitting their pre-order release dates.
People didn’t trust those rankings, and I don’t blame them. My suspicion was that Break Point got a bunch of fans to submit positive reviews. I didn’t run that survey so I can’t be sure. (We could rerun the 2010 data, but I think the new results make this unnecessary.)
While designing the 2011 survey, I was determined to be able detect these things, as well as snoop out potential fraud. If an odd brand made it to the top again, I wanted to be able to justify it.
One of the simplest things to monitor are referring URLs to see how people are getting to the survey. From this, I found that many companies had linked to the survey from Facebook, which had been expected (and has been found to not significantly affect rankings.)
X-Guard was one of these companies, but they went a step further. They were running a contest to raffle off a sold out gi to fans who took the survey and voted for them. And boy, did it work.
By their nature, online polls and surveys can be overly influenced by a company with a strong internet presence. We tried to account for this bias, but we were unprepared to deal with the high concentration of extremely positive reviews driven in by X-Guard’s “gi giveaway” contest.
How do I handle this? X-Guard didn’t do anything against the rules, since there weren’t any. Frankly, I hadn’t planned on this. I didn’t feel X-Guard meant any harm (and after talking with them, I’m sure they didn’t.) And I don’t think their fans were being dishonest in their reviews. X-Guard just wanted to send their fans over and offer a cool prize for showing support.
This was a sticky situation, because I didn’t want to punish X-Guard for something they didn’t know they shouldn’t do. After discussing the issue at length with my stats guy and a dozen other BJJ gear bloggers, we arrived at a decision to remove X-Guard from the official rankings, but to explain what happened and publish the unedited results with them at the top (which is what you’re reading now.)
The survey was designed more as a census than a popularity contest. But the truth is we can only collect data on gis that people want to talk about. In a magical Christmas land, rather than doing an open survey, we’d have a panel of 1000 expert reviewers with no affiliations to the gi companies. Or better yet, we’d get 1000 randomly selected jiu-jiteiros to rate all their gis, not just the one they choose!
We don’t want to set a bad precedence for future surveys where companies could bribe their way to the top. The next survey will carry rules like “Companies can link to the survey, but can’t offer any incentives to get good reviews” (unless we find a way to get the randomly selected reviewers plan to work.)
(While we’re talking about dropping brands, you should know that another company was removed after we found that half of their reviews came from an IP address associated with the company. These reviews all rated their brand unusually high and other brands unusually low.)
I was very sad about having to remove X-Guard from the rankings. They are just the kind of small company I want to get exposure from the survey. Their owner, Elmer Santelices, was surprisingly understanding when I told him how I had to drop him, and he even apologized to me about running the contest. (He also did an interview with me.)
After taking the survey, a reviewer wrote me to ask why I didn’t ask about quality of customer service. He said he’d stopped buying gis he liked because he hated dealing with the company.
This had been considered when designing the survey, but ultimately scrapped. It only worked if you could only buy gis directly from the company and not through many other sources like online retailers, brick-and-mortar martial arts stores, school pro shops, Craigslist, eBay, person-to-person barter and so on. Maybe we’ll try to tackle that in the future.
Nevertheless, he brought up a very good point. With the quality of gis converging, the product can take a backseat to other factors that affect buyers’ decisions. These include:
- Marketing, advertising and branding
- Fantastic customer service
- Quick and courteous communication
- Community engagement
- Word-of-mouth reputation
- A motivated fan base
One of the advantages a small company has is the ability to more easily interact with their customer base. As a company grows, this becomes more difficult, especially as they start being sold through distributors and lose the direct connection to their customers. At the same time, more sales means more potential for problems or quality control issues.
This could explain why so many small companies made it into the Top 33 ahead of the big names like Atama, Koral and Gameness. The quality of the small guys probably isn’t so much better, but if they stay on top of their game, they can keep a higher percentage of their customers satisfied and fanatical.
X-Guard’s ability to leverage this fact definitely shows what’s possible when your customers are behind you.
Don’t miss our interview with Elmer Santelices, the owner of X-Guard Fight Wear!
Pingback: A Better Way to Compare BJJ Gis | BJJ Gear | 2011 BJJ Gi Survey Results | Aesopian BJJ