The more I think about it, the more I’m excited about the idea of a personality matching shootout. It’s about time we had some true scientifically proven, apples-to-apples comparisons between the various systems in use on today’s dating sites.
Any shootout of this magnitude runs the risk of failure at any time. If we’re going to do this, we need vendors and participating dating sites to be “all in.” I say “we” because its going to take several people to help set it up, run the tests and verify the results. I’m more than happy to participate and help make it happen in whatever role makes sense. Obviously there are privacy concerns, non-disclosure agreements and other hurdles to overcome, but if people are really serious about getting involved, I think we can find common ground for the sake of moving forward.
It’s important to make the shootout meaningful, not just for dating sites, but more importantly, the singles that use them.
Open it up to as many vendors as possible. Here’s who I would like to see participate in the shootout: PerfectMatch, Chemistry, eHarmony, sites running ThomasKnowsPeople’s test and Yahoo! Personals (powered by WeAttract). Let us not forget Fernando Ardenghi, our esteemed colleague in Argentina. I hope he brings his best matching system to the party, as we’re all tired of hearing about him go on about it for years ;-) This is not a complete list by any means. Who else should be invited, who wants to participate?
All companies involved are going to have to set aside some resources to make this happen. We’re not evaluating word processors, but a highly complex systems with lots of moving parts that no-one has ever reliably compared out in the wild successfully, at least to my knowledge. Databases need to be connected to, audit trails tracked and software needs to be written. We need to be able to define and ultimately track progress, bot successful and unsuccessful matches. Then there is the issue of identifying participating websites and a small group of overseers to keep things fair, balanced and on track.
I can see the potential for problems such as creating a list of evaluation criteria and how the results are measured. I’m not a Dr. But I hope to facilitate the discussion between participants about what gets measured and how. Then there is the issue of ranking the results. These are complicated issues that will require all involved to give a little bit in order to make the evaluation as fair as possible.
Perfection is not the goal, but I think we can all get along enough to have a solid set of results that can be used for business deals, marketing materials and a baseline for improvement. And more happy people that can change their Facebook status to “In a relationship”, because thats the end goal, right?
Dating sites will want a list of integration criteria. What resources are required to integrate with each matching platform? What are the expected overall integration costs? How many man-hours from start to finish? Are we talking complicated database queries or “Userplane simple” plug-and play? Where does the matching platform reside? On the dating site’s servers or offsite? How much work is it to pull data from the dating site into the matching system? What does it take to get the results back into member’s profiles? Can you search based on the test questions or are the results baked into search only? I’m just listing a few things off the top of my head here. Any CTO or IT department at a dating site is going to have a lot more questions that what I’ve rattled off here.
Are systems based on monthly licenses or per-query fees? How will a dating site’s costs be affected as they grow? What happens if something blows up in the middle of the night? Who’s responsible for fixing problems?
If your matching platform is wildly accurate, but costs $100k a year and takes 6 months to set up, how does that compare to a $1,000/month service thats less accurate but takes a week to get working? I’m speaking in generalities here, but you get the idea.
Better matches for singles are based on completely different criteria than two business dancing around the table trying to figure out how to enter into a lucrative long-term partnership and all of the associated business rules.
Are the matching platforms a drop-in replacement for a site that already has a system in place, or better suited for sites that don’t have personality testing?
Part of me thinks the tests should go on brand new sites, in part to see how tests work with small numbers of members and scales as the sites grow.
Maybe I’m thinking too much, maybe we make the shootout as simple as possible. For every 1,000 singles, how many meet someone and go on more than 10 dates within three months? Rank the results and we’re done.
As a consultant, hand-waving about tests performed in a controlled environment are not realistic compared to the reality of being part of a live Internet service. That’s why I want to know about the details. And this is a dating industry blog after all. But still, it’s all about more effective matching. That is the end game here.
Added bonus: these general criteria will also be useful to background check, profile and content management/compliance companies.
There will always be a certain amount of inexactitude, but we have to start with some core concepts and ideas and build from there. There is always the chance that the major dating sites won’t want to participate, but I see that as their loss.
There are several top-ten dating sites that have no matching system in place. they spend 10′s of millions of dollars on advertising and couldn’t care less about what people do on their sites, only that they click ads and their credit cards re-bill every month. Perhaps the results of the shootout will put them in a position to consider integrating a matching platform.
What do you think?
No related posts.
{ 23 comments… read them below or add one }
Joel, that wasn’t mocking, that was an accurate reflection of your original post. ;-)
I’m happy to participate, although I would ask each participating company to contribute to cover my time if I were to run the whole process.
These things don’t happen by themselves and this shootout could easily take a month or two to set up. I will ponder this in my next post on the subject.
Shoot me an email with how we would integrate your system with a dating site. Might as well start with you.
David, is that you? If so, you ask, “What do I think?”
I think, after mocking me a bit, finally, you get it.
I’m a classic introvert. When it comes to people, I really, really love dogs. So, rather than me defaulting into my “through and through” NY attitude (actually, Bensonhurst), why don’t you, central to the dating business, do the organizing?
Is should be obvious–I’m in!
Joel
Dave,
You’re proposing something I proposed myself several years back. No surprise, there were no takers then. Of course, virtually all companies will not participate in something like this now for obvious reasons (let alone the logistical issues that would be substantial). It’s not their loss if they refuse to participate; only a chance for a loss if they do.
That said, I’ve several impartial colleagues of extremely high standing who understand well the nuances of this type of research and would probably help. It’s more complex than you envision. A large grant would be also be required — the costs will be exceptionally high.
On a related note, I’m glad — nay, relieved — that Dr. Joel is finally becoming amenable to the topic of “challenges” (I initiated the notion with him not too long ago but was stonewalled).
Something big IS about to happen. Expect an announcement soon.
Thanks,
James Houran, Ph.D.
http://www.OnlineDatingMagazine.com
David, you don’t like the term “mocking” as an accurate reflection of your language in relation to my challenge?
I’ll leave it as very jiggy that you have come to think “The more I think about it, the more I’m excited about the idea of a personality matching shootout.”
I have your email somewhere and will have my tech guy get in touch with you. I’ll contact him tomorrow.
Regards, Joel
Saving majority of my comments for a new post. I am however interested in hearing more about this big happening. Big as in what?
Joel,
Don’t be silly! You’re new to the arena of compatibility testing and research. I even had to correct several of your claims on your website.
The fact is, there’s nothing new or earth shattering about your “challenge.’ It’s also not a new idea. I issued that challenge several years ago, and it was the basis for my peer-reviewed article in the North American Journal of Psychology.
Moreover, tests and measurements experts do validate applications in the real-world; not labs. It’s standard practice, not some “ultimate research.” And the constant knocking of math is laughable — as if “dismissive attitude, fluffy words and clinical insight” somehow take the place of hard data. They don’t.
More on your challenge very soon!
Thanks,
James Houran, Ph.D.
http://www.OnlineDatingMagazine.com
James, I remember it differentlty. Your challenge was math; mine, posted on Mark’s site two weeks ago without response, was “let’s move into the real world. ‘
Nonetheless, I am appreciative of the complexity. In fact, if you go back to my original challege, posted on September 5–I suggested something that I think is important, but will make things even more complicated.
For those who haven’t read it, here it is edited for brevity:
Dear James,
I CHALLENGE YOU!
Rocky 7!
Have you noticed how often pharmaceuticals that have been thoroughly researched and passed FDA scrutiny are found either ineffective (e.g. Vytorin) or worse, harmful (e.g. Celebrex) in real life?
In so many areas of life a surprising percentage of research that has been peer reviewed is a disappointment in real life. Even in an area far from the conventional, Wall Street, many people have lost money with very sophisticated, well researched mathematical models. It seems that in all areas research is best proven in real life.
You challenge me to expose my stats, as most other test developers have refused to do—most prominently E-Harmony and Chemistry, according to John Tierney’s science column in The New York Times: “Hitting It Off, Thanks to Algorithms of Love as reported by Daniel Ehrenhaft on his blog.
Once I do what many others have not, you go for the knockout punch. Given that so much research looks good on paper and fails in real life, here’s my challenge:
Develop a compatibility test that has the engaging style mine has, that has a brief completion time, as mine has, and do it in the next 6 months. Not something you’ve simply worked on, but a new instrument created by you to compete with mine.
My test and yours will be placed on mutually agreeable I-dating sites. Let’s see how we do in real life—the arena where theory turns functional—or not! I will lend my clinical services to support the matches and you will lend yours.
We’re talking Rocky 7!
Yes, I know, you think introducing clinical services will confound the research, but I believe that matches require support—that’s part of the reason so many I-dating subscribers are discouraged and that’s the package I offer.
Are you ready to lace up for Rocky 7? Or are you going to cite some statistics and dance around?
That is the ultimate research—outside the lab and into real life!
Joel (aka “Rocky”)
Once again, guys, I initiated a challenge that is obviously drawing energy.
If you put it together, very cool. I wouldn’t know where to start. I’m an idea guy,
a practicing clinician and a writer, those are my strengths. Organizing and managing a large project is not a strength. In fact, the mere thought of it gives me the jitters.
Come on, how would I find time to play?
JB
Hi Dave, if you want to compare apples to apples it would be great if you can contact each and every online dating site offering compatibility matching methods and ask:
A) TEST
A.1) Brain behind the test.
A.2) test? proprietary or commercial / ipsative or normative / validation / reliability, etc
B) MATCHING ALGORITHM
B.1) Brain behind matching algorithm.
B.2) the ENSEMBLE (the whole set of different valid possibilities)
B.3) About compatibility.
If compatibility means a high degree of similarity between several variables (religion, education, income, personality, personal preferences, likes and dislikes, etc.) between prospective mates
OR
if compatibility means a mix of similarity and complementarity between variables
OR
What does exactly compatibility mean?
B.4) What does exactly similarity mean? and What does exactly complementarity mean?
B.5) How exactly the compatibility matching method works, specifying main matching equation/ formula (without revealing proprietary information).
E.g:
“a proprietary Dyadic Adjustment Scale”
“a proprietary multivariate linear regression equation”
“a proprietary multivariate logistic regression equation”
“an adapted quantum mechanics math equation”
B.6) How compatibility results are displayed
E.g.:
only a number: like 110 (Cybersuitors), 96% (PlentyofFish Chemistry Predictor), 65 (Parship), etc.
only a graphic system: like a set of up to five empty/half_full/full hearts icon; a set of up to ten empty/half_full/full squares, a set of semaphores, etc.
or a written report like ThomasKnowsPeople
to indicate the degree of compatibility.
The method I had invented
“The pattern 6.7.6.8.9.6.7.7.8.7.2.5.8.7.3.4
is 92.55033557% +/- 0.00000001%
similar to the pattern 7.7.6.8.8.7.6.5.8.7.4.5.7.7.3.4″
B.7) Average number of “compatible real persons” for one person over the entire database.
Eg.
Many actual online dating sites offering compatibility matching methods could reach “as low as” 3, 4 persons high compatible per 1,000 persons, so in a 10,000,000 persons database, you have as many as 30,000 / 40,000 persons to contact, that means a whole precision LESS than anyone could achieve by searching on one’s own!!!!
30,000 persons is the population of an average small city!!!
The method I had invented can reach
3 most compatible persons in a 100,000 persons database,
12 most compatible persons in a 1,000,000 persons database,
48 most compatible persons in a 10,000,000 persons database,
DIVORCE RATES
C.1) “Divorce rates” of the couples matched by the compatibility matching method after 1 year, 2 years, 5 years, 10 years.
In August 2005, Dr. John A. Johnson told me, “There are probably undiscovered factors beyond similarity of any type that determine relationship quality.”
I think lot of persons could be interested in meeting/contacting other persons sharing nearly the same personality because they will be *predictable* for them.
Kindest Regards,
Fernando Ardenghi.
Buenos Aires.
Argentina.
ardenghifer@gmail.com
I think a few people need to get outside more.
Evan, I recall your seminar in Miami. Excellent! And, about getting out more–Yes!
Best wishes, Joel
But first this: James, all I was saying in my too long note is that I issued the challenge to you 2 weeks ago without response. Come on, as you guys are prone to saying, stay with the data.
Okay, now I am going out to play…JB
Some of us do get outside and work in the real world with real clients and have a real life.
That said, I think a few people need to learn to base website claims on research more and purportedly “scientific tests” on actual science more. Seriously everybody, are we in the business of helping people or scamming them?!
There are no standards for compatibility testing in the online dating industry, so it’s the responsibility of those with the expertise and experience to protect consumers from unsubstantiated rhetoric related to science.
And last this: Joel, the thunder is coming. Wait and see.
Thanks,
James Houran, Ph.D.
http://www.OnlineDatingMagazine.com
DR. JAMES HOURAN ACCEPTS “DR. JOEL BLOCK’S CHALLENGEâ€! –
ACADEMIC COMMUNITY NOW WAITS TO SEE IF DR. JOEL HAS SCIENTIFIC
STAMINA TO ACTUALLY BACK UP HIS UNSUBSTANTIATED CLAIMS
Challenge Accepted
YOU’RE ON! My team would be delighted to demonstrate the art and science of compatibility testing. Let’s first take care of housekeeping. Dr. Block, you must have heard about the distinction between reliability and validity, and that reliability comes first, because it’s mathematically impossible for an unreliable test to be valid (check any book on test theory). So, we’ll HAVE TO start with the research, mathematics and statistics. Serious testing companies, like the ones making the ACT, SAT, GRE, etc do this, and so should we – not only is this logical, it’s also the prescribed way in the APA’s manual for test design. Your website claims to be offering a scientific test, so, to do your “real world test” challenge justice, we’ll adopt the standard approach that’s advocated by all 101 level test and measures courses (and beyond). First establish reliability, then validity, then outcomes. Again, your own website makes bold claims about its “science,†so continued attempts to downplay or dismiss this step would be hypocritical on your part at the very least.
Of course, validation of the system should be done before releasing to the public. Even here math and stats cannot be avoided: You now talk about a “real-world test” or challenge based on outcome metrics related to customer experience, satisfaction, usability and ROI to the online dating service. That’s all well and good, but doesn’t this imply that mathematics and research is needed after all? The answer is “yes, it is.” Since we all agree on this, let’s start the challenge with research and mathematics. To be good citizens, and to protect ourselves and the public, before we launch the head-to-head comparison, please submit a test manual showing your “test” is even safe for the public to use. As an example, your manual should follow standard protocol. As an example, check out this link: http://www.goertzel.org/dynapsyc/2005/True.htm.
Specifics
My team has developed several proprietary matching systems on some of the biggest dating sites. That said, proper test construction is not a simple “just add water” process, so your “six month†time line might need to be extended. Also, the application will be built with an eye towards being “engaging and shorter” test than your questionnaire. Of course, you have yet to show that online daters require matching systems to be short and “engaging” (whatever that means). For example, eHarmony is phenomenally successful, yet their questionnaire is neither short nor probably engaging according to your definition. So, from a business standpoint these features may be meaningless. Also, researchers have long known that even “bad tests” can be perceived as effective due to placebo-type effects. Being a clinician, you already know this, but just in case I recommend this article: http://www.2020skills.com/asts/Barnum%20effects%20in%20assessment.pdf. Thus, it’s again imperative to start at the beginning of any scientific instrument – the reliability and internal validity.
With this all in mind, my team is confident we can build an original customized application that’s shorter and more entertaining test than your questionnaire. Shorter tests that don’t sacrifice psychometric quality are arguably best constructed using a Computerized Adaptive Testing (CAT) approach. Fortunately, my team includes pioneers in this as well, see:
http://iknowsys.com/PDF/NO-MARKS_Rense_Lange_CAT_rating_scales.pdf.
My team is well established in this industry and in the academic community because of our customized applications that drive revenue. Indeed, we have presented at iDate and published peer-reviewed scientific articles about the evidence (and lack of) for compatibility tests. It was even our team that first initiated a challenge to compatibility test makers to show the public any and all evidence for their specific methodologies. Some started to do this, and in keeping with the initial standards by Glenn Wilson and Jon Cousins, my team was arguably the first to bring testing standards to this aspect of cyberspace. Even Dr. Mark Thompson at the pioneering online testing company weAttract acknowledges this.
However, we don’t work for free. The estimated cost to combine all of the features described above in a new application, to meet APA testing standards, and being ready within approximately six months will cost approximately USD $350,000.00. You already have my private email address. Please write me so I may provide you with the mailing address to send a check. We will require full payment before the project will begin. We can also accept a wire transfer. The nice part is that you would own the application we build and be free to sell or license it as you see fit (we can always build another). We would insist, though, that the results be published in a peer-reviewed journal so the field can learn from our head-to-head comparison.
Personal Note to Dr. Joel Block
You welcomed debate and questions from me on OnlinePersonalsWatch.com. It was clear from your comments that you had no idea who I was or that my recognized expertise was compatibility testing. It was also clear you had little to no grasp of the literature and history behind compatibility testing itself. I even had to correct you on several naïve claims on your website. Then when I started asking hard questions about the “scientific†matching system you were selling/advertising on that blog you became less cooperative. It was framed in niceties like, “let’s take this offline; this stuff is probably boring people.†But, I didn’t go away and continued to ask hard questions for which you obviously had no credible answers or data. Now, you’re trying a different method of misdirection by insulting the general public by asserting the public can’t or won’t appreciate all this science and research “stuff†and therefore we should ignore it and simply see “what people like.†People like candy but it’s not nutritional, whereas medicines can be life saving though not have the best “user experience.â€
You poise yourself as someone who cares about the outcomes on people, but you seem to selling snake oil. Your dismissive, cavalier attitude about science is neither consistent with the messaging you have on your site nor with your clinical education and credentials. As an academic, I find it appalling and bordering on unethical. I’m even tempted to contact all professional organization to which you belong and notify them of the situation. Your website states your test is scientific and effective, although you repeatedly avoid providing any data of professional quality that establishes you even have a scientific test in the first place. I take strong offense to your advertised claims, and they should be modified immediately. Other researchers who have seen our exchanges side with me on this. You might have a “fun questionnaire,†but such an application shouldn’t be equated with an effective scientific instrument in which people should place their trust.
Everyone may not appreciate the science behind compatibility testing, but is that any reason not to adhere to professional standards? I’m incredibly relieved that experts who make and improve aircraft, household appliances, vehicles, seat belts and medical equipment don’t adhere to your point of view. I challenged you a while back to provide evidence for your scientific claims. Jon Cousins – a published compatibility researcher who deals with real world applications – also asked for similar data. These data apparently don’t exist, since you’ve never answered our original challenge to you. Amateur hour is over – you’ve been called out. Let’s start the challenge immediately and follow professional standards for evaluation. Frankly, I don’t expect you to actually participate. What I do expect is more stone-walling. The academic community is waiting – are you going to put up or shut up? My team started this challenge for dating sites to show clear and convincing evidence for their purported “scientific matching testsâ€, and I’m committed to see it through.
The idea of comparing compatibility systems is intriguing. But, I wonder if an apples-to-apples comparison is possible. Matching people is as much an art as a science.
Most of the compatibility tests I take match me based on stats: religion, # of kids I want, hobbies, willingness to relocate, etc. But, I can skim a person’s profile for less than one minute and assess if he meets my “stats criteria.” I don’t feel that I am getting more insight with these tests.
Most importantly, I want to know if we experience life in a similar way, because sharing this fundamental perception of the world will let us transcend relationship issues that we will no doubt later confront but cannot anticipate, no matter how much information we have now about our mutual love for dogs, kids, French food and skydiving.
At A Sound Match, I use music — a common element in all of our lives that reflects our emotional and social selves — to suss out the likelihood that two people share a similar experience of how they walk through the world. The stats are requested, too, but they are not primary in determining compatibility.
Here is my off-the-cuff idea for a “real-life” challenge: Get a sample size of singles. Each single takes all of our personality tests and receives lists of ranked matches from each of us. Put them in a room for a huge speed-dating-like event. Have them meet and rank their dates on score cards. Let them tell us which system introduced them to people they liked and want to see again. We can have periodic follow-up interviews to measure success.
Think it’s possible to run a project like this at iDate in January? ;-)
Be well,
Lynne
***************************************************************************************************
My Music Personality is a Diamond in the Rough. Are we compatible? Take the 3-min quiz.
Lookup: tunedin
Lynne Sandler
415.285.8935
http://asoundmatch.com
*music matches us*
Lynne, great idea. I think you’re on to something.
Interesting comments Dr. Houran. Where to begin. I think I’ll wait for my next post on the topic to address.
Hi Dave,
“Interesting comments?” With all due respect, Dave, there’s nothing more to address. Either Dr. Joel has a “scientific test” or he doesn’t. I appreciate your expertise on the business of online dating, but you’ve admitted before that the subject of tests and measurements is not within your expertise.
It’s also outside Dr. Block’s. He’s made bold claims that he refuses to substantiate with professional quality data. There’s not a gray area about the criteria for justifying the label “scientific test” from a professional standards perspective. He wants desperately to take the issue out of the scientific arena, but that’s not going to fly.
There’s nothing more for anyone to say, except for Dr. Block — it’s his turn to stop stonewalling and substantiate his claims or go back and do meaningful psychometric research and come back with the results. End of story.
Thanks,
James Houran, Ph.D.
http://www.OnlineDatingMagazine.com
Hi All—
My response to Houran’s “Thunder†(which is less thunder than it is a blast of putrid hot air) is in three parts. The first is to the General Public—a perspective on tests and research in general. The second is a Memo to Houran and the third is a Major Offer to all site owners, except those involved with Houran.
Part One to the General Public:
Yes, I am the new kid on the block and I have attitude—well earned, I might add. But I’m dealing with a guy full of self-righteousness. I have a broader perspective and I have seen that research needs to be viewed with caution. There are reams of research that have been peer-reviewed and later found wanting. Sometimes with major costs. Remember Viet Nam? It was the Rand Institute’s research that pointed the way into that disaster.
You know the screening tests we take, both men and women? PSA for prostate cancer—controversial; mammogram, several randomized controlled trials have not supported its utility. Remember Vioxx? How about Vytorin? Both passed peer-reviewed scrutiny. One is likely to kill you and the other doesn’t work.
Ironically, although the consequences in some areas of research, like medicine, could be disastrous, the research is in many ways simpler than social science research. The variables in social science are actually harder to pin down. As Einstein said, (and this is a paraphrase) “What can be counted (in research) often doesn’t count.†But don’t get simplistic and take this as a put down of science. It is about providing perspective. Houran uses research as a tool to diminish others work, not to support legitimate efforts.
My issue is not with science, but with a zealot that puts down others without making it clear that all research, including his, has limitations. His attitude discourages innovation and gives the impression that science is exacting. It aims to be exacting, but it almost always falls short. In fact, as I state directly on http://www.ButterfliesAgain—the nature of lasting love (compatibility) is such that other factors, like having the tools to make a relationship work, is critical. No test should stand alone. I have a clear statement about the limits of compatibility testing. An instrument that hopes to match couples for lasting love is a combination of art and science—in equal measure. I bring both to the challenge.
Houran is like a right winger of test construction. His outlook is extreme, focusing on one element of an issue and not taking in the full picture with all its implications. ButterfliesAgain is a creative innovation that is as much art as it is science. All important research begins with observation, mine is 30 years working with couples. But to argue the art/science issue with Houran would be like arguing the merits of stem cell research with Sarah Palin. There isn’t an open-mindedness that is so necessary to discussion. Houran has demonstrated a negative default. He comes off as having a bottomless reservoir of certitude. It is not about what is right, it is about playing, Gotcha!
I have left myself open to his wrath (and that is exactly what it is!) by putting my data on the website in an easy-to-read and understand manner. His criticisms are too broad because he does not have all the info he needs to make a proper assessment. I purposely didn’t provide that to him in our exchange. If you went to a psychologist who hammered you with negativity, would you be enthusiastic about being open? Me neither.
By the way, this is from the forum you are now reading, it was posted about a year ago (July, 2nd, 2007):
Dr. James, Houran has written an article at Online Dating Magazine warning singles to avoid falling for scientific matching hype that has permeated the online dating industry. Houran comes across as the tell-all scientist who knows the dark secrets of the dating industry while managing to stick up for his work at True.com and putting down the Chemistry.com and eHarmony blogs.
Just what is Dr. Houran’s angle here? I don’t know who is paying his bills, who his clients are or what he’s trying to accomplish in this anti-dating industry/blog rant. Going on ABC and doing Time magazine interviews is more self-serving than coming up with a real solution to help singles focus on realistic expectations
My point—Houran approaches these issues with a hammer and views his competitors as nails.
Part Two: Memo to Houran
$350,000? This is a joke, right? James, seriously, you really have to get over yourself. You are not essential to me or to the world of dating. You give yourself way too much importance. The problem with zealots is that their delivery, like yours, generally sucks. They are condescending, arrogant, snide and more than a little unpleasant. Those characteristics don’t play well. And that is exactly why I didn’t go further with you as to specifics. Your default is negative and, frankly, I don’t trust you. Your arrogance masked as “Master of the Test Construction Universe†is hard to take.
Your snide comments and condescending attitude was really a major turn off. If you are paying attention, that backing-away-from- you response must be familiar. Your spin on my suggestion to take it off line due to boredom was that I was afraid of the forum? That’s the kind of negative default that turned me off.
Why didn’t I trust you?
Here are a few examples of unnecessary bites that give away who you are:
You said this on 8/30: “My criticism is that you never specify evidence for why people should believe your assessment is reliable and valid. You neither address it on the site nor here on OPW.â€
Actually, your registration on my site states that you didn’t see my data until September 1!
This raised my suspicions about you. A guy that promotes “integrity†doesn’t have a margin of error. Especially about an outright lie.
Another example. In your last note you state: “Jon Cousins—a published compatibility researcher …also asked you for similar data.â€
James, you made that accusation in an earlier comment (8/31) and here’s how I replied then:
“Yes, I know, a colleague of yours was on the site, Jon Cousins, but he did not agree with the standard NDA and wrote me to that effect. I wrote back immediately and invited him to speak with me—he never called me.â€
Now you are going to make that accusation, already explained to you, once again? These are the things that support my contention you are not looking to protect the public, but to protect your turf. Once again, your integrity comes into question. Your agenda is to hammer the competition.
You chided me for saying nothing is new, a stretch just to maintain your critical, adversarial attitude. I gave you multiple examples of what is a retreading of old things. Want yet another?
The “new†swing to positive psychology posited by Dr. Marty Seligman is an update of Abe Maslow’s take that clinicians need to look at what’s right with people.
I also followed the long thread about security that True.com was pushing security. First, as you know, the company you were associated with had a plethora of complaints while you were associated with it. Reviewing that “security†thread, you seemed once again not to really listen to those (many!) that disagreed with you. When you were questioned about your research, here’s what you said:
“Please by all means sponsor some research studies if you feel the studies that have been done are inadequate. I doubt you would cite them as inadequate if they had shown the majority of surveyed people to be against this legislative effort.â€
For a research guy, I would think you would have responded precisely (about your biased sampling) rather than the indirect response that you accuse me of.
Here’s the link to the many complaints on the thread for True.com:
http://forums.plentyoffish.com/2844031datingPostpage2.aspx
And here’s what was said by a person who apparently has been involved with I dating. Not a researcher? That doesn’t mean his view should be ignored:
I think the answer to “how can you tell that tests are true and correct” doesn’t have anything to do with independent auditors or scientific journals. The answer should be based on the ratio of couples matched who find love to those who were incorrectly matched. If a test is published with all the publishing background in the world it still won’t hold a candle to one that accurately predicts who will fall in love with whom.
I personally have met many of the matches that the true.com test has made for me and many matches that eharmony made for me and neither has found Ms Right for me yet despite the lack or abundance of literature and scientific review. Imagine if Dr Houran’s answer to the question “How can you be sure True.com’s tests are true and correct?” was “The couples we match fall in love with each other twice as often as the couples eHarmony matches.” THAT would be a selling point and would make me send a few more winks on their system again tonight…
- Glenn Gasner
I think Glenn is implying that we aren’t dealing with a cancer drug here. He’s suggesting you lighten up! My sentiments exactly.
That’s it James. I’ve put in way too much time on this as it is. I’m done. Take your last shot. You can have the last word. I won’t be responding. Enough!
Part Three: A FREE Offer to Dating Sites
Here is an offer for all I-dating site owners—except those associated with Houran. Don’t be put off by the Gotcha King, follow your entrepreneurial path, find out for yourself.
1. Try ButterfliesAgain FREE. The only cost to you will be integration of the instrument into your site. We will supply the necessary documents and codes. Let’s get the feedback from your subscribers. I will be more open to their comments, then to those of Houran who says he is protecting the public but is more likely protecting his own turf.
2. I will make myself available (also FREE) to answer select dating/relationship emails from your subscribers. My 19th book on the subjects of love and sex will be out next year. I have specialized in these areas in my clinical practice for nearly three decades, so your subscribers are getting a service that is not typically available from an expert with my credentials—at least not for FREE.
3. I will send you an advanced copy of my forthcoming book, The Real Reasons Men Commit: Why He Will—or Won’t—Love, Honor, and Marry You (Adams Media, December). The book introduces new psychological concepts and preview readers have found it extraordinarily helpful. Based on my publisher’s permission, there is the possibility of posting some of the content for your subscribers.
Joel Block, Ph.D.
Okay, I really am finished, so in anticipation of Dr. H’s defensive reply: Yeah, yeah.
And that’s all he wrote.
JB
Interesting post & feedback in comments. I guess the big question is whether these assessments work & how to measure that success? I would invite users comments here rather than marketing puff from companies themselves. Also can I add that from a coach perspective these are not tests but assessments (there is no right or wrong answers to the questionnaires!) The other question is what is scientific when it comes to dealing with people & their preferences & values???? Surely matching systems that have been around for centuries like astrological matching is more reliable than science which has only been around for a shorter while & often doesn’t look at people but stats? I have started posting reviews of different matching systems but from the human point of view, asking for customers feedback. This is a big market – supporting good matches for people who use online dating sites or social networks. I’m sure every system has its strengths & weaknesses. I would like to invite information about different assessments from various companies on the web & get some reality feedback from customers. You can check out the first post in this series at http://tinyurl.com/4fyj2w If you have an assessment that you would like reviewed let me know.
Best Pemo Theodore
Your’re right, they are not tests as much as compatibility assessments.
I’ve mentioned astrology before, off to check out your site.
Just to clatify…
Tests and assessments are both gauges or measures of underlying traits or characteristics. Both can use a myriad of different question types to establish trait levels. The only real difference is that with assessments there is evaluative feedback or clinical/practical application. Tests simply give descriptive “grades†or scores, not neccesarily evaluations of those grades per se. That is what an assessment does.
Thanks,
James Houran, Ph.D.
Dave just gave me the heads up on this latest round of assessment science bun throwing.
I think Evan Mark Katz has the best advice. Little wonder his comment has the highest rating so far – a whopping 3.4.
Here’s our two cents…
1) first and foremost the test has to be compelling (i.e. fun) to the end-user. If not, they won’t complete it – no matter how validity/consistent the results are. Attrition is the first challenge in test construction.
2) questions should customizable so that the test works on a variety of sites. We’ve found that the same test doesn’t play the same way in the UK as it does in the US or on a Catholic site the way it does on a Jewish site.
3) the test should be accessible in terms of price. $350K is laughable. I’m not sure how many takers Dr. Houran is getting but we have to work to get clients accept a $5K set-up fee.
4) finally, the test must be profitable (i.e. make money for the website). It’s no longer enough to simply promote your test on the basis of member conversion. It needs to contribute to subscriber conversion. Why else would you add it? Benevolence is not a trait we’ve encountered much in the dating site owner’s in the last 7 years.
Have a good one.
Patrick
Hi Patrick,
Thanks for participating. Some reactions to your feedback:
1) first and foremost the test has to be compelling (i.e. fun) to the end-user. If not, they won’t complete it – no matter how validity/consistent the results are. Attrition is the first challenge in test construction.
A: Simply not so. The market bears this out. eHarmony neither has a fun nor compelling questionnaire, yet they dominate the market. Actually, users will complete a questionnaire or legitimate assessment if they have intrinsic motivation to tolerate the process. Making it “engaging” is nice, but if the process doesn’t work then all those cosmetics were useless in the end — both for that user and for driving PR for the site.
2) questions should customizable so that the test works on a variety of sites. We’ve found that the same test doesn’t play the same way in the UK as it does in the US or on a Catholic site the way it does on a Jewish site.
A: Not sure what you mean by “cuztomizable.” It’s true that some tests/assessments need to have different content for different audiences, although proper test construction at the onset would’ve revealed this in the first place and designers can make revisions so that one application does work with diverse groups. Test makers in personality and educational testing do this all the time. But, if you mean change formats, questions, etc on a whim, then I’d say that’s a dangerous approach. Changing the wording even slightly on some or all of the questions without a follow up validation is a recipe for a poor test and hence poor user experience.
3) the test should be accessible in terms of price. $350K is laughable. I’m not sure how many takers Dr. Houran is getting but we have to work to get clients accept a $5K set-up fee.
A: Not so. Companies that can afford an original and valid application should be free to enjoy the competitive advantage that application provides. Any company can build their own questionnaire and matching method for free (5K is incredibly expensive for a fun “questionnaire”), but tests and measurements companies routinely get millions of dollars for their work. Building a legitimate assessment that ‘s original, has strong psychometrics (it works) based on modern test theory and is user friendly is costly in terms of time and money. Of course is scalable according to the exact scope and business requirements.
4) finally, the test must be profitable (i.e. make money for the website). It’s no longer enough to simply promote your test on the basis of member conversion. It needs to contribute to subscriber conversion. Why else would you add it? Benevolence is not a trait we’ve encountered much in the dating site owner’s in the last 7 years.
A: Yes, no argument here. Although I do know of some sites that have “tests” simply to compete in the minds of users. It seems that almost every site has a “test” nowadays — so some companies may want one for a value-add to their customers. I guess benevolence is not entirely dead, which is encouraging given this is an industry purportedly devoted to love and happiness.
Thanks,
James Houran, Ph.D.
http://www.OnlineDatingMagazine.com
Every time i come here I am not dissapointed, nice post
{ 3 trackbacks }