The OkCupid blog is back, and as usual, it’s ruffling feathers. This just a few weeks after Facebook came clean about doing experiments where they manipulated user data to explore the contagion of emotion, among other things.
It’s great to see OkCupid publishing research again. They crunched some numbers, tweaked features, hid photos, wrote a blog post and the result is once again a huge amount of free exposure. As an added bonus we all get some more transparency into how singles use dating sites. That’s exactly what they wanted, and they nailed it. Remember, the last time they opened up their Big Data kimono they were acquired for $90 million.
My mother, of all people, called me in a huff about the OkCupid article in the NY Times, OKCupid Plays With Love in User Experiments. She was concerned that dating sites were more sensitive and important than other sites, and that manipulating users was wrong. She, like so many people, couldn’t get her head wrapped around the fact that websites and apps test and iterate all day long every day. Otherwise they would die off as competitors take advantage of their inability to react to what’s learned during the iterative development process. Regardless of the scope of the alterations or how they may be telegraphed to users, that’s how it works and many mainstream websites and the ensuing trillions of dollars of value have been created in the last 20 years based on this fact. Amazon did it better than their competion, eBay too, and countless others.
OkCupid’s Christian Rudder:
I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out.
…At OkCupid if the algorithm changes, yeah, they go on different dates, discover different people, maybe even marry somebody different. But that’s not me playing god, that’s just a fact of the service. Any decision the site makes has those implications because people are really using these services in their lives.”
Nailed it. Add a pop-up disclaimer every time you visit a website that is testing something and we’re done. I kid, this is confusing and complicated, lets dig in.
OkCupid is basically a room full of self-described math nerds. They live and die by the data they gather on users and algorithms and make no qualms about it. OkCupid is going to get you a date and if you don’t like it you are free to leave. Go pay one of Match’s 20+ other dating sites for the opportunity to meet someone. Or POF, eHarmony or some niche that just maybe has a few people in your area code.
You want to scream at a company that continues to stiff-arm consumers and gathers, manipulates and sells your information? Read The Insidiousness of Facebook Messenger’s Mobile App Terms of Service.
- Allows the app to take pictures and videos with the camera. This permission allows the app to use the camera at any time without your confirmation.
- Allows the app to read you phone’s call log, including data about incoming and outgoing calls. This permission allows apps to save your call log data, and malicious apps may share call log data without your knowledge.
- Allows the app to read data about your contacts stored on your phone, including the frequency with which you’ve called, emailed, or communicated in other ways with specific individuals.
Facebook messenger has been downloaded a billion times. I’m not trying to redirect focus away from the dating industry, but there are many, many other sites doing decidedly more sketchy stuff with your information and generally pulling the wool over your eyes than a dating site with a few millions users, a tiny fraction that having been exposed to a handful of tests. And those users received an email about the tests!
Think of how many millions of people have swiped left instead of right while searching for their Tinderella? (Hat tip to Tyler Kingkade for this). Millions of people a day moaning out loud as the match of their dreams passes by into the digital ether, all due to swiping the wrong direction. Millions of people missing their potential life partner, and the feature responsible for this nightmare is the primary functionality and reason why people use the app! Are we complaining about Tinder too from a functionality standpoint? Of course not, plenty of other reasons to hate on Tinder, but that’s for another post.
I have been pushing startups to do data/functional/behavioral-based testing for 15 years. The phrase “Set it and forget it” has always driven me crazy. People tend to approach launching a dating site like building a house. You build it and people move in and every five years you paint the living room. But dating sites are dynamic living breathing entities. There are always ghosts in the machine that need attention, and lots of opportunities to do things differently, and better. This is what OkCupid, Facebook and thousands of other websites and mobile apps do every day. Every. Single. Day.
Guess what, everything online is dynamic these days, built on-the-fly from vast databases and templates. Static html simply doesn’t exist anymore. This makes it easy to test new fonts, wording, pricing, page layout, search functionality, messaging, etc.
Every dating site worth it’s salt aggressively tests new features, functionality and matching algorithms. If they don’t, they’re doing it wrong and I wouldn’t even bother using the site.
Before we go nuclear on OkCupid, let’s unpack the situation a bit, head off into the weeds with some ancillary information, and then you can each decide how right or wrong companies are when it comes to improving services with user testing without explicitly telling users.
A component of user testing is called A/B testing. Companies like Unbounce and many others offer services to test the performance of entirely different pages, or test pages where sections of the page are different (called multivariate testing).
Match or POF could have multiple home pages, depending on your geographic area, gender, age or some other attribute they know about you. Or maybe they are testing colors, or the types of people shown to non-logged-in new visitors. Or 1,000 other things.
They could move the Inbox section to a different part of the page, or change the font. Maybe a nag screen to get you to upload more photos, but the person next to you in the coffee shop wouldn’t see the same thing, even if they were on the same site at the same time. You see version A, they see version B.
Pretty basic stuff. OkCupid went a few steps further, but not so far as to really raise eyebrows if you’ve been around online communities for a while. Here’s where I get in trouble, I’m so deep into the industry practices that absolutely nothing surprises me in the dating industry. I can see how a regular Joe could freak out at OkCupid, but this is an awareness issue, not an ethical one.
What if a guy is acting like a complete tool on a dating site, the dating site could shadow-ban the person. Hard to tell if this is currently in vogue with dating sites, but it sure works on Reddit. That’s absolutely lying to a user, and it’s an industry best practice on many forums.
Reddit is the evil master because they shadowban awful people? Is that ok or not? I think it’s ingenious and perfectly legit. Oh how I would like to see dating sites do this more and then alert the user and offer them a way to “make things right”. But that sort of interactivity with users is absolutely unheard of in the dating industry.
There is an old story about eBay trying to change the background page color that is relevant here.
eBay.com: Satisfying their Users with Gradual Evolution.
At eBay, they learned the hard way that their users don’t like dramatic change. One day, the folks at eBay decided they no longer liked the bright yellow background on many of their pages, so they just changed it to a white background. Instantly, they started receiving emails from customers, bemoaning the change. So many people complained, that they felt forced to change it back.
Not content with the initial defeat, the team tried a different strategy. Over the period of several months, they modified the background color one shade of yellow at a time, until, finally, all the yellow was gone, leaving only white. Predictably, hardly a single user noticed this time.
More recently, eBay recently underwent a redesign of the forms that people use to post items for sale. These pages are critical to the success of the site, used more than 1,000,000 times every week. Knowing that their users are sensitive to sudden, dramatic changes, the team came up with a novel approach to phasing in the forms.
After verifying the changes through usability testing, the team made working versions of the new pages available for previewing with a link on the existing seller’s page. After looking at the preview, users had the option of making the new forms their default. By tracking how many users converted to using the new forms, the team could measure it’s acceptance.
Great story that illustrates the nuances of public opinion when it comes to our favorite destinations online. A slightly more benign example, but you see where I’m going with this.
Here’s another example closer to home. Clients always ask me what their marketing message should be (fresh launch or in some more advanced variation, old-growth dating sites). My response is that Google wants your money and provides many tools to make your life easier, and easier to get your credit card number. I tell them to put a range of ads on Google, and use the verbiage that converts the best in their marketing message. Perfect? Of course not, but it’s cheap, fast and often good enough to launch. Or you can waste a lot of money on a branding expert, but at the startup phase, that’s a waste of time 99% of the time.
Should I feel bad that at certain times my client may be misrepresenting their brand while they test marketing messages? Of course not. How else are they going to figure out the right messaging to drive signups, a Ouija board?
Where is the line that a dating site must cross to be considered evil? Is showing variants of home pages ok? What about different versions of the signup process (manual, Facebook-enhanced, different questions for different people)? God forbid Match or eBay improves their search results display, right? But you get better prices and people in the long run. See the tradeoff? If it’s not worth it to you, go hit the bars or go to church.
At the other end of the spectrum we have outright changing up a person’s profile, hiding people from them on purpose, not letting their messages go through (see Shadowbanning) or outright using different matching algorithms for certain geographic areas to test changes. That definitely happens in the dating industry.
The main issue is people’s ignorance about how Internet companies operate and evolve.
Another example: People ask how much they should charge for a monthly subscription or when to turn on the paywall. I smile and say I have no idea, let’s find out. This is often awkward at first, but when a client is looking at a full dashboard of data along with a sales funnel, it all becomes clear. “Oh we throw a bunch off stuff at the wall (educated guess) and see what sticks?” Exactly. This is Lean Startup 101 stuff. If you don’t get this 110% from the get-got, you are basically screwed, regardless of if you run a dating site or otherwise. Or listen to your marketing person, because they’ve launched tons of dating sites, right?
OkCupid Labs has probably shuttered more dating site concepts than the competition has launched as a whole. Throw something up, learn from it, iterate, take the best ideas and then shut it down. That’s one way they learn what people want and how to give it to them.
But you don’t have to be a big multi-national conglomerate like Match/OkCupid to benefit from the Lean Startup model. Many dating sites are now being launched on WordPress and BuddyPress. Easy to set up, easy to tweak, easy to test. While you are spending a year trying to raise millions to take on Match, we’ve launched 10 sites in 10 different niches around the world, and it all runs off the same platform. Does it scale easily to 5 million members? Probably not but who cares? If you have 5 million profiles, you can have your site written in Assembly by fantastically beautiful programmers working poolside with you in Belize.
The value of white label dating sites is that anyone can run a site. A side effect of this straightforward-and-simple dating site startup model is that white label dating sites become less appealing when some kid in the Philippines can crank out a basic version of your dating site in a few weeks, complete with an easy-to-operate testing environment that anyone can use.
Is spending $25k on a WordPress-based site better than spending the same amount of money on marketing a white label site (albeit without the cool 20-something marketing mavens like they have at Global Personals)? The answer is that it depends on a number of factors I’m happy to discuss with you.
Massive amounts of A/B testing means days of relying on a third party for profiles to launch a dating site are over. It’s too easy to onboard a few thousand people the first few weeks of existence. Hello Launchrock. Initial marketing message and basic functionality with a super-slippery on-boarding process, questionable quality to the database if you stray to far afield of your real messaging, but at least it’s not empty and you didn’t do something stupid like buy a CD of profiles. Then you switch mid-stream to you core branding and functionality. This doesn’t work for everyone, but I’ve seen it time and time again and it works.
For an extreme example Zoosk got tons of new members when they bought the Oregon Trail Facebook app and redirected all users to the Zoosk app. That was infuriating and brilliant. Are their investors and the other people trying to make huge bank off their IPO worried about some app purchase years ago and how that literally pissed off millions of people? Of course not, we have short attention spans.
Remember back when eHarmony was branded as an anti-gay Christian site a decade ago? There are now people on the site who were eight years old when that happened and they just added another few hundred thousand subscribers in the last year. Time heals (most) affronts to our sensibility when it comes to the Internet. Well except for Digg.com, which attempted to “replace the entire plane engine while flying” and subsequently crashed into the badlands of public opinion.
Ok, enough history and on-boarding chatter. How is a dating site going to test matching algorithms, marketing messages, communication functionality and other key aspects of services without throwing a wrench into their relationship with their users? They could do it internally like Facebook, effectively “eating the dogfood” before pushing new features and functionality to users. But Facebook isn’t a dating site and its single employees are likely too small for a reliable sample size.
Are we at a place with online dating that people spend an inordinate amount of time looking at match percentages and letting those percentages dictate with whom they communicate? Match percentages can be displayed many different ways, ranging from numerical representations to the order of search results. What if OkCupid had re-ordered search results instead of changing match percentages? Is that easier to stomach in terms of user manipulation?
Match percentages are merely an indicator based on what OkCupid knows about matching people up until now. If we paid close attention to compatibility scores on dating sites, we’d all be single because they really don’t have a very good idea how to match people. Hearing the co-founder of OkCupid admit they really don’t know what they are doing was a solid vindication of what I’ve been saying for a looong time. This doesn’t mean they aren’t brilliant and driven and crazy-smart, but they’re fumbling around in the semi-darkness just like the rest of the industry. And that’s perfectly fine, because if I ask for more I just get depressed at the whole notion of online dating. It’s so incredibly imprecise and complex and full of guesstimates and if you think about it too long you’ll end up in a ball in the corner.
Remember, dating sites run far away from actually being called dating sites. You don’t date online, you are exposed to a certain subset of a large group of people based on some arbitrary match cooked up by some pretty smart people. It’s up to you to initiate contact or respond. But you sure as hell aren’t dating on a dating site. You’re looking at static billboards for the most part, with little value past whatever the person felt like writing six months ago when they wrote the profile at 1am, with a buzz, after another bad date.
Back in the day eHarmony expressly stated that the service was for entertainment purposes only, probably at the behest of lawyers terrified at the potential for lawsuits. How fitting that such a serious attempt at online matchmaking was considered a novelty.
Tweaking search results (hiding good matches for example) is a lot different than looking at a number and performing a specific action because the number is some value (30% match vs. 90% as per the OkCupid research). Plus, guys, who really looks at these numbers? If she fulfills whatever criteria you have (or not), no matter how ridiculous, she gets an email, regardless of the number next to her photo. Please, prove me wrong.
Let’s not forget fate. Maybe it’s fate that OkCupid changed up some things and a certain man didn’t meet a certain woman. Or maybe they did meet. If you want to get upset about fate, it’s your prerogative and above my pay grade.
What’s the final answer here so that people don’t feel taken advantage of while dating sites improve their services? Better Terms of Service? Pop-ups saying that “your experience is being modified for testing purposes” or something else entirely?
I like the idea of having beta.datingsite.com or some other sort of parallel experience where the user can opt-into new features or be subjected to whatever artificial intelligence fuzzy quantum math that sites like OkCupid come up with next.
Until then:
Everything changes and nothing stands still.
– Heraclitus
