Ben Settle Headline Split Test Results
Tuesday, 6:33 AM
Dear Friend,
Last week I sent a message endorsing Ben Settle's new hard copy newsletter.
Since I know that negative news always gets more viewership than positive news, I split-tested two "negative" headlines on one of my subscriber lists.
They were…
- Ben Settle SUCKS (By the way, it was Ben who suggested this.)
- Why I HATE Ben Settle
Here's a screen shot from my Aweber account with the results:
Notice that the headline "Why I HATE Ben Settle" got a 37.2% open rate and a 18.8% click-through rate.
The headline "Ben Settle SUCKS" got a higher open rate, 44.6%… but a lower click-through rate, 12.8%.
I have no idea how many sales it brought in because the link in the e-mail was not an affiliate link. It just linked directly to Ben's newsletter site. (I believe so strongly in Ben's new newsletter I endorsed it without any compensation.)
Looking back, we SHOULD have used a tracking link from Ben's 1shoppingcart account so we could see how many new subscribers he got from this endorsement.
Oh well.
Anyhoo… would you like to know why one headline got a higher open rate but the other got a higher click-through rate?
So would I.
The truth is… I have no earthly idea.
We can guess why we THINK it got these results, pontificating for hours… but the truth is, we don't really know why. The only thing we know for sure is the results.
And why do we know the results?
Because I TESTED it.
You can theorize for days and try to rationalize why you think this or that headline is better… but in the end, you're just guessing.
The only way to know for sure is to run that puppy up the flagpole and see how many people salute it.
And it's so easy to test this stuff online nowadays there's no excuse for not doing it.
All the best,
P.S. If I were sending a direct mail pitch and had access to a list of customers or subscribers like this… this is EXACTLY what I would do to figure out the best headline for my direct mail piece.
If I didn't have access to a list I would do a headline split test on Google Adwords.
It's a lot cheaper and faster to split test your headlines this way than in direct mail.
Hi Dan
The sad thing is Dan most people use this kind of negativity in there every day life without being fully conscious of it or ever really noting why. I noted a few things that my Mother said to me and swore to myself that I would never use them in, around or on my children…did I? You bet and with the same tonal inferences as well. With me though it is more who the email is from, rather than the headline that's where my eye rests first. But a few great headlines really have mustered me into quick action.
Dan, it's good to know at least someone out there can profit from my suckiness… I've tried unsuccessfully to do that for years but to no avail…
Doesn't it kinda hurt your feelings to see people opening my e-mails to find out why I hate you… or why I think you suck? lol
We should have split tested a 3rd headline with something positive like "Why I LOVE Ben Settle's Newsletter" or something like that.
Knowing human nature I have a feeling the negative ones would have outpulled.
Good points, Byron.
That's cool that you're consciously avoiding using negative stuff like that. It's not all that easy, is it?
I'll save the details about this for a future blog post but my wife just started watching a two year old girl a few days a week while the mom is at work. It's amazing… she is like a little sponge. I'm watching every thought, action and word around her… including my tone of voice.
I've often wondered what kind of adult a person would become if they were only surrounded by parents and other people who only spoke positive, encouraging and uplifting things. Now is my chance to try it… even if it is only a few days a week.
DD
I think it's just cause Ben sucks SO MUCH, that those two headlines resonated with everyone on your list hahahaha
Soon to be traded in for something a lot more powerful… better duck haha
I agree that split testing is vital and the numbers don't lie.
However, there's one thing all the testing in the world won't reveal:
how the use of negative headlines such as these impact the level
of trust between you and your subscribers in the long run.
The fact is that you don't hate Ben Settle and most will agree that
Ben Settle doesn't suck.
But I guess you found a way to tie-in the subject line with the body
copy with skill.
If I'm a subscriber do I believe your next headline, claim or offer
more or less next time?
I don't use these types of headlines and I couldn't give two flying
f*cks if they increase or decrease open rates. I want my subject
lines to be credible and continue to build trust and the relationship
with my subscribers along the way.
Some things that really matter, can't always be measured.
Shaun
Good point, Shaun.
What sayeth the subscribers?
How did you feel when you received one of those headlines? Did I lose your trust? Will you never again listen to the free content I provide?
DD
I agree with Shaun, I don't think Ben Settle really sucks either… but then again, I'm a tad biased…
lol
He does bring up a good point though.
I agree with Caleb. lol. I received "Why Ben Settle SUCKS" and all I could think was….YOU DAMN RIGHT DOBERMAN!
HELL YEAH!
lol…just kidding
We shouldn't talk junk about Ben…I joined your list from his blog. But there is a reason they call him Ben "Bougie" Settle
To Ben: I didn't know you played with guns. Look…it's not worth going back to prison. Ok….I was just kidding.
I like guns too, but I don't have a handgun yet. And I don't play with them while I"m on the phone – but sometimes I wave them around in public. (Kidding about that too)
You're talking about a guy who plays with a 9mm while he's doing phone interviews. I don't think it's a good idea to make him mad.
I think I got the "Ben Sucks" one .. I only opened it because I wanted to sign your petition / leave negative comments in agreement on your blog etc.
When I found out you were actually mates I joined the people who bounced.
Would I have opened the "Why I HATE Ben Settle" one .. not so likely because it sounds more you are going just going to have a rant and not take action about it.
OK .. maybe I don't really hate Ben Settle. And to be fair, I was expecting the switch in the copy because it is a fairly common pitch with the guru gaggle – "so and so sucks … because they outsold me" etc.
I think Shaun makes a valid point. Bit like the little boy who cried wolf…
Frankly, Dan, I don't… er… open your emails purely because of headline. I open all of your emails because I love your content. Having said that, it does make me a tad bit more curious to open it with an interesting headline, so I click the link two milliseconds faster than I was going to. But, you could've said something about Ben giving you intestinal disorders or some such oddity about him being a gun control advocate or something and I would've opened it anyways.
So, to be completely on the fence about it, I think you've built the trust and credibility to get your readers to open the email anyways, but for those who are new to you, a compelling headline gets them to open the email to give you the chance to prove yourself to them and build that trust and credibility.
- John
My personal opinion…
The relationship is dependent on far more than the email subject lines.
I personally found the email entertaining. I knew Dan doesn't hate Ben Settle, but I still opened the email out of curiosity, to see what the story was. And Dan did payoff the subject line in the email, so I didn't find it deceptive.
On the other hand, there are the usual "gurus" who bombard their list with sales pitches, affiliate offers, product-launches-of-the-week in almost every email. When I get a headline from them that says "XYZ sucks", I know it's just the "guru" trying to sneak up another sales pitch on me. It's so predictable, I don't bother opening the email.
Same subject line, different reaction. Because of how the different publishers treat me as a subscriber.
Kinda like if a friend punches you in the arm, you know it's a sign of affection. But if a sleazy salesman punches you in the arm, you think "whoa…. he's trying to fake rapport to manipulate me".
So my opinion is, the better someone usually treats their subscribers, the better the general reception to pretty much anything they do (such as "XYZ sucks" subject lines). There will always be some who get offended, but hey, life would be boring if everyone always reacted the same way
Thanks, Jerry.
That's pretty much what I thought, too.
2 1/2 years of publishing free content that many really big players in direct response have said is better than most PAID content… and actually LOSING money to invest my time in doing this… and one subject line can destroy people's trust in me?
Huh?
If that was people's reaction then they don't know me very well.
Just to be clear…
I didn't say or mean to imply that one subject line can destroy people's
trust in you. It's more of a drip, drip effect – especially for newer
subscribers.
It's like a bank account. Things add up, and things taketh away.
People who've been with you for years already know like and trust
you enough to open your e-mails reflexively and read whatever
you say – sometimes regardless of the headline.
But someone who is new, and doesn't yet know the quality of
your writing may pigeon hole hole you along with other marketers
who use similar attention-grabbing subject lines where they don't
really mean what they say.
I prefer to not to do that.
A couple of days ago Perry Marshall sent what I thought was a
totally sick e-mail that tied in his promotion with the news story
of the tsunami.
Were it not for the fact he's delivered great content in most of
his e-mails for years I'd have unsubbed right away.
See, there's more than one way to skin the cat.
There are other ways where you can use a subject line the generates
curiosity, gets opens WITHOUT potentially decreasing trust (especially
with newer subscribers).
That's how I look at it anyways.
True dat.
But if a new subscriber chooses to be offended by something and ignores the message because of the messenger, I'm truly glad to see them go.
I can't stand to be around the easily offended. They used to be a threat to my survival. I wrote an article about it here:
http://dobermandan.com/dont-discard-the-message-b…
Dan,
Two reasons I opened the email with whichever headline it had: The name Ben Settle rang a faint bell and I wanted to find out why and second, I open every email you send me. Trust, like Shaun said.
I suspect your split test would have been more revealing if you were sending only to NEW subscribers or a fresh list. Of course, your open percentage would have sucked worse than Ben.:-)
Overall, I agree with Shaun's comments. And Jerry's, too. The negative headline tactic is so overused, it's gotten stale for me. I would have used a headline like "Why you NEED to subscribe to Ben Settle's newsletter." That, coming from someone I trust for good advice would hook me in a nanosecond.
Owen
A couple years ago one of the gurus (I think it was Frank Kern) did a video or blog post about what a great open rate he got with the subject "Bad news". For months after that, everybody and their brother in this incestuous little IM niche was using that headline. It made me laugh every time I saw it.
And therein lies the problem.
If it looks like a duck, quacks like a duck, then people perceive it
as a… duck (whether it is or not).
The "bad news" and "I HATE Joe Blow" type headlines can have
this adverse side-effect.
Using a stale or overused headline potentially labels you as one
of 'those'…
As just one of many instead of one of a kind with something
interesting, unique and valuable to say. The latter will get you
opened and read again and again.
Part of my job as a marketer is to continually communicate and
reinforce my uniqueness and – yep – subject lines is a small part
of that.
As a marketer you can't ignore human nature. People gravitate to the negative much more than the positive. That's why the news media always says "If it bleeds, it leads."
So this kind of stuff, as I've proven in many tests, gets a much higher response… but as you've pointed out, at what long term expense?
Sure thing.
I didn't mean to imply not using negative subject lines.
You're absolutely right – negative subject lines can often get
higher open rates.
Well, I actually did run a positive Ben Settle headline back in January 2009, and it got a 29.1% open rate and 16.7% click-thru rate. The subject line was:
Why I Think of Ben Settle Every Time I Take a Shower
http://www.ryanhealy.com/ben-settle-says-take-mor…
I thought the whole thing was hilarious. Then again, I've got a sense of humor that would fit right in with junior high kids.
Ryan
I would click on that just to find out if you're a pervert or something!
Seriously, though… good example, Ryan. Curiosity is a powerful thing we copywriters and marketers can use for all kinds of things.
Would you be able to share this result over on A/B Tests dot com?
Hi Dan,
Actually, I don't think you had enough actions taken to really judge whether there was any difference in open rates or click throughs.
I sometimes split test the same headline against itself.
Now that may sound weird – but what it lets me do is test random variability. I've had open rates 20% higher on one side of a split test for exactly the same headline.
So what it means is that with relatively small sample sizes (the last one I did was with something over twice the size of the list you're reporting results for) you can get big variations in open rates just to random factors. it's nothing to do with any difference in the headlines.
So I'm always cautious now in overintrpreting the results of a split test. For my list I ignore any differences less than 20% as they could be completely random.
With a bigger list that number is going to go down. But just be wary of jumping to conclusions.
Interestingly, the numbers were still very different after a week – but closed up to about 10% apart after 2-3 weeks.
Ian
Leave your response!
These email marketing secrets
work like crazy
Just use these fill-in-the-blank
email marketing templates
to sell more stuff right now...