Split testing (or A/B testing) is a very interesting topic that I’ve read about many times, however sadly I hadn’t found the time and opportunity to try out any services for conducting such testing until this week. As many of you already know, one of the main products from the WP Mayor team is the WP RSS Aggregator plugin and its premium add-ons.
Since launching the premium add-ons around a year ago I’ve been wanting to do some conversion testing with the website, so after checking what options were available I decided to test the waters with the Nelio A/B Testing solution. In this post I’ll be sharing my review of this service based on this experience. If you’d like to know more about split testing in general Nelio have created a very good slideshow about the topic which you can check out.
Why Nelio A/B Testing?
There are a number of split testing services online, but the first point in favour of Nelio is that it is a native solution for WordPress. That means that you don’t have to login to some other website to create new tests or view tests. Everything is viewable from within your WordPress dashboard. You only go to Nelio’s website for the initial sign up stage, from then onwards everything is done through the dashboard.
So lets see how the thing works.
You first need to install the Nelio A/B Testing plugin from WordPress.org, following that head over to the Nelio website to subscribe to a plan. The service includes a 14-day trial period to get you started and you can use that to see if its the right service for you. There are also a number of plans which we’ll talk about in a moment, but for the purposes of this review, I got the Professional plan.
Once you’ve subscribed and filled in the details in the Nelio screen within your WordPress dashboard, you’ll end up with a Settings screen like mine:
What that means is that you’re ready to start off with your first experiment! Exciting times indeed. Head over to the Experiments screen and you’ll see all the experiment types available:
Let me take some time to explain what each of these are for, so you’ll get a better idea of what A/B Testing is and how it works.
Keep in mind that for every test the Nelio service will measure how two alternatives stack up against each other. What we are testing for is which of the alternatives lead to a higher conversion rate.
A conversion is defined by an action on the part of the user, and that action resulting in the opening of a particular URL in the browser. So for example from the products page I could have the goal to have users add products to their shopping cart and then click the Purchase button, upon which they will end up on the Checkout page. So the Checkout page URL will be my goal in this case.
A/B Testing for Page or Post Titles
The aim here is to see whether a change in the post or page titles will convert better. You might think this is a trivial thing but when you actually try things out the results might surprise you.
A/B Testing for Pages or Posts
Here we’re going one step further and changing the content on that page or post. So for example on the original page I could have a paragraph of text and an image to the right, while I would then create an alternative with a single column consisting of an image at the top, a paragraph and a Buy it now button. Then I would send the two versions into battle through Nelio, and see which one performs best.
A/B Theme Test
This test allows you to pit two themes against each other. Can be a very powerful test when you want to implement many changes or want to switch themes altogether. Switching themes is a big thing for most websites so you want to make sure that the new themes will get better results than the old one, else there is no sense in changing.
I’ve heard so many stories of new themes and designs actually having dramatically worse business performance, even though the site owners thought they were a much better design than the previous version. So don’t take things for granted, test before you implement.
Heatmap Experiment
This is one of my favorites. The heatmap and clickmap allows you to see which areas or your page or post are attracting the most attention, and where users are clicking. This is invaluable information and helps you realise what users are really searching for on your site, and whether they are missing out on what you really want them to be seeing and clicking on.
One thing I noticed is that the heatmap showed clicks in places where I really would not have been expecting any. After some discussion with the plugin’s developers, I discovered that there are several scenarios where it happens:
- Getting the focus on the page (even when the page has the focus), when returning from another window.
- Clicking before scrolling. We have detected that some visitors have this habit (specially with touchpads).
- Trying to click the vertical scroll bar, but they click on the wrong place (closer to the bar, but not in the bar).
- Misunderstanding between what is clickable and what is not. This is an important scenario that can help us improve our site.
- Simply because they clicked in a meaningless way.
Therefore, common sense needs to be applied when interpreting the results, and you need to keep in mind the above points before taking any decisions following the results of a clickmap test.
A/B CSS Test
This test allows you to make CSS changes and see what effects they have on your users. Very handy for quick things like changing button colors.
Whichever experiment you try, you will get fresh information about the evolution of the experiment every day, with nice graphics about visitors and conversions. Detailed statistical information is also available if you want it. Otherwise, the plugin summarizes the key information for you.
Don’t worry about your server performance. The gathered information about experiments and statistical calculus are stored and performed in Nelio’s backend servers, thus saving your server from a lot of load which would otherwise result if those processes were run on your server.
Nelio A/B Testing is built very sturdily and will work smoothly with all popular WordPress hosting providers.
With the professional plan, Nelio also offer a personalized service where one of our company experts will suggest possible tests for your site (e.g. based on their and the community experience on what has worked in other sites in the same domain) and even create and monitor the tests for you.
User Interface
One of the things the Nelio team clearly values is the importance of a native experience, which means that the average user should not even realise that he is using something that is not part of WordPress itself. How do we achieve that? By making sure that our plugin’s interface is composed of the same elements that WordPress itself users.
Now Nelio A/B Testing is one of those plugins that require a custom interface due to their nature, and it’s very hard to find developers doing things the right way when building custom interfaces within WordPress.
I was pleasantly surprised to find that Nelio adopted all the standard WordPress UI elements, and in the areas were something totally custom had to be built, they retained the same look and feel of WordPress, so you don’t even notice the introduction of this custom interface. The guys ta Nelio clearly know what they are doing and did a great job with the interface, top notch stuff, one of the best I’ve seen lately.
Support
Support is amazingly fast and curteous. I had a few problems getting the plugin to work on my site due to my particular SSL setup and the guys at Nelio were super helpful. They offer support via email and phone in both English and Spanish. You can open a support request from the plugin interface itself, there is a submenu item which opens a form which you can use for this purpose.
The Getting Started guide is very well done and using this one resource as my guide I was able to figure out how to conduct all of the tests, no problem at all here. The FAQ page also does a great job at clearing out most of the questions one might have when using such an A/B testing service, so make sure you check that out as well
I give Nelio top marks for support and documentation.
Code Quality
As a developer myself, I like to take a look at the code of the plugins I use. I did the same with Nelio and found some really beautiful and well-written code. Definitely a great sign.
Code is Poetry indeed!
Most of the code that drives this service is of course hosted on the Nelio server, the plugin is just the connector between your website and the service, however it still has a good amount of code, enough to give you a clear idea of the standards that the Nelio development team abide by, and I’m impressed. If only all plugin developers wrote beautiful, well commented code like this.
Pricing
Nelio A/B Testing comes in three flavours, two of which can be purchased outright and other is a custom solution.
The Basic plan is valued at $19/month and includes 1 site and 5,000 tested page views.
The Professional plan is $59/month and allows you to have the plugin installed and active on 5 sites at the same time. There is a limit of 35.000 tested page views across all sites plus the ability to use it on a multisite installation. You also get the added bonus of priority email and free consulting hours.
The third plan is called Enterprise and is basically a tailor made plan for your needs. To get this plan you’ll need to get in touch with Nelio and tell them about your requirements, following which they’ll get back to you with a quotation.
For the value this service provides, I think the pricing is right. You also have the trial period which I am sure will serve to convince any skeptics out there.
Conclusions
Nelio AB Testing is a fairly new plugin that only has a thousand or so downloads from the WordPress.org plugin repository at the moment, but I’m sure that as it gains more coverage that number will grow exponentially. I really love using this plugin and the potential impact it can have on your business is huge, and we talking in pure financial terms here. Even a simple improvement of your conversion rate can mean a difference of thousands of dollars each year.
Nelio A/B Testing has all the features you’d expect from a serious A/B testing tool, this is no quick buck plugin but a really well thought out solution for a problem that every site owner on the internet has. Optimizing your website is one of the best things you can do to improve your business’s bottom line.
If you own an e-commerce shop (and not only) you should definitely be doing a/b testing, and this is one of the best tools I’ve come across for this purpose. I therefore highly recommend Nelio A/B Testing and look forward to seeing your feedback. Try out Nelio for 14 days, then come back and let us know what improvements to your conversion process you managed to make by using this plugin/service.
Looking forward to hearing about your success stories!
