I really enjoyed last night’s panel discussion on user research — and I learned a lot (hence for the long, long notes!)
The event took place at Texas Coworking — what a beautiful facility! After a bit of networking, the panel was introduced by Julie Lowe, current president of the Austin Usability Professionals Association and Usability Manager at Sentient Services. The rest of the evening went as follows:
- Amy L. E. Jones -Senior User Researcher at Convio, Inc – on Morea & UserVue
- Will Meurer – UI Developer for Mumboe – on Silverback, ClickTale & UserFly
- Tanya Payne – Usability Test Lead at OpenText – on VMWare & WebEx
- Audience Questions & Comments
Amy Jones
Senior User Researcher at Convio
Convio offers online software and services to help inspire and mobilize people to support nonprofits. They have a pool of 300 opt-in usability participants, who they screen via SurveyMonkey when new testing is needed. Right now, they use Morea & UserVue to conduct usability testing remotely.
Morae is the premier (read: priciest) software for usability testing and market research. The suite ($1495) is actually composed of 3 software components:
- Morae Recorder: Captures audio, video, on-screen activity, and keyboard/mouse input during a research session. Depending on the research methods you’re using, you might record camera video only (no screen recording).
- Morae Observer: Enables team members to watch the customer’s experience, make notes, and flag tasks in real time. Observer connects to the computer running Recorder via your network (LAN/WAN/VPN). Multiple Observers can be connected to a single Recorder. Purchase additional Observer licenses to give more people on your team a chance to observe sessions…or to support multiple, simultaneous sessions.
- Morae Manager: View and analyze Morae recordings, automatically calculate metrics, generate graphs, and create highlight videos to share with stakeholders. Manager is used after the session; not during.
UserVue is TechSmith’s solution for remote recording & observation. It is optimized to work seamlessly with Morae manager. Testing is recorded, and observers may watch while the test is in progress; testers never see them.
The downsides UserVue include its price, its platform (it works only with PC and always launches an IE window); and most importantly, its longevity. Rumors abound on the web that TechSmith intends to discontinue UserVue in Summer 2010.
Amy likes to present video to the developers she works with, and she has found that series of 5 short clips of the same task are the most compelling — nobody can argue with the findings. As such, she appreciates that with Morae & UserVue, she is able to review at up to 4x the speed of the video if she chooses — playback stays very clear. She also reports that jumping around in the video and moving back and forth is easy to do, without buffering. She exports the video to .AVI or .WMV format and packages it in PPT.
When asked how she convinced her company to invest in Morae, Amy explained that she installed the trial version to record tests at the client conference. Once they had all the footage, she showed her boss and said “Hey, we’re going to lose all this in 30 days!”
Will Meurer
UI Developer for Mumboe
Mumboe offers web-based contract management. Will is one of about 15 employees at the startup. As a UI developer, he is technically on the development team. He wears multiple hats – JS, HTML, interaction design, occasional graphic design, usability testing & UX research. The company is too small for employees to spend a lot of time on any one thing, so they need cheap, quick, fast, & easy usability products.
Silverback
Silverback was developed by and for ClearLeft, one of the premier UX agencies in the UK, to be a low-cost (read: $49) screen capture software for exclusively for the Mac OS which markets itself as a tool for guerrilla usability testing. Ten percent of the purchase price goes to “Save the Geurillas” campaign.
Like Morea or Camptasia, it records the screen, webcam & audio input and can show clickspots and keystrokes – cmd or enter, etc.
Will describes Silverback as really easy to use because of its simplicity; He can set up a test in about 20 seconds. The software allows him to transform his MacBook into a mobile usability lab, running test anywhere on the cheap. In fact, he put it quite bluntly; “We actually do usability testing because we have this tool.” The recordings allows him to test by himself, focusing on moderation during the test and analyzing the results afterward.
On the downside, Silverback is for Mac only. Although it’s coded for cross-browser consistency, his caompany’s product does call browser dialogs at times, which can throw a user off if he or she is not familiar with Mac dialogs.
Another weakness of Silverback is that it doesn’t offer metrics or analysis. All you have is your video and its timeline. And exporting from silverback takes forever, but you cannot play the video inside of the software. You have to export to see what you recorded.
Clicktale & UserFly
ClickTale and UserFly are direct competitors, both tracking every mouse move, click and scroll to create playable videos of customers’ entire browsing sessions. Will described the tools as “kinda like Google Analytics on steroids”. Each is implemented by adding a simple line of JavaScript to the pages of your site.
Will described the data output as fascinating. But they’re not using it.
Why not?
- Uncertainty. Since there’s no leading or interaction with the subject at all, you cannot be certain about what the user was trying to do or whether they succeeded. This can create more trouble than it helps.
- Replay. Replay of each page shows only the initial capture, so any dynamic interactions that happen once the pages have loaded are not clear in the video.
- Privacy. Customers trust you with your data. At Mumboe.com, users store uber-private contracts. Letting ClickTale collect and save images of those contracts on its servers would violate the users’ expectations of privacy.
Tanya Payne
User Experience/User Research Contractor at OpenText
Tanya is tasked with testing highly interdependent stacks of technology, including LDAP, Oracle, Apache, and the content in their product, Vignette. As an enterprise-level provider, they sometimes test highly secure technology environments.
VMWare
With VMWare, Tanya can access the necessary systems without spending hours setting up the technology. She no longer has to borrow development resources to configure the testing station. And, of course, she doesn’t require physical servers. She also appreciates that with VMWare, she can customize an instance and then dispose of it after the test — no long uninstalls to deal with.
On the flipside, she’s found VMWare remarkable unreliable. She encounters an error every 3-4 tests and finds herself needing to restart the server a lot. And she also laments the fact that she has no option but to contact a VMWare specialist when there’s a serious problem — the internal resources cannot help her.
WebEx
On top of VMWare, Tanya uses WebEx to go remote. Her company already had a site license to support online meetings and presentations. Unlike LiveMeeting and GoToMeeting, WebEx installation does not require administrative privileges.
WebEx lets clients know when new participants join the conference, which Tanya perceives as beneficial full disclosure. With it, she can record sound & screen.
Unfortunately, Tanya reports that its slow. Really, really slow. It struggles with different sized monitors, which often results in skewed presentations. And recordings are made in a .ARF file format, which requires conversion. Conversion actually takes longer than the initial recording.
Audience Questions & Comments
In response to Tanya’s trouble with VMWare, one attendee suggested Sun’s Virtual Box as a VMWare replacement. It’s free and easily configured from the GUI. Windows 7 also offers virtualization but Paul Daly has found it to be problematic.
Concept Testing
One attendee asked about concept testing and whether panelists use the same tools when they are tasked with collecting qualitative feedback on aesthetics. In the software space, the idea of a “concept” might not resonate as clearly as it does for those on the marketing side.
- I explained the desirability testing white paper and how eager I am to try that method of concept testing.
- Someone also mentioned a standard survey regarding “attractiveness” but I didn’t catch the details.
- Will suggested trying fivesecondtest.com, an easy & low-cost way to identify the most prominent elements of your user interfaces. You can invite your own users or depend on the community’s participation. Mumboe posted their test on Twitter to get feedback from their follows. With the tool, you can ask users what they remember about your design or have them click on the things they find most prominent. Will suggested that it’d be useful for A/B or even A/B/C tests.
- Someone else suggested another tool in which Noah Masterson, another attendee, is a silent partner on. Grupthink was described as an open-ended way to capture thoughts and let the great ones bubble up. Noah said that it’s a way to harness qualitative feedback on anything in question & answer format. I wonder how it might be applied to concept testing.
TreeJack
I asked attendees if they’d ever used TreeJack, OptimalWorkshop’s tool for testing hierarchy/taxonomy. Lauren Moler, from the Web Content Group at National Instruments, explained that TreeJack is paid monthly and offers 1 free 5-question test with as many users as you want. Other attendees expressed interested in OptimalWorkshop’s other products, which include:
- OptimalSort is a well-known tool for digital cardsorting.
- ChalkMark offers rapid and high simplistic usability feedback. With one task, one image and one click, testers get a heatmap to show the concentration of clicks.
KJ-Technique
Will mentioned a technique called the KJ-Technique, a silent exercise used to quickly reach rock-solid group consensus. I’m definitely interested in learning more about this technique for moderating group interviews and gathering requirements.
Incentives for Usability Test Participation
When the topic of incentives arose, we learned that panelists are each in unique situations.
- Tanya has tried to offer incentives, but has gotten some kickback about paying customers to participate. At OpenText, nothing valued over $25 can transfer from a staffer to a customer.
- Since Amy has an eager base of client volunteers, she doesn’t generally offer incentives for usability participation. She did use a sweepstakes offer for a $150 gift card to motivate survey completions and got a whopping 1500+ responses.
- Will conducts quick and dirty usability testing with anyone he can find. Sometimes his participants are friends and family, and other times they people he finds at local coffee shops. He will buy them a cup of coffee in exchange for participation, as he only needs 5 or 10 minutes of their time to get feedback.
Panelists seems to agree that incentives should be offered when non-clients or the general public are needed for user research; as such users have no intrinsic motivation to offer feedback.
Presenting the Findings
Video
While Amy depends heavily on video to create presentations that have impact and prove her point to developers, Robin and Tanya report that they’ve managed to get enough authority with their developers that they no longer have to prove each point. Their issues are stack-ranked against all the issues that the team handles.
Will also confirmed that although he has the footage, he doesn’t spend a lot of time splicing the video for presentations. He works in a quick-turn environment so they take findings and run with them.
Quantitative
When asked how they measure success quantitatively, panelists and the audience all seemed to agree that although numbers might not be the part of job that excites us most, they are useful for proving changes over time, especially if management or the consumers of the data are numbers people.
Graphs can showing changes over time in success rates (% of users who are able to complete a task), time to complete task, first instinct accuracy, and an interesting metric arose with which I was not familiar: the SUS.
Here’s what I know so far: SUS is a measurement common in software, and it stands for system usability scale. SUS should be determined via survey and should be revisited with every release. In order for the SUS to be meaningful, you’ve gotta have a benchmark.
According to Amy & Tanya, a recent study uncovered a correlation between the SUS and the net promoter score (NPS) — the key metric for many marketing and experience teams today. From what they said, an average score (70) on the SUS has been found to correlate to a neutral score (0) on the net promoter. Of course — correlation is not causation, but future research around these 2 metrics could potentially shed some light on the importance of usability.
Since usability tests involve limited user sets and often cover small areas within the overall user experience, Amy expressed concerns about using usability testing to collect SUS. Tanya agreed — numbers from any particular round of usability testing do not represent the SYSTEM overall.
Tanya framed it well: “Usability exists in context: a particular task at a particular time and a particular user.”
Collecting SUS feedback during usability might be a way to identify which individual features are dragging down the system’s overall usability.
Analysis Time
Another attendee asked how long it takes panelists to analyze each usability session.
- Tanya explained that analysis has historically taken about as long for her at the test did – a 1:1 ratio. Since OpenText is moving toward an Agile development workflow, she & colleagues are looking for ways to move more quickly, which means she is now taking notes in a spreadsheet as she does when possible.
- Amy splices videos into persuasive presentations for the development teams. This takes her 2 to 3 times as long as the test itself.
- Will works fast — performing up to 5 5-minute tests, then writing up the feedback all within an hour.
All in all, the event was a success. The turnout was great, the panelists were humble yet knowledgeable and as always, I was impressed by all the talent and professionalism Austin has to offer. As a tech hub, Austin has a large but mysteriously silent community of UXers. The idea of tapping into and coordinating this community is so exciting, and I really look forward to getting more involved in just that.
See also
Thanks Erin! Thorough and informative write-up of the panel!
Very well done, Erin. Thank you! If anyone is interested in the pro edition of Grupthink it’s at http://www.grupthinkpowered.com.
Wow, now that is a pretty complete write up of the event! It was certainly a great group and insightful.