How we handle user requests for Wild Apricot software changes
How exactly do we handle suggestions from our users about product changes and enhancements? I have written a bit on this in the past, but this question comes up quite frequently in surveys we receive every month, so I figured I would do another, more detailed description of our current process.
There are quite a few inputs into our decision-making process, such as:
- performance (ongoing work on maintaining and improving product speed)
- refactoring (rewriting the old code to make it easier to maintain and update)
- marketing department needs (e.g. to monitor stats for new trials and upgrades)
- technical support needs (e.g. tools to review system logs)
- billing administrators’ needs (e.g. handling payments by checks and overdue payments)
- ideas from our product design team on how to make the system easier to learn and use
But to keep this post manageable, I will focus on the process for handling suggestions and requests we receive from our clients.
The main starting point for suggestions is our wishlist forum - we constantly encourage our clients to submit their ideas through the wishlist forum. We also receive suggestions during support interactions with clients and in the surveys we send out to our clients every month (we receive a few hundred surveys each month from the administrators of accounts whose upgrade anniversary falls into the given month).
Unless the suggestion is already well known to us and is clearly understood, we normally request clients to go to the wishlist forum and create a new discussion thread with their request - or add their comments to an existing one. This is done to funnel all the requests through the same process so that we can collect feedback from other clients - and also have a centralized source of inputs for our product design team to process. Also, in this way we can dialogue with the originator of the suggestion to clarify the details.
The wishlist forum is open to the public to view. However, creating new threads or adding comments does require a registration, since the discussion forum software we use, unfortunately, can't use the login information from the Wild Apricot system itself. We know this prevents some people from posting their feedback and have been researching options to replace our discussion forum with an easier to use and more sophisticated piece of software. We have finally picked the replacement and will be planning and executing the transition in the next 3-6 months.
Every post to our Wishlist forum triggers an email to our product design team. I used to process the bulk of these myself, but nowadays my colleague Evgeny, who runs our product design team, handles most of the suggestions himself.
The initial evaluation is to make sure we understand the suggestion and that it makes general sense within our vision for Wild Apricot. Most of the time we do a quick reply to acknowledge the request and ask for some clarification / additional details.
Quick tip: what really helps us and other clients understand a particular request, is a detailed example - please do describe your situation in detail so that we can understand the underlying reasons.
Evaluating which suggestions are ‘popular’
We already have over 6,500 paid accounts on Wild Apricot, so there is a wide range of types and sizes of organizations using our system (even though the majority fall under the general umbrella of associations, clubs and charities). If we were to try and address each individual request from each client, we would quickly go crazy - and Wild Apricot software would quickly become a total confusing mess of buttons and screens. So a very important first step for all suggestions is to evaluate how many other other clients have a similar need.
Let’s say someone requests that we integrate Amazon payments to handle online payments - and someone else requests QuickBooks merchant service. How do we decide which one should take priority? The simple yardstick we use now is counting the number of people who commented on the suggestion - or gave it high rating.
Here is an example:
This particular suggestion has 34 comments and has been rated 3 times at 5 stars. It’s not totally precise, since the same person might have posted multiple times - plus there are comments from our team - but it is a decent approximation.
The total number of threads in our Wishlist forum has been growing steadily every year - it currently stands at 720 threads. However the majority of requests have little support (e.g., only one or two comments or ratings). So upon analyzing our database, we have come up with a rule of a thumb - we consider a suggestion ‘popular’ if it has been commented on or rated at least 10 times. The number of these ‘popular’ threads is currently at about 80 and has been growing very slowly:
These threads are the main focus of our product design team.
The biggest weakness of this stage in the process, is that many people simply might not have had a chance to voice their support for a particular suggestion. One way we try to address this is to pay special attention to requests mentioned in support emails and surveys, search for existing wishlist threads and then send a direct link and an explicit request for the client to add their comments/support to the thread in question.
Analyzing complex requests
Some requests are relatively straightforward - as the example in the previous section. Most of the requests are actually much harder to deal with.
Let’s say someone comments: “You should redesign Wild Apricot to better support family memberships”. And a bunch of other people pipe up with “Yeah, you should do that” and “Wild Apricot sucks for family memberships”.
This gives us a pretty good idea that handling family memberships in Wild Apricot has to be improved - but very little input into what exactly we should change. So now our product design team jumps in and tries to get the additional details by posting clarification questions like:
- What specific aspects of family memberships would you like to see improved?
- How would it ideally work for you?
- How are you currently dealing with it?
- Could you describe the scenario of how you typically use family memberships?
As additional feedback comes in, we can start formulating tentative theories and post our description of how we understand the problem and our ideas for possible ways to solve it (which might be quite different from ideas provided by other posters).
It is not uncommon for us to realize at this stage that we are actually talking about several very different scenarios and we might end up splitting the original request thread into multiple threads.
Depending on how quickly we get the replies and or the speed of our own reaction, this stage of the process can take from a few months to a couple of years, until we narrow down the exact problem and come up with a possible high-level solution.
So now we have a popular suggestion and a tentative solution that has been validated with a number of clients.
The next stage is for our product design team to develop the detailed design of the proposed solution. This includes design mockups of all the affected screens, descriptions of changes in the processing logic and so on. This is a very time-consuming stage and is pretty much hidden from you. We have been thinking how to improve on this - one idea we have been trying out is posting a summary of our designed solution back onto the relevant Wishlist forum thread for final validation by the interested people, e.g. see mobile administration preview.
Scheduling and development
Now the baton passes to our development team who review the various requests that have been analyzed and designed, estimate the development work involved, map to our available resources and decide which ones are we going to take into development for the next release.
When a particular request has been selected for the next release, only then can we start talking about possible dates for the solution to be released. Unfortunately, even if a feature is well analyzed and designed by our team, it still might not be possible to include it in the release -- sometimes this feature might conflict with other features or other code in the system.
Our product design team stays involved during the development process to see it through.There are also a few of cycles of testing involved:
- testing by our QA team of each particular feature
- review and acceptance of each feature by the product design team
- ‘regression testing’ of the whole new version with all the features involved to ensure they mesh well with each other - plus retesting the whole system to ensure that changes haven’t broken anything else.
As Wild Apricot has gotten bigger and more complicated, these testing and acceptance phases have been getting longer and longer (currently regression testing alone takes at least 2 months).
We have learned the hard way not to skimp on these stages, after our eagerness to release new updates and address user requests led us to rush through the testing phases a couple of years ago. That was a huge pain for our clients, support and development teams.
One bright light in this tunnel is that for the regression testing we have found a way to reduce the time needed by creating automated tests for common workflows in the system. Also, we have started the work to change our system code to be less ‘monolithic’ so that we can develop, test and release parts of the system without affecting other parts. This has been done for the payment system - and we will continue with other modules throughout this and next year.
I hope this was a useful glimpse into how we approach handling product requests from our clients. As you have seen, it can be a pretty long time between receiving your ideas and being able to add them to the system. But we always welcome new ideas - they are what helps us make the system better over the long term. I would encourage you to contribute to the wishlist forum with your ideas.
Feel free to post your questions here and I will be happy to elaborate on this process.