How to Evaluate Software and Vendors for Selection

You know you need new software, for whatever purpose, and every piece of software comes with a vendor.  Wading through the possibilities may be daunting, particularly with the “how do I know what I don’t know?” factor.  On what do you base your selection?  How do you objectively evaluate, rather than just going with a gut feeling about the software and its vendor (and how much it will cost)?

InfoTech Research Group (ITRG), with whom Cimatri partners on industry technology research, suggests the following concepts on which any software and vendor selection should be based.  In this article, we will look at those criteria and the questions you should be asking when considering any software purchase.

Business Value Created

Ah, this one--- right at the top of the list.

The purpose of any software should be to create value for staff, users, constituents, customers, members, and stakeholders.  “It is imperative that any software selection be aligned with the organization’s business needs and deliver enough business value to justify the cost.” (InfoTech Research Group) A further factor to consider is consistency--- will this software and its vendor consistently provide value that justifies the cost now and in the future? 

Breadth of Features

Does the software provide more than just the basic features to accomplish a task?  While your focus initially may be to answer requirements or issues that are causing your users pain, it is important to look beyond the immediate.  What if your requirements change drastically over the life of the product?  Is the product feature-rich enough to be able to change with your needs, and are its vendors able to support that change without a major impact to the structure of the product?  In other words, are customizations going to be the only answer to change?

Quite often, users will say, “oh, we don’t need that right now,” and review of features beyond the immediate need are dismissed.  Your mindset should be, “we don’t use that feature right now, but we could choose to use it in the future.” Be sure you are looking for solutions that possess the flexible features to accommodate the next good idea, should you decide to use them. 

Quality of Features

The features of a solution that are most important to you should behave reliably and as expected, without workarounds or unexpected “blips.”  When considering selection, the vendor should be able to demonstrate this reliability, and it is a good idea to ask the vendor’s references about their experience in this area.  Does it work as expected, and, if not, what is required to make it work consistently?

Product Strategy and Rate of Improvement

With the rapid pace of technology in general, it is expected that a solution will keep up with tested trends.  For instance, software should be on the most current version of its underlying foundation, its “built on.”  But, more than that, the vendor should demonstrate an understanding that their software must be improved upon regularly.  What is the roadmap for continually modernizing the solution?  How often are updates or upgrades done?  Are customers required to accept those upgrades, or are they allowed to lag behind on versions? Does the vendor demonstrate insight into new technology trends that may improve (or impede) their product?  And how does this rate of improvement directly affect the solution you have in place once you have purchased? 

Usability and Intuitiveness

In plain English--- is the software easy for its intended user to figure out?  This question may cover a variety of users, and it will be important for you to understand what users will provide the “sway” factor, i.e., whose use of the system will most impact the organization and its goals.  Is it your C-suite user or the call center user?  Which functional area holds the most ownership of the data and use of the system?  Is it external constituents or members? Or is it everyone?

Another piece to this, is what do your users know already?  Is the software familiar, like something else used in the organization or similar to the previous software so users can adapt to it easily?  If the new software is completely new to them and will require training to use effectively, that is a factor—and a cost both in terms of budget and effectiveness--- to consider. 

Vendor Support

Vendor support is key, of course.  When you pick a piece of software, plan to build a relationship with its vendor.  Ask and understand how they communicate, what the cadence of communications are, how accessible they are to you when issues arise, and ask how they will handle your organization’s expectations. If necessary, consider contractual commitments to a mutually agreeable communication structure.

Some vendors will, in the implementation plan, ask for commitment to typical task turnaround times.  For instance, the vendor may offer a window of x number of days to complete standard tasks and expect you to complete review of that output within x number of days.  Feel free to ask about these at any time in the selection process, This will likely become a contract or SOW point, so make sure you are comfortable with the proposed cadence.

Ease of Data Integration

No system stands alone in today’s technology landscape, so integrations with other systems to expose data to either side are expected.  The first step here is understanding what you have (or plan to have) that will need to be integrated into the software being purchased.  A useful tool that Cimatri uses is an Application Portfolio Assessment to gather the landscape of what software is actually being used in your organization. 

Whatever tool you use to accomplish this, make sure that the vendor has, wherever possible, standard integrations with the other platforms external to it.  If they aren’t standard, what will the cost be to develop?  Is your new system consuming the integration or is the external system consuming the integration from your new system?  For any of the integrations, who will maintain the integrity of the integration when changes or upgrades are done?  Will there be a cost to that? 

Ease of Customization

If your new software functionality doesn’t quite fit your needs as is, out of the box, the word “customization” frequently comes up.  Any customization should be well-defined and as simple to consume as possible.  However, before you commit to the concept of customizing any piece of software, it’s necessary to understand a bit of terminology.

“Customization” means there is code that is created by developers outside the standard software to accomplish some requirement you have for the system.  This entails a cost to develop and typically a cost to maintain, perhaps every time there is a change or upgrade to the base product.  And by cost, I mean both dollars and efficiency.  If a customization breaks, time and effort are required to correct that, and while you’re waiting, the feature may be unavailable.  Depending on the criticality of that feature, this could be a major issue.

Different from customizing software is the vendor’s ability to “configure” the software.  Configuration means the option to have the software to behave in a certain way has already been built into its capabilities; it just needs to be told to behave that way.  There is little risk of breakage upon upgrade or changes and no additional cost to configure the system. 

You should look for configuration rather than customization wherever possible.  However, if it must be done, make sure you understand the costs and responsibilities when changes are required. 

Ease of Administration

The administrator(s) of the new software should get some special attention.  While many software solutions concentrate (and need to) on end user experience and functionality, it is important to be sure that those who work in the “back end” administrative functionality of a system have adequate tools to do so easily.  Make sure that your intended software provides adequate administrative tools and training in those tools.  These are the users who are responsible for ongoing software health and internal support. Make sure they can be fully comfortable in their administrative abilities within the system and can understand how everything works together, and that they will have continued administrative level support from the vendor. 

Availability and Quality of Training

Unfortunately, this topic always seems to be at the end of the list of things to consider when selecting a new software and vendor.  It should be among the first.  Training on a new piece of software typically begins with implementation, when a select group of users who are responsible for vetting the system set up are trained in its use and continues into training all users around “go-live.”

However, too many organizations and vendors let training dwindle after go-live.  Through the life of the engagement, the vendor should be willing to provide: 1) new user training, 2) periodic refresher training, 3) training on new features as they become available, and 4) remedial training on features or procedures.  These may come in a variety of channels, from in-person or video training to contextual help to standard courses available online.

Charge your organization with committing to training.  While there may be costs to consider, it is important to understand that dissatisfaction with a system most frequently comes not from its lack of features, but the user’s lack of understanding on what the features are and how to use them.  So, think of your training budget and time as health insurance, and make sure the vendor is committed to user education using channels that your organization is comfortable consuming.

Ease of Implementation

Once a selection is made and the celebration over, you must get on with the work of implementation.  As you are evaluating software and vendors, understanding your business and system requirements, and giving the vendor that insight is critical to a solid implementation plan.  With that information, the vendor should be able to offer an expected implementation timeline and cost.  Be aware of where there will likely be needs for customization, and be sure, in the selection process, to meet the implementation team. 

The team demonstrating and selling you the software may not be the team with whom you will have the close relationship for implementation, so ask to meet them.  How comfortable are you with their implementation philosophy and attitude?  How much work do they anticipate your users will need to do to assist them in getting the system off the ground?  What are their expectations, and what are yours?  Make sure you have a match here.  It will impact the success of your new system. 


There are many factors to consider when selecting a new software and its vendor.  It is important to remember that the two go together, and you must be confident in your choice of both.  The topics above give you a basis on which to make decisions. 

As strategic technology consultants concentrating in the association industry, Cimatri provides intensive assistance to help you work through the important decisions around the selection of key systems and their integration. Focusing on people, process, and then technology, we use both quantitative and qualitative methodology to inform our engagements.  

Contact Cimatri for more information on how we can help you select your new system, confident that you have made the best choice available.

Subscribe to our Newsletter

Contact Us