April - 2002

Technical Evaluation Report

10. Evaluating Vendor Supplied Information

Patrick J. Fahy
Centre for Distance Education,
Athabasca University

Abstract

Distance educators are not normally equipped by their training or experience for the complex task of evaluating technologies. One of the areas of potential disadvantage is in interpreting information provided by vendors themselves, and in relating effectively with sales, marketing and technical representatives. An objective and thorough product evaluation requires that information be selected, and sometimes generated, to aid the process. Vendors may agree to provide additional information, including direct experience with their products, if evaluators know what to ask for and what to expect from vendors.

Introduction

In evaluating potential distance delivery and support technologies, researchers may have occasion to use information provided by vendors. Historically, education in general and distance education particularly have been considered niche markets by most major technology developers and providers; however, as educational technology investments in general increase, and as distance training in the private sector grows, more vendor resources are being directed to promotion of distance learning applications of all kinds.

Vendor promotion of and publicity about their products, services and achievements creates both challenges and opportunities for distance educators attempting to make intelligent technology choices. Because distance educators are not usually experienced purchasers of technologically complex and expensive systems, and because they usually are not aware of common practices in marketing and sales in these fields, some caveats and an exploration of possible strategies for gaining reliable information to inform the selection process may be helpful. While providing some cautions, this advice might also give individuals who lack marketing or purchasing experience some understanding and some strategies, permitting them to interact on a more equal basis with professional sales and marketing people.

While evaluating the capability of the technology to meet the organization’s needs and expectations, evaluation of software and hardware should also consider the readiness of the organization for adoption of the technology. Welsch (2002) reminds evaluators that innovations of all kinds encounter financial, structural and cultural challenges in any organization, and that an assessment must be made of the organization’s readiness to change in order to successfully incorporate new technologies. The evaluator’s task includes both looking at a technology’s potential usefulness to the organization, and assessing the organization’s readiness to adopt the technology successfully. A mismatch between corporate readiness and the technology’s demands as an innovation could be disastrous for an unprepared organization.

Assumptions and Strategies for Evaluators

A key assumption in the following discussion is that vendor-supplied information can be used effectively in evaluating a technology, if the potential purchaser understands in advance some facts, and, where strategically appropriate, engages in some timely activities.

1. Truth vs. the whole truth in vendor publications. While there should be no misinformation, vendor-supplied information is almost never complete. Vendors know they are not on the witness stand, and they therefore do not feel they have to tell the “whole truth.” (They may be following St. Paul’s insightful advice to the citizens of Corinth: “All things that are true for me may not be expedient.”)

• It is not reasonable to expect vendors to provide complete, objective information on their own products; the evaluator must seek comparative information and evaluative data (caveat emptor).
• The evaluator should also be prepared, if doubtful of any claims, to challenge vendors to prove them (see 2, 3, 4 and 6, below.)
  

2. Real vs. ideal performance reports. Similar to the above, information provided in marketing and sales publications commonly highlights only the strengths of the products and the company, usually in a non-technical sense, often in comparison with known weaknesses of the competition, and sometimes referring to performance under laboratory (rather than “real-world”) conditions of use.

• Marketing and sales information should be regarded with caution, even scepticism, until corroborated by independent testing.
• Objective technical information should be sought, to replace merely qualitative or comparative product descriptions.
  

3. Access to technical information. Related to the above point, evaluators should be able to access quality, objective information from a vendor, consulting, as necessary, appropriate internal and external expert sources. Technical specifications include credible engineering information describing the performance or requirements of the technology under controlled conditions.

• Technical specifications should reflect benchmark checks, or should be referenced to externally recognized standards, where applicable.
• The evolutionary history of specifications and performance of a product should be made available, so evaluators can see how the product’s functionality has evolved.
• “Next release,” or otherwise promised future features, should not be mentioned unless they actually exist, and may be evaluated in some form.
  

4. Reference-site information. Reference sites are customers of the vendor who know (and presumably are happy with) the product, and who are also regarded by the vendor as using the product intelligently, effectively and appropriately. Reference sites can be regarded as demonstrating the most effective potential outcomes likely achievable with the product.

• Evaluators should have access to a choice of reference sites. Vendors should assist potential purchasers in selecting reference sites to contact or visit.
• Reference sites who are partners of the vendor, who have a financial interest in the product, or who derive any financial reward from referrals should not be used without disclosure of these facts.
• After purchase, users of a product who provide the vendor with valuable feedback, especially feedback leading to product refinements or development of new products with commercial potential, should receive some benefit, especially if student, staff or faculty assistance contributed significantly to the enhancements. Evaluators should seek advantages for the organization when partnering with a vendor to improve the vendor’s product.
  

5. Business facts. Information about the company itself, as a corporation, can sometimes show how well managed and accepted the company and its products are in its field. This information is usually only readily available on public companies, which by law must make public certain information about their business dealings. While information on public companies should be entirely accurate (federal regulations apply), the data may be difficult to interpret, and may also be incomplete.

• Company size and reputation may indicate corporate health, and ability to deliver ongoing product support and services. (Reference sites may be a better source of this information; see 4, above.)
• Despite the potential difficulties in acquiring and evaluating it, information about the size of a company, the location of its offices (potentially important for support considerations), its technical history, its pattern of growth (market share, product acceptance), and its product plans may be useful in assessing corporate health and “fit” with the potential purchaser’s intentions.
• Quarterly and annual reports of public companies should be available for perusal.
  

6. The right to pilot-test. Because the acquisition of complex technologies, often unfamiliar in the organization, represents both a major investment and a high risk, the opportunity to pilot-test the product may be of value to some purchasers.

• Vendors should approve “no-obligation” pilot-tests, and assist the potential purchaser with orientation, installation and other associated tasks. Tests should be of sufficient length, should be conducted under conditions of experimental rigour, and should correspond well enough to actual conditions of use, that results are likely to reflect closely actual program performance. (No surprises should occur after purchase, for want of testing before.)
• An agreement about costs related to testing should be reached before the pilot commences; for ethical reasons, the vendor’s direct costs should normally be borne by the potential purchaser, to assure impartiality in the evaluation.
• Use and publication of the results of pilot- or evaluative tests should be agreed upon in advance of any pilot-testing.
  

7. Price and support stability. Sales practices in relation to technologies often change as marketing opportunities and products evolve; in some companies, these may also change if sales representatives need to book revenue at the end of a quarter or a fiscal year. To potential purchasers, frequent changes in product packaging or in sales strategies may lead to questions about the product or the company itself.

• Pricing should be quoted with no hidden or deferred costs, and should permit perpetual use by the purchaser of the version purchased, including any ongoing necessary product support. (Educational customers do not normally have resources for annual renewals of licenses, or mandatory upgrade fees.)
• No product should normally be “orphaned” by the vendor. While the product ages, it should continue to be fully supported, even if it is “obsolete” in relation to more current versions.
• If an older version of a product eventually becomes embarrassing to the company, or too expensive to support, the vendor should make an appropriate upgrade possible within the resources of the purchaser. (The principle is that the vendor incurs a moral responsibility to each educational purchaser to maintain product integrity so long as the user is satisfied with and is using the product.)

Conclusions

Vendors competing for business in the highly competitive and often financially lean, even distressed, distance education marketplace, provide product information of different kinds in a variety of forms. As well, the volume and intensity of these kinds of representations may be considerable: marketing and sales expenditures in relation to educational technologies, especially of software products, may comprise a major part of the cost of doing business, sometimes significantly in excess of the cost of the product itself. For example, in 2001, PLATO Learning Inc. reported that its costs for “selling, general and administrative expenses” had increased 30 percent, to 63 percent of revenue, up from 61 percent of revenue in 2000; in 2000, the increase had been 29 percent. Bearing in mind that PLATO is a mature company, with established products and customer based, and therefore less in need of core development and support expenditures, for the same two years “product development and customer support” comprised 11.3 percent and 12.4 percent of revenue, respectively.

Evaluators should know how to judiciously assess and use vendor-provided material, including knowing when to ask for more or different kinds of information. Gathering needed product and organizational data, generating new data where necessary, and helping colleagues see how the data relate to intended outcomes, are core tasks of the technology evaluation process in a training or educational environment. Because of the long-term organizational implications, the evaluation and selection process must generate an accurate and realistic description both of the technologies’ capabilities and of potential impacts on the organization. Vendors may legitimately help in this process if their role and contributions are properly managed by the evaluator.

References

PLATO Learning Inc. (2002). 2001 Annual Report. Retrieved March 14, 2002, from: www.plato.com

Welsch, E. (2002). Cautious steps ahead. Online Learning 6(1), 20 – 24.

This series of software evaluation reports will continue with reviews of other online collaborative tools.

N.B.Owing to the speed with which Web addresses are changed, the online references cited in this report may be outdated. They can be checked at the Athabasca University software evaluation site: cde.athabascau.ca/softeval/. Italicised product names in this report can be assumed to be registered trademarks.

JPB. Series Editor, Technical Notes