Evaluation of Acme Audio's Online Workflow and Software Feature Packages

Background

Organization
Acme Audio (a pseudonym) provides CAD / CAM (Computer Aided Design / Computer Aided Manufacturing) solutions to companies who create custom in-ear products. These companies range from hearing aid manufacturers to in-ear monitor and hearing protection providers.​
Program and Stakeholders
To support Acme Audio customers, Acme created videos and PDF descriptions for new features and workflows that customers might find relevant. These resources are referred to collectively as "packages." The packages include both a video outlining the workflow or software feature and a PDF document outlining step-by-step instructions on how to implement the workflow or software features into their own production environment.
Accessing the packages requires the customer to login to the Partner Portal with a username and password, which is supplied to the customer once they have signed a quote for purchase of software from Acme. The customer is then free to choose which packages they would like to view or download. The packages do not replace the need for formal software and workflow training they are merely a support tool to help customers fill skill gaps that are identified by Acme Product Management and Technical Support and the Training and Application Specialist.

There are three types of stakeholders for the Online Workflow and Software Features Packages.
Upstream Stakeholders: Several stakeholders played a role in selecting, designing, developing and making the packages available to customer via the Partner Portal. The key upstream stakeholders are the following:
  • Audio Director
  • Technical Account Manager
  • Training and Application Specialist
  • Two Product Managers
  • Partner Portal Content Team
Direct Impactees: Team members at Acme Audio and customers involved in the manufacturing of custom ear products via the digital production process. These include:
  • Around 150 Production Managers
  • Around 80 3D Specialists
  • Possibly upwards of 80 Software Trainers
  • Anywhere between 400 to 500 actual users of the software in a production environment
Indirect Impactees: Successful implementation of learnings from packages into customer production processes will not only impact the direct Acme Audio end customers, but also other groups of people:
  • People purchasing custom in-ear products
  • Dispensers of custom in-ear products
  • Governments
  • Companies with users of custom hearing protection
  • Acme Dental Academy Manager
Within Acme, an indirect impactee is the Acme Dental Academy Manager. He is currently designing and implementing an online training platform that will not have any direct impact on the Audio business. However, he is very interested in understanding the impacts the Audio packages have had on Audio customers.​
Evaluation Request
The packages include both a video outlining the workflow or software feature and a PDF document outlining step-by-step how to implement the workflow or software feature into their own production environment. Acme Audio engaged in an evaluation to provide conclusions on the overall quality and worth of the packages to end customers. The client for the evaluation is the Audio Director of Acme Audio.​

Evaluation Methods

Evaluation Purpose and Type
To determine the evaluation methods, the evaluation team (Crane et al. 2019) worked with the following stakeholders:
  • Product Management
  • Technical Support Manager
  • Director of Acme Audio
With significant input from the Training and Application Specialist, who was the lead of the evaluation team, it was decided that the evaluation findings would be used to:  
  1. Determine the impact on customers, both positive and negative, intended and unintended, of the current packages available to customers on the Partner Portal.
  2. Make better decisions on what material to develop.
  3. Identify changes required to the material development process.
Because the focus of the evaluation was to determine positive and negative impacts of the packages and based on the fact the packages have been available from the Partner Portal since August 2017 (which speaks to the maturity of the product for feasibility reasons), the evaluation team and client decided to conduct a goal-based summative evaluation with a goal-free approach. This provided conclusions on the overall quality and worth of the packages to customers, as well as fulfilling the secondary focus of the evaluation, which was to uncover potential improvements to packages and processes involved in developing the packages. We followed Chyung's (2019) 10-step evaluation process with identification, planning, and implementation phases (Figure 1).
Picture
Figure 1. 10-step evaluation process with three phases. Note. Adapted from Chyung  2018, p. 15
Dimensions
Through initial communication with the client (John Smith) and based on the Program Logic Model (PLM) for the Online Workflow and Software Feature Packages, the evaluation team began to develop a list of specific program dimensions to investigate.

Based on input from key stakeholders, the evaluation team lead proposed several dimensions and met with the Director of Acme Audio and members of Product Management to finalize key dimensions to investigate. During the meeting, the stakeholders and the evaluation team consulted the PLM. They initially agreed on investigating five dimensions in relation to the packages, two of which were not specifically outlined in the PLM (this was later whittled down to four dimensions, with two dimensions merged to ensure ease of data collection. The update was agreed to by key stakeholders).

Once the evaluation team established the dimensions, they discussed with the stakeholders how they intended to make use of the evaluation findings. Based on their input, the evaluation team helped them identify the relative degrees of importance weighting (IW) among the dimensions:
  1. Package Design and Development (Important)
  2. Customers Accessing Packages (Very Important)
  3. Customer Knowledge of Package Usefulness and Customer Utilization of Packages (Extremely Important)
  4. Other Outcomes (Extremely Important)


​Data Collection Methods
As the overall approach to this evaluation, the evaluation team followed Chyung’s (2018) 10-step evaluation procedure. The 10-step procedure assists the evaluation team to design an evaluation based on the stakeholders’ needs and the stakeholders’ use of the evaluation findings.

The evaluation team used the ARCS model (Keller, 2010) to develop a checklist to determine how well the packages were designed, specifically about what elements of attention, relevance, confidence and satisfaction (ARCS) were present in the packages and what areas may be worth adding. Due to limitations in time and the importance placed on this element (ranked least important by stakeholders), the team evaluated the top ten downloaded elements only.

As a primary focus of the evaluation, the evaluation team used Brinkerhoff’s (2006) Success Case Method, which provided structure and direction for the team to investigate Dimension 3 and allowed the team to collect information to find success cases and non-success cases.
​
While incorporating these frameworks, the evaluation team used multiple sources of data that were collected from stakeholders, including Product Management, the Technical Support Manager, the Director of Acme Audio, resellers, and customers. Additionally, the evaluation team used multiple types of data collection methods:
  • Survey—Web-based survey was sent out to customers.
  • Email—Correspondence with unknown Acme staff and resellers of Acme Audio products who had downloaded / previewed packages.
  • Interview—Scripted phone interviews were conducted. Also, one-on-one Interviews were planned to capture success cases and non-success cases. 
  • Extant data review— number of packages previewed and downloaded. ​ ​

Results

​Each of the four dimensions was evaluated using data and rubrics to determine the overall quality. The rating range used is Poor, Barely Adequate, Good, Very Good, and Excellent. The dimensional results are presented in Table 1.

Dimension

Importance Weighting

Results

1.       Package Design and Development

Important

Very Good

2.       Customers Accessing Packages

Very Important

Good

3.       Customer Knowledge of Package Usefulness and Customer Utilization of Packages

Extremely Important

Good

4.       Other Outcomes

Extremely Important

Very Good

Table 1 Dimensional Results

Conclusions
The overall quality of the packages was obtained by analyzing dimension results through a final rubric (see Table 2). The evaluation team determined that the overall quality of the online workflow and software feature packages for customers were rated Good, but improvement needed.

If both Extremely Important dimensions are rated

and

If both Very Important and Important dimensions are rated

then

The overall quality is:

Excellent

Both Excellent or Very Good

Very High Quality

Very Good or Excellent

Very Good (one or both) or Excellent

High Quality

Good or Very Good

Good (one or both) or Very Good or Excellent

Good, but improvement needed

Good or Barely Adequate

Barely Adequate (up to only one), Good, Very Good, or Excellent

Much Improvement Needed

Barely Adequate or Poor

Poor (one or more), and Barely Adequate, Good, or Excellent

Significant Improvement Needed

Table 2 Final Rubric

​The evaluation team made the following recommendations to improve the overall quality of the online packages:
  • Use the ARCS checklist to review the remaining Online Packages and identify those that require updating (overall scores less than 8).
  • Update materials from evaluation review that have an average below 8.
  • Engage with customers to find out what would make the packages more useful for them.
The evaluation team also recommends that Acme Audio Management evaluate whether the Partner Portal is the correct place for storing of the online packages. A total of 18 customers have directly downloaded materials from the Partner Portal since its inception, which equals 14% of total Audio customers with a Partner Portal login.
If Acme Audio wishes to continue using the Partner Portal as the touchpoint for customers accessing the online workflow and software feature packages, then it is strongly recommended that they make coordinated efforts to communicate this to customers, as well as the value and benefits of having this material available.

References

Brinkerhoff, R. O. (2005). The success case method: A strategic evaluation approach to increasing the value and effect of training.

             Advances in Developing Human Resources, 7(1), 86-101. https://doi.org/10.1177/1523422304272172

Crane, E., Shires, J., & Stevens, J. (2019). An evaluation of the online workflow and software feature packages for customers [MS
​
               Word document]. OPWL 530 course site. https://Blackboard.boisestate.edu

Chyung, S. Y. (2018). 10-step evaluation for training and performance improvement. Sage.

Chyung, S. Y., Wisniewski, A., Inderbitzen, B., & Campbell, D. (2013). An improvement- and accountability-oriented program

              evaluation: An evaluation of the Adventure Scouts program. Performance Improvement Quarterly, 26(3), 87-115.

              http://.doi.org/10.1002/piq.21155

Keller, J. M. (2010). Motivational design for learning and performance: The ARCS model approach. Springer.