International Review of Research in Open and Distributed Learning

Volume 18, Number 3

May - 2017

Improving the K-12 Online Course Design Review Process: Experts Weigh in on iNACOL National Standards for Quality Online Courses

David Adelstein¹ and Michael K. Barbour^2
1Jinsong Schools, ²Touro University, California

Abstract

Within the K-12 online learning environment there are a variety of standards that designers can utilize when creating online courses. To date, the only research-based standards available are proprietary in manner. As such, many jurisdictions have begun adopting online course design standards from the leading advocacy organization, which that have yet to be validated from a research perspective. This article reports on the second phase of a three-stage study designed to examine the validity and reliability of the iNACOL National Standards for Quality Online Courses. Phase two utilizes two groups of expert reviewers to examine and provide feedback with goal of further refining these standards (after the standards had been scrutinized through the lens of the available K-12 online learning literature).

Keywords: K-12 online learning, K-12 distance education, virtual school, cyber school, online course design

Improving the K-12 Online Course Design Review Process: Experts Weigh in on iNACOL National Standards for Quality Online Courses

K-12 online course designers have numerous options when contemplating standards to guide their development of asynchronous course content; however, not all standards are freely accessible. Some institutions, such as the Virtual High School (VHS), have their own publicly available, in-house process (Kozma, Zucker, & Espinoza, 1998), while other institutions use standards that are part of a proprietary system used by certified experts, such as the Quality Matters (QM, 2014). In 2007, and then later updated in 2011, the International Association for K-12 Online Learning (iNACOL, 2011) released their National Standards for Quality Online Courses. These standards were largely based on standards released earlier by the Southern Region Education Board (SREB), with some additions due to iNACOL's involvement in the Partnership for 21^st Century Skills initiative (National Association for K-12 Online Learning [NACOL], 2007). The iNACOL standards used a rubric that covered five different areas (i.e., content, instructional design, student assessment, technology, and course evaluation and support) to review the overall quality of a course (iNACOL, 2011). Since its initial release, the standards have been implemented in a variety of jurisdictions, including for use in states such as Michigan and Texas ("Making Online Learning Accessible," 2015; Oakland Schools, 2015). However, even as the standards remain popular with legislators and policymakers, there has been no research published on the validity of the standards or a review as to how they relate specifically to online course design.

The study reported in this article follows an earlier phase in the validation of the iNACOL standards (see Adelstein & Barbour, 2016). Phase one of this larger research initiative reviewed the construct validity of the iNACOL standards (Drost, 2011). Using contemporary research, each of the 52 elements found in the iNACOL standards were reviewed to determine the level of support each standard had within the research literature. Each standard was compared to research into K-12 online learning, as well as the broader field of online learning and course design. The following article describes phase two of this validation process, which consisted of three rounds of expert review over the revised iNACOL standards from the first phase. The authors will begin by briefly discussing the current state of K-12 online course design literature. The three phases of the expert review will be outlined, detailing the process and results. Finally, the revised K-12 online course design rubric will be discussed.

Literature Review

K-12 online learning is not a new concept. Prior to the widespread use of the World Wide Web, students and instructors would be able to connect via telephone or correspond through the postal service (Clark, 2013). As the opportunity for K-12 online learning increased, it should not be surprising that many courses were designed using the same principles that designers applied to these legacy distance models, as well as to face-to-face courses (Barbour & Adelstein, 2013a; Barbour, Morrison, & Adelstein, 2014). Instead of telephones and the postal service, chat rooms and email were utilized (Perrin & Mayhew, 2000). As websites and learning management systems (LMS) came into existence, courses began to take and copy from traditional face-to-face courses (Barbour, 2007). However, it became apparent that there were widening differences between the two environments. Effective online educators, for example, had to utilize skillsets better suited for K-12 online environments (Davis et al., 2007). As educators had to shift their way of thinking, the demand for an overhaul in course design began to form.

Research, specifically about K-12 online course design, has been limited (Barbour, 2013; Barbour & Adelstein, 2013b). There have been studies conducted that focus on specific programs, such as VHS or the Florida Virtual Schools (FLVS) (Kozma et al., 1998; Zucker, 2005). In both instances, the design of the online course is strongly considered along with other aspects. VHS requires its educators to take a mandatory graduate level course that has a focus on design within the LMS (Zucker & Kozma, 2003). The FLVS utilizes a team approach consisting of subject matter experts, project managers, instructional designers, and web developers (Johnston, 2004). The team process has proven successful for FLVS, but it is a very unique system (Barbour & Reeves, 2009).

As K-12 online learning has continued to mature and evolve, best practice standards that include aspects of course design have also been released (iNACOL, 2011; QM, 2014). Some of these standards are proprietary, such as those found in the QM system. Beginning as a 3-year Fund for the Improvement of Postsecondary Education grant in 2003 (Legon & Runyon, 2007), the first QM rubric was formed in 2004. QM gradually became an entire process for online course review (Shattuck, 2007). The current rubric utilizes eight general standards (i.e., course overview and introduction, learning objectives, assessment and measurement, instructional materials, learner interaction and engagement, course technology, learner support, and accessibility), while the program offers to train staff for peer reviews, course design, and more (MarylandOnline, 2013). However, even though they have never been tested for validity, the iNACOL (2011) standards are an easy place for K-12 online course designers to begin because the standards and rubric are publically available and non-proprietary.

Methodology

Upon completion of the construct validity phase of this research initiative (see Adelstein & Barbour, 2016), the next stage was the content validity of the revised rubric. The purpose was to test the design of the new rubric through expert review. It was recommended to involve content-area experts, as content validity is a result of their verification that the rubric meets the standards as outlined in phase one (Roblyer & Wiencke, 2003; Taggart, Phifer, Nixon, & Wood, 2001). Roblyer and Wiencke (2003) denoted that a properly designed rubric used in educational technology is a meaningful way to both assess and guide practitioners. It should not be surprising to see a leader in the field, such as QM, used a rubric for their proprietary design standards during the creation process (Hixon, Barczyk, Buckenmeyer, & Feldman, 2011).

Eight experts, who were divided into two groups, reviewed the standards over the course of three rounds, examining each standard from a course design perspective. The experts were selected based on their background and experience in K-12 online education (see Table 1).

Table 1

Description of the Two Expert Review Groups

Specifically, each group consisted of a researcher, administrator, designer, and teacher; all of whom had been directly involved with K-12 online learning.

During round one, each of the experts received a document containing the 52 iNACOL elements listed under the five main standards based on the results of the first phase of this research initiative. The document was color coded to indicate the nature of research supported for each standard (i.e., green for significant K-12 online learning research support, yellow for limited K-12 online learning research support, or orange for supported only by non-K-12 literature). There were also two additional sections added to the end of the document. The first section offered four new standards that were found to be present in the K-12 online learning research, while the second suggested combining elements that were seen as similar in scope. In round one, the experts were asked to rate the importance of each standard as it related to course design using a basic Likert scale (i.e., 1 for low relevancy, 2 for some relevancy, and 3 for significant relevancy). An area for comments was also included for each section.

After compiling the ratings from round one, a second document was created that listed the average rating for each of the standards and the comments that experts made. Based upon both the raw rating, as well as expert suggestions, the researcher made suggestions about revising or removing certain standards. Experts were asked to select one of four options (i.e., keep the standard as is, revise the standard, combine with another standard, or delete the standard) and to provide a written rationale for that decision.

The responses from round two were again compiled in a new document that consisted of three sections:

1. standards where there were general agreement that should be kept as written,
2. standards where the expert feedback from the previous two rounds that had a clear consensus for either revision or deletion, and
3. standards that did not have a clear consensus from the experts and would require further discussion.

The experts' feedback from the previous rounds was listed under each standard. Round three consisted of 60-minute discussion with each expert group using Google Hangout that focused on the second and third sections of the round three document.¹ During the Google Hangout, the researcher facilitated discussion around the standards recommended for revision or deletion until a consensus was achieved on whether to revise or delete a particular standard, as well as the specific wording for any revised standards.

Results

In this section, we organize the data using the complete iNACOL National Standards for Quality Online Courses, broken down by section (see Tables 2-7). The revised iNACOL standards based on the expert review are provided in Appendix.

Table 2

Section A: Content

Section A was highly regarded by the experts in terms of significance to course design. In the first round, there was overwhelming agreement to keep the majority of the elements in some form, with the exceptions of A5 (i.e., Multiple learning resources and materials to increase student success are available to students before the course begins) and A12 (i.e., Online instructor resources and notes are included). Both elements were further discussed in round two, with experts still divided on how to move forward. All experts shared in round three that the phrasing of A5 was problematic, questioning how realistic it was to have all materials present before the course begins. Ron mentioned that due to the logistics of certain courses, having all material available:

is technically not possible in some settings. Because you do an enrollment and that's when the students are there and they can't get access to the course until they are enrolled and they are enrolled at date of start. So it's not physically possible.

It was suggested, and agreed upon by Group A, to change the wording to "All course materials are available to students at the course start." Group B, on the other hand, did not think the element was appropriate. Amanda noted:

you really don't know what's going to work until the students start the course and actually, you know, get their feedback as to what's working and what's not working. The other thing is, is that your course materials could be specific to that student as well, like some students may better at, um, a virtual lab or something else, and another student might learn better by watching a video or doing something else.

With other elements in the rubric discussing additional materials, Group B moved to delete A5 (i.e., Multiple learning resources and materials to increase student success are available to students before the course begins).

The round two discussions of A12 (i.e., Online instructor resources and notes are included) lead to a suggestion of combining it with A13 (i.e., Assessment and assignment answers and explanations are included) or keeping it as is. Both groups were quick to lean towards combining the elements. Ron suggested a further revision to include the pedagogy behind the material, as this understanding would help teachers "to grade [the assignment] appropriately, but they would also be given the grading rubrics which they would then communicate clearly in an easy to understand manner to the students and parents." The rest of the Group A members agreed.

Round one did include two suggestions from experts that were put forth in round two. The first looked to combine A1 (i.e., The goals and objectives clearly state what the participants will know or be able to do at the end of the course; the goals and objectives are measurable in multiple ways), A6 (i.e., A clear, complete course overview and syllabus are included in the course), and A7 (i.e., Course requirements are consistent with course goals, are representative of the scope of the course and are clearly stated) due to similarities. Experts were unanimous on combining the elements, with both groups agreeing on the suggested wording put forth in round three. The other suggestion was to delete A4 (i.e., Information literacy and communication skills are incorporated and taught as an integral part of the curriculum), with an expert wondering if it was better suited at a program level and not at the course level. Group A had little discussion, as all agreed it was too broad and not a part of the course design. Group B strongly thought that the communications piece was already handled in element B9 (i.e., The course design includes explicit communication/activities (both before and during the first week of the course) that confirms whether students are engaged and are progressing through the course. The instructor will follow program guidelines to address non-responsive students.), but perhaps the information literacy should remain. For example, Kelly liked the idea that information literacy should be

embedded in the course design... I really do think that this might be a program related piece because it is overarching whole content areas, so it's not specific to a course design, but it should be interwoven into the courses specifically.

Group B agreed, and revised A4 to read, "Information literacy is incorporated as an integral part of the course."

Eventually, a decision had to be made regarding the differences between Group A and B results for element A4, A5, and A12. This was accomplished by reviewing the current K-12 literature against the expert comments from all three rounds. Element A4 was deleted, with the thought that information literacy should have a focus at the program or curriculum level, and not in the course design. Element A5 was kept in the rubric using Group A's wording. Research showed it was important for the students to have access to the materials before the course begins, allowing them time to make sure everything is compatible with personal technology. Finally, A12 and A13 were combined using Group A's suggestions as well.

Table 3

Section B: Instructional Design Elements

Much like Section A, Section B only had two elements, B9 (i.e., The course design includes explicit communication/activities (both before and during the first week of the course) that confirms whether students are engaged and are progressing through the course. The instructor will follow program guidelines to address non-responsive students.) and B11 (i.e., Students have access to resources that enrich the course content), which required further discussion in round two. The wording of B9 was a point of contention for a few experts, with the element only mentioning the importance of checking engagement before and during the first week. There was a strong overall push to revise the element to include practices throughout the course, which lead to the proposed rewording, "The course design includes explicit communication/activities at multiple intervals throughout the course that confirms whether students are engaged and are progressing through the course. The instructor will follow program guidelines to address non-responsive students." Group B was in agreement with suggestion, while Group A continued the discussion. Joanne mentioned that courses also come with tools to assess engagement, and these tools should be mentioned and used. She was also concerned with the length of the first sentence, so it was split into two sentences for final consideration.

B11 was debated at length in both expert groups. There were numerous suggestions from round two, such as revising to include examples or combining with either A5 (i.e., Multiple learning resources and materials to increase student success are available to students before the course begins) or B2 (i.e., The course is organized by units and lessons that fall into a logical sequence. Each unit and lesson includes an overview describing objectives, activities, assignments, and resources to provide multiple learning opportunities for students to master the content.). To start the conversation, the researcher offered the suggested revision, "Course design provides students with resources (e.g., alternate assignments, multimedia, simulations) that enrich course content". Group A agreed with the suggestion, with one edit recommended from Ron to include mention of Universal Design for Learning (UDL). It was suggested to include it both in this standard and at the beginning of the new rubric.

Group B had already eliminated A5, knowing that B2 and B11 covered much of the same territory. A B2 revision was previously agreed upon, but the group was quick to see similarities. Jason summed up the group's thoughts when he commented

B2 seems to be talking about the overview, and B11 is what is actually there, I guess. Or at least describing the opportunities, then. To go along with the overview.... It just seems like they need to be focused together, to make them one.

Further, Kelly suggested, "Course design provides students with resources (e.g., alternate assignments, multimedia, simulations) that enrich course content. Each unit and lesson includes an overview of the key objectives that incorporate a variety of activities, assignments, and resources to provide multiple learning opportunities for students to master the content." The experts in Group A agreed.

There were two expert suggestions from round one that impacted B2 and B10 (i.e., The course provides opportunities for appropriate instructor-student and student-student interaction to foster mastery and application of the material). The concern over B2 stemmed from the use of the word logical, which appeared to lock the element into a traditional mode of design. Group B was quick to agree upon the revision, which simply eliminated "that fall into a logical sequence" from the end of the first sentence. Group A shifted their conversation to the use of units and lessons, with Joanne offering up modules. Ron agreed, adding

When we design courses, we design them around weeks. Not units not lessons, but around weeks. And I don't know if units and lessons precludes weeks, but I'm also not sure that it encourages that. And units and modules is better. But I would go around, I think organized by modules and take out the units.

Louise was unsure of eliminating units, but came to an understanding that the delivery depends on the instructor and mechanisms used. Therefore, modules could stand alone.

B10 was questioned by an expert for the use of foster, which implied that mastery only comes from the suggestions listed in the element. A revision, "The course provides opportunities (e.g., instructor-student and student-student interaction, assessments, access to resources) for mastery and application of the material," was suggested to the experts. Group A unanimously agreed, while Kelly had a further revision for Group B. Her thought was to keep the examples listed in the element similar to one another by relating each interaction to the student. The list was changed to "student-instructor interaction, student-student interaction, student-course content, student-LMS," and experts were content to move on.

After collecting the expert suggestions, a final decision was made on how to phrase B2, B9, B10, and B11. The most complex of the revisions involved B2 and B11. The similarities brought up by Group B were logical, and the reasoning from Jason was enough to move forward with a combination. Group A's suggestion of changing units and lessons to modules was taken under consideration and added to the final wording. Group A's addition of tools and punctuation were accepted for B9, and Group B's wording was used for B10.

Table 4

Section C: Student Assessment Elements

Taken as a whole, Section C was positively viewed by the experts, with only C3 (i.e., Ongoing, varied, and frequent assessments are conducted throughout the course to inform instruction) averaging below a cumulative 2.5 score. The use of the word "frequent" was an issue for the majority of experts, and it was suggested to replace it with "quality." Both groups were quick to agree with the new wording.

Three expert suggestions were taken from round one and shared with the group as a whole. The first was C2 (i.e., The course structure includes adequate and appropriate methods and procedures to assess students' mastery of content), with the wording "adequate and appropriate" seeming too vague, leaving some experts to wonder who determines this. The initial comments from round two were fairly split between keeping the original wording and revising the element. Group A promptly decided that the original, while a bit vague, gave enough direction for design. Group B, on the other hand, moved to eliminate and not replace "adequate and appropriate."

C6 (i.e., Grading rubrics are provided to the instructor and may be shared with students), according to one expert, suggested that the word "may" implies the rubric does not need to be shared with students. Another expert was concerned that a rubric will be forced upon a qualified teacher. A rubric must be supplied in the course, but a qualified instructor should have final say over which rubric to use. While there was unanimous agreement amongst the experts that the instructor will share the rubric with students, there was some discussion as to the phrasing of the final revision. The suggested wording supplied used, "Suggested grading rubrics are provided to the instructor. The instructor will share a final grading rubric with students." Group B accepted the revision, but Group A was concerned over misinterpretations about the word "final," as some might view it in the context of a final exam. Ultimately, "final" was replaced by "chosen" in the element.

Experts were also concerned over language in C7 (i.e., The grading policy and practices are easy to understand), and looked to replace "easy to understand" with "clearly communicated." Group B unanimously agreed, while Ron from Group A suggested both phrases should be used. Louise and Ron offered continued revisions by adding "to students and parents" at the end, as they are the stakeholders who will interpret the policies.

C2, C6, and C7, had minor revision details that had to be accounted for. C2 was kept as is, as the wording, even though vague in nature, gives some direction to the designer. Group A's version of C6 was kept to avoid misinterpretation, and C7 was also finalized by group A. Much like C2, the wording gives appropriate direction to the designer.

Table 5

Section D: Technology

Section D proved to be one of the most contentious for the experts throughout the process. D1 (i.e., The course architecture permits the online instructor to add content, activities and assessments to extend learning opportunities), D2 (i.e., The course accommodates multiple school calendars, e.g., block, 4x4 and traditional schedules), D6 (i.e., Prerequisite skills in the use of technology are identified), D7 (i.e., The course uses content-specific tools and software appropriately), D8 (i.e., The course is designed to meet internationally recognized interoperability standards), and D9 (i.e., Copyright and licensing status, including permission to share where applicable, is clearly stated and easily found) were all flagged for further discussion coming out of round one. There was concern that D1 was not appropriate for all instructors, so adding "where applicable" at the end of the element was suggested by an expert. Both groups unanimously agreed with the revision. D9 was quickly agreed upon as well, with both groups acknowledging the importance of copyright laws.

D2 was a lengthier discussion for both groups. All experts agreed that giving calendar examples limited what an online course could fit into. Group B suggested that the element was not needed at, and voted to delete the element. Group A, on the other hand, simplified the wording and related it to the module design previously mentioned in Section B.

In round two, the experts were split on how to handle both D6 and D7. For both groups, the conversation began with a possible combination the elements, using the suggestion "Prerequisite skills, course tools, and course software are identified and appropriate in relation to the students and course." Louise mentioned to group A that the notion of prerequisite skills should be part of communication, but was not a function of course design. The other experts agreed, and removed "prerequisite skills" from the revision. Group B was fairly adamant that D6 had to remain in some form or another. As Amanda put it

I'm looking at this from trying to explain to a parent, you know, why their student shouldn't take this specific course because maybe they don't meet those prereqs.... prerequisite skills in the use of technology are identified. That is, that is something that they need to know how to do. How to navigate, you know, different parts of the course. And it might be course specific, meaning different courses will have different prereqs, but I don't think you can delete this.

After a bit more discussion, Group B approved the combination of D6 and D7 as suggested.

From the round one and two comments, it appeared that some experts were not familiar with what D8 (i.e., The course is designed to meet internationally recognized interoperability standards) was referring to. Even after further explanation, Group A was quick to delete the element, not viewing it as a necessary part of design. Group B took a different stance, viewing D8 as something that will be important in the future of design. Jason brought up that as instructors and students move from one proprietary software to another, it is important they have the ability to keep communicating and creating. The other experts agreed, and opted to keep the element.

As with the previous elements, a final version of the suggestions had to be obtained for D2, D6, D7, and D8. D2 (i.e., The course accommodates multiple school calendars, e.g., block, 4x4, and traditional schedules) appeared to have middling support from both research and the experts, so the decision was made to eliminate the element. The additional thought was that the modules in the course could be manipulated to fit any calendar, so there was not an overwhelming need to mention this as a design requirement. It would instead fall to the instructor and institution to make the course work for them. There was a strong argument for keeping D6 (i.e., Prerequisite skills in the use of technology are identified), and the suggested combination of D6 and D7 (i.e., The course uses content-specific tools and software appropriately) was used. Finally, Group B's suggestion that D8 (i.e., The course is designed to meet internationally recognized interoperability standards) would be relevant in the future of design was enough to keep the element intact.

Table 6

Section E: Course Evaluation and Support Elements

Much like the previous section, E brought about much discussion as to how the elements pertained to course design, or if they did at all. E4 (i.e., Course instructors, whether face-to-face or virtual, are certificated and "highly qualified." The online course teacher possesses a teaching credential from a state-licensing agency and is "highly-qualified" as defined under ESEA), E6 (i.e., The course provider offers technical support and course management assistance to students, the course instructor, and the school coordinator), E7 (i.e., Course instructors, whether face-to-face or virtual, have been provided professional development in the behavior, social, and when necessary, emotional aspects of the learning environment), E8 (i.e., Course instructors, whether face-to-face or virtual, receive instructor professional development, which includes the support and use of a variety of communication modes to stimulate student engagement online), and E10 (i.e., Students are offered an orientation to taking an online course before starting the coursework) were all forced into the discussion for round two. E2 (i.e., The course is evaluated using a continuous improvement cycle for effectiveness and the findings used as a basis for improvement) and E3 (i.e., The course is updated periodically to ensure that the content is current) were suggested to be combined by an expert in round one, and E4 through E10 were all put up for deletion in various round one suggestions. The conversation in round three began with combining E2 and E3. Group A believed the standards did not fit into design and should therefore be eliminated. Group B saw it differently, believing that the findings from the evaluation should be used to improve and update the course. However, there was concern over the use of periodically and what that actually meant. Kim suggested adding "as needed" to the end of the combined revision, and the rest of Group B agreed.

During round one, it was suggested that elements E4 through E10 could be completely eliminated, as they do not relate to course design. Group A quickly moved to eliminate all remaining elements, with the exception of E6 (i.e., The course provider offers technical support and course management assistance to students, the course instructor, and the school coordinator). Louise was adamant that support should be built directly into the course, and not remain at the institution level

It's the program that is delivering, designing, and then delivering this online course that makes the determination how the technical assistance is going to be provided. But the given is that within that course design, is the tool for technical assistance. But it's a programmatic decision. We don't care...who provides the assistance. As long as it can be found.

The rest of Group A agreed, and a revised version of E6 remained. Group B, however, came to the conclusion that E6, as well as the rest of the elements in the suggestion, could be eliminated. Kelly summed up the collective thought

I think too there's a lot of them that are...higher level program. The course provider in terms of technical support, they're going to provide that. I see a lot of program level, like orientation for students, I think that's program related piece, too. That should be for all students taking any online course within the program or whatever it might be.

Group B concurred, and elements E4 through E10 were deleted.

Reviewing data and all reviewer comments, the suggested combining of E2 (i.e., The course is evaluated using a continuous improvement cycle for effectiveness and the findings used as a basis for improvement) and E3 (i.e., The course is updated periodically to ensure that the content is current) was accepted into the final rubric. Course design can be continuous and ongoing, meaning there should be an evaluation and improvement process in place. The revised version of E6 (i.e., The course provider offers technical support and course management assistance to students, the course instructor, and the school coordinator) was also added. Group A made a strong case for the need of technical support to be located in each course. While it does not matter who eventually supplies the support, there should be access to help for every instructor and student directly within the course.

Table 7

Section F & G: Suggested Elements and Revisions

In round one, the experts were presented with four additional elements and two revisions. Elements F1 (i.e., The syllabus promotes a student plan of work with attainable expectations), F2 (i.e., Technology is used to help increase self-efficacy of students), and F3 (i.e., Activities are designed to encourage students' individual interests and goals) were all readily accepted. In round two, there was a strong consensus to delete F4 (i.e., The instructor understands student goals and personalizes support). When mentioned in round three, there was no call for discussion from either group, and F4 was eliminated. F1 and F3 were placed in Section B under the Instructional Strategies and Activities subsection. F2 was located in Section D under the User Interface subsection.

The revisions were split with the experts. There was strong support in round one to combine B4 (i.e., The course and course instructor provide students with multiple learning paths, based on student needs that engage students in a variety of ways) and B6 (i.e., The course provides options for the instructor to adapt learning activities to accommodate students' needs). The suggested revised wording was not brought up by experts for further discussion and was added to the final rubric. However, most experts did not believe E6 and E7 were closely related, and the combined suggestion was dropped. In the end, both elements were ultimately recommended for deletion by both expert groups.

Discussion

The first section of the iNACOL National Standards for Quality Online Courses (i.e., Section A: Content) received a relatively high level of support from the experts during all three rounds of review. This result was not surprising, given the fact that these standards were primarily centered on structural and preparatory aspects of the online course. For example, in his study of course developers at a province-wide supplemental virtual school, Barbour (2005, 2007) reported several principles of effective course design that were focused on items like the consistency of navigation and structured course content. Similarly, students have also stressed the importance of structural and preparatory material in an online course. Gallini and Barron (2001-2002) reported that students preferred "a course structure with clear guidelines along with opportunities in the course to suggest alternative approaches to meeting course objectives" (p. 149), all aspects of structural and preparatory material found in an online course. Even most of the QM general standard areas (i.e., course overview and introduction, learning objectives/competencies, assessment and measurement, instructional materials, learner interaction and engagement, course technology, learner support, and accessibility) were focused on what online course designers would describe as structural and preparatory items (MarylandOnline, 2013).

Considering the significant tie between instructional and course design, expert support for the majority of the Section B (i.e., Instructional Design) elements was not unexpected. There was agreement that opportunity for higher order thinking, differentiating, and active learning be taken into consideration when designing the course. This was also supported by Mastropieri et al. (2006), who discussed how differentiating helped middle school science students achieve higher score on both in-class unit and state exams. The largest obstacle in Section B was actually related to the wording of certain elements. Experts agreed that resource materials could help with mastery, as have been seen in the K-12 online learning literature with algebra students who used virtual manipulatives (Cavanaugh, 2013). The wording and redundant nature of certain elements led to combining parts of Section B.

As a whole, the Section C (i.e., Student Assessment) elements were agreed upon and accepted by the experts in the revised rubric. This level of agreement was consistent with DiPietro (2010), who interviewed 16 online educators and found that participants agreed that assessment and feedback helped students engage with the content, along with meeting their individualized needs. In fact, as students become engaged with the learning, they are generally open to hearing feedback on how to improve and reach mastery of the subject material (Naidu, 2013). This feedback can be aided by the use of various resources, including rubrics, and by viewing course rubrics students become aware of expectations (Rice, 2012). As with Section B, the experts were mainly concerned with the wording of various elements, and moved forward with the section largely intact.

Unlike the previous three areas, Section D (i.e., Technology) garnered more discussion with regards to both wording and how the elements pertained to course design. The experts agreed that flexibility was important to scheduling online courses, a notion that Wicks (2010) also supported. However, the experts thought an element specifically about different calendar types was unnecessary, and that element was subsequently deleted. Further, there was also open debate over interoperability of the course, with some experts not seeing the necessity of integration. However, Watson and Watson (2007) noted that LMSs needed to "truly become systemic, integrating systems seamlessly to allow for improved collaboration across systems among stakeholders" (p. 32). While many of the remaining elements were eventually reworded or combined, the experts were generally agreeable with the general sentiment found in Section D (i.e., that understanding that the technology used played an important part in course design). This is consistent with earlier studies into the design of K-12 online courses. For example, Barbour (2007) interviewed six online educators who found that minimal and simple navigation gave a consistency that was appreciated by the students. The same group of educators, however, didn't feel that a course should shy away from multimedia and interactive elements, which could be used to enhance the curriculum.

Section E (i.e., Course Evaluation and Support Elements) was by far the most retooled area from the experts, but this is not to say that the elements were not important when creating an online course. As a few of the experts pointed out, all the elements were significant, but simply belong to different rubrics, as opposed to one focused on online course design. For example, the VHS required a 26-week class in how to design a course that utilized the LMS (Zucker & Kozma, 2003). Further, the Illinois Virtual High School (IVHS) used a similar practice shell as well, as was pointed out by Barbour, Kinsella, Wicks, & Toker (2009). IVHS also realized that continuous support was needed, and offered monthly professional development using face-to-face, synchronous, or asynchronous methods. However, that did not mean that the need for professional development and support should be included in a rubric designed to measure quality online course design. Yet, not all elements from Section E were deleted. For example, continuous course updates were fully supported by experts, and was a practice utilized by many K-12 online learning programs (Ebert & Powell, 2015).

The final areas were Section F and G (i.e., Suggested Elements and Revisions), which focused on the elements that were suggested for addition or elements that should be revised. The suggested elements looked to include student motivation in the course design structure, which was not a part of the original iNACOL standards. Three of the four suggestions were strongly supported by the experts, and found their way into the revised rubric without revisions. Both Chen and Jang (2010) and Kim, Park, and Cozart (2014) reported that motivation was an essential part of education, particularly in the online learning environment. As such, it was important that components that fostered student satisfaction in autonomy and self-efficacy were planned for within the online course design.

Conclusions and Implications

After examining the iNACOL National Standards for Quality Online Courses (2011) based on current literature and research in phase one of the study (see Adelstein & Barbour, 2016), a revised set of standards were reviewed by eight experts for phase two. The review occurred over three rounds, with the first two happening via e-mail and the third through a video conference. During all three rounds the experts recommended that certain elements be kept, combined, or deleted using their knowledge and understanding of online course design as a guide. The end result was a K-12 online course design rubric based off the original iNACOL quality standards that was further revised and refined.

The iNACOL standards, while praised by the experts, are purposefully broad, covering all aspects of online courses. The results of phase two of this study helped to bring essential online course design standards into focus. This specialized and more focused view may be able to help curtail how overwhelming the standards can appear, especially for those new to the field of K-12 online learning and designing online courses for a K-12 population. The revised rubric will allow stakeholders, including educators, course designers, and administrators, to focus specifically on the aspects of online course design, creating a stronger base upon which to build asynchronous online course content.

Having said that, the researchers would recommend that further expert review be conducted. Due to time and resource constraints, the experts for this study were limited to eight individuals. Also, while the first two rounds were vital to giving the experts some guidance, the majority of the discussion related to and refinement of the individual elements occurred during the video conference; which was limited to approximately one hour. It would be beneficial to provide the experts multiple opportunities to video conference over the course of the refinement of the standards. Finally, the iNACOL standards were chosen due to their open, non-proprietary nature. However, there are also other widely used standards that could be used or supplemented as the basis for this model of expert discussion. As for our own line of inquiry, with the expert review completed, the next phase of this particular study will test the application of the rubric. Three to five teams of two reviewers will apply the rubric against current K-12 online courses. Using inter-rater reliability, the researchers will examine the reliability and validity of the rubric.

Notes

1. Due to a last minute emergency situation, one expert from Group A (Connor) was unable to attend the Google Hangout.

References

Adelstein, D., & Barbour, M. K. (2016). Building better courses: Examining the construct validity of the iNACOL national standards for quality online courses. Journal of Online Learning Research, 2(1), 41-73.

Barbour, M. K. (2005). The design of web-based courses for secondary students. Journal of Distance Learning, 9(1). 27-36.

Barbour, M. K. (2007). Principles of effective web-based content for secondary school students: Teacher and developer perceptions. Journal of Distance Education, 21(3), 93-114. Retrieved from http://www.ijede.ca/index.php/jde/article/view/30

Barbour, M. K. (2013). The landscape of K-12 online learning: Examining what is known. In M. G. Moore (Eds.), Handbook of distance education (3rd ed.) (pp. 574-593). New York: Routledge.

Barbour, M. K., & Adelstein, D. (2013a). High-school students' perceptions of effective online course design. The Morning Watch, 41(1-2), 56-65. Retrieved from http://www.mun.ca/educ/faculty/mwatch/vol41/fall2013/michaelBarbour.pdf

Barbour, M. K., & Adelstein, D. (2013b). Voracious appetite of online teaching: Examining labour issues related to K-12 online learning. Vancouver, BC: British Columbia Teachers Federation. Retrieved from http://www.bctf.ca/uploadedFiles/Public/Issues/Technology/VoraciousAppetite.pdf

Barbour, M. K., Kinsella, J., Wicks, M., & Toker, S. (2009). Continuum of change in a virtual world: Training and retaining instructors. Journal of Technology and Teacher Education, 17(4), 437-457.

Barbour, M. K., Morrison, J., & Adelstein, D. (2014). The forgotten teachers in K-12 online learning: Examining the perceptions of teachers who develop K-12 online courses. International Journal of Online Pedagogy and Course Design, 4(3), 18-33.

Barbour, M. K., & Reeves, T. C. (2009). The reality of virtual schools: A review of the literature. Computers & Education, 52(2), 402-416.

Cavanaugh, C. (2013). Student achievement in elementary and high school. In M. G. Moore (Ed.), Handbook of distance education (3^rd Ed.) (pp. 157-168). New York: Routledge.

Chen, K. C., & Jang, S. J. (2010). Motivation in online learning: Testing a model of self-determination theory. Computers in Human Behavior, 26(4), 741-752.

Clark, T. (2013). The evolution of distance and online education in American schools. In M. G. Moore (Ed.), Handbook of distance education (3^rd Ed.) (pp. 555-573). New York: Routledge.

Davis, N., Roblyer, M. P., Charania, A., Ferdig, R., Harms, C., Compton, L. K. L., & Cho, M. O. (2007). Illustrating the "virtual" in virtual schooling: Challenges and strategies for creating real tools to prepare virtual teachers. The Internet and Higher Education, 10(1), 27-39.

DiPietro, M. (2010). Virtual school pedagogy: The instructional practices of K-12 virtual school teachers. Journal of Educational Computing Research, 42(3), 327-354.

Drost, E. A. (2011). Validity and reliability in social science research. Education Research and Perspectives, 38(1), 105.

Ebert, J., & Powell, A. (2015). A case study of Clark County School District's Virtual High School. In T. Clark, & M. K. Barbour (Eds.). Online and distance education in schools: Global perspectives on policy and practice (pp. 131-143). Sterling, VA: Stylus Publishing.

Gallini, J., & Barron, D. (2001-2002). Participants' perceptions of web-infused environments: A survey of teaching belies, learning approaches, and communications. Journal of Research on Technology in Education, 34(2), 139-156.

Hixon, E., Barczyk, C., Buckenmeyer, J., & Feldman, L. (2011). Mentoring university faculty to become high quality online educators: A program evaluation. Online Journal of Distance Learning Administration, 14(5). Retrieved from http://www.westga.edu/~distance/ojdla/winter144/hixon_Barczyk_Buckenmeyer_feldman144.html

International Association for K-12 Online Learning. (2011, October). National standards for quality online courses version 2. Vienna, VA: Author. Retrieved from http://www.inacol.org/wp-content/uploads/2015/02/national-standards-for-quality-online-courses-v2.pdf

Johnston, S. (2004). Teaching any time, any place, any pace. In C. Cavanaugh (Ed.), Development and management of virtual schools: Issues and trends (pp. 116-134). Hershey, PA: Idea Group, Inc.

Kim, C., Park, S. W., & Cozart, J. (2014). Affective and motivational factors of learning in online mathematics courses. British Journal of Educational Technology, 45(1), 171-185.

Kozma, R., Zucker, A., & Espinoza, C. (1998). An evaluation of the Virtual High School after one year of operation. Arlington, VA: SRI International. Retrieved from http://thevhscollaborative.org/sites/default/files/public/Evaluation%20after%20yr%201.pdf

Legon, R., & Runyon, J. (2007). Research on the impact of the quality matters course review process. In 23rd Annual Conference on Distance Teaching & Learning (pp. 8-10). Madison, WI: Division of Continuing Studies, University of Wisconsin-Madison. Retrieved from http://www.uwex.edu/disted/conference/resource_library/proceedings/07_5284.pdf

Making Online Learning Accessible for Students with Disabilities: Lessons from Texas [Blog Post]. (2015, September 4). Retrieved from http://www.inacol.org/news/making-online-learning-accessible-for-students-with-disabilities-lessons-from-texas/

MarylandOnline. (2013). Quality matters overview. Annapolis, MD: Author. Retrieved from https://www.qualitymatters.org/sites/default/files/pd-docs-PDFs/QM-Overview-Presentation-2014.pdf

Mastropieri, M. A., Scruggs, T. E., Norland, J. J., Berkeley, S., McDuffie, K., Tornquist, E. H., & Connors, N. (2006). Differentiated curriculum enhancement in inclusive middle school science: Effects on classroom and high-stakes tests. Journal of Special Education, 40(3), 130-137.

Naidu, S. (2013). Instructional design models for optimal learning. In M. G. Moore (Ed.), Handbook of distance education (pp. 268-281). New York: Routledge.

National Association for K-12 Online Learning. (2007). National standards for quality online courses. Vienna, VA: Author.

Oakland Schools. (2015). FAQ for section 21f of the State School Aid Act . Waterford Twp, MI: Author. Retrieved from http://media.mivu.org/institute/pdf/21F_FAQs.pdf

Perrin, K. M., & Mayhew, D. (2000). The reality of designing and implementing an Internet-based course. Online Journal of Distance Learning Administration, 3(4). Retrieved from http://www.westga.edu/~distance/ojdla/winter34/mayhew34.html

Quality Matters. (2014). Quality matters (QM ). Retrieved from https://www.qualitymatters.org

Rice, K. (2012). Making the move to K-12 online teaching. Boston, MA: Pearson Education, Inc.

Roblyer, M. D., & Wiencke, W. R. (2003). Design and use of a rubric to assess and encourage interactive qualities in distance courses. The American Journal of Distance Education, 17(2), 77-98.

Shattuck, K. (2007). Quality matters: Collaborative program planning at a state level. Online Journal of Distance Learning Administration, 10(3). Retrieved from http://www.westga.edu/~distance/ojdla/fall103/shattuck103.htm

Taggart, G. L., Phifer, S. J., Nixon, J. A., & Wood, M. (2001). Rubrics: A handbook for construction and use. Lanham, MD: Scarecrow Press.

Watson, W. R., & Watson, S. L. (2007). An argument for clarity: What are learning management systems, what are they not, and what should they become? TechTrends, 51(2), 28-34.

Wicks, M. (2010). A national primer on K-12 online learning, version 2. Vienna, VA: International Association for K-12 Online Learning. Retrieved from http://www.inacol.org/research/bookstore/detail.php?id=22

Zucker, A. (2005). A study of student interaction and collaboration in the Virtual High School. Naperville, IL: Learning Point Associates

Zucker, A., & Kozma, R. (2003). The virtual high school: Teaching generation V. New York: Teachers College Press.

Appendix

Revised Rubric

Improving the K-12 Online Course Design Review Process: Experts Weigh in on iNACOL National Standards for Quality Online Courses by David Adelstein and Michael K. Barbour is licensed under a Creative Commons Attribution 4.0 International License.