View in EDS HTML Full Text PDF Full Text

Fostering Excellence in Blended Learning, a Mixed Methods Investigation of an Academic Learning Design Support Tool

Saved in:
Bibliographic Details
Title: Fostering Excellence in Blended Learning, a Mixed Methods Investigation of an Academic Learning Design Support Tool
Language: English
Authors: Helen Benson (ORCID 0000-0002-6918-5045), Keith Heggart (ORCID 0000-0003-2331-1234), Kylie A. Williams (ORCID 0000-0002-5094-243X), Chloé Smit (ORCID 0000-0002-3037-5837), Patrick Abela (ORCID 0009-0007-8663-3317)
Source: TechTrends: Linking Research and Practice to Improve Learning. 2025 69(1):71-83.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 13
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Blended Learning, Learning Management Systems, Higher Education, Usability, Computer Software Evaluation, Electronic Learning
DOI: 10.1007/s11528-024-01019-z
ISSN: 8756-3894
1559-7075
Abstract: This article investigates the effectiveness of a newly developed online tool for assessing and improving the design and accessibility of Learning Management System (LMS) sites in Higher Education. The study employs a mixed-methods approach, incorporating both quantitative and qualitative data. Results from 25 subject coordinators and 3 peer reviewers reveal the strengths and weaknesses in LMS site design at a particular institution. It identifies recommendations for enhancements such as clearer signposting of educational materials, improved organisation of reading lists, and increased opportunities for collaboration. In addition, the research identifies key areas for improvement in institutional LMS site design, including consistency, organisation, and ease of navigation. These findings underscore the importance of engaging prior knowledge, fostering active learning, and ensuring clear communication of student expectations. The tool is shown to be a valuable resource for educators to assess and enhance their online teaching environments. The study's findings contribute to the ongoing discourse on optimising blended learning experiences, emphasising the need for ongoing improvement in LMS site design and accessibility.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1460904
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHNM2lzkACqOc81GpI3LmNsAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDG3IqMKTgT_GSBGefAIBEICBmzzBCsJ_toKhUwG9dzw7gnBP8MVNkYYYrgSbqh0ZanpLW-0J7Xonzw9eKK_xfQEQtdFJzF9ZcGshHeOVpBZZy5VVB6uNYx__9hJnzc-QGI_E3yRSDhazD1hME__QISvx_xEpwe2RpwPWYwUizqcJEODvUfNLFnoGww4hohGN0heZtfvDEZUVIqV8cWUYCDA2-RD-LveFl6P6k7qu
Text:
  Availability: 1
  Value: <anid>AN0182468365;ttr01jan.25;2025Jan29.03:45;v2.2.500</anid> <title id="AN0182468365-1">Fostering Excellence in Blended Learning, a Mixed Methods Investigation of an Academic Learning Design Support Tool </title> <p>This article investigates the effectiveness of a newly developed online tool for assessing and improving the design and accessibility of Learning Management System (LMS) sites in Higher Education. The study employs a mixed-methods approach, incorporating both quantitative and qualitative data. Results from 25 subject coordinators and 3 peer reviewers reveal the strengths and weaknesses in LMS site design at a particular institution. It identifies recommendations for enhancements such as clearer signposting of educational materials, improved organisation of reading lists, and increased opportunities for collaboration. In addition, the research identifies key areas for improvement in institutional LMS site design, including consistency, organisation, and ease of navigation. These findings underscore the importance of engaging prior knowledge, fostering active learning, and ensuring clear communication of student expectations. The tool is shown to be a valuable resource for educators to assess and enhance their online teaching environments. The study's findings contribute to the ongoing discourse on optimising blended learning experiences, emphasising the need for ongoing improvement in LMS site design and accessibility.</p> <p>Keywords: Learning design; Higher education; Quality assurance; Blended learning; Education Specialist Studies In Education</p> <p>Copyright comment Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</p> <hd id="AN0182468365-2">Introduction</hd> <p>Blended learning, which can be defined as the combination of face to face and online education, is becoming a mainstream approach in Higher Education (Halverson et al., [<reflink idref="bib17" id="ref1">17</reflink>]). To support this change, Learning Management Systems (LMS) are becoming increasingly sophisticated, moving away from administrative repositories for course information to next generation digital learning environments where students can collaborate, engage and learn actively online (Koh & Kan, [<reflink idref="bib21" id="ref2">21</reflink>]). Fathema and Akanda identified that to ensure large scale LMS adoption it is important for universities to ensure they have procedures in place for the design, development, implementation and evaluation of LMS sites (Fathema & Akanda, [<reflink idref="bib15" id="ref3">15</reflink>]). This has led to a surge in the number of learning designers employed in higher education (Heggart & Dickson-Deane, [<reflink idref="bib18" id="ref4">18</reflink>]) and a focus on the kinds of skills and knowledges that learning designers should be able to employ. There have been numerous attempts to formulate this into a set of standards: the best known of these are those produced by The International Board of Standards for Training, Performance and Instruction (IBSTPI) which describe the standards of practice for those working in adult education (for more information, see https://ibstpi.org/about/). While a focus on the standards of practitioners is important, it is also necessary to consider the product: that is, the LMSs developed by learning designers and other parties. It is with these sites that students will engage, and thus it is necessary to have a mechanism to assess their utility for students. Universities need to adopt an evidence-based approach for the benchmarking and assessment of Technology-Enhanced Learning (TEL) including the evaluation of LMS (Cappelli & Smithies, [<reflink idref="bib7" id="ref5">7</reflink>]). While there is some general agreement on what constitutes the accepted standards for a good technology-enhanced learning experience (and these are briefly introduced below), there is relatively little extant research into how these might be, firstly, operationalized in a specific setting, such as an LMS or, secondly, how they might be assessed and by whom. It is this area that this current research project seeks to address.</p> <p>One well known tool for evaluating the online learning environment is the ASCILITE Technology Enhanced Learning Accreditation Standards (TELAS). TELAS identifies eight important areas to assess (ASCILITE, [<reflink idref="bib3" id="ref6">3</reflink>]). These include inclusivity and accessibility; ensuring a functional, logically sequenced site layout; providing learner support in technical and administrative aspects of learning; fostering teacher presence by including learner and teacher interactions that support learning; providing clear links between expected outcomes, learning activities and assessments; allowing students to engage online and with each other and facilitating formative and summative feedback; using digital technologies and facilitating the development of digital literacy; ensuring learning resources are available, functional, comply with copyright and are appropriately attributed and providing context for learning resources and utilising a range of teaching modalities. These principles are a useful starting point for discussing the value of an LMS, but they are more general in nature; they apply to all facets of technology enhanced learning, rather than a single application, such as an LMS site. Nevertheless, these were used as a starting point for the design and development of the online tool and hence are discussed below to provide context for the discussion of the tool itself.</p> <hd id="AN0182468365-3">The ASCILITE Technology Enhanced Learning Accreditation Standards (TELAS)</hd> <p>The first standard relates to inclusivity and accessibility. Accessible design is a key factor in ensuring that all learners can participate in online learning experiences. The development of accessible LMS sites benefits all learners, not just those with accessibility needs (Coy, [<reflink idref="bib12" id="ref7">12</reflink>]; Pearson & Koppi, [<reflink idref="bib24" id="ref8">24</reflink>]). It is worth noting that LMS should also adhere to the Web Content Accessibility Guidelines 2.0 (WCAG 2.0), a globally recognised set of recommendations for "making Web content more accessible to a wider range of people with disabilities, including blindness and low vision, deafness and hearing loss, learning disabilities, cognitive limitations, limited movement, speech disabilities, photosensitivity and combinations of these." However, LMS sites, being of an organizational nature, can go beyond simple accessibility to foster inclusivity and belonging (Haddow & Brodie, [<reflink idref="bib16" id="ref9">16</reflink>]). Well-designed LMS sites can engender belonging by creating online communities and ensuring that students feel known both as individuals and as part of the student cohort. Using supported activities such as discussion board icebreakers, collaborative online tasks and group chats may be one way of increasing social engagement and a sense of belonging in a subject (Ahn & Davis, [<reflink idref="bib1" id="ref10">1</reflink>]).</p> <p>The second standard relates to the effect that visual design of a LMS site may have on how learners understand information and construct knowledge, and how they judge the value and usability of an online experience (Reyna, [<reflink idref="bib27" id="ref11">27</reflink>]). Elements of visual design that are relevant here include symmetry, colour, typography, layout consistency and aesthetics. A related concept is modularisation, also known as sequencing and chunking. By ensuring content is clearly divided in logical and coherent arrangements the cognitive load for students may be reduced (Houston, [<reflink idref="bib19" id="ref12">19</reflink>]), thus improving the overall learning experience.</p> <p>The third standard is focused on providing learner support. Ensuring academics are clearly communicating expectations for learners in the LMS site ensures that all students understand what they are required to do and what they should expect from the teaching staff in relation to delivery of content, assessment, communication and feedback provision. Clearly communicating expectations for learners in the LMS site helps them prepare for study at the beginning and throughout the subject (Cho, [<reflink idref="bib11" id="ref13">11</reflink>]). Adult learners are likely to be studying alongside work and other commitments. Understanding key information such as the subject structure and volume of learning per week allows learners to schedule study time in advance (Carter Jr et al., [<reflink idref="bib8" id="ref14">8</reflink>]).</p> <p>The fourth standard is teacher presence. Teacher presence refers to the support given to students online from the teaching staff. This could include facilitating online introductions, conducting welcome activities and/or videos when a course begins, actively facilitating discussion boards as well as providing prompt feedback on students' contributions and assessment tasks (Stone & Springer, [<reflink idref="bib30" id="ref15">30</reflink>]). When teaching staff implement these strategies, students are assured that their lecturer is 'present' and supportive of their learning.</p> <p>Evaluating engagement of student learning (the fifth standard) can assist academics in ensuring they are facilitating knowledge construction. Learning is constructed using previous knowledge to build new links and create understanding. Using the LMS site to understand what students already know can help to engage their prior understanding and support their construction of new concepts (Emami, [<reflink idref="bib13" id="ref16">13</reflink>]).</p> <p>The sixth standard examines the use of digital technologies to foster active learning. Active learning is an instructional approach that engages learners in the learning process (Smith Budhai & Skipwith, [<reflink idref="bib29" id="ref17">29</reflink>]). There is a broad range of active learning techniques and digital technologies that can be integrated in LMS sites including embedded discussion boards, social polls and comments boxes amongst others.</p> <p>Another standard that relates to LMS site design is facilitating access to subject learning content such as readings. Reading lists incorporated in the LMS site allow students to access their required, recommended, and further readings via links to the University library. This functionality increases copyright compliance and improves access for students (Tooth, [<reflink idref="bib31" id="ref18">31</reflink>]).</p> <p>The final standard recommends the use of a range of modalities for learning resources and placing them in a context. This is supported by pedagogies such as Universal Design for Learning (UDL) which recommends educational designers provide multiple means of representation, action and expression and engagement (CAST, [<reflink idref="bib9" id="ref19">9</reflink>]). Checkpoint 7.2 of these guidelines highlights the importance of ensuring learners are engaged by information and activities that are valuable and relevant to their interests and goals (CAST, [<reflink idref="bib9" id="ref20">9</reflink>]). Explaining the context and relevance of a learning resource and whether it is essential or recommended can facilitate learner engagement with subject content (Seymour, [<reflink idref="bib28" id="ref21">28</reflink>]). Signposting is one method of structuring content learning designers can use to provide context and allow learners to comprehend the relevance of information (Fabian et al., [<reflink idref="bib14" id="ref22">14</reflink>]). Utilising a range of interactive learning modalities also aligns with UDL principles by allowing learners multiple means of engagement, action and expression (CAST, [<reflink idref="bib9" id="ref23">9</reflink>]).</p> <hd id="AN0182468365-4">The Need to Develop a Tool to Undertake Quality Assurance in Online Learning Environments</hd> <p>As stated earlier, there has been relatively little research into ways to ensure compliance with quality standards in LMS sites. What there has been is of a more general nature, like the TELAS standards themselves, and focused on blended or online learning, rather than specifically on LMS compliance. Nevertheless, this research does offer some insights into how one might focus on improving LMS sites. A recent systematic review of quality assurance interventions in blended learning design identified learning design toolkits as a key intervention (Benson et al., [<reflink idref="bib4" id="ref24">4</reflink>]). However, and this is common with other literature as well, these toolkits were predominantly aimed at supporting initial course development rather than assessment and improvement of existing subjects (Benson et al., [<reflink idref="bib4" id="ref25">4</reflink>]). There were few examples of tools that could be used to assess the quality of existing sites. While one might make the case that such sites should conform to the same standards as the initial sites, as espoused by the toolkits, such an approach ignores the often time-constrained nature of academic work, or the temporality of courses and sites, which are constantly changing and being updated. This suggests that there is a need for a tool that can assess existing LMS sites efficiently; ideally, any such tool should be a light-touch mechanism that can be used by academics regularly to check their compliance.</p> <p>This was the challenge addressed by the project team of academics and learning designers that led to this paper. The team worked together to co-design and implement an online tool to assess the design and accessibility of LMS sites within a specific faculty, although they were of the view that the final tool should be applicable across the university. The results from the tool would then be used to provide targeted feedback and support to academics in relation to their LMS site design, and thus improve the overall student experience. The tool was constructed to assess multiple aspects of the LMS site as informed by both the university's teaching and learning framework and the TELAS benchmarks as described above. The results from the tool data would also allow for the identification of gaps in learning design and accessibility knowledge amongst academics. The results from a mixed methods study evaluating the usefulness of the tool to determine LMS compliance, and academics' perceptions of it, forms the basis of the rest of this paper.</p> <hd id="AN0182468365-5">Research Questions</hd> <p>The research questions were:</p> <p></p> <ulist> <item> RQ1: What are the current strengths and areas of improvement needed in relation to LMS site design within the Faculty of Health?</item> <p></p> <item> RQ2: What were the perceptions of academics and peer reviewers in relation to the usefulness of the development tool?</item> </ulist> <hd id="AN0182468365-6">Methodology</hd> <p>This study was focused on the perceptions of the usefulness of the tool by academics and peer reviewers, as well as the strengths and weaknesses of LMS site design within the faculty as identified by the tool. This is a change in approach to most evaluations of tools. In most cases, when evaluating a tool, a survey of the users and some analytics about use of the tool would be sufficient to make an assessment (such as that used by Chen et al., [<reflink idref="bib10" id="ref26">10</reflink>] or Poldoja et al., [<reflink idref="bib25" id="ref27">25</reflink>]). However, in this case, the researchers were also interested in the perceptions of this new tool: a subject which was, considering the relative newness of the tool, more exploratory than confirmatory. For this reason, a concurrent convergent mixed-methods approach (Heigham et al., [<reflink idref="bib20" id="ref28">20</reflink>]) where qualitative and quantitative data were collected simultaneously, analyzed separately, and then merged for comparison and interpretation was adopted to "provide a depth and breadth that a single approach may lack by itself"" (p. 136). Drawing on Ali et al. ([<reflink idref="bib2" id="ref29">2</reflink>])'s methodology which was used to undertake the qualitative evaluation of a learning analytics tool, the survey invited both scaled and open-ended responses about the usefulness of the tool.</p> <p>This study was conducted at the Graduate School of Health, a School within the Faculty of Health at the University of Technology Sydney. The Graduate School of Health is a postgraduate school offering clinical master's programs in seven disciplines including pharmacy, clinical psychology, genetic counselling, physiotherapy, speech pathology, good manufacturing practice and orthoptics. Before commencing the study, research ethics approval was obtained from the Human Research Ethics Committee at the University of Technology Sydney (ETH 23–8320).</p> <p>The first phase of the study was the development of the tool. The project team met on several occasions to undertake this. Ultimately, it was decided that the most efficient mechanism would be a survey that could be distributed using an online Qualtrics survey. (Qualtrics Provo, [<reflink idref="bib26" id="ref30">26</reflink>]). The questions in the survey were aligned both with the TELAS guidelines, but also the faculty goals. In addition to the survey questions, the tool provided academics with educational information in relation to each learning design assessment area and then prompted them to review their subject LMS sites to answer the survey questions. The tool also allowed academics to indicate if they would like to be contacted to arrange ongoing learning design support by engaging with institutional learning designers.</p> <p>All 43 subject coordinators from the faculty, who coordinated a subject in either Semester 1 or 2 (or both) of 2023, were approached via e-mail to participate in the study. Once the survey was completed by the subject coordinator, to validate the results and deepen the level of feedback provided, an independent peer reviewer was assigned to assess the LMS site and record responses to the survey questions. Thus, each subject was reviewed twice.</p> <p>A report was then sent to the academic providing feedback on their responses and those of the peer reviewer, highlighting key findings of the review and providing links to relevant education and pedagogical references to allow them to learn how to address areas requiring improvement.Data from the tool was exported to a spreadsheet and thematically analyzed using qualitative content analysis (Bowling, [<reflink idref="bib5" id="ref31">5</reflink>]; Vilhelmsson, [<reflink idref="bib32" id="ref32">32</reflink>]). To avoid peer reviewers' bias, only academic responses were extracted in relation to the usefulness of the tool. A bubble plot was used to visually represent the frequency of each theme for both the high standard areas of the site, and also those that needed improvement. Qualitative analysis was undertaken using Braun and Clarke's ([<reflink idref="bib6" id="ref33">6</reflink>]) six-step process of thematic analysis which involved data familiarisation; interim code generation; seeking themes; reviewing themes; defining and naming themes; and write-up (Braun & Clarke, [<reflink idref="bib6" id="ref34">6</reflink>]).</p> <p>Two researchers reviewed the data to familiarise themselves with the content, make notes and then discuss the findings. Then, data was analysed by CS, with emerging themes coded utilising NVivo (Lumivero, [<reflink idref="bib22" id="ref35">22</reflink>]) and exported to Microsoft Excel for analysis of quantitative data. The data were organised into themes, tabulated, and interpreted. Quotes supporting each theme were captured and discussed to allow exemplars to be extracted. The study has been reported per the standards for reporting qualitative research (SRQR) guidelines (O'Brien et al., [<reflink idref="bib23" id="ref36">23</reflink>]).</p> <p>Before discussing the results, it is important to comment on the positionality of the researchers. Both the researchers and the participants of the study are university academics within the same university and this means that many of the participants have a working relationship with some of the research team. To avoid any potential effects on the data due to these relationships, independent peer reviewers were assigned to conduct the peer review of the subject sites in addition to the reviews conducted by the subject coordinators. CS, a PhD candidate and academic from the faculty, was employed as a research assistant on the project. Three of the research team (PA, HB and KH) who work in different roles and disciplines consented to act as peer reviewers.</p> <hd id="AN0182468365-7">Results</hd> <p>Fifteen subject coordinators responsible for 25 subjects consented to participate in the research. The academics were from the disciplines of pharmacy (5 academics), genetic counselling (3 academics), physiotherapy (5 academics), speech pathology (1 academic) and orthoptics (1 academic). Background information on Semester 1 and Semester 2 subjects included in this study is shown in Table 1. The context and volume of learning content delivered in the subjects varied widely. Modalities of offering ranged from an online-only focus to a blended design combining content from the LMS and on campus classes to minimal online content with a focus on practical face to face lectures and workshops, work integrated learning or research.</p> <p>Table 1 Academic Participants and Subjects Reviewed</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Academic</p></th><th align="left"><p>Discipline</p></th><th align="left"><p>Semester Year</p></th><th align="left"><p>Number of Students</p></th><th align="left"><p>Subject Name</p></th><th align="left"><p>Delivery Mode/ Subject Type</p></th></tr></thead><tbody><tr><td align="left"><p>Academic 1</p></td><td align="left"><p>Genetic Counselling</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>29</p></td><td align="left"><p>Genetic Counselling Subject 1</p></td><td align="left"><p>Online</p></td></tr><tr><td align="left"><p>Academic 2</p></td><td align="left"><p>Genetic Counselling</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>31</p></td><td align="left"><p>Genetic Counselling Subject 2</p></td><td align="left"><p>Online</p></td></tr><tr><td align="left"><p>Academic 3</p></td><td align="left"><p>Genetic Counselling</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>31</p></td><td align="left"><p>Genetic Counselling Subject 3</p></td><td align="left"><p>Online</p></td></tr><tr><td align="left"><p>Academic 3</p></td><td align="left"><p>Genetic Counselling</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>26</p></td><td align="left"><p>Genetic Counselling Subject 4</p></td><td align="left"><p>Research Project subject- minimal online content</p></td></tr><tr><td align="left"><p>Academic 4</p></td><td align="left"><p>Orthoptics</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>50</p></td><td align="left"><p>Orthoptics Subject 1</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 5</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>81</p></td><td align="left"><p>Pharmacy Subject 1</p></td><td align="left"><p>Work Integrated Learning subject- minimal online content</p></td></tr><tr><td align="left"><p>Academic 5</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>79</p></td><td align="left"><p>Pharmacy Subject 2</p></td><td align="left"><p>Work Integrated Learning subject- minimal online content</p></td></tr><tr><td align="left"><p>Academic 6</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>81</p></td><td align="left"><p>Pharmacy Subject 3</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 6</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>79</p></td><td align="left"><p>Pharmacy Subject 4</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 7</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>81</p></td><td align="left"><p>Pharmacy Subject 5</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 7</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>74</p></td><td align="left"><p>Pharmacy Subject 6</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 7</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>73</p></td><td align="left"><p>Pharmacy Subject 7</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 8</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>74</p></td><td align="left"><p>Pharmacy Subject 8</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 8</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>73</p></td><td align="left"><p>Pharmacy Subject 9</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 9</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>81</p></td><td align="left"><p>Pharmacy Subject 10</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 9</p></td><td align="left"><p>Pharmacy</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>79</p></td><td align="left"><p>Pharmacy Subject 11</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 10</p></td><td align="left"><p>Physiotherapy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>77</p></td><td align="left"><p>Physiotherapy Subject 1</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 10</p></td><td align="left"><p>Physiotherapy</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>77</p></td><td align="left"><p>Physiotherapy Subject 2</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 11</p></td><td align="left"><p>Physiotherapy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>72</p></td><td align="left"><p>Physiotherapy Subject 3</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 11</p></td><td align="left"><p>Physiotherapy</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>64</p></td><td align="left"><p>Physiotherapy Subject 4</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 12</p></td><td align="left"><p>Physiotherapy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>74</p></td><td align="left"><p>Physiotherapy Subject 5</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 12</p></td><td align="left"><p>Physiotherapy</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>76</p></td><td align="left"><p>Physiotherapy Subject 6</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 13</p></td><td align="left"><p>Physiotherapy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>66</p></td><td align="left"><p>Physiotherapy Subject 7</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 14</p></td><td align="left"><p>Physiotherapy</p></td><td align="left"><p>Semester 1 2023</p></td><td align="left"><p>63</p></td><td align="left"><p>Physiotherapy Subject 8</p></td><td align="left"><p>Blended</p></td></tr><tr><td align="left"><p>Academic 15</p></td><td align="left"><p>Speech Pathology</p></td><td align="left"><p>Semester 2 2023</p></td><td align="left"><p>65</p></td><td align="left"><p>Speech Pathology Subject 1</p></td><td align="left"><p>Blended</p></td></tr></tbody></table> </ephtml> </p> <p>The results of the tool for each of the eight Technology Enhanced Learning Accreditation Standards (TELAS) areas of the site design are shown in Table 2. This table showcases the number of academics or peer reviewers who indicated that they felt the subject being reviewed demonstrated that area of the assessment.</p> <p>Table 2 LMS Site Development Tool Results</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Tool Area</p></th><th align="left"><p>Site Specifics</p></th><th align="left"><p>Academic Results</p><p>n (%)</p><p>25 (50)</p></th><th align="left"><p>Peer Reviewer Results</p><p>n (%)</p><p>25 (50)</p></th></tr></thead><tbody><tr><td align="left"><p>1. Inclusivity/ Accessibility</p></td><td align="left"><p>Site includes opportunities to interact with peers</p></td><td align="left"><p>11 (44%)</p></td><td align="left"><p>12 (48%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site includes activities that support collaboration</p></td><td align="left"><p>10 (40%)</p></td><td align="left"><p>11 (44%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site includes opportunities to engage in collaborative tasks</p></td><td align="left"><p>10 (40%)</p></td><td align="left"><p>6 (24%)</p></td></tr><tr><td align="left" /><td align="left"><p>Headings on the site are designed to enable screen reader use for students with visual impairment</p></td><td align="left"><p>20 (80%)</p></td><td align="left"><p>16 (64%)</p></td></tr><tr><td align="left" /><td align="left"><p>The tables on the site are designed to enable screen reader use for students with visual impairment</p></td><td align="left"><p>8 (32%)</p></td><td align="left"><p>7 (28%)</p></td></tr><tr><td align="left" /><td align="left"><p>All video and media contain captions to make these accessible for students with hearing/ cognitive impairment</p></td><td align="left"><p>11 (44%)</p></td><td align="left"><p>10 (40%)</p></td></tr><tr><td align="left"><p>2. LMS Visual Site Design</p></td><td align="left"><p>Site clearly divided in modules</p></td><td align="left"><p>18 (72%)</p></td><td align="left"><p>19 (76%)</p></td></tr><tr><td align="left" /><td align="left"><p>Logical flow of site design</p></td><td align="left"><p>19 (76%)</p></td><td align="left"><p>18 (72%)</p></td></tr><tr><td align="left"><p>3. Learning support</p></td><td align="left"><p>Site provided information on how feedback will be delivered, and requirements for student participation</p></td><td align="left"><p>16 (64%)</p></td><td align="left"><p>16 (64%)</p></td></tr><tr><td align="left" /><td align="left"><p>All learning tasks have clear instructions and expected outcomes</p></td><td align="left"><p>19 (76%)</p></td><td align="left"><p>16 (64%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site includes information on what students can expect from teaching staff</p></td><td align="left"><p>15 (60%)</p></td><td align="left"><p>17 (68%)</p></td></tr><tr><td align="left" /><td align="left"><p>Clear instructions for assessments are provided on site</p></td><td align="left"><p>20 (80%)</p></td><td align="left"><p>19 (76%)</p></td></tr><tr><td align="left" /><td align="left"><p>Assessment marking criteria provided on site</p></td><td align="left"><p>22 (88%)</p></td><td align="left"><p>15 (60%)</p></td></tr><tr><td align="left" /><td align="left"><p>Assessment exemplar(s) provided on site</p></td><td align="left"><p>11 (44%)</p></td><td align="left"><p>2 (8%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site provides opportunities for formative feedback prior to assessment submission</p></td><td align="left"><p>10 (40%)</p></td><td align="left"><p>9 (36%)</p></td></tr><tr><td align="left" /><td align="left"><p>Student learning outcomes are provided on site</p></td><td align="left"><p>9 (36%)</p></td><td align="left"><p>12 (48%)</p></td></tr><tr><td align="left"><p>4. Teacher Presence</p></td><td align="left"><p>Site has multiple opportunities for students to interact with teaching staff using online tools</p></td><td align="left"><p>12 (48%)</p></td><td align="left"><p>13 (52%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site includes learning materials and subject resources introduced by teaching staff</p></td><td align="left"><p>18 (72%)</p></td><td align="left"><p>16 (64%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site includes places where teaching staff clearly relate content to their academic and professional experience</p></td><td align="left"><p>10 (40%)</p></td><td align="left"><p>8 (32%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site uses first person language for the lecturers and tutors</p></td><td align="left"><p>14 (56%)</p></td><td align="left"><p>15 (60%)</p></td></tr><tr><td align="left"><p>5. Engagement of learning</p></td><td align="left"><p>Site supports students to engage prior knowledge</p></td><td align="left"><p>14 (56%)</p></td><td align="left"><p>12 (48%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site shows students how the subject fits into the course</p></td><td align="left"><p>17 (68%)</p></td><td align="left"><p>10 (40%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site includes activities that support knowledge creation & application</p></td><td align="left"><p>10 (40%)</p></td><td align="left"><p>13 (52%)</p></td></tr><tr><td align="left"><p>6. Digital Technologies</p></td><td align="left"><p>Site includes social polls/ comment boxes/ discussion boards</p></td><td align="left"><p>16 (64%)</p></td><td align="left"><p>17 (68%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site uses university supported online technologies</p></td><td align="left"><p>24 (96%)</p></td><td align="left"><p>25 (100%)</p></td></tr><tr><td align="left"><p>7. Facilitating Access to Content</p></td><td align="left"><p>Subject readings are clearly organised on site</p></td><td align="left"><p>12 (48%)</p></td><td align="left"><p>11 (44%)</p></td></tr><tr><td align="left"><p>8. Context for Learning Content Provided</p></td><td align="left"><p>Site provides clear signposting of educational materials</p></td><td align="left"><p>10 (40%)</p></td><td align="left"><p>10 (40%)</p></td></tr><tr><td align="left" /><td align="left"><p>Site communicates the relevance of learning content</p></td><td align="left"><p>8 (32%)</p></td><td align="left"><p>5 (20%)</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0182468365-8">Research Question 1: LMS Site Design Strengths and Weaknesses</hd> <p>The results from the tool proved to be useful in identifying the strengths (Fig. 1) and weaknesses (Fig. 2) of the current LMS sites in the school. These are presented below, before they are discussed further.</p> <p>Graph: Fig. 1 LMS Site Strengths</p> <p>Graph: Fig. 2 LMS Site Weaknesses</p> <hd id="AN0182468365-9">Inclusivity and Accessibility</hd> <p>Inclusivity and accessibility proved to be significant areas of weakness for many of the LMS sites. The majority of LMS sites were identified as requiring improvement on both the provision of collaborative tasks and opportunities to interact with peers (and hence provided limited opportunity to engender feelings of belonging and inclusivity). The main reasons given for this were that collaborative engagement mainly occurred during the face-to-face workshop activities and was not deemed a requirement of the LMS site: <emph>"Opportunities for collaboration are assumed to be in-class" (Peer Reviewer 3).</emph> Two subject coordinators stated collaboration and interaction with peers occurred through Microsoft Teams (a different platform to the one being reviewed) rather than the assessed LMS site. One subject coordinator expressed their ignorance about how they might do this and noted that they would like guidance on fostering belonging on the LMS site: "<emph>Happy to hear how to improve a sense of belonging if there are some helpful ways to do this</emph>" (<emph>Academic 11).</emph></p> <p>The questions around accessibility were also a weakness. Perhaps most concerning was the fact that nine academics did not answer the relevant questions of the survey designed to assess the accessibility of their subject sites. This might be because they were unsure (even with the educational material provided) how to assess this. There was also an issue with the access privileges for some of the peer reviewers: <emph>"Unable to check correct use of headings/alt text on images as I was added to the subject as an observer" (Peer Reviewer 1).</emph> The comments that were made about accessibility were predominantly around the use of colour. "<emph>The use of colour and contrast may be an issue for students with visual impairment (blue text on blue background, green and red text)" (Peer Reviewer 1).</emph></p> <hd id="AN0182468365-10">Quality of LMS Visual Site Design</hd> <p>Another theme that emerged from the tool data was that there was wide variation between subjects in terms of the quality of LMS site design. Tool results for five subject sites indicated that the LMS site design did not need any improvement and four stated that the site had a good layout and structure: "<emph>This is a well-structured and well-designed site. I could find nothing missing</emph>" (<emph>Peer Reviewer 2).</emph> In particular, the flow of the site was noticed by the reviewers: <emph>"Structured very well for this type of subject. One module for list of Virtual Placement Week lectures, and another module for weekly workshops, with a single page for each week. Nicely done, very clear" (Peer Reviewer 3).</emph> Reviewers noticed when this wasn't the case, as well: <emph>"Inconsistent look and feel. The structure is difficult to follow at times" (Peer Reviewer 1) and</emph> "<emph>Many pages had no banners, fonts, spacing and colour differed on many pages</emph>" (<emph>Peer Reviewer 1).</emph></p> <p>Suggestions to improve modularisation were focussed on summarising the content per module: <emph>"I felt that I was able to follow the structure of the course. My only suggestion would be that some of the modules were very long—lots and lots of pages. This might be a bit intimidating to students (and staff)" (Peer Reviewer 2) and "There is a very large amount of written content in this site—perhaps some headers to indicate the weeks would be beneficial here to help flag the focus and break up content areas a little more" (Peer Reviewer 3).</emph></p> <p>However, some of the academics noted that there were reasons for not adopting a common format. These reasons often related to the subject delivery mode. Research subjects, work integrated learning subjects or those mainly focussed on face-to-face teaching (as opposed to blended or online) were all seen to influence the viability of the LMS site: "<emph>This subject does not use modules as a typical subject would, because it is a major research project subject. The Canvas site is mostly used to manage assessment tasks and weekly workshops</emph>" (<emph>Academic 2).</emph></p> <hd id="AN0182468365-11">Learning Support: Communicating Expectations of Students in the Subject</hd> <p>Most subject sites consistently included clear instructions on learning tasks, assessments and outcomes, and provided students with marking criteria for all assessments. However, there was a lack of exemplars provided to students. Only 13 (26%) of respondents indicated that exemplars were used in the subject. Reasons given for this mainly related to the subject coordinators determining that this would not be appropriate: "Don't really want to provide student assessment exemplars as this would lead to them using it as a template. Students are provided with clear guidelines for assignment tasks" (Academic 16) and "<emph>No assessment exemplars but an opportunity to practice the assignments in class is given" (Peer Reviewer 1).</emph></p> <p>One important point to note is that while most subjects did have learning objectives listed somewhere on the LMS site (including in the subject outline) these were not often clearly communicated to students at the start of each module: <emph>"Most modules contain, in the overview page, a certain indication of the main topics covered, but this is not formally and consistently linked to the SLOs" (Academic 8).</emph></p> <hd id="AN0182468365-12">Teacher Presence</hd> <p>The majority of subject sites demonstrated teacher presence by including learning materials and subject resources introduced by the teacher and using first person language for teachers and tutors. Additionally, opportunities for students to interact with teachers was provided through discussion forums, comments boxes, polls and widgets in half of the subject sites reviewed. Opportunities for formative feedback prior to assessment were less frequently available.</p> <p>The requirement for lecturers and tutors to relate content to their professional experience was also identified as a potential area for improvement. Some academics stated that students do not need teacher expertise to be explicitly communicated in the subject site as students are aware that the academics are experienced clinicians: <emph>"Incorporation of content into practice is implied in this subject as tutors are practicing pharmacists that bring real world experience to the students learning" (Academic 8).</emph></p> <p>Some suggested improvements for this section included adding a welcome video as only 3 of 25 sites had one. For those that did have a video, there were some improvements suggested for it, too: <emph>"Subject welcome video is not directed at students or delivered by staff, but video provided on welcome page provides some basic background on the role of pharmacist. However, it doesn't contain any information about the subject or learning expectations, nor introduces any of the academic staff" (Peer Reviewer 3).</emph></p> <hd id="AN0182468365-13">Engagement of Learning</hd> <p>The results from the tool recorded that just over half of the subjects made links to previous subjects and invited students to consider past skills. Several academics stated linking to previous subjects was less relevant as the subject was an introductory subject:</p> <p>"<emph>As this is an introductory subject links to prior subjects and knowledge aren't very frequent. There are no explicit references to future subjects but there are implications that the skills picked up in this subject will be used in later subjects further on</emph>" (<emph>Peer Reviewer 3).</emph></p> <p>Demonstrating how the subject related to the university course and the relevance of subject learning content were identified as areas for improvement in 20 of the 25 subjects: "<emph>Integration of linkage to future subjects to be completed may be helpful for students to demonstrate big picture focus of course overview</emph>" (<emph>Academic 9).</emph></p> <p>One academic noted although links did not currently exist, it would be beneficial to link the final semester subject within the course and link to its future application in the workplace: "<emph>This subject is a final semester subject in the degree. It may be beneficial to foreshadow how this will be used in future practice</emph>" (<emph>Academic 9).</emph></p> <hd id="AN0182468365-14">Digital Technologies</hd> <p>In relation to active learning facilitated by digital technology, most sites embedded discussion boards, social polls or comments boxes. Sites less frequently used activities that supported collaboration or knowledge creation and application. Several academics stated that they were not sure if the students would use these if they were available: <emph>"I don't think the discussion board is used well by me or by students. Not sure how to increase engagement here. I don't know if it's needed though, because the classes are all very interactive and engaging (based on SFS feedback)" (Academic 11).</emph> One of the peer reviewers also indicated.</p> <p>this was a relevant concern stating: "<emph>Most of the comments boxes were empty so I'm not sure how much student engagement with the online content was occurring" (Peer Reviewer 1).</emph></p> <p>Reasons for not using recommended technologies included that the subject delivery mode required less online active learning: <emph>"As a work integrated learning subject, it appears the majority of the coursework is in a lecture and assessment format, whilst the rest is completed on placement" (Peer Reviewer 3).</emph></p> <p>Some academics identified that although they felt that opportunities for online learning were potentially beneficial for students that the time required in digital learning object development made this impractical:</p> <p> <emph>"The Canvas site is considered very secondary to the in-class interactions, and so the learning tech is mostly embedded into classes rather than the Canvas site. I am very happy for someone to add more tech into the Canvas page as it would likely add value, I'm just currently not convinced enough that it is worth the time" (Academic 12).</emph> </p> <p>Some peer reviews identified that adding interactive learning as potentially beneficial despite the subject delivery mode: "<emph>The delivery of content on the LMS is primarily text, image, and video focused. Whilst the primary active coursework would take place in the workshops, perhaps some of these larger text pages could be converted into interactive learning objects to help break up larger pages of content</emph>" (<emph>Peer Reviewer 3).</emph></p> <hd id="AN0182468365-15">Facilitating Access to Learning Content</hd> <p>Results from the tool indicated that reading lists were a common area of LMS site design that was highlighted as needing improvement (Fig. 1). An example of feedback from peer reviewers includes: "<emph>Library Reading List should be added to subject resources page. All articles should be linked to reading list where relevant"</emph> (<emph>Peer Reviewer 1).</emph></p> <p>Most sites did use readings, but these were not set up with links to the library. For some subjects the readings included links to external websites for organisations that required a log in; some subjects used textbooks but were not linked to the library, and some academics indicated that the practical subjects did not need any readings.</p> <hd id="AN0182468365-16">Provision of Context for Learning Content</hd> <p>Providing clear signposting for learning content was identified as an area requiring improvement in 60% of LMS sites by both academics and peer reviewers. Communicating the relevance of learning content to students was also identified as an area requiring improvement in most LMS sites with 68% of academic reviews and 80% of peer reviews identifying this as a gap in the LMS site design. An example of feedback from peer reviewers includes <emph>"There could be some more context provided, such as a brief intro and summary page to better establish the context for the week's learning" (Peer Reviewer 3).</emph></p> <hd id="AN0182468365-17">Differences Between Academic and Peer Reviewer Responses</hd> <p>Tool responses from academic and peer reviewers were generally aligned. Where discrepancies existed, academics' assessment of their subject sites were more positive than peer reviewers. The largest discrepancy between the results of the peer reviewers and academics was in relation to consideration if assessment exemplar(s) were provided on the LMS site with eleven academics (44%) stating that the LMS site included this, whereas only two (8%) peer reviewers agreed. Other areas in which the academic opinions differed from peer reviewers included: in the provision of assessment marking criteria ((22 (88%) sites identified by academics versus 15 (60%) identified by peer reviewers)), in the provision of information in relation to how the site shows students how the subject fits into the course ((17 (68%) academics versus 10 (40%) peer reviewers)), in relation to if opportunities to engage in collaborative tasks existed in the site with ((10 (40%) of academics versus 6 (24%) peer reviewers)), and if the site communicated the relevance of learning content ((8 (32%) of academics versus 5 (20%) of peer reviewer responses).</p> <hd id="AN0182468365-18">Requests for Institutional Learning Design Support</hd> <p>Provision of additional learning design support was requested by 11 of 15 (73%) academics who indicated they would like to by contacted by the university learning design support team. Areas of learning design academics identified that they would like more information on included inclusivity and accessibility (11/15) LMS site design (3/15), communicating expectations of students in the subject (11/15), teacher presence (11/15), engaging knowledge and facilitating active learning (11/15) and the provision of context for learning content (6/15).</p> <hd id="AN0182468365-19">Research Question 2: Academic Feedback in Relation to the LMS Site Development Tool</hd> <p>Most of the subject coordinators and peer reviewers were positive about the tool with 10/25 (40%) indicating that the tool was very useful and 12/25 somewhat useful (48%). Only two subject coordinators were neutral about the usefulness (8%), and only one subject coordinator indicated that it was somewhat not useful (4%).</p> <p>All the academics who stated that the tool was very useful co-ordinated blended subjects. Academics who rated the tool as somewhat useful came from blended subjects, work-integrated learning subjects and research subjects. The subject coordinators who were neutral about the usefulness of the tool both co-ordinated blended subjects. The academic who indicated that the tool was somewhat not useful coordinated a research subject.</p> <p>When asked which aspects of the tool required development four respondents (16%) stated they did not think the tool required improvement with one stating:</p> <p> <emph>"I don't think it needs improvement as it covers all the main aspects of what a Canvas subject site should be. It actually provided me with a very good insight on how to improve my Canvas sites moving forward—Thanks!" (Academic 6)</emph> and another stating:</p> <p>"<emph>It is great already—and has given me good insight. Completing this has been more helpful than the three or four workshops I've completed on Canvas. As a teaching tool, if it had examples it would be fantastic" (Academic 10).</emph></p> <p>When asked for proposed areas of improvement of the tool, responses were centred around ensuring clarity of questions 5/25 (20%). "<emph>I think some prefacing, including some definitions would help" (Academic 13).</emph> Feedback from one subject coordinator included: <emph>"My gut feeling is that a little video from you to explain each point of the tool would be amazing for the teacher learning to be interactive. E.g. a here's a script for a welcome video </emph>etc<emph>." (Academic 11).</emph></p> <p>Two suggestions from an academic were focussed on improving implementation: "<emph>after completion provide an output in form of checklist of exactly what to do. Provide a link to the resource that clearly states how to do this. This can be checked once completed</emph>" (<emph>Academic 12)</emph> and: "<emph>It should be used as a tool with time set aside to actually make the changes, many of the suggested changes will take weeks to actually develop and implement" (Academic 12).</emph> Aligning application of the tool with a time in the university semester that has a lower teaching load was identified as a facilitator for implementing the recommended changes identified by the tool.</p> <p>Academics also highlighted areas for improvement of the tool such as that the tool was not as relevant for subjects that had a focus on in class learning (5/25) with statements such as: "<emph>I thought the tool was good. But perhaps it applies more to subjects that are taught entirely online? So maybe a different version for where classes are all in-person? I am probably wrong here but keen to hear more about this" (Academic 11).</emph> Some were also curious if engagement from students on the site can be traced back to improve the site: <emph>"It would be great to be able to quantify student engagement with these activities as a way of reflecting on which of these activities are working well/need amendment</emph>" (<emph>Academic 4).</emph></p> <hd id="AN0182468365-20">Discussion</hd> <p></p> <hd id="AN0182468365-21">Evaluation of LMS Site Design</hd> <p>Although LMS site design flow and structure were found to be strengths of the LMS sites, results from the tool identified that academics required the most support in improving the signposting and provision of context for learning content. In their paper examining design of an online educational module using UDL principles, Seymour et al. found that improving the clarity of the weekly structure and signposting supported students engagement and preparation for learning (Seymour, [<reflink idref="bib28" id="ref37">28</reflink>]). This was achieved by providing a weekly preparation document and follow up e-mail that highlighted the relevance of upcoming content and clearly outlined the preparation tasks that were required (Seymour, [<reflink idref="bib28" id="ref38">28</reflink>]). Assisting academics with understanding how to improve their signposting of content and communication of content relevance using strategies such as these may be a useful focus for future interventions.</p> <p>Subjects that were focussed on practical skills such as work integrated learning and research project subjects did not consistently follow the recommended LMS site structure. Feedback from one academic included "I think the tool would be more useful for a subject that has a different kind of structure, since this one is a research subject and does not include much learning activity on Canvas." Another academic expressed a similar view stating "The majority of the content for this subject site is self-paced research, so the blended format isn't as applicable here." This may indicate that an adapted LMS site structure is required for these subject types and a tool that uses a "one size fits all" approach to assessing LMS site structure may not be appropriate. Alternatively, the results may indicate that further development of LMS site for research projects or work-integrated learning subjects is required. As students are often working independently when participating in research or work-integrated learning subjects, there is potential for LMS sites to be leveraged more effectively to facilitate a place for collaborative learning to occur.</p> <p>Addressing accessibility and inclusivity requirements was an area identified as a weakness of many of the LMS sites reviewed. Upskilling academics in activities such as adding captions to videos, alternative text to images, considering colour contrast and ensuring screen reader compatibility should increase the accessibility and inclusivity of LMS sites for all students. Pearson and Koppi identified the importance of a structured approach when supporting academics to use the available accessible features of the learning environment successfully (Pearson & Koppi, [<reflink idref="bib24" id="ref39">24</reflink>]). Education in relation to current accessibility guidelines and how to practically ensure compliance with them would help to address the gaps identified by this intervention.</p> <p>Creating belonging and opportunities for online collaboration and engagement were also identified by the tool as a potential focus for future improvement interventions. Haddow and Brodie ([<reflink idref="bib16" id="ref40">16</reflink>]) identified the need for creating an authentic community and facilitating student participation and ownership of the community (Haddow & Brodie, [<reflink idref="bib16" id="ref41">16</reflink>]). In terms of LMS site design, potential methods to foster belonging could be creation of opportunities for online collaboration and engagement for students. Providing academics with training to support an understanding of evidence-based strategies to foster inclusivity and belonging may also improve this aspect of LMS site design. It should be noted that in blended subjects there is often an opportunity for significant face to face interaction between students and academics allowing collaboration and the creation of a sense of student belonging outside of the LMS site.</p> <hd id="AN0182468365-22">Implications</hd> <p>The positive response of academics to the learning design support tool provides evidence for the need for institutional support in relation to LMS site design. Academics identified that additional training in LMS site design and institutional support to address identified gaps would be useful. This accords with research by Fathema and Akanda ([<reflink idref="bib15" id="ref42">15</reflink>]) that concluded that universities should make LMS evaluation routine and that universities should offer general and advanced training for academics with different levels of LMS experience (Fathema & Akanda, [<reflink idref="bib15" id="ref43">15</reflink>]).</p> <hd id="AN0182468365-23">Strengths and Limitations of the Study</hd> <p>This study describes a novel learning design quality assurance intervention aimed at academics. The development tool was well received and facilitated comprehensive assessment of LMS sites. Weaknesses identified included the time required to conduct the review of the subject site with some academics identifying a need for more education on the tool before being asked to implement it.</p> <p>Our findings indicate that despite sites containing the basic information needed for learning, any mode of subject delivery (online, face to face or blended) could benefit from implementing extra information around the teaching staff, assessments (examples and feedback prior to submission), relevance of the subject in relation to the whole course/ profession and organization of readings and signposting education materials. Use of a tool such as the one developed in this project is an efficient and effective way to support quality assurance and assist academics with targeted feedback and support.</p> <p>In relation to assessing accessibility and inclusivity there were some limitations found in relation to the level of access that the peer reviewers had when assessing the LMS sites. Future iterations of the tool will include ensuring that the reviewers have administrator access to each LMS site ensuring they can view all relevant accessibility elements of the site such as captions on videos.</p> <p>Differences between the academic responses and the peer reviewer responses may have occurred for a range of reasons. The tool itself may not have been clear in what was being assessed and how academics should conduct that assessment. Academics participating in this project were not given any training or support in relation to the use of the tool and this may be a learning for similar projects in the future. Clear and specific guidelines on how to assess each aspect of the LMS site should improve consistency in results and support academics using the tool.</p> <p>When there were differences in the assessment between peer reviewers and academics, the academic assessment was generally more favourable than that of the peer reviewers. This indicates that it may be potentially difficult for subject co-ordinators to conduct a completely independent review of the subjects they co-ordinate. Alternatively, the academics could have had a better understanding of their own subjects and context and this may have contributed to the variation in assessment results. The use of peer reviewer assessment allowed for validation of the results and was a strength of the intervention.</p> <p>Future research could look to assess the impact of implementing improvements to learning design of subject sites and academic knowledge of learning design by considering aspects such as student engagement with online content and academic grades and performance before and after tool implementation. Investigating student experiences in relation to the use of LMS sites would also be potentially beneficial to further inform future tool development.</p> <hd id="AN0182468365-24">Conclusion</hd> <p>This project confirmed that academics are willing to use a learning design support tool and that it is effective in identifying potential areas of improvement of LMS sites. Key areas of academic professional development identified in relation to learning design include equipping academics with the skills to improve accessibility and inclusion of LMS sites and foster belonging with opportunities for online collaboration between students. Adapting future LMS site development interventions to the context and delivery mode of each subject may improve their relevance and effectiveness.</p> <hd id="AN0182468365-25">Acknowledgements</hd> <p>The authors would like to acknowledge the University of Technology Sydney Faculty of Health who supported this research through a faculty teaching and learning scholarship. The authors would also like to acknowledge the time and contributions of the project team who were instrumental in the design of the development tool and the academics who participated in the research.</p> <hd id="AN0182468365-26">Authors Contribution</hd> <p>All authors contributed to the study conception and design. Material preparation and data collection were performed by Helen Benson, Keith Heggart and Patrick Abela. Data analysis was performed by Helen Benson and Chloé Smit. The first draft of the manuscript was written by Helen Benson and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.</p> <hd id="AN0182468365-27">Data Availability</hd> <p>The data that support the findings of this study are available on request from the corresponding author, [HB]. The data are not publicly available due to ethics approval requirements.</p> <hd id="AN0182468365-28">Declarations</hd> <p></p> <hd id="AN0182468365-29">Competing Interest</hd> <p>The authors have no competing interests to declare.</p> <hd id="AN0182468365-30">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0182468365-31"> <title> References </title> <blist> <bibl id="bib1" idref="ref10" type="bt">1</bibl> <bibtext> Ahn MY, Davis HH. Four domains of students' sense of belonging to university. Studies in Higher Education. 2020; 45; 3: 622-634. 10.1080/03075079.2018.1564902</bibtext> </blist> <blist> <bibl id="bib2" idref="ref29" type="bt">2</bibl> <bibtext> Ali L, Hatala M, Gašević D, Jovanović J. A qualitative evaluation of evolution of a learning analytics tool. Computers & Education. 2012; 58; 1: 470-489. 10.1016/j.compedu.2011.08.030</bibtext> </blist> <blist> <bibl id="bib3" idref="ref6" type="bt">3</bibl> <bibtext> ASCILITE. (2020). TELAS Technology Enhanced Learning Accreditation Standards. Retrieved April 8, 2024, from https://<ulink href="http://www.telas.edu.au/wp-content/uploads/TELAS-Framework-Complete-v1.6.pdf">www.telas.edu.au/wp-content/uploads/TELAS-Framework-Complete-v1.6.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib4" idref="ref24" type="bt">4</bibl> <bibtext> Benson, H, Williams, K. A, & Heggart, K. (2024). Quality assurance interventions in blended learning design: A systematic review of the literature. Australasian Journal of Educational Technology. https://doi.org/10.14742/ajet.9362</bibtext> </blist> <blist> <bibl id="bib5" idref="ref31" type="bt">5</bibl> <bibtext> Bowling, A. (2002). Research methods in health: Investigating health and health services. Open University Press.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref33" type="bt">6</bibl> <bibtext> Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006; 3; 2: 77-101. 10.1191/1478088706qp063oa</bibtext> </blist> <blist> <bibl id="bib7" idref="ref5" type="bt">7</bibl> <bibtext> Cappelli, T, & Smithies, A. (2021). Building organisational capacity for blended learning: An evidence-based approach. Academic Conferences International Limited, 73–79. https://doi.org/10.34190/EEL.21.026</bibtext> </blist> <blist> <bibl id="bib8" idref="ref14" type="bt">8</bibl> <bibtext> Carter RA Jr, Rice M, Yang S, Jackson HA. Self-regulated learning in online learning environments: Strategies for remote learning. Information and Learning Sciences. 2020; 121; 5/6: 321-329. 10.1108/ILS-04-2020-0114</bibtext> </blist> <blist> <bibl id="bib9" idref="ref19" type="bt">9</bibl> <bibtext> CAST. (2018). Universal design for learning guidelines (Version 2.2). Retrieved April 8, 2024, from <ulink href="http://udlguidelines.cast.org">http://udlguidelines.cast.org</ulink></bibtext> </blist> <blist> <bibtext> Chen YC, Hwang RH, Wang CY. Development and evaluation of a Web 2.0 annotation system as a learning tool in an e-learning environment. Computers & Education. 2012; 58; 4: 1094-1105. 10.1016/j.compedu.2011.12.017</bibtext> </blist> <blist> <bibtext> Cho M-H. Online student orientation in higher education: A developmental study. Educational Technology Research and Development. 2012; 60; 6: 1051-1069. 10.1007/s11423-012-9271-4</bibtext> </blist> <blist> <bibtext> Coy, K. (2017). Post-secondary educators can increase educational reach with universal design for learning. Educational Renaissance, 5(1). https://doi.org/10.33499/edren.v5i1.94</bibtext> </blist> <blist> <bibtext> Emami Z. Facilitating student learning through engaging prior learning: An exercise in development of agency. International Journal of Pluralism and Economics Education. 2015; 6; 2: 198-205. 10.1504/IJPEE.2015.072594</bibtext> </blist> <blist> <bibtext> Fabian K, Smith S, Taylor-Smith E, Meharg D. Identifying factors influencing study skills engagement and participation for online learners in higher education during COVID-19. British Journal of Educational Technology. 2022; 53; 6: 1915-1936. 10.1111/bjet.13221</bibtext> </blist> <blist> <bibtext> Fathema N, Akanda MH. Effects of instructors' academic disciplines and prior experience with learning management systems: A study about the use of Canvas. Australasian Journal of Educational Technology. 2020; 36; 4: 113-125. 10.14742/ajet.5660</bibtext> </blist> <blist> <bibtext> Haddow, C, & Brodie, J. (2023). Harnessing innovation approaches to support community and belonging in higher education. Innovations in Education and Teaching International, ahead-of-print, 1–14. https://doi.org/10.1080/14703297.2023.2176907</bibtext> </blist> <blist> <bibtext> Halverson LR, Graham CR, Spring KJ, Drysdale JS, Henrie CR. A thematic analysis of the most highly cited scholarship in the first decade of blended learning research. The Internet and Higher Education. 2014; 20: 20-34. 10.1016/j.iheduc.2013.09.004</bibtext> </blist> <blist> <bibtext> Heggart K, Dickson-Deane C. What should learning designers learn?. Journal of Computing in Higher Education. 2022; 34: 281-296. 10.1007/s12528-021-09286-y</bibtext> </blist> <blist> <bibtext> Houston L. Efficient strategies for integrating universal design for learning in the online classroom. The Journal of Educators Online. 2018; 15; 3: n3. 10.9743/jeo.2018.15.3.4</bibtext> </blist> <blist> <bibtext> Heigham, J, Croker, R, Heigham, J, & Croker, R. A. (2009). Qualitative research in applied linguistics: A practical introduction (1st ed.). Palgrave Macmillan UK. https://doi.org/10.1057/9780230239517</bibtext> </blist> <blist> <bibtext> Koh JHL, Kan RYP. Students' use of learning management systems and desired e-learning experiences: Are they ready for next generation digital learning environments?. Higher Education Research & Development. 2020; 40; 5: 995-1010. 10.1080/07294360.2020.1799949</bibtext> </blist> <blist> <bibtext> Lumivero. (2023). NVivo 12. Retrieved April 8, 2024, from <ulink href="http://www.lumivero.com">www.lumivero.com</ulink></bibtext> </blist> <blist> <bibtext> O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine. 2014; 89; 9: 1245-1251. 10.1097/ACM.0000000000000388</bibtext> </blist> <blist> <bibtext> Pearson, E. J, & Koppi, T. (2002). Inclusion and online learning opportunities: Designing for accessibility. Research in Learning Technology, 10(2). https://doi.org/10.3402/rlt.v10i2.11398</bibtext> </blist> <blist> <bibtext> Põldoja, H, Duval, E, & Leinonen, T. (2016). Design and evaluation of an online tool for open learning with blogs. Australasian Journal of Educational Technology, 32(2). https://doi.org/10.14742/ajet.2450</bibtext> </blist> <blist> <bibtext> Qualtrics. (2024). Qualtrics software (Version January 2024). Provo, Utah, USA. Retrieved April 8, 2024, from https://<ulink href="http://www.qualtrics.com">www.qualtrics.com</ulink></bibtext> </blist> <blist> <bibtext> Reyna, J. (2013). The importance of visual design and aesthetics in e-learning. Training & Development, 40(5), 28–31. Retrieved April 8, 2024, from https://<ulink href="http://www.proquest.com/trade-journals/importance-visual-design-aesthetics-e-learning/docview/1462494447/se-2">www.proquest.com/trade-journals/importance-visual-design-aesthetics-e-learning/docview/1462494447/se-2</ulink></bibtext> </blist> <blist> <bibtext> Seymour M. Enhancing the online student experience through the application of universal design for learning (UDL) to research methods learning and teaching. Education and Information Technologies. 2023; 29: 2767-2785. 10.1007/s10639-023-11948-6</bibtext> </blist> <blist> <bibtext> Smith Budhai S, Skipwith KB. Best practices in engaging online learners through active and experiential learning strategies. Taylor & Francis Group. 2021. 10.4324/9781003140405</bibtext> </blist> <blist> <bibtext> Stone C, Springer M. Interactivity, connectedness, and 'teacher presence': Engaging and retaining students online. Australian Journal of Adult Learning. 2019; 59; 2: 146-169</bibtext> </blist> <blist> <bibtext> Tooth, P. (2022). How reading lists improve student experience and copyright compliance. LX Blog. https://lx.uts.edu.au/blog/2022/05/24/reading-lists-improve-student-experience-copyright-compliance/</bibtext> </blist> <blist> <bibtext> A Vilhelmsson 2018 Using qualitative content analysis to study consumer reports on adverse drug reactions SAGE Publications Ltd https://doi.org/10.4135/9781526439949</bibtext> </blist> </ref> <aug> <p>By Helen Benson; Keith Heggart; Kylie A. Williams; Chloé Smit and Patrick Abela</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib17" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib21" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib15" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib18" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib12" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib24" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib16" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib27" firstref="ref11"></nolink> <nolink nlid="nl9" bibid="bib19" firstref="ref12"></nolink> <nolink nlid="nl10" bibid="bib11" firstref="ref13"></nolink> <nolink nlid="nl11" bibid="bib30" firstref="ref15"></nolink> <nolink nlid="nl12" bibid="bib13" firstref="ref16"></nolink> <nolink nlid="nl13" bibid="bib29" firstref="ref17"></nolink> <nolink nlid="nl14" bibid="bib31" firstref="ref18"></nolink> <nolink nlid="nl15" bibid="bib28" firstref="ref21"></nolink> <nolink nlid="nl16" bibid="bib14" firstref="ref22"></nolink> <nolink nlid="nl17" bibid="bib10" firstref="ref26"></nolink> <nolink nlid="nl18" bibid="bib25" firstref="ref27"></nolink> <nolink nlid="nl19" bibid="bib20" firstref="ref28"></nolink> <nolink nlid="nl20" bibid="bib26" firstref="ref30"></nolink> <nolink nlid="nl21" bibid="bib32" firstref="ref32"></nolink> <nolink nlid="nl22" bibid="bib22" firstref="ref35"></nolink> <nolink nlid="nl23" bibid="bib23" firstref="ref36"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1460904
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Fostering Excellence in Blended Learning, a Mixed Methods Investigation of an Academic Learning Design Support Tool
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Helen+Benson%22">Helen Benson</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-6918-5045">0000-0002-6918-5045</externalLink>)<br /><searchLink fieldCode="AR" term="%22Keith+Heggart%22">Keith Heggart</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-2331-1234">0000-0003-2331-1234</externalLink>)<br /><searchLink fieldCode="AR" term="%22Kylie+A%2E+Williams%22">Kylie A. Williams</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-5094-243X">0000-0002-5094-243X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Chloé+Smit%22">Chloé Smit</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-3037-5837">0000-0002-3037-5837</externalLink>)<br /><searchLink fieldCode="AR" term="%22Patrick+Abela%22">Patrick Abela</searchLink> (ORCID <externalLink term="http://orcid.org/0009-0007-8663-3317">0009-0007-8663-3317</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22TechTrends%3A+Linking+Research+and+Practice+to+Improve+Learning%22"><i>TechTrends: Linking Research and Practice to Improve Learning</i></searchLink>. 2025 69(1):71-83.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 13
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Blended+Learning%22">Blended Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Management+Systems%22">Learning Management Systems</searchLink><br /><searchLink fieldCode="DE" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="DE" term="%22Usability%22">Usability</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Software+Evaluation%22">Computer Software Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Electronic+Learning%22">Electronic Learning</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s11528-024-01019-z
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 8756-3894<br />1559-7075
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This article investigates the effectiveness of a newly developed online tool for assessing and improving the design and accessibility of Learning Management System (LMS) sites in Higher Education. The study employs a mixed-methods approach, incorporating both quantitative and qualitative data. Results from 25 subject coordinators and 3 peer reviewers reveal the strengths and weaknesses in LMS site design at a particular institution. It identifies recommendations for enhancements such as clearer signposting of educational materials, improved organisation of reading lists, and increased opportunities for collaboration. In addition, the research identifies key areas for improvement in institutional LMS site design, including consistency, organisation, and ease of navigation. These findings underscore the importance of engaging prior knowledge, fostering active learning, and ensuring clear communication of student expectations. The tool is shown to be a valuable resource for educators to assess and enhance their online teaching environments. The study's findings contribute to the ongoing discourse on optimising blended learning experiences, emphasising the need for ongoing improvement in LMS site design and accessibility.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1460904
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1460904
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s11528-024-01019-z
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 71
    Subjects:
      – SubjectFull: Blended Learning
        Type: general
      – SubjectFull: Learning Management Systems
        Type: general
      – SubjectFull: Higher Education
        Type: general
      – SubjectFull: Usability
        Type: general
      – SubjectFull: Computer Software Evaluation
        Type: general
      – SubjectFull: Electronic Learning
        Type: general
    Titles:
      – TitleFull: Fostering Excellence in Blended Learning, a Mixed Methods Investigation of an Academic Learning Design Support Tool
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Helen Benson
      – PersonEntity:
          Name:
            NameFull: Keith Heggart
      – PersonEntity:
          Name:
            NameFull: Kylie A. Williams
      – PersonEntity:
          Name:
            NameFull: Chloé Smit
      – PersonEntity:
          Name:
            NameFull: Patrick Abela
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 8756-3894
            – Type: issn-electronic
              Value: 1559-7075
          Numbering:
            – Type: volume
              Value: 69
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: TechTrends: Linking Research and Practice to Improve Learning
              Type: main
ResultId 1