Part I: The 3 TвЂ™s of Online Assessment: Tools, Techniques, and (Saving) Time Curtis J. Bonk, Professor, Indiana University President, CourseShare http://php.indiana.edu/~cjbonk; firstname.lastname@example.org Vanessa Paz Dennen, Assistant Professor Florida State University http://www.vanessadennen.com, email@example.com Session Objectives пѓ� Detail online assessment techniques пѓ� Discuss how to match learning activities with learner assessments пѓ� Examine instructor time and comfort issues пѓ� Discuss ways to limit and detect cheating and plagiarism пѓ� Document online tools and resources for assessment Online Assessment Techniques (with some time-saving tips added inвЂ¦) Is this motivating? How would you feel? пѓ� пѓ� пѓ� пѓ� пѓ� пѓ� пѓ� You take an online class. You read some Web pages. Maybe you watch some videos or hear some audio clips. Maybe you ponder some study review questions. You take a multiple choice test online. You receive an automated score on the test. Class is over. How about this scenario? пѓ� пѓ� пѓ� пѓ� пѓ� пѓ� пѓ� You take an online class. You вЂњmeetвЂќ your fellow students on the d-board. You read some materials. You find and share some materials too. You participate in some discussions of course concepts. You take a multiple choice test. You receive an automated score on the test. Class is over. Commentary on Scenario 1 пѓ� No interaction with peers. пѓ� Students donвЂ™t feel вЂњmissedвЂќ if they donвЂ™t participate. пѓ� Not clear why course is online (except perhaps for media elements). пѓ� Potential for immediate feedback is nice -but assessment format is limiting. Commentary on Scenario 2 пѓ� Interaction with peers is great. Serves as a motivator. пѓ� Community is likely to develop. пѓ� Students will feel involved and important if they share examples and resources. пѓ� Assessment format may not be well aligned given the activities. пѓ� Class lacks closure in a manner appropriate to the activities. Assessment and Learning пѓ� Course objectives, activities, and assessments should be in alignment пЃ¬ This tends to be an issue in courses regardless of medium. пѓ� Example: пЃ¬ пЃ¬ In class students conduct a debate Students are tested on their ability to recall facts Mis-aligned Online Learning and Assessment пѓ� A not-uncommon scenario пЃ¬ пЃ¬ пЃ¬ пѓ� Discussion is used as a learning activity Students are required to participate Participation is noted by how many messages were composed by a student But does this method measure learning? Common Online Assessment Complaints пѓ� Instructor пЃ¬ пЃ¬ пЃ¬ пЃ¬ perspective ThereвЂ™s too much to assess! I donвЂ™t know what activities to assess! I donвЂ™t know if students really are ready for the test! How do I know the student actually did the work/took the test? Common Online Assessment Complaints пѓ� Student perspective пѓ� If theyвЂ™re supposed to discuss, why doesnвЂ™t that count as part of their grade? пѓ� If theyвЂ™re just supposed to do something, why does quality matter? пѓ� I just got a number, no feedback. пѓ� I didnвЂ™t get participation feedback. The Feedback Issue пѓ� Students participating in online activities look for feedback to know: пЃ¬ пЃ¬ пЃ¬ пѓ� A. the instructor is reading their contributions B. their participation is valued C. their participation is adequate, in terms of quality and quantity Feedback need not be individualized to be effective пЃ¬ Whole class commentary provided on a regular basis was found to be just as satisfactory from the student point of view (Dennen, 2001) The Assessment Issue пѓ� Often, online activities go unassessed пЃ¬ вЂњAdd-onвЂќ syndrome: Adding an online activity to a previously designed class because it sounds like a good idea The Assessment Issue пѓ� Students are more likely to participate when then know there is impact on their grade пЃ¬ Direct impact: graded on participation (quality, quantity or both) пЃ¬ Indirect impact: participation should bolster performance on other assessments пЃ¬ Students quickly become aware if an online activity is not related to assessed learning objectives The Assessment Issue, Cont. пѓ� Sometimes the wrong things are assessed пѓ� Examples: пЃ¬ Assessing studentsвЂ™ online moderation skills when the course topic/learning objectives have nothing to do with online moderation пЃ¬ Assessing quantity of participation, but not quality вЂў which, granted, is easier, but encourages sloppy message posting rather than thoughtful learning dialogues BloomвЂ™s Revised Taxonomy пѓ� A useful tool for checking alignment пѓ� Also great to guide your course design! QuickTimeв„ў and a TIFF (Uncompress ed) decompressor are needed to s ee this picture. To Find Out MoreвЂ¦ A Taxonomy For Learning Teaching and Assessing By Anderson And Krathwohl QuickTim eв„ў and a TIFF (Un co m press ed ) de com pres so r are n eed ed to se e this p ictu re . Assessment Techniques OptionsвЂ¦ пѓ� Formative or Summative пѓ� Student-led (Self or Peer) or Teacher-led пѓ� Public or Private пѓ� Process or Product Other issues to considerвЂ¦ пѓ� вЂњObjectiveвЂќ or Interpretive пѓ� Rubric-based or Wholistic Formative vs. Summative Assessments Formative пѓ� Alleviate student anxieties re: expectations пЃ¬ пѓ� Seem especially high in online classes Encourage working toward mastery пѓ� Can be informal Summative пѓ� Used for studentвЂ™s grade пѓ� Assumed to be вЂњbestвЂќ effort Example: Online Formative Assessment пѓ� Paper пЃ¬ пЃ¬ draft discussion forum Start a discussion forum for papers-inprogress Have each student start a thread and post elements of their papers as they complete them (e.g., topic, major claims, research sources) вЂў A schedule for each element is useful пЃ¬ Monitor and provide feedback Student vs. Teacher Led Teacher-led пѓ� вЂњTraditionalвЂќ assessment пѓ� Most often summative Student-led пѓ� Students may assess self or peers пѓ� May be formative or summative пѓ� Can greatly relieve instructor burden пѓ� Students reinforce concepts through feedback process Examples: Online SelfAssessment пѓ� Self-tests: Use test tool to create self-tests (multiple choice, true false) пЃ¬ пЃ¬ May wish to track student efforts Can incentivize use (essentially, use as a learning tool) пѓ� Reflection papers: Have students submit brief, focused papers expressing the strengths and assessments of their assignment(s) Example: Online Peer Assessment (Formative) пѓ� Feedback groups: Assign students in groups to provide formative feedback on projects and papers пЃ¬ пЃ¬ пЃ¬ пЃ¬ Often raises quality of assignments Need a structure with clear deadlines Need prompts and models to guide students May wish to assess feedback process/contributions Example: Online Peer Assessment (Summative) пѓ� Conference Presentations: Have students вЂњpresentвЂќ their work and ask questions/provide feedback to others. пЃ¬ пЃ¬ пЃ¬ пЃ¬ In d-board, have students attach papers to messages; post a message with a synopsis; or attach a powerpoint presentation Each student/team should have their own thread Feedback should occur during a defined period of time. May consider allowing students to rate assignments on certain dimensions Public vs. Private Private пѓ� Work is submitted to the teacher only пѓ� Entire burden of feedback is on teacher пѓ� Important if assessing at fact level Public пѓ� Peers can see each othersвЂ™ work (either in process or completed) пѓ� Peers may comment on each othersвЂ™ work пѓ� Often increases quality of work submitted Process vs. Product пѓ� Product пЃ¬ пЃ¬ The end deliverable Look for polish, accuracy пѓ� Process пЃ¬ пЃ¬ How the student got there Look for thoughtfulness of approach, intent Assessing Process пѓ� Easy to do пЃ¬ пЃ¬ пѓ� Many technology tools will archive student work/interactions Students create a document trail in process Helps students develop metacognitive knowledge пЃ¬ пЃ¬ Instructors structure/model/encourage productive work processes Students learn how to manage their own work processes Why Assess Process? пѓ� For пЃ¬ пЃ¬ пЃ¬ the instructor вЂ¦ Provides formative feedback on course (e.g., helps gather data about why students have difficulty with product-oriented assessments) Clarifies who is doing most work in small group assignments Helps prevent cheating Why Assess Process? пѓ� For пЃ¬ пЃ¬ пЃ¬ пЃ¬ the student вЂ¦ Typically improves the quality of their products Helps them develop productive work processes Puts on a schedule Shows that you care about individual growth Assessment Project Cycle пѓ� From Classroom Assessment Techniques by Angelo & Cross (1993) пѓ� Step 1: Plan пЃ¬ пЃ¬ пЃ¬ Choose class Focus on assessable question Design project to answer question Assessment Project Cycle  пѓ� Step пЃ¬ пЃ¬ пЃ¬ Teach target lesson Collect assessment data Analyze data пѓ� Step пЃ¬ пЃ¬ пЃ¬ 2: Implement 3: Interpret results Communicate results Evaluate assessment project I. Term Papers пѓ� How пЃ¬ пЃ¬ пЃ¬ пЃ¬ to do online: Have students each start their own thread and post topic of interest Peers and instructors give feedback Students post thesis statements, research sources, etc., with iterations of feedback Final paper is posted Term Paper Assessments пѓ� Product: the paper пѓ� Process: quality and timeliness of student work from time when paper is assigned пѓ� Process: quality and timeliness of feedback provided to peers пѓ� Process: responsiveness to feedback received from instructor and peers II. Discussion Assignments 1. Chain of thought пЃ¬ пЃ¬ пЃ¬ Have students develop a solution to a problem Have students indicate what led them to a particular conclusion, method or approach Can be done in a discussion board Discussion Assignments 2. Theory to Practice пЃ¬ пЃ¬ пЃ¬ Have students match up theories you are learning about to actual problems Present students with problems and have them explain what theories they would use to solve these problems and how they would approach it Debrief the assignment Discussion Assignment 3. Synthesizer (i.e., wrapper) пЃ¬ пЃ¬ Have students take roles being the weekly synthesizer of class discussion Add a вЂњmetaвЂќ level in which students narrate their own experiences while reading the weekly discussion пЃ¬ Reflect on how life relates to discussion III. Group Projects пѓ� Tools used пЃ¬ Chat: brainstorming ideas, making group decisions, regular way to feel connected (should be archived) пЃ¬ пЃ¬ пЃ¬ пЃ¬ Discussion board: commenting on drafts E-mail: quick feedback File exchange: sharing project files MS Word: Track changes Group Project Assessments пѓ� Product: project files that are turned in пѓ� Process: online archive demonstrating пЃ¬ пЃ¬ пЃ¬ пЃ¬ Who contributed what Who provided peer feedback Who worked in a timely manner How collaborative a group was пѓ� Process: peer ratings пѓ� Process: interim instructor consultations Group Project Assignments 1. Work Plans пЃ¬ пЃ¬ пЃ¬ Have students develop a plan of work for their project Make them outline topic, schedule, resources needed, division of labor and anticipated form of final deliverables At end of project, have students evaluate how well they followed their own plan and how useful it was Project Assignments 2. Research Trail пЃ¬ пЃ¬ Have students document the steps they took in the research process and the results Ask for a brief reflection on how effective their process was and what they might change the next time Project Assignments 3. Process Presentations пЃ¬ пЃ¬ пЃ¬ Have students focus on their process as well as their product in class presentations To maintain focus, ask them to share 3 main lessons learned Might ask for some process documents to be shared, like an early draft Project Assignments 4. Design Journal пЃ¬ пЃ¬ пЃ¬ Have students maintain a journal of all ideas related to their project Encourage sketches, lists, organizational charts, etc. Require journals to be turned in with final projects IV. Reflection Assignments пѓ� Have students keep a weekly journal of their thoughts on readings and course content AND real-world related instances that they noticed пѓ� May make these public, with each student having their own discussion thread Making it Happen пѓ� Learners need to see that process is valuable: пЃ¬ Model appropriate processes пЃ¬ Provide students with scaffolding (guide sheets) to structure their processes пЃ¬ Give students feedback on their process пЃ¬ Require students to reflect on their processes пЃ¬ Grade students on process VanessaвЂ™s Top Time-Saving Tips Before you assign it, ask yourself вЂњcan I reasonably assess it?вЂќ пѓ� Rely on students/peers for providing some feedback пѓ� Let students know what to expect up front пѓ� Choose the right tool for the job пѓ� пЃ¬ пѓ� Get to know the editing and commenting features of your favorite programs Use rubrics! пЃ¬ I make them in Word, and then while grading I highlight or bold the section that applies to the studentвЂ™s projects. Add a few comments at the bottom = speedy grading!