Standard 2: Program Assessment and Unit Capacity
The unit has an assessment system that collects and analyzes data on the applicantís qualifications, candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs.
2.1.1 The Assessment System reflects the conceptual framework.
The Teacher Education Governance Unit (TEGU) has an assessment system that reflects the theme "Teachers As Reflective Decision-Makers", the conceptual model of "ARK", and the focus on the INTASC standards as the initial program goals. The Unit has also identified eight dispositions indicative of effective teaching professionals, and candidates are assessed on how well they exhibit these dispositions. The assessment system has evolved with the assistance and review of the Program and Policy Committee (P&P), the Teacher Education Administrative Council (TEAC), the Stakeholdersí Advisory Committee (SAC) and by all Teacher Education faculty and TEGUís partners during annual retreats. At these retreats, the assessment data is presented, discussed, and future objectives are considered in relation to the theme and model.
Each advanced program has agreed to follow the conceptual framework used by the initial programs but with the use of standards applicable to their special focus. Some of the programs have recently been developed and some have recently become a part of the Unit and as a result the assessment system is not as complete as the initial programs. Faculty members complete annual assessments that are sent to the Office of the Vice-President of Academic Affairs.
Meetings are underway to incorporate these new programs into the assessment system and to build a similar database to assess the candidates, the faculty and the program as a whole. Several of the advanced programs hold current national accreditations that require specific assessment procedures.
2.1.2 The Assessment system incorporates candidate proficiencies outlined in professional and state standards.
TEGUís assessment system is designed to gather data on the qualifications of candidates in the BSE programs, on the Unit faculty, and on the Unitís effectiveness. As soon as an individual registers for a core course a file is opened for them in the Unitís databank. Data collected for each individual include: demographic data, a copy of their transcript (added when students are admitted to the program), identification of a teaching major, demographic data including age, ethnicity, performance assessments based on INTASC standards, self-assessments of the INTASC standards, self-assessments of the teaching dispositions, evaluations of student teaching by the cooperating teacher and the university supervisor, and assessments of teaching abilities during their initial years of teaching in P-12 classrooms. At the initial level candidate proficiencies are assessed using INTASC standards and performance based-assessments.
Each course provides proficiency scores for candidates on the INTASC Standard (or Standards) that the course most clearly addresses. These data are based on authentic performances by the candidates. As the instructor submits the INTASC data, they also identify the performances used to make the assessments. See Appendix B, page 92 & 93, for a sample of how faculty report INTASC data by class.
In the advanced programs the standards that guide each programís purpose also form the basis for measuring candidateís proficiencies. Each advanced program conducts its own assessment in accordance with its applicable standards as illustrated in the Conceptual Framework for Advanced programs. The standards set by the state and the respective professional organizations guide the program design and the assessments of candidate performance.
2.1.3 The assessment system continuously examines the validity and utility of the data produced.
Each year a summary of all data is presented to the Teacher Education faculty and the Unitís school partners at a daylong retreat. During the retreat, faculty and partners review these data and the assessment system itself. They decide if there is need for change in the program, the assessment system, or both. The validity and utility of the data are discussed in this process. When issues are identified during a retreat, the Program and Policy Committee (P&P) formulates the appropriate solutions so that these issues can be resolved. P&P subsequently makes recommendations to TEAC to approve any changes formally.
New information is added to the database each semester. Faculty submits assessment data based upon candidates completion of performance-based assignments. Faculty assesses all candidates using a common rubric adapted from Wiggins and McTigheís material on Understanding by Design (Wiggins, G. & McTighe, J., 1999).
Data represent both the candidateís understanding of the particular Standard as well as their ability to apply this understanding in actual teaching situations. Candidates must be able to perform with students as the ultimate goal of the program.
As candidates are assessed, faculty uses a four-point rubric to report their proficiency to TEAC. The performance-based activities of each course constitute the basis for judgment. Following is a clarification of the four-point rubric.
Assessment Rubric for Unit Standards
|Shows a sophisticated understanding of the relevant ideas or processes. The concepts, evidence, arguments, qualifications made, questions posed, and methods used are advanced, going well beyond the grasp of the subject typically found at this age level.||The Performance or product is highly effective. The ideas are presented in an engaging, polished, clear, and thorough manner, and are mindful of the audience, context, and purpose. The final product or performance shows high quality craftsmanship.|
|Shows a solid understanding of the relevant ideas or processes. The concepts, evidence, arguments, and methods used are appropriate for addressing the issues or problems. Response shows no misunderstandings of key ideas or overly simplistic approaches.||The performance or product is effective. The ideas are presented in a clear and thorough manner, showing awareness of the audience, context, and purpose.|
|Shows a somewhat naïve or limited understanding of the relevant ideas or processes. The concepts, evidence, arguments, and methods used are somewhat simple, crude, or inadequate for addressing the issues or problems. Response may reveal some misunderstandings of key ideas or methods.||The performance or product is somewhat effective. Some problems with clarity, thoroughness, delivery, and polish are evident. It is unclear whether the audience, context, and purpose have been considered.|
|Shows little apparent understanding of the relevant ideas or processes. The concepts, evidence, arguments, and methods used are inadequate for addressing the issues or problems. Response reveals major misunderstandings of key ideas or methods.||The performance or product is ineffective. One of two situations is evident: The performance is unpolished, providing little evidence of planning, practice, and consideration of purpose and audience; or the presentation is so unclear and confusing that the key points are difficult to determine.|
For the annual retreat each department with teaching majors or advanced programs also reviews their programs in terms of changes in state accreditation standards and share any changes within their program. While the annual retreat focuses on a review of the data collected for admission, retention, and exit requirements as well as INTASC data, the agenda may also include issues relative to recruitment and retention of candidates at the university, professional development of Teacher Education faculty, and any policies or procedures that require input from faculty and the Unitís P-12 partners.
Graduate program faculty is responsible to assess their programs and candidates using the Unitís conceptual framework and the standards they use to guide their programs. If the program seeks a national accreditation, then their assessments are geared to that end. Again, several of our programs are new to the Unit and are only now designing their assessment systems.
2.1.5 Decisions about candidates are based on multiple assessments at multiple points.
When a candidate applies for admission to Teacher Education they are required to submit a packet that shows evidence of the candidateís academic progress, evidence of experience working with children, evidence of basic literacy and numerical skills (PPST), evidence of an assessment of speech and hearing, a minimum GPA of 2.5 and overall, a minimum GPA of 2.5 in the Communications block, and a minimum GPA of 2.5 in their major. A summary of each candidateís application packet is presented to the Teacher Education Administrative Council (TEAC) who are charged with the responsibility of insuring that all candidates are qualified for admission. See Appendix A (page 89) for a copy of the admission packet used by the Unit.
The next formal checkpoint for teacher education candidates comes as they apply for student teaching. In the application for student teaching a candidateís academic performance is reviewed and they complete a self-assessment of the INTASC standards and the eight teaching dispositions using the same rubric used by the Teacher Education faculty. (See Student Teaching Handbook)
The final checkpoint for the Unitís candidates comes in the completion of student teaching, the application for graduation and recommendation for licensure. The candidate must meet all of the exit requirements, including a minimum 2.5 GPA overall, in the Teacher Education sequence, and in the major. When all program requirements are satisfied the Unit will recommend that the Education Standards and Practices Board (ESPB) grant the candidate initial licensure. In the Student Teaching Handbook copies of the assessment forms used in student teaching and the forms used by cooperating teachers to assess our program are included.
The last entries in the Unitís database are gathered as the graduates complete a program assessment at the end of their first year of teaching. Since not all of the Unitís graduates are immediately employed in the teaching field, we follow these graduates for maximum of three years and attempt to gather data and at the end of their first year of teaching. Graduates are asked to complete the same self-assessment form that they complete at earlier phases of the program. The forms used to collect data from the graduates are the same used by candidates as they complete their self-assessments throughout the program, although the data collection may involve a telephone interviewer completing the form from answers provided by the graduates.
As explained previously, each individual who applies for a graduate program must meet the minimal entrance requirements set by the Graduate School, plus additional requirements set by the respective programs (See Graduate Catalog). As a candidate progresses through the advanced programs they have regular checkpoints that require information to be submitted to the Graduate School Office.
At the advanced level each program collects data based upon the professional standards they use as guides. Those who seek licensure are required to provide data in accordance with the accrediting body and their reports are on file with the Unit and will be available in the evidence room for the team members to peruse. The Education Specialist in School Psychology, the Master of Science in Communication Disorders, and Master of Music Education are examples of these programs and their institutional reports for their respective accreditations will be on file in the evidence room.
Programs that do not have national accreditation must submit an annual assessment report to the university. They must identify their learning outcomes and provide data on how these outcomes are met. Some advanced programs are funded by grants and are required by the funding agencies to show the accomplishment of outcomes identified in the grant. For example, the MAT in Mathematics is such a program whose annual report will be placed in the evidence room. Three of the programs that will be reviewed are new to the assessment process and, while they provide an annual assessment, they are still developing and revising their assessment systems to meet the Unitís conceptual framework to the fullest extent. The Master of Education is a new program and new concentrations are being added each semester; the development of their assessment system is still in process.
2.1.6 Data show a strong relationship between candidate performance assessments and candidateís success.
As data from candidates currently in the program are compared with data from the same candidates as they complete their initial teaching assignments, TEGU will be able to look for correlations between candidate performance on program assessments and candidate assessments during teaching. Initial comparisons with elementary education candidates, the only group for which data is currently available, showed a very high correlation between INTASC ratings and candidate success as rated by cooperating teachers and university supervisors.
Advanced programs that prepare candidates for licensure in professional areas require their students to pass national assessments and/or to complete internships that allow faculty to compare internal data with data from external sources such as clinical supervisors in the field.
2.1.7. The unit conducts studies to establish fairness, accuracy, and consistency of its performance assessment procedures.
The faculty review Unit data at their annual retreat and examine its fairness, accuracy, and consistency. Several changes in the assessment system may be traced back to the facultyís examination of data during the annual retreat. One example is the use of the portfolio as an exit requirement and as a source of data on how well the candidates performed. The data extracted was subjective and inconsistent; the workload in both monitoring and correcting the portfolios was untenable in larger departments. As a result, the Unit in its annual retreat discussed alternative ways to collect meaningful data. TEGU decided to move to the collection of more objective data measured by authentic assessments throughout the course of study.
The annual assessment reports that are to be provided the VPAA provide an avenue for faculty to consider their assessment procedures each year. Across the university an Assessment Day is provided each semester to allow faculty to study these issues in their programs.
2.1.8 The unit makes changes consistent with the study of data collected.
Initial and Advanced Programs
Comments and suggestions made by faculty at the annual retreat are the basis for many policy and procedural changes that are addressed by the Program and Policy Committee. P&P makes recommendations to TEAC on how to alleviate concerns expressed during the retreat. Departments can institute changes in their programs based on data they gather, but changes in Unit programs or coursework must be approved by TEAC, the University Curriculum Committee, and the MSU Faculty Senate. The elimination of the portfolio as an assessment tool is an example of this process.
2.2.1 The unit is implementing the assessment system and providing regular and comprehensive data on program quality, unit operations, and candidate performance, including first years of practice.
2.2.2 Data is based on multiple assessments from both internal and external sources.
Data are collected relative to each candidate with a focus on performance assessments for each of the INTASC standards. (See Appendix D, page 95) Additionally, a university assessment committee annually collects reports from each department. The university also publishes a Fact Book every year that provides data on administration, students, number of degrees conferred, faculty and staff, and finance. Below is a list of reports generated for the Teacher Education faculty and the Unitís P-12 partners at their fall 2003 retreat:
1. Summary of self-assessments by Teacher Education candidates on the INTASC standards and on Unit-defined dispositions. This summary is broken down into four categories: assessments and its mission, assessments at application for student teaching, assessments following the completion of student teaching, and assessments at the end of their first year of teaching.
2. Summary of candidates cited for inappropriate dispositions.
3. Summary of evidence of student learning.
4. Summary of diversity among candidates, among faculty, and among P-12 students encountered by Teacher Education candidates.
5. Summary of data collected by candidates applying for admission to Teacher Education. This includes a summary of GPAs, summary of dispositions reported by candidate references, summary of PPST scores, and a summary of demographic data.
6. Summary of candidate performance by INTASC Standards. Overall scores are reported plus subgroups by course, discipline, and by portion of the program completed.
7. Summary of student teaching final evaluations.
8. Summary of the cooperating teachersí program assessments.
9. Summary of the cooperating teachersí assessment of candidate dispositions.
Each of these summaries or reports is shared with Teacher Education faculty, members of the Stakeholders Advisory Committee, and university administration. During their annual retreat faculty and school partners review the data and discussion follows on whether these data seem to indicate a need for program change or an adjustment of the assessment system to gather more pertinent data. In some cases the faculty may request further analysis of data to determine if differences in scores are statistically significant. A summary of the data and suggestions for revision is written and shared with members of the local educational community.
In addition to the analysis carried out at the annual retreat, the database is used by faculty to help advise candidates and by candidates themselves to review progress through their program. Administration may also make use of the database when it is deemed necessary to review a candidateís record. As this database is maintained and expanded it will provide invaluable data to review factors relative to Unitís success in developing candidates who will help students learn.
Graduate program assessments are designed to meet the needs of the agencies that provide their accreditation. New programs with assessment systems still under development are expected to follow a model similar to that of the initial programs.
2.2.3 The unit maintains a record of formal candidate complaints and documentation of their resolution.
Initial and Advanced Programs
Teacher Education candidates use the university process if they have complaints or concerns about faculty, the program, or its delivery. The candidates follow the process outlined in the undergraduate and graduate catalogs that include an initial visit with the instructor to try to resolve the issue, then a visit with the chair or Dean, and finally an appeal process conducted by Student Rights Committee if a resolution is not reached earlier. Records of the complaints submitted and their resolution are kept by Dr. Richard Jenkins, Vice-President for Advancement and Student Affairs.
As an alternative, candidate complaints may be registered with a candidateís adviser or directly with TEAC, if it is a Unit matter. If a candidate complaint is not resolved during a visit to the advisor it will be referred to TEAC. The complaint and subsequent actions taken will be recorded in TEAC minutes. At the graduate level the candidate works through the Graduate School policies for these issues.
2.2.4 Data are regularly and systematically collected, and compiled, summarized, analyze, and reported publicly for the purpose of improving candidate performance, program quality, and unit operations.
Initial and Advanced Programs
As reported in 2.2.2, a summary of all data collected in TEGUís assessment system database is shared with all Teacher Education faculty and school partners at the annual retreat.
2.2.5 The unit is developing and testing different technologies to improve its assessment system.
Initial and Advanced Programs
The Unit has established a database using "Filemaker Pro" to collect information on each candidate, Teacher Education faculty member, and clinical teacher. The reason for this choice was the programís ability to allow secure online access to data, also allowing for online data collection.
The Office of Teacher Advisement and Field Placement also makes use of the database for placing practicum and student teachers and mail merge to communicate with prospective cooperating teachers and candidates while they are in the field. Minot Public Schools allow faculty to be a part of their "First Class" private email system for communication between school personnel and faculty. Plans are in place to make the database accessible to faculty and candidates and school partners through secure on-line connections
2.3.1 The unit has developed evaluations and continually searches for stronger relationships in the evaluations, revising both the underlying data systems and analytic techniques as necessary.
Initial and Advanced Programs
As discussed under Elements 1 and 2 of Standard 1 the Unit has described how data are collected and analyzed. Often faculty or stakeholders make suggestions that are discussed at the annual retreats and direction is then provided to the Program and Policy Committee who refines those ideas and recommends policy and procedural changes to TEAC for ratification. As the assessment system matures and the Unit is able to track candidates from entry through initial years of teaching it will be possible to analyze statistically the many variables involved and hopefully identify ways to strengthen our program even further.
2.3.2 The unit systematically studies the effects of any changes to the program to ensure that the intended program strengthening occurs and that there are no adverse consequences.
Initial and Advanced Programs
The Unitís annual retreat specifies that faculty review the impact of program changes. At this time most of the changes that faculty has recommended are included in the assessment system. The universityís requirement to submit annual assessments to the VPAA requires each program to review the impact of any changes made within their programs.
2.3.3 Candidates and faculty regularly review data on their performance and develop plans for improvement.
Initial and Advanced Programs
In the semester prior to the teamsí visit all candidates were given access to the database to allow these data to be used to complete their self-assessments. Advisors have also used the system for better academic advising. You may go to the following website to see a sample student file that would be typical of what a candidate or their advisor would see. http://www.minotstateu.edu/tegu/