NSF 1. What progress have you made toward addressing your center's research agenda?

The Technology Enhanced Learning in Science (TELS) Center creates resources so that the field can investigate the impact of technology on science learning and instruction. Since receiving funding in October, 2003, TELS has created resources for researchers, defined a research agenda, and conducted investigations to advance the agenda. Here we briefly summarize our resources and activities. The posters developed for the PI and AERA meetings provide more detail (See TELS AERA posters). TELS has created and tested center-wide resources including:

•        The TELS technologies. TELS merged WISE (from UC Berkeley) and Pedagogica (from The Concord Consortium) to offer the research community a platform for design, development and delivery of inquiry science curriculum that includes highly interactive dynamic visualizations and models. WISE supports the design and delivery of inquiry projects, scaffolding students and teachers, and collecting all data for purposes of assessment and analysis. Pedagogica takes advantage of Dynamica, BioLogica™, and Molecular Workbench to support development of richly interactive modeling and visualization activities and also logs students' actions within these environments.

•        TELS benchmark assessments and knowledge integration rubric. Starting in December, 2003, TELS surveyed teachers in partner schools to identify science topics that perplex students, align with standards, and could benefit from powerful visualizations and models. TELS selected 12 topics, 2 each from middle school life science, physical science, and earth science and from high school biology, chemistry, and physics. For each topic, TELS designed assessments that tapped the integrated understanding shown to lead to lifelong learning in prior research (Linn & Hsi, 2000). These assessments included items from prior research as well as items from standardized tests. TELS administered these assessments in all classrooms of 30 teachers in 9 participating schools. TELS created a knowledge integration scoring rubric to code responses to the items (see Figure 1.1).

•        TELS curriculum projects. At the Summer Retreat in 2004 12 multidisciplinary partnerships were formed to design inquiry activities for each of the 12 topics delivered using the TELS technologies. These partnerships reviewed student performance on the TELS benchmark assessments and designed activities intended to help students develop more coherent and linked ideas about the TELS topics (see Figure 1.2 for middle school projects and Figure 1.3 for high school projects).

•        TELS design principles database. TELS represented research on the design of projects to promote knowledge integration in a design principles database (see TELS 5). The database includes features gathered from over 50 technology-enhanced innovations, all the principles synthesized by Linn, Davis, & Bell (2004), and additional principles added by researchers participating in workshops and courses. The database serves as a community tool, supporting cumulative research in the field and aggregating knowledge about effective design.

•        TELS perspectives. To build on the research in the field, each of the 12 multidisciplinary partnerships reviewed the research on their topic and created a short perspective written for a general audience to identify the challenges of learning that topic, the common intuitions students bring to science class, any relevant prior research, and promising directions for design (see Figure 1.4 TELS Perspectives). The perspective also explained how the partnership made design decisions in authoring their inquiry project, and suggested effective ways to use the project.

•        TELS design review. To ensure that the TELS projects take advantage of research in the field as well as the collective expertise of the center, TELS created a design review process (see Figure 1.5 Design Review) (Holmes & Linn, 2005). Projects received multiple reviews at the summer retreat, were pilot tested and reviewed in the fall, and continue to be reviewed and revised as they are used by more and more teachers.

•        TELS collaborative courses. To build a research community among TELS postdoctoral scholars, fellows, and researchers, TELS offers annual collaborative courses. These courses, co-designed by leaders at two institutions, meet face-to-face at each site and on-line for discussions and project reviews. In addition, courses take advantage of teleconferencing technologies. In Spring 2004, TELS offered a course on Assessment and Technology and in Fall 2004 the course was on Metacognition and Technology (see http://www.telscenter.org/grad_opp/gradopp_onlinecourses.html for syllabi and other materials).

•        TELS professional development. To support teachers in their use of TELS curriculum materials, TELS created a professional development program supported by activities delivered with TELS technologies. TELS builds on a professional development model developed to help teachers use WISE effectively (see Slotta, 2004; Higgins, 2005; Varma, 2005; NSF 3).

•        TELS knowledge integration framework. TELS has continued to refine and elaborate the knowledge integration framework (see Linn, in press) that grew out of research leading up to WISE (Linn & Hsi, 2000) and from research on WISE and its predecessors (Linn, Davis, & Bell, 2004). Linn & Eylon (in press) wrote a chapter for the Handbook of the American Psychological Association synthesizing the extensive research in science education and identifying 10 basic patterns that promote knowledge integration. As these publications illustrate, knowledge integration connects the developmental, sociocultural, cognitive, and constructivist research traditions to offer pragmatic guidance for science education researchers.

•         TELS research agenda. In a series of collaborative activities TELS participants have identified six research themes and associated questions (see Figure 1.6 Research Themes). The research themes involve collaboration among TELS members from multiple universities, school districts, and research organizations. TELS has begun to explore the six research themes. To illustrate:

•        Science curriculum design. TELS technologies enable researchers to make precise comparisons between alternative versions of instructional materials. To explore knowledge integration instruction Richland, et al. (in press) shows that students learn more when asked to integrate ideas from two distinct topics than when asked to generate ideas for just one topic and that students who integrate across topics perform better on both single and multiple integration test items, suggesting the benefit of knowledge integration instruction even when the outcome measure is a standardized test with primarily items on single concepts.

•        Scientific modeling and visualization. Teachers, administrators, and students universally agree that modeling and visualization are essential aspects of scientific literacy. Yet students grapple with models and visualizations often finding them difficult to understand and epistemologically complex. Research on Airbags (McElhaney, 2005) suggests that students need instruction that enables them to understand the strengths and limitations of models in the discipline they are studying.

•        Knowledge integration assessment. Research on the knowledge integration items shows that items scored using the knowledge integration rubric form scales that meet all the criteria for IRT models (see Liu et al., 2005; TELS 1). Establishing consistent norms for knowledge integration across topics strengthens TELS assessments and has implications for standardized test development.

•        Professional and leadership development. TELS has successfully prepared 40 teachers to use technology-enhanced materials. Teachers report satisfaction with the instruction and frustration with the under-supported technology environments at their schools (see Varma, 2005; NSF 3). The 13 principals of TELS partner schools report support for TELS but limited resources for technology. These findings underscore the importance of working with the policy community.

•        Science learning for diverse contexts and populations. TELS schools serve diverse learners who face economic and personal challenges. In many TELS schools few students go home at night to speak English (see Figure 1.7 TELS School Partners). In spite of these obstacles TELS projects have succeeded in improving knowledge integration in each implementation. In addition, studies of the success of students from each quartile of the class indicate that students progress equally (see Chiu, 2005; Tate, 2005).

•        Partnership research and methods. TELS has developed a suite of community resources to support center partnerships as well as the field in general (Slotta, 2004). Studies of the perspectives (Hoadley & Rajan Sockman, 2005), the design review process (Holmes & Linn, 2005), and the design principles database (Ronen-Fhurmann-Fuhrmann & Kali, 2005) indicate promising impacts on the community. These studies also suggest ways to make the resources more useful. TELS is beginning to synthesize the research methods that help take advantage of the resources, partnerships, and new communication channels and will report on this work in the future.

In summary, TELS has embarked on a collaborative and comprehensive research program (see NSF 6). Only with the dedicated, collaborative, and consistent contributions of each TELS participant has all this progress been possible. It is a pleasure to work with such an energetic, flexible, and creative group of people. We hope that this enthusiasm is infectious.

NSF 2. What progress have you made in developing the leadership cadre in your field?

a. Recruitment of a diverse group of doctoral students

TELS has recruited as Fellows fifteen doctoral students and five postdoctoral scholars. Twenty percent of the Fellows are males, 20% are African American, 60% Caucasian, and 20% Asian (see figure TELS Doctoral and Postdoctoral Fellow Diversity). As a group, the Fellows hold undergraduate degrees in Science Education, Biology, English and Philosophy, Psychology, Engineering (Product Design, Electrical), Materials Science, Physics, Astronomy, Urban Planning, Language and Culture, and Information Systems Management.

Figure TELS Doctoral and postdoctoral fellow diversity.

		Gender		Ethnicity
	TELS Partner	Male	Female	African American	White	Asian	Asian American
Doctoral Students	UCB, ASU, PSU, Mills, Technion	3	12	2	10	1	2
Post Doctoral Scholar	UCB, Concord Consortium	1	4	2	2	1
Total		4	16	4	12	2	2

b. Providing appropriate research opportunities for these students

Both the graduate students and the post-docs were given responsibility for TELS activities early in TELS' first year, and both groups have already made significant contributions to educational research. These contributions were effectively represented at a poster session recently at the annual TELS meeting at AERA in Montreal.

Doctoral and post doctoral students conducted research on projects, professional development, and systems. Fellows in the project research track identified a specific area of science curriculum and designed a project to assess and improve student understanding. They developed content in the WISE environment, often inserting Pedagogica-based steps into projects. They worked closely with technology developers to create simulations and activities that matched their pedagogical needs. The fellows worked with teachers to help them incorporate projects into their curriculum. Each worked with two to four teachers, including teachers in geographically remote locations. They gathered data from classroom runs, conducted teacher interviews, and analyzed and reported on the data resulting from these activities. Fellows engaged in professional development and systems research used surveys and interviews with teachers and administrators as their instruments for data collection.

When surveyed about their roles, doctoral and post doctoral fellows responded with the following:

"As a first year Ph.D. student in TELS, I have had the opportunity to conduct research early in my graduate career. I have been able to perform this research in a largely self-guided capacity, forming my own research questions and making decisions about curriculum and methodology." (Doctoral Student)

"In terms of research, we, myself and colleagues at Mills, designed and are now conducting a study with all thirteen of the TELS principals. I am learning how to design a study appropriate to our research agenda, learning data collection and analysis strategies, and am beginning to learn how to write for publication. I am continuously reminded to think about how our research piece at Mills connects to the larger TELS research agenda to provide a more holistic view of considerations for technology enhanced learning in science." (Doctoral Student)

"I lead the implementation of the middle school physics force and motion curriculum, "Hanging with Friends, Velocity Style!" Using the preliminary curriculum designs from the retreat and my research questions as a guide, I authored most of this TELS project.  I coordinated the efforts and feedback of researchers, teachers, and technology developers. The unit has undergone evidence-based revisions and has been run in several classrooms.  I was able to demonstrate leadership in my research activities—development and refinement of research questions, assessment items, data analysis, and write-up." (Doctoral Student)

"I am directing professional development efforts for school district partnerships in Durham, North Carolina and Norfolk, Virginia. This involves organizing, supervising, and leading professional development activities—including mentoring middle and high school science teachers, designing an online TELS Teacher Professional Development project, and implementing ongoing teacher training workshops." (Post Doctoral Scholar)

"Currently working on multiple projects including:

•   Extending KI to other constructs such as epistemological beliefs about science and learning.

•   Connecting three different assessment data sources (embedded assessments, pre/post tests, and benchmark assessments).

•   Comparing students' KI over different grade levels.

•   Developing a framework to document longitudinal trajectories of student learning." (Post Doctoral Scholar)

"I have been given leadership opportunities in curriculum development, research and administration. I was immediately given a leadership role in curriculum development as a designer of a middle school unit about global climate change. I have developed pre- and post- student assessments and helped design research projects to measure the effectiveness of the unit." (Post Doctoral Scholar)

c. Offering specific experiences to develop leadership potential

Throughout the research process we have been sensitive to the need to challenge students at both levels to assume appropriate leadership roles, while also ensuring that they receive the level of assistance that they need in carrying out these roles. Opportunities for leadership have been created in several areas:

• creation and pursuit of an original research agenda

• design of curriculum and technology tools

• implementation of an inquiry project in science classrooms

• acquisition, and analysis of data from a project

• communication of research results to colleagues and to the community at large.

Comments from doctoral students and postdoctoral scholars testify to the value of specific TELS experiences that help develop leadership potential:

"The TELS fellowship has enabled me to take multiple positions of leadership. I was able to develop a high school chemistry curriculum and TELS allowed me to take leadership of my own research agenda." (Doctoral Student)

"The leadership opportunities are one of the most important aspects of being a TELS fellow—a rare opportunity to get to know people with a variety of areas of expertise: teachers, researchers, designers, and educational technology experts and leaders of the field." (Doctoral Student)

"My leadership capacity has developed in terms of my ability to be a visionary, adapt to the demands, needs, and interests of our subjects, and contribute to our research interests as a team player... I have had opportunities to create research goals, design investigative strategies, share research tasks, and receive and offer constructive criticism and feedback concerning my reactions, insights, and uncertainties about our work. I have enjoyed every minute of my experience as a TELS fellow so far." (Doctoral Student)

"The learning structure joined with meaningful research, collaboration in community, and envisioning future steps are necessary proficiencies for leadership. I am learning these skills through the TELS fellow program. Armed with experience, I hope to use the skills to propel positive change in education's future." (Doctoral Student)

"My leadership opportunities came while I was collaborating with assessment scholars in TELS. I facilitated the design, administration, and coding of the first year benchmark assessments, developed a construct for knowledge integration by connecting learning theories, TELS curriculum materials, and assessment methods, and helped perform statistical analyses on benchmark assessment items based on item response modeling." (Post Doctoral Scholar)

"I have had opportunities to design and conduct studies with leaders in science education research and also to share my knowledge and experience with the TELS fellows—served as a mentor for graduate students as they design and implement research studies; lead the process for designing and enacting the TELS model for professional development; designed and implemented research on the model, including developing an interview protocol, classroom observation rubric as measures of the model's effectiveness." (Post Doctoral Scholar)

NSF 3. What effect is your center having on the community of practitioners (teachers, administrators, etc.) with whom you are working?

TELS curriculum projects typically require 5 or more class periods of computer-based activity, with new pedagogical practices for teachers. Most teachers require practice and mentoring to learn how to monitor and scaffold their students in a technology enhanced inquiry curriculum, surveying ideas then periodically addressing the whole class to help guide its overall progress. The design of TELS inquiry projects is nuanced, blending the delivery of content with challenging collaborative inquiry activities that emphasize modeling and investigation.

The first year of classroom trials provided the opportunity to initiate a program of professional development that engages teachers, mentors and administrators in an exchange of ideas and approaches. The program was informed by earlier designs of mentored professional development (Slotta, 2004; Linn, in press) where teachers are guided through milestones of planning, enactment and refinement of the curriculum. Using the evidence-based approach teachers and mentors evaluated evidence of student performance on assessments to establish learning goals for their students and to customize instruction. The mentor played a vital role in helping teachers plan and reflect, and in guiding their use of the technology. The goals of the model are to promote autonomy within a small cohort of teachers using TELS curriculum in any school setting and to establish a sustainable community within that school, leading to repeated cycles of planning, enactment and evidence-based reflection.

The TELS professional development model focuses on mentor supported project enactment. As projects are enacted, professional development mentors and teachers engage in several cycles of planning, enactment and reflection activities. The main goal of the TELS professional development model is to provide support for teachers' knowledge integration as they enact the TELS technologies in their classrooms. The TELS professional development model helps teachers take full advantage of technological tools embedded in proven, powerful, science curricula. This year we focused on documenting the experiences of the participating teachers. Data collected about their experiences will inform revision of the model.

All TELS teachers participated in professional development and taught at least 1 TELS project. To date, thirteen teachers have participated in a structured interview or, in cases where an interview was not possible, responded to a written survey version of the interview questions.

Overall, interview responses show that TELS teachers are excited to have the opportunity to use a new technology in their classrooms, and expressed a desire to extend their experiences with TELS units. Sample comments:

-        The opportunity to use computer based learning was exciting, fascinating, and truly rewarding. It was great to see the students involved in their own learning and engaged in topic. This is the type of program I am more interesting in running in my classroom more often, as long as I have the technology.

-        I was very impressed with the students' progress from beginning to end of the project. I was glad to see their knowledge increase.

-        The graphs were difficult for students to understand. I am not sure if it is a developmental issue or not. I found it valuable to have them sit there and figure it out.

-        (One student) wrote that the position about where you're sitting in the car determines the safety of it (airbags). And I was like, "Yeah!" So that was really what I wanted them to get. Plus sort of a change of pace and a different you know, enthusiasm.

Initial analyses of the post-enactment interviews show that TELS Teachers are able to:

• Use technology in an innovative way for teaching science

• Experience new teaching practices

• Reflect on student learning in a technology enhanced environment, and

• Reflect on the impact of the technology on their teaching.

Figure Teacher Response to TELS. Brief summaries of the responses to some of the interview questions reveal how participating in TELS impacts the community of middle and high school teachers.

Question: What was the most successful aspect of the project?	-        Having student using the graphs and having to interpret the graphs was really good. -                 I liked the focus on writing in science. I could see an increasing amount of descriptive words that they used in their writing. -                 One thing that I did see was students actually getting engaged and having discussions and being involved in the content, which was really cool. Because this class especially was having some trouble being engaged by content, and that was one of the reasons why I chose this particular class because they needed a change. -                 They write these sentences explaining what they see, and I think that was fantastic. That was the best part of it.
Question: How did using the technology in your class influence your teaching practice?	-                 It allowed the students to go at their own pace and to have direct instruction on what they needed help with. -                 It (the TELS lesson) just helps me to bring different models to them and show them that they have access to stuff like this outside of their classroom. It's not just in class, and it's not just in the textbook. Lots of sources, so I think that was a good thing. -                 It (the TELS lesson) helps with group work and asking their peers for help instead of constantly asking the teacher. -                 I was able to back out, watch my students read, participate, engage, confer with each other and connect to concepts to their own understanding.
Question: What was the most frustrating aspect of the project?	-                 In our department we're really focused on coverage, and we have to get through all these topics. So, I have this pressure to cover all this material. So, I felt like sort of impatient. I'm spending too much time on this. -                 I found that when students were not all progressing at the same pace and at different places in the activity, I had to keep going back to grade or always checking and re-checking what I had already graded.

TELS Teaching Certificate

The TELS certificate is currently under development by TELS leadership and professional development researchers, in partnership with administrators from TELS school districts. Initially a California version of the certificate that meets California HOUSSE requirements will be developed (see http://www.cde.ca.gov/nclb/sr/tq/documents/nclbresourceguide.pdf). The TELS certificate will focus on the need for teachers to demonstrate "Standards aligned professional development" and "leadership and service to the profession" (HOUSSE - Part 1: Assessment of Qualifications and Experiences). These assessments are targeted at "not new" teachers who must demonstrate professional leadership and expertise. The TELS certificate will also target the requirements of HOUSSE - Part 2 ("Assessment of Current Qualifications through Classroom Observation and Portfolio Development").

70% of the teachers interviewed expressed an interest in pursuing TELS certification. Example reflections are below:

-     I included my involvement in TELS as one of my portfolio entries (for National Board Certification). I am just scratching the surface on this project. I could see how a certification program could be helpful for newer and more experienced teachers. Younger teachers would get more experience teaching science content in innovative ways. More experienced teachers would get more support in the technology end of things. I don't know how (TELS) proficient I want to be, but there are areas like understanding the technology behind the projects and understanding the teacher tools (that would be useful).

-     I would love to learn to do these sorts of interactive programs for my student. Seeing them engaged and having fun learning only reinforced my desire to provide these opportunities for my students. I would not only create projects myself, but I would have students creating projects as well. I just need to know how to do some of the neat interactive activities. I would very much like to participate in a certificate project.

NSF 4. What are the contributions being made to the center by STEM disciplinary faculty (or non-faculty researchers) and their departments?

TELS has assembled an interdisciplinary membership that includes STEM faculty, post-doctoral scholars and researchers who participate in the design of materials, advise us on content-related issues, and collaborate in joint proposals (see figure TELS STEM Contributors)

Our center is strengthened by its inclusion of discipline experts for each domain of science, whose role is to participate in the design of TELS curriculum and assessments. Several of these scientists attended the summer retreat where they consulted with TELS investigators, graduate fellows, and science teachers concerning science issues.

To encourage disciplinary involvement, we have presented at the APS, ACS, and AAAS meetings during the last year. A technical paper on the inclusion of chemical bonds in molecular dynamics has been accepted for publication in the Journal of Chemical Education, which has a disciplinary readership.

STEM contributors in the Physical Sciences

We are fortunate that the TELS PI and co-PI from Concord bring a wealth of knowledge and experience. Dr. Paul Horwitz and Dr. Robert Tinker, both hold doctorates in physics. Dr. Horwitz's expertise is in elementary particle theory, atomic physics, and laser-induced isotope separation; Dr. Tinker is an experimentalist and an expert in low-temperature physics. Each has substantial research experience in physics, as well as in physics education, which is a rare combination that allows them to contribute sophisticated ideas to the design of TELS materials as well as to the research program.

TELS also benefits from the contributions of physicist Eugene Stanley, University Professor at Boston University and Director of BU's Center for Polymer Studies. Professor Stanley was involved with the inception of our center and supervised the scientific research of doctoral student Alfonso Lam, a member of the first cohort of TELS graduate students.

John Belcher, Professor of Astrophysics in the Physics Department of MIT, runs the TEAL/Studio Physics project where he offers a technology-rich version of introductory Electricity and Magnetism (See: http://web.mit.edu/8.02t/www/802TEAL3D/teal_tour.htm). The visualizations he uses in his teaching are legendary (See http://jlearn.mit.edu/museum_images/visphysics.htm) but he needs a way to guide student interactions with these and mechanisms for embedded assessment. We are working with him to do this by embedding his software in WISE/Pedagogica/SAIL.

STEM Contributors in the Life Sciences

TELS benefits from the participation of Gail Hollowell, a professor at North Carolina Central University with a background in molecular biology. Hollowell currently works on educational research progress, and is one of the TELS leaders from NCCU. She has advised our designs of life science curriculum and assessments at the middle and high school levels, and offers important insight concerning the design of materials for diverse student populations including those served by NCCU, such as Durham Public Schools.

TELS is also well advised by Dr. Elisa Stone, who received her PhD in molecular biology from The State University of New York, Stony Brook. Dr. Stone has recently completed a masters program in education at University of California, Berkeley, where she contributed to the preparation of TELS life science materials. She now continues to advise our designs of middle and high school curriculum in topics of meiosis and mitosis, and was a contributor to the TELS Perspective in life science conceptualizations.

Dagmar Ringe, Professor of Biochemistry, Chemistry, and Rosenstiel Basic Medical Sciences at Brandeis University, has consulted with Concord Consortium on design and functionality of the Molecular Workbench software tool, used by TELS in several of its projects. Her presence on the team has ensured that the model embedded in the software does not distort or overly simplify the real-world science.

Trish Morse, a biology professor the University of Washington, is a TELS advisor who has reviewed the life science curriculum and consulted with TELS graduate fellows concerning their designs.

Marsha Matyas, a physiologist from the American Physiology Society, has been an ongoing collaborator in the design and implementation of health science curriculum for middle school. Dr. Matyas has employed TELS projects in her own research with professional development for new physiology doctorates as well as K-21 teachers in topics of physiology.

STEM Collaborators in Earth Science

TELS is fortunate to have as co-PI Yael Kali, a well known earth science education researcher whose work emphasizes strong disciplinary content and who worked closely with the geoscientist Nir Orion. Kali has advised on all designs of earth science curriculum in TELS, particularly the rock cycle project for middle school, for which she designed several innovative resources.

Catherine Manduca is a professor of geology at Carlton College (Minnesota) who served as the outreach coordinator of DLESE (Digital Library of Earth Science Education), and currently directs the Science Education Resource Center. In collaboration with Manduca and David Mogk, geology professor at Montans State University, Jim Slotta has designed global climate change curriculum that served as a basis for that currently used within TELS. Manduca and Mogk have also collaborated with Slotta on several grant proposals to extend TELS to research in geoscience education, have organized several geoscience visualization workshops, and annual presentations at the American Geophysical Union meetings. Manduca, Mogk and Slotta co-organized an NSF-funded NAGT visualization workshop in the geosciences in 2004.

STEM Collaborators in Engineering

Ronald Miller, Professor of Chemical Engineering at Colorado School of Mines has collaborated with Jim Slotta in preparing a grant proposal (for NSF ROLE program) to apply TELS instructional principles to the design of educational materials for nanoscience concepts in engineering. Dr. Miller has observed that his students have great difficulty in understanding certain concepts that are concerned with the emergent behavior of complex dynamic systems, and proposes to design a new set of technology-enhanced lessons to help students overcome such barriers.

Michael Clancy, from the University of California School of Engineering and Computer Science, has collaborated with Marcia Linn and Jim Slotta in designing innovative applications of the TELS technology platform, and has contributed to the design of our Second Generation architecture. Clancy has collaborated in the preparation of several grant proposals recently to fund the design of introductory computer programming courses using the TELS technology platform. He is currently using the TELS First Generation software in his instruction of introductory computer science at University of California, Berkeley.

Alice Agogino, professor of mechanical engineering at University of California, Berkeley, has collaborated with Marcia Linn on numerous projects, and contributes expertise in the use of visualizations and the design of digital content. She has served as a TELS advisor, and has worked with one of the TELS graduate fellows in guiding the design of research materials for middle school physical science.

Figure STEM Contributors to TELS. Science, Technology, Mathematics, and Engineering Faculty who participate in TELS.

NSF 5. What information has your evaluation provided your center about impact, changes needed, and effectiveness? How have you used this information?

TELS has implemented a multi-layered evaluation system to gain insight from the TELS evaluators (Heller Research Associates), national and international leaders including the advisory board, and center participants, concerning our initiatives and direction. Heller Research Associates (HRA) provided feedback at the retreat, summarized the comments from the advisory board, interacted with TELS participants, and developed the TELS logic model (see Figure 5.1 TELS Logic Model). TELS has organized external reviews and also engaged each of its partners in reviewing Center activities (see Figure 5.2 TELS Timeline).

External Reviews

At the inaugural meeting at NCCU in Durham, NC December 2003 TELS brought together leaders from universities, research organizations, and school districts, and sought comments from the broad coalition of minority-serving institutions affiliated with NCCU. We identified the science topics that science teachers found (a) challenging for students, (b) amenable to technology enhancement, and (c) central to the curriculum. These topics became the focus of benchmark assessments, curriculum designs, and research investigations. TELS classroom teachers and administrators said they were committed to effective use of technology in their schools, eager to participate in the summer retreat, and concerned about professional development as well as technology support. This feedback alerted TELS to a major challenge — ensuring that teachers could use the technologies at their schools effectively.

In March 2004, TELS held a technology review at the Concord Consortium, where technology leaders presented their plans and received feedback from external reviewers Kate Bielaczyk, Jim Kaput, and Bob Panoff. These reviewers encouraged TELS to engage in a rapid development process and to use off-the-shelf materials whenever feasible.

In April, 2004, in conjunction with the American Educational Research Association (AERA) meeting, TELS held a review of the research plan. Roy Pea, the chair of the TELS Advisory Board, and Xiaodong Lin, a member of the TELS Advisory Board, responded to plans for investigating the impact of TELS curriculum designs on students and teachers. Lin and Pea endorsed the plan for benchmark assessments that could inform curriculum design. They also stressed the importance of keeping the research coherent and feasible.

At the summer retreat in 2004, TELS formed partnerships spanning all of its member institutions to design curriculum materials that address the 12 topics, and to plan research (see Figure 5.3 TELS Retreat). At the end of the retreat, the full Advisory Board joined us, listened to our progress to date, and provided us with extensive feedback. In addition, the TELS evaluator, HRA, provided feedback and suggestions during the retreat. For example, Heller observed small group interactions during curriculum design sessions and provided Center staff with information about the pace and content of the groups. After a few days' work, the groups were making steady but slow progress in planning their units. Heller suggested postponing the research discussion until more groups had developed the overall structure and content of their units. This recommendation was implemented.

At AERA 2005, we held a TELS-wide review. Prior to the meeting, each postdoctoral scholar, Fellow, and leader developed a poster summarizing their research over the past year (see TELS AERA 2005 Posters). Draft posters were developed by March 15, reviewed by 2 other TELS members, revised, and reviewed on March 21 at a TELS teleconference. At AERA participants presented the 22 posters and gave feedback to others. Roy Pea, the chair of the TELS advisory board, attended the session and made comments to individuals and to the group as a whole.

 

TELS Reviews

The TELS partners have carried out extensive evaluations of Center work. TELS established a design review process for curriculum materials that includes multiple forms of feedback (Holmes & Linn, 2005). TELS has designed benchmark, pretest, posttest, and embedded assessments for curriculum materials that help make comparisons across projects as well as contribute evidence to guide revision (Lee, 2005; TELS 1). TELS has studied the form and effectiveness of its communications and is using the information to refine the communication process (Hoadley & Rajan Sockman, 2005). TELS has also reviewed the professional development program in meetings with school leaders and by conducting interviews with participating teachers and administrators (Varma, 2005; Fauvre et al., 2005).

Using Evaluation Information

Feedback received in all these reviews has been constructive, supportive, and helpful. Reviewers encouraged TELS to develop high -level as well as detailed accounts of the vision and plan for the work. TELS has created a logic model (see Figure 5.1 TELS Logic Model) to provide a comprehensive perspective. Additionally, TELS has created visual, written, and web-based documents describing the Center model, philosophy, design process, and activities. These include perspectives (see TELS Perspectives and the TELS web site at http://www.telscenter.org/) that provide the design rationale for each TELS curriculum project, academic papers describing the framework, design process, and assessment model (see Figure 1.1 TELS Knowledge Integration Levels), presentations (see figure TELS Papers and Presentations), and curriculum units (see Linn & Slotta, 2005) as well as the answers to these questions. TELS has also developed the Design Principles Database to synthesize design knowledge and is adding current research findings to the corpus. We highlight several examples:

Curriculum Design Feedback. The project developed by TELS partnerships have received extensive review, most recently in classroom trials, and have been revised based on these reviews. A final test of the impact and effectiveness of these projects will occur when the benchmark assessments are administered to the current cohort of students and performance is compared to performance of the prior cohort. Reassuringly, each project that has been tested and assessed in classroom settings has demonstrated progress for student understanding of its main topics (see Figure 5.4 Project Assessments).

Professional Development Feedback. The TELS professional development activities have been extensively revised as summarized in the response to question NSF-3. Feedback from teachers and school leaders suggests that the TELS professional development efforts have been successful, albeit intense. Perhaps the major achievement from the first year of TELS implementation is our documentation and characterization of the difficulty of establishing a working and functioning technology environment in each of the TELS partner schools. The single biggest challenge faced by TELS school partners concerns technical support. Schools have computers, they have labs, they have networks, but they do not have technical support. Computers are often locked in closets. When they are available, they often lack the most recent software. Networks have arcane and bizarre firewalls that often prevent students form doing even the most rudimentary activities. Access and scheduling of technology for use by teachers is often poorly coordinated.

To respond to this challenge, TELS has put considerable human and financial resources into every one of its schools to ensure that the technological environment is functioning smoothly. This one-time effort has paid off and many teachers have run multiple TELS projects, once the technology environment was in place. TELS is working to establish a system to help new schools establish effective technology environments.

Each of the 13 principals involved in TELS, report enthusiasm for technology use in science and support for science education programs in their schools. Nevertheless, limited resources prevent them from providing sufficient technology support. They appreciate the assistance provided by TELS and have promised to diligently ensure that the environments we have installed remain operational. At the same time, the principals report that impending budget crises bode poorly for long-term support for technology at their schools. This lack of technology support in the nation's schools affects much more than just TELS. It is a major crisis and deserves national attention.

In summary, the TELS Center has benefited from extensive external feedback and these evaluative comments have been integrated into TELS planning and communication. The ability to take advantage of each partner's expertise and to learn from numerous cycles of feedback and refinement has resulted in improved research, professional development, curriculum materials, assessment models, communication strategies, and leadership opportunities for doctoral students and post-doctoral scholars as well as teachers and administrators.

NSF 6. What evidence do you have that your project is becoming a Center? (e.g., the degree to which the partners are coalescing around your CLT mission, policy changes and new structures that will persist, connections with other innovative projects — including other CLT's)

At the TELS meeting at AERA in April 2005, 22 posters created by over 35 TELS participants provided compelling evidence that TELS has become a center (see TELS AERA posters). As participants asked each other about families and research designs almost in the same breath the collegiality and the commitment to the goals of TELS were clear. Heated debates about interpreting student responses to models and disputes about the importance of specific representations for a discipline convinced observers that TELS has a progressing research program. When the group turned to planning the research program individuals generated stunning questions across the six themes. And, when each participant was asked to select just 2 of the 6 themes as their primary focus, many balked, saying they were working in at least 3 areas. The summary of the research questions in figure Research Themes lists the questions that emerged from the process.

The posters show that TELS participants are researching a cohesive and mutually supportive set of research themes, using outcome measures that enable researchers to carefully compare results. Participants at multiple institutions are exploring the same theme yet taking different perspectives. For example, researchers at Arizona State University, UC Berkeley, Concord Consortium, and Norfolk are all looking at the impact of powerful modeling environments using diverse populations, curriculum materials, and research methods.

The posters show that TELS participants are developing resources such as the professional development model, TELS technologies, and design principles database for the community. Each participant is also using two or more resources created by others (see TELS 1 for a description of the center-wide resources).

Each poster offers insights into the emergent TELS knowledge integration framework (Linn, in press). The knowledge integration construct, summarized in figure Knowledge Integration Levels, guides the analysis of student work, classroom assessments, and classroom observations. Research collaborators develop rubrics for new assessments using the framework. Designers report that they are finding patterns in their projects that promote linking and connecting of ideas.

Evidence for TELS as a center includes:

TELS Community and center-wide research agenda

•        The TELS activities (see Figure 5.2 TELS Timeline) have enabled participants to develop shared interests and research agendas. Participants at the 2004 retreat formed cohesive partnerships that have persisted beyond the retreat. Participants provide thoughtful, constructive feedback on ideas presented at meetings and for the poster review.

•        A series of research planning activities have identified 6 research themes associated with the overall TELS research question. Each of the research themes has primary participants from multiple TELS sites.

•        All the TELS fellows and postdoctoral scholars and most of the researchers participated in both of the TELS collaborative courses. Course activities shaped the research plans for the TELS fellows and postdoctoral scholars.

TELS Center-wide resources used by each partner institution.

As described in the response to NSF 1, all the TELS participants use the center's wide resources.

•        By merging the WISE and Pedagogica environments, TELS has created a flexible, shared platform that is currently in use by all the TELS partners and serves to link the 12 curriculum projects, enabling research designs to test ideas in multiple science disciplines.

•        The TELS design principles database enables researchers in TELS to compare and contrast features of the curriculum materials they design and to take advantage of designs created by others (see TELS 5).

•        The TELS Professional Development model, supported by an on-line instructional project, has been tested and refined in each participating TELS school. TELS researchers are learning how to customize professional development to school and teacher needs.

TELS Knowledge integration framework

•        Development of the TELS assessments, knowledge integration construct, and scoring rubrics, helped refine the framework.

•        The TELS professional development model implements the framework and connects to related work using the framework (Davis, 2003).

TELS Connections to other CLT funded projects.

•        CILS researchers are exploring the design principles database and the design patterns.

•        CCMS and TELS have held a joint meeting and identified common disciplinary topics and research questions.

•        CAESL and TELS have common interests in assessment research and are sharing a research fellow.

Views of TELS from the field

•        Groups have approached TELS to form collaborations. Many groups want to use the TELS technologies (see TELS 2). The Kaleidoscope Project of the European Union, for example, has met with TELS staff to discuss the possibility of adopting TELS' SAIL technology in order to ensure interoperability between software tools and components intended for educational use. The ability of technology to log students' actions and make inferences from them — a feature of both WISE and Pedagogica, and supported by SAIL — is increasingly valued not only by the research community but by teachers and administrators as well.

•        At the TELS symposium held at the AERA Annual Meeting in April, several groups expressed interest in using the TELS infrastructure .The Maine Mathematics and Science Alliance, in collaboration with the Maine Laptop Initiative discussed the possibility of using TELS for conducting large-scale educational experiments in a ubiquitous computing environment.

•        Jon Singer and Brianna Timmerman at the University of South Carolina are using the TELS BioLogica™ software in their research.

•        Peter Garik and his colleagues at Boston University used Pedagogica-based quantum mechanics activities in a 650-student freshman chemistry course in 2004, and they intend to do so again in Fall, 2005.

•        Countries around the world are creating versions of WISE in new languages. SAIL will support multiple languages and enable Pedagogica to use multiple languages.