TELS 1. The TELS project relies on baseline data so the effect of interventions can be measured and understood.  What is the status of that data collection? Since the types of interventions that can be investigated are limited by the parameters included in the baseline data collection, explain these constraints.

Overall TELS Assessment Research Goals

TELS Assessment research provides a framework to understand more effectively student science learning through the use of TELS materials. It involves the collection and interpretation of evidence in learning to (1) support curriculum design and instruction, (2) develop longitudinal trajectories and (3) conduct cross-sectional comparisons. In investigating the effects of interventions on student learning, TELS collects and analyzes assessment data from various sources including embedded assessments, pre/post tests, and annual benchmark tests.

The embedded notes and pre/post tests measure whether students acquire inquiry learning relevant to a particular science topic address in each TELS unit. The benchmark assessments include items from embedded notes, pre/post tests, and the previous year's benchmark assessments. TELS compares cohorts using item response modeling techniques.

TELS Benchmark Assessments 2004

The first year TELS benchmark assessments were administered in May through June in 2004 (see Figure 5.2 TELS Timeline) prior to TELS instruction. The benchmark assessments had six content versions: middle school physical science, life science and earth science; and high school physics, chemistry, and biology. Classes were tested on the content the teacher would be covering the following year. The tests in the 6 content areas included paper and pencil tests and technology-enhanced online tests. The paper and pencil benchmark assessments consisted of multiple choice and open-ended items from NAEP/TIMSS, and Knowledge Integration (KI) items previously used in related research projects or newly created for the benchmark assessments. We administered these tests to 1700 students in 9 schools, taught by 30 teachers.

Analysis of the paper and pencil benchmark tests yielded a construct that measures knowledge integration. Knowledge integration is a learning perspective that values the process of linking ideas as students continue to develop a more coherent and normative view on scientific phenomena. In addition, KI items and scoring rubrics were compared with multiple choice items and scoring rubrics from NAEP/TIMSS. Item response modeling analyses indicated that KI items and scoring rubrics differentiate student performance much more consistently than standardized items and scoring rubrics.

TELS Benchmark Assessments 2005

During the 2004-5 year period, TELS teachers implemented one or more TELS units in their science content area. This year's 6 content areas of benchmark assessments include selected embedded-note prompts from target TELS units, corresponding pre/post tests, and items from the previous year's assessments. To implement a more cost-effective testing procedure, significant portions of assessment data will be collected online. Assessment goals include: (1) establishing connections among embedded assessments, pre/post tests, and benchmark assessments, (2) analyzing assessment data based on IRT models to estimate relative standings of all administered items, and (3) comparing student performances by grade level, content area, and teacher implementation pattern.

Constraints and Opportunities

TELS assessment research is designed to explore potential constraints imposed by the kind of assessment practices implemented. Five such constraints and how we transform them into fruitful opportunities for curriculum design, instruction, and research follow:

Constraint 1: Measuring KI

TELS focuses on fostering the process of knowledge integration when students learn science. From the knowledge integration perspective, the science learner is viewed as having a repertoire of ideas, rather than a single notion concerning any scientific phenomenon. Effective science learning happens when instruction helps the student's process of linking, connecting, promoting, reconsidering, adding, and demoting ideas. In facilitating this knowledge integration process, TELS units introduce new ideas into the repertoire that stimulate the process of ideas bumping up against each other and raising issues about how disparate notions of a scientific phenomenon might fit together.

A suite of assessments used in TELS research is designed to measure knowledge integration. Aligning curriculum, instruction, and assessment with knowledge integration has several instructional benefits. First, assessment results can be directly used to inform the revision of TELS units and the teacher's instructional planning for the next activity in the same unit as well as for the next run of the TELS unit. By emphasizing knowledge integration, assessments can help teachers recognize what should be taught and learned.

Since knowledge integration can happen in many different forms, a question of whether knowledge integration is a single dimensional, measurable construct can be raised from a psychometric standpoint. On the other hand, breaking inquiry learning into a number of narrowly-defined constructs may not preserve the kind of scientific inquiry needed in solving real-world problems. We assume that knowledge integration is a fundamental cognitive process that is involved in any scope of inquiry activities such as modeling, formulating explanations, and conducting investigations. Therefore, important dimensions of TELS assessment research concern defining knowledge integration as a measurable construct, developing scoring rubrics, and generating KI item design principles.

Constraint 2: Selecting Items from Standardized Item Pools

TELS assessment research attempts to document the effects of technology-enhanced instruction on student learning through internal and external measures. We use internal measures to mean assessments that are closely related to the scientific topics and the type of inquiry thinking TELS units directly address. We also use external measures that appeared in influential standardized tests such as NAEP/TIMSS on the same science topics.

In selecting external measures from nationally released item pools, we need to address two issues. The first issue is related to content coverage. TELS identified 12 topics for middle and high school science and developed units around these topics. However, not all 12 topics are equally represented in the national item pools. In particular, few items exist to tap into real-world inquiry learning. The second issue concerns the lack of standardized items that can measure KI. When we analyzed the first year benchmark assessments, we found out that multiple choice items were not differentiating students as consistently and reliably as open-ended items with KI scoring rubrics. We also discovered that open-ended items from standardized tests can be used to measure KI when they were coded with the KI scoring method, instead of the suggested scoring rubrics by NAEP/TIMSS.

Constraint 3: Comparing with National Populations

We recognize that the best way of communicating the effects of TELS on student learning with educational stakeholders is to demonstrate how TELS students perform compared to national populations. We use Item Response Modeling techniques to estimate the psychometric parameters of instruction sensitive items used in embedded assessments and pre/post tests as compared to those of standardized test items. This allows us to compare the quality of KI items with that of standardized items. In addition, we recalibrate student performance based on his/her response pattern to the items. By using these recalibrated scores, we can compare TELS students with populations whose performance statistics were reported nationally.

Constraint 4: Developing Longitudinal Trajectories

We plan to develop longitudinal trajectories of students' knowledge integration in two ways. First, we compare students' knowledge integration levels on selected KI items over time to characterize the nature of integration in particular problem contexts. The improvement over time can indicate that certain revisions made to TELS units relate to students' increased knowledge integration. Second, we use recalibrated KI ability scores to follow the same cohorts of students who learn different TELS units from year to year. In this comparison, maintaining and tracking the same cohorts of students are difficult as students do not usually remain in the same classroom from year to year.

Constraint 5: Addressing Logistics of Assessment Administration

In 2004, we administered two different forms of benchmark assessments: paper and pencil tests and online tests. Though paper and pencil tests were much easier to administer and collect than online tests, they turned out to be costly as the assessment data had to be individually entered to the computerized database. In addition, relying too much on paper and pencil tests may not be an ideal way to measure student learning due to the use of learning technologies in TELS.

We plan to move away from the paper and pencil testing format to the online testing format. This year, we will use online benchmark assessments for most of the participants. However, we still have to administer paper and pencil tests to accommodate a small portion of students who cannot take benchmark tests online. The most common problem is related to the fact that the number of computers with reliable internet connections in schools is not sufficient enough to allow all students to take online tests simultaneously. In this case, we plan to assign randomly half of the students to online tests and the other half to paper and pencil tests. With these data, we can answer a question of whether two different test administration formats are comparable. We think that up to 80% of all TELS participants will take benchmark assessments online this year. Our expectation is that all benchmark assessments will be administered online by 2006.

TELS 2. The analogy of the national lab depends on a review board to screen for important ideas that can be tested in the lab.  What is the status of the national board?  Describe the process they have used or will be using.

Process and Board

TELS has established a process for working with national and international partners to build a cohesive community that conducts cumulative research on technology-enhanced learning in science. TELS' role as a national center involves providing resources for all researchers, forming partnerships funded by new proposals, and supporting a select set of partnerships that add innovations to the TELS technologies. TELS relies on the national advisory board to guide decisions about which partnerships to pursue. These decisions are informed by the criteria described in the TELS proposal. We seek partners developing research programs that are:

• Based on cognitive research findings and studies in diverse classrooms
• Address central topics in the science curriculum for which IT offers unique advantages.
• Consistent with the TELS research agenda and curriculum framework.
• Involve collaboration of two or more researchers.
• Investigate student learning in schools with diverse populations
• Implement professional development that builds on TELS research
• Assess inquiry using items that require integration of ideas and align with standards.
• Tailor TELS technologies to needs and opportunities of participating teachers.
• Communicate results to multiple audiences including minority serving institutions

Resources for all researchers

TELS is developing the resources outlined in NSF 1 to encourage researchers to investigate technology enhanced learning supported by models and visualizations. These resources such as the TELS technologies, TELS curriculum projects, and knowledge integration assessments give new investigators a head start on their research program. The resources enable researchers to test instructional ideas by customizing an existing curriculum project. Investigators can focus on new ways to teach a disciplinary topic and rely on TELS professional development to communicate with teachers. Those interested in research synthesis or community building can use the design principles database. Spontaneous and continuing users of TELS technologies include:

• Graduate students at universities around the world who use TELS resources for their research.
• Scott McDonald, an assistant professor at Pennsylvania State University who created TELS curriculum materials for use by members of a nanotechnology center.
• Science education faculty members who use TELS curriculum designs in preservice courses.
• Doris Jorde, a researcher in Norway, who uses TELS resources for research on science learning.
• Teachers around the world who access TELS curriculum materials on the web and use them in their classrooms.

Collaboration on new proposals

TELS resources can contribute to proposals from new institutions and research groups. These collaborations can extend the community and create new technologies. Often these collaborations result from presentations to new groups interested in science education (see TELS Papers and Presentations). Some recent examples of projects that have been funded or submitted for funding include:

• Tinker has initiated a series of collaborations with nanoscientists interested in modeling these processes. The preliminary models (See http://www.telscenter.org/research/research%20update/perspectives_newstyle/NanoPersp.html) were presented by Linn at a materials science conference in Doha, Qatar recently. Researchers expressed interest in creating an Arabic version of WISE and using the models in introductory engineering courses.
• UCLA included TELS as a partner in a Science of Learning Center proposal.
• Linn collaborates with Robert Bjork at UCLA on a project submitted to IES to explore connections between laboratory learning studies and classroom contexts. TELS technologies enable implementation of comparison studies in the same classroom.
• Slotta collaborates with Professor Ron Miller from the Colorado School of Mines on an NSF grant proposal to the ROLE program to study students' learning of advanced engineering concepts in nanotechnology using TELS technologies.
• Marsha Matyas from the American Physiology Society has recently won a grant from the National Institute of General Medical Sciences (NIGMS) to develop an online interactive professional development community for minority trainees in physiology and plans to use TELS technology resources.
• The Networked Collaborative Inquiry Learning (NetCoIL) project was recently funded by the German DFG to enable international collaborations between three technology oriented research projects: Co-Lab (Thorsten Bell, Ulrich Bosler, and Ton deJong), TELS (Jim Slotta and Paul Horwitz), CoOL Modes (Andreas Harrer, Niels Pinkwart).

Support for new innovations

Embedding Pedagogica-based activities (with rich, interactive models – see TELS 3) within a WISE project provided a test case for the eventual scenario where TELS would support the integration of a wide range of technology approaches. Development of the first generation solution has motivated TELS technologists to define a comprehensive framework for the integration of technologies (Slotta, 2004; Slotta, Aleahmad, Zimmerman, 2005). This new framework, known as the Scalable Architecture for Interactive Learning (SAIL), will support a more seamless integration of WISE and Pedagogica and enable the integration of additional innovations, resulting in an increasing level of integration among the various TELS technology components. The technology framework will support connections between TELS technology and curriculum materials and important standards for digital content (e.g., SCORM) and eLearning systems (e.g., OKI, LD). SAIL (described in TELS 3) will provide a common technology framework for numerous research projects, allowing innovations to be easily exchanged across projects or compared within investigations (e.g., using two different versions of a Java-based concept mapping tool within a TELS curriculum project to explore the implications of specific design features).

To select potential collaborators the TELS advisory board will consider alternatives and apply the criteria mentioned at the outset. The developers of BGuILE (Brian Reiser, Northwestern University), ChemSense (Patricia Schank, SRI International) and WorldWatcher (Daniel Edelson, Northwestern University) have expressed an interest in embedding their technology innovations within the TELS platform. Similarly, the other two projects within the NetCoIL grant (CoLAB and CoOLModes) are planning to develop new versions based on the SAIL architecture, allowing their innovations to interoperate with those of TELS. Such flexibility in the technology infrastructure will support a greater range of curriculum innovations, and will allow the TELS platform to accommodate more diverse functionality and learning designs.

TELS 3. Describe the technologies that are being merged with Pedagogica and WISE and the trajectory for these technical innovations. You described the combination s "essential to creating classroom-ready learning experiences that realize the potential of models and tools."

TELS is building a technology platform to support researchers who seek to develop technology-enhanced materials that include assessments aligned with instruction, scaffolding of students and teachers who adopt and use these materials, design principles to guide innovation, and support for localization or customization. Below, we describe the current functionality of the technology platform, in which WISE and Pedagogica are loosely coupled, as well as the trajectory of our technology development effort, which aims for more integration and modularity and will result in a more complete merging of functionality from the Berkeley and Concord innovations.

We refer to the current technology platform as TELS First Generation, which has been used to support the development of the twelve inquiry projects created in 2004, including their authoring by distributed teams, their delivery via WISE to classrooms, the collection of student data, and professional development supports for teachers.

We refer to the new technology infrastructure that is under development as TELS Second Generation, which responds to technical limitations of our first generation platform. This new software system will be developed according to an architecture designed over the past year by technology specialists from Berkeley and Concord known as SAIL: Scalable Architecture for Interactive Learning (Slotta, Aleahmad, & Zimmerman, 2005; Berge & Slotta, in press). SAIL will help to specify the design of all elements within the TELS Second Generation technology platform, providing a coherent framework for the exchange of data between modular technology elements, and specifying the structure of each object as well as any data it reports and user interface elements it may contain.

TELS First Generation Technology Platform

TELS has achieved a somewhat rudimentary merging of Concord and WISE technologies to allow TELS researchers to design inquiry projects and explore how models or simulations can promote learning of science concepts. TELS expands the functionality of WISE by integrating the rich models, visualizations, and probeware from the Concord Consortium (e.g., Pedagogica, Molecular Workbench and CC-Probe). TELS expands the effectiveness of Concord models by supporting the design of science inquiry projects, their delivery to classrooms, the capture of student assessments (in the WISE environment), and the scaffolding of students using the materials.

The First Generation technology platform is a hybrid solution. WISE has been relatively unaltered. The Concord models are embedded within WISE projects and launched using an off-the-shelf protocol called "Java WebStart." These models are sizeable (e.g., 5 or 6 Megabytes) Java applications which are downloaded from the Concord server and stored remotely on each client machine (so that they do not have to be downloaded again). Upon completion of the Concord modeling step, the student closes that application window, which then disappears and returns the student to WISE without logging any data.

Before TELS, the Concord modeling technologies were used in standalone lessons that could log student interactions but could not add other inquiry features. Before TELS, WISE projects offered inquiry activities and scaffolding for students but lacked the sophisticated, interactive models developed by Concord. This merging also has some downsides. Scripting Concord models is more difficult than authoring WISE projects. Concord programmers need to help with any customizations. Data logging in Concord models cannot be supported using the launching mechanism in TELS. However, WISE can gather embedded assessments and other information from students.

To illustrate the capabilities of the TELS First Generation technology platform, we describe the project titled "Airbags: Too Fast, too Furious," Kevin McElhaney, a first-year TELS graduate fellow led the partnership to create a TELS inquiry project for high school physics that would help students learn the concept of relative velocity. The group selected an existing Concord Pedagogica model that supports study of the relative motion of two objects using simulations linked dynamically to graphs and data tables. The inquiry theme of "why airbags can sometimes be dangerous," focused students on exploring the speed at which the airbag deploys and the fact that great damage will result if the passenger meets the airbag before it is fully deployed (because the velocity of the passenger in the reference frame of the car will be added to the velocity of the expanding airbag). Students are introduced to the Concord model early in the project, with increasingly sophisticated versions that enable investigations of the airbag properties, the car's crumple zones, and the distance of the driver's head from the steering wheel. Students manipulate this model, gaining experience that WISE could not have provided. WISE added an inquiry context and a scaffolding environment that the Concord models did not have before. This example illustrates how the TELS technology platform supports investigations of how models, simulations, and linked graphical representations can help students develop an understanding of difficult science concepts like that of relative velocity (see McElhaney, 2005).

TELS Second Generation Technology Platform

The First Generation technology platform has been successful in a wide range of school settings, student populations, science topic areas, modeling activities, and end-user technology configurations. However, some challenges have been identified, particularly concerning Web browser response times, firewalls, security of data, student registration, and data logging. These challenges have revealed limitations in the central Web server and browser architecture, and have helped to guide our designs of the next generation of TELS technology.

As a result Concord and Berkeley technology developers began designing a new technology platform that merges the various software systems into a more coherent architecture of interoperating java applications which run on local student computers. The use of a common Java framework will enable a greater level of integration between the rich modeling activities (i.e., from Concord) and the scaffolding environment (i.e., from Berkeley) and a greater flexibility in terms of the curriculum and assessment materials that can be designed.

While the technologists from Berkeley and Concord recognized the advantages of a new architecture that emphasized modular, interoperating Java "objects", a substantial problem was the specification of those objects in terms of their properties, data structures, and rules for combining them. Collectively, these specifications and rules can be interpreted as an architectural model, according to which a very wide range of learning technologies could be created. While supporting the First Generation technology platform, TELS has progressed in defining a Scalable Architecture for Interactive Learning (SAIL). While SAIL is not a usable technology in itself, it offers a level of specification that will allow developers to create new Java-based learning technologies (or to re-write existing ones) with confidence that their creations will be able to smoothly interoperate.

Many of the technology tools within WISE are Java-based (e.g., the tools for drawing, concept mapping, graphing, and argument editing), and all Concord models are implemented in Java. A key aspect of the Second Generation technology platform will be the replacement of monolithic control applications (e.g., WISE or Pedagogica) with modular, object oriented systems where well-defined learning objects present themselves to the learner, or to some higher level object, managing their own data according to the SAIL specifications. This object oriented approach is common and well respected in the software development community, and is well suited to the modular, step-wise curriculum that is characteristic of TELS.

The Second Generation platform, currently under development, will appear fairly similar to the existing software, with a WISE-like inquiry map that results in various kinds of steps, each of which can be authored and customized to collect various types of data from students. While the user experience of Second Generation software will be relatively unchanged (although with somewhat improved performance), the operations of managing content and user data will be quite different. In departing from the standard Web server/Web browser paradigm, the Second Generation platform will adopt a more sophisticated treatment of important issues such as user accounts, authentication and security, re-usability of content and interoperability with other technology systems. TELS will begin piloting Second Generation systems in Fall, 2005, with a gradual scaling within our center as the technology stabilizes during Spring, 2006.

We hope that our adoption of a more scalable, sustainable architecture will increase the capabilities of TELS to share its innovations within a gradually expanding community of researchers and practitioners. Ideally, other research projects will adopt the open source framework of SAIL, which would expand the available functionality for innovations, and hopefully promote a dynamic community of developers. The Second Generation platform should also enable a more dynamic evolution of the TELS technology environments, leading to more diverse curriculum designs with adaptive flow of activities and greater capabilities for student collaboration.

TELS 4. Among the promised outcomes are that "TELS will create a large body of research that provides new and stronger evidence of the benefits and pitfalls of technology-based innovations for student learning in diverse settings" and "TELS will create a community of informed and able researchers who understand how technology can enhance learning. Describe progress or revisions in your thinking on these points.

TELS is simultaneously building a research base and stimulating a larger community to develop the skills and capabilities to contribute relevant research findings. TELS also has a research agenda concerned with building partnerships and motivating new research directions.

Creating a body of research

TELS researches how powerful technologies such as models and visualizations can be integrated into technology-enhanced inquiry projects to help improve scientific understanding and scaffold students, teachers, and administrators. The TELS research agenda has six themes summarized in the response to the question about TELS Research Themes. Research to date has focused on the challenge of providing new and stronger evidence of the benefits and pitfalls of technology-based innovations for student learning in diverse settings. This research has benefited from several dimensions of TELS, including:

Diverse school partners and research on equity. Investigations of student, teacher, and administrator learning are taking place in collaboration with schools that serve a diverse population. The 34 teachers at these schools have implemented 11 of the 12 TELS curriculum projects with about 3700 students. Overall, these projects lead to improved understanding of complex scientific topics such as electricity, chemical reactions, and mitosis.

This research shows that TELS succeeds with a broad range of students. When researchers break down performance based on pretest results, they report equal progress for students with low, medium, and high entering knowledge (see Chiu, 2005; Tate, 2005). Results for language learners show opportunities to customize TELS projects for those learning English as well as benefits from this customization (see Clark et al, 2005).

These studies raise new questions that the TELS school partners enable us to pursue: How can powerful models contribute to language proficiency for students from varied cultural groups? What is the impact of TELS on students and schools that emphasize memorization? What is the impact of TELS curriculum materials on students who feel that their cultural group is unwelcome in science? TELS researchers are currently designing materials that address science learning in the context of career awareness.

Varied science topics and visualization. TELS projects introduce complex models for 12 different science topics. Preliminary findings show the benefit of these projects for understanding science. Researchers who look at the impact of the models find that some students struggle to understand the nature of scientific models and therefore benefit less than those who appreciate the role of models in reasoning (see Gobert, 2005; McElhaney, 2005).

These findings suggest new questions that TELS can address due to the varied topics that students study with scientific models: What are the advantages of using a common modeling framework for teaching varied topics in chemistry, physics, and other science domains? And, if students become familiar with modeling environments early in their instruction, will this familiarity provide them with a head start on learning from subsequent projects?

Multiple representations across science topics. Prior research has called for teaching with multiple representations of scientific phenomena, but often students simply become confused with a complex screen that contains multiple representations. The many representations and models available in the TELS technologies, including flexible Flash animations, provide an opportunity to obtain richer and more detailed information about students' understanding of representations, allowing researchers to explore this important question (see Clark et al, 2005).

School leadership in diverse contexts. TELS school partners are set in six different states and 13 districts, each with their own leadership and professional development norms. TELS has an amazing opportunity to explore the impact of school policies across these contexts. Already, research on the commitments of the principals suggest ways to get more leadership support for technology enhanced science instruction (see Fauvre, Harrison, & Bowyer, 2005).

In summary, these research directions cut across all TELS sites and take advantage of many different aspects of the TELS resources. They also provide illustrations for researchers in the field to contribute findings that connect to the research of TELS.

Community building

TELS has begun to build a community of informed and able researchers in the areas of science education, technology, computer science, engineering, professional development, and student learning who investigate the impact of technology on learning. TELS stimulates the field to create more cohesive and cumulative research with its resources, partnerships, and communication activities:

• Resources. The field benefits when researchers use up-to-date technologies and test ideas in new technological environments supported by TELS. The resources now widely used by the TELS community serve as one vehicle for enticing new and established researchers to participate in a more coherent enterprise to TELS (see NSF 1 and NSF 6). For example, new communities have adopted the design principles database as a way to capture design knowledge (see TELS 6).
• Partnerships. Partnerships supported by TELS or by new funding are another way that the community is growing (see NSF 6, TELS 2 and TELS 5).
• Communication activities. TELS uses multiple communication paths including a web site, collaborative courses, perspectives, the TELS certificate, professional development workshops, research posters, publications, presentations, curriculum guides, assessments, and design patterns. Collaborative courses designed by TELS researchers and open to students at each site have already served over 50 students and offer a way to build the community. TELS is in the process of creating a TELS certificate for teachers that aligns with federal and state requirements for certification (see NSF 3).

Nanoscience case study. TELS resources, partnerships, and communication activities are contributing to a new research direction in the area of nanoscience education. This has occurred through TELS publications and presentations and illustrates how TELS can appeal to broad audiences. Tinker has prepared a draft TELS Perspective on nanoscience that shows how TELS resources such as Molecular Workbench can connect to researchers in NSF funded centers, science museums, and universities who want to bring the exciting findings in this area to life. Linn combined the information in the Perspective with the supports in WISE to motivate participants at a nanoscience conference in Doha, Qatar to form a partnership that would create Arabic versions of WISE and allow them to research the impact of the materials in university courses. Researchers at the Lawrence Hall of Science, seeing the models running on the TELS website, have indicated interest in using these models for a new exhibit as part of their nanoscience area.

Researching partnerships

An important TELS research theme concerns determining strategies for motivating new investigators to join in conducting research on technology-enhanced learning. Difficulties in communicating design knowledge to new groups, obstacles to using innovations in new contexts, and limitations in the human capital in the field have all traditionally impeded efforts to conduct cumulative, influential research. Several examples illustrate how TELS has progressed in this effort

Design knowledge. TELS investigators are studying advances in capturing design knowledge using the design principles database (see Kali & Fuhrmann, 2005; TELS 5). As the resources in the database grow, the interest shown by new groups expands. Kali is forming a partnership funded by NASA to use the database to capture knowledge that is currently only available by talking to individual designers. This work raises new questions such as, how can the design principles database support partnerships in customizing or designing projects?

Partnership practices. TELS supports many forms of partnerships including software design partnerships, professional development partnerships, open source communities, international collaborations, collaborative course design partnerships, and research partnerships. TELS is studying how these partnerships progress and identifying tools and practices to improve the process (see Hoadley & Stockman, 2005). This research looks at the role of varied TELS resources including the perspectives and the summer retreat and identifies both obstacles and successful strategies. The partnerships that designed the 12 projects illustrate the range of successful approaches as well as the limitations of technology-enhanced communication tools.

Conclusions

Building a successful, cohesive research program for investigations of technology-enhanced learning requires a firm foundation as well as flexible and innovative approaches. TELS is developing the foundation and forming partnerships that are strong and influential. TELS is exploring this new, challenging and ultimately incredibly important question with both enthusiasm and trepidation. Many external forces such as research funding, school funding, federal and state policies, and competing research opportunities make this by far the most high risk, high gain aspect of the TELS center. We seek insight, support, and sustained interest from policy leaders to make this a cumulative research program a reality.

We are creating an open research center that promotes good learning design and important research in the context of sophisticated science enabled by flexible technology. We share a vision of science learning of difficult, core concepts through student exploration and problem solving. The technology facilitates this by simplifying the authoring and delivery of materials that promote this kind of learning, by providing tools (including probeware) and models for exploration, and by monitoring and logging student learning in this context. This is an ambitious, long-term vision that will take a decade. We are only beginning to assemble the required technology, to prepare the schools to undertake this research in their classrooms, to train the cadres of researchers needed, and to assemble the additional human and financial resources required.

TELS 5. One of the innovations of TELS is that you will codify "design knowledge in the design principles database and [test] its value for developers and customizers."  What progress is being made on this front?

TELS leader Yael Kali and fellow Tamar Ronen-Fhurmann-Fuhrmann, at the Technion, are coordinating a new professional practice for research synthesis supported by the TELS Design Principles Database. Specifically, TELS is:

• Synthesizing TELS "design knowledge" by studying how individuals and teams design curriculum materials using TELS technologies.
• Exploring how the architecture of Design Principles Database can help designers create effective technology-enhanced materials.
• Developing a collaborative course to prepare designers of educational materials. Instructional activities are being pilot-tested in a course at the Technion and will be used in the Fall 2005 TELS Collaborative Design course.

Synthesizing TELS design knowledge

To investigate how the database helps identify research questions that need to be answered by each of the curriculum development teams and to understand how the Design Principles Database can provide a cumulative record of design knowledge, the Technion design team reviews TELS projects and identifies features that could contribute to community understanding of design. The evidence from this research is still being gathered and analyzed, so these findings are preliminary.

Each Feature in the database includes the following information: a) rationale behind the feature (also called specific design principle), b) detailed description, c) context of use, d) field-based evidence (including successes as well as failures), and e) technical information (author, category, subject, kind, grade, software URL, reference URL). Each Feature also connects to Pragmatic Design Principles, which are already in the database, or are contributed as new TELS design knowledge to the database.

To establish the entry for the database, the Technion team generates a draft summary. The authors of the project then review and refine the draft. This interchange provides an excellent record of how two groups describe the same design feature. Before entering the information in the database the two groups negotiate a final version. These features are then published in the database.

This research is currently describing features from the twelve projects, and collecting the field-based evidence from each (36 features are currently in authoring/revising stage). Six TELS features and two Pragmatic Design Principles have already been published in the database. Thirty more TELS features and several more Pragmatic Design Principles are in writing/review process.

Locate the features by following these URLs:

Active watching in a time-travel machine animation

Zooming from Macro to Micro animation

Graphing student collected data

Dynamic zooming between macro and micro phenomena (phase change)

Manipulable models of molecules in Molecular Workbench

Preliminary findings point to several design principles that unite TELS work. These principles illustrate decisions arising as a consequence of merging WISE and Pedagogica. For instance, we see that many of the projects implement Design a clear and engaging flow of activities based on the demands of WISE. Projects also implement Enable manipulation of factors in models and in simulations to incorporate the Pedagogica models. To see how this infrastructure can help synthesize field-based evidence, follow these URLs to the design principle "Enable manipulation of factors in models and simulation", which connects with the TELS features a) Manipulable models of molecules in Molecular Workbench, and b) Global Warming Model. We see the combination of these principles in many of the projects, as an indicator that TELS has successfully merged the technologies, and thus created an innovative direction to STEM education.

Improving the architecture of the Design Principles Database

The Technion team is focusing on how the architecture of the Design Principles Database can design. Preliminary studies show that the database can serve as a metacognitive tool, helping partnerships become more aware of the rationale for features they design.

The database includes three levels of principles: Meta-Principles, Pragmatic-Principles, and Specific Principles (see http://www.design-principles.org/dp/index.php#figure1). The Meta level includes four principles, built into the database, which summarize the knowledge integration framework and define instructional guidelines based on the framework (Linn, in press; Linn & Hsi, 2000). The Pragmatic Principles represent syntheses of design research which have common evidence from diverse settings and disciplines about impact on learning. The Specific Principles include the descriptions of individual software features and field-based evidence of their implementation in the classroom.

Researchers are exploring how the connections between these levels can help designers reflect on design knowledge. TELS designers can use research-based pragmatic design principles and understand the theories behind them. They can also use the database to capture new features, and evidence collected from their implementation. This evidence can strengthen the research basis of existing design principles or in some cases give rise to new pragmatic principles.

Creating the TELS Collaborative Design course

The process of creating an effective collaborative course that builds a community and harvests design knowledge involves trial and refinement. The three trials conducted by TELS are:

a)     Exploring the usability of the database in the design process. The value of the Design Principles Database was explored with four case-study graduate students, who participated in a technology-based curriculum development course at the Technion. The outcomes of the study informed the design of learning materials which are currently in pilot testing (see next section).

b)    Pilot-testing the learning materials. The learning materials are structured in three main activities: a) Brainstorming, in which students use the Meta-Principles to generate ideas for their design, b) Planning, in which students use design patterns and pragmatic principles to make decisions about how to build an engaging flow of activities, and c) Designing, in which students review relevant features in the database, select specific features, incorporate the features in the flow of activities, and produce a detailed design document.

c)     Implementation and data-collection. In this stage, the revised learning materials will be tested in the TELS Collaborative Design course.

In summary, TELS is synthesizing design features and research evidence in the Design Principles Database. The database serves as a community tool to combine evidence from standalone projects into some more general research-based design principles. In addition, the database captures current design knowledge and makes it useful for designers and customizers.