Tech Tally Approaches to Assessing Technological Literacy (2006) / Chapter Skim
Currently Skimming:
APPENDIX E Instrument Summaries
APPENDIX E Instrument Summaries
Pages 265-346
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 265... ...
, 293 ICT Literacy Assessment, 296 Illinois Standards Achievements Test -- Science, 299 Industrial Technology Literacy Test, 301 Infinity Project Pretest and Final Test, 304 Information Technology in a Global Society, 307 Massachusetts Comprehensive Assessment System -- Science and Technology/Engineering, 311 Multiple-Choice Instrument for Monitoring Views on Science Technology-Society Topics, 313 New York State Intermediate Assessment in Technology, 317 Praxis Specialty Area Test: Technology Education, 320 265
|
From page 266... ...
Provincial Learning Assessment in Technological Literacy, 322 Pupils' Attitudes Toward Technology (PATT-USA) , 325 Science and Technology: Public Attitudes and Public Understanding, 327 Student Individualized Performance Inventory, 330 Survey of Technological Literacy of Elementary and Junior High School Students, 333 Test of Technological Literacy, 336 TL50: Technological Literacy Instrument, 337 WorkKeys -- Applied Technology, 340 266 T E C H T A L L Y
|
From page 267... ...
Scope The U.S. Department of Defense maintains and administers the Armed Services Vocational Aptitude Battery (ASVAB)
|
From page 268... ...
Minimum percentile scores are required for enlistment; combinations of scores from the eight areas are used to qualify test takers for specialties in each branch of the military. Sample Items Readers wishing to get a sense of the types of items on ASVAB are encouraged to look at an ASVAB test-preparation book, such as ASVAB, The Armed Services Vocational Aptitude Battery, 2004 Edition (Simon and Schuster)
|
From page 269... ...
was established in 1975 to monitor student achievement in British schools. Over time, the focus of APU shifted from assessment to providing support for curriculum development.
|
From page 270... ...
Sample Items · Developing-solutions, concept-model activity for the 90-minute paper and-pencil test. (Concept-model activities were presented to students in physical form.
|
From page 271... ...
") Committee Observations This instrument reflects a curricular emphasis on "design and technology" in the U.K.
|
From page 272... ...
approach to assessment offers real insights into design competency but does not address technological literacy in the broad sense defined by the committee. Awareness Survey on Genetically Modified Foods Background Sponsor/Creator Jane Macoubrie, Patrick Hamlett, and Carolyn Miller, North Carolina State Uni versity, with funding from the National Science Foundation Purpose Research on public involvement in deci sion making on science and technology issues What is measured Knowledge and attitudes toward geneti cally modified foods Target population American adults Item format Multiple choice Sample size 45 adults in North Carolina Frequency of administration Once, in 2001 Availability Jane Macoubrie, Department of Commu nication, North Carolina State University Scope This project was inspired by the Danish practice of providing opportunities for citizens to participate in "consensus conferences" to discuss science and technology issues and make policy recommendations to the government.
|
From page 273... ...
This survey, which was administered to participants prior to the conference, included 20 multiple-choice questions addressing ethical and scientific issues, as well as current practices in the farming of genetically modified crops. Sample Items2 · Can genes escape from genetically modified crops and jump to other plants?
|
From page 274... ...
It would be interesting to administer a survey like this before and after participation in a consensus-type conference to determine what, if any, learning has taken place. Design-Based Science Background Sponsor/Creator David Fortus, University of Michigan Purpose Curriculum development, research What is measured Science and technology knowledge and transfer of design skills to new situations Target population 9th- and 10th-grade students in the United States Item format Multiple-choice and open-ended ques tions and design skills projects Sample size 92 students in 9th and 10th grade in one Michigan public high school Frequency of administration Once, in 20012002 Availability Dissertation held at University of Michi gan and an article describing the instru ment (Journal of Research in Science Training 41(10)
|
From page 275... ...
The structures for extreme environments unit was followed by a design project requiring the design of a kite that could fly a mile high. The environmentally safe batteries unit was followed by a project requiring the design of a battery for an artificial heart.
|
From page 276... ...
Assume that the voltmeter and connecting wires are not broken, and that you can add or change materials to the setup if necessary. · Structures for extreme environments unit assessment of design skills Can you design a kite that will fly one mile high?
|
From page 277... ...
Davis, Washington State University Purpose Assess students' knowledge, performance, and evaluation of the design process; evalu ate student teamwork and communica tion skills What is measured Student knowledge and skills in engineer ing design Target population Baccalaureate engineering students Item format Constructed-response, team design exer cise, reflective essay Sample size Unknown Frequency of administration Unknown Availability http://www.tidee.cea.wsu.edu/assessment tools/ Scope This three-part assessment was developed in 2002. The first component, intended for early-stage baccalaureate engineering students, A P P E N D I X E : I N S T R U M E N T S U M M A R I E S 277
|
From page 278... ...
(Suggested correct answer mentions gathering informa tion, defining requirements, generating ideas, evaluating ideas, making decisions, implementing ideas, and devel oping a process.) · Component 2: Ability to perform engineering design processes (in groups of four)
|
From page 279... ...
· Component 3: Understanding of the engineering design process and analysis of team design performance Prepare a 2-page essay, double-spaced in 12-point font, demonstrating your understanding of team-based engi neeringdesignprocessesfocusedonmeetingacustomer's needs. Reflecting on your recent team design experience A P P E N D I X E : I N S T R U M E N T S U M M A R I E S 279
|
From page 280... ...
Students could perform well on this assessment without believing in any of the lessons of teamwork or the design process. That is, by memorizing jargon and the school-learned steps of the design process, a student could do well without demonstrating higher order thinking skills.
|
From page 281... ...
The syllabus for the standard-level design technology course stipulates that the curriculum must cover six areas of design technology: designers and the design cycle; the responsibility of the designer; materials, manufacturing processes, and techniques; production systems; and clean technology and green design. The curriculum for higher level courses covers these additional topics: raw material to final product; microstructures and macrostructures; and appropriate technologies.
|
From page 282... ...
Student performance is also evaluated by a teacher-directed "internal assessment," which includes a design project. The rubric for the internal assessment includes student planning, data collection, data processing and presentation, and manipu lative and personal skills.
|
From page 283... ...
Committee Observations This instrument does a very good job of assessing knowledge related to the IB curriculum. Paper 1 tests basic knowledge at the A P P E N D I X E : I N S T R U M E N T S U M M A R I E S 283
|
From page 284... ...
The assessment seems particularly effective at teasing out knowl edge of the design process. All three papers require technical knowledge beyond what might be considered basic technological literacy.
|
From page 285... ...
collects demographic data, including gender, ethnicity, age, type of school, years of teaching experience, and family and friendship connections to an engineer. Sample Items · Indicate whether you strongly disagree, disagree, are neutral, agree, or strongly agree with the following statement: Engineering can be a way to help teach students lan guage arts.
|
From page 286... ...
Directorate General for Press and Communication Purpose Monitor changes in public views of sci ence and technology to assist decision making by policy makers What is measured Opinions about science and technology Target population People 15 years and older in the EU Item format Survey Sample size 16,029 people in all 14 EU member states Frequency of administration Surveys on various topics conducted regu larly since 1973; this poll was taken in May/June of 2001 Availability http://europa.eu.int/comm/research/press/ 2001/pr0612en-report.pdf 286 T E C H T A L L Y
|
From page 287... ...
responsibilities and accountability of scientists; (4) genetically modified food; (5)
|
From page 288... ...
Most questions are straightforward and focus on current issues, but a few questions in the "perceptions of scientific methods" section appear to require some higher order thinking. In addition, the poll has a much stronger emphasis on environmental and biorelated science and technology issues (e.g., mad cow disease and genetically modified food)
|
From page 289... ...
food based on genetically modified organisms; (6) the scientific profession; (7)
|
From page 290... ...
For example, correlating opinions and knowledge with religion and educational level, among other factors, may be useful for this type of assessment. In terms of technological literacy, however, the poll does not assess design or technology skills of any kind.
|
From page 291... ...
f u t u r e c i t y . o r g / d o c s / 2004JudgesManual.pdf Scope In the Future City Competition, teams composed of three students, a teacher, and an engineer-mentor create a computer city design with SimCity software and a physical, scale model of part of the city.
|
From page 292... ...
Committee Observations The Future City Competition allows students to combine an open-ended, engineering design task with communication skills, use of 292 T E C H T A L L Y
|
From page 293... ...
Background Sponsor/Creator International Technology Education As sociation (ITEA) , with funding from the National Science Foundation and National Aeronautics and Space Administration Purpose Determine public knowledge and percep tions of technology to inform efforts to change and shape public views What is measured Public understanding, opinions, and atti tudes about technology and technological literacy Target population American adults Item format Survey Sample size 1,000 people in 2001; 800 in 2004 Frequency of administration Twice, in 2001 and 2004 Availability Contact ITEA, which commissioned the poll, at http://www.iteawww.org A P P E N D I X E : I N S T R U M E N T S U M M A R I E S 293
|
From page 294... ...
The 17-question telephone poll of 1,000 randomly selected Americans resulted in three major conclusions: (1) Americans believe technological literacy is important for everyone; (2)
|
From page 295... ...
(9 percent of respondents said a great deal; 32 percent said some; 40 percent said very little; and 19 percent said no influence.) Committee Observations Both polls addressed aspects of the ITEA Standards of Technological Literacy related to the nature of technology, technology and society, and abilities for a technological world.
|
From page 296... ...
; ris ing juniors at four-year colleges (advanced assessment) Item format 14 4-minute and one 15-minute simu lated, scenario-based tasks delivered via the Web Sample size Approximately 4,500 examinees at 31 campuses (January through April 2005 administration of advanced assessment)
|
From page 297... ...
Tests cost $35 each, and initial orders must include a minimum of 100 tests. Sample Items Actual test items are not publicly available.
|
From page 298... ...
Given the target population for the assessment, this may not be a significant worry. The assessment was not designed with the ITEA Standards for Technological Literacy in mind, but it addresses some of the benchmarks in ITEA Standard 17: Information and Communication Technology.
|
From page 299... ...
The 70-question exam is administered in 80 minutes and covers science inquiry, life sciences, physical sciences, earth and space sciences, and science, technology and society. The committee reviewed only sample test items because the Illinois Board of Education does not release actual test items.
|
From page 300... ...
Gray D Green (Suggested correct answer: A)
|
From page 301... ...
Industrial Technology Literacy Test Background Sponsor/Creator Michael Hayden, Iowa State University Purpose Assess the level of industrial-technology literacy among high school students What is measured Knowledge in systems, applications, and interpretations of industrial technology Target population American high school students Item format Multiple choice Sample size 806 high school and 265 college students A P P E N D I X E : I N S T R U M E N T S U M M A R I E S 301
|
From page 302... ...
Sample Items · Multiple choice The space shuttle and the Alaskan pipeline have as their most common characteristic the fact that: A they were both the center of accidents B
|
From page 303... ...
Committee Observations Considering the broad range of questions about industrial technology in this instrument, it may be appropriate for measuring industrialtechnology literacy for high school students who have taken courses in the field. A few of the questions require higher order thinking, such as interpreting a graph; however, the majority of questions require factual recall.
|
From page 304... ...
What is measured Cognitive skills and curriculum-related knowledge Target population American high school students Item format Open-ended and multiple-choice questions Sample size Thousands of students in 20 states Frequency of administration Ongoing since 1999 Availability Samples are available at ftp://ftp.prenhall. com/pub/esm/sample_chapters/engineering_ computer_science/orsak/index.html Scope Geoffrey Orsak, dean of the School of Engineering at Southern Methodist University, founded the Infinity Project in 1999 to interest more high school students in pursuing careers in engineering.
|
From page 305... ...
7The Infinity ProjectTM, Institute for Engineering Education, School of Engineering, Southern Methodist University.
|
From page 306... ...
9#333#21312#1151#21312#333#9 (Suggested correct answer: The letter "O") · Signal analysis -- end-of-year basic test (May 2003)
|
From page 307... ...
Information Technology in a Global Society Background Sponsor/Creator International Baccalaureate Organization (IBO) , Geneva, Switzerland Purpose Student evaluation What is measured Students' knowledge of information tech nology terminology, concepts, develop ments, trends, social significance, and ethi cal issues Target population 1619-year-old high school students who have taken the Information Technology in a Global Society course A P P E N D I X E : I N S T R U M E N T S U M M A R I E S 307
|
From page 308... ...
ibna@ibo.org Scope The International Baccalaureate Organization (IBO) oversees the IB Diploma Program, which offers intensive pre-university courses and exams.
|
From page 309... ...
As an added benefit, some local stores give a 5% discount to 8Information Technology in a Global Society, Standard Level Paper 1, November 2002 © International Baccalaureate Organization 2002. The assessment model for the subject was changed in May 2004, and this style of question is no longer used.
|
From page 310... ...
The open-ended questions require higher order thinking in cog nitive and affective dimensions. It was not possible to judge the capabili ties portion of the assessment because the external assessment did not include skills tasks.
|
From page 311... ...
Massachusetts Comprehensive Assessment System -- Science and Technology/ Engineering Background Sponsor/Creator Massachusetts Department of Education Purpose Monitor individual student achievement, gauge school and district performance, sat isfy requirements of No Child Left Behind Act What is measured Knowledge of technology and engineering Target population 5th-, 8th-, and 10th-grade students in Massachusetts Item format Multiple-choice and open-response items Sample size 74,605 5th-grade and 78,686 8th-grade students in Massachusetts Frequency of administration Annually since 1998 Availability h t t p : / / w w w .
|
From page 312... ...
Fifth-grade students are tested on their knowledge of materials and tools and their understanding of the design process. The 8th-grade test requires deeper knowledge of the nature of technology, as well as specific domains of technology, such as construction, communica tion, manufacturing, transportation, and biotechnologies.
|
From page 313... ...
Most of the questions are multiple choice and focus on knowledge of course content, identification of terms, and rote memorization. A few open-ended questions require higher order thinking or design thinking to solve problems.
|
From page 314... ...
Sample Items · Science and technology, defining technology Defining what technology is can cause difficulties because technology does many things in Canada. But MAINLY technology is: 314 T E C H T A L L Y
|
From page 315... ...
J None of these choices fits my basic viewpoint.
|
From page 316... ...
H None of these choices fits my basic viewpoint.
|
From page 317... ...
Purpose Student evaluation; curriculum improvement What is measured Students' knowledge and skills in seven areas Target population 7th- and 8th-grade students in New York who have taken a technology education course Item format Multiple choice and extended response Sample size Unknown Frequency of administration Unknown Availability http://www.emsc.nysed.gov/ciai/mst/pub/ tqsample.pdf Scope In 1986, New York schools began offering an Introduction to Technology course for 7th- and 8th-grade students. The State Department of Education developed this instrument in 2001 to test their knowledge.
|
From page 318... ...
hydraulic (Suggested correct answer: 4) · Tools, resources, and technological processes multiple choice Which device produces power by means of a chemical reaction?
|
From page 319... ...
Committee Observations This assessment includes a balance of multiple-choice questions on technological concepts and open-ended questions that require a reasonable level of higher order thinking. Although this is a paper-andpencil assessment, some of the open-ended questions touch on the capabilities and ways of thinking and acting dimension of technological literacy.
|
From page 320... ...
In recent years, ETS has modi fied the test to bring it into alignment with the ITEA Standards for Technological Literacy. The test has 120 multiple-choice questions di vided into five categories: pedagogical and professional studies, informa tion and communication technologies, construction technologies, manu facturing technologies, and energy/power/transportation technologies.
|
From page 321... ...
use leading questions to help the student review and analyze the relative merits of several assembly procedures E refer the student to a reference on assembly procedures (Suggested correct answer: D)
|
From page 322... ...
Committee Observations Although the items in the assessment are aligned with the ITEA Standards on Technological Literacy, they do not probe higher order thinking. Most questions focus on terminology, recall of definitions, and the identification of basic concepts.
|
From page 323... ...
Student performance was assessed on a scale of 1 to 5 with a well defined rubric for each task (see sample question)
|
From page 324... ...
To sell more product E None of the above (Suggested correct answer: D)
|
From page 325... ...
Dugger, with funding from Vir ginia Polytechnic Institute and State Uni versity, the state of New Jersey, and the International Technology Education Association Purpose Assess student attitudes toward and knowledge of technology What is measured How gender, age, parents' professions, technology in the home, and courses in technology influence students' conceptions of and attitudes toward technology Target population American middle school students Item format Multiple-choice and open-ended questions Sample size 10,349 students in seven states Frequency of administration Once in 1988 Availability http://www.iteawww.org (under Confer ence Proceedings)
|
From page 326... ...
Nearly 77 percent of the stu dents who took the PATT-USA survey were enrolled in or had previously taken a technology education or industrial arts class. The open-ended question asked student to describe what technology is.
|
From page 327... ...
Committee Observations The PATT-USA focuses on assessing attitudes of students toward technology. The questions that test students' understanding of technological concepts do not require higher order thinking.
|
From page 328... ...
A major conclusion of both the 2001 and 2004 surveys was that, although Americans are interested in scientific discoveries and new technologies, they do not feel well informed or know a lot about technology-related issues. Sample Items (From the 2001 survey)
|
From page 329... ...
adult knowledge and attitudes toward science and, to a lesser extent, technology. Most of the questions focused on attitudes, and the knowledgerelated questions did not require higher order thinking.
|
From page 330... ...
Burke, with funding from the Council on Technology Teacher Edu cation, International Technology Educa tion Association, and the Technical Foun dation of America Purpose Develop a model to assess the problem solving capabilities of students engaged in design activities What is measured Student achievement in 12 areas of design and problem solving Target population American high school students Item format Rubric Sample size Two small high school classes of 12 and 15 students Frequency of administration Several times for research purposes Availability Rodney Custer, Department of Technol ogy, Illinois State University 330 T E C H T A L L Y
|
From page 331... ...
Student achievement was correlated with a number of factors, including geographic location, technology education experience, grade level, mathematics and science achievement score, personality type, problemsolving style, and gender. Sample Items (Because this instrument is a rubric, rather than an exam, there are no sample questions.
|
From page 332... ...
Model/prototype is crude, with little or no refinements made. Committee Observations The Student Individualized Performance Activity is a well con sidered tool for assessing design skills, and the rubric adheres to the ITEA Standards for Technological Literacy.
|
From page 333... ...
According to the test developers, the 80-question exam includes questions in 10 areas of abilities in technological literacy and six technology systems.
|
From page 334... ...
Between 1995 and 2000, this assessment was used in several local studies and a number of master's theses. However, it is not used regularly or widely to assess technological literacy.
|
From page 335... ...
The tool has multiple shapes (Suggested correct answer: A) Committee Observations For the most part, the items in this assessment appear to be appropriate for the target populations.
|
From page 336... ...
Sample Items · A manufacturing control question Which of the following items needs the highest design safety factor?
|
From page 337... ...
TL50: Technological Literacy Instrument Background Sponsor/Creator Michael Dyrenfurth, Purdue University Purpose Gauge technological literacy A P P E N D I X E : I N S T R U M E N T S U M M A R I E S 337
|
From page 338... ...
Slightly more than half of the items address technological procedures. Sample Items · Technological procedures: systems analysis and synthesis questions Consider a typical factory's automated spray paint station that uses a robot to paint parts passing on a conveyor.
|
From page 339... ...
Vision system, auto-conveyor, light system, transfer robot (Suggested correct answer: C) · Fundamentals of materials and processing technologies: materials tech nology basics question The process of tempering material: A
|
From page 340... ...
Assessments may be taken ei ther through an employer licensed by ACT or a licensed WorkKeys site, typically an educational institution. Availability Test details, sample items, and informa tion about ordering a practice test are avail able at http://www.act.org/workkeys 340 T E C H T A L L Y
|
From page 341... ...
10These sample items appear on the WorkKeys website at http://www.act.org/ workkeys/assess/tech and are reprinted with permission.
|
From page 342... ...
About 4 feet from the floor at location B (suggested correct answer)
|
From page 343... ...
2 C 3 (suggested correct answer)
|
From page 344... ...
You check the control valve and it is fine. Also, there is no hydraulic 344 T E C H T A L L Y
|
From page 345... ...
Although not designed with the ITEA Standards for Technological Literacy in mind, the sample items are consistent with benchmarks described in the ITEA standards related to energy and power (Standard 16) , using and maintaining technological products and systems (Standard 12)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.