The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy APPENDIX E Instrument Summaries Contents      Armed Services Vocational Aptitude Battery,   267      Assessment of Performance in Design and Technology,   268      Awareness Survey on Genetically Modified Foods,   272      Design-Based Science,   274      Design Team Assessments for Engineering Students,   277      Design Technology (Higher Level),   280      Engineering K–12 Center Teacher Survey,   284      Eurobarometer: Europeans, Science and Technology,   286      European Commission Candidate Countries Eurobarometer: Science and Technology,   288      Future City Competition—Judges Manual,   291      Gallup Poll on What Americans Think About Technology (2001, 2004),   293      ICT Literacy Assessment,   296      Illinois Standards Achievements Test—Science,   299      Industrial Technology Literacy Test,   301      Infinity Project Pretest and Final Test,   304      Information Technology in a Global Society,   307      Massachusetts Comprehensive Assessment System—Science and Technology/Engineering,   311      Multiple-Choice Instrument for Monitoring Views on Science-Technology-Society Topics,   313      New York State Intermediate Assessment in Technology,   317      Praxis Specialty Area Test: Technology Education,   320

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy                 Provincial Learning Assessment in Technological Literacy,   322      Pupils’ Attitudes Toward Technology (PATT-USA),   325      Science and Technology: Public Attitudes and Public Understanding,   327      Student Individualized Performance Inventory,   330      Survey of Technological Literacy of Elementary and Junior High School Students,   333      Test of Technological Literacy,   336      TL50: Technological Literacy Instrument,   337      WorkKeys—Applied Technology,   340

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Armed Services Vocational Aptitude Battery Background Sponsor/Creator U.S. Department of Defense Purpose Assess potential of military recruits for job specialties in the armed forces; provide a standard for enlistment What is measured Knowledge and reasoning skills in eight areas Target population Young Americans interested in military careers Item format Multiple choice Sample size More than 900,000 high school students annually Frequency of administration Ongoing in its present form since 1968 Availability Sample items available from various test preparation books (e.g., Kaplan ASVAB 2004 edition, Simon and Schuster) Scope The U.S. Department of Defense maintains and administers the Armed Services Vocational Aptitude Battery (ASVAB)1 to assess the potential of military recruits for enlistment and various specialties. ASVAB is currently administered in three forms. High school students, the most common test takers, can take the form 18/19 version of the ASVAB as early as 10th grade. Recruiters can also administer a paper version or computer-adapted exam to prospective recruits who are no longer in 1 ASVAB is a registered trademark of the U.S. Department of Defense.

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy school. ASVAB includes eight sections: general science, arithmetic reasoning, word knowledge, paragraph comprehension, auto and shop information, mathematics, mechanical comprehension, and electronics information. Scores are reported in each area, and a simple equation is used to calculate a raw score, which is converted into a percentile score. Test takers also receive composite scores in verbal ability, math ability, and academic ability. Minimum percentile scores are required for enlistment; combinations of scores from the eight areas are used to qualify test takers for specialties in each branch of the military. Sample Items Readers wishing to get a sense of the types of items on ASVAB are encouraged to look at an ASVAB test-preparation book, such as ASVAB, The Armed Services Vocational Aptitude Battery, 2004 Edition (Simon and Schuster). Committee Observations The ASVAB exam is an appropriate instrument for the military to assess a broad range of knowledge and abilities among high school students and young adults. The sections on spatial reasoning, mechanical comprehension, and auto and shop information seem relevant to technological literacy. Despite the emphasis on technological topics, however, most of the items are very narrow in scope and require only factual recall or low-level application of knowledge. The auto and shop questions favor males, who tend to have more exposure in these areas. Assessment of Performance in Design and Technology Background Sponsor/Creator Richard Kimbell, et al. at the Technology Education Research Unit, Goldsmiths College, University of London, with funding from the U.K. Department of Education and Science

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Purpose Curriculum development, research What is measured Design capabilities Target population 15-year-old students in the United Kingdom Item format 90-minute open-ended design tasks, half-day modeling tasks, and a long-term project Sample size Approximately 10,000 students from more than 700 schools in the United Kingdom Frequency of administration Once, in 1989 Availability Kimbell, R., K. Stables, T. Wheeler, A. Wozniak, and V. Kelly. 1991. The Assessment of Performance in Design and Technology: The Final Report of the APU Design and Technology Project. London (D/010/B/91): Schools Examinations and Assessment Council/Central Office of Information. 285 pp. Scope The Assessment Performance Unit (APU) was established in 1975 to monitor student achievement in British schools. Over time, the focus of APU shifted from assessment to providing support for curriculum development. In 1985, APU commissioned an assessment of design and technology achievement to gauge how well students performed in design and technology activities. The assessment had three parts. The first part, administered to approximately 9,000 students, was a 90-minute pencil-and-paper test on which students were asked to complete a structured activity. Twenty-one activities were created for the assessment each involving one of three contexts: people, environment, or industry. Each activity had a specific focus: starting point, early idea, development of a solution, evaluation of a product, or modeling. A

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy “starting-point” activity might ask students to suggest new or improved products or systems that could be designed for the garden. A “developing-solutions” activity might ask students to design a self-watering plant pot that could be stacked and interlocked. The activities ranged from closed, well-defined questions to open, loosely defined tasks. In the second part of the assessment, about 1,500 students who completed the paper-and-pencil test, took part in a half-day, team-based modeling activity in which they could use various soft and rigid modeling materials, such as rubber bands, beads, string, and fabric, to create a prototype design. The final assessment component involved approximately 70 of the 1,500 students from the second test. The students participated in long-term (up to nine months) school projects. Students were regularly interviewed to develop a long-term history of individual performance. In all three assessment components, researchers tried to determine how well students formed ideas, organized their time and resources, considered alternative solutions, and modeled solutions that could be evaluated against the user’s needs. Activities were evaluated in three areas: the processes in design and development, the quality of communication, and conceptual understanding. Holistic marks indicating a student’s overall capabilities were awarded based on pre-established characteristics of good and poor performance. Individual discriminators of capability, questions to ascertain if a student’s responses included certain predetermined components, were also used to evaluate performance. Sample Items Developing-solutions, concept-model activity for the 90-minute paper-and-pencil test. (Concept-model activities were presented to students in physical form. Ready-made ideas presented in half-developed form allowed students to proceed quickly into the design stage of the project.) When considering the needs of the elderly, a group of young people recognized the weakness of built-in cooker timers and decided to make one that was more suitable. A member of the team came up with the idea that a portable timer could be designed that was set by a twisting action. Design a timer for the elderly that:

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy is portable is set with a twisting action will sound an alarm when the time runs out is a suitable size and shape Your task today is to take this idea and develop it as far as you can in the time available. (This activity included a drawing of a twist-type timer.) Modeling activity for the half-day, team-based assessment The team has decided to make a bird scarer for use in gardens and allotments. A member of the team came up with the idea that “spinning in the wind” advertising could be developed for scaring birds. Design a bird scarer that: Has sails or vanes that catch the slightest breeze Makes “bird scaring” movements Gives off “bird scaring” sounds from a sound box Fits into the environment Your task today is to take this idea and develop it as far as you can in the time available. (This activity included two drawings. The first was of a wind-sail mounted on top of a sound box. The second depicted a garden and was accompanied by a number of thought-provoking questions, such as “what about high winds?”, “what makes the sound?”, and “is it safe?”) Committee Observations This instrument reflects a curricular emphasis on “design and technology” in the U.K. educational system. Assessment activities seem to require higher order cognitive capabilities. The evaluation framework, which includes holistic, procedural, communication, and conceptual elements based on four domains (task clarification, investigation, solution generation, and appraisal), is conceptually robust. This instrument is complex and would be difficult and expensive to administer, score, and report on a large scale. The task-centered

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy approach to assessment offers real insights into design competency but does not address technological literacy in the broad sense defined by the committee. Awareness Survey on Genetically Modified Foods Background Sponsor/Creator Jane Macoubrie, Patrick Hamlett, and Carolyn Miller, North Carolina State University, with funding from the National Science Foundation Purpose Research on public involvement in decision making on science and technology issues What is measured Knowledge and attitudes toward genetically modified foods Target population American adults Item format Multiple choice Sample size 45 adults in North Carolina Frequency of administration Once, in 2001 Availability Jane Macoubrie, Department of Communication, North Carolina State University Scope This project was inspired by the Danish practice of providing opportunities for citizens to participate in “consensus conferences” to discuss science and technology issues and make policy recommendations to the government. Conference participants are non-experts who are provided with extensive background information on a subject and then

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy convened to discuss the issue. Researchers at North Carolina State University conducted a Danish-style consensus conference in 2001 to assess the feasibility of consensus conferences in the United States. This survey, which was administered to participants prior to the conference, included 20 multiple-choice questions addressing ethical and scientific issues, as well as current practices in the farming of genetically modified crops. Sample Items2 Can genes escape from genetically modified crops and jump to other plants? Yes and they often do Only to some crops, but those crops aren’t genetically modified Only during rare climatic conditions No, genes cannot move from species to species without human intervention I don’t know (Suggested correct answer: A) To keep genetically modified crops separate from traditional crops, farmers are currently required to do which of the following? Use different machines to harvest each field Use different storage bins and silos Transport separately to the production facility None of the above I don’t know (Suggested correct answer: D) Ethical arguments against the genetic modification of food products include: Genetically modified crops violate species integrity Biotechnology changes too fast to effectively understand and regulate it The belief that scientists should not “play God” All of the above I don’t know (Suggested correct answer: D) 2 Reprinted with permission of the North Carolina Citizens’ Forum Project Team

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Committee Observations This content-specific survey does not require higher order thinking skills. In addition, the level of factual knowledge required to perform well is likely to be beyond the capability of most individuals in the target population. It would be interesting to administer a survey like this before and after participation in a consensus-type conference to determine what, if any, learning has taken place. Design-Based Science Background Sponsor/Creator David Fortus, University of Michigan Purpose Curriculum development, research What is measured Science and technology knowledge and transfer of design skills to new situations Target population 9th- and 10th-grade students in the United States Item format Multiple-choice and open-ended questions and design skills projects Sample size 92 students in 9th and 10th grade in one Michigan public high school Frequency of administration Once, in 2001–2002 Availability Dissertation held at University of Michigan and an article describing the instrument (Journal of Research in Science Training 41(10): 1081–1110).

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Scope David Fortus developed the design-based science (DBS) curriculum as part of the dissertation for his Ph.D. The DBS curriculum has three units: structures for extreme environments, environmentally safe batteries, and safer cell phones. The course instructor (not Fortus) started each unit by administering a pre-instruction content-knowledge test. The test was followed by a number of weeks of classroom teaching on the science and technology related to the unit, as well as instruction in the design process. At the end of each unit, students were given an exam that included 13–15 multiple-choice questions and 2–5 open-ended questions. Multiple-choice questions required low, medium, and high cognitive skills; open-ended questions required medium and high cognitive skills (as determined by Fortus). To test the transfer of design skills, students in groups of four were asked to apply knowledge from each unit to a new situation. The structures for extreme environments unit was followed by a design project requiring the design of a kite that could fly a mile high. The environmentally safe batteries unit was followed by a project requiring the design of a battery for an artificial heart. The unit on safer cell phones was followed by a project requiring the design of a hearing protector for rock musicians. Groups were evaluated in five categories: design variables; gathering of information; comparison of options; model, drawing, or diagram; and design evaluation. All four students in each group earned the same grade on the project. Sample Items3 Safer cell phones unit multiple-choice question A cell phone is similar to a microwave oven because: Both have been proven to be dangerous to your health They both emit microwaves They operate on the same voltage 3 From Fortus, D., R.C. Dershimer, J.S. Krajcik, R.W. Marx, and R. Mamlok-Naaman. 2004. Design-based science (DBS) and student learning. Journal of Research in Science Teaching 41(10): 1081–1110.

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Test of Technological Literacy Background Sponsor/Creator Abdul Hameed, Ohio State University Purpose Research What is measured Knowledge in four areas: construction, manufacturing, communication, and transportation technologies Target population 7th- and 8th-grade American students Item format Multiple choice Sample size 1,350 students from 20 schools Frequency of administration Once in April 1988 Availability Dissertation by Abdul Hameed held at Ohio State University Scope This test was developed by Abdul Hameed in the late 1980s as part of his Ph.D. dissertation in technical education (industrial arts) at Ohio State University. The 64-question exam, which is intended to be completed in a single class period, tests students’ understanding of using, making, and controlling technology. Sample Items A manufacturing control question Which of the following items needs the highest design safety factor? Airplane Gasoline Engine

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Radio Bicycle (Suggested correct answer: A) A construction making question Steel reinforcement is placed in concrete in order to Help keep the concrete from breaking and separating Improve the appearance Provide holes for ventilation Increase the weight (Suggested correct answer: A) An understanding transportation question The first manned rocket to enter space was launched in the late 1930s and early 1940s late 1940s and early 1950s late 1950s and early 1960s late 1960s and early 1970s (Suggested correct answer: C) Committee Observations This assessment covers a broad range of general knowledge about technology, but few questions require that students do anything other than recall information. The test does not require problem-solving, decision-making, or technology-related skills. A number of test items refer to specific technologies that were state of the art in the early 1980s but would not be familiar to many students today. TL50: Technological Literacy Instrument Background Sponsor/Creator Michael Dyrenfurth, Purdue University Purpose Gauge technological literacy

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy What is measured General knowledge of technology in eight areas Target population High school students, university students, and adults Item format Multiple choice Sample size Unknown Frequency of administration Unknown Availability Michael J. Dyrenfurth, College of Technology, Purdue University Scope This 50-question, multiple-choice instrument is designed to assess technological literacy in eight areas: (1) working with technology; (2) technological procedures; (3) overview of technology; (4) overview of industrial technology; (5) fundamentals of communications technology; (6) applications of energy and power technologies; (7) fundamentals of materials and processing technologies; and (8) impact of technologies on society. Slightly more than half of the items address technological procedures. Sample Items Technological procedures: systems analysis and synthesis questions Consider a typical factory’s automated spray paint station that uses a robot to paint parts passing on a conveyor. Which of the answers contains the best list of subsystems of such a work station? Controlling computer, transfer robot, auto-conveyor, cell perimeter Instrumentation unit, auto-conveyor, warehouse unit, read-out and input unit

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Auto-conveyor, controlling computer, spray robot, read-out and input unit Vision system, auto-conveyor, light system, transfer robot (Suggested correct answer: C) Fundamentals of materials and processing technologies: materials technology basics question The process of tempering material: Softens the metal and removes internal stresses Increases the metal’s resistance to scratching and abrasion Toughens the material Is not described (Suggested correct answer: C) Technological procedures: technology assessment/evaluation (impacts) question To properly judge the effects of a technological innovation, one should: Measure the dollar effects resulting from it Estimate the impacts of it on our society Identify its impact on the people using it All of the above (Suggested correct answer: D) Committee Observations Although this instrument includes questions that require interpretations of simple graphs and analog scales, the majority of questions rely heavily on memorization and knowledge of terminology that may become outdated or may not transfer well among population groups. The instrument may not be appropriate for university students in most science or technology fields because much of the content is basic and does not require a higher education.

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy WorkKeys—Applied Technology Background Sponsor/Creator ACT Purpose Determine workforce readiness; identify skills gaps in current and potential employees. What is measured Practical reasoning and problem-solving skills related to four applied-technology domains: electricity, mechanics, fluid dynamics, and thermodynamics Target population High school and community college students, adults transitioning to the workforce, current workers in technology-dependent businesses Item format 32 items at four levels of difficulty administered over 55 minutes (online) or 45 minutes (paper and pencil) Sample size Since 1992, when ACT introduced the WorkKeys Program, some 9 million individuals have taken one or more of the program’s 10 assessments (M.J. Klemme, WorkKeys consultant, personal communication, December 20, 2005). Frequency of administration There is no fixed schedule of test administration. Assessments may be taken either through an employer licensed by ACT or a licensed WorkKeys site, typically an educational institution. Availability Test details, sample items, and information about ordering a practice test are available at http://www.act.org/workkeys

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Scope The Applied Technology Assessment is one of 10 assessments offered by WorkKeys. The others are Reading for Information, Applied Mathematics, Business Writing, Writing, Locating Information, Teamwork, Observation, Listening, and Readiness. Although the assessments can be given individually, they were originally designed to be part of a larger ACT job-skills program. The program includes a component to help employers identify the skills necessary for specific jobs and a training element to close skill gaps revealed by the assessment. Test items are grouped into four difficulty levels, 3–6, based on the number and complexity of the skills required to answer each item correctly. A Level 3 item, for instance, describes a simple system with three to five components, portrays a problem with one variable, and includes all the information necessary to solve it. A Level 6 item describes a complex system with 10 or more components, presents a variety of possible problem sources, and includes considerable extraneous information. Assessment results can be presented as a level score, which ACT says should be used for employee selection, promotion, or other high-stakes purposes, or a scale score, which can show individual improvement over time, provide for group comparisons, or show a likelihood of benefit from educational opportunity. ACT charges $4 per test for educational and government institutions that use the assessment with their own students. The rate is higher for businesses. Sample Items10 Level 3 You are building a greenhouse like the one shown in Figure 1 for a local nursery. The owners specified that the greenhouse should have automatic vents, controlled by a thermostat, which will open when the temperature in the greenhouse gets too high for the plants. Figure 2 shows the floor plan of the greenhouse. 10 These sample items appear on the WorkKeys website at http://www.act.org/workkeys/assess/tech and are reprinted with permission.

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy Figure 1 Figure 2 A thermostat will control the opening and closing of the automatic vents. It is a temperature-sensitive device that can be set to activate when the air around it reaches a certain temperature. The owners of the greenhouse want to have the vents open when the air around the majority of the plants reaches 90°F. At what height and location in Figure 2 should you install the thermostat so it gives the desired results? About 4 feet from the floor at location A About 4 feet from the floor at location B (suggested correct answer) About 8 feet from the floor at location C About 8 feet from the floor at location D Near the peak of the roof at location E Level 4 Your industrial services company has been hired to deliver a small but heavy gearbox. The container is too small to justify renting a large truck and too heavy for the company’s pickup truck. You decide to rent a heavy-duty utility trailer

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy and pull it with the pickup truck. At which spot, labeled 1–5, on the trailer shown should you place the container to pull the load most easily and safely? 1 2 3 (suggested correct answer) 4 5 Level 5 The band saw where you work will not start. This saw uses 240 volts, draws 25 amps, and has 30-amp cartridge fuses. These fuses (see diagram shown) are designed to protect an electrical circuit. Their main component is a fuse wire made of a low-resistance, low-melting-point alloy. When a higher than tolerable current goes through such a fuse, this fuse wire melts. Your supervisor has told you to check the fuses in the band saw. By looking at the fuses, you cannot tell if they are good or bad. You have turned off the power to the saw and removed one of the fuses. You check this fuse with a volt-ohmmeter (a device that measures resistance to the flow of electrical current). If the fuse is good, the resistance (measured in ohms) for the fuse will be:

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy 0 ohms (suggested correct answer) 10 ohms 50 ohms 100 ohms infinite Level 6 The garage where you work is equipped with a hydraulic lift, like the one shown, that you use to raise cars off the floor so it is easier to service them. An air compressor capable of generating pressures of 120 pounds per square inch (psi) powers the lift. The air regulator releases a steady amount of air pressure (usually 30 to 40 psi), and the control valve directs the flow of that air through the lines. Pushing the control valve forward (as shown in the figure) allows air into the lines, raising the lift. Moving the valve to the middle position seals the line so no air can escape, and pulling the valve back releases air from the line, lowering the lift. The air from the compressor exerts a force on a tank of hydraulic fluid, which, in turn, transmits this force to the bottom of the lifting piston. *Figure adapted from Principles of Technology Teacher’s Guide, Year 1, Unit 7, Force Transformers (Waco, TX: Center for Occupational Research and Development, 1991), 94. Used with permission. You have been working on a car up on the lift for about an hour. When you raised the car, the lift worked normally, but now the lifting piston has begun to creep down. You check the control valve and it is fine. Also, there is no hydraulic

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy fluid on the garage floor or in the lift pit below the garage floor. The next thing you should check to determine the problem is the: air compressor air regulator air line between the compressor and the control valve air line between the control valve and the hydraulic fluid reservoir (suggested correct answer) line between the hydraulic fluid reservoir and the lifting piston Committee Observations This assessment is notable for its focus on problem solving and reasoning in technological systems. Although not designed with the ITEA Standards for Technological Literacy in mind, the sample items are consistent with benchmarks described in the ITEA standards related to energy and power (Standard 16), using and maintaining technological products and systems (Standard 12), and problem solving and trouble-shooting (part of Standard 10). The sample items suggest that the assessment requires that examinees have basic knowledge of fundamental scientific concepts and cause-and-effect relationships in technological systems. The items also require a fairly high degree of reading skill, which may pose challenges for examinees learning English. The scenarios presented

OCR for page 265
Tech Tally: Approaches to Assessing Technological Literacy This page intially left blank