By Robert Krampf
Over the past few weeks, I have discovered some major scientific errors in the guidelines that are used to develop questions for the fifth and eighth grade Science FCAT tests.
The Science FCAT is Florida’s high stakes test that assesses all the science concepts and information that students should have learned by the end of fifth grade. Schools and districts are subject to financial incentives or penalties, depending on their students’ FCAT scores, so this is a very important test.
A few weeks ago, I started developing FCAT practice questions to help students review concepts and prepare for the test. To develop those questions I used the Florida Department of Education’s FCAT 2.0 Science Test Item Specifications. These documents are used as “a resource that defines the content and format of the test and test items for item writers and reviewers.”
I expected the Test Item Specifications to be a tremendous help in writing simulated FCAT questions. What I found was a collection of poorly written examples, multiple-choice questions where one or more of the wrong responses were actually scientifically correct answers, and definitions that ranged from misleading to totally wrong.
I suggest that you read over the entire document, but here are a few of the problems that I found in the FCAT 2.0 Science Test Item Specifications for grade 5.
A glossary of definitions (Appendix C) is provided for test item writers to indicate the level of understanding expected of fifth grade students. Included in that list is the following definition:
Predator—An organism that obtains nutrients from other organisms.
By that definition, cows are predators because they obtain nutrients from plants. The plants are predators too, since they obtain nutrients from decaying remains of other organisms. I have yet to find anyone who thinks that this is a proper definition of a predator. In the same list we find:
Germination—The process by which plants begin to grow from seed to spore or from seed to bud.
Sample Item 2 for SC.5.N.1.6 (page 32), which assesses the following benchmark.
SC.5.N.1.6: Recognize and explain the difference between personal opinion/interpretation and verified observation.
This sample question offers the following observations, and asks which is scientifically testable.
- The petals of red roses are softer than the petals of yellow roses.
- The song of a mockingbird is prettier than the song of a cardinal.
- Orange blossoms give off a sweeter smell than gardenia flowers.
- Sunflowers with larger petals attract more bees than sunflowers with smaller petals.
The document indicates that 4 is the correct answer, but answers 1 and 3 are also scientifically testable.
For answer 1, the Sunshine State Standards list texture as a scientifically testable property in the third grade (SC.3.P.8.3), fourth grade (SC.4.P.8.1), and fifth grade (SC.5.P.8.1), so even the State Standards say it is a scientifically correct answer.
For answer 3, smell is a matter of chemistry. Give a decent chemist the chemical makeup of the scent of two different flowers, and she will be able to tell you which smells sweeter without ever smelling them.
While this question has three correct answers, any student that answered 1 or 3 would be graded as getting the question wrong. Why use scientifically correct “wrong” answers instead of using responses that were actually incorrect? Surely someone on the Content Advisory Committee knew enough science to spot this problem.
For this one, you have to go to the document and scroll down to Sample Item 7 for SC.4.E.6.2 (page 42)There is nothing in the drawing or written information that indicates if the square object is a streak plate (to test streak) or a glass plate (to test hardness.) Scratching a glass plate is one of the most common tests for hardness, and it appears as a graphic or photograph in most textbook units on minerals. C would be just as valid an answer as B, but a student that answered C would be graded as giving a wrong answer. This flaw could have easily been avoided by simply not listing hardness as one of the choices.
These are just a few of the problems that I found. I contacted FLDOE’s Test Development Center, and sent them a list of the errors. Their response for error after error was: “This item was reviewed and deemed appropriate by our Content Advisory Committee.”
I asked for contact information of someone from the Content Advisory Committee, so I could find out how these errors made it past scientific review. Steve Ash, Executive Director of the Test Development Center, told me that FLDOE would not give out that information.
Even more troubling was their response to the example questions that had more than one correct answer. In response to the example above for SC.5.N.1.6, Christopher Harvey, the Mathematics and Science Coordinator at the Test Development Center told me: “We need to keep in mind what level of understanding 5th graders are expected to know according to the benchmarks. We cannot assume they would receive instruction beyond what the benchmark states. Regarding #1 – While I don’t disagree with your science, the benchmarks do not address the hardness or softness of rose petals. We cannot assume that a student who receives instruction on hardness of minerals would make the connection to other materials. The Content Advisory committee felt that students would know what flowers were and would view this statement as subjective. Similarly with option 3, students are not going to know what a gas chromatograph is or how it works. How a gas chromatograph works is far beyond a 5th grade understanding and is not covered by the benchmarks. As you stated most Science Supervisors felt that student would not know this property was scientifically testable. The Content Advisory Committee also felt that 5th graders would view this statement as subjective. We cannot assume that student saw a TV show or read an article.”
The response to my comments on Sample Item 7 for SC.4.E.6.2 (page 42), I was told: “Here again I don’t disagree with your science; however, elementary educators consistently told us that glass plates are not used in elementary classrooms for safety reasons. They did not feel that 5th graders would be familiar with using glass plates to test hardness.”
So according to the Test Development Center, it appears that it is acceptable to use scientifically correct answers for wrong responses on the Science FCAT as long as FLDOE does not expect a fifth grader to be educated enough to realize that the wrong answers are scientifically correct.
I wonder how many students got “wrong” answers on the FCAT because their teachers taught them too much. How many “F” schools would have higher grades if those scientifically correct “wrong” answers were counted as correct answers. How many “B” schools would get the extra funding that “A” schools get, if those scientifically correct “wrong” answers were counted as correct answers?
We may never know the answers to those questions. The Test Item Specifications are the guidelines that are used to write the test questions. If the Science FCAT test is reviewed by the same Content Advisory Committee that reviewed the Test Item Specifications, then it probably has similar errors. But as much as I would love to check the accuracy of the questions from the actual Science FCAT, I can’t. Teachers, scientists, and the general public are not allowed to see actual test questions, even after the tests have been graded and the penalties for those grades have been imposed.
Robert Krampf, a long-time science educator, is based in St. Augustine and writes The Happy Scientist blog, where this piece originally appeared. Reach him by email here.
Kyle says
This is unacceptable, penalizing children who are ahead of the curve is paramount to penalizing an athlete for being more skilled than his peers, which I may add would never happen in American society. What is most troubling is the complete lack of accountability the system seems to have in place. Science is by definition “the systematic knowledge of the physical or material world gained through observation and experimentation”, to disregard those simple principles when developing a test shows an utter disregard for the subject at hand. This is very disturbing indeed and is helping me make my decision to home school my children much easier.
Samuel Smith says
I’ll preface this by saying that I have an advanced degree in a hard science.
That said, I’m looking at appendix C of this document, that provides terms and definitions for grades 3-5, and most of them are ridiculously wrong. Here’s an example of a few:
Absorb—To take up and store energy without reflecting or transmitting that energy.
Wrong. Firstly, absorb isn’t confined to energy, and secondly just about every object in nature routinely absorbs, transmits, and reflects at the same time. Glass, for example, does all three.
Asteroid—An object that is found in the solar system, orbits the Sun, and is much smaller than a planet.
Wrong. This includes things that distinctly aren’t asteroids, e.g. comets, satellites, and so on.
Attraction—A term used to describe the magnetic force exerted by oppositely charged objects
or to describe the gravitational force that pulls objects toward each other.
Wrong wrong wrong wrong. Oppositely charged particles attract via the electrostatic force, not the magnetic one.
Comet—An object made of rock, ice, dust, and gas that revolves around the Sun.
Oh you mean a planet, right? Wrong.
Conduct—To transmit heat or electricity through a medium.
Wrong. In heat, conduction only works in solids via contact.
I could go on for hours with this.
Broccoli Rob says
I have compiled the Florida definitions of some words you mention here.
*Asteroid: Things you develop at the end of the alimentary canal due to lack of fiber in your diet.
*Comet: An abrasive cleanser.
*Conduct: Something a kid does in school that’s bad.
Add these to the list:
*Disfiger (sic) I don’t know what that is. Saw that in a comment yesterday.
*Douch Bag (sic): Ditto, authored by a lobotomized person yesterday.
*Principle: Wrong spelling for “principal.”
*Rederick (sic): Flagler county rhetoric.
Yellowstone says
I, too, had an assignment to review a series of elementary math questions (and answers) on a placement test.
It was a illuminating experience once I got started at how brilliant these young minds can be. The problems and questions presented gave faith to me these kids are really learning something.
It wasn’t long before I got to a series of problems that dealt with two separate drawings – one on one page – the other on the backside of the same page.
Not knowing the the second figure even existed I was faced with a time/distance problem and was refered to the drawing via the figure number beneath.
I felt stupdified! With a major in mathematics – I couldn’t answer the question! New Math? I thought. Trick question – I resolved. After stumbling around I simply turned the page and started the next question.
Again, “refer to drawing Figure x-x and answer the following questions:” Again, I stumbled. More New Math?? Wow, was I out of it.
Then the lights came on. The figure numbers were in correct sequence – but the drawings were interchanged. Ha, the joke was on me.
After reviewing the test, I mentioned it to the administrator. “Kids get it”, she said with a shrug of her shoulders.
New Math? Trickery? No, extremely poor editing – both in wording and arrangement.
How can a significant test pass through so many intelligent hands and be so lacking?
NortonSmitty says
I posted this late on another thread so many of you may not have seen it. It amazed me. An 8th grade final exam from 1895:
What it took to get an 8th grade education in 1895…
Remember what our grandparents and great-grandparents stated that they only had an 8th grade education? Well, check this out. Could any of us have passed the 8th grade in 1895?
This is the eighth-grade final exam from 1895 in Salina , Kansas , USA . It was taken from the original document on file at the Smokey Valley Genealogical Society
And Library in Salina , and reprinted by the Salina Journal.
8th Grade Final Exam: Salina , KS – 1895
Grammar (Time, one hour)
1. Give nine rules for the use of capital letters.
2. Name the parts of speech and define those that have no modifications.
3. Define verse, stanza and paragraph
4. What are the principal parts of a verb? Give principal parts of ‘lie,”play,’and ‘run.’
5. Define case; illustrate each case.
6 What is punctuation? Give rules for principal marks of punctuation.
7 – 10. Write a composition of about 150 words and show therein that you understand the practical use of the rules of grammar.
Arithmetic (Time,1 hour 15 minutes)
1. Name and define the Fundamental Rules of Arithmetic.
2. A wagon box is 2 ft. Deep, 10 feet long, and 3 ft. Wide. How many bushels of wheat will it hold?
3. If a load of wheat weighs 3,942 lbs., what is it worth at 50cts/bushel, deducting 1,050 lbs. For tare?
4. District No 33 has a valuation of $35,000.. What is the necessary levy to carry on a school seven months at $50 per month, and have $104 for incidentals?
5. Find the cost of 6,720 lbs. Coal at $6.00 per ton.
6. Find the interest of $512.60 for 8 months and 18 days at 7 percent.
7. What is the cost of 40 boards 12 inches wide and 16 ft. Long at $20 per metre?
8. Find bank discount on $300 for 90 days (no grace) at 10 percent.
9. What is the cost of a square farm at $15 per acre, the distance of which is 640 rods?
10. Write a Bank Check, a Promissory Note, and a Receipt
U.S. History (Time, 45 minutes)
1. Give the epochs into which U.S. History is divided
2. Give an account of the discovery of America by Columbus
3. Relate the causes and results of the Revolutionary War.
4. Show the territorial growth of the United States
5. Tell what you can of the history of Kansas
6. Describe three of the most prominent battles of the Rebellion.
7. Who were the following: Morse, Whitney, Fulton , Bell , Lincoln , Penn, and Howe?
8. Name events connected with the following dates: 1607, 1620, 1800, 1849, 1865.
Orthography (Time, one hour)
[Do we even know what this is??]
1. What is meant by the following: alphabet, phonetic, orthography, etymology, syllabication
2. What are elementary sounds? How classified?
3. What are the following, and give examples of each: trigraph, subvocals, diphthong, cognate letters, linguals
4. Give four substitutes for caret ‘U.’ (HUH?)
5. Give two rules for spelling words with final ‘e.’ Name two exceptions under each rule.
6. Give two uses of silent letters in spelling. Illustrate each.
7. Define the following prefixes and use in connection with a word: bi, dis-mis, pre, semi, post, non, inter, mono, sup.
8. Mark diacritically and divide into syllables the following, and name the sign that indicates the sound: card, ball, mercy, sir, odd, cell, rise, blood, fare, last.
9. Use the following correctly in sentences: cite, site, sight, fane, fain, feign, vane , vain, vein, raze, raise, rays.
10. Write 10 words frequently mispronounced and indicate pronunciation by use of diacritical marks
And by syllabication.
Geography (Time, one hour)
1 What is climate? Upon what does climate depend?
2. How do you account for the extremes of climate in Kansas ?
3. Of what use are rivers? Of what use is the ocean?
4. Describe the mountains of North America
5. Name and describe the following: Monrovia , Odessa , Denver , Manitoba , Hecla, Yukon , St. Helena, Juan Fernandez, Aspinwall and Orinoco
6. Name and locate the principal trade centers of the U.S. Name all the republics of Europe and give the capital of each..
8. Why is the Atlantic Coast colder than the Pacific in the same latitude?
9. Describe the process by which the water of the ocean returns to the sources of rivers.
10. Describe the movements of the earth. Give the inclination of the earth.
Notice that the exam took FIVE HOURS to complete.
Gives the saying ‘he only had an 8th grade education’ a whole new meaning, doesn’t it?!
Smitty again. I bet that most Master Degree grads today would fail this test.