|
"Basic Statistics
for R&D", a report written by Dr. Jay Apt, has been graciously
provided in PDF form by Bill Spadafora of NARTS.
It is useful in conjunction with Joyce
Guzik's article.
The following article was written for the March/April
2004 issue of Sport Rocketry Magazine (copyright 2004). It is used with
permission.
R&D Advice
by
Joyce Ann Guzik, NAR 37191
Editor's note: Joyce has
judged the R&D event at 13 NARAMs. She is a staff scientist at Los
Alamos National Laboratory and holds a Ph.D. degree in Astrophysics.
This year at NARAM-50, cash prizes of up to $1000 for first place are
being awarded to B-division winners of the Research and Development
competition. Since the stakes are high, we thought it would be good to give
you an idea of what the judges are looking for, and conversely, what the
judges have often found lacking about R&D reports, and also give you
some useful tips and advice.
First, it is important to read the rules in the 'Pink Book' on this
event that can be found at www.nar.org/pinkbook. You must submit three
copies of your R&D report for judging. The rules give a list of
elements that must be included in a report in order to qualify for the
event, such as stating how much it cost to do your project. You must also
submit a separate 250-300 word written summary of the report. You must be
prepared to give an oral presentation before the judges if your project is
in contention for 1st through 4th place.
Getting an idea for a project
According to the Pink Book, an R&D project is supposed to either
advance the state of the art of model rocketry, or use model rocketry as a
research tool. Follow your curiosity. A good project has an idea that shows
creativity and originality, is feasible to execute in a reasonable amount
of time, and (sometimes fortuitously) leads to a result that is useful, or
advances everyone's understanding.
Is there something that has been mysterious, that you have always wanted
to understand? Have you wanted to improve the performance of your rocket?
Is there something that has always annoyed you about your rocket (misfires,
the way your parachute tangles or won't deploy, red barons, not being able
to find it after launch, the difficulties of altitude tracking, lack of
guidance, safety concerns, etc.) for which you might have a novel solution?
Is there something that you could use model rocketry for (e.g., taking
atmospheric science data, taking aerial photos for a particular use,
developing a science curriculum to teach some aspect of physics or math or
chemistry, etc.)? For examples of what R&D projects have been done in
the past, you can search the World Wide Web, NARTS publications, yearly
NARAM issues of Sport Rocketry, and ask other rocketeers.
Start early!
Don't plan on doing your project a few days before NARAM! The winning
R&D projects in the upper divisions often take months to over a year to
complete, with multiple aspects/parts/approaches. Building your test
rockets or equipments, testing, allowing time for failures and revising
your approach, and getting a statistically significant amount of quantitative
data from experiments and flight tests (see below) all take time.
Keep a logbook chronicling your progress, ideas, data, and observations.
It will be useful when it comes time to write your report.
The written report
Be sure that everything listed in Pink Book rule 63.5 is included in
your report. Your entry could be disqualified if it does not contain all of
the required elements in some form. Include an introduction about why you
chose this project and what was your initial (and sometimes evolving)
objective for the project.
A written summary of your report is also required. This is not simply a
repeat of the introduction, or an advertisement for the rest of the
project-it is a succinct summary of your objective, a brief statement of
the means you used to meet it (e.g., calculations, flight tests,
engineering tests, or other data gathering), and a statement of the
conclusions and significant results (What did you learn? Did you meet the
objective?). If a reader had time to read only your summary, what would you
want him/her to know about your project?
Keep a bibliography and list the sources that you used for reference. Do
go to the library (surprisingly, there is stuff there that isn't on the
web!), look at the web, and ask other experienced modelers. But make sure
your writing is original-DO NOT cut and paste entire paragraphs from
reference sources on the web or from books unless you have a purpose for
them in the context of the report. Material taken from a source should
appear within quotes in your report, and a reference to the original source
should appear in your bibliography. Even if you paraphrase material, or put
it in your own words, you need to reference the original source. Also,
stick to the point of your project, and do not include extensive irrelevant
information or descriptions. You will not earn extra credit for padding out
your project with more pages.
Periodically during your preparations, you should try to step back and
think carefully about your experimental setup and think about what else
could account for or affect your results (was Rocket B heavier than rocket
A by a little? Did Rocket B have slightly crooked fins and so not fly as
straight or have more drag?). Include these thoughts in your report. Even
if you don't know how to correct for such systematic errors, just doing
some thoughtful analysis and itemizing other possible causes for the
results will earn you the esteem of the judges!
It would also be good after the conclusions in your report to include a
retrospective on what you would do differently if you could start over,
your thoughts on follow-up work that could be done to improve the project,
or other ideas that this project inspired.
Please have several other people read your report before you enter it in
the R&D event. Have them make suggestions for improvements. Find out
what things they thought were confusing or missing. Also, be careful to
eliminate all of the grammar and spelling mistakes (a good proofreader
helps here too!).
The significance (or lack thereof) of
your results
One thing that the judges are especially looking for is that the
contestant is not drawing incorrect or unsupported conclusions from their
observations and experimental results.
This can arise for two reasons. First, the contestant may not have
enough controls or checks in his/her experimental process. For example, if
you are doing a lot of flight tests with several different models, you need
to check for and record possible differences between the models. Do they
have the same mass, surface finish, fin area, frontal area, etc.? Also,
follow procedures that help to 'randomize' the effect of influences that
you may not consider or can't control. For example, if you are comparing
the performance of A vs. B engines using what you think are two identical
models, don't fly all of the A engines in one model, and all of the B
engines in the other model; divide the two types of engines evenly between
the models. Similarly, if you are comparing two different types of models,
don't make all the flights of model #1 in the cool, clam morning, and then
make all the flights of model #2 in the hot, windy afternoon; mix the
flights together so that external changes affect both groups equally.
The second reason people draw unsupported conclusions is that they don't
consider whether any differences in their results are statistically
significant or not. For any quantitative data it is a must to have some
statistical analysis. Most importantly, you need to have enough trials of
an experiment to get statistically significant results. One to three trials
is much too few to get significant results; ten to twenty would be much
better. At a minimum, you should calculate the 'mean' (average) and the
'standard deviation of the mean' ( ) for all sets of experimental data you want to compare. Note
that is NOT the
standard deviation of the data sample, but the standard deviation of the
mean of the data sample (which is the standard deviation of the sample
divided by the square root of the number of trials in the sample).
The short technical report Basic Statistics for R&D by NASA
astronaut and rocketeer Dr. Jay Apt, available from NARTS
(www.nar.org/narts), discusses in detail the calculation of mean and
standard deviation of the mean. This is an estimate of how far your calculated
mean is likely to differ from the true mean that would be found by doing a
very large number of trials. There is a 68% chance that the true mean lies
in the range +/- on either side
of your sample mean; the chance is 95% that it's in the range +/-2.
If the error ranges of
two sets of experimental data overlap, then you cannot claim there is a
statistically significant difference between the two sets. For example, as
Dr. Apt points out, if you have a set of trials of rocket A with an average
altitude 320 +/- 20 meters, and a set of rocket B with 310 +/- 20 meters,
you cannot conclude that rocket A reaches higher altitudes than rocket B!
Including more trials in a data set gives a narrower error range for and can result in you being able to
determine a statistically significant difference that fewer trials will not
reveal.
Figure 1. Rocket data example spreadsheet
Rather than duplicate Dr. Apt's report by
demonstrating how to calculate the mean and  for your data, we
show you how to do it in an Excel spreadsheet (see Figure 1). The bottom of
the figure shows what formulas are in the cells of column B (the same
formulas are copied across into column C). The "track lost" entry
in cell C6 is there to show that the formulas ignore cells that are blank
or contain non-numerical data.
The spreadsheet shows two sets of rocket altitude measurements and the
mean (average) of each set. But is there a significant difference between
the means? Using the values from row 10, we can
see that the error ranges 191.0 +/- 5.2 and 202.0 +/- 4.7 do not overlap,
so the data are significantly different (at least to 'one sigma'-one
standard deviation of the mean). The difference in the results would be
more convincing if the data were different at the 'two sigma' level (95%
confidence) but they are not, since doubling the error ranges causes them
to overlap. By the way: if the we only had the first three flights in each
set, the results are 191.7 +/- 11.1 and 204.3 +/- 8.0, which does not show
a clear difference between the two sets since those ranges overlap.
The Student's t-test formula in cell B13 is another statistical test
that Excel can perform. The result shown tells us that there is only a 13%
chance that the two data sets come from the same population, another
indication that the data are significantly different.
Oral Presentation
In addition to the written report, the contestants that are in line to
place must do an oral presentation in front of three NARAM R&D judges
and a friendly audience of the interested NARAM attendees, and answer
questions following their talk from the judges and audience. The
presentation can significantly affect the judges' opinions, but remember
that you need to do well enough to impress the judges on the written report
to earn the chance to present the oral in the first place!
Prepare your oral talk well: Have vugraphs (transparencies) summarizing
your report (proofread them!), and use large font sizes for the text so
people in the back of the room can see them. If you are using a computer
projector, the same applies-but have transparencies as a backup. Have some
props as well (samples of your rocket, apparatus, etc.) to show the judges
and pass around. Practice in front of your family and friends two or three
times before coming to NARAM and ask them for their honest constructive
criticism regardless of how much it hurts your ego!
The R&D rules state that the oral presentation shall not exceed 15
minutes in length, but the constraints of time at NARAM usually lead to the
judges to request talks with a length of 5 or 10 minutes, so you should be
prepared for either length. Contestants that go on past the time limit may
be cut off from necessity without finishing. One viewgraph per minute is a
good rule of thumb. You could have an additional 5 minutes of material in
reserve if it turns out there is time for a longer format. The judges and
audience will not interrupt you with questions during your presentation, but
there will typically be 5 minutes of questioning after you have finished
your talk. Practice answering questions from your family and friends.
The judges may ask you to demonstrate some aspect of your project on the
field on Friday, so be prepared for this if you think your project might
warrant it.
Good luck-we are hoping to see many outstanding projects at NARAM!
[end of Joyce's article]
return
to top
|