[ww-devel] Big Data Table

[Geoff Goehle]

The "Big Data Table" is my name for the main table for the data mining
stuff that is connected to the grant extension.  Basically the idea is
to get a bunch of answer data from various universities and see if we
can learn anything from it.  

I suppose I should back up a bit and ask a question I thought I knew the
answer to.  Are we designing the table to with the data we want, or are
we designing the table to make use of the existing past answers
databases?  If we are designing a new past answer table then the
question is what fields do we want to store.  I already now know that we
want to store the problem seed, the status of the attempt, and
fractional credit for individual answers.  We could also get demographic
information and store it somewhere, both at the university/instructor
level and at the student level.  One issue with this approach is that we
would design the new past answer tables, and then we would roll them
out, and we would have to wait a semester (or probably more) before we
had data to analyze.  

The second approach is to make use of the past answer tables that are
already sitting on university servers.  This lets us start collecting
data right away, but limits us to the data we have been collecting.
E.G. we can't use fractional answers, we will have to post-compute
status' of individual attempts (which could very well be too expensive),
and demographic data would have to be entered manually (which probably
rules out student demographic data).  

Of course there is no reason we couldn't pursue both.  However if we do
that I would start with the existing data so we can really understand
what the deficiencies are before we build a new table.  

Geoff.