As usual, there is very little documentation in place for such things. The main information is in the POD file for the AnswerHash object. The AnswerHash is the object passed to the filter, and its properties and methods are described at the top of the POD file. Filters are discussed (briefly) near the bottom.
Basically, you get passed the AnswerHash, and can get the data you are interested in out of that object, manipulate it in whatever way you need, and return the modified AnswerHash so that it can be passed on to the next filter.
Earlier filters can modify the data, or add to it (as can the pre-filters and answer checkers themselves). So there may be there data in the AnswerHash that is not described in the POD file. For example, the MathObject answer checkers add the
student_value, student_formula, and correct_value properties, which store the MathObjects for the student's answer and the correct answer. So it is often easiest to just pretty-print the contents of the AnswerHash during your debugging so that you can see what it contains. If you set debug=>1 in the cmp() call (e.g., ANS($m->cmp(debug=>1)))), I think that is done automatically before each filter is run.