GoodProblems

Heuristics

Problems have a clear sense of what they are trying to do (e.g., develop skills, develop understanding, evaluate student understanding, etc.)
Problems follow Best Practices
Problems have "nice enough" numbers
The problems are clean and clear (and well-written)
- It is clear from the problem what answer is expected of the student
The concepts that are being communicated and evaluated are clear
They have hints and solutions: support for students who are stuck or who lack other support structures
The written solutions inform the manner in which the problem is framed
They are stable and well tested
The problem has a clear learning objective, e.g., as a COMMENT()
- The problem is written to promote students' accomplishment of the learning objective
Problems are accessible to screen readers and other accessibility tools
- The problems provide a good idea of what is being asked when a hardcopy is generated (drop down messages, colors on graphs, graph scaling, and table size)

Related Ideas

Add some sort of ranking system in the NPL
NPL branching: should/could there be a curated version of the NPL that would try to eliminate duplication? What standards could we use? (This list of heuristics, or a rubric score?)

Heuristics, Reorganized

Category	Considerations
WeBWorK Coding	Problems follow Best Practices They are consistent over possible randomizations, and well tested Problems are accessible to screen readers and other accessibility tools The problems provide a good idea of what is being asked when a hardcopy is generated (drop down messages, colors on graphs, graph scaling, and table size)
Technical Issues	Problems have "nice enough" numbers The problems are clear and well-written, and it is clear from the problem what answer is expected of the student They have hints and solutions: support for students who are stuck or who lack other support structures
Learning	Problems have a clear sense of what they are trying to do (e.g., develop skills, develop understanding, evaluate student understanding, etc.) The problem has a clear learning objective, e.g., as a COMMENT() The problem is written to promote students' accomplishment of the learning objective The written solutions inform the manner in which the problem is framed

Problem Type	Problem Should Have	Problem Should Not Have
Skill/practice (e.g., find derivative)
Conceptual/multipart

Alternate formulation

Problem Type	Problem Should Have	Problem Should Not Have
Fill in the blank/Select an answer
Numerical
Formula