Learning Science Alex Chen

Adaptive concept mapping and why PR grading needs to know what you already know

Static rubrics miss half the signal. The concepts a developer already understands should change what feedback they receive on every new PR.

Adaptive concept mapping and why PR grading needs to know what you already know

When we started grading Go PRs, we used a flat rubric. Every PR got checked against the same list of concepts: error handling, goroutine lifecycle, interface usage, struct design, and so on. The feedback was accurate but frequently irrelevant. A developer who had already written five PRs demonstrating solid error handling kept getting error handling feedback because the rubric didn't track what they'd already demonstrated they knew.

The problem with a flat rubric is that it treats every learner as being at the same position. But developers switching stacks aren't blank slates. A developer with seven years of Python experience already understands function decomposition, separation of concerns, and working with concurrent primitives. They don't need lessons on those concepts in their Go learning path. What they need are lessons on the specific places where Go diverges from Python in ways that will produce bad code if the Python pattern is applied directly.

What a concept map is

A concept map is a graph of skills and their prerequisites. In Go, understanding channel ownership depends on understanding goroutines, which depends on understanding that goroutines are not OS threads and are multiplexed by the Go runtime scheduler. These relationships are real: a developer who doesn't know how goroutines are scheduled will have a harder time understanding why channel semantics matter.

The concept map doesn't need to be exhaustive or formally specified to be useful. It needs to represent the main dependencies clearly enough that a lesson about goroutine lifecycle doesn't get queued before a developer has seen basic channel usage. And it needs to track which nodes a developer has visited so those nodes don't surface again as active feedback targets.

When a PR arrives for grading, the system doesn't check the code against the full concept list. It checks the code against the concepts that are currently active for this developer: the ones they haven't yet demonstrated, or the ones they demonstrated incorrectly in earlier PRs.

Why prior knowledge changes the feedback

A developer who already understands interfaces well doesn't need a comment on their Go PR explaining that interfaces are satisfied implicitly. They need the comment to go one level deeper: if their interface definition is too broad, if it has methods that aren't needed in the current context, if they're passing a concrete type where an interface would make the function more testable.

If the grader doesn't know that this developer already understands the basics, it produces beginner feedback on an experienced developer, which is at best useless and at worst insulting. The experienced developer stops reading the feedback carefully because it doesn't match their level.

On the other side: a concept that looks like an advanced topic might be exactly what this developer needs first, because of something in their prior stack experience. A developer with extensive concurrent programming experience might be ready to engage with goroutine ownership concepts earlier in their Go learning path than a developer who has only written single-threaded scripts. The concept map handles both cases by using what we know about the developer's demonstrated knowledge to calibrate which feedback is likely to be useful right now.

The prerequisite graph in practice

We represent concepts as nodes and prerequisites as directed edges. When a developer receives a lesson on a concept and subsequently demonstrates understanding in a PR, that node gets marked. Downstream concepts whose prerequisites are now satisfied become available as feedback targets.

This means the first few PRs for a developer look very different from later ones. Early PRs produce feedback on foundational concepts: error handling idioms, basic struct design, straightforward goroutine usage. As those get marked, later PRs start surfacing more specific concepts: context propagation, error wrapping chains, channel directions in function signatures.

The developer isn't moving through a fixed curriculum. They're moving through their personal gap list, which gets updated every time they submit a PR.

What static rubrics get right, and where they fail

We're not arguing that rubric-based feedback is wrong. Static rubrics are excellent for surface-level issues: missing error checks, wrong fmt.Errorf verb, unused imports. These are the same for every developer and don't require knowledge of learner state to flag correctly. We still check for them on every PR.

The rubric fails when it's the only mechanism. When every PR surfaces the same list of potential issues without regard for what the developer has already demonstrated, the signal drowns in noise. Developers start ignoring comments because they expect them to be things they already know, and in doing so they miss the comments that actually matter.

Adaptive concept mapping is not a replacement for a good rubric. It's the layer on top of the rubric that decides which of the rubric's findings are worth surfacing right now, for this developer, in this PR.

Where the model is imprecise

To be clear about what adaptive concept mapping doesn't do: it doesn't predict which concepts a developer will struggle with in the future. It only tracks what they've demonstrated and what they haven't. A developer might demonstrate correct goroutine lifecycle usage in five PRs and then produce a goroutine leak on the sixth, because the sixth PR had a different context that revealed a gap in their model.

When that happens, the node reverts to active. The system doesn't penalize or flag regression. It simply notes that the concept is relevant again and queues appropriate feedback. The model is descriptive, not predictive. That's a real limitation and we're honest about it.

What it does well is prevent the most common failure mode of static grading: wasting developer attention on things they already know while the things they don't know get buried in the same feedback list. That waste is what we set out to reduce, and the concept map is the mechanism we built to reduce it.