I've had the same comment surface in several, unrelated conversations I've had with colleagues lately. All math teachers. In each case we were discussing some aspect of the curriculum and at one point they invariably say: "Y'know, you just have to memorize that."
Driving my son and a friend of his home yesterday we were talking. They started talking about tests they have coming up in various classes. They listed those classes where "you just have to memorize that stuff." Again, math was one of those classes.
So really, what do we absolutely have to memorize in math? I do not teach memorization, although I too have told my students that they have to commit certain things to memory. In each case I emphasize they should not memorize individual facts, rather, they should identify patterns and recall the patterns. I teach mnemonics. (Is that splitting hairs?) For example, how many patterns can you find in these two columns of numbers?
So really, what do we HAVE to memorize in math?