In the work that I’ve done for 1 Across magazine, you’ll find all manner of tricky theme devices: mystery words, partial definitions, long key phrases, links between clues and structural methods (I think I may have invented a new form in the Alphabetical Takeaway, in which a different letter of the alphabet is ignored in the wordplay for each clue) … anything goes, in other words, and—crucially—you have to solve the theme in order to complete the puzzle.
Now, I like compiling that way, but it’s a high-risk approach: make a key phrase too hard, and too many people will give up without finishing the puzzle; too easy, and half the puzzle is done in one go. For weekday puzzles, in particular, it’s just not fair.
When I started compiling for the Guardian, I wanted to do something in which the theme adds to the fun (that’s from my point of view, as well as yours) without getting in the way of finishing the puzzle. From that, my style developed into the strange thing that you now know.
Look carefully, and you may see a subdivision between two types of theme: those that explore related words (milk products or football, say) and meditations on an individual word or phrase (“head” or “in a sense”, for example).