A little while ago I reviewed a manuscript that had somewhat obviously missed my published work on a similar topic. There were other things afoul with the paper, so I wrote a referee report stressing those other issues without mentioning my work. Much to my horror, the other referee report was a rather ill-tempered negative recommendation that must have mentioned the failure to engage my work a half dozen times.
Guessing who wrote that negative review of your work is a time-honored tradition in academia. Moreover, people often seem very confident that their guesses are correct. I can’t remember how often I have heard assertions of the type “This referee told me to cite X more so it must have been written by X.” Yet, many referees may be like me and write their referee reports precisely to avoid this kind of association. Indeed, Lee once told me that when he was the editor of the American Political Science Review he received many letters from authors claiming that referee Y had long held a grudge against him. Yet, almost without exception those letters wrongly guessed the identity of the anonymous referee. When you think about it, this is not so surprising given that referee reports are largely unobserved so we know very little about how people approach this task.
Guessing the identity of anonymous referees just seems like an activity with very little upside. If you guess wrong (which you are likely to do despite your convictions to the contrary), you may wrongly believe that someone is “against you.” You will never know whether you have guessed right and even if you have, how useful is that information really? Obviously, people will continue to do it anyway. All I can say is that you should leave open the possibility that you are wrong even if your identification of the referee seems obvious given your working assumptions about how referees write their reports.