Monsters are distinguished from other creatures using two primary criteria: dissimilarity to humans and lack of sentience or obvious intelligence. Because of this, some sentient races are considered monsters because they are not humanoid and some decidedly non-humanoid creatures are considered sentient races.
Geralt has his own set of guidelines for judging what constitutes a monster. One of his tests is their reaction to silver. Most magical beasts cannot bear it. Other metrics he uses include intent and circumstance. Overall, the witcher is far more fair in his classification of creatures into monsters and non-monsters.
Even in the game, Geralt must use his judgment to decide what or who is and is not a monster. A list of game monsters can be found here.