Maybe you can use a code for your town, so you only play with players who know the code. Maybe use something that needs different keyboard like ä ö ü or characters that are not often used in a language like:
Code for a japanese town: ヰ
Code for a chinese town: 爨
So each time you get a baby, ask for the code. If the player dosen’t give you the right code, let them die.
Once i had players i raised, but then they used a code. So in the end i couldn’t join anymore, because i didn’t had the code to my own town
Also use the curse button, so you wont get them as baby anymore or as mother anymore.