what I would do is take a risk and have more extras than necessarily needed, and if someone repeatedly disobeys your direct orders and ruins takes, kick them out. prove a point.
you also need to be ruthless-- yell “quiet on set”, command attention, glare down people who are talking, tell them the sooner they shut up the sooner everyone can leave. you don’t have to be harsh about it if you don’t want to be, but you need to make it clear that this is your set and they play by your rules. if they don’t listen to that, switch people around in the crowd. if it’s the same two people having a conversation, separate them. if the conversation persists (one of them keeps talking), call that person out by name or specifically pull them aside/have someone pull them aside. if they’re publicly shamed for not listening, they’ll probably feel guilty and start listening.