First create a dummy group with no users in it
<record id="make_invisible" model="res.groups">
<field name="name">Invisible</field>
</record>
Replace all users in the group. The eval statement here means, replace all ids in the groups_id field with the id for group make_invisible.
<record model="ir.ui.menu" id="module.menu_name">
<field name="groups_id" eval="[(6,0,[ref('make_invisible')])]"/>
</record>