Should i go back to the court?

I have custody of my children and their father has visitation rights. On a recent visit to their fathers, the police were called by a neighbour because the children were hanging out of a bedroom window throwing stuff out. When the police arrived, they found dog muck smeared into carpet, filth and mold everywhere. The children were removed into the care of their grandparents until they came back home to me. I was unaware of the state of his home until the children came home and i got told about it. I've tried to be accomodating towards the father in regards to the children, but after this incident, should i go back to the court to prevent them going again? or should i just leave the issue and just let him see his kids at his parents house?