I mean if all someone does is go to communities to downvote every single post, it is clear that they are having an agenda, rather than interacting with the content in a fair way.

As there is an algorithm pushing things to “hot” or “active” in the local and all feeds based on votes, it does have an influence on the visibility of posts.