Yeah, downvotes 100% aren’t private on Lemmy, because federation requires the ability to track votes in order to prevent a user from voting multiple times. There are even some mobile apps that allow you to view who has up/downvoted something. Off the top of my head, I think Sync For Lemmy and Friendica both allow you to inspect votes.