who

I guess rsync can be told to remove removed files on the destination, too?

Yes. The --delete family of options are relevant here.

Git is for text files and retaining a history of every change and every state that has ever existed. It is the wrong tool for what you want, because it would be wasteful of resources.

I suggest automating lossy encodings locally (there are quite a few approaches you could use here, such as a cron job with the encoder of your choice), and automating an rsync job to keep your server updated.