You can attach the following to posts:
Videos (MP4, M4V, MOV, WebM) up to 40MB. Video will be transcoded to H.264 MP4 with a maximum bitrate of 1300kbps and framerate of 60fps.
Audio (MP3, OGG, WAV, FLAC, OPUS, AAC, M4A, 3GP) up to 40MB. Audio will be transcoded to MP3 using V2 VBR (roughly 192kbps).
Images (PNG, JPG, GIF) up to 8MB. Images will be downscaled to 1.6 megapixels (enough for a 1280x1280 image). Up to 4 images can be attached.
Animated GIFs are converted to soundless MP4s like on Imgur/Gfycat (GIFV). You can also upload soundless MP4 and WebM, which will be handled the same way.
More info here: https://docs.joinmastodon.org/user/posting/
@support HECV (h.265) is also a very efficient codec. But I don't know how difficult it is to convert videos to this format, I have only made it local.