Video data basics

(Digital) video data is comparatively complex. A file that allows you to play a video with sound on a computer or other device contains different types of data: Parts of the data are used to display moving images on the screen, while other parts of the data are used to play sound through the speakers of your device. Both has to happen synchronously. You can easily come up with examples with even greater complexity, such as video and audio accompanied by captions, or a movie which has been dubbed and allows to switch between different audio languages.

The high complexity of time-based audiovisual media is exacerbated by the high variability in technical solutions and standards for storing digital A/V data. This page collects some resources to provide an overview over file and encoding standards. Software like ffmpeg, Handbrake or VLC can convert video data between different formats to varying extents/degrees of success, but the results can vary widely based on the parameters chosen for a converting or transcoding data.

Media container formats (file types)

Both audio and video files are often referred to by their file name extension, or file type, for instance MP4, MP3, WAV, etc. It is important to understand two things:

  1. A file name extension is just a matter of convention and convenience, and it is only indicative of the type of data a file contains; the file name in general is just a label. No common operating system strictly enforces the use of file name extensions: it is very easy to simply rename a file to bear an .MP4 extension; that does not magically turn a text file into a video (to use a kitchen metaphor: If I made strawberry jam, fill it in a jar and put a label on it that says 'blood pudding', the jam doesn't actually turn into something else). Only Windows appears to use the file name extension to decide which program to use when opening a file. 
  2. The file type indicated by the file name extension can refer to either the main type of data contained in the file, or a type of container, or both. Examples: An MP3 file is in fact an MPEG-1 file container, storing (only) audio data that was compressed using the MPEG 1/2 Audio Layer III (MP3) encoding scheme. MP4 'video files' are container files conformant to MPEG-4 specifications, which can contain different combinations of video and audio data streams, using various encoding/compression schemes (including, for instance, MP3-compressed audio). 

For a good overview over Media Containers currently supported by modern browsers, see https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Containers

Commonly found container formats for video data are: MP4 (actually MPEG-4 Pt. 14, typically using .mp4 file extension), QuickTime (typically using .mov, .qt), AVI (Audio Video Interleave), Windows Media (Advanced Systems Format, typically using .wmv, .asf), and more recently Ogg (typically using .ogv) and Matroska (typically using .mkv).

Encoding image data for digital video

'Video' is essentially a series of images played in rapid succession, to give a human spectator the impression of motion (motion picture, moving image). For a movie, this typically amounts to 25 images ('frames') per second (frame rate). (The story is more complicated for TV, for the sake of this explanation we will stick with the analog motion picture example.)

Mozilla Developer Network page on color encoding and chroma subsampling in digital video: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_concepts

"A short guide to choosing a digital format for video archiving masters": https://web.archive.org/web/20221128105156/https://scart.be/?q=en/content/short-guide-choosing-digital-format-video-archiving-masters

Storing "uncompressed" video data, instructive overview over common color subsampling methods: https://superuser.com/a/347434 

See https://en.wikipedia.org/wiki/Video_file_format for a list of Video file formats (= combinations of container formats with one or more codecs)