Let's get to roots of SCM - it is for sources. Now sources are not only text files, they are everything that you use as a source (source codes, scripts, PSD files, sounds). Everything that can be repeatedly generated are not sources (spritesheets, executables, resource packages).
You need to store all sources in one repository for the sake of consistency. Because each and every revision ought to be self-sufficient. You also need to have build scripts to produce application and all its assets from sources. Builds are not stored in SCM because you can always regenerate them.
EDIT: Once you put size out of the question, it makes sense to keep all sources in SCM because it's quite often when engine behavior depends on lookup textures, models sizes and shapes, assets count and location (also think TDD). When you have source codes revision A you ought to be very confident in your engine to allow it to work with any other assets revision. Otherwise you need to make a tie that code A works only with assets A, B-B, C-F etc.. and then when you rollback the code you need to rollback assets as well. That ruins SCM idea.
From practice: we were storing paletted textures, but since r2879 we switched our format to full-color textures. Having source textures (some unchanged since r1000) and generation scripts under SCM we don't have a problem of finding right assets/exe combo.