glEnable(GL_CULL_FACES);

Simple one liner that applies back face culling. This is excellent for trimming down polygon count. Also look into Frustum/Occlusion culling for reducing poly counts.

Another thing to look into is SIMD for your vectors and quaternions (mostly used in camera rotations). If you don't want to mess with ASM then check out libSIMD which is a great library for handling SIMD in C.