The problems that you are having looks like a triangulation problem. In this book,
Mathematics for 3D Game Programming and Computer Graphics - Eric Lengyel chapter 9, you can find information that can be usefull for you. Here it is a screenshot of the chapter.
