The problem your having is called gimble lock. I think what your looking to do is called arcball rotation. The math for arcball can be abit complicated.
A simpler way of doing it is finding a 2d vector perpendicular to the 2d swipe on screen.
Take the vector and project it onto the camera near plane to get a 3d vector in world space. Screen space to World space.
Then create a quaternion with this vector and multiply it to gameobject. Probably with some slurp or lerp transition. –
Edit:
Unity Example: In the unity example, the internal state of the gameobjects rotation is a quaternion not a matrix. The transform.rotation method generates a quaternion based on the vector and angle provided and multiplys that quaternion with the gameobjects rotation quaternion. It only generates the rotation matrix for rendering or physics at a later point. Quaternions are additive and avoid gimble lock.
Your Code Should look something like this:
private float[] rotationMatrix() {
final float[] xAxis = {1f, 0f, 0f, 1f};
final float[] yAxis = {0f, 1f, 0f, 1f};
Quaternion qY = Quaternion.fromAxisAngle(yAxis, angleX);
Quaternion qX = Quaternion.fromAxisAngle(xAxis, -angleY);
return (qX * qY).getMatrix(); // should probably represent the gameobjects rotation as a quaternion(not a matrix) and multiply all 3 quaternions before generating the matrix.
}