What's wrong with my normal map implementation?

I'm trying to write my own 3D engine from scratch in C, and right now it can render spheres, but I wasn't able to perfectly implement norl maps.

All of the resources I've found online about normal maps are about applying them to meshes and talk about making sure yso I just kind of guessed the right way to do it for a sphere. The basic algorithm I'm using to perturb the normal vector is:

struct _Vec3 {
    double x;
    double y;
    double z;
};

struct _Rgb {
    unsigned char r;
    unsigned char g;
    unsigned char b;
};

void adjustNormal(Sphere* sphere, Vec3* point, Vec3* normal) {
    double* arr = getSphereCoordinates(sphere, point); // get coordinates of point, where 0 < x, y < 1
    double x = arr[0];
    double y = arr[1];
    Rgb* color = getPixel(sphere -> normalMap, x, y); // get the data from normal map
    x = (color -> r) / 255.0;
    y = (color -> g) / 255.0;
    double z = (color -> b) / 255.0;
    Vec3** ortho = getOrthogonalVectors(normal); // basis of tangent space
    Vec3* tangent = ortho[0];
    Vec3* bitangent = ortho[1];
    scaleVec3(normal, z);    // scale normal vector by z
    scaleVec3(tangent, y);   // scale tangent vector by y
    scaleVec3(bitangent, x); // scale bitangent vector by x
    normal = add3(normal, tangent, bitangent); // add together three vectors
    normalize(normal);
}

Now, I feel pretty confident that getSphereCoordinates and getPixel works because applying a regular texture (not normal map) to a sphere works fine. I'm less confident about getOrthogonalVectors because even if I change that, I still have an issue with the normal map. Just in case, here it is:

Vec3** getOrthogonalVectors(Vec3* vec) {
    Vec3 temp;
    copyVec3(&temp, vec);
    temp.x += 10; // no matter how I change
    temp.y += 1;  // this to get a nonparallel vector,
    temp.z += 1;  // the normal map isn't correct

    Vec3* v1 = cross(vec, &temp);
    Vec3* v2 = cross(vec, v1);
    scaleVec3(v2, -1); // necessary for orientation

    normalize(v1);
    normalize(v2);

    Vec3** ans = malloc(sizeof(Vec3*) * 2);
    ans[1] = v1;
    ans[0] = v2;
    return ans;
}

Below is the normal map being used and what the sphere looks like after the map is applied.

This picture is with the light source being at the same location as the camera, so in theory the entire front of the sphere should be lit. Is there something I'm doing wrong?