1

I am wanting to:

  1. Open the texture from an image via cv2 instead of via ModernGL's load_texture_2d method.
  2. Save the resulting image (in the write method) via cv2 rather than Pillow.

I currently have the following code:

from pathlib import Path
from array import array

import cv2
import numpy as np
from PIL import Image

import moderngl
import moderngl_window


class ImageProcessing(moderngl_window.WindowConfig):
    window_size = 3840 // 2, 2160 // 2
    resource_dir = Path(__file__).parent.resolve()

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.image_processing = ImageTransformer(self.ctx, self.window_size)

        # Not working:
        #img = cv2.imread("test6.png")
        #self.texture = img.astype('f4')
        
        self.texture = self.load_texture_2d("test6.png")

    def render(self, time, frame_time):
        # View in Window
        self.image_processing.render(self.texture, target=self.ctx.screen)

        # Headless
        self.image_processing.render(self.texture)
        self.image_processing.write("output.png")


class ImageTransformer:

    def __init__(self, ctx, size, program=None):
        self.ctx = ctx
        self.size = size
        self.program = None
        self.fbo = self.ctx.framebuffer(
            color_attachments=[self.ctx.texture(self.size, 4)]
        )

        # Create some default program if needed
        if not program:
            self.program = self.ctx.program(
                vertex_shader="""
                    #version 330

                    in vec2 in_position;
                    in vec2 in_uv;
                    out vec2 uv;

                    void main() {
                        gl_Position = vec4(in_position, 0.0, 1.0);
                        uv = in_uv;
                    }
                """,
                fragment_shader = """
                    #version 330

                    uniform sampler2D image;
                    in vec2 uv;
                    out vec4 out_color;

                    void main() {
                        vec4 color = texture(image, uv);
                        // do something with color here
                        out_color = vec4(color.r, 0, 0, color.a);
                    }
                """,          
            )

        # Fullscreen quad in NDC
        self.vertices = self.ctx.buffer(
            array(
                'f',
                [
                    # Triangle strip creating a fullscreen quad
                    # x, y, u, v
                    -1,  1, 0, 1,  # upper left
                    -1, -1, 0, 0, # lower left
                     1,  1, 1, 1, # upper right
                     1, -1, 1, 0, # lower right
                ]
            )
        )
        self.quad = self.ctx.vertex_array(
            self.program,
            [
                (self.vertices, '2f 2f', 'in_position', 'in_uv'),
            ]
        )

    def render(self, texture, target=None):
        if target:
            target.use()
        else:
            self.fbo.use()

        texture.use(0)
        self.quad.render(mode=moderngl.TRIANGLE_STRIP)

    def write(self, name):

        # This doesn't work:
        raw = self.fbo.read(components=4, dtype='f4')
        buf = np.frombuffer(raw, dtype='f4')
        cv2.imwrite("OUTPUT_IMAGE.png", buf)

        # But this does:
##        image = Image.frombytes("RGBA", self.fbo.size, self.fbo.read())
##        image = image.transpose(Image.FLIP_TOP_BOTTOM)
##        image.save(name, format="png")


if __name__ == "__main__":
    ImageProcessing.run()

Currently, when the code is run as-is, no image is saved whatsoever. The window just hangs and nothing happens. I am not sure if I have something wrong in my code or if the datatypes are wrong.

The pillow code (if you uncomment it) works to save it, but please note: While I could convert to a numpy array from Pillow, I would prefer not to in my use-case.

Clarification: The window loads and shows the image result just fine, but doesn't save correctly in the write method.

7
  • is your call to imread() meant to be commented out? Commented Nov 29, 2020 at 1:11
  • if the window just hangs, you need to simplify your program and debug it. maybe start from a minimal program that works, then work your way up and pay attention to when it fails again. Commented Nov 29, 2020 at 1:12
  • O.k. Thank you for helping. imread is meant to be commented out because that was my attempt to load from cv2. The window hangs as in "it works just fine if you run it, but the image doesn't get saved". The code is completely runnable as-is, if you'd like to see what I mean. Clarification: The window loads and shows the image result just fine, but doesn't save correctly in the write method. Commented Nov 29, 2020 at 1:14
  • @Rabbid76 and @Christoph Rackwitz, both of your answers work for saving the image, so Thanks! :) However, I have been unable to get good results with self.ctx.texture(img.shape[:2], img.shape[2], img). I am not sure how or if I should attach a photo or something, but the image appears banded. I have tried on multiple images with the same result. load_texture_2d doesn't have this problem. Thanks! Commented Nov 30, 2020 at 15:48
  • @correctsyntax How does the other answer work? In the other answer, no Texture object is created. Commented Nov 30, 2020 at 15:51

2 Answers 2

2

There is som code missing in your application

The method load_texture_2d creates a moderngl.Texture object. Hence the method loads the file, creates a texture object and loads the texture image from the CPU to the GPU.

cv2.imread just load the image file to a NumPy array, but it doesn't create a moderngl.Texture object.

You have to generate a moderngl.Texture object from the NumPy array:

img = cv2.imread("test6.png")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # optional
img = np.flip(img, 0).copy(order='C')      # optional
self.texture = self.ctx.texture(img.shape[1::-1], img.shape[2], img)

Before writing the buffer into an image, you must reshape the NumPy array according to the image format. For instance:

raw = self.fbo.read(components=4, dtype='f1')
buf = np.frombuffer(raw, dtype='uint8').reshape((*self.fbo.size[1::-1], 4))
cv2.imwrite("OUTPUT_IMAGE.png", buf)
Sign up to request clarification or add additional context in comments.

2 Comments

Thank you. It works, but I had to convert the colors with cv2.COLOR_BGR2RGB and remove the np.flip to get the output image with the correct colors and rotation. Should I edit your answer to account for this?
@correctsyntax I see. That depends on the source image and system. Thank you. You're welcome.
0

can you tell, in the write() method, what buf.shape is? I think it's a 1-d array at that point and it probably is height * width * 4 elements long.

imwrite() needs it shaped right. try this before imwrite():

buf.shape = (self.size[1], self.size[0], 4)

that should reshape the data as (height, width, 4) and then the imwrite() should accept it.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.