Bytes from unpredictable data in Python

Question

We receive data from an API call and we can't predict the nature of that data. The challenge is just to code a bullet-proof conversion (which I naively thought would be very easy) resulting in an iterable bunch of bytes (bytearray or bytes). I've tried a variety of approaches, even trying io.BytesIO, but some input types still get rejected (a code type kicks out, for example). Any suggestions? It shouldn't be this hard just to treat data as bytes. Thanks for any help.

    def process_binary_object(self, api_data):
        api_dat_size = sys.getsizeof(api_data)
        if api_dat_size < 4 or api_dat_size > 4096:    # somewhat arbitrary
            return '0123'
        if isinstance(api_data, bytes):
            self.binary_seq = api_data
            print("Conversion technique aa succeeded.")
        else:
            try:
                self.binary_seq = api_data.tobytes()
                print("Conversion technique bb succeeded.")
            except (AttributeError, TypeError):
                try:
                    self.binary_seq = bytearray(api_data)
                    print("Conversion technique cc succeeded.")
                except (AttributeError, TypeError):
                    try:
                        mem_view = memoryview(api_data)
                        self.binary_seq = mem_view.tobytes()
                        print("Conversion technique dd succeeded.")
                    except (AttributeError, TypeError):
                        self.binary_seq = io.BytesIO(api_data)
                        print("Conversion technique ee succeeded.")
        # some types still don't make it
        first_four = self.binary_seq[:4]
        return first_four

Christopher Hoffman · Accepted Answer · 2018-09-13 13:45:28Z

1

After time and working with Java and reading a code book I came across this for the strategy of handling bytes.

Bytes were originally interpreted by the Java Virtual Machine. If your source of bytecode is from java implemented web services, then launch a JVM and configure the stream for the JVM to parse and the JVM will do the rest in this on demand basis in real time.

This is done by attaching an JIT(Just in Time) compiler to your JVM and the bytecode will compile into executable native code in real time. Any other bytes that are not code will be interpreted as such.

Biography: Java Seventh Edition, The Complete Reference

answered Sep 13, 2018 at 13:45

Christopher Hoffman

1921 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Christopher Hoffman · Accepted Answer · 2019-10-09 20:39:53Z

0

You could regular expression the common word byte for your true/continue case

edited Oct 9, 2019 at 20:39

answered Jun 8, 2018 at 16:11

Christopher Hoffman

1921 silver badge9 bronze badges

9 Comments

Ben_Z Over a year ago

Perhaps I wasn't clear. The input is not always one consistent type, it changes, and I need a reliable way to always have iterable bytes. But thank you for your help in any case. I hate to admit it, but I think I'll be forced to write the data to a file and read it back in as bytes, putting a huge, embarrassing delay into the process. Depressing.

Christopher Hoffman Over a year ago

Thats what the built in command 'type' is for. It will identify the type for you. Try type(api_data) If type(api_data) == str: do something ....if python cant identify the object, you will need to setup your environment with the api_libraries that is sending the data in

Ben_Z Over a year ago

Say I identify the 15 or 20 types we've received over the past several weeks. Some types use an obj.tobytes() function, so I could hard-code that for those types. Some types need obj.to_bytes(), so I could hard-code that for those types. Some types only have obj.to_bytearray(), so I could hard-code that for those types. Some types only have a obj.to_byte_array(), so I could hard-code that for those types. Now we go into production and then a week later we get a new type that needs obj.convert_to_bytes(), but we didn't code for that type, so our process blows up. That is the issue.

Christopher Hoffman Over a year ago

escalate up, get someone on the phone, create a standard template for your interface and distribute it, make it their responsibility to provide the converter. Have a meeting with all your providers as a team, and get everyone to understand and on the same page

Ben_Z Over a year ago

I could have dozens of meetings, have dozens of escalations and get dozens of people on the phone. The answer would be, "You are responsible to convert the data into bytes, no matter what comes in. That is your job. Why is that so hard? If Python can't do it, throw it out and get a better language. Don't invent a new requirement and try to force it on us. We told you the input could have any possible layout and it was never our responsibility to conform to any particular format." Everyone is on the same page.

|

Collectives™ on Stack Overflow

Bytes from unpredictable data in Python

2 Answers 2

Comments

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related