Write and read multiple byte[] in file with Java

Question

I have to write byte arrays in a file. I can't do it in one time so I can't put my arrays in a container. Also the size of my arrays is variable. Secondly, the file is very huge, so I have to split it, in order to read it array by array.

How can I do that ? I tried to write line by line my byte arrays but I haven't been able. How can I put a separator between my arrays and after split it over this separator ?

EDIT :

I tried this :

ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = new ObjectOutputStream(bos);
out.writeObject(byteArray);

But, I execute this code several times, so the ObjectOutputStream adds each time a new header which corrupt the file.

I also try :

out.write(byteArray);

but I couldn't separate my arrays. So I tried to append a '\n', which didn't work. After I was looking for library like FileUtils in order to write byte[] line by line but I didn't find.

Show some code, and tell us what you have tried. otherwise, I predict a downvote frenzy coming. — Colin D
– Colin D, Commented Oct 19, 2012 at 15:15
@pith Have a look at second code snippet in the answer. I think this is what you need. — Amit Deshpande
– Amit Deshpande, Commented Oct 19, 2012 at 16:19

Amit Deshpande · Accepted Answer · 2012-10-19 16:21:12Z

You can use existing collections Like e.g. List to maintain List of byte[] and transfer it

    List<byte[]> list = new ArrayList<byte[]>();
    list.add("HI".getBytes());
    list.add("BYE".getBytes());

    ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(
            "test.txt"));
    out.writeObject(list);

    ObjectInputStream in = new ObjectInputStream(new FileInputStream(
            "test.txt"));
    List<byte[]> byteList = (List<byte[]>) in.readObject();

    //if you want to add to list you will need to add to byteList and write it again
    for (byte[] bytes : byteList) {
        System.out.println(new String(bytes));
    }

Output:

   HI
   BYE

Another option is use RandomAccessFile. Which will not force you to read complete file and you can skip the data that you don't want to read.

     DataOutputStream dataOutStream = new DataOutputStream(
            new FileOutputStream("test1"));
    int numberOfChunks = 2;
    dataOutStream.writeInt(numberOfChunks);// Write number of chunks first
    byte[] firstChunk = "HI".getBytes();
    dataOutStream.writeInt(firstChunk.length);//Write length of array a small custom protocol
    dataOutStream.write(firstChunk);//Write byte array

    byte[] secondChunk = "BYE".getBytes();
    dataOutStream.writeInt(secondChunk.length);//Write length of array
    dataOutStream.write(secondChunk);//Write byte array

    RandomAccessFile randomAccessFile = new RandomAccessFile("test1", "r");
    int chunksRead = randomAccessFile.readInt();
    for (int i = 0; i < chunksRead; i++) {
        int size = randomAccessFile.readInt();
        if (i == 1)// means we only want to read last chunk
        {
            byte[] bytes = new byte[size];
            randomAccessFile.read(bytes, 0, bytes.length);
            System.out.println(new String(bytes));
        }
        randomAccessFile.seek(4+(i+1)*size+4*(i+1));//From start so 4 int + i* size+ 4* i ie. size of i
    }

Output:

BYE

Aubin · Accepted Answer · 2012-10-19 15:19:05Z

1

You have to described your data in your encoding. i.e. add some metadata.

For example, the length of the array, then the data of the array.

This is called serialization.

Array of int: length(4 bytes), data[0] (4 bytes), data[1] (4 bytes), data[2] (4 bytes)

answered Oct 19, 2012 at 15:19

Aubin

14.9k11 gold badges67 silver badges89 bronze badges

2 Comments

Sujay Over a year ago

Really? That's serialization for you?

Peter Ilfrich Over a year ago

It's a form of serialization, although you would normally only add as much meta-data as necessary to de-serialize and re-create the object (or array).

Jilles van Gurp · Accepted Answer · 2012-10-19 16:03:19Z

0

There are several ways of doing this. Basically first thing to point out is that a file is just an unstructured sequence of bytes. Any bytes. Basically you want o store several byte arrays of variable length. That means you have to add structure in some way, which you then parse when you read.

The easiest is probably to use some sequence of bytes that you treat as a separator (specific enough that it has a low probability of showing up in your data otherwise).

Then on writing, first simply write your bytes, then the separator

out.write(myarray);
out.write(separator);
out.write(anotherarray);

On reading, you need to use some kind of sliding window that you can inspect so that you can figure out if you have read the separator. Basically just iterate over the bytes one by one and keep the last few you've read in some buffer. When you see the separator in your buffer, you have just found the end of your array.

An alternative to this could be writing blocks of a fixed length with some header that describes the length and maybe a bit for denoting whether there is another packet for the current array. Then you simply read and write whole blocks.

answered Oct 19, 2012 at 16:03

Jilles van Gurp

8,4225 gold badges42 silver badges48 bronze badges

1 Comment

Aubin Over a year ago

How to you deal with the problem: I have separator INTO an array?

Collectives™ on Stack Overflow

Write and read multiple byte[] in file with Java

3 Answers 3

Comments

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related