1

I am working in Javascript and creating an offline application.

I have a very large array, over 10mb. I need to save elements from the array where a user has made a change, but writing / reading the array to disk is too slow.

I would like to create a second array that holds only the changes made by the user, so I can save that smaller array and then when I need to load changes it can mirror its changes to the large array.

For example:

lib[x][1][y][6] = "no";
libVars[x][1][y][6] = "no";

libVars would simply hold the element changes in lib and would not be the same size as the large array lib as the user will only interact with a small portion of the large array.

Obviously this doesn't work, as libVars doesn't have the same structure as lib, and if it did it would also take up a lot of memory (i think so anyways!)

I would then like to loop through libVars and update lib at the same element points where the changes have been saved in libVars.

Is there a way to save some type of pointer to a location within lib that I could store in libVars with the associated element value?

Any help would be much appreciated.

Thanks.

======================

Sample of my large array

var lib = [
[241,[[221119,"sample data","sample","no","no",131,"no"],
[221121,"sample2 data","sample2","no","no",146,"no"],
[221123,"sample3 data","sample3","no","no",28,"no"],
[221626,"sample4 data","sample4","no","no",26,"no"],
[221628,"sample5 data","sample5","no","no",88,"no"]]],
[330,[[305410,"2sample data","sample 2b","no","no",197,"no"],
[305412,"2sample 2 data","sample2 2b","no","no",147,"no"],
[305414,"3sample 2 data","sample3 2b","no","no",10,"no"] ...

Attempt at solution posted by Z-Bone

I have added the tempLib var, but I don't see how you eventually update the actual lib object

Object.entries(libVars).forEach(([path, value]) => {

  var tempLib = lib;
  var pathParts = path.split('#');

  pathParts.forEach((pathPart, index) => {

  if (index == pathParts.length-1) {

    tempLib[pathPart] = value;

  } else {
    //I don't fully track what you are doing here. 
    //I assume you are building the array path, but to me it looks like
    // it's getting wiped out and not built upon?    
    tempLib = tempLib [pathPart];
  }
});
});
Improve this question
2
  • 1
    npmjs.com/package/json-pointer Commented Jan 27, 2019 at 0:21
  • @JoeWarner Thanks but I would like to avoid any libraries. Commented Jan 27, 2019 at 21:08

4 Answers 4

Reset to default
2

Update 28/01/2019 Based on nomaam's test data

@nomaam Please check the following snippet. I made it change an arbitrary path in lib array. The path I chose is this: lib[0][1][3][4]. You can see how it initially returns the value "no" and then after execution of the modification tracking solution, it returns the value "yes".

// First Step: Original Data (lib)
var lib = [
[241,[[221119,"sample data","sample","no","no",131,"no"],
[221121,"sample2 data","sample2","no","no",146,"no"],
[221123,"sample3 data","sample3","no","no",28,"no"],
[221626,"sample4 data","sample4","no","no",26,"no"],
[221628,"sample5 data","sample5","no","no",88,"no"]]],
[330,[[305410,"2sample data","sample 2b","no","no",197,"no"],
[305412,"2sample 2 data","sample2 2b","no","no",147,"no"],
[305414,"3sample 2 data","sample3 2b","no","no",10,"no"]]]]

// Log initial value in arbitrary selection lib[0][1][3][4]
console.log(lib[0][1][3][4])

// Second Step: Pointer's object
var libVars = {}

// Example of changing a value via an artificial "pointer" to lib[0][1][3][4]
libVars['0#1#3#4'] = 'yes'; // original value is "no"

// Third Step: Run the modifier - update changes from libVars to lib
Object.entries(libVars).forEach(([path, value]) => {

  var tempLib = lib;
  var pathParts = path.split('#');
  
  pathParts.forEach((pathPart, index) => {
    
    if (index == pathParts.length - 1) {
      
      tempLib[pathPart] = value;
      
    } else {
      
      tempLib = tempLib[pathPart];
      
    }
  });
});


// Log the change fro lib
console.log(lib[0][1][3][4])

Original Answer

If I understand your challenge correctly, you could technically hold an object with the path to the property you want to modify as the key and the modified value as its value.

let's say your original object looks like this:

var lib = {a: {b: { c: { d: "yes"}}}}

You could keep a log of changes in a backup object (e.g. value changes from "yes" to "no"). It could look something like this, with . marking a nested property.

var libVars = {};
libVars['a.b.c.d'] = "no";

Then when you would like to update the original large array/object of values, you could do this:

Object.entries(libVars).forEach(([path, value]) => {
    var tempLib = lib;
    // split by nesting indicator -> dot
    var pathParts = path.split('.');
    // iterate all parts of the path to the modified value
    pathParts.forEach((pathPart, index) => {
        if (index == pathParts.length-1) {
            // once you made your way to last index, update value
            tempLib[pathPart] = value;
        } else {
            tempLib = tempLib[pathPart];
        }
    })
})

console.log(lib); // outputs => {a: {b:{c: {d : "no"}}}}
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Doesn't this assume that a key cannot have a . in its name? e.g. {'a.b': {'c.d': 'yes'}}
@customcommander This is an example of a possible approach. You could replace the nesting representation dot with any other character. You could also escape it and then split with the escaping operator.
I am attempting to get your example working. I am getting undefined errors on section tempLib = tempLib[pathPart]. I like the concept. Can you please clarify for me, if i normally access an array like lib[0][1][3][6], is there a way to convert a string, without looping, like lib[ "0,1,3,6" ], if possible then the direct "pointer" string paths could just be inserted and I wouldn't have to worry about undefined errors or indexing issues in the loop.
I have posted my attempt at your solution with comments in my original post
@nomaam First thing I see is you're using the original lib to make your way to the final value (assigning it to itself each time). This way you lose the original lib. In each iteration use a temp var that is set to lib and iterate on it. (see my example: var tempLib = lib; ). It would be very helpful if you could also, post a partial example of your lib array/object. Notice that in my example, I used an object and not an array.
|
1

Update 2

Updated my answer, to get rid of lodash, and it appeared to be equal to Z-Bone's answer. His variant is even more clear than mine.

Updated @ 28.01.2019

var lib = [
[241,[[221119,"sample data","sample","no","no",131,"no"],
[221121,"sample2 data","sample2","no","no",146,"no"],
[221123,"sample3 data","sample3","no","no",28,"no"],
[221626,"sample4 data","sample4","no","no",26,"no"],
[221628,"sample5 data","sample5","no","no",88,"no"]]],
[330,[[305410,"2sample data","sample 2b","no","no",197,"no"],
[305412,"2sample 2 data","sample2 2b","no","no",147,"no"],
[305414,"3sample 2 data","sample3 2b","no","no",10,"no"]]]];

// object to track edits
var edits = {
  '0.1.2.2': 'edited'
};

// sample/simple implementation of lodash's 'set' function
function applyEdit(arr, path, value) {
  var items = path.split('.');
  var last = items.length - 1;
  var current = arr;
  items.forEach(function (item, i) {
    if (i !== last) {
      current = current[item];
    }
  });
  current[items[last]] = value;
}

// our restoration function
function patch(arr, edits) {
  Object.keys(edits).forEach(function(key) {
    var newValue = edits[key];
    applyEdit(arr, key, newValue);
  });
}

console.log(lib[0][1][2][2]) // "sample3"

// now we can restore our edits
patch(lib, edits);

console.log(lib[0][1][2][2]) // "edited"

// --------------------------------------------------------

// to prevent forgetting to track the updates you can use utility function
// to make changes to your 'lib' array

function update(arr, path, value) {
  applyEdit(arr, path, value);
  edits[path] = value;
}

Original answer:

One possible solution would be to have an object where you track all the changes. const edits = {}; And when you update your object with lib[x][1][y][6] = "no"; you also save your edit:

edits[`${x}.1.${y}.6`] = "no";

Basically you just create a string holding the path of the change.

After you loader your edits object from file you can apply all changes to the original array with the following code:

import { set } from 'lodash';

function patch(arr, edits) {
  Object.keys(edits).forEach(key => {
    const newValue = edits[key];
    set(arr, key, newValue);
  });
}
Improve this answer

2 Comments

Thank you for the post. It is similar to @Z-Bone 's idea. Just trying to get it to work on my end. See my follow up posts attached to his solution.
I am trying to test your solution but I am unable to use external libraries such as lodash.
0

Assuming that the paths in libVars are the same in lib then you could simply record the changes into an array of functions that you reapply on your lib object later on:

const data = {foo: 'aaa', bar: 'bbb'};

console.log('data before:', data);

const updates = [];

updates.push(obj => {
  obj.foo = 'foo';
});

updates.push(obj => {
  obj.bar = 'bar';
});

updates.forEach(fn => fn(data));

console.log('data after:', data);

Improve this answer

2 Comments

Thanks, but I think this would raise the issue I mentioned in the question, that the smaller array that holds the changes will pretty much be a similar size to the main array, as empty array values still take up memory
The only thing that updates will contain are functions not the actual data on which they operate.
0

This may be handled with emulated autovivification using Proxy. See:

https://en.wikipedia.org/wiki/Autovivification

Autovivification and Javascript

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.