0

I'm learning Map() function of javascript. I want to map rules for declarativeNetRequest but I need a bit of help.

How I can create a map using this model as output:

{
    "id" : 1,
    "priority": 1,
    "action" : { "type" : "block" },
    "condition" : {
        "urlFilter" : "abc",
        "domains" : ["foo.com"],
        "resourceTypes" : ["script"]
  }
}

At the moment I'm fetching a list of filters from a remote server and I'm parsing it using this code, after the parsing I want to map the results to have an object that I can use with declarativeNetrequest api

let def = {};
let out;
let defMap = new Map();

const fetch = async () => {
  await axios({
  url: "https://raw.githubusercontent.com/easylist/easylist/master/easylist/easylist_adservers.txt",
  responseType: "text"
  })
  .then( (response) => {
      let parsed = response.data.split("\n");
    out = parsed.filter( (item) => {
      if( !item.startsWith("!") ){
        return item.replace(/([\|\|\w\d\.\^]+)(?:[\w-\?=~,\$]+)$.*/, "$1");
      }
    });
  });
  return out;
}
// debug only 
fetch().then( (out) => {
  console.log(out);
});

will be this possible?

2
  • Can you please explain what is the final object output that you are trying to achieve? Commented Sep 7, 2020 at 16:53
  • @VimalPatel the desired output is already into the question and it's the model I've posted Commented Sep 7, 2020 at 17:01

1 Answer 1

1

See below example that outputs a list of JSON objects based on the rules. Here are some other pointers:

  • You don't need a Map(), just a regular JSON/Javascript object structure
  • You don't need to use .then() (Promises syntax) and await (async/await syntax) together - just chose one - await is the more modern way to deal with asynchronous responses - I used await
  • If you had used .then(), then the return statement would have completed before the inner callback function had been called, so out would not have been set
  • I didn't look into the syntax of the file too much - but I presume fragments after ^ are useful, perhaps applying them to only certain subdomains
  • The block rule accepts an array of domains - so you may be able to create one rule that applies to many domains instead of one rule for each domain
import axios from 'axios'

async function fetch() {
  const resp = await axios({
    url: "https://raw.githubusercontent.com/easylist/easylist/master/easylist/easylist_adservers.txt",
    responseType: "text"
  })
  return [...parseDomains(resp.data)].map(blockRule)
}

function* parseDomains(data) {
  for (const line of data.split("\n")) {
    const match = line.match(/^[\|]+(?<domain>[^\^]+)/)
    if (match) yield match.groups.domain
  }
}

function blockRule(domain, id) {
  return {
    id,
    priority: 1,
    action: { type: "block" },
    condition: {
      urlFilter: `for_${domain}`,
      domains: [domain],
      resourceTypes: ["script"]
    }
  }
}

// debug only 
fetch().then((out) => {
  console.log(out)
})

Additional explanation:

  1. Get the data:
resp = await axios(...)
  1. parseDomains splits the data into lines (as you did before), and does a regular expression match on it. If it matches, it'll yield another domain. The function* makes this a generator expression - basically it can be iterated over (like an array) the elements are whatever you yield. If each line would yield a domain then you could have used .map() to convert each line to a domain, but only some lines yield domains so you need something else. An alternative would have been to create an empty array and then add a new domain element to it each time you found a valid domain.

The regular expression itself is a bit simpler than the one you had:

  • match one or more |'s
  • followed by a one or more non-^ (so will end/terminate when ^ or end-of-string is encountered) - this is put into a match group (enclosed by parentheses, like you did) and given the name domain
  • if the return value, match is truthy, it means you have found a match, and the matches can be found in match.groups - since I named it domain it's in match.groups.domain
  1. Since parseDomains is a generator expression, which are lazily evaluated, to run a map/filter on it you'll need to eagerly evaluate it - that's what [...expr] does - it basically forces the generator expression to be evaluated and converted into an array. We could have kept evaluating things lazily like this instead:
async function* fetch() {
  const resp = await axios(...)
  for (const domain of parseDomains(resp.data)
    yield blockRule(domain)
}
  1. Finally we map the domains through blockRule to output the actual rules, so you'll have an array of rule objects in the end.
Sign up to request clarification or add additional context in comments.

3 Comments

can you explain a bit the code please?I'm not very confident with JS. Also a note about the block rule you've mentioned, the docs are clear about how a rule can be declared, for each rule only one url is accepted for the urlFilter field, block is only an action that will be taken if the url pattern is detected. Maybe you've confused this api with the webRequest one :)
I figured since domains is an array that it could accept several, but it doesn't matter - as it is now it's one rule per domain/line :)
I added some more explanation in the answer body above.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.