2
$\begingroup$

I want to backtest a large number of algorithms on the same dataset, one stock ticker. The algorithms are actually variations of one algorithm with various combinations of parameters, amounting to about 10,000 variations.

Is there a way to test all these algorithms truly in parallel, with a shared cache storing various indicators computed in the process?

I'm new to parallel computing, but I think GPUs and cloud computing providers might have solutions. However, the largest number of cores I've seen a provider offer is 128.

In very high-level pseudocode, I'm looking for something like,

// allocate a thread/core per algo
for (i = 0; i < algos.length; i++)
  threads[i] = CPU.createThread(algos[i])
  threads[i].initAlgo(param1, param2, ...)

// Pass each tick to all algos
for (i=0; i < ticks.length; i++)
  for (j = 0; j < algos.length; j++)
    signal = threads[j].algoProcessTick(ticks[i])
    // do something with signal
Improve this question
$\endgroup$
4
  • $\begingroup$ You can definitely have more core if you are willing to pay more, you might want to look into the companies doing HPC (high performance computing) and things such as solver as a service with scalable capacities to fit your needs $\endgroup$ Commented Nov 4, 2019 at 9:08
  • $\begingroup$ Why must you have them run in parallel? Are you dealing with strictly historical data? What stops you from looping over every algorithm, saving the results, and then loop though the same dataset with the next algorithm? $\endgroup$ Commented Nov 8, 2019 at 18:03
  • $\begingroup$ @Jasonp: calculating custom indicators for the sliding window would have to be done every single time if we looped over every algorithm. By running them in parallel, only the first algorithm from a group with similar parameters (that affect the indicator) that sees the new tick needs to calculate the indicator. $\endgroup$ Commented Nov 16, 2019 at 6:14
  • 1
    $\begingroup$ @Gascoyne Would you need all 10,000 variations running simultaneously as new data is being sent through every variation so you can validate each in realtime? If there is this need for many cores, I am wondering if there's a way to mitigate the needs for so many cores: run the minimum number of algorithms that produce all needed parameters (as you said there are groups of various parameter needs), save those parameters alongside your historical data, and apply the rest of the varied algorithms over that stream of data. Mayb many or all algorithms can be applied to a core thatd export indicators $\endgroup$ Commented Nov 17, 2019 at 2:09

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.