Short short answer for your question: You're doing it wrong.

The longer answer:

Your bottleneck is most likely file reading and parsing, something you should only do once.

So rather than loading the file and running the code directly, you should instead define callbacks in your Lua code, which are then executed (rather than the whole script).

For example, you might have some lua script to apply movement to some entity:

this_x = this_x + this_vx * dt
this_y = this_y + this_vy * dt

To run this script, you could simply run it using lual_dofile() over and over again, but this isn't very efficient (as you've noticed).

So instead, wrap that code in a function:

function onUpdate(this, deltaTime)
    this.x = this.x + this.vx * deltaTime;
    this.y = this.y + this.vy * deltaTime;
end

You'll once again load and execute this code once to actually initialize/load that script. This won't have any noticeable effect or change other than defining the functions (or any locals) in your current context.

Then, once every frame, you'd just call onUpdate() once every update rather than reloading/reparsing the whole script (and simply don't call it, if it isn't defined; a state which you can save for later, so you don't have to keep trying).

This should improve your engine's performance dramatically.

I think part of the problem (which also caused me to misinterpret the question at first) is the lual_dofile() calls.

This sounds like all scripts are always loaded in the same global state, overwriting each other, so loading/parsing causes a significant overhead.

Instead, I'd suggest you use lua_newthread() to create a separated context once per script file loaded.

This allows you to load all your script files once and then simply switch between them before calling update() or any other Lua function, without having to reload/recompile anything.