Minetest asyncio

A lua coroutine wrapper thing. Inspired by Python's asyncio. As with Python's asyncio, coroutines run in the main thread and must voluntarily give up control.

Note that this doesn't come with I/O-related functions, it is just called asyncio to not conflict with the similar-but-different async mod.

Example code

"Plain" lua

local function iter(a)
    local stage = 0
    return function()
        if stage == 0 then
            stage = 1
            return 'Iteration started.'
        elseif stage == 2 then
            return
        elseif a < 100 then
            a = a * 2
            return a
        else
            stage = 2
            return 'Finished iterating.'
        end
    end
end

function itertest(a)
    for i in iter(a) do
        print('Iterator test: ' .. i)
    end
end

print('Non-async code started, deferring...')
minetest.after(0, function()
    print('Non-async code resumed, using iterator.')
    itertest(1)
    print('Finished iterating, waiting 1 second.')
    minetest.after(1, function()
        print('Waiting another two seconds.')
        minetest.after(2, function()
            print('Done!')
        end)
    end)
end)

"Plain" lua syntax with asyncio

local iter = asyncio.coroutine(function(a)
    asyncio.yield('Iteration started.')
    while a < 100 do
        a = a * 2
        asyncio.yield(a)
    end
    asyncio.yield('Finished iterating.')
end)

itertest = asyncio.coroutine(function(a)
    for i in asyncio.iter(iter, a) do
        print('Iterator test: ' .. i)
    end
end)

asyncio.run(function()
    print('async code started, deferring...')
    asyncio.defer()
    print('async code resumed, using async iterator')
    itertest(1)
    print('Finished iterating, waiting 1 second.')
    asyncio.sleep(1)
    print('Waiting another two seconds without asyncio.sleep.')
    asyncio.await(minetest.after(2, asyncio.resume))
    print('Done!')
end)

Entirely asyncio lua

local async function iter(a)
    asyncio.yield('Iteration started.')
    while a < 100 do
        a = a * 2
        asyncio.yield(a)
    end
    asyncio.yield('Finished iterating.')
end

async function itertest(a)
    async for i in iter(a) do
        print('Iterator test: ' .. i)
    end
end

print('async code started, deferring...')
asyncio.defer()
print('async code resumed, using async iterator')
itertest(1)
print('Finished iterating, waiting 1 second.')
asyncio.sleep(1)
print('Waiting another two seconds without asyncio.sleep.')
asyncio.await(minetest.after(2, asyncio.resume))
print('Done!')

How it works

This mod is a lightweight-ish wrapper around lua coroutines. The async function and async for syntax can only be used with asyncio.dofile and is run through a primitive transpiler(?) that converts the syntax into valid lua.

API

The following functions are available both inside and outside of coroutines:

• asyncio.run(function(...)): Executes a function inside an asyncio coroutine.
• asyncio.coroutine(function(...)): A function wrapper that runs the function inside a coroutine.
• asyncio.exec(...): Similar to asyncio.loadstring(...)().

asyncio.loadstring, asyncio.loadfile, and asyncio.dofile work the same way as their built-in counterparts, however these functions also accept async function and async for syntax, and the code itself is executed inside a coroutine.

The following functions only work inside coroutines:

• asyncio.await(...): Suspends execution of the current coroutine until asyncio.resume is called. If asyncio.resume is never called, the coroutine will be garbage collected by lua. Example: asyncio.await(function_with_callback(params, asyncio.resume))
• asyncio.resume: This should not be called inside the coroutine itself, instead used as a callback function. asyncio.resume changes depending on what coroutine is running, and is nil outside of coroutines.
• asyncio.sleep(delay): Suspends execution of the current coroutine for delay seconds. Internally, this just calls asyncio.await(minetest.after(delay, asyncio.resume)).
• asyncio.defer(): Defers execution of this coroutine until the next globalstep.
• asyncio.optional_defer(): Allows asyncio to defer, however leaves the decision to actually defer up to asyncio. Currently, it will only call defer() if it has been over 5ms since the previous "successful" optional_defer(), however this may change in the future. Returns true if asyncio.defer() was actually called, otherwise false.
• asyncio.odefer(): Alias for asyncio.optional_defer.
• asyncio.iter(func, ...): Calls func as an asyncio iterator.
• asyncio.pairs(obj, defer_every): Similar to pairs(), but calls asyncio.defer() every defer_every iterations. Must be called in an async for loop or with asyncio.iter(). If defer_every is unspecified, asyncio.optional_defer() is called on every iteration instead.
• asyncio.ipairs(obj, defer_every): The same as asyncio.pairs, but with ipairs.
• asyncio.yield(...): Used to yield values in asyncio iterators. This function is slightly different from coroutine.yield and should not be used interchangeably with it.
• asyncio.running: A variable, set to true when inside a coroutine. Do not manually set this, or it will break.
• asyncio.run_async(function(...)): Runs function(...) in a separate coroutine, so it does not block the current one.
• asyncio.domodfile(file): Calls something similar to asyncio.dofile(current_modpath .. '/' .. file).

Custom lua syntax

The custom lua syntax introduced by this mod is "transpiled" back into regular lua, and can only be used with asyncio.dofile etc.

Pre-runtime transpiling

If you are going to be using Minetest's insecure environment from a file using this custom syntax, I strongly recommend avoiding asyncio.loadfile etc.

To invoke the transpiler outside of Minetest, you can simply run lua transpile.lua /path/to/file.async.lua. This should print the transpiled code. If you want it to write the transpiled code to a different file, provided you are on a UNIX-like OS, you can run lua transpile.lua file.async.lua > file.lua.

Note that async for loops will always be mangled ensure compatibility with Lua 5.1.

Coroutines

asyncio coroutines can be created in one of two ways:

func1 = asyncio.coroutine(function(a, b, c))
    return a + 1, b + 1, c + 1
end)

async function func2(a, b, c)
    return a + 1, b + 1, c + 1
end

The async function sytnax will only work in lua code loaded with asyncio.dofile etc.

The return value of coroutine functions will be lost if they are called from other non-coroutine functions.

You don't have to await coroutines from other coroutines, and calling coroutines from non-asynchronous code will work correctly.

Iterators

asyncio iterators can only be used inside coroutines.

for value in asyncio.iter(iterable_function, parameters) do
    print(value)
end

-- "async for" is recommended over asyncio.iter and should be used where
--  possible.
async for value in iterable_function(parameters) do
    print(value)
end

Without LuaJIT, because of coroutine limitations in Lua 5.1, asyncio.iter calculates and stores all iteration values before starting the loop. This greatly reduces performance, especially when using asyncio.pairs or asyncio.ipairs. async for does not have this limitation, however, and is therefore recommended over asyncio.iter.

async for k, v in asyncio.pairs(minetest.registered_items) do
    -- Some resource-intensive thing here.
end

For another iterator example, see here.

HTTP(S) requests

As to not circumvent mod security, there are no asynchronous HTTP fetch functions, and there are no helpers to create fetch functions.

To send an HTTP(S) request from an asyncio coroutine, you can call asyncio.await(http.fetch(req, asyncio.resume)), where http is the name of the HTTPApiTable. This will return the response table.

Alternatively, if you are calling http.fetch multiple times, it may be easier to define an asyncio HTTP fetch function:

local function http_fetch(req)
    return asyncio.await(http.fetch(req, asyncio.resume))
end

wait until

asyncio introduces a wait until <condition> statement, that checks condition every globalstep until it evaluates to true, and then continues executing the thread.

Do not use wait until unless there is no other way, it can and will create lag.

Locks and events

asyncio adds asyncio.Locks and asyncio.Events.

These are similar to Python's asyncio locks.

Example:

local lock = asyncio.Lock()
async function do_something()
    print('Acquiring lock...')
    lock:acquire()
    print('Acquired the lock, doing something...')
    asyncio.sleep(5)
    print('Releasing the lock...')
    lock:release()
end

for i = 1, 5 do
    asyncio.run_async(do_something)
end

-- Wait for the above functions to finish.
lock:acquire()
lock:release()

-- Events
local event = asyncio.Event()
async function do_something_else()
    print('Waiting on the event...')
    event:wait()
    print('Event triggered, waiting 5 seconds.')
    asyncio.sleep(5)
    print('Done.')
end

for i = 1, 5 do
    asyncio.run_async(do_something_else)
end

-- Set the event
asyncio.sleep(5)
print('Running event:set()...')
event:set()
print('Finished running event:set().')