WIP: [retarget upstream] Task Scheduler #4716

Draft
teknomunk wants to merge 18 commits from scheduler into automated-tests
Member

This is part of a series of PRs that will be updating redstone functionality. This adds a general purpose scheduler to VoxeLibre. While this does not update redstone behavior directly, I plan on using priority levels 1, 3 and 4 as part of the follow-on redstone updates. In particular, the planned change from mesecons-based redstone power flood-fill to a netlist-based circuit with eventual-update redstone visuals would use priority levels 2 (circuit update) and 3 or 4 (visual update).

The scheduler includes these main parts:

  1. 4-priority first-in, first-out queue for tasks that will run the next timestep.
  2. 32 entry ring buffer for tasks scheduled for tasks 2-33 timesteps in the future
  3. A structure that is a sort of hybrid between a finger tree and a Array mapped trie for tasks scheduled 34 or more timesteps in the future.

The four priority levels are intended for these uses:

  1. Asynchronous tasks that are always started this timestep. This is first for maximum overlap with the remainder of the timestep.
  2. Synchronous tasks that are always run this timestep.
  3. Asynchronous tasks that are started this timestep provided there is remaining time budget.
  4. Synchronous tasks that are started this timestep provided there is remaining time budget.

The scheduler replaces the builtin core.after() and takes over scheduling these tasks. The scheduler takes the functions in core.registered_globalsteps, creates a code generated function for dispatch and execute this as part of dispatching priority 2 tasks.

Tasks

  • Fix crashes:
  • Add fixed-time-budget tasks with accounting
    • Use for mob spawning (#4468)
  • Add task save/restore across server restarts
    • Implement vl_scheduler.register_serializable(identifier, callback)
  • Add documentation

Testing

  1. Unit testing: cd mods/CORE/vl_scheduler; busted unit-test.lua
  2. Play testing: start a game up using this branch and ensure it doesn't crash.
This is part of a series of PRs that will be updating redstone functionality. This adds a general purpose scheduler to VoxeLibre. While this does not update redstone behavior directly, I plan on using priority levels 1, 3 and 4 as part of the follow-on redstone updates. In particular, the planned change from mesecons-based redstone power flood-fill to a netlist-based circuit with eventual-update redstone visuals would use priority levels 2 (circuit update) and 3 or 4 (visual update). The scheduler includes these main parts: 1. 4-priority first-in, first-out queue for tasks that will run the next timestep. 2. 32 entry ring buffer for tasks scheduled for tasks 2-33 timesteps in the future 3. A structure that is a sort of hybrid between a [finger tree](https://en.wikipedia.org/wiki/Finger_tree) and a [Array mapped trie](https://en.wikipedia.org/wiki/Bitwise_trie_with_bitmap) for tasks scheduled 34 or more timesteps in the future. The four priority levels are intended for these uses: 1. Asynchronous tasks that are always started this timestep. This is first for maximum overlap with the remainder of the timestep. 2. Synchronous tasks that are always run this timestep. 3. Asynchronous tasks that are started this timestep provided there is remaining time budget. 4. Synchronous tasks that are started this timestep provided there is remaining time budget. The scheduler replaces the builtin `core.after()` and takes over scheduling these tasks. The scheduler takes the functions in `core.registered_globalsteps`, creates a code generated function for dispatch and execute this as part of dispatching priority 2 tasks. ### Tasks * [ ] Fix crashes: * [X] [Recursive task queuing](https://git.minetest.land/VoxeLibre/VoxeLibre/pulls/4716#issuecomment-85856) * [X] [Job cancel](https://git.minetest.land/VoxeLibre/VoxeLibre/pulls/4716#issuecomment-85869) * [ ] Add fixed-time-budget tasks with accounting * [ ] Use for mob spawning (https://git.minetest.land/VoxeLibre/VoxeLibre/pulls/4468) * [ ] Add task save/restore across server restarts * [ ] Implement `vl_scheduler.register_serializable(identifier, callback)` * [ ] Add documentation ### Testing 1. Unit testing: `cd mods/CORE/vl_scheduler; busted unit-test.lua` 2. Play testing: start a game up using this branch and ensure it doesn't crash.
teknomunk added this to the 1 - Now milestone 2024-11-29 01:33:20 +01:00
teknomunk added the
API
enhancement
redstone
labels 2024-11-29 01:33:20 +01:00
teknomunk added 2 commits 2024-11-29 01:33:22 +01:00
teknomunk added a new dependency 2024-11-29 01:33:40 +01:00
teknomunk changed title from WIP: Redstone Updates #1: Scheduler to WIP: Redstone Update Part 1: Scheduler 2024-11-29 01:34:10 +01:00
teknomunk added this to the Redstone Update project 2024-11-29 01:36:12 +01:00
teknomunk changed title from WIP: Redstone Update Part 1: Scheduler to WIP: Task Scheduler 2024-11-29 01:37:36 +01:00
Author
Member

Had crash when running overnight:

AsyncErr: Lua: Runtime error from mod '??' in callback environment_Step(): "Runtime error from mod '??' in callback l_add_item(): ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:197: attempt to index field 'last' (a nil value)
stack traceback:
	...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:197: in function 'queue_task'
	...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:230: in function 'vl_scheduler_after'
	...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:236: in function 'after'
	...t/games/VoxeLibre/mods/ENTITIES/mcl_item_entity/init.lua:831: in function 'set_item'
	/usr/share/minetest/builtin/game/item_entity.lua:9: in function </usr/share/minetest/builtin/game/item_entity.lua:3>
	[C]: in function 'add_item'
	...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:137: in function 'item_drop'
	...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:477: in function 'death_handle'
	...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:578: in function 'real_func'
	...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:228: in function 'func'
	...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:139: in function <...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:81>
	/usr/share/minetest/builtin/common/register.lua:26: in function </usr/share/minetest/builtin/common/register.lua:12>"
stack traceback:
	[C]: in function 'add_item'
	...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:137: in function 'item_drop'
	...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:477: in function 'death_handle'
	...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:578: in function 'real_func'
	...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:228: in function 'func'
	...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:139: in function <...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:81>
	/usr/share/minetest/builtin/common/register.lua:26: in function </usr/share/minetest/builtin/common/register.lua:12>
Had crash when running overnight: ```` AsyncErr: Lua: Runtime error from mod '??' in callback environment_Step(): "Runtime error from mod '??' in callback l_add_item(): ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:197: attempt to index field 'last' (a nil value) stack traceback: ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:197: in function 'queue_task' ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:230: in function 'vl_scheduler_after' ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:236: in function 'after' ...t/games/VoxeLibre/mods/ENTITIES/mcl_item_entity/init.lua:831: in function 'set_item' /usr/share/minetest/builtin/game/item_entity.lua:9: in function </usr/share/minetest/builtin/game/item_entity.lua:3> [C]: in function 'add_item' ...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:137: in function 'item_drop' ...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:477: in function 'death_handle' ...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:578: in function 'real_func' ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:228: in function 'func' ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:139: in function <...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:81> /usr/share/minetest/builtin/common/register.lua:26: in function </usr/share/minetest/builtin/common/register.lua:12>" stack traceback: [C]: in function 'add_item' ...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:137: in function 'item_drop' ...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:477: in function 'death_handle' ...etest/games/VoxeLibre/mods/ENTITIES/mcl_mobs/physics.lua:578: in function 'real_func' ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:228: in function 'func' ...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:139: in function <...minetest/games/VoxeLibre/mods/CORE/vl_scheduler/init.lua:81> /usr/share/minetest/builtin/common/register.lua:26: in function </usr/share/minetest/builtin/common/register.lua:12> ````
teknomunk added 1 commit 2024-11-29 18:44:32 +01:00
Author
Member

Next crash (occurred when killing myself with arrows):

AsyncErr: Lua: Runtime error from mod '??' in callback luaentity_Step(): "Runtime error from mod '??' in callback on_player_hpchange(): ...est/games/VoxeLibre/mods/HUD/mcl_death_messages/init.lua:242: attempt to index field 'job' (a nil value)
stack traceback:
	...est/games/VoxeLibre/mods/HUD/mcl_death_messages/init.lua:242: in function 'func'
	.../.minetest/games/VoxeLibre/mods/CORE/mcl_damage/init.lua:64: in function 'run_callbacks'
	.../.minetest/games/VoxeLibre/mods/CORE/mcl_damage/init.lua:69: in function 'run_damage_callbacks'
	.../.minetest/games/VoxeLibre/mods/CORE/mcl_damage/init.lua:159: in function 'func'
	/usr/share/minetest/builtin/game/register.lua:506: in function </usr/share/minetest/builtin/game/register.lua:491>
	[C]: in function 'set_hp'
	...nk/.minetest/games/VoxeLibre/mods/CORE/mcl_util/init.lua:582: in function 'deal_damage'
	.../.minetest/games/VoxeLibre/mods/ITEMS/mcl_bows/arrow.lua:260: in function 'on_step_old'
	...ames/VoxeLibre/mods/ENVIRONMENT/mcl_void_damage/init.lua:24: in function <...ames/VoxeLibre/mods/ENVIRONMENT/mcl_void_damage/init.lua:23>"
stack traceback:
	[C]: in function 'set_hp'
	...nk/.minetest/games/VoxeLibre/mods/CORE/mcl_util/init.lua:582: in function 'deal_damage'
	.../.minetest/games/VoxeLibre/mods/ITEMS/mcl_bows/arrow.lua:260: in function 'on_step_old'
	...ames/VoxeLibre/mods/ENVIRONMENT/mcl_void_damage/init.lua:24: in function <...ames/VoxeLibre/mods/ENVIRONMENT/mcl_void_damage/init.lua:23>
Next crash (occurred when killing myself with arrows): ```` AsyncErr: Lua: Runtime error from mod '??' in callback luaentity_Step(): "Runtime error from mod '??' in callback on_player_hpchange(): ...est/games/VoxeLibre/mods/HUD/mcl_death_messages/init.lua:242: attempt to index field 'job' (a nil value) stack traceback: ...est/games/VoxeLibre/mods/HUD/mcl_death_messages/init.lua:242: in function 'func' .../.minetest/games/VoxeLibre/mods/CORE/mcl_damage/init.lua:64: in function 'run_callbacks' .../.minetest/games/VoxeLibre/mods/CORE/mcl_damage/init.lua:69: in function 'run_damage_callbacks' .../.minetest/games/VoxeLibre/mods/CORE/mcl_damage/init.lua:159: in function 'func' /usr/share/minetest/builtin/game/register.lua:506: in function </usr/share/minetest/builtin/game/register.lua:491> [C]: in function 'set_hp' ...nk/.minetest/games/VoxeLibre/mods/CORE/mcl_util/init.lua:582: in function 'deal_damage' .../.minetest/games/VoxeLibre/mods/ITEMS/mcl_bows/arrow.lua:260: in function 'on_step_old' ...ames/VoxeLibre/mods/ENVIRONMENT/mcl_void_damage/init.lua:24: in function <...ames/VoxeLibre/mods/ENVIRONMENT/mcl_void_damage/init.lua:23>" stack traceback: [C]: in function 'set_hp' ...nk/.minetest/games/VoxeLibre/mods/CORE/mcl_util/init.lua:582: in function 'deal_damage' .../.minetest/games/VoxeLibre/mods/ITEMS/mcl_bows/arrow.lua:260: in function 'on_step_old' ...ames/VoxeLibre/mods/ENVIRONMENT/mcl_void_damage/init.lua:24: in function <...ames/VoxeLibre/mods/ENVIRONMENT/mcl_void_damage/init.lua:23> ````
teknomunk added 1 commit 2024-11-29 19:14:56 +01:00
teknomunk added a new dependency 2024-12-02 16:22:04 +01:00
teknomunk force-pushed scheduler from 8dac0a34ef to b6869c9211 2025-01-01 19:45:50 +01:00 Compare
teknomunk added 1 commit 2025-01-01 20:11:57 +01:00
teknomunk force-pushed scheduler from e75be34a89 to fca66afac2 2025-01-05 19:55:50 +01:00 Compare
teknomunk force-pushed scheduler from fca66afac2 to af493b6b0f 2025-01-10 12:27:10 +01:00 Compare
teknomunk force-pushed scheduler from af493b6b0f to 7eafdfbf62 2025-01-11 15:15:49 +01:00 Compare
teknomunk changed title from WIP: Task Scheduler to WIP: [retarget upstream] Task Scheduler 2025-01-20 20:48:12 +01:00
teknomunk force-pushed scheduler from 7a798f261f to 7176ad30f9 2025-01-23 13:59:45 +01:00 Compare
teknomunk force-pushed scheduler from 7176ad30f9 to 307d35febb 2025-01-29 03:25:35 +01:00 Compare
teknomunk added 1 commit 2025-01-29 04:16:54 +01:00
teknomunk added 1 commit 2025-01-29 13:13:22 +01:00
teknomunk force-pushed scheduler from d44096fa6b to dde4a0235c 2025-01-29 13:20:28 +01:00 Compare
This pull request has changes conflicting with the target branch.
  • .forgejo/workflows/test.yml
  • test.sh
  • tests/docker/Dockerfile
  • tests/lib/mock.lua
  • tests/luacheck/test.lua
  • tests/luacheck/test.sh
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b scheduler automated-tests
git pull origin scheduler

Step 2:

Merge the changes and update on Gitea.
git checkout automated-tests
git merge --no-ff scheduler
git push origin automated-tests
Sign in to join this conversation.
No reviewers
No Milestone
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: VoxeLibre/VoxeLibre#4716
No description provided.