Question - Looking for an experienced VV / MacroQuest Lua dev to help with a data-collection script

[shatter]

What I want to build​

I'm building an external-observer AI agent for EverQuest — screen capture + vision model + LLM + mouse/keyboard actions, running entirely outside the EQ process. No injection, no hooks, no memory reads. To get an idea of what I want to toy around with check out the Starcraft community. https://github.com/google-deepmind/pysc2. AlphaStar didn't learn to play StarCraft II from scratch via self-play — it was bootstrapped on ~1 million human replays via supervised learning, and only then refined with RL and league play. Every serious StarCraft bot that came before it (CherryPi, SAIDA, TStarBot) also leaned heavily on imitation or hard-coded expert systems as the foundation. The blunt lesson learned during the 2010s was: pure RL on a complex game from a cold start does not work at a practical compute budget. MacroQuest can provide a great foundation for this. The real question is can it be partially driven by an LLM alongside smaller RL models and a custom build expert system hosted on the same machine that runs EQ natively. My guess is yes!

What I want to build is a one-time data-collection harness that runs under MQ on an emulator (TAKP / Quarm / P99) or Live, logging perfect in-memory game state alongside timestamped screenshots. That becomes labeled training data for the external vision pipeline. The final system never touches MQ — I just need it once to bootstrap the dataset. Target: ~20–40 hours of clean data across a couple of low-level old-school zones like West Freeport and various class/level combos.

Yes, the broader project shouldn't need MacroQuest once it gets going. Not trying to hide that. But MQ gives perfect structured ground truth, which is exactly what you want to train vision models against screenshots. This is a scoped and finite not ongoing automation work.

__​

Background: I'm a Principal Machine Learning Engineer by day (demanding job, without much free time). I use agentic AI tools daily at work and can produce the Python side of this project quickly. I'm also EQ player since 1999 — classic-era grognard, 6-box RG alum — though I've barely played in recent years. In grad school, I studied Reinforcement Learning and NLP.

The ML/AI side I've got covered. I've also read the MQ Lua docs and the bindings/ source enough to write the spec below. What I don't have is production Lua / VV experience — I've never shipped an MQ macro or plugin, and the RG community has people who can build this in a weekend where it would take me a month of fumbling.

If existing plugins cover part of this already, I'd rather build on top of them than reinvent. Happy to hear what you'd reuse — no need to rebuild an event manager from scratch if something already does 60% of the job.

The script at a high level​

Runs on a test character on an emulator while that character plays normally (manual, VV, or CWTN). Emits two synchronized log streams:

1. State ticks — structured game-state snapshot every 200 ms (player, target, merc, pet, buffs, gems, disciplines, positions, group, nearby spawns).
2. Event stream — combat, spell, chat, loot, death, zone, XP events with structured fields.

Plus a capture-sync signal so an external Python process can screenshot the client on each tick. Timestamps align → (screenshot, state, events) tuples → labeled training set.

Requirements​

I've read through the MQ Lua docs and the src/plugins/Lua/bindings/ source, so I know most of the state fields below are one-line TLO reads. The real work is event authoring, robust file handling, and not starving the frame loop. Framing the ask accordingly.

Environment

• Modern MQ with Lua on an emulator (TAKP preferred; Quarm or P99 fine)
• Lua only — no .mac
• Must coexist with VV / CWTN / Muleassist without interference

State tick — ~200 ms cadence, JSONL

Drive tick cadence on mq.gettime() deltas (not raw mq.delay(200)) so drift stays bounded under load. Each tick: high-res timestamp + monotonically increasing tick ID, then standard TLO reads for:

• Me: HP/Mana/End (cur/max/pct), X/Y/Z/Heading, Zone.ShortName, Class, Level, Combat/Sitting/Invis/Mount/Casting
• Target: CleanName, Type, PctHPs, ConColor, Distance3D, Level, Class, Casting
• Me.Mercenary.*, Me.Pet.*
• Me.Buff(i) — {name, duration_s, caster}. Caster can be nil/unreliable for buffs you didn't cast; keep the field but don't paper over the gap.
• Me.Gem(slot) + GemTimer + SpellReady
• ActiveDisc + CombatAbilityReady / CombatAbilityTimer for the class standard set
• Group.Member(i) — accept that Mana/End for non-self members is sparse until inspected
• SpawnCount('npc radius N los') + NearestSpawn(1,'npc radius N xtarhater') — exact search string is a config knob

Full field schema doc welcome; bike-shedding the exact field names isn't the point.

Event stream — JSONL

There's no built-in event catalog in MQ, so this is the part I'd most like your judgment on. Target events: melee hits/misses, spell cast/land/resist/fizzle, death, loot, XP/AA gains, faction, chat (all channels), zone transitions, engage/disengage, res. Each event: {ts, tick_id, kind, actor, target, amount, detail}, unknowns preserved as kind: "other" with raw text.

If you have an existing event library (Mighty Lua Event Manager, a private one, whatever) that I can build on, that's ideal. mq.event + regex-ish patterns + mq.doevents() in the main loop is the expected pattern.

Capture-sync

Each tick, signal the current tick ID out of MQ so the external Python process can match a screenshot to it. Options I'm open to, in rough preference order:

1. mq.actors message on a named mailbox — cleanest if Python can receive it (I may need to write a tiny bridge)
2. Write tick ID to a sidecar file (atomic rename or single fixed-size line)
3. Named pipe / local UDP

Use whatever's simplest and doesn't block the tick.

Config

Lua table in a file (config.Lua) or INI — whatever would be fastest. Tick rate, field-group toggles, event-type toggles, output dir, rotation size, spawn-search strings.

Action log

Ideal: log every /keypress and /doKey issued by any running script — {ts, key, source_script, command_text} — for behavior cloning. I understand /keypress goes straight through MQ without a Lua-visible hook and observing other scripts' keystrokes isn't doable in pure Lua. So: if this is a C++ plugin ask, flag it and skip. Logging this script's own commands is fine.

If anyone has any insights on how to best structure this, let me know!
Happy to answer questions about the broader project. I recognize that the data collection will require upwards of 200GB for several hours of gameplay, but that's Ok!

 

[vsab]

You didn't cover why?

 

[kaen01]

isnt takp like the mac version of eq that no one but like 5 people use anymore? and only supported by one emu and the client for that is not supported by mq?

 

[B_I_G__D_A_D_D_Y]

You didn't cover why?

I got the impression the project would be to have an LLM capable of assisting players in EQ without any form of MQ just by "watching" the video output. It would probably be undetectable using current methods because nothing in the EQ file library would be modified. It sounds like a fun project and maybe EQ is just a training ground or proof of concept for more financially rewarding games.

 

[Douler]

I’m so lost you can’t use MQ on P99 (or Quarm), but it’s one of your target clients and must have VV. (Plus you want actions not available in those clients.

 

[shatter]

"why" is a fair question. Longer version, because it also answers the "is this just a ChatGPT-reads-the-screen toy" questions I'm sure people will ask:

The target architecture is three layers, and the heavy lifting is the middle one — not the LLM.

1. Vision model — pixels → structured game state. HP/mana/endurance bars, target info, buff icons, spell gem readiness, combat log text. Cheap CV + OCR for the easy parts, a small VLM (possibly PaliGemma (3B), which is lightweight, open-weights model from Google designed for varied vision-language tasks. These models can be run locally). It's a perception problem, not a reasoning one. Given enough compute, you could run multiple small VLMs as part of a larger system in conjunction with CV + OCR tooling.
2. Reinforcement-learning policy — state → action. This is the actual decision-maker, and it's the piece I care most about. I'm leaning explicitly on what the StarCraft research community built out before LLMs became the trendy answer to everything — DeepMind's AlphaStar, Facebook's CherryPi, Tencent's TStarBot, Samsung's SAIDA. That whole lineage showed the recipe for real-time games with partial observability, long horizons, and large action spaces: imitation learning from human replays first, then self-play RL on top. EQ's action space is much smaller than SC2's (no build orders, no economy), but resource management, pull pacing, and positioning have the same shape. The playbook should transfer. There's multiple RL tools that can be exploited for this, like https://gymnasium.farama.org. My assumption is each zone will have to have multiple RL sessions to train policies.
3. Expert system — hardcoded safety rails and structural rules that shouldn't be learned. "HP < X and no heal incoming → gate/camp." "Never engage a red con solo at classic levels." Things where the failure mode is a corpse run, and where an RL agent shouldn't have to discover "don't die" from ten thousand death samples. The MQ community, and especially Red Guides with CWTN, DanNet, etc. can offer plenty of guidance and data here.

The LLM is in the stack but in a narrower role — parsing rare chat events, high-level planning ("low on hp/mana, find a safe med spot"), fallback for situations the RL policy hasn't seen. Not the primary driver.

The decade old Starcraft AlphaStar recipe is underexplored on MMOs compared to RTS. The research question ("does imitation-learning-then-self-play work on a game with 6 controlled units and a 24/7 world instead of a 40-minute match?") is interesting to me independent of whether the agent ever replaces my 6-box. Heck, I'd like to see if I can get it work with just one account!

And yeah, I want to test this out on either Live or a friendly EMU.