Beginning Scripting - EQ Tutorial Example - Looking for Feedback

[AutomateEverything]

Hi Everyone,

I'm new to this scene. I've never really had interest in "cheating" while playing Everquest, but after several years of not playing, I found myself moderately interested in seeing what I could script within the game, for the challenge of that itself. So I have never really used MQ2 on any real character, or a character above level 3, really. My end goal is to, well, Automate Everything. Can I create a character, and hit go, and have them level up, AA up, equip themselves, buy spells, sell loot, etc until they are max? I'm sure I won't get to the destination completely, but I'm interested in seeing how far I could get. There seem to be a lot of small scripts, but not much with everything hooked together.

That said, I've been playing with the macro scripting language, and seeing what the program is capable of. One of my first goals was to automate the tutorial. Not because it was extremely useful to do, but because it required a good variety of types of scripting to accomplish, exposing me to different APIs.

Since I didn't really find anything like this when I started out, I wanted to share, and also get some feedback if there are easier or newer ways for some of these things. I actually started doing this a year or two ago and lost interest upon realizing how terrible MQ2Navigation worked for larger pathing, since I didn't have any interest in recording every path. When I noticed it was getting fixed (MQ2Nav), I decided to revisit a bit.

This macro is run when you very first login to the game on a new character in the tutorial. It currently handles everything from there until you get the charm after talking to everyone.

Some of the things this includes, for examples:

• MQ2 Navigation use, including waiting until you reach a destination before continuing
• Loops, if statements, and other flow-control
• Functions, with and without explicitly typed parameters
• Interacting with the tradeskill window, including recipe searching and combining
• Finding a specific item slot
• Accepting tasks and their rewards
• Interfacing with the bank
• Auto attacking, minor MQ2Melee
• Handing in quest items
• Finding and picking up ground spawns
• Understanding and integrating spell scribing
• Working with bags
• Holding the script until done zoning
• Placing an augment

I know a lot of these aren't rocket science, but most of them required quite a bit of research to figure out the first time. I wish I had a few better notes from them I started to credit, but unfortunately I did not take many. The one credit I remember is that the item pickup was from some script that worked out of Kaladim I found.

I'm looking to slowly plug along from here, but wanted to introduce myself, share some learnings, and open myself up to some feedback. Is there anything I'm doing here that is particularly bad, or there is an easier/cleaner way? I'd certainly appreciate the feedback.

Note I've broken my code out to a number of includes just for clarity to myself. I'm sure I could do with a bit more cleanup. That said, it is what it is for now.

Run the "Tut.mac" to start. I have included all my custom scripts, hope I don't have any older dependencies floating around I'm not thinking of.

I hope this helps someone, and look forward to some discussion.

edit by redbot: updated N_Moving.inc for new MQ2Nav commands

 

[PeteSampras]

I did some macro writing videos if youre into that sort of thing (that matches up to your first few bullets). I didnt exactly finish because i got a new computer after the first 4 and never installed the recording software on the new computer. But if you are new to macros they are probably of use.

https://www.youtube.com/channel/UCwfrlqbh8nYEEnxuDTR64yw

otherwise, good to see people trying to help others learn to code!

 

[Warl0ck45]

for what it is worth, kudos for making the attempt and not giving up.

As time marches on there is a bigger and bigger gap between the folks who have been writing for scripts for years and those just now trying their hand to it. A lot of time there is the "look at old code" response, but the main code has changed so much over time that what code was doing excellent in 2004 is not even working now. Not being able to find "working code" to build your understanding on, let alone expand to do more, makes for a real obstacle I find. At least, for us folks who do not tinker with code outside of MQ2 =)

No matter where this adventure takes you, give a holler if you need help. Lotsa folks here to help =)

 

[underdog]

I STARTED WATCHING THOSE YOU SHOULD FINISH AND MAKE MORE WITH LINKS AND UPDATES , YOUR REALY CLEAR ON YOUR EXPLANATIONS , EVEN IF ITS JUST A SMALL TOPIC YOU TAKE TIME AND ARE VERY CLEAR

 

[AutomateEverything]

for what it is worth, kudos for making the attempt and not giving up.

As time marches on there is a bigger and bigger gap between the folks who have been writing for scripts for years and those just now trying their hand to it. A lot of time there is the "look at old code" response, but the main code has changed so much over time that what code was doing excellent in 2004 is not even working now. Not being able to find "working code" to build your understanding on, let alone expand to do more, makes for a real obstacle I find. At least, for us folks who do not tinker with code outside of MQ2 =)

No matter where this adventure takes you, give a holler if you need help. Lotsa folks here to help =)

Thanks sir.

As a software engineer professionally, I don't mind the non-working examples as much as the average person, I think. I get the intent a little easier. That said, there is a lot to find, so in the end it all just comes down to how much time you want to invest in trying to find the way it works.

I'm rather astonished there isn't some sort of helper library (that I've found at least) for common, everyday tasks. That's why I started to break these into includes, to start building that instead of just writing the same thing over and over.

As it grows I can see namespacing becoming a larger issue, though. It is still scripting.

 

[Eltai86]

Pete,

Should also bring your other write ups you have done in the past over to here as they are very helpful in combination with the videos.

 

[Warl0ck45]

As a software engineer professionally, I don't mind the non-working examples as much as the average person, I think. I get the intent a little easier. That said, there is a lot to find, so in the end it all just comes down to how much time you want to invest in trying to find the way it works.

-=- notes new person to pester with script posers....-=-

 

[nyghteyes]

Welcome to the party!!

Reminds me of the Dain macros.
9 -10 year old code, but might help with some insights or trigger some thoughts.

 

[AutomateEverything]

Jesus. That Dain macro is crazy. Seems like a time before there were any utilities - the basically made their own recorded AdvPath, along with a bunch of other little utilities.

I bet it worked, but damn is it ugly!

I will say that I appreciate the share, though. I'll read that a bit more to give me ideas on certain things. Certainly seems to have some reasonable examples on netbots for a group. Not 100% that's the way I want to go, but that type of thing makes sense.

On an unrelated note, anyone happen to have an example of how to pass an array to a subroutine? I can't seem to get that working, and I cannot seem to find an example anywhere, including the documentation. I'd like to avoid the massive quantities of global variables if I can help it that I see in most of these scripts.

 

[PeteSampras]

Jesus. That Dain macro is crazy. Seems like a time before there were any utilities - the basically made their own recorded AdvPath, along with a bunch of other little utilities.

I bet it worked, but damn is it ugly!

I will say that I appreciate the share, though. I'll read that a bit more to give me ideas on certain things. Certainly seems to have some reasonable examples on netbots for a group. Not 100% that's the way I want to go, but that type of thing makes sense.

On an unrelated note, anyone happen to have an example of how to pass an array to a subroutine? I can't seem to get that working, and I cannot seem to find an example anywhere, including the documentation. I'd like to avoid the massive quantities of global variables if I can help it that I see in most of these scripts.

you should never need to pass an entire array if it is an outer. you can just send the array or array locations.

/declare this[10,10] string outer
/varset this[1,2] taco

/call routine ${this[1,2]}

Sub routine(whatever)
/echo ${whatever}
/return

or
/call routine this 1 2

Sub routine(array,int x, int y)
/echo ${${array}[${x},${y}]}
/return

both would /echo taco

maybe i just dont get what youre asking.

 

[AutomateEverything]

Hey Pete,

Perhaps its just my background, but I write mostly in higher level language. Globals aren't real good, which outers basically are (for that script instance). I'm looking to keep my scope appropriate.

In the case I was playing with right now, I was trying to create a generic 'find mobs to kill function' which would be passed an array of mobs I wanted to kill. This is particularly useful for kill tasks I'm doing. I define an array of what I want to kill and pass it in, rather than continually needing to update a global one.

Standard arrays, really. As this is based in C++, though, where arrays aren't exactly the most explicitly defined constructs at times, I wonder if the scripting portion doesn't port their parameterization well.

 

[PeteSampras]

Hey Pete,

Perhaps its just my background, but I write mostly in higher level language. Globals aren't real good, which outers basically are (for that script instance). I'm looking to keep my scope appropriate.

In the case I was playing with right now, I was trying to create a generic 'find mobs to kill function' which would be passed an array of mobs I wanted to kill. This is particularly useful for kill tasks I'm doing. I define an array of what I want to kill and pass it in, rather than continually needing to update a global one.

Standard arrays, really. As this is based in C++, though, where arrays aren't exactly the most explicitly defined constructs at times, I wonder if the scripting portion doesn't port their parameterization well.

memory allocation doesnt really matter for what you're talking about. a string is a string is a string for mq2 purposes. they are 2048 characters. even my ultra high end macros that do everything only lag on load. after they are loaded with everything stored as outers i can still run 12+ instances with macros and plugins going full throttle. by all means use local as needed, but for all intents and purposes just use sub routines as the equivalent of functions and use outers for anything that needs passed between subs. also fun little fact is each line in a macro processes basically the same regardless of length. i wrote a diagnostic macro to test all the different /write, /read, and processing, and TLOs. accessing ini and /echos are by far the biggest cycle killers and will lag the crap out of your macros if overused.

 

[AutomateEverything]

Fair enough, Pete. I appreciate the details on the behind the scenes.

I'm still interested in passing the array, though, for good programming practice over the performance. I also have concerns over name collisions as my scripts grow, which is the pinnacle problem of outers/globals. Creating a "library" is not particularly useful if all the standard names collide in any scripts people write with it. I'd like to keep as much as I can out of global scope.

If there is a way to pass them that someone has found, I'm still interested in hearing it. Otherwise I think my next step is trying to find and understand the command parsing code in the source, where I've already admitted that my C++-fu is not super. Such an easy language to make complicated.

~~

Edit: As an aside, writing to "Console" and reading from disk are common performance bottlenecks in most any program that are frequently overlooked. Can't tell you how many times I've had 50x decrease in run-time by turning off all the console writes in simple little utilities or scripts, such as Robocopy and similar.

 

[PeteSampras]

Fair enough, Pete. I appreciate the details on the behind the scenes.

I'm still interested in passing the array, though, for good programming practice over the performance. I also have concerns over name collisions as my scripts grow, which is the pinnacle problem of outers/globals. Creating a "library" is not particularly useful if all the standard names collide in any scripts people write with it. I'd like to keep as much as I can out of global scope.

If there is a way to pass them that someone has found, I'm still interested in hearing it. Otherwise I think my next step is trying to find and understand the command parsing code in the source, where I've already admitted that my C++-fu is not super. Such an easy language to make complicated.

~~

Edit: As an aside, writing to "Console" and reading from disk are common performance bottlenecks in most any program that are frequently overlooked. Can't tell you how many times I've had 50x decrease in run-time by turning off all the console writes in simple little utilities or scripts, such as Robocopy and similar.

can you give an example when you need to pass an array rather than just the name of the array? i mean you could always just /call it i guess if that works.

/call routine this

Sub routine(this[10,10])


The problem is you are limited to what is in mq2macrocommands.cpp in mq2main plugin. storing it as an outer and just accessing it via name would be inversely proportional in speed to the size of the array. when i say i dont get when you would pass an array is because you would store an array just like any other vector/deque/struct in c++. you would do that without hesitation so why would you not have an outer for an array and call it done? im only an amateur coder, so maybe I am missing something, but i cant see any version of passing an array being better than just having it stored as an outer.

 

[AutomateEverything]

I'm going to revisit in the next day or three and try more options, but what you have proposed does not work from what I tried. Unfortunately not as straight forward as I hoped.

I gave an example above, where I was trying to use an array as a parameter, but I think your question really boils down to why I wouldn't want to just use outer.

In typical programming you want to keep globals (which outers really are, for most intents and purposes. "globals" in MQ seem to be far beyond normal globals) to an absolute minimum. A professional program has a grand total of zero if at all possible, which should be possible in most cases.

There are a few reasons for this. Global memory scope is typically a consideration, as globals get loaded at program start and last for the entire duration, versus instantiated variables that are dynamically allocated and deallocated throughout the program life. This was, in fact, a giant limitation of Everquest itself for many years. A lot of the character models that had to be shared across zones (such as illusions) were stored in global memory scope, and it got so bad over time the team had to completely re-write it, as they couldn't reasonably add more to the global memory space. That was the time they added the memory options to Everquest options, if I recall. And illusions took waaaay off afterwards, since they had the ability.

The second major reason revolves around namespace collisions. If I name a global "Mobs" and its hidden in an include file, this can lead to both strange behavior and a lot of debug time wasted when someone tries to declare a local, or even another global, called "Mobs." By keeping the variables used local, you keep them out of the global namespace and prevent these collisions. It saves a lot of debug time and prevents silly errors that happen far more often than you would think.

At the end of the day, the C++ behind the scenes is going to pass this by reference regardless as an array. The way C++ really treats this is that the variable holding the array contains little more than a pointer to the starting location of the array in memory. When you pass the array, you are really passing a reference to the original object, as you are in pretty much any dynamically allocated construct. So it's not expensive - it's actually just as cheap as passing a string or something - but it keeps the variables out of the global namespace, and still lets you access the data on the other side of the function call.

So, programming 101 - globals are evil and they should die. They are going to be considerably harder to avoid in a scripting construct (classes would make this way easier in places, along with static variables, which solve *some* of the global problems), but I'm doing my part to try to keep things clean.

 

[PeteSampras]

The passing of variables causes more system delay than storing the info in the outer because of the mq2macrocommand.cpp. So you can put in all the effort for almost no actual effect on system performance, or you can just use the best process with the tools actually at your disposal. Good luck with whatever choice you go with!

 

[playj]

OP, really nice macro stuff there! Nicely organized too.

I had the same idea, you just took it 10x further than I did.

I did a simple finish Tutorial A in 2006: http://www.redguides.com/community/showthread.php/30303-tut-mac-Finish-Tutorial-A

I had the idea of doing more but got lazy as I do not keep my characters in tutorial, since there are faster ways of getting xp.
I am a big fan of MQ2AdvPath for automating most routine running. (I had some macros that run over 6 zones)

The ultimate would be the macro that does everything up until 105, but does not use any outside assistance. That is type tut.mac in Tutorial A and come back 2 months later to a Level 105

 

[AutomateEverything]

The ultimate would be the macro that does everything up until 105, but does not use any outside assistance. That is type tut.mac in Tutorial A and come back 2 months later to a Level 105

Thanks playj. That's the goal, quite literally. I'm absolutely sure I won't have the time or patience to completely get there, but I'm also sure it will be interesting regardless.

- - - Updated - - -

The passing of variables causes more system delay than storing the info in the outer because of the mq2macrocommand.cpp. So you can put in all the effort for almost no actual effect on system performance, or you can just use the best process with the tools actually at your disposal. Good luck with whatever choice you go with!

Hi Pete,

I actually wanted to look into this. I never really mentioned system performance (memory hogging, sure, but not performance), so this wasn't my goal to begin with.

That said, what you said had me moderately concerned. I do want to make sure that what I write performs reasonably. I generally prioritize clean code over performance pretty heavily unless I absolutely need to do otherwise, but this is a real-time game, so I figured it would be worthwhile to check in on this.

I wrote a quick and dirty script to test this out, which also gave me the opportunity to learn a bit more about timers, which is cool. Looked like this:

Sub Main



	/declare val1 int outer
	/declare val2 int outer
	/varset val1 4
	/varset val2 3
	
	/declare valP1 int inner
	/declare valP2 int inner
	/varset valP1 4
	/varset valP2 3
	
	/declare i int inner
	/declare j int inner

	/declare start timer inner
	/varset start 1000
	/for i 0 to 10000
		/call NoParams
	/next i
  	/echo Seconds to call without parameters: ${Math.Calc[(1000-${start})/10]}
	
	
	/declare secondTimer timer inner
	/varset secondTimer 1000
	/for j 0 to 10000
		/call WithParams ${valP1} ${valP2}
	/next j
  	/echo Seconds to call with parameters: ${Math.Calc[(1000-${secondTimer})/10]}
/return

Sub NoParams
	/declare val3 int inner
	/varcalc val3 ${val1} + ${val2}
/return

Sub WithParams(int param1, int param2)
	/declare val3 int inner
	/varcalc val3 ${param1} + ${param2}
/return

I don't have a very fast CPU - I'm actually running a first gen i7 920, so its quite old (almost convinced myself to upgrade!), so this wasn't tested with something blazing test to throw off the numbers.

The numbers I came out with were 2 sets of tests. I first ran this with loops of 1000, then increase to loops of 10000 you see in the code above.

In the 1000 iteration loop, both ran at 3.30 seconds, absolutely no difference in 1000 iterations.

In the 10000 iteration loop, the call without parameters ran at 32.90, while the call with parameters ran at 33.10. A difference of only 0.2 seconds over 10,000 calls! I think I can handle that and reasonably say I'll definitely prefer the cleaner code, as the performance is not impactful.

I hope this helps someone else who is looking at similar decision possibilities.

 

[PeteSampras]

I think you misunderstood what i meant. As previously mentioned, a line of code is a line of code to mq2 macro parser. a string is a string is a string for memory purposes. a turbo block directly related to the numbers of lines of macro code are parsed per mq2 pulse. #turbo 20 will read 20 lines of code per pulse. #turbo 500 will read 500. It varies based on system, running applications, and more directly plugin/macros that are running. MQ2 does about 15 pulses per second on my system with what i typically run.

So if you are passing a single set of variables, its the same as passing no variables. Because a line is a line is a line. However, if you are /declaring locals in each sub, then those are eating cycle time. You should avoid declaring over and over if you can help it to save cycle time. If you no longer need an outer at all, then simply /deletevar.

Sub hi
/declare i int local
/declare x int local
/varset i 3
/varset x 1
/return

will take 4x as long to process as:
Sub hi
/multiline ; /declare i int local ; /varset i 3 ; /declare x int local ; /varset x 2
/return

because that is just how it is. So using outers saves you cycle times. And the memory of just storing arrays is tiny. You are worrying about single trees in a vast forest that have practically no bearing on performance compared to other issues because of how the .cpp is set up to process macros. For that very same reason, plugins run much faster because they dont have to use 80 lines of code to processes every line of script.

 

[AutomateEverything]

I'm now following what you are saying.

Looking at it that way, and with some minor testing with your exact functions, I'm seeing the second one run at 3/5 of the time or so.

Either way, that still means I can execute a thousand lines of code in under a second and a half, at the default 20 Turbo. Running it at the 80 Kiss does (I still don't have a good concept for what is a stable figure, so using that as an example), I can run nearly 3000 in a second.

So in the end, I guess I'll just have to disagree on this one. While I find the information interesting and appreciated, the performance trade off is not enough to turn me away from the maintainable code. If it sheds any light, I'll throw out one of my favorite computer science articles of all time - http://www.flounder.com/optimization.htm. I firmly do not believe in optimizing code before it is necessary.

Either way, the dialog has led me to learn more about some of the underlying structure, and that is appreciated.

 

[PeteSampras]

I'm now following what you are saying.

Looking at it that way, and with some minor testing with your exact functions, I'm seeing the second one run at 3/5 of the time or so.

Either way, that still means I can execute a thousand lines of code in under a second and a half, at the default 20 Turbo. Running it at the 80 Kiss does (I still don't have a good concept for what is a stable figure, so using that as an example), I can run nearly 3000 in a second.

So in the end, I guess I'll just have to disagree on this one. While I find the information interesting and appreciated, the performance trade off is not enough to turn me away from the maintainable code. If it sheds any light, I'll throw out one of my favorite computer science articles of all time - http://www.flounder.com/optimization.htm. I firmly do not believe in optimizing code before it is necessary.

Either way, the dialog has led me to learn more about some of the underlying structure, and that is appreciated.

so it is that 3/5 because the sub name, the return, and the call all take the exact same amount of time as each of the lines:

1. /call hi
2. Sub hi
3. /declare i int local
4. /declare x int local
5. /varset i 3
6. /varset x 1
7. /return

1. /call hi
2. Sub hi
3. /multiline ; /declare i int local ; /varset i 3 ; /declare x int local ; /varset x 2
4. /return

it should take 4/7 of the time to run the second one, not counting :goto loops, /for iterations, etc that also add individual lines to the processing.

 

[Incognito]

I think it's important to come right out and say that we are bound by the limitations of the platform we are writing within. I certainly don't think Pete is suggesting spaghetti code, but rather pointing out due to the organism in which we are writing there are inherent limitations that are not overcome by passing in references. It is beyond our control, it is what it is. Thx Pete for so clearly outlining the beast we are up against.

 

[AutomateEverything]

I think it's important to come right out and say that we are bound by the limitations of the platform we are writing within. I certainly don't think Pete is suggesting spaghetti code, but rather pointing out due to the organism in which we are writing there are inherent limitations that are not overcome by passing in references. It is beyond our control, it is what it is. Thx Pete for so clearly outlining the beast we are up against.

Yes, it was a little lost on me at first that literally the only thing being passed was the potential value. Considering there were reference-able types, this didn't jive for me mentally. I get it now, though it's an unfortunate limitation. Honestly wonder if it would be that hard to implement back in the main source, but really doubt it would be used enough to be worth the effort.

 

[eqmule]

Honestly wonder if it would be that hard to implement back in the main source.

If you figure that out id be happy to add it to the main core.

 

[Redbot]

Updated N_Moving.inc for new MQ2Nav commands

 

[Rocker]

Guessing the latest patch broke this macro. When I tried using it this morning the macro would end right after moving to the glooming deep jailor. Seems to stop at the while loop in N_Moving.inc Sub WaitUntilMovedNavPathing(targetSpawn)

Anyone know how to fix this?

Sub WaitUntilMovedNavPathing(targetSpawn)
/nav locxyz ${Spawn[${targetSpawn}].X} ${Spawn[${targetSpawn}].Y} ${Spawn[${targetSpawn}].Z}
/while (${Navigation.Active}) {
 /delay 1
 /doevents
}
/return

 

[Redbot]

Guessing the latest patch broke this macro. When I tried using it this morning the macro would end right after moving to the glooming deep jailor. Seems to stop at the while loop in N_Moving.inc Sub WaitUntilMovedNavPathing(targetSpawn)

Anyone know how to fix this?

Sub WaitUntilMovedNavPathing(targetSpawn)
/nav locxyz ${Spawn[${targetSpawn}].X} ${Spawn[${targetSpawn}].Y} ${Spawn[${targetSpawn}].Z}
/while (${Navigation.Active}) {
 /delay 1
 /doevents
}
/return

I believe rephaite has been working on this

 

[Rocker]

I could not get it working using /while so below is the macro modified to get around the /while issue. Also it does the merc portion of the tutorial quest. I had to unload MQ2Autogroup so it didn't suspend the merc instantly but the macro loads MQ2AutoGroup when it finishes.
Large toons tend to get stuck. Creating a mesh of the two tutorial zones using tile sizes around 60 fixes most of the issue but ogres will still occasionally get stuck.

Modified files:
N_GeneralUtil
N_Moving
N_TutorialIntroQuests
Tut

Not changed:
N_Env
N_Spells
N_UIInteractions

 

[kaen01]

when making mesh set agent height to 10, and width to 2.5 ogres should be able to get around then.

 

[jackstraw01]

I know this thread is old, but I made some updates for those that may be interested. I've used it for a few toons to get them out and ready for plvl in CR.

Added:

Subs (methods) that take you up through Innkeeper in CR. Also added code to level you up to 6, but that is still being tested.

Flow: Zone to CR -> Get welcome quest --> Go to Guild Master - Turn in letter and equip tunic --> Go to quest giver --> Go to Innkeeper --> Go to early mob area outside of CR and start fighting (after /mqpause)

Files:


earlylevels.inc - Contains the subs for the above flow
N_GeneralUtil.inc - Added a basic GM check (not sure if it works).
N_Moving.inc - Added a couple of new moving methods
N_UIInteractions.inc - Added a couple of new UI Interactions
loot.inc - Modified version of Maskoi's LootIt. Requires NinjaAdvLoot from Level 2.
equipment.inc - used to equip chest piece
combat.inc - Contains Kill/Pull methods
tut.mac - Mac that runs you through to CR noobie killing field


Hope this helps.