AngelScript: The Revenge

[DarkDragon]

Almost exactly a year ago there was discussing about switching ZC to using AngelScript instead of ZScript, and it had some support at the time, and Gleeok and Saffith did quite a bit of work on incorporating AngelScript and porting over some of the engine internals... and then it was backtracked.

Gleeok in another thread posted

This would allow new systems to be created that can simply piggy-back off the old ones but also be tightly coupled with scripts that can actually rewrite built-in behaviors without having major issues or losing compatability (or creating bugs with existing quests). The new npc scripts and supporting of various global callbacks I envisioned would be a good test of that experiment before doing anything else with that. ...It's not been worked on really, but it should be. .. :/

I think I'm going to make putting in the compiler/script processor/script engine my next priority besides the few other things I have here. Even if it's not used immediately, if I can get a ZScript->angelscript conversion utility working you might take it more seriously.

and so I wanted to (re)start a discussion about AngelScript in ZC. Some points in particular I'm curious about:

1. What were the lessons learned last year during the effort to incorporate AngelScript? What are the big challenges?
2. Is it feasible to switch to AngelScript while maintaining compatibility with old quests (presumably, this would require only supporting the ZASM interpreter, and ZScript could be dropped entirely)? Or would an AngelScript-driven ZC live in its own ecosystem separate from the existing program and quests?
3. What concrete steps would need to be taken to adopt AngelScript, to minimize the time until we have a functional release candidate, if we decide to do this?

Making a plan sooner rather than later seems prudent, as it will help dictate whether effort should be put into expanding the power of ZScript, vs. working on its replacement.

[ZoriaRPG]

As I said at the time, I'll mostly be focused on ZScript in any regard. Even if you want to shift over, supporting both, for a while, seems mandatory; and I do not see the problem with crosscompiling. tuning the ZScript parser, introducing some new syntax over time, and slowly migrating would make the whole transition cleaner. I also see no issue with supporting compiling ZScript into AS> The AS model is designed to accept additional syntax types, which is why it was originally an attractive model.

The other end of the question, is that is the change better than say, just making ZScript far more robust, and implementing the types of things that we could do with AS, using the existing language.

I do not see ZScript as something separate from ZC compatibility issues. If you want to support all the qussts ever made, you need to support all the ZScript ever made with them.

Another approach, is to integrate both. I could throw together a block diagram of this, if you want. You could pretty much have a core with two lines of communication, and some interrupts, to allow parsing instructions from both AngelScript, and ZScript, into a core engine; or you could base the core engine on AngelScript entirely, and use ZScript inputs to that, as we do with the present ZC engine. That would probably be my approach to it.

I've said many times, that I do not object to AS in premise, only to the older planned implementation, which was essentially 'We don't want to spend the time to do this properly.'.

Just about everything that AS does, can be implemented in ZScript too, which could allow the user to select their preference, and cross-compile between the two, for the most part, for a while. From there, you can slowly phase out support for ZScript, if you desire; or whatever. I think I plotted out a multi-release timeline for that, and no-one ever wanted to look at it, because they didn't want to ensure that a high level of compatibility exists between old, and new scripts.

There are other options, too. Gleeok wanted to do a C# thing, and I am not a fan of this, mostly because the C# syntax is very much a Microsoft thing, and cross-platform support is--or at least was--farcical. If we wanted to do this, I would like to see the core ZC processes minimised to very basic things, and the script parser take over to handle most events. We can have that conversation separately, but right now you have a total of what, five of us involved?

I would like to focus on ZScript for the next two versions, at least, possibly integrating something new, in part, starting with the second, and slowly progress into it, if we want to do it, over the following two. (That equated to 2.55, 2.60, 2.65, and 3.0.)

I would love to be able to offer both over the long term too, but how problematic that might be, I do not know. The flip side, is that this falls into the same category as shifting to Allegro 5, or other radical changes. The present state of the engine needs encapsulation of some kind first.

@Saffith was the only one who worked on shifting the engine to an AS core. There was a branch with his enemy code, where the internal enemy code was replaced with AS scripting stuff. I recall that he purged it because it was some kind of unworkable nightmare for him, at the time, but I could be mistaken. It would be nice if he documented what he tried to do, what worked, and what did not.

[DarkDragon]

The other end of the question, is that is the change better than say, just making ZScript far more robust, and implementing the types of things that we could do with AS, using the existing language.

Right! This is the key question. I could focus energies on some of the changes that have been discussed on the forums and in chat (making inter-script communication easier, tracking pointers over time easier, scripting weapons and Link, etc). But if it sounds like transitioning to AngelScript makes more sense, I want to avoid introducing new features that will make cross-compilation harder.

I do not see ZScript as something separate from ZC compatibility issues. If you want to support all the qussts ever made, you need to support all the ZScript ever made with them.

Well, it's true that *ZASM* must be supported. But not necessarily ZScript (we guarantee that old quests will play in new versions; I agree it would be very nice if old quests can be edited in new versions too, but this is not as bedrock of a principle IMO.)

Let me ask you this: is there any reason to support ZScript, *other* than backwards compatibility of old scripts? If there were a ZScript cross-compiler, would this be sufficient? Or is there a fundamental benefit to supporting both languages?

There was a branch with his enemy code, where the internal enemy code was replaced with AS scripting stuff. I recall that he purged it because it was some kind of unworkable nightmare for him, at the time, but I could be mistaken. It would be nice if he documented what he tried to do, what worked, and what did not.

Agreed. Taking stock of past lessons is a key first step to planning for how, and if, to transition to ZScript.

[Dimentio]

Let me ask you this: is there any reason to support ZScript, *other* than backwards compatibility of old scripts? If there were a ZScript cross-compiler, would this be sufficient? Or is there a fundamental benefit to supporting both languages?

To please people who use both languages. I myself have not tried AngelScript, but I am to understand that there are quite a few things different with the format of AngelScript then ZScript? Muscle memory is a hard thing to rewire. Isn't AngelScript a more object-based language? Does AngelScript no longer have ints and floats be the same thing like ZScript did? It would take quite a bit to get used to.

Also, there's the fact that people would have to write 2 versions of the same script: one for Angelscript to please the current version users, and one for ZScript to please people still using 2.50.x. People still use 2.50.0, for crying out loud. If we keep ZScript support, people don't have to write 2 versions of the same script.

Finally, think about it: ZScript? As in ZC, or ZQuest? The name just fits.

[ZoriaRPG]

Right! This is the key question. I could focus energies on some of the changes that have been discussed on the forums and in chat (making inter-script communication easier, tracking pointers over time easier, scripting weapons and Link, etc). But if it sounds like transitioning to AngelScript makes more sense, I want to avoid introducing new features that will make cross-compilation harder.


Well, it's true that *ZASM* must be supported. But not necessarily ZScript (we guarantee that old quests will play in new versions; I agree it would be very nice if old quests can be edited in new versions too, but this is not as bedrock of a principle IMO.)

Let me ask you this: is there any reason to support ZScript, *other* than backwards compatibility of old scripts? If there were a ZScript cross-compiler, would this be sufficient? Or is there a fundamental benefit to supporting both languages?


Agreed. Taking stock of past lessons is a key first step to planning for how, and if, to transition to ZScript.

(Emphasis, mine.)

This is precisely why AS was abandoned over every other concern. The ability to download an existing script, and compile it in ZC, must be preserved, whether we change the extant script engine by adding new things to it, or change it out to something else.

Simply supporting ZASM in quests, is blatantly insufficient. I won't even contribute to a ZC version that breaks this capability. 99% of my output for ZC in the last five years, is script-based. It'd be the biggestr slap int he face, to the community, to say, we won't suppose any of that anymore, and it reduces the amount fo available resources for the questmaker to zero.

If cross-compiling is invisible, all the existing quirks are preserved, and all the extant types are preserved in identical syntax, implementing both engines at one time is not as critical. It would be nice, but not critical. If even one capability of ZScript is un-suppprtable, then we need both. Simple as that. It doesn't prevent compiling down to the same functions on both ends of things.

You saw the reaction of just changing a directive. Magnify that tenfold, for any change as radical as this. The only users interested in AS were hardcore programmers, FWIW.

I honestly can't think of any nuance of the language that is not in use.

If you want to drop ZScript entirely, and can't support its codebase, then you may as well just start from scratch with ZC, make it clean, make it new, make it Z3 scrolling banana cream pie, and enable classic quest emulation, as ZC 3.x. This was the exact plan a year and a half ago. 3.x would be wholly new, with AS, and those of us who wanted to continue developing 2.,x would do that, which is where 2,future came from, and the origin of its name.

If you want to go that route, then I'm pretty much out of it until it is reasonably put together. Building an engine from scratch on top of everything else that I'm doing just isn't feasible for me, but once the basic structure is in place, I'd be willing to work on it. Then again, you need more than two or three people for that to be a realistic venture; so you can see why this was dropped. It went in circles for a year, arguing about flipping implementation, until we resolved just to expand what we have at present, and decide on AS after we get a bunch of other stuff out of the way, and do one or two updates; then bring it back to the table.

I really don't feel like arguing for or against it again until we reach that point, and can do at least a minimalistic implementation in a cleaner engine, and see how it works.
@Dimentio : I could call it PuddingScript-Z if you want. The name isn't the issue. I agree that the only way we could do this is if existing syntax is maintained. I also have zero interest in doing it now.

TBH, it just feels as if we're all reaching here, and that we're getting very far ahead of ourselves.

I should note, that on a purely personal level, I favour a more robust ZScript implementation. As long as we're able to do this, and it seems that we arwe, with three of us willing to work on it, then the point of the AS migration--nobody wanted to work on ZScript--is less of an issue. Over time, I think we could support just about everything that AS supports, and possibly more, with a performance gain at the end of the day.

The main push for AS was additional C Syntax, and stuff like structs, that we are on the verge of adding to the existing language.

I just don't have the answers that you want, I suppose, but I don't think that what we're trying to do would make cross-compilation harder if in three to five years, we decide we want this.

[Gleeok]

I'm against coalescing ZScript/Zasm/ and angelscript (hereby referred to as "AS") into some kind of hideous amalgamation (at least initially), mainly due to the fact that AS will have no compatibility concerns or other integration nightmares to worry about. Pounding on the keyboard with a free mind can be a powerful tool after all. :) The 18-bit "fixed" experiment that is a ZScript integer/float/bool is going the way of the dinosaurs though. If anything ever needs to interact with allegro's fixed it can be done in c++ code.

Since everyone has their own agendas for now, I think I'll just start it and see where it goes. I believe I have the most experience with AS ((I've used it for 2 projects in the past) and I don't particularly like getting into theoretical arguments so we'll just see what happens. I'm a bit rusty with AS however, so it's not like there's going to be a new script setup overnight; the bindings alone can be very time consuming. I will "play nice" with everyone and not integrate into ZC anything into the main branch, so there's no fears of extra libraries or anything that will affect what anyone else is doing, so no one has to worry about it.

The priority at first is getting all the basic stuff together and have the compiler to be separate from ZQ so that eventually scripts can be modified on-the-fly--I imagine a scripter writing a script while testing it in ZC and then just pressing 'R' and within a second it can recompile the scripts for them. We'll see how hard this is in practice. I'll have to look into how the module system in AS might work with this as well, so there's enough preliminary stuff to get out of the way before anything major.

[Gleeok]

1. What were the lessons learned last year during the effort to incorporate AngelScript? What are the big challenges?
2. Is it feasible to switch to AngelScript while maintaining compatibility with old quests (presumably, this would require only supporting the ZASM interpreter, and ZScript could be dropped entirely)? Or would an AngelScript-driven ZC live in its own ecosystem separate from the existing program and quests?
3. What concrete steps would need to be taken to adopt AngelScript, to minimize the time until we have a functional release candidate, if we decide to do this?

1) No idea. Although I'm in the "existing enemies should remain for compatibility reasons, and 2.6 scripted enemies get a whole new class and interface" camp, Saffith showed off a taste that you could indeed script every single enemy type ZC has directly in AS, which is pretty amazing. What I'd like to do is just make it easier to do this.
2) No idea!? What do you think?
3) No idea!!!

FWIW: There is a horizon effect at play here DD.

[Tamamo]

1. What were the lessons learned last year during the effort to incorporate AngelScript? What are the big challenges?
2. Is it feasible to switch to AngelScript while maintaining compatibility with old quests (presumably, this would require only supporting the ZASM interpreter, and ZScript could be dropped entirely)? Or would an AngelScript-driven ZC live in its own ecosystem separate from the existing program and quests?
3. What concrete steps would need to be taken to adopt AngelScript, to minimize the time until we have a functional release candidate, if we decide to do this?

1. All enemies can be recreated in angelscript, but problably shouldn't be since they die after one frame for some reason.
2. Yes, as you said support the ZASM interpreter and I think we're good.
3. A detailed design doc how it's going to work would be nice. Yeah I know most you hate those.

[Saffith]

As far as I got with it, it was very easy and worked fine. There was a little issue with the wall masters - for some reason, they made Link scroll to the next screen instead of warping, but it only took a minor adjustment to fix that. Aside from that, everything continued to work perfectly, ZScript and all.

The only real problem I ever had was that it didn't work at all for Gleeok. That seemed to have something to do with how return values were handled, but we never ruled out things like compiler issues, as far as I recall. Really, the reason it was rolled back has more to do with my personal problems than anything else.

There is some question of how much overlap you want between old and new stuff. Should new things continue to be based on the same sprite class, with its fix coordinates and miscellaneous garbage? It would be feasible, and I think not even difficult, for compatibility scripts and new ones to use totally separate base classes and script engines. That's a big plus, in my view - a lot of the legacy cruft can just be moved neatly out of the way.





As for ZScript, it has a variety of issues. The number system doesn't match data types used internally, and the conversion is inefficient and produces unexpected results (2.0/3.0==0). It's difficult to add new built-in functions, and overloading them is currently impossible, as far as I can tell. But the single biggest issue is the extent to which it depends on ZC's internal behavior. In many areas, ZScript was not designed with the notion that things might change.

Consider this simple bit of code:

for(int i=0; i<176; i++) {
    if(Screen->ComboT[i]==CT_WATER) {

This iterates over every combo on the screen and does something whenever it finds water, right? Wrong. It iterates over combos 0-175 and does something whever it finds a combo of type 3.

Say scrolling is implemented. Now instead of screens of 176 combos, you've got rooms with varying numbers of combos. This code doesn't necessarily cover the whole screen anymore. It will still work, but not correctly.

But that's the easy problem. Suppose the combo system changes. Instead of a limited number of well-defined combos, you can now give each combo an arbitrary set of properties. Maybe you make combo #3 a magic-reflecting, poundable hookshot trigger. The script isn't checking for anything like that; it just assumes combo #3 is water. Again, it will still work, but not correctly.

ComboF has similar issues and more. What if it becomes possible to place multiple flags on the same combo? ComboF can't handle that at all.

What about DrawZOffset? Currently, it doubles as HitZOffset. That should obviously change, but existing scripts that uses it were made with that quirk in mind. They won't work correctly without being updated.

New warp system? Warp functions break.

Quirks, bugs, and old behavior can often be imitated, but there are limits. In particular, when a feature is used in multiple different ways, they can't all work. For instance, going back to scrolling again, what happens to drawing functions? Currently, coordinates on the screen and coordinates in the world are the same. If the screen can scroll, that's no longer true. It will have to be one or the other in existing scripts, and some will break.

To some degree, these problems are inevitable, and I'm certainly not under the impression that switching to AngelScript automatically fixes everything forever. The point is that the notion that updating ZScript instead will keep existing scripts working is wrong. Many scripts will break, and it's going to be every bit as difficult and confusing for average users, if not more so. Yes, it will be easy to update these scripts most of the time, but it won't be significantly easier than converting them to AngelScript.

There's nothing AngelScript does that can't theoretically be added to ZScript. But it's a lot more work, and the benefit to users is dubious.

[ZoriaRPG]

I think we all know my views.

(1) Any implementation that does not suport taking a raw, existing zscript file, and processing it, in ZC, is out.
(2) I think that the engine needs heavy reworking before we consider changing the status quo.
(3) I do no tthink that the benefits of AS outweigh the implications of what adding it might do.
( a ) This includes the potential for an exponential number of new bugs.
( b) I do not think that the time involved* versus the benefit is a net gain.
(4) Any option that sacrifices compatibility of any sort, is to me, a bad idea. I do not mean quest binary compatibility, solely.
(5) I worry about the performance hit, which is likely to be notable.
(6) Moreover, I do not have the time to dedicate to properly learning the AS sources at present. i can work on ZC until, probably April. After that, I will likely have a six month or greater gap before I can focus on it readily, and I will be tapering off slowly toward April, as well. I expect it to take months for me to truly understand the AS underpinnings, so that's over a year before I could even know if I would want to work with it.

*I would expect to spent the time required to fully support cross-compiling with no syntax changes, and implement all of the existing features, or quirks. That kind of effort is absurd, and massive, when compared against what changing the engine gains for us. Anything short of this is just a sign that we lack the kind of dedication, or manpower, involved with the project to move it forward from its present engine condition.

This is something that we can still do, a few years down the road, if we still have an interest in it by then. With people actually working on the parser, and ZScript, most of what people want out of AS can be done with what we already have, with fewer risks, better stability, and better acceptance.

Please remember that the old 'plan' was to let us work on 2.x stuff, and make AS a 3.x thing that @Gleeok and @Saffith did, and that they supposedly weren't concerned with some compatibility rifts between these two branches of ZC. I have always wanted to ensure that new versions of the programme are compatible with all of the old materials, including scripts as raw files, not as assembly. I think that we have a plan now, we have focus, and we have people willing to do the work. Rocking the boat, for the fifth time, right now, is just madness.

This changed several times, with lack of communication being the underlying thorn. I'll also note, that seeing a working proof of concept is something I would need, to ultimately decide, but I'm just tired of the whole debate, and want to proceed with whatever amounts to our present, viable plan.