Implement scripting capabilities?

#1

Hi there,

in the “Where to start with BIG projects” thread there was some talk going on about implementing scripting capabilities in larger 3D-applications. What is the basic idea behind that? I mean not the idea behind scripting itself, but how would you actually tackle implementing such a thing from a developer’s point of view?

Suppose you want Python for your scripting language. Do you have to code the application in Python itself to allow for scripting? Or do you just sort of parse Python commands to be interpreted by the language (e.g. C++) behind your application?
I hope you know what I mean :slight_smile: and sorry if that sounds stupid or something but I’m just curious…

cheers, Martin

#2

I’m also interested in this, for a start check this out:

http://www.gamedev.net/reference/list.asp?categoryid=76#118

#3

Well you can kinda do either.

Personally I’d favour the approach of writing the individual modules of you applicaiton in C as python modules, then using python to “glue” them together into an application.

Writing python modules can be painful tho…

#4

Originally posted by M-J
in the “Where to start with BIG projects” thread there was some talk going on about implementing scripting capabilities in larger 3D-applications. What is the basic idea behind that? I mean not the idea behind scripting itself, but how would you actually tackle implementing such a thing from a developer’s point of view?

It would depend a lot on the design and architecture. For example MEL and MAXScript are custom languages that bear some resemblence with other languages but are interpreted by the host application/core. In MAXScript’s case it’s based on a design that mimic’s the COM model of the Windows architecture (but isn’t, in fact using COM directly). Another design approach would be to expose core functionality in (dynamic) library form so that scripting languages that can access function hooks directly from a library will call upon it (this being a design where the scripting is decentralized and external to the main core).

An important design and architecture decision has to be between the two options; a proprietary interpreted language with an internal messaging system or an external language with.

It would not be advizable to mimic or clone the entire syntax and semantics of a language because it’d require a lot of development and maintenance so external support of existing languages would be preferable. So, no, you don’t go and write an interpreter that reads lines of Phyton code that would then execute functions in your host application, instead you’d write modules specifically against the target language in a way the target language can use them.

The future wave of scripting langauges for applications will, very likely, take on a completely new direction in terms of managed code and exposure of public functions/object. Instead of having to support multiple interfaces or force only compatibility with a single scripting langauge there’d be choice for the end-user in the langauge used to script. We have had a lot of success implementing such a design based on .NET and Mono whereby any .NET enabled language (C#, C++, VB.NET, Perl.NET, Phyton.NET, etc.) can be used as a scripting langauge against the public libs. Internally we rely mostly on C# when used as a scripting language because we can compile it in memory directly to executable code (on the fly) and gain a lot more speed over the current and more traditional interpreted scripting languages. Not having to code against different external specs, being able to change the design at any point and expose those aspects for scripting purposes, and being able to have a choice in the langauge without extra effort (less effort, in fact) along with (currently limited but effective) in-memory compilation support are definitely the way of the future… though it’s still like taking steps out of infancy at the moment (but we’re going to finally say goodbye to the architecting styles of the 80’s!).

#5

Originally posted by M-J
Suppose you want Python for your scripting language. Do you have to code the application in Python itself to allow for scripting? Or do you just sort of parse Python commands to be interpreted by the language (e.g. C++) behind your application?

You’d do it pretty much the way described here: http://www.python.org/doc/current/ext/embedding.html

#6

thank you all for the information :slight_smile: these are some really useful articles and tutorials!

@StefanDidak:
Concerning .NET:
Does that mean that I could write an application with C# (based upon the .NET Framework) and any .NET scripting language (e.g. Python.NET) could call objects and methods directly without me having to worry about writing modules in a way my scripting language is able to “understand” them?
AFAIK all .NET languages are supposed to be based on the common language runtime and common type system and therefore should be able to call any object/method even if they’re written in another language. If that’s the case, is it right then that I just have to implement an API for my application, say in C# or Managed C++, and Python.NET is able to use this API directly? That would be really simple, wouldn’t it? :slight_smile: well, at least from a design point of view…

#7

It would be really simple if all your users were using .NET…

I’d suggest doing it the hard way first… it could be as simple as just providing a bunch of functions to manipulate your app in a python module. Then you can guarantee anyone can use it. Not just the poor sould still using Windows.

#8

Also check out Boost.Python if you are writing Python modules in C++. It looks like it would make things a lot simpler.

http://www.boost.org/libs/python/doc/

Simon

#9

Originally posted by M-J
If that’s the case, is it right then that I just have to implement an API for my application, say in C# or Managed C++, and Python.NET is able to use this API directly? That would be really simple, wouldn’t it? :slight_smile: well, at least from a design point of view…

Correct. And yep, it’s that simple really. Though the approach is more like an SDK and less like an embedded scripting language. However, in a lot of cases you can use the langauge directly. What we’ve been doing involved using C# as a scripting langauge where we feed the source directly into the app which compiles it on the fly and executes it. This gets around the common performance issues that are often related to having embedded scripting (interpreted) langauges. With Windows we’ve had no trouble architecting that though on Linux we had to make quite a bit more effort to get it to work exactly the same way. Eventually, looking at where development of scalable apps are headed, the historical boundaries seperating scripting langauges and plugin architectures are slowly fading away. You could even expose public types and methods in a ‘script’ and have yet other components use them. :slight_smile:

#10

thanks for the link, simon. that’s really impressive… looks quite simple :slight_smile:

@playmesumch00ns: well, yes that’s a point :slight_smile: and with .NET even more so because you not even would have to use Windows but you would have to install the .NET Framework as well to get these applications to run.

@StefanDidak: Just out of curiousity: did you have to actually code this compilation on the fly or is this some neat trick provided by .NET? Is there MSIL involved somehow?
Btw, using C# as a scripting language and compiling it on the fly sounds really cool, but I think that would take away the “ease-of-use” of scripting languages particularly for non-programmers, wouldn’t it?

#11

Originally posted by M-J
did you have to actually code this compilation on the fly or is this some neat trick provided by .NET? Is there MSIL involved somehow? Btw, using C# as a scripting language and compiling it on the fly sounds really cool, but I think that would take away the “ease-of-use” of scripting languages particularly for non-programmers, wouldn’t it?

We didn’t have to code the compilation process since those aspects are part of the class libs (e.g. calling the in-memory compilation process is a matter of using the right objects and methods) but we did have to make a framework around it to sync development between Linux and Windows without introducing structural differences between the source base.

I only mentioned C# because that’s what we’ve been relying on for the most part but you could just as easily add VB.NET support or any other CLR compliant language support that has in-memory compilation support. It’s not been a major requirement for us at the moment but when I asked my lead engineer to look into VB support it looked like it was just as easy to add that by using the System.CodeDom.Compiler.* components and the VBCodeProvider in particular. A good CLR compliant language would support its CodeDom as well so while there are probably some that don’t most likely are already at that stage. Those that do not would require a compilation done by the end-user whereby the “script” is essentially a plugin assembly while those that do could function as full embedded scripting languages that will compile to fast code on the fly.

#12

Hi there. I have similar experience to Stefan, I implemented comparable functionality in one of my apps and wrote a little script editor/manager for it. I used JScript.NET as the primary choice because of it’s ease of use and similarity to normal JScript. It’s both very powerfull (JIT compiled code) and easy to implement. Actually one of the (few) things microsoft got right.

If for some reason or the other (performance maybe? although then scripting is almost always a bad idea anyway :slight_smile: ) you don’t want to go with .NET you can still go with COM and implement the IDispatch interface (Softimage did this for XSI). The beauty of this is that you have native interfaces for C++ (via COM) and another interface for scripting languages that interfaces the Windows Scripting Host, so you can use any WSH compatible scripting language (vbscript, jscript are standard, python, perl and ruby adaptations exist, e.g. ActiveRuby). This route is definitly harder to do but still allows for a range of languages to choose from (so the user can use the one he already knows).

Then you can of course make your own, but that is both a lot of work and rather limited most of the time. There are some scripting languages that were designed with interfacing C/C++ in mind (LUA is one of them, I think). You can often find them in games for level AI scripting or such stuff. Also worth to check out maybe.

HTH,
Daniel

#13

ok, now I got it :slight_smile: thanks for all your explanations. they really gave me some insight…

#14

Simon: Had a go at that boost python thing. It looks really, really simple until you actually have to build the bloody thing! Aaargh! The bjam build tool is a nightmare!

#15

I’m reasonably happy with extending python using the standard system…

It has quite a bit to get your head around initially, but once you understand working with the PyObjects it’s not all that bad…

I’ve done a simple module (with static functions) as well a my own type (think class…)

#16

I think that link was from a link further up the discussion - it just looked pretty neat. Shame if it’s a pain to compile (I have never tried anything like this yet - would like to though).

This is one of my nitpicks with MTOR/RAT. It would have been much better if they had exposed some lower level functions and then script their fundamental blocks. This would mean that they could still keep their code theirs, but be able to extend it better. eg ribattribute is very powerful for adding RIB attributes into the code, but the list it supports is quite limited (unless you only want to do raytracing of course!). I would love to be able to just add an extra case for Matte so that it could be added into the shader code rather than the kludge of RIB boxes to make a simple holdout matte.

Plus now I wouldn’t go with Tcl - more liklely Ruby or Python (if Python had Perl style regex thyen it would be an even easier choice).

I do have to admit that I like scripting stuff as a hell of a lot more than writing it in C/C++.

Simon

#17

too true mate, I’ve been getting deeper into wxPython recently - absolutely brilliant. You can get an app going in no time at all!

Tried Ruby with wxRuby… it was dog slow. Shame, the language itself looks quite nice.

#18

Originally posted by playmesumch00ns
[B]too true mate, I’ve been getting deeper into wxPython recently - absolutely brilliant. You can get an app going in no time at all!

Tried Ruby with wxRuby… it was dog slow. Shame, the language itself looks quite nice. [/B]

I’ve never had a look at wxPython myself… been getting used to using Tkinter, but am not such a huge fan of it…

gotta do a UI this afternoon for a python script, so I might just look into wxPython…

#19

The best approach still today to add scripting languages to your program is to go thru SWIG.
SWIG offers the ability to expose functions and classes thru a very minimal syntax and by reading C/C++ include files. Currently, it supports: Perl, Python, Ruby, Ocalm, TCL, and others. Code produced is usually very optimal and addresses many of the gotchas of each language, unlike something like boost.python which is a pig.
For exposing inheritance and similar, there is usually some more work involved in adding some custom wrapper code (and why something more automatic like CLR could indeed be a godsend), but it is usually rather painless.

CLR (and .NET) is indeed an interesting proposition which could indeed change the way we work…or be replaced or forgotten as so many other microsoft stuff of the past. I think the consensus is that the ideas are solid, but the implementation may still need some more polishing. Microsoft overall does not have a great reputation with languages and interfaces (COM, DCOM, Jscript, VB, etc.) and it is trying to regain a lot of ground lost to others (Java, python, web programming, etc). But it IS the best thing I’ve seen come out of MS in a long time. Albeit CLR is a microsoft standard, it was made freely open. Currently it still remains a windows only thing, albeit mono on linux is slowly making progress on supporting all of it and that could make it a more viable multiplatform paradigm. CLR takes some concepts from Java and others but improves on them (like allowing compiling to native code). The current new languages that supports CLR well, C# and VB.NET indeed shows its potential benefits (lighting fast execution equivalent to C++ when compiled to native code) and the drawbacks of CLR (garbage caching which is slower than references or having no memory management, no multiple inheritance and the CLR is considered not that good for supporting dynamic languages such as perl, python or ruby). The ability of interoperability of languages is possible, but under a certain set of restrictions described as CLS (ie. you may not be able to use some features of your favorite language if you expect the function to interoperate nicely with other CLR languages). Overall, CLR certainly seems like a much more interesting alternative than, say, Java and as mono becomes a solid framework on linux, and other nicer languages than those of microsoft appear, CLR could indeed catch on pretty fast. Of course, don’t expect to write something like a renderer with CLR. Applications where speed are a top priority will likely still remain in the realm of C/C++, but the type of applications where this is needed may become less and less with time.

Regarding wxRuby, forget it for now as the port only has a couple of months. Use rubyFOX, rubyFLTK which are very well supported (and, in my opinion, also better – assuming you don’t mind non-native widgets). Or you can use rubyTK, too, of course, if you are more traditional.

#20

Originally posted by gga

CLR (and .NET) is indeed an interesting proposition which could indeed change the way we work…or be replaced or forgotten as so many other microsoft stuff of the past.

When Visual Studio 7.0 was delayed and the Microsoft “DNA” transformed itself a few times I had serious doubts about the direction they were going into. After having exposed to a lot of it and jumped into it with part of the business I’m fairly confident that it’s not going to be forgotten all that easily (80% of all efforts were *nix related at that point). The impression I got after meeting and talking to key engineers on various CLR/NET portions is that Microsoft is taking a big step and a big risk for their own future by going CLR/NET themselves all the way (and then some). Even if I weren’t confident about the technological aspects of it all I would probably still consider it a ‘safe bet’ given how reliant Microsoft itself is going to be on it all for their future (and however much everyone wishes they go away that won’t happen anytime soon). Then again, if it wasn’t for Mono the weight of any related decisions would probably have weighed in more heavily. :slight_smile:

The initial 1.0 and 1.1 releases have been interesting and as you say there is still quite a lot to be done but it’s shaping up (and for the first time I’ve experienced Microsoft actually listening to developer input as well… refreshing). Things like the garbage collection is shaping up, that and real generics, in 2.0. There are also new weirdnesses to deal with like XAML which takes some getting used to in order to see the benefits.


Of course, don’t expect to write something like a renderer with CLR. Applications where speed are a top priority will likely still remain in the realm of C/C++, but the type of applications where this is needed may become less and less with time.

You raise a very good and important point. Performance is great but not the best you can achieve. But how much can one sacrifice in development effort and cost and how many benefits and performance gains will it bring. For areas like games where every last bit of performance is squeezed out of the system it’s not sufficient yet and probably won’t be for a long while. On the other hand we’ve had great experiences with realtime rendering (not of the game-engine kind but in areas where we deal more with memory bandwidth bottlenecks than CPU crunching limits) by having certain small portions in C++ where we need to do some low level access to hardware, yet are calling most of it through Interop for pretty much everything else (UI, extensions, communications, networking, synchronization with other seperate app components). After four months the results are just 2-3% under the performance of the previous C++ code base. Yet the small percentage frees up two other engineers who can dedicate their efforts in other areas. Cost savings, efficiency, easier code and management, it’s worth the small performance hit in this particular case. For a more traditional (e.g. not realtime) renderer I can see it be applicable in the near future (based on current hard and software direction) though the benefits would probably come in expandability, programmability, control and quicker turn-around cycles in development (good for vendors, good for customers), at the cost of a performance hit that will probably be dealt with by newer/faster systems. A small step backward to make a bigger leap forward.

Looking at software development over the past decade from a technology point of view and then looking at the business and markets in general there’s a huge demand for having things more managable, quicker to market, easier to fix/adapt/refactor, and something that does actually fit in with the concept of reusability (something that has been a buzzword that didn’t apply to code as much as it should have). CLR fits in with that concept quite nicely but the past two years have only been the initial steps. The question is, based on a project that has the ability to start fresh, and depending on the project lifecycle (in x number of years) the weight of CLR/NET is definitely something to consider and examine. As an old-timer it’s been refreshing to actually be able to get things done and choreograph development in ways that used to have alot more hair-pulling and nail-biting moments. :slight_smile: