Monkey2 Update for June

Hi everyone,

It’s mainly been a month of tweaking and tooling around, but there have been a few semi-major developments along the way.

For starters, weak references have been added to the language via the new WeakRef class. It’s taken a while to add these as I honestly wasn’t that clear on how to implement them without incurring major overhead on ALL GC, but it turned out to be not so big a deal and I think they will be really useful.

The main motivation for weak references is to allow ‘caches’ to be implemented. For example, a hypothetical Pixmap.Open could match a filepath to an already loaded pixmap via a StringMap, so if you open the same pixmap multiple times, only the first Open actually does any work (not to mention all the memory saved!). You could do this with a plain old StringMap<Pixmap> and it would sort of work, except that the cache would just grow and grow over time as the StringMap would itself be keeping all open pixmaps ‘alive’ so they would never actually get garbage collected.

I faced this issue with the ‘themes’ system in mojox and allowing for themes to be changed on the fly. The result I came up with was a rather convoluted ‘ResourceManager’ system (which is likely to be deprecated soon) where resource managers had to be explicitly discarded to effectively just remove their enttries from a global cache. It sort of worked, but weak references should be able to achieve the same thing in a much simpler/cleaner way.

Evgeniy Goroshkin, the author of Monkey2’s mighty ‘Ted2go’ IDE, has achieved the near impossible feat of getting auto-completion working with local variables. This was a bit of a headache due to monkey2’s ‘:=’ operator – which allows you to initialize variables without explicitly specifying their type – but after a bit of a tweak to the way ted2go uses mx2cc to generate typeinfo for autocompletion, we came up with a viable solution that seems to work pretty well. Note that you must have a ‘locked build file’ (or set an ‘active project’ using ted2go’s new project system) for auto-complete/typeinfo to work now. The new system this will also create a tmp ‘.mx2’ dir in each *.monkey2 file dir. We hope to be able to move this into the main project .buildv dir eventually, will hopefully have a hack at this very soon.

On the mx2cc front, I think I have a relatively simple way of achieving a form of threading monkey2 without incurring major overhead on the incremental GC system which, IMO, works really well, even though it’s kind of overkill for 99% of monkey2 uses.

The basic idea would be to treat threads more like processes in that each thread would effectively have it’s own (logical) memory space. This would be achieved by ‘deep copying’ any values sent between threads. The exact mechanism for how to do this isn’t quite clear to me yet, but one simple approach would be to give each thread a message queue, and provide Thread.Post, Thread.Wait etc style messaging functions. Whenever a ‘message’ is posted to a thread, it’s contents would be automagically deep copied so the receiver would end up with it’s own copy of the message. An even sexier approach would be to somehow allow users to ‘execute function/method foo on a new thread’, and deep copy the parameters in and return value out.

All globals variables would probably also need to be made ‘thread local’ (ie: each global var would have a per thread copy), at least initially, and this itself would incur some overhead but hopefully not much – more testing required here. This would mean new threads wouldn’t be able to access the App or Keyboard or Mouse etc instances, but that’s OK for now, it just means we’d start small ‘n’ safe and would have to think careful about how/if to share this sort of stuff. It should be relatively easy to shared things like std.filesystem and libc as these are all (?) thread-friendly. I actually prefer this approach to just making everything available to everyone from the outset – which would be impossible without seriously impacting GC performance anyway, something I really, really do not want to do.

Eventually some mechanism for shared global variables could be added (at least for ‘simple’ global types that don’t require deep copying) which could even be made atomic if shared (ie: threads would effectively ‘auto lock’ any shared global vars when reading/writing).

All this deep copying sounds like it would have a lot of overhead, and if not used carefully-ish it could, but I think it would be more than suitable for a wide range of really useful ‘coarsely grained’ parallelism, for example: async resource loading (currently not implemented), playing music (currently implemented in c++), async socket IO (also c++) even possibly handling physics/rendering (would be sweet!). Also note that blocks of raw memory (ie: UByte Ptr stuff like pixmap/audio data etc) would NOT have to be copied as the goal here is purely to prevent GC problems. It would probably (?) be unsuitable for ‘finely grained’ stuff where you do the same thing many times in parallel, eg: fragment shading, ray tracing etc and need to share stuff more often so the copying overhead may be more of an issue. The potential is also there for the system to work inter-process or even across a network (although in that case you would have to copy the UByte Ptr stuff somehow).

User Ethernaut has released a mojo3d scene exporter for Houdini! Hola! Here is the github project: https://github.com/DoctorWhoof/Houdini-To-Mojo3D-Exporter

I’ve been working with him on aspects of this for a while now and we have developed a very cool system IMO. This is hopefully just the start and there should be more to come in the future.

Bye!
Mark

***** Monkey-v2018.06 Mx2cc-v1.1.14 Ted2go-v2.11 *****

Added WeakRef class. Must be constructed with an Object and has a single Target property that returns Null if object has been GC’d.

Added Stream.SharedPath property for obtaining a path to an already open stream.

Changed to stb-image for hdr support so hdr images can now be loaded from any stream.

Added Cancel() method to HttpRequest.

Added Timeout property to HttpRequest.

Implemented HttpRequest on all targets – only GET tested. Windows/Linux currently use wget behind the scene – had a quick look at building curl again and went back to my corner.

Added cheezy godrays effect – see mojo3d/tests/pbhspheres.monkey2

Fixed deprecated warnings in android project template.

Updated android SDK to: Android Studio 3.1.2 ; Android SDK 27 (Oreo 8.1) ; NDK 16.1.4479499 ; Android SDK Tools 26.1.1

Point/spot light now render bounding quads instead of fullscreen quads.

3+

Leave a Reply