Monkey2v1.1.08 now available!

Posted on November 1st, 2017

Hello all,

Monkey2v1.1.08 is out now at blitzresearch.itch.io!

First up, there are a number of cool language additions:

The makedocs system built into mx2cc has been rewritten from scratch because the old one had become such an unholy mess I couldn’t really do anything with it any more! Module ‘manual’ docs should now be placed in a ‘newdocs’ dir inside the module dir, and all docs are now built into the same location, ie: the docs/modules dir, which IMO is a whole lot more useful than the old system which used to build docs into the individual module dirs. You also no longer have to makedocs before uploading a module via the module manager page.

A simple ‘component system’ has been added to mojo3d, although not much has been done with it yet – I have converted the Animator, Collider and RigidBody types to components, but more as a proof of concept than anything else (that said, I’m quite happy with it and can’t see it changing much). I plan to spend considerable time over the next month getting this stuff into shape – The non-rendering side of mojo3d has all been a bit ‘half done’ for too long now! I am relatively happy with the current state of the renderer (it generates some weird warnings in the android emulators which needs fixing, but it does at least work on all emulators I throw it at), but there is of course more to 3d engines than just rendering – esp. these days! That said, I *have* been experimenting with a more modern terrain system for mojo3d in my spare time and think I’ll eventually be able to do something pretty cool terrain-wise.

There’s a new PlayMusic method in AudioDevice for streaming audio from file – see new musictest banana. This should work on all targets except for emscripten, which doesn’t support threads so will need to use a different mechanism, probably involving some calls into native javascript. Audio must currently be in ogg format, although mp3 support is now a possibility due to several mp3 patents having apparently expired. PlayMusic returns a ‘channel’ object which can be used to control music volume, pause music etc. With the addition of music streaming, I do feel like monkey2 is finally ‘fully equipped’ for writing 2d games with, although of course there will be many refinements and bugfixes to come!

Apart from more work on mojo3d, my next goal is to try and put together a more ‘finished product’ by the end of the year. This will involve more work on both the docs and examples, esp. for the monkey, std, mojo, mojox and mojo3d modules, which really form the ‘core’ of monkey2.

Finally, apologies for the late ‘monthly’ release….no excuses really, just been a weird couple of months in various ways (IRL too).

Bye!
Mark

12 Comments


Monkey2-v1.1.07 out!

Posted on September 14th, 2017

Hi,

After a bit of a delay, Monkey2 V1.0.07 is now out (from http://blitzresearch.itch.io/monkey2 as usual)! Actually, I was a wee bit unwell near the start of August which took me out of commission for a while, but I’m back now baby!

I did manage to get some pretty useful stuff done though, including:

Improved forward renderer

The forward renderer has been majorly overhauled/improved. This can now handle any number of lights, with or without shadows, but there is still room for improvement here. Currently, each light involves an additional rendering pass, which is not strictly necessary if shadows are not enabled, in which case several lights could be rendered in a single pass. This optimization will have to wait for later, but for now, as long as you keep number of lights/shadow casters down, the forward renderer should perform well.

The forward renderer actually runs in one of 2 modes – direct or indirect. Direct forward rendering draws directly to the display which means the app can’t apply ‘post effects’ unless they’re very simple. With deferred rendering, post effects can effectively ‘read’ the color/normal/depth buffers after the scene is rendered, but with direct forward rendering these buffers are not available so all post effects can really do is alpha blending/additive etc style effects. So I added the indirect forward rendering mode to allow for a wider range of post effects. With indirect forward rendering, the same rendering algorithms are used, only the screen is rendered to an internal offscreen color/depth target which can be accessed by post effects. It’s sort of like a half-assed deferred renderer. The range of effects possible with the
indirect forward renderere is still more limited than with a full on deferred renderer whcih has complete access to output colors, positions, normals and specular, but it’s better than nothing.

Activating the forward renderer is a little klunky right now. You need to do something like this in Main():

This is a bit of a mess and will be cleaned up in future.

Point light shadows

I’ve also finally got point light shadows working. This wont be terribly fast yet (shadows aren’t) because it needs to render the entire scene to 6 side of a cube map. Future optimizations here will include culling (only render what’s in light range) and the ability to ‘mask out’ cube faces. This masking can also be used to create spot lights where only one cube face really needs to be rendered. You could also create ‘dual spot lights’, ie: 2 spot lights pointed in opposite directions (which only requires 2 cube faces).

The algorithm here currently packs depth values in RGBA texels, as webgl doesn’t support depth cube maps. This works surprisingly well (at one point I had the whole engine using RGBA depth maps) although should be optional for HW that support depth cube maps (packing/unpacking RGBA pixels is a little ugly).

Linear vs SRGB colors

Another mojo3d issue which has been lurking for a while is the renderer’s use of ‘linear’ colors. As some users have discovered, the colors you assign to materials etc in mojo3d don’t look quite the same as they do in paint programs. The reason for this is that the mojo3d shaders assume color parameters are actually in ‘linear’ color space, whereas colors are usually specified in ‘sRGB’ color space. The main difference here is that color intensity (brightness) is linearly proportional in linear color space, but not in sRGB space. Mojo3d internally converts texture colors from sRGB->linear space, but it doesn’t convert color constants – well it did’t. I have modified the mojo3d APIs so that any color property (eg: Material.ColorFactor, Scene.ClearColor) now assumes colors are always in sRGB space, and converts them as necessary. So now you can assume that any texture or constant color you pass to mojo3d will look the same as it does in photoshop etc.

As a side effect of all this, note that the direct forward renderer uses additive blending for lighting effects, but in sRGB color space intensity is not proportional, so the lighting effects in the direct forward renderer (which assumes target buffer is sRGB) are technically wrong! However, the difference is subtle and no more wrong than Blitz3D etc were. It’s just a compromise that needs to be made if you want to render directly to an sRGB buffer.

Particle systems

There’s a new mojo3d PaticleSystem entity too. This is actually just a container for a particle buffer and a particle material. A particle buffer contains the actual vertices that makes up the particle ‘cloud’, and has properties such as duration (how long each particle ‘lasts’), min/max velocity and cone angle. Particles are automatically added and removed to a particle buffer as necessary each time the scene is rendered. A particle buffer can be shared amongst multiple paticle systems, much the way a mesh can be shared amongst multiple model entities.

Currently, a particle system’s origin and orientation are locked to the actual particle system entity, so if you move the entity, all particles move with it. This can can look unrealistic depending on what you are doing, so particle systems are currently best suited to static things like torches on walls etc. However, it should be possible for each particle to have individual position/orientation in future, allowing particle systems to be attached to thrusters etc and to able to leave ‘trails’ etc. Actually, you can probably attach a particle system to a thruster right now and it’ll probably look OK with the right tweaks.

Mojo outline mode

I also added a simple outline mode to mojo. It doesn’t do anything terribly flash like outlined fonts or image edge detection (outline of an image is just a rect border) but I think it shuld be useful for visualization stuff. For a simple demo, I added outlines to the hellochipmunk/chipmunkdebugger banana.

Speaking of bananas, Ted2go now has a banana samples viewer which launches on the first run of Ted2go (or can be relaunched via help menu). This is already looking very nice, and will look even better once I’ve cleaned up the bananas a bit. Plus we need some 3d bananas, currently all 3d sample code is in tests dir in vaiours mojo3d module dirs.

And speaking of IDEs, the monkey1 mollusk IDE now has early monkey2 support. This is a nice, clean, very easy to use IDE (Windows only) and you can find it here: http://grudlux.com/mollusk/.

Finally, please also consider throwing nerobot (the author of Ted2go) a few bucks via paypal here: https://www.paypal.me/engor/10. He has done an amazing job of tweaking and polishing the original Ted2 IDE, a huge feat considering it uses a completely unique GUI system and a completely unique language. But it has proven to be an excellent ‘test case’ for monkey2 I think. It runs virtually identically on windows, macos and linux (to a degree not even Qt could manage) and has a ton of intersting things going on under the hood.

Anyway, that’s enough from me for now – here’s the new stuff in v1.1.07 VERSIONS.TXT:

***** Monkey2 v1.1.07 *****

Android min-sdk now 14 for android projects.

Added Scene.DestroyAllEntities.

Added new ‘instance’ uniformblock type for better (not great yet) instancing.

Enabled assimp FBX and BLEND loaders.

Sprites now compare with depth buffer.

Added Sprite.TextureRect:Rectf property for simple 3d sprite animation.

Fixed Texture.Flags setter.

Added simple outline drawing to Canvas, see: Canvas.OutlineMode, Canvas.OutlineColor and Canvas.OutlineWidth. See chimpmunkdebugger.monkey2 in hellochipmunk banana for demo.

Implemented point light shadows using rgba depth cube texturing for webgl compatibility. Slow, needs to do some culling.

Added ParticleSystem entity subclass. A ParticleSystem contains a ParticleBuffer and a ParticleMaterial, so you can creating multiple instances of the same ParticleBuffer with different materials. See mojo3d particles test.

Added LinearColors property to UniformBlock and Renderer. Material color properties now always in sRGB space. Changed Color.Gold etc.to compensate (they were linear colors).

Bye!
Mark

7 Comments


Monkey2v1.1.06 now available!

Posted on August 3rd, 2017

Hello everone!

I have just uploaded a prebuilt binary release of Monkey2 V1.1.06 to itch.io here: https://blitzresearch.itch.io/

This is mainly a ‘first pass’ over various problems and issues with the new 3d stuff, which has included a few big, breaking changes (note that mojo3d is still in it’s infancy and there may be more) in addition to various bugfixes and and feature additions.

The breaking changes include…

Mojo3d (and only mojo3d, not monkey.math or std.geom) now uses degrees instead of radians for angular units. This makes it easier to visualize angles when intializing entity rotations and so on ‘by hand’, which is the most common usage of angles when using mojo3d.

Entity properties and methods now default to ‘world space’. For example, Entity.Position now returns an entity’s position in world space, and Entity.LocalPosition must be used to get it’s position in local space. The rationale here is that game logic is primarily concerned with world space coordinates, so these should be easier to use.

I have split RigidBody into StaticBody, DynamicBody and KinematicBody subclasses, all of which extend RigidBody, but each with a simpler interface that features only the relevant methods. I will probably do the same thing to Camera and Light at some point.

As for additions…

I have added a bunch of new properties (eg: Entity.Rx for entity rotation around x axis, Entity.LocalRx etc) and methods (eg: Entity.PointAt) to the Entity class via a set of class extensions (more to come). I have also moved some of the old Entity properties/methods into extensions in order to clean up the core entity source file a bit. This has worked really well, the ‘core’ entity class now only contains a handful of members for dealing with the ‘tricky bits’. This also means many Entity members are now docced in the (new) ‘extensions’ section of the mojo3d docs. I eventually plan to link classes to extensions in the docs, but for now extension docs are separate from class docs.

The physics World class now has RayCast and ConvexSweep methods (still very WIP) that appear to work really well (thanks bullet!). In particular, I have managed to use the convex sweep to emulate Blitz3D style sliding ‘FPS’ collisions, only with capsules instead of spheres. You can in fact pass any ConvexCollider to ConvexSweep (Sphere, Capsule, Cylinder, More?) and it should just work. Being able to use capsules and cylinders means more realistic player shapes of course.

Currently, it works very much the way Blitz3D’s sliding collision worked, and therefore suffers from similar limitations, eg: ‘step size’ limitations, unintentional sliding down slopes etc. But the exciting thing IMO is that it’s written *in* monkey2 so can be tweaked to behave however you want it to! I do intend to improve on it quite soon so that mojo3d will include a useful quake style collision system.

Have a look at modules/mojo3d-physics/test/fpsthing.monkey2 to see b3d style sliding collisions in action, with the original b3d castle model no less! I have tried it with an actual quake level, but I can’t include that in monkey2 due to licensing issues.

I also spent a bit of time on an IAP module for monkey2. Tony from Playniax sort of forced my hand here by actually writing a mobile game. This has been theoretically possible for some time now, but very few people seem to have even tested the mobile stuff, let alone tried to do something serious with it, so it was actually kind of a shock!

The game itself looks really cool, check it out here: https://playniax.itch.io/revengestar

IAP should be ready in a few weeks. It will be a paid module available via itch.io for 29.00.

(And thank you to the user who sent a donation to cover the ios app store fee!).

Anyway, my plans now for the rest of the year are roughly:

* More work on 3d: point light shadows, terrain material, terrain engine, usable animator, physics etc.

* More work on reflection, especially selective reflection. Also, an EditObject for mojox. The combination of the 2 will enable some very cool stuff I think.

* More work on docs, with the goal of eventually publishing a monkey2 ebook.

Ok, not exactly a roadmap but better than nothing?

Bye!
Mark

12 Comments


Monkey2 v1.1.05 now available!

Posted on June 27th, 2017

Hi everyone,

I have just uploaded prebuilt binaries of monkey2-v1.1.05 for windows and macos to itch.io at https://blitzresearch.itch.io/monkey2

This is mainly a ‘pre-3d’ release. It contains several under the hood tweaks to mojo that will be required by the soon to be released mojo3d module(s). I have tried to make these tweaks as unobtrusive as possible, but there’s a chance something may not work as it used to. Please post any problems you may have at monkeycoder.co.nz and bugs to the github repos.

Another semi-major change I made was to remove the reference counted ‘resource’ system. Hopefully no one was actually using this (it was a bit of a mess) but I eventually decided it just wasn’t working. What I ended up doing instead was adding a native ‘OnFinalize’ virtual method to Resource that will be invoked if a resource becomes unreachable by app code AND hasn’t had Discard called on it. Note that finalizers execute in a ‘delicate’ phase of GC collection, so have some severe restrictions on them (which is why I haven’t yet added finalizers to the language, and may never). In particular, finalizer code should NOT make use of any object, array or function pointer fields of ‘Self’ (as they may not even exist any more) and should NOT assign ‘Self’ anywhere (you should go a runtime ewrror if you try). So you’re pretty limited in what you can do inside a finalizer, but it’s enough to clean ups FILEs, GL objects, system memory and other kinds of native resources.

This means it’s not strictly necessary to use Discard to cleanup resources any more, GC will eventually do it for you. GCCollect() is still slow so Discard is recommended in realtime situations, but on the whole it should be possible to write apps without worrying about using Discard at all now. The only thing you might have to be careful about is situations where a variable is keeping a resource alive after you’ve finished with it (say, a value in a Map). As always, test yer stuff for mem leaks!

This release also contains a few other goodies, including:

* Added std.requesters.OpenUrl support for android and ios.

* Added WIP assimp and bullet modules – mainly for future use by mojo3d but IMO its also nice to have the standalone modules.

* Added tinyfiledialog library for linux requesters, except for Proceed which uses kdialog as there’s no equivalent in tfd.

* Added WIP MSVC support for faster compile times on windows. To enable, use ‘MX2_USE_MSVC=1’ in bin/env_windows.txt and rebuild all modules.

Mojo3d is progressing nicely, and a version fit for public consumption should be available to the general public ‘very soon’. There will also be a ‘sneak peek’ release for monkey2 patreon supporters ‘coz they rule and the whole project would not be possible without them.

This first release will not be very usable, but off course ‘usable’ depends on your needs. But in general, there IS still some way to go with mojo3d. More on this later…

Bye!
Mark

9 Comments


Monkey2 3d update!

Posted on May 2nd, 2017

Hi everyone!

There have been some important ‘behind the scenes’ developments recently that have been hard to show via screenshot so apologies for the twitter silence lately, but here’s a hopefully informative blog update instead…

Deferred HDR rendering

Up until this point, all rendering has been performed directly to the app window, in a single pass, and with a single ‘fudged in’ light. This is commonly known as ‘forward rendering’. It’s a pretty simple approach and (can be) very fast.

Another approach is ‘deferred rendering’. With deferred rendering, rendering is instead performed to an offscreen texture, and instead of trying to do everything in one pass, a deferred renderer generally does things in multiple, often simpler passes. Deferred renderers also usually maintain a number of ‘gbuffers’ – special buffers that contain ‘per pixel’ geometric information, such as surface color/metalness/normal etc. This extra information is usually rendered on the first pass, and is used by subsequent passes for lighting, shadows, special effects and so on.

This sounds complex, and it kind of is, but it offers several really nice benefits. First, lights are very fast to render, eg: for pointlights you just render a quad per light and the shader uses the existing gbuffers and depth buffer to performing all lighting calculations. Second, it allows you to write all sorts of cool ‘post processing’ effects that can also use the gbuffers/depth buffer for effects like fog, ‘god rays’, screen space reflections, bloom, depth-of-field and a lot more. This typically involves drawing a fullscreen rect that has access to the ‘current’ color buffer, the gbuffers and the depth buffer. Shader code then has the ability to inspect the normal of a single pixel, or depth value (and therefore view/world etc position) and any material specific info like roughness/glosiness (perhaps for reflection effects?).

But this does come at a cost – if your game only has a few lights and doesn’t do much post processing, forward rendering could well be the better/faster approach (I think it’s generally the recommended approach for VR). Also, mobile is unlikely to have the grunt just yet to make good use of deferred rendering, so forward rendering might make more sense on mobile too. But for desktop developers, deferred rendering is IMO the way to go as it’s more powerful, and if you have lots of lights can be faster. Given most mx2 users are desktop users, it’s what I’m going to start with. Adding a forward renderer in future should not be a big deal but I don’t want to have to maintain 2 renderers right now.

In the process of moving to a deferred renderer, I added some new floating point pixel formats for textures, so the main ‘color buffer’ texture that ultimately contains what you see is now in 16 bit floating point format. This gives you ‘hdr’ (high dynamic range) colors, where the color buffer can actually contain color RGB values >1. You can’t actually show such colors on a normal monitor (this is what the new HDR TVs are all about) but you can smear the over-bright colors around a bit so they brighten up neighboring pixels, making the entire scene appear brighter (or ‘glow’) in the process. This is a post processing effect known as ‘bloom’, and is the only one I currently have going!

New PBR material system.

I also ending up the changing the PBR material system used. Previously, I had been using what’s known as a ‘specular/gloss’ system. This basically means your materials have 2 color textures – diffuse and specular – and a single (1 component) ‘gloss’ texture. Instead, I am now using a slightly different system known as ‘metalness/roughness’ (ultimately, these are both PBR systems, they really just store material parameters differently). The metalness/roughness approach actually uses fewer parameters than specular/gloss as it combines diffuse/specular texture into a single ‘color’ texture, only with an additional (1 component) ‘metalness’ texture. Metalness effectively says how much of color is diffuse and how much is specular, eg: if metalness=1, then color is really just specular. If metalness=0, then color is really diffuse. This system also has a ‘gloss’ texture, only it’s stored as 1.0-gloss and called ‘roughness’ instead. Not sure why the difference here, but it seems to have become a standard of sorts so I’ll go with it.

There are 2 advantages to the metalness/roughness system. First, it is theoretically harder/impossible to accidentally produce ‘unrealistic’ colors that can’t occur in the real world, because basically you’ve got less parameters to tweak, ie: color, metalness and roughness, and pretty much all combinations of these are ‘valid’ and should produce a realistic color. Not so if you have sepearate diffue/specular textures where (apparently), some combos of diffuse/specular just don’t exist IRL. Second, it simplifies the deferred gbuffer system as there is like to store in the gbuffers, ie: instead of having to store diffuse/specular (6 floats), you only have to store color/metalness (4 floats) which can be stored in a single RGBA texture (ie:RGB=color, A=metalness). so all-in-all, the metalness/roughness system only require 2 offscreen textures/gbuffers for deferred rendering: A color buffer (RGB=color, A=metalness) and a normal buffer (RGB=normal, A=roughness).

So all in all, the ‘mostly finalized’ default PbrMaterial class allows you to play with the following material properties:

ColorTexture:Texture – RGB color texture.
ColorFactor:Color – color factor. ColorTexture is multiplied by this. Ditto for all factors below.
AmbientTexture:Texture – RGB ambient texture – really ‘ambient occlusion’. Allows you to mask out ambient light.
AmbientFactor:Color – ambient factor.
EmissiveTexture:Texture – RGB emissive texture. Doesn’t actually add a light source (could it?) but pixels will be at least this bright!
EmissiveFactor:Color – emissive factor.
MetalnessTexture:Texture – single component metalness texture.
MetalnessFactor:Float – metalness factor.
RoughnessTexture:Texture – single component roughness texture
RoughnessFactor:Float – roughness factor.
NormalTexture:Texture – normal texture.

You can also write your own material classes that take different parameters to this, but if you’re using the deferred renderer and want lighting, your material shader will have to somehow produce the following PBR params:

These are then passed to an internal main0() function. For example, the default fragment shader for the built in PbrMaterial looks a bit like this:

main0() is actually a function #included by the shader that computes/writes the correct values out to the color buffer and gbuffers.

So you could write a super simple shader that just did this:

Or, you could include the PBR params in less textures for a faster shader, or computer PBR params etc. etc.

Anyway, going WAY off topic here!

* Release date

I *hope* to have a v0.1 released this month. It will be far from finished, but this *is* an open source project and I feel bad about not releasing stuff already! But as always, my concern has been to have things ‘far enough’ along so I don’t have to massively change things *after* people are doing things with it. To this end, I will likely be releasing less than I would have liked (or we’ll be here forever!) but it should be more finished.

Bye!
Mark

12 Comments


Minor 3d update!

Posted on March 1st, 2017

Hi,

Ok, a little update on monkey2’s 3d progress!

I’ve only really spent a couple of weeks on it full time so far (and have had some kind of nasty flu-bug thing over the last few days so have been taking it easy this week) so there’s not all that much done, but in a nutshell:

* There’s a simple model/surface/material system going. Surfaces have a material, vertices and indices and are attached to models, very similar to b3d’s mesh/surface/brush system. Also CreateSphere and CreateBox for testing. All this may or may not yet be thrown away. Also, simple skybox support.

* There’s a pretty cool ‘PBR’ shader going. PBR stands for ‘physically based rendering’, which, as far as I can work out, is really just a slightly different approach to how diffuse/specular/normalmap etc are parameterized, and a few additions/tweaks to the good old ‘blinn-phong’ shader code to make it more realistic. Crash course in PBR is here:

http://renderwonk.com/publications/s2010-shading-course/hoffman/s2010_physically_based_shading_hoffman_b_notes.pdf

I don’t understand all of that(!) but I had a play around with it a few years ago (in c++) and worked out enough to get it ‘sort of’ going so had a bit of a head start. (It was fun getting back into the ‘shader groove’ too!) Also, I recently discovered this which clarified a few things that had me guessing first time around…

https://seblagarde.wordpress.com/2011/08/17/hello-world/

I’m now pretty happy with the results although I’m sure there will be heaps of tweaks to come – esp. from people who *really* understand all the math!

In practice, PBR affects the parameters/colors/textures etc used by materials, but not a lot really. The way I’ve done it, materials have diffuse/specular/gloss/normal colors/maps so it’s almost the same as b3d brushes (although the ‘meaning’ of some parameters differs a little). There are other approaches too such as diffuse/metallic/roughness/normal, which can easily be provided for via a different shader if necessary – and may be better, dunno yet!

What’s the point of this PBR stuff anyway? Well, the biggy is it just plain looks better because the underlying algorithms are better, but IMO PBR is also easier to use. This page has some hints on how PBR colors work and basically they’re just hardcoded diffuse intensities for non-metalic (or ‘dielectric’) materials (with about .04 specular – everything has specular!), and hardcoded specular colors for metalic materals (with black diffuse).

https://seblagarde.wordpress.com/2011/08/17/feeding-a-physical-based-lighting-mode/

PBR can (theoretically) achieve these sorts of results:

https://www.google.co.nz/search?q=physically+based+rendering+examples&tbm=isch

Note that mx23d may not be capable of this level of quality just yet, but the ‘pieces’ are mostly all there I think.

I realize that not all mx23d users are gonna be doing stuff to this level (or are even particularly interested in ‘realism’ in the first place), but even for casual users, I think PBR will be a big win regardless of what you’re doing.

* There’s also a simple gltf loader going. No anim stuff yet, and bizarrely glTFv1.0 has no built-in (ie: non-extension) support for normal maps (glTFv2.0 does, apparently coming very soon) although I’m sure we can kludge something in here if necessary.

Next up, I am going to start work on the entity system, including basic bullet physics integration and animation. This is a pretty exciting time as it should mean I have some stuff flying around relatively soon!

Everything is still gles2 and I haven’t yet decided whether to go to gles3. Nothing I’ve done so far *needs* gles3 so gles2 is still an option at this point, so I don’t see the need to make any firm decisions about this right now. There are some good points on the pros/cons of using gles3 or gles2 here:

http://monkey2.monkey-x.com/forums/topic/gles3-0-vs-gles2-0-for-mx23d/

Feel free to add your own thoughts/comment(s) but please be nice!

Also, please follow me on twitter @blitzmunter for occasional screenshots of my 3d progress.

Finally, there is no source code yet (and wont be until I have a clearer picture of how it’s all gonna end up) but I’d like to remind everyone that the monkey2 3d module *will* be open source! There seems to be a bit of confusion about this going by some posts on the forums.

Bye,
Mark

9 Comments


Happy new year everyone!

Posted on January 3rd, 2017

Happy 2017 everyone!

Sorry I’ve been a bit quiet lately, but I had a bunch of real life stuff to deal with there for a bit, including both a broken down car AND motorbike so I’ve been stranded at home with just a windows laptop since Christmas. I should (surely?) be sorted by the end of the week.

Part of the problem here is that I moved into an office at the start of December. This is something I was reluctant to do for a long time due to the expense, but ultimately decided it was probably worth it for the sake of my sanity – I was getting just a bit *too* comfortable with not talking to ‘real’ people for weeks at a time! Besides, not moving would have allowed me to stretch things out a bit longer, but not long enough to make a real difference – what I really need is MORE INCOME!

The office itself is very cool though. It was very cheap thanks to a friend’s help (thanks Joe from The Arcade!) and best of all is in the same location as a bunch of talented young game developers (none of whom have ever heard of blitz, let alone monkey), so it’s got a great gaming-ish vibe to it!

The office moved disrupted December a bit but I did manage to clean up some of the mobile stuff – in particular, you can now build all required architectures (not just arm) for android/ios. Still in development and not too elegant yet as the archs to build are driven by env_platform.txt, but it does mean mobile has reached a kind of ‘minimum viable product’ IMO. I also put in a bit more time on cxxtomx2 for converting c++ headers to mx2 modules. I did in fact have it successfully converting the ‘bullet’ lib, even adding ‘glue’ code for mx2 unfriendly stuff like ‘Blah &’ params, ‘const Blah &’ returns etc. Not finished yet, but looking very finishable now.

And as for 2017:

as mentioned above, priority one is definitely to MAKE MORE MONEY. My current plans here are to:

This does mean I probably wont be spending as much time directly on mx2 as I have been in the past, but I don’t think there’ll be a whole lot of difference.

Finally, one last thing I’ll be trying: I will soon being adding a ‘3d module’ goal to patreon. The idea here is that if patreon hits 1000/month, I’ll be back full-time on mx2 and concentrating on a 3d module for mx2. It’s a fair bit of money, but it’s about the amount I think would allow me to forget all the money crap and just get stuck in and do it! And besides, his figure may change over time depending on how other stuff goes.

I have in fact been playing around with a hypothetical 3d module for mx2 and like what I’ve come up with so far and would love to take it further. I wont go into too much detail right now, except to say I’m aiming for something blitz3d-ish that’s fun to just tool around with, in an mx2 style (eg: making use of properties where sensible as opposed global functions), with built-in support for bullet physics (it’s cool and free and open source and I can almost build it!), and with a more modern approach to rendering (eg: shaders instead of ‘multitexturing’; PBR shaders etc).

The truth is that something 3d will probably eventually happen some day, it’s just a matter of when. I would love to be working on it right now, but it’s a relatively big project and if I were to do that while ignoring the current income situation the trip would be over pretty quickly.

Bye!
Mark

21 Comments


Website update and new binary releases!

Posted on November 11th, 2016

Hi,

As you may have noticed I’ve given the website a bit of a tidy-up. I definitely struggle with web design stuff so it’s not nearly as good as I’d like it be, but still, I think it’s an improvment! If nothing else, it makes for a nice change.

There are also some prebuilt v1.1.01 releases now available from the files section to celebrate, although for some reason, the windows version has turned into a 250M beast! That’s getting too big for my liking so I’ll probably look into getting Ted2 to build modules/documentation the first time it’s run instead of shipping them with releases in future, but for now you’re stuck with a rather large download sorry.

I am also pleased to announce that the new releases also include ‘Ted2go’ as the default IDE. Ted2go is a Ted2 fork by Evgeniy Goroshkin (otherwise known as nerobot on the forums) who has done near miraculous stuff with Ted2 considering the rudimentary state of  things!

Ted2 will remain in the monkey2 repos for a while yet, but Ted2go has also been added as a ‘submodule’. For people who like to build from source, ‘rebuildall’ still builds old Ted2 but you can now also use ‘rebuildall2go’ to build Ted2go instead. Unfortunately, dealing with submodules is a little complicated (for one thing, they’re not included in the zips you download from github) so I haven’t made ted2go the default for ‘rebuildall’ just yet.

Finally, Tony of playniax has submitted his Pyro framework module for publishing – it’s gots lots of useful stuff in it and it’s up now at the ‘modules’ page. Definitely lookng forward to whole enchilada Pyro-wise!

Whew! I’ll probably write more tomorrow, but it’s been a LONG couple of days and I’m a bit shagged so for now…

Bye!
Mark

21 Comments


Reflection is in!

Posted on October 23rd, 2016

Hi,

Ok, it’s still a bit WIP, but the core of the reflection system is now in!

It introduces 3 new ‘builtin’ types:

I’ve added a simple reflecttest banana too.

Variants

The ‘Variant’ type is a primitive type that can be used to ‘box’ values of any type.

The easiest way to create a variant is to cast a value to Variant (much like casting an Int to String etc), eg:

An uninitialized variant will contain a ‘null’ value (of type Void) until you assign something to it:

A variant is ‘true’ if it contains any value with a non-void type (including a bool false value!) and ‘false’ if it is uninitialized and has no (void) type.

Any type of value can be implicitly converted to a variant, so you can easily pass anything to a function with variant parameters:

To retrieve the value contained in a variant, you must explicitly cast the variant to the desired type:

Note that the cast must specify the exact type of the value already contained in the variant, or a runtime error will occur:

The one exception to this is if the Variant contains a class object, in which case you can cast the variant to any valid base class of the object.

Typeof and TypeInfo

The Typeof operator returns a TypeInfo object that contains various properties and methods for inspecting types at runtime. There are 2 ways to use Typeof:

The use of seperate () and <> delimeters is to prevent the parser getting confused by complex expressions/types.

TypeInfo objects have a To:String operator (mainly for debugging) so can be printed directly:

Typeof returns the ‘static’ type of a class object. To get the actual instance type, use the Object.InstanceType property:

You can retrieve the type of the value contained in a variant using the Variant.Type property:

TypeInfo also includes functions for inspecting all user defined types:

Returns the TypeInfo for a named type. A named type is a namespace or class declared by your app – it does not include primitive types, pointer types, array types etc. Class names must be prefixed by the namespace they are declared in.

To get an array of ALL named types:

DeclInfo objects

TypeInfo objects for namespaces and classes also contain a set of DeclInfo objects. A DeclInfo represents the member declarations inside of a namespace or class. Currently, only global, field, method and function members are supported. DeclInfo objects also have a To:String operator to help with debugging.

You can inspect the member decls of a type using the TypeInfo.GetDecls method:

You can retrieve a single unique member using TypeInfo.GetDecl:

There may be several decls with the same name due to method and function overloading, in which case the simple GetDecl above will fail and return null. In this case, you either need to inspect each decl individually to find the one you want, or you can pass an additional TypeInfo parameter to GetDecl:

This will return the default constructor for MyClass, assuming there is one.

Getting and setting variables

Member decls that represent variables (ie: fields and globals) can be read and written using the DeclInfo.Get and Decl.Info.Set methods:

The first parameter of Set and Get is an object instance, which must be non-null for getting and setting fields.

The second parameter of Set is a variant, and is the value to assign to the variable. The type of the value contained in the variant must match the variable type exactly, or a runtime error will occur.

Note that since any value can be cast to a variant, we can just provide the literal value ’10’ for Set and it will be implictly converted to a variant for us. On the other hand, we must explicitly cast the result of Get() from a variant back to the type of value we want.

Invoking methods and functions

To invoke methods and functions, use the DeclInfo.Invoke method:

The first parameter of Invoke is an object instance, which must be non-null for invoking methods.

The second parameter of Invoke is an array of variants that represents the parameters for the call. The types of these parameters must match the parameter types of the actual method or function exactly, or a runtime error will occur.

Limitations

Currently, typeinfo is only generated for non-generic, non-extension, non-extern 100% pure monkey2 globals, fields, function, methods, classes and namespaces. You can still use other types (structs etc) with variants etc, but you wont be able to inspect their members.

Typeinfo may be stripped out by the linker. I’ve added a little hack to mojo to keep module typeinfo ‘alive’, but there is still work to do here. If you find the linker stripping out typeinfo, you can prevent it doing so for now by adding a ‘Typeof’ to Main() referencing the type you want to keep alive. Or, you can set MX2_WHOLE_ARCHIVE in bin/env_blah.txt to ‘1’ to force the linker to include ALL code, but this will of course produce larger executables.

2 Comments


C2mx2 lives!

Posted on October 17th, 2016

Hi,

I finally got around to taking a good look at using ‘libclang’ to help with converting native ‘c’ header files to monkey2 externs, and it turned out to be way easier than I thought!

Libclang is a helper lib for the clang c/c++ compilers that provides a simple interface for walking the nodes of an app’s syntax tree. The idea with a c->mx2 convertor is simply to rewrite these nodes into equivalent mx2 ‘extern’ decls.

I did try this a while back but just couldn’t llvm/libclang building from source. This time around, I installed the binaries in the vague hope that libclang might be included – and it is! I don’t think it used to be…but anyway, this made things a lot easier.

And the actual libclang library itself is very nice and clean and easy to use – well, it would be if it weren’t for the fact it’s dealing with c/c++ which has a ton of weird wrinkles.

I’ve managed to use it to put together a little ‘c2mx2’ app (now added to ‘src’ dir) that seems to work pretty well. I’ve used it to convert the chipmunk 2d physics lib API (now a new module), SQLite (also a new module) and a few others. I’ll probably redo the SDL2  API and others too once it’s a bit more tested. It can also convert the ‘libclang’ API itself.

Note that libraries converted this way are still really c/c++ libraries – you still have to deal with pointers and a certain degree of krust. But it makes an excellent starting point for monkey2-ification without having to get your hands too dirty with c/c++ etc. And in the case of nice APIs like chipmunk, it’s still very usable ‘as is’.

There are also things you need to be careful of – in particular, c2mx2 will generate function pointers but these are ‘plain’ C function pointers and should only be used with global functions, not methods or lambdas, otherwise you’ll probably get a bunch of ugly c++ errors when building. More work required here…

Another thing I’ve done in the process (which may affect some native code) is streamline the CString system. In terms of native string types, there is now only CString. Utf8String has gone. Ditto String.FromUtf8String/ToUtf8String have gone.

String.FromCString() now assumes utf8 (falling back to ‘byte per char’ if utf8 is invalid) and String.ToCString() now emits utf8. This just simplifies so many things, esp. with native code where it’s not always clear what encoding const char *’s are, and utf8 is so prevalent now that I think it’s worth the loss of some dubious/confusing extra functionality (that can go elsewhere if necessary) to simplify things significantly.

I will be (re)adding String.FromChars() with arrays/pointers of ints for situations where you want to create strings directly from chars, but as far as CString goes, it’s utf8 all the way.

Finally, you can now return CStrings from extern functions and store them in extern vars/fields, so no more “String.FromCString( SDL_GetError() )” style – SDL_GetError can now return a CString which will be automatically converted to a monkey2 String.

Bye!
Mark

2 Comments


Minor update: Ted2/Mojox improvements

Posted on October 13th, 2016

I spent some time recently working on adding some features to ted2 and mojox, including:

Also. the canvas font system now supports rendering ALL characters in a font. Well, up to 65536 characters anway.

It used to only handle chars from 32-127, and I was a bit worried about how to add support more as the default DejaVuSans font alone has something like 4000 characters in it, and I didn’t want to generate ALL of them everytime as 99% of apps will only use a fraction of them.

I ended up going with a ‘page’ system, where each page contains up to 256 chars and pages are created ‘on demand’. This probaly isn’t ideal as it will produce 256 textures worse case, one for each page. A better approach is probably to use a dynamic atlas, but the current system will do for now.

This, combined with a tweak to mx2cc, should mean it’s now possible to display any char a font can handle. Check out the new fonttest banana that allows you to view all font pages.

Bye!
Mark

2 Comments


Semi-major mojo update

Posted on September 29th, 2016

Hi!

Ok, I’ve just committed/pushed some pretty major improvements/changes to mojo.

First up, mojo can now do 2d normal-map style lighting! It’s also very easy to use:

There are few restrictions on what you can do in lighting mode: you can’t modify the viewport, scissor or ambientlight properties, or Clear() the render target – this stuff must all be setup before you enter lighting mode – but apart from that you can use the full canvas API.

There are 2 simple lighting demos in the bananas folder: simplelight and lightsparks.

Secondly (and possibly breakingly) I’ve modified the texture filtering system. TextureFlags no longer contains ‘Filter’ or ‘Mipmap’ enums (currently, it only contains ‘Dynamic’). I considered leaving these in as NOPs for backward compatibility but I think that’d just confuse things. Instead, texture filtering is selected on the fly via Canvas.TextureFilter (similar to Canvas.DisableTextureFiltering – now deprecated), or you can force an image to using a particular filter mode via Image.TextureFilter. This is probably the change that will the affect the most projects (I had to tweak a few bananas) but it should be easy enough to deal with – basically, forget about texture flags unless you’re creating a dynamic image.

Thirdly, I’ve added some renderstate stuff to the Image class including BlendMode, TextureFilter and Color properties. BlendMode and TextureFilter default to ‘null’, which basically means ‘use the canvas property’, ie: old mojo behavior. But if you set these to non-null, they will override the canvas properties. Color defaults to white, and is modulated by canvas color.

Finally, the entire shader system has been completely rewritten. I severely doubt anyone was using the dodgy old undocced (and @hidden) system, but if you were, sorry, but it had to be done. The new system isn’t quite ready for public use yet, but it should be much easier for peple to play with shaders once it is.

Whew! I’ll probably do a follow up post tomorrow going into a bit more detail on how this stuff works, but that’ll have to do for now as I’m stuffed!

Bye,
Mark

13 Comments


New stuff now up!

Posted on September 5th, 2016

Hi,

I’ve had a few internet problems over the last week, so I didn’t end up doing anything (directly) on the ios target as I couldn’t face even attempting it without stackoverflow support – nevermind xcode docs! Internet is now back at full speed though so it’s back into ios tomorrow.

Without much in the way of internet, I instead got to work on a bunch of stuff that *I* have complete control over, which is always the most fun! I’ve just pushed the results to github, and there is some cool stuff in there IMO. But first…

Breaking Changes Alert!

The __TARGET__ preprocessor var now works differently with desktop targets. It is now set to one of the following values: “windows”, “macos”, “linux”, “emscripten”, “android” or “ios” – ie: the *precise* target. You can still pass ‘-target=desktop’ to mx2cc (and it’s still called “desktop” in ted2) but this now gets converted to one of “windows”, “macos” or “linux” before anything’s compiled.

I have also added the following boolean preprocessor vars: __DESKTOP_TARGET__ (true if __TARGET__ is “windows”, “macos” or “linux”), __MOBILE_TARGET__ (true if __TARGET__ is “android” or “ios”) and __WEB_TARGET__ (true if __TARGET__ is “emscripten” or, hopefully one day, “webasm”!).

Changing __TARGET__ like this is a semi-major change (and apologies if I’ve borked your code, but it should be easy to fix) but the way it used to work forced you to use __HOSTOS__ everywhere to find the real target, which is not correct if we eventually want to be able to support cross-compilation in future.

I considering adding a new var, but it’s already confusing enough and I think this simplifies things down to a very clear set of vars. I’ve also added a little fudge to the parser, so if you have code that compares __TARGET__ with “desktop” you should get an error.

__HOSTOS__ is still there, but don’t use it. It will probably change to plain __HOST__ with the same potential values as __TARGET__.

Next up, I have moved the Timer and Process classes from mojo to std. If you are ‘Using std…’ and ‘Using mojo…’ this should have no effect on your code, but if not you will need to make changes if you use these.

I did this mainly because I never really wanted these to be in mojo in the first place, and other stuff is starting to ‘leak’ into mojo too! The main problem here is that mojo is currently the only bit of mx2 that manages an event queue, and both Timer and Process depend(ed) on being able to post messages (via internal c++ code in mojo) from a separate thread to the main ‘app’ thread.

Things started getting ugly here when I wanted to add Fiber.Sleep, which needed to do the same thing, but I didn’t really want to have to move Fiber to mojo too (and I definitely don’t want to make std dependant on mojo) – it’s getting crowded in there! And I can see a lot of future cases like this too, eg: synchronous TcpStreams that politely block the current fiber etc.

So I ended up adding a simple ‘pseudo’ queue system to std (in std.async – it’s mostly extern code) that can be used to post messages from threads. However, it currently does nothing on its own – it depends on mojo to ‘hook’ it to provide a real event queue. But it should be pretty easy for std.async to provide a real/default queue, which means you’d be able to use all the async stuff from plain console apps…one day.

The upshot of all this is that Timer and Process are now in std, but you still need to be building a mojo app for them to work!

Class Fiber now has a Sleep( seconds:Double) function to pause the current fiber, and I added a Now:Double() to std.time (which uses an arbitrary time base). I’ve gone with doubles here because it makes a LOT of timing logic much easier. For example, the old Timer code used to do some nasty bresenham stuff to produce correct millisecond delays (eg: 16,17,17,16,17,17…for 16.66666 period). This is a whole lot simpler with doubles (although, yes, the result is not 10000% accurate) and means you can now create a Timer with hertz<1. Woohoo!

Fiber.Sleep has also meant I’ve been able to implement Timer in 100% mx2, which is nice.

New Language Features!

These are are currently still very WIP, but I’ve added preliminary ‘Operator To’ and class extension support to mx2cc.

Operator To allows you to add an implicit conversion operator to a class or struct (not interface). The basic idea is:

While I’m not generally a huge fan of implicit conversion operators like this, there are some situations where they can be very useful.

For example, up until now there’s been no easy way to convert Vec2<Int> to Vec2<Float>, and I ended up adding a few special case functions to let you do things like multiplying a Vec2<Int> by an AffineMat3<Float> etc. But by adding this to Vec2<T>…

…you can now pass a Vec2 of any type to something that wants a Vec2 of any other type.

You can’t ‘chain’ operator To yet (I think – and maybe never) and I’m not totally sold on the name yet (IMO it looks very clear/obvious, but does ‘borrow’ a keyword and I haven’t thought through parser issues yet) but I like it so far!

There is no ‘Object.ToString()’ yet (this is a bit complex, as I don’t want to force To:String() to be virtual for all objects) but for debugging this would of course be very nice.

I also do not plan to also allow ‘constructor based’ conversion. IMO, this is where things in c++ start to get really wacky (which conversion is being used?!? I still have to google it…) (also, I don’t really like it when the compiler calls ‘New’ for you…) and with the next feature I don’t think it’s really necessary…

Class extensions allow you to add non-virtual methods (and other members except fields) to an existing class, ie:

You can’t yet add extensions to built-in types, and there are several issues with ‘search order’, but as far as I can tell this is working pretty well, even with generic types, eg:

Where things get really interesting is combining both of these. For example, I was working on some c++ a while back that used both the ‘glm’ math library, and the ‘bullet’ physics library. Both these libs had their own ‘vec3’ etc types, but of course they were incompatible.

As far as I know, there is no way to add an implicit conversion operator ‘outside’ of a type in c++, so unless you were willing to tweak either the bullet or glm source code, you were stuck with having to wrap a bunch of stuff in to_glm() or to_bt() functions.

But with extensons and operator to, you can do it this way:

Which, according to my calculations, should allow you to freely pass to glm_vec3’s to bullet and vice versa!

New Build Stuff!

Building an app in Ted2 now builds the app in a separate ‘blah.products/TARGET’ dir, eg: if you build ‘commanche.monkey2’ for the desktop target on windows, you end up with the exe/assets/dlls etc in a “commanche.products/Windows/” dir. By default, mx2cc on the command line works the same as it used to, ie: the product is placed in the ‘.buildvblah’ folder. There is a new mx2cc ‘-product’ option that ted2 uses, but leave it alone! Also note that none of this affects makemods, only makeapp.

Yes, this is another ‘junk’ dir, but I think it’s the best solution (until I can think of a better one) as the build products now get their own ‘clean’ dir that contains only what you need to run (or build in android studio/xcode etc) the app. The ‘.buildv’ dirs are temporary and contain all sorts of junk, so it all ends up a bit of a mess cramming more in there.

And Ted2 now offers simple per-target ‘product settings’ options (pretty much only App Name right now) and these are stored in a blah.products/products.json file that you generally don’t want to accidentally trash, as it’s effectively the ‘project’ file for all targets.

This is all very tentative stuff right now, but I like where it’s heading.

Bye!
Mark

9 Comments


Android support phase1 complete!

Posted on August 24th, 2016

Hi,

I have just committed a bunch of stuff that includes preliminary support for android in it – but don’t get too excited! It’s in a pretty rough state right now! There’s an ‘android’ dir in the root monkey dir that explains the various spells and incantations that must be performed to produce an mx2 android app.

Building an android app is not fun at this point by any means (you need to copy output files to a ‘real’ android project), but it’s far enough along that curious users should be able to get it working and give some feedback.

It’s also highly unstable, but the guts of it are in including touch input and audio (the scary ones). Touch is only currently generating fake mouse events – which required low level tweaks to mojo to get all the mouse down/enter/leave/up event stuff working sensibly so I hope I haven’t broken anything there!

What I’ve ended up deciding to do is leave the ‘project’ stuff OUT of mx2cc altogether, so this phase is complete in as far as that’s pretty much all I need/want mx2cc to do. So mx2cc’s job will be to just generate .exe/.so files and assets folders, without caring what’s going to be done with them. These currently end up in hard coded locations (ie: inside the .buildv dir) but there’ll probably be mx2cc args to allow you to change this.

I am now planning to implement the project side of things as ted2 ‘plugins’ (now that there’s the beginnings of a plugin system in Ted2). I’m not sure exactly how just yet, but it should along similar lines to my earlier post on ‘project templates’ etc. And with mojox, there’s a reasonably decent gui system in place to handle the front end stuff.

But before that, I’ve decided to forge ahead with adding ‘plain’ iOS support to mx2cc so I know exactly what the project stuff has to deal with. All going well, the end result should be much like android, ie: it produces an .so file and an assets folder and that’s it. The only tricky bit I can think of is possibly dealing with the different architectures (although I think there are only 2?). I managed to side step that with android by using the ndk-build system which builds arm, x86, mips etc. for you in one hit (currently only armv7a, but it’s easily configurable). I don’t think there’s anything like that on mac(?) but it shouldn’t be too hard to deal with.

Bye!
Mark

8 Comments


New Stuff Up!

Posted on August 15th, 2016

Hi,

Ok, I’ve just pushed a bunch of new stuff to the github repos, some of which may not be 100% stable.

However, you should now be able to use the new module manager in ted2 to install the solitary module currently available online, skidracer’s portmidi module. You can also try uploading modules if you want – see link in help menu.

A brief summary of changes:

So, with that lot out of the way, it’s time to get stuck back into more targets! This will initally mean a big cleanup of the fast-becoming-a-monster builder.monkey2 file in mx2cc (which will likely inspire cleanups in other areas) but the overall plan is for the next majorish release to have android support (hope I don’t regret saying that!).

Bye,
Mark

6 Comments


Sidetracked!

Posted on August 5th, 2016

Hi,

Ok, after an initally promising start with the android target (I can compile/build/link stuff now…but it’s all very hacky) I decided to suspend work on that for a while in order to finish up the module manager system, which is IMO only really half-finished right now – pretty much like every other module manager I’ve ever attempted.

Android is still happening, but it will be a Big Job and right now I think getting a useful module manager system togther is more important, as people are starting to write some really cool stuff in mx2.

The current module upload system works fine IMO, but without some way of being able to update/install modules ‘in’ Ted2 as easily as possible, it’s all kind of redundant – users may as well just download the zip from the project repos if they are required to unzip/install/updatemods etc manually.

So I set about adding a module manager dialog to ted2, which really needs some come of TableView – something mojox (the micro-gui used by ted2) didn’t have. And things kind of snowballed from there – since I was tweaking mojox I decided I might as well clean it up a bit and turn it into a standalone module, given that a few people have shown some interest in actually using it. And I may have got a bit carried with that…

I also ended up making some tweaks to Ted2 using some of the new mojox features as I added them, eg: draggable/closable tabs, right-click popup menu in the file tree to create/delete files/dirs (miss that so much from ted1!), and a tidy up of the secret-no-longer helptree so it works like the online searchable treeview. F1 now also attempts to bring up some meaningful help for the current keyword – nothing fancy, basically just an index search.

I also moved the html help view to the same pane as the console (which is now tabbed) which I thinks works pretty well. You have to drag the splitter up to see the whole page, but that can go on a hot key and it’s definitely nice having docs/code visible at the same time some. You frequently only want to see a declaration anyway, and having that pop up in a little window at the bottom of the page works well I think.

Anyway, the actual module manager is now going quite nicely I think. You can select modules to update/install from a module manager dialog in ted2, and the download/unzip/copy/updatemods/updatedocs stuff is all done for you. It also makes a backup of the modules you are modifying before making any changes in case something goes wrong (mostly I’m thinking updatemods here…).

It’s pretty simple – no update depedancy checking yet, and it’d be nice to be able to rollback multiple times not just once, but IMO it’s a good start.

I hope to commit this work early next week. I did plan on commiting more as I went, but given some of changes I was making, getting mojo+mojox+ted2 working together sweetly has been a bit of a volatile process and I didn’t want to commit too much stuff that could well disappear tomorrow or break something I haven’t tested with. Perhaps I was overly cautious here? Or perhaps I should have done all this work on a branch…

Bye!
Mark

modman

9 Comments


Latest developments

Posted on July 16th, 2016

Hi,

Here’s a quick update on some things I have been working on since the v1.0.0 release:

New fiber system.

The fiber system has always been a bit ropey – it simulates fibers with OS threads, using semaphores to sync them. This is robust, but has considerable overhead and I had always planned to change to ‘real’ fibers at some point – ie: when fibers are switched, all that needs to be done is the CPU registers (including stack pointer) have to be saved/restored to/from a chunk of memory somewhere. The overhead for this shouldn’t really be much more than for a plain function call.

I wasn’t expecting to have to tackle this for a while, but an interesting bug popped up that pushed it to the front the queue – when you tried to close an ‘image document’ in Ted2, it would abruptly crash.

This was happening because the ‘close document’ code in Ted2 actually runs on its own fiber and, being well behaved, attempts to discard the image. However, discarding an image ultimately results in a call to glDeleteBuffers – but GL context’s are bound to the current thread so, since a fiber is/was really a thread, there was no context bound when glDeleteBuffers was called and kaboom! This could have been worked around, but what it really meant that, with the old fiber system, NO rendering code could be called on a fiber.

Fortunately, there is an excellent boost ‘context’ library that provides support for fiber-like code. I eventually got this building on Windows but of course, being a boost/c++ library, it was pretty huge. But there was an interesting ‘asm’ directory in there that looked like it just might contain the *exact* asm code need to perform fiber switches. So I added asm support to mx2cc (ie: you can now #import .s and .asm files), played around with things for a couple of hours and it worked!

The new fiber system is therefore in and should be *much* faster. And as far as the OS is concerned, there’s only one thread so you can render (and do other stuff like use system requesters, which you also couldn’t before) on fibers no problem.

It appears to have the asm code for a ton of target ABIs too (and it’s likely better written than I could have managed on  my own) including android and ios. But alas, emscripten is likely to remain fiber-less for the foreseeable future.

OpenAL Audio

Money2 v.1.0.0 used SDL_Mixer for audio, but it quickly became apparent that was pretty limited and wasn’t going to cut it.

After looking at a few of the options, I decided to go with OpenAL (again). There are number of reasons for this: I’m pretty familiar with it, it’s relatively high level and easy to use, and it runs everywhere I currently care about (desktop, emscripten, android, ios). For Windows, I am using openl-soft, an open source ‘fork’ (I think) of the original OpenAL. It’s LGPL licensed, which means you can’t static link with it and have to ship an openal.dll with you app (or depend on the user having it installed) but I don’t consider this a biggy.

So the mojo.audio code has now been upgraded to openal so the Audio, Sound and Channel classes should now be fully functional – although currently only WAV and OGG files are supported.

I also had a play around with ‘dynamic streaming’ of audio in openal, something skidracer already had working in SDL – see the vsynth banana.

It’s currently highly experimental so the docs are @hidden, but the basic idea is you start a fiber and queue chunks of raw audio data to a channel on that fiber. But before you queue a chunk, you generally called WaitQueued() to ‘sync’ with the channel. Here’s a quick pseudo example:

There is, however, one major drawback to this: The system is effectively polled. I start a 20hz timer in the background to check when chunks become unqueued and, of course, 20hz ain’t very fast so it forces a certain amount of latency.

And there are OS events that can completely stop mx2 executing altogether – in particular, dragging a window around by the title bar seems to just halt EVERYTHING (not so window resizes).

I can up the timer rate, but probably not much past 60hz as once you hit the framerate (if SwapInterval=1), the app ends up being blocked by ‘SwapBuffers’. I have a slightly crazy idea here for moving SwapBuffers to a thread which I want to play with in future, but for now SwapBuffers sort of introduces a ‘minimal polling rate’.

Ideally, this stuff should be done with REAL threads (although of course that introduces its own problems) but even then, openal isn’t really designed for this as it has no mechanism to signal a semaphore or something when a buffer is unqueued. But with threads, you would at least be able to poll at any rate you wanted.

Native GLES20 rendering

This was something I kind of decided to have a go at on a whim, mainly because a number of people asked about it.

There is now a ‘GLWindow’ class (that extends Window) that can be used to render native GLES20. To do this, just extend GLWindow instead of Window and override the GLWindow.OnRenderGL() method.

You can also combine GLES20 and mojo rendering, ie: you can still override OnRender( canvas:Canvas ), do some mojo rendering, call Super.OnRender (which will ultimately call OnRenderGL), do some more mojo rendering etc.

Some caveats though:

What’s next?

Android support is my current focus, and after a bit of experimentation I don’t think it’ll be too hard.

I had a play around with the android ndk a few days ago, and it turns out there’s a ‘tool-chain’ creator in there that generates what is effectively a ‘devtools’ style folder that contains the correct gcc-ish compilers for building native code on a particular host so I already have the basic ‘tools’ problem solved.

The ‘big picture’ idea here (and with ios) is to build an entire android app into a ‘dll’ instead of an ‘exe’, and simply copy it into a ‘working’ android studio project.

To deal with things like manifest settings, info.plist config etc, I think I’ve come up with a pretty good compromise between monkey1’s unmaintanable ‘hack everything to death’ approach, and having to use android studio etc to configure *everything* on the target project side. It would basically work like this:

This is IMO probably the sanest approach to take and what I’m aiming for right now re: ‘target projects’.

Money!

As some of you have no doubt worked out, mx2 is not generating a whole bunch of income for me at the moment.

Fotunately, it’s not as bad as it probably looks. There have been several very generous one off donations, and some people have chosen to donate regularly via paypal instead of patreon, so the pateron page at least doesn’t tell the whole story. If it did, things would be pretty grim right now!

I think ultimately, if it ends up working (ie: I can keep doing it fulltime), this is likely to be how it will work – ie: there wont be a single source of income like patreon, but multiple sources of income combining to provide me with something approaching a livable wage.

But things aren’t quite there yet, so if you have been using or are interested in mx2 but haven’t dontated yet, please consider doing so.

And spread the word!

Bye,
Mark

2 Comments


Monkey 2 V1.0.0 now available!

Posted on June 30th, 2016

Hi,

It’s been a while coming, but monkey 2 v1.0.0 is now up at github and available as a prebuilt binary (Windows only sorry) from the new files page here: http://monkey2.monkey-x.com/monkey2-files/!

Pleae note that although this is a ‘version 1 release’ it is by no means the end of anything. It’s really just a snapshot of where things stand, but with the added security of knowing that I’m not gonna ‘break’ anything any time soon (some stuff *will* inevitably be deprecated over time though). To this end, quite a bit of stuff I’m not too sure about has been left out of the docs. Feel free to play around with anything – and most of it will likely stay in some form or another – but generally speaking, only the stuff that is docced is guaranteed to keep working the way it does.

Monkey2 now also includes Ted2 – a minimal ‘IDE’ designed mainly so users have *something* to build monkey 2 apps with, and on any platform they can build monkey 2 for. If you find Ted2 a touch *too* minimal, you can still use  Ted from monkey1, or perhaps try out weibow’s Sublime text 3 package – more info here: http://monkey2.monkey-x.com/forums/topic/sublime-text-3-language-file-and-snippets/. I’m sure more add-ons for other editors will happen over time too.

I do plan to continue development of Ted2 – there’s a pretty cool potential gui module lurking in there, and I still love the idea of being able to host any future potential ‘maplet2’ style tools in a common IDE.  But I did probably bite off a bit more than I could chew taking on a GUI/IDE, given I’m also responsible for the language, modules, documentation, website etc. Yet…I still like the fact that mx2 is (finally) a ‘self contained’ product, and you can just download it and use it without any stuffing around!

Ted2 also includes a simple debugger, along the lines of the b3d, bmx and mx1 debuggers. Again, nothing too flash – there’s ‘pause’, ‘step’, ‘step-in’ and ‘step-out’ (although you still need to manually add DebugStop’s to the code, and no debugging of globals yet sorry) but it’s still a huge improvement on ‘print’ style debugging (although that’s often useful!). I do think the ultimate solution to a debugger is probably a ‘gdb parser’ style thing, but that’s a relatively big job that I’m not up to undertaking just yet. There’s also plenty I can improve on with the current system anyway.

Finally, the docs are still a little rough around the edges. I will keep hacking away at them, but there’s still a fair bit to do and I didn’t want to hold up a ‘release version’ release any longer.

I’ll be making another post soon concerning ‘what’s next’ (getting mx2 working on emscripten/android/ios are high priority though) but right now I’m off for a long relaxing bath!

Bye,
Mark

15 Comments


Little update….

Posted on May 7th, 2016

Hi,

Just a little update to let everyone know I’m still here!

I’ve been working furiously on the v1.0 release and things are progressing nicely.

Mojo2 has been massively cleaned up and mx2-ified but should still remain largely recognizable to mx1 mojo2 users. Still no lighting or shadows, but all the basics are in including render-to-image.

I’ve dropped the ‘DrawList’ class entirely – well, more precisely, it’s been merged into Canvas (which ended up pretty much empty after I ripped out all the lighting/shadow ‘hooks’ used by Renderer).  The old Canvas extended DrawList anyway so most users wont see any major difference.

You’ll still be able to use a canvas as a drawlist though (hopefully sometime later today). By default, canvas works in a sort of retained mode, batching up all drawing calls and building vertexbuffer data for when you call ‘Render’, so it will be possible to ‘draw’ one canvas’ draw ops to another. I may eventually add an immediate mode that causes the canvas to flush itself automatically when renderstate changes (and perhaps when some threshhold of vertices has been reached) but the general idea is for it work the same either way – but with lighting/shadows in retained mode.

It’s tempting to add proper deferred rendering, so only shadowcasters need to be retained, but I’d also need to add fallbacks to the current system for crappier HW, so that will just have to wait if I’m to get V1.0 out sometime in the near future! Besides, maybe this stuff is better left for a future 3d module.

Finally, I do plan to get some of this up on github before V1.0 so people can play around with it a bit. I realize there haven’t been many updates recently, but things are definitely happening behind the scenes!

Bye,
Mark

Leave a comment


Are we there yet?

Posted on April 26th, 2016

…Not quite, but I think we’re in the final stretch now!

I now have a usable debugger up and running. It’s currently pretty simple but the basics are in, ie: you can run/pause/step/enter/leave mx2 code and see all the variables on the stack as they are updated. And wow, you forget how much easier a debugger can make life when you’re so used to going without!

The bad news is that I wont be ‘releasing’ this just yet. The debugger – all the GUI stuff in fact – is based on mojo2, and mojo2 is IMO not quite ready for release. I did consider releasing a ‘mojo2-old’ style module, but I have decided against that as it would also need to include an ‘old’ version of the app/window stuff and would just confuse things in the long run I think.

I have abandoned plans for a whole new 2d module, and have instead decided to soup up mojo2. The biggest change I am planning to make here is getting rid of the mojo2 ‘Renderer’ class, which is currently used for lighting/shadows (well, you can ‘sort of’ do these directly with canvas but it isn’t pretty). Instead, lights will be ‘added’ directly to drawlists, in the same way ShadowCasters are. The Renderer class was a pretty good idea, but it forced you to use/create a scenegraph design which IMO just didn’t ‘fit’ with the rest of Mojo2’s ‘immediate mode’ approach.

Most of the ‘grunt work’ stuff like images, fonts, drawing etc will remain the same (with several enhancements) so most of your mojo2 code will look the same/similar, unless you’re using Renderer of course. A backward compatibility Renderer class is a possibility here if there is enough demand.

As for the IDE, I have scaled back my ambitions here a bit. I was (and in the long run, still am) planning on building a ‘plugin’ based IDE, that would allow you to write plugins for stuff like map/scene editors, file conversion tools etc, but I think I got a bit ahead of myself here. This is something best done with a reasonably mature set of APIs, and mx2 just isn’t there yet, to put it mildly!

So the ‘new’ Ted2 is simpler than I originally intended, and in fact bears a stricking resemblance to Ted1. This is pretty much complete now, although it lacks ‘options’ dialogs (you will need to edit JSON files) and a few other bits and pieces. Also, although I wanted to do ‘native’ menus, I ended up doing custom ones instead – it was just faster/easier than getting native menus going on Windows/MacOS/Linux and of course will work everywhere.

I also get that not everyone will use Ted2, so I am also planning to do something like ‘mx2db’ that includes the debugger but is used the same way as mx2cc.

My current goal is to get a V1.0 of mx2 together so people can start actually using it. In addition to what’s already up on github, this will include:

* Ted2 (with usable debugger)
* Custom Gui module.
* Sexier Mojo2 module.
* Decent set of docs.
* Usable help system.
* mx2db for people using 3rd party IDEs.

Things that will not, alas, be in V1.0:

* Threads
* Reflection
* Module manager (soon!)
* Android/iOS support (soon!)
* Custom 3D Module

It’s been about a year now since I started mx2 and yes, like seemingly all software projects, it’s taken longer than originally planned. I have probably taken the ‘long way around’ with a few things, but I do think the end result is going to be pretty cool. I have used it a LOT myself over the last 6 months (something I probably didn’t do enough of with monkey1) and, while there is stil lots to do, I am really happy with the general direction it’s taking.

Bye!
Mark

6 Comments


Update time!

Posted on April 7th, 2016

Hi,

As you’ve probably guessed by now, I’ve been spending quite a bit of time on the website and docs system.

This involved a crash course in wordpress, but I think I’ve mostly got my head around it now. I’m pretty pleased with the results, although it certainly still has a ‘designed by a programmer’ look about it. But that’s OK – it’s a programming site after all! With a bit of luck, once monkey2 is a bit more widely used some more professional looking sites will popup. Still, I like having a site that *I* know how to tweak and will (hopefully) be low maintenance.

I’ve re-enabled post comments for the site and you can now also post comments on doc pages. This is strictly for testing purposes right now. I will probably scrub all the comments before the ‘final’ docs are uploaded so don’t get too carried, but please do feel free to give it a go!

You can also edit comments after you’ve made them. However this side of things is a bit  wonky, as wordpress insists on editing them in the admin page, so you don’t get the nice visual editor. I ‘sort of’ got it going, but it was ‘double escaping’ the posts or something weird so I gave up. Will revisit this later.

But it hasn’t all been web fun and games. Doing the docs also forced me to fixed something I was kind of dreading – aliases. Up until recently, aliases have been ‘resolved’ immediately when their symbols are looked up. For example:

The actual ‘type’ of CameraPosition as far as the compiler is concerned is Vec3<Float>, since the alias is resolved when the type ‘Vec3f’ is worked out. But this means the docs are ‘wrong’, ie: you get this:

Ok, you could argue that’s just as correct, and in this case it’s not so bad. But aliases are often used to simplify complex types, eg:

Having ‘OpenFunc’ expanded to its full type everywhere in the docs just didn’t do it for me!

So it’s fixed now, although it involved writing a ‘proxy’ class which can be quite tricky (esp. when done ‘after tha fact’), so there *may* have been a few issues introduced in the process, although everything *seems* to be fine…

While I was in the guts of the alias system, I also added generic aliases. This was also partly docs inspired, but mostly because I wanted to clean up the ‘StringStack’, ‘StringMap’ convenience types (which I use a lot). These used to be implemented with something like this:

However, this is kind of nasty/cheating (the empty classes it also looked ‘weird’ in the docs). It mostly works, but in reality a StringStack is *not* a Stack<String>. This becomes an issue when you try and do something like:

The last line gives an error because ‘Slice’ is a member of Stack<T> and returns a Stack<T>, not a StringStack, so it can’t be assigned to buf. Even a downcast wont help.

So StringStack and co. are now implemented via Alias, eg:

Which allows buf=buf.Slice( 1 ) to work no problem.

But I ran into problems implementing StringMap as an alias, which used to look like this:

The problem here is that a simple alias isn’t enough, because there is still the unknown type parameter ‘V’. The solution is a generic alias:

This can be used in exactly the same way the old ‘extends’ version could, eg:

I also spent a bit of time on the extern system. You can now #import local “.dylib” and “.framework” libs on Macos, and the compiler tries a bit harder to think up a useful symbol for extern decls that are declared without one. For example, this…

…now generates ‘C::Test’ as the native symbol for ‘Test’. It used to just use the decl identifier, which in this case would have been plain ‘Test’ which was pretty much useless. Note that if you explicitly declare a symbol using =”blah” syntax, mx2 doesn’t touch it at all.

Enums are a tricky case. There several ways to declare enums in c/c++, so one system that deals with all of them is not easy, but I’ve given it a shot. By default, this…

Will map V1 to ‘V1’, V2 to ‘V2’ etc. This is to handle c/c++ enums which are actually not added to the enum scope, but the enum’s outer scope. Similarly, this…

Will map V1 to ‘C::V1’ and V2 to ‘C::V2’.

You can also declare ‘anonymous’ enums using this kludge…

This will map V1 to ‘E::V1’ and V2 to ‘E::V2’, BUT it will use ‘int’ as the enum type, not ‘E’. This is to handle c++ code such as:

Ideally, all this interface code should really be automatically generated by some cool tool (llvm lib looks promising), but until such a tool exists I think these little extras will be worth it.

So, what next for monkey2?

My immediate goal is get a usable debugger going, and that means resurrecting the gui/graphics system.

The debugger will, for now, remain based on the simple approach I took in bmx – ie: code as added to the translated output to store current function/statement location. This side of things is already working to a degree – in debug mode you can now use GetDebugStack() to find out where you are in the stack. However, debugging without *some* kind of gui is no fun at all so getting mojo working again is a proprity right now. And yes, debug info WILL include globals this time ’round!

This style of debug info is not the most efficient, and ultimately it’d be nice to somehow translate c++ debug info to monkey debug info, and parse lldb debugger output etc. But given my limited knowledge of either c++ debug info or lldb, that’s a relatively big job that I wont be attempting right now.

Besides, this ‘hacky’ approach to debug info has some benefits – it can be used to do a simple profiler, custom code can be called every function/statement etc. And an interesting idea: it should be possible to track the last N statements executed in a cicular queue, and find out how code got to a particular statement. You wouldn’t be able to ‘undo’ var changes, but just seeing what the last N statement executed before a crash might be useful?

There have also been a few questions asked recently about how monkey2 will work on ios/android. And the truth is I don’t know yet. My current thinking is for monkey2 to generate a static (or dynamic) lib that contains all the app code (and only needs to expose a bbMain), that can be dumped into a ‘target project’, ie: an Android Studio or XCode project. The SDL code might end up in the target project too, so it can be properly initialized at the right time and hacked if necessary to deal with app suspension or whatever.

I also think target specific code and libs such GooglePlay, AppStore, Ads etc should be in the target project, NOT monkey. These often have weird requirements for adding them to projects such as having to modify linker settings etc, and while I tried in monkey1 to modularize this stuff I think ultimately it just didn’t work very well.

I also want to avoid the situation I got into with the ‘app config’ settings stuff in monkey1. This basically allowed you to modify a limited subset of project properties (such as android manifest settings or ios info.plist vars) via ‘#SOME_VAR=SOMETHING’ in monkey code. The compiler then attempted to hack these settings into the target project using a variety of unholy techniques.

This was based on my initial flawed thinking that ‘it would be nice if the user only had to deal with monkey’. But it just wasn’t worth it – it was a nightmare to maintain and IMO just made it harder when you ineveitably needed to add something to the target project that wasn’t supported via app config settings. So in monkey2, if you want to modify the android manifest or ios info.plist, you will need to do so OUTSIDE of monkey2. The simplest way being of course to use android studio or xcode – which are designed for exactly this stuff.

Bye,
Mark

4 Comments


Early version of docs now up!

Posted on March 25th, 2016

Hi,

I spent most of my time this week making a start on the docs and creating a ‘docmaker’ system.

It’s early days yet (isn’t it always?) but you can check out the results here:

http://www.monkey-x.com/mak/mx2/mx2docs/docs/

There are a few things I want to change around, but not many so I think considerable chunks of it are finally reaching a stable state!

My design skills still suck, but I think it looks pretty good. The treeview script needs some work but I’m OK with the general layout. It may not work too well on HDPI displays yet though.

I quite like the monkey silhouette ‘logo’ too. It’s pretty distinctive and the fact it’s a single color means it can be used with any colored background.

There’s some new code up on github too. There’s a new ‘makedocs’ script in /src that builds the docs. It currently only does monkey, libc and stc modules, but you can modify to do all modules by removing the params after ‘mx2cc makedocs’.

Bye!
Mark

Leave a comment


Monkey2 Progress Report

Posted on March 8th, 2016

Hi,

Apologies for the silence, but I’ve been busy cleaning up and finalizing 1001 ‘little things’ I have encountered while converting the compiler to ‘pure’ mx2.

It hasn’t been very exciting stuff but the end result is that the compiler – which is indeed now pure mx2! – is a LOT more solid than the last version. It picks up a lot more errors than it used to (stuff that used to just break at the c++ compile stage) and hopefully reports them in a more useful way.

There are also a few additions that, while they seem outwardly sensible and ‘easy’ – were in fact quite tricky to implement. This includes stuff like allowing you to use assignment operators such as += etc with a class that only provides an operator+ (you *can* provide +=,  but if you don’t mx2 will generate one for you), being able to use += etc with properties and operator[]=, and several other similarly helpful shortcuts.

A lot of this is stuff that c++ and (I think) c# don’t allow, but they’re the kind of thing that, when you’re learning about operator/assignment overloading (well, when I was anyway) you often wonder why the compiler doesn’t help out a bit more. So mx2 does!

I have also finalized how objects/values are compared, and how comparison operators work. I did have this ‘sort of’ working in the last version but, like many things, kind of just got it working and moved on to the next thing, vaguely aware there were holes in it.

Anyway, the system I have settled on works like this:

Note that, like all operators, comparison operators are ‘statically typed’. This means that if you create an object of a class with custom operators, and assign it to a plain ‘Object’ variable, the operator is effectively lost as the compiler can only see a plain object. However, operators can still be virtual (or not).

I also spent a fair amount of time speeding up the whole build process. The real killer here currently is c++ compile times, so the new mx2 translator tries much harder than old mx2 did to #include as little as possible. A lot of translated files now just start with a bunch of simple ‘forward type declarations’ where the old files would have used #include. This reduces dependencies so modifying one file has less of a knock on effect on other files.

Other tweaks include: massively improved overload resolution algorithm; support for invoking generic methods declared in an external module (never worked!); support for ‘generic generic parameters’ (WIP!); full support for all numeric types; addition of native CString, WString and Utf8String types for use with external APIs.

I plan to release new mx2 as a ‘language only’ release sometime this week, probably via github (finally!).

This wont be of much interest in people wanting to write apps, but I’ll be getting back into that side of things soon so please be patient. People who are interested in stress testing the language side of things though, please check it out!

Bye,
Mark

2 Comments


Minor update on new compiler

Posted on February 10th, 2016

Hi,

I’m currently in the process of converting the mx2 compiler from monkey1 to monkey2. It can’t *quite* compile itself yet, but I think it’s about 90% of the way there and I’m hoping to have it all up and running sometime next week.

In the process, I have cleaned up quite a few issues with mx2, some of which I knew about, some of which were news to me!

For starters, in old mx2 you couldn’t explicitly specify a generic function (which is an incredibly nasty phrase but much worse than it sounds. If you know a little about generic stuff, the following should make some sort of sense…).

For example, take the simple generic ‘Min’ function:

In old mx2, the ‘version’ of Min used is entirely inferred from the parameters you call it with. For example, if you call Min( 10,20 ) the int version of Min will be used (ie: type ‘T’ will be ‘Int’ inside the body of Min) because 10 and 20 are ints. Ditto, if you call Min( 10.0,20.0 ) the float version will be used.

But things get a bit messy if the parameters are of different types, for example Min( 10,20.0 ) where one parameter is int and one is float. In this case, the compiler gets fatally confused because it can’t determine whether to use the int or float version of Min.

One solution is to ‘cast’ one of the parameters to the type of the other, but the clearest way is to just ‘select’ the version of Min you want to use up front, and you can now do this using Min<Blah>. For example, Min<Float>( 10,20.0 ) will use the float version of Min, regardless of what parameters you pass. The compiler wont get confused because there is only one float version of Min to choose from so it knows to convert both parameters to Float and life is good!

This feature effectively allows you to ‘pass a type’ to a function, and it becomes super useful when used inside other templates. But more on that later…

Another thing I wanted to have a go at (but wasn’t too sure I could do!) was solving the problem of how to resolve generic function overloads. For example, this will fail under old mx2:

The problem is that the call to Test with an Int[] parameter is actually a viable match for *both* generic Test()s. In the first version, ‘T’ would be ‘Int[]’, while in the second ‘T’ would just be ‘Int’.

However, it ‘feels’ like the second version should be called – and indeed c++ does this. But I couldn’t get my head around an algorithm to decide what made it deterministically better than the first! I wasted a couple of days+ on this trying some very ugly stuff that involve counting ‘type depths’ and so on, but eventually gave up.

Anyway, I revisited this recently and it turns out the solution is remarkably simple (although perhaps not to explain) and I feel like kind of an idiot for not spotting it sooner.

The basic idea is to see if you can ‘pass’ function 1’s parameters to generic function 2. If you can, but can’t do the opposite (ie: pass function 2’s parameters to function 1), then function 1 is a better match.

For example, if we try to call Test(2) with Test(1)’s parameters, it wont work because Test(2) wants a ‘T[]’ but we are just passing a ‘T’. However, the opposite DOES work, ie: we can pass a ‘T[]’ to a generic function that takes a ‘T’. So the second version is a begtter match, ie: the first version is ‘more generic’.

Note that this ordering is completely independent of the ‘real’ types involved, ie: the functions could be sorted beforehand into a best->worse order, although new mx2 doesn’t do that yet.

Ok, that probably didn’t help at all (it’s pretty weird to pass generic types to generic functions!) but the upshot is that it works and works very well!

There are quite a few other cleanups and improvements in new mx2, but I think it’s probably time for a bit more ‘Soma’…

Bye,
Mark

7 Comments


Welcome to the new Monkey 2 devblog!

Posted on February 1st, 2016

Hi,

I have decided to move the monkey2 development blog to wordpress, mainly so I can add forums to the blog but also with the expectation that this will eventually grow into a complete monkey 2 site.

I’m new to wordpress but have so far managed to avoid any coding what so ever! It’s all been click ‘n’ play which has been great, as I’m not a huge fan of php.

The site looks a bit rubbish right now, but the forums should at least work fine. There are only 3 forums right now – “Monkey 2 FAQs”, where I plan to add little bits and pieces of info from time to time (and will eventually give me something to copy and paste into the docs!), “Monkey 2 Development”, which will replace the current ‘monkey 2’ forum at monkey-x.com and a “General programming” forum. To post to the forums, you will need to register at this site. See the ‘register/login’ links at the bottom of this page.

And the nice thing about using wordpress is that it should be relatively easy for someone who actually knows what they’re doing to play with it a bit and make it look 100 times better. Which is another excuse/incentive for me not to play with anything under the hood…

Bye!
Mark

Leave a comment


Fun with fibers!

Posted on February 1st, 2016

Hi,

I spent a bit of time over new years playing around with ‘fibers’.

This was partly in response to requests for me to look into async/await style functionality, but also touched on a bunch of other stuff I was interested in checking out such as coroutines/generators.

But just what is a fiber? The term is pinched from the win32 API, and they are in effect ‘lightweight’ threads. Like real threads, each fiber has it’s own stack pointer/program counter etc.  However, unlike real threads, fibers don’t run ‘in parallel’. Only one fiber is ever running at a time, and a running fiber must explicitly ‘yield’ to allow another fiber to run.

My first stab at a fiber API was very simple and looked something like:

With this alone, you can do some pretty cool stuff. For example, you could cause a fiber to ‘wait’ for the App.Idle signal, faking a ‘VWait’ of sorts:

(Note: fiber ‘0’ is reserved to mean the ‘GUI’ fiber…this should NEVER block!)

Probably looks  a bit weird, but it’s actually pretty simple. All this does is add a ‘signal listener’ function to the App.Idle signal (which is emitted by the app framework when the app is idle)  which, when called, wakes up the fiber that is VWaiting. Note also that VWait() will not work when called on the GUI fiber, as in this case the SwitchToFiber( 0 ) at the end becomes a NOP.

The messy bit is the switch back to fiber 0 at the end of VWait(). This is fine in a lot of situations, but not all. The simplest example of this is a fiber that starts another fiber that vwaits. In this case, the ‘child’ fiber shouldn’t just switch back to the gui fiber, but back to the ‘parent’ fiber, ie: the fiber that started it.

The problem here is that ‘pure’ fibers only really have 2 states – running or blocked. What’s really needed is a third state – ‘ready’. A ready fiber is not running, but it’s not blocked either – it wants to run again in the future. And it turns out this is actually quite easy to do, via the addition of 2 functions that work a bit like this:

…where ‘readyStack’ is just a global stack of ints.

There are a few subtleties that mean these aren’t just implemented as wrappers around SwitchToFiber (in particular fiber creation/destruction) but that’s about it. Fibers don’t need an explicit ‘state’ var added to them, as their ‘ready’ state is implicit in whether they’re on the ready stack or not.

VWait looks almost the same…

…but can now be safely called from pretty much anywhere  – except, still, the GUI fiber! In which case, SuspendCurrentFiber will attempt to pop an empty stack anyway, so again, do not block the GUI!

But when should you actually use fibers? Well, IMO one of the coolest uses for fibers is to ‘linearize’ game logic. If you check out the ‘defender’ source code in xmas monkey2 demo, you’ll notice a ton of ugly stuff in the ‘OnRender’ method for handling game state changes and dialog timers. This is due to the fact that the app is callback based, and callbacks must return promptly and without blocking for everything to keep going, so you’re stuck with having to update a bunch of state vars incrementally every tick.

However, with fibers you can do the sequential game logic stuff in a much more natural manner. The latest version of defender now has a ‘GameLoop’ method that looks something like:

This code is run on a fiber using CreateFiber( GameLoop ) and as such, it can freely block (eg: vwait) without blocking the GUI thread. The dialogs can therefore ‘wait 10 seconds’, PlayGame() can VWait etc without any problem. All in all, it’s suspiciously like coding in b3d/bmx again!

Another use for fibers is in writing ‘generator’ style code. In fact, it is remarkably easy to build a generic generator class with fibers:

…and a simple example ‘int range’ generator…

Finally, there’s a catch: fibers don’t currently work in emscripten. There are features being added to emscripten that means they may eventually happen, and I have had a demo going (only works on firefox ‘nightly’ though) although it was a bit limited (you could only start a fiber from the main/gui fiber) but realistically I don’t think fibers in emscripten will be seriously viable in the near future.

But one of my ‘conditions’ for providing emscripten support was that I was NOT going to let it hold back the other targets, and this is a very good case in point. So monkey2 will eventually have some sort of fiber support, but probably only for non-emscripten targets. Bummer, but what can ya do…

Bye!
Mark

2 Comments


XMAS Demo (V3)!

Posted on December 23rd, 2015

Hello and happy Christmas!

I have put together a little monkey2 ‘xmas demo’ – downloads here:

Windows: monkey2_xmas2015v2.exe

MacOs/Linux: monkey2_xmas2015v3.zip

The Windows release is a self extracting archive that should ‘just work’!

The MacOs/Linux release is source code only (the windows release includes source code) that will need to be built.

On MacOs, this means the ‘xcode command line tools’ will need to be installed and on Linux, it’s off to apt-get land. However, if you already have monkey1 (or blitzmax?) installed and working, you should be able to build monkey2 without any problems.

To rebuild the source code, you will need to open a shell, change dir to the monkey2/src dir, and enter ‘./rebuildall.sh’ After a time of wonderment, Ted2 should pop up and you’re good to go.

It’s still early days, but I’m kind of proud of this! There’s more info on what’s coming next etc inside the demo.

V2 now up

I’ve just uploaded a v2 that fixes a windows ‘mingw build’ issue that caused builds to fail, and a crash bug with ‘Open’ in Ted2.

Also, I may a few blind tweaks to the source version that may allow it to build on older versions of MacOS.

Known issues

Bye,
Mark

Leave a comment


Makemods Update

Posted on November 17th, 2015

Alrighty, I finally have what I think is a pretty good module/library system going!

The mx2cc app now has a ‘makemod’ option in addition to ‘makeapp’, which creates a ‘.a’ archive file (‘.so’ to come…) from all .monkey2 files reachable (via #import) from a given root .monkey2 file. For now, you will need to include these source .monkey2 files if you want to redistrubute a module in ‘prebuilt’ form, but I plan to eventually provide an (optional) way to convert these to a pre-processed format.

The syntax I ended up with for importing a lib is simply #Import “<modname.monkey2>”. This has a nice symmetry with #Import “localfile.monkey2” and with the existing mechanism where you enclose system lib/framework etc files in “<>”. In this case, the system file is a precompiled monkey lib.

Libs still currently go in the ‘modules’ dir, and it might be time to revisit the naming of particular concepts. Currently, ‘module’ is roughly equivalent to ‘monkey source file’ but it should probably mean ‘precompiled set of .monkey2 files’, as it does in bmx. Modules don’t have usable names from inside the language (if you want the contents of a module to go into a different namespace, you need to use ‘Namespace’) so the concept of a ‘module’ is really only of interest to the compiler anyway.

I was expecting this to be a bit tricky, and indeed it was! While I had the ‘guts’ of it all going pretty quickly, there was a fairly large fly in the ointment – generics. The problem here is that adding new ‘types’ of List, Stack, Map etc (eg: List<Actor>, List<View>…) really means you’re changing the ‘std’ module on the fly, as all these new types end up in the std namespace. But you don’t want to have to recompile the std module because of a new generic type, so the source code for these new types has to go in the module they are declared in – which means potential multiple definitions.

But I think it’s all going pretty well now, and solves one of my biggest gripes about monkey1 – the way it had to rebuild everything all the time! Compile times for larger projects should be MUCH better under monkey2 than they were under monkey1.

Bye,
Mark

Leave a comment


Build System Update

Posted on October 30th, 2015

Hi,

In preparation for ‘makelibs’ I’ve spent the last few days rearranging the build system so each app gets it’s own build dir. Boring, fiddly stuff, but necessary.

I’m using a .build dir system similar to monkey1’s, with a subdir for each ‘profile’ that will contain the output app, and a build_cache dir for temporary link files, eg:

Some of this will eventually be configurable, via settings such as PRODUCT_DIR, PRODUCT_NAME etc.

A ‘profile’ in this case is just a combination of ‘build flags’, eg: target; debug/profile/release; static/dynamic etc. It’d also be nice to allow people to add custom profiles in addition to the built-in/default ones. Custom profiles would be to define and override compiler and linker options.

Already, just giving each app it’s own build dir is giving me a better idea of how monkey2 builds should perform in practice, and once the initial ‘build everything’ is done, it’s looking pretty good!

Bye ,
Mark

Leave a comment


Pre-pre-pre Alpha Release

Posted on October 27th, 2015

Hi,

Ok, new monkey2 demo time!

As with the last demo, this should be installed along side the latest monkey release as it is still IDE-less. It will also work with the latest free version of monkey.

This is still very much pre-alpha stuff, but it should be border-line usable now and has reached a state where I reckon it’s worthwhile getting some feedback on how it’s shaping up.

The zip file is available here: http://www.monkey-x.com/mak/mx2/monkey2_28_Oct_2015.zip. I’ve also started on a little ‘monkey1 to monkey2’ doc  here: http://monkey2.monkey-x.com/2015/09/09/monkey1-to-monkey2/

Most monkey1 language features should now be more or less functional (though I have tested some way more than others) and a few of the new monkey2 features are proving to be VERY useful. In particular, I am loving function values and structs! There are some other new features in there, but they have received far less testing and some are very underdeveloped.

There is still no thread support, and I’ve pretty much discarded any stunt parsing to deal with EOL issues (for now). I’ll revisit this later, but right now my priority is to get error handling, line numbering etc working correctly and EOLs complicate this somewhat.

There is quite a bit of module code in there, some of which will be familiar to monkey users, and some of which will be familiar to blitzmax users! However, please consider this stuff to be mainly ‘brainstorming’ code – I find writing modules to be both an excellent way of debugging the compiler, and trying out freaky ideas. Most of the module stuff will remain in one form or another, but some of it will change as language features are added.

So I still don’t recommend writing ‘real code’ with this version…please!

There is some fairly interesting stuff in the modules though, including:

* A pretty complete implementation of mojo2.

* The return of blitzmax’s stream system, where you can specify a ‘protocol’ when opening a stream.

* A quick ‘n’ nasty implemnation of a ‘zip::’ stream protocol, courtesy of the miniz lib. Not cached or anything so woefully inefficient.

* A minimal custom GUI. This will probably be a bit controversial, but it’s happening. It already provides a few features I think are sorely missing from monkey, such as easy support for virtual resolutions (including dealing with mouse coords and window resizing) and support for multiple windows (which SDL supports so why not?). These are things that are kind of ugly to do without some sort of ‘view’ architecture.

* A very simple HtmlView based on the litehtml library. Don’t expect to be able to write a browser with this, but for displaying docs/instructions I think it will be very useful.

The ‘big picture’ is shaping up a bit more now, and my next big job is to work on some sort of ‘library’ system.

Currently, there is no library system to speak of. Source file changes will automatically trigger a recompile of dependant files, but it’s all still just a massive soup of source files at heart. This works pretty well once the ‘build cache’ has warmed up, but it’s already hitting limits. When you switch to a different project, far too much stuff is recompiled (partly due to the generics system) and on Windows at least, linking 100-ish object files is brutally slow.

So it’s time to clean up the build system and tackle libraries, and what I’m planning to do here is something like blitzmax’s ‘makemods’. This will behave similarly to ‘makeapp’, except the result will be a static .a archive file OR a dynamic .so library. You’ll be able to import libraries using something like #import “<sdl2.monkey2.so>”  – or, IDEs could perhaps allow you to select libs and pass them on to mx2cc, ala msvc and c#’s assemblies.

The main idea behind supporting .so libs is to make it possible for monkey2 apps to support plugins, something I think would be useful for both games and game editors. By placing all the GC and memory managment stuff in a single shared .so, it should be possible to pass objects etc around between plugins with no problems (ha!). This is all likely to impact the module archictecture considerably so I think it needs to be done before I get any deeper into module/library development.

Other big picture stuff includes reflection and debugging, but that’s a story of another post!

Bye,
Mark

Leave a comment


Monkey1 to Monkey2

Posted on September 9th, 2015

Hi,

A quick list of things to consider when converting monkey1 code to monkey2…

Strict mode not an option!

All functions that return something must return something! In monkey1, you could omit return statements even in functions that returned non-void. Not so in monkey2 (although this isn’t actually implemented in the compiler yet..).

Variables no longer default to int – all variable declarations must either specify a type, or be initialized with a value, eg:

Method and function declarations can still omit a return type, however the default return type in monkey2 is void (as opposed to int in monkey1).

The monkey1 shortcut type specifiers (eg: % for int and # for float) are not supported in monkey2 – you must use ‘:Int’, ‘:Float’ etc.

Using namespaces and importing files

Monkey2 adds a ‘namespace’ system to monkey1.

This basically means all declarations belong to some namespace. There are no ‘global’ declarations.

The easiest way to deal with this is to add this to the top of your source files:

This will allow you to access ALL decalarations in the std and mojo namespaces.

To import monkey2 source files into a build, use the preprocessor #Import directive, eg:

If you don’t specify an extension, #Import assumes ‘.monkey2’ so this actually imports “file2.monkey2”

Function call parameters must be enclosed in parenthesis

In monkey1, you could (usually…) omit the paranthesis around function parameters, eg:

In monkey2, this is not allowed – all function parameters must be enclosed in parenthesis, eg:

Array creation syntax changes

Monkey1 allowed a ‘quick’ form of array constructor:

In monkey2, you need to use the longer (but equivalent) form:

Also, to create an array with existing elements you must now use:

This replaces the old [value0,value1,value2…] syntax.

Virtual and overriding methods must be explicitly marked

In monkey1, all non-final methods are automatically virtual and can (sometimes inadvertently) dynamically override existing methods. In monkey2, virtual and overriding methods must be marked as such, eg:

Property syntax changes

Code that declares properties will need to use the new syntax. Instead of this…

…you will to need to use…

End-of-line handling

Monkey2 is currently VERY strict about end-of-lines.

All ‘block’ declarations (Class, Method, Property, Function) and block statements (If, While, Repeat, Select) must be of the following form:

_header_
…_content_…
_footer_

_header_ and _footer_ must start on a new line, and be the only thing on that line. Declarations can no longer be ‘mashed up’ onto a single line. For example:

Multiple statements can appear on a single line, as long as they are separated by the statement separator token ‘;’, eg:

Finally, monkey2 ignores any end-of-line tokens that appear after any of the following: ‘(‘, ‘[‘ and ‘,’. This allows you to split function parameter lists and auto array elements over mulitple lines.

No fancy slices.

Instead of this…

…use this…

Bye,
Mark

1 Comment