Sic Semper VisualBasicus

Something I overheard today at work:

We should hold a big VisualBasic conference, and once everybody is inside we gas the place.

It still boggles my mind that so much code is written in VisualBasic.

Silverlight Website

Yesterday I mentioned that I'm writing a new web application with Silverlight. I also expressed a few misgivings about the current level of support for Silverlight in VisualStudio and other tools. Today, I'm going to talk a little bit about the design of this new website and some of the decisions that we made to arrive at this design.

Our current web application is written in ASP.NET/C# and uses a series of third-party plugin controls to add improved aesthetics, performance, and capabilities. This strategy works reasonably well, though I've personally found it to not be maintainable as-is. I've found that complexity is growing much more quickly than capability does, and that our limited development resources quickly get strained. This is especially true when we start running into impossible-to-debug errors in hundreds of lines of generated javascript. Our entire team has ground to a halt on more than one occasion while one intrepid developer suffered through long sessions of phone tech-support to resolve these kinds of issues.

If we were using a combination of PHP and JavaScript, with a comprehensive client-side framework like jQuery things might be a little different. But, we're not. That ship has long-since sailed and there is no going back now.

The fact is that my office is a C# house. All our developers know C#, all our projects are written in C#. Trying to switch our main web application to use PHP instead would require major retooling time and effort, retraining for several developers, lots of lost time and productivity, etc. C# is as good a language as any, and if we can get moving with the right tools and the right technologies it should work just as well as anything else.

The problem I have with ASP is that it doesn't run directly, it generates HTML, CSS, and JavaScript. The result is that the source of the actual page which the client receives bears only the vaguest resemblance to the code which the developer wrote. Find a bug somewhere in the CSS? Unhandled JavaScript exception? Unless it is painfully obvious where the problem is coming from, it might be impossible for anybody to trace that problem directly back to the offending ASP code. Producing a robust and portable solution may be even harder.

This is not to mention that there are major differences in JavaScript, HTML, and CSS implementations between browsers and there's no guarantee that anything you see on your development machine will look the same when the client gets it. I can guarantee that Microsoft tries hardest to make things look spiffy in Internet Explorer, but even then I have found some severe problems with ASP websites not rendering properly in various versions of IE. For modern browsers this usually isn't a huge problem anyway, but we deal with several clients who use IE6 by corporate mandate and refuse to update for "security concerns". Go figure.

[Side Note: If your IT personnel claim that upgrading from IE6 to something newer is a security concern, fire them. This is the first sign, and not a trivial one, that the person in charge of your IT department is incompetent.]

JavaScript differences between such a huge range of browsers is really a pretty serious concern. There's a reason why I like working with VMs, which provide a standard interface across multiple platforms. Parrot is my favorite, of course, but .NET/Mono earns a strong honorable mention too. The more we can write in a managed language like C#, and the less it gets converted to something less reliable, the better. Generated code is the enemy here, and HTML/CSS/JavaScript is really a terrible choice for target "language".

These are the reasons why, in a nutshell, that I picked Silverlight for the new website. Toolchain support is sophomoric at best, but it does give us what we need otherwise: We can write everything in C#, we don't have to worry about differences in rendering, performance, or capability between different browsers. Even the poor IE6 folks have a Silverlight plugin installed. We can leverage the strengths of our existing team too, which is deadline-friendly. We have a good opportunity here to mature with our tools, and I've already filed a few bug reports and sent in some feedback to Microsoft about some of the issues I've found. All we need now is to wait for service pack 1 without killing ourselves or each other.

The website requirements are a pretty nebulous beast, as I'm sure they are for any sufficiently-usable web application. Different customers all want to see different things and have different tools at their disposal. What we needed was a framework that would allow us to create a pluggable interface and support/enforce encapsulation and abstraction like any modern MVP application wants. To do all these things, we decided to use Prism. Prism is part framework and part "guidance" which provides tools, lessons, and examples to follow for building modular MVP or MVVM applications. Basically, it's a nice booster to make sure your team is following some modern best-practices when it comes to scalable and maintainable application design.

Like any methodology, adhering too strictly to MVP or MVVM design can put you in an uncomfortable place, but if you take the lessons to heart and use the tools that you need, I've found that it works out pretty well. One particular nit that I do have about it is that aggressive decoupling of your classes leads  to referring to modules by name with strings, or System.Type objects, and you miss out on entire classes of analysis and error-detection tools that your compiler's type system would provide otherwise. "Be really careful" is a strategy that doesn't really work well in life and doesn't really work well in programming either. There's nothing worse than trying to track down an error which raises no compiler warnings and fails silently at runtime too without throwing an exception. Combine this with my inability to debug Silverlight from VisualStudio and I've run into a few situations where I almost pulled some hair out. Once I get my debugging issues sorted out this won't nearly be so much of a problem, but right now it makes for long days.

Kakapo and Parrot-Linear-Algebra

Parrot's 2.6 release is out and now everybody is holding their breath in anticipation of the new Rakudo Star release tomorrow. I'm still fighting with my own release project: Kakapo.Kakapo got broken pretty severely when Parrot was changed to no longer store methods in namespaces, and I have been unable in several weeks to make it work again. Austin has been busy with other projects during this time too, so it's been me by my lonesome to try and get this thing working again.

The problems are three-fold: First, Austin's plan for Kakapo was ambitious and the framework does a hell of a lot of things, not all of which I fully understand. Second, NQP and P6object have progressed in some significant ways since Kakapo was originally inked out, and much of the workaround code that Austin wrote to bypass some missing features no longer seems necessary. Third, we have this issue with methods not working nicely with NameSpaces anymore, which makes all of Kakapo's method-injection logic break.Without method injection happening, and happening early enough, all the code in Kakapo which relies on the new methods is broken.

Tene put together a patch for NQP which allows methods marked "our" to be stored in NameSpaces again. This should simplify some of the new logic in Kakapo though I've been too busy in the past few days to test it. Hopefully this patch of his represents a large portion of the fix to get Kakapo working again. I doubt it will be everything that's necessary, but it will be a nice start.

Without working Kakapo I haven't done any work whatsoever on Parrot-Linear-Algebra. PLA uses Kakapo to implement all it's unit tests, and I'm happy enough with the new testing framework that I do not want to move it to use anything besides Kakapo. In the worst case I would consider trying to break out a testing sub-framework from Kakapo if we can't fix the entire framework, but I haven't gotten to that point yet and likely won't have the time for it in the near future. I would much rather fix the framework as a whole first, and then maybe talk about splitting out the unit test stuff into a separate sub-library later, if I still feel like that's a beneficial path to take.

What I would really like to be doing is getting back into PLA development. I had hoped to get some wrappers together to make it work with Rakudo Star, but without working tests I haven't cut a suitable release and without a release it won't be included in the Rakudo Star bundle anyway.

My hope is that within the next few days I can get Kakapo working again. At least I need it working enough that I can get the unit tests for PLA going. With Tene's fix, some hard work, and some trial-and-error I think it will be possible.

VisualStudio 2010

The new web application at my work is going to be Silverlight-based, which gives me an opportunity to learn a fancy new technology and add another line of gibberish to my resume.

As anybody who's worked with Silverlight will tell you, VisualStudio 2008 really doesn't help you at all. I don't have many nice things to say about XAML, and VS2008 requires you to write it by hand. We could use Expression Blend, but without a paid license you're using a crippled product, and that doesn't make any sense when you already have an expensive license to VisualStudio. Luckily, the beautiful IT people got us licenses for VisualStudio 2010, which boasts massively-improved Silverlight support.

Notice that I said "Massively-improved", not "perfect" or even "good". Yes, VS2010 is head-and-shoulders above VS2008 in terms of Silverlight support. However, I'm having a hard time really appreciating those improvements in light of the other glaring deficiencies. And there are many deficiencies still.

In the marketing and documentation VisualStudio 2010 is sold as having "Silverlight support". What does that even mean? Yes there is a new WYSIWYG editor to help improve Silverlight GUI design which produces mostly-correct XAML code. It's far better than nothing, but I find it to still be ridiculously frustrating to do even simple operations. Working with grids is tiresome. Working with controls which need to dynamically size is even worse. I find myself several times an hour wishing I could use some CSS to add style information. I find myself wishing I could specify height in percent, instead of a fixed number, the ambiguous "Auto" or as a weighted ratio. None of these are even among the most damning criticisms I have.

The set of built-in controls available to the Silverlight developer is paltry at best. If you want to write anything more complicated than a tic-tac-toe game, you're going to need to download some kind of add-in or third party package. Even some of the most basic controls that a Windows developer has grown accustomed to in the past 20 years are missing with no replacements in sight. All the best documentation, even documentation from Microsoft, recommends you download the Silverlight Toolkit to get back some of these basics. You have to wander why some of these necessary and recommended controls weren't just included in the standard libraries.

Silverlight has support for several powerful graphical features like transforms, animations, storyboards, and projections, but no good ones are available by default. Get within 20 feet of PowerPoint and some knucklehead is going to abuse the star-wipe, but the basic Silverlight library doesn't even provide you with a basic wipe, fade, slide or expand/shrink. Combine that with the fact that VisualStudio2010 doesn't really provide you with any decent tools to create these things yourself, and you find a huge new source of frustration.

And I've left my biggest complaint for last: Debugging. Let me add some emphasis here for clarity: You cannot debug a Silverlight application in VisualStudio 2010 without downloading and installing a separate, and finicky, add-on. You cannot debug Silverlight applications, by default, in an IDE which boasts "Silverlight support". If you want to debug your Silverlight application (or do other advanced operations like "unit testing"), you need to download the Silverlight Tools for VisualStudio 2010 package. I can't even imagine why they wouldn't have included this functionality directly in VisualStudio itself, if every Silverlight developer is going to need to download and install it separately for even the most basic tasks. That is a decision that boggles the mind and leaves me completely flabbergasted.

But wait! The problems have not ended once you install those tools. If you install your add-ons in the wrong order, or pick the wrong settings in VisualStudio or even your browser, or unknowingly anger the gods of programming, you still won't be able to debug. I know, it happened to me. I haven't been able to set a breakpoint since I started this web application two days ago, and it is seriously hampering my coding mojo.

I like Silverlight, generally. The reliance on XAML is a pretty lousy and short-sighted design decision, in my opinion, but isn't a killer. You do get rich client-side applications without having to write them in ActionScript or Java. I wish that VisualStudio's touted "support" was a little bit more impressive and comprehensive.  In short, I really hope that this platform and toolchain matures significantly in the coming years, because we have a growing web application that we're going to need to be expanding and maintaining for the foreseeable future.

Lorito: A Design

Cotto has been putting together a really great checklist for Lorito, the new microcoding approach that the Parrot developers are hoping to implement soon. The first step on the checklist, creating a proper compiler for our ops code DSL is already complete. Today at #parrotsketch Allison also mentioned some things about Lorito, and how we should start focusing more effort on it. I, for one, am very happy about that.

In this post I'm going to play architect and spell out a vision for Lorito. At the very least, this will be a good play-excercise. I don't expect everybody to like all the things I say here, but I do expect this post to generate some thought and discussion.

On the ground floor, Lorito needs to be able to do what C code does now. Actually, it really needs to be able to do much of what the underlying hardware machine does now. C is just a useful abstraction over the hardware, one that people are familiar with. What we don't want to be doing is creating our own ABI, or creating new low-level calling conventions or anything like that. Those kinds of problems are already resolved and if we want to be able to tap into the myriad of existing libraries we will want to stay compatible with existing conventions.

We want Lorito to interact with raw integers and pointers, including pointer dereferences and pointer arithmetic. This is important so that we can easily work with C-level arrays and pointers. I can't think of a modern architecture where pointers and integers are different sizes and are treated differently, but I can't say I'm familiar with every architecture in current use. I don't foresee any huge problems with creating an opset that works transparently with integers and pointers.  We also want Lorito to be able to call C-level functions, including indirect calls through a function pointer. There's no hard requirement yet stated that Lorito needs to be binary compatible with compiled C code, but I think we can all agree that such would be a major benefit. In fact, if we had a utility that could compile Lorito directly into C code, that would be the best intermediate step we could take. As I will discuss next, such a tool would make the conversion of Parrot to using Lorito easier in the long run. We probably don't want Lorito-to-C conversion to be a permanent part of our build, but it does get us moving now.

The PASM ops are currently written in C. If Lorito does everything that C can do, eventually they could all be rewritten in Lorito instead. That raises the interesting idea that groups of Lorito ops can be composed into more complex higher-level ops. PASM then becomes little more than a set of predefined convenience compositions, though it certainly does not represent the complete set of possible compositions. Dynops would just be custom predefined Lorito compositions and, since they wouldn't be defined in machine-code libraries, their definitions could be included directly in bytecode for later use.

In fact, while we are moving down this though path, we get to the idea that we could identify repeated patterns of Lorito ops at compile time, and define custom compositions on the fly. This allows us to compress packfiles for size without really adding all the overhead of general-purpose compression algorithms. When optimizing code, if we can apply an optimization to any composed op, even those identified on the fly, those optimizations can be immediately used anywhere the composition op is. This allows us to prune branches out of the parse tree when optimizing quickly.

Any language for which a compiler exists to convert it to Lorito can be considered an "overlay" language for Lorito. We can then use any overlay language any place where we can use Lorito. For instance, if NQP is converted to output Lorito (and composition ops) directly, we can then use NQP to define all the PASM ops, all the core PMCs, and several parts of Parrot's core. That would be quite the turnaround from how NQP is used currently.

Conversely, almost any C code can be rewritten in Lorito, so long as we have a step in the build to preprocess Lorito back into valid C code. In fact, I see no reason why we cannot interleave code written in both languages, in a manner analogous to how the Q:PIR construct allows PIR code to be written in NQP source. If we had a preprocessor that scanned code files and converted Lorito ops into equivalent C code line-by-line, we could start slowly rewriting much of Parrot, if not all of it, in Lorito. Then, once we have converted all the code over that we need, we could change the build process to convert those code files to bytecode instead of machine code, and run large parts of Parrot's code through the runcore.

Alternatively, instead of writing core code in Lorito directly, we could write core code in any Lorito overlay language. NQP comes to mind here.

The huge benefit to be had is that if user programs are written in a Lorito overlay, and Parrot itself is written primarily in Lorito or an overlay, our eventual JIT can record traces across the API boundary, and optimize hot sections of the Parrot core on the fly at runtime.

Here ends the part of the blog post concerned with overarching architecture discussions. Let's get down to the nitty-gritty details.

Lorito is going to want some basic arithmetic operations: addition, subtraction, multiplication and division on integers (pointers) and floating point numbers. We're also going to want modulus, address-of (C &) and pointer dereference (C unary *) for integer/pointer types, int-to-float and float-to-int cast operations. Logical operations are probably essential as well: and, or, xor, not. Logical inversion (C unary !) would probably be necessary as well. Assuming we can find a way to share ops for int/pointer and float operands (which I doubt we can do in an elegant way), that's still about 20 ops we're going to want to even perform basic manipulations on numbers. I can't imagine any modern, robust, mature virtual machine which doesn't perform these operations readily.

On top of basic mathematical operations, we're going to need some important operations for calling subroutines: Passing and retrieving arguments, calling subroutines, returning with values. I don't know how low-level Lorito intends to get, but let's face the reality of the world: Hardware machines are stack-based. Every library function we want to call is probably going to be passing arguments on the system stack. If we want to be defiant and call these ops "pass_arg" and "retrieve_arg" to hide the fact that Parrot is using the system stack, that's fine by me. The "stacks are evil" mantra, while occasionally misguided, is definitely firmly ingrained in Parrot culture.

In a minimalist world, the only ops we would really need are ops to call functions with passed arguments. We could implement every other op as a huge library of functions to call. Of course this would be extremely sub-optimal, but it does open a new possible avenue: Any operation which is sufficiently uncommon could be turned into a function call instead of having a dedicated op. We could encapsulate the call in a composite op. There are lots of options here, we have to weigh the desire to have a smaller op set against the need to have ready access to a huge myriad of low-level operations.

Having ops to do basic ops is necessary but not sufficient. I don't think we can sit back on our laurels and be happy with an op set that only does a subset of what C can do. That's worthless. Parrot needs to provide access to it's PMC and STRING types, and also to its interpreter. We want ops to load registers with values from the constants table in the bytecode file. We want to treat PMCs and STRINGs as opaque pointers at the user-level. They aren't like other pointers which can be manipulated, they are GCable objects that Parrot treats specially. We don't, for instance, want to be passing a PMC pointer willy-nilly to an arithmetic operation. We also don't want such a mistake to imply a call to one of the addition VTABLEs.

That said, we want to be able to get the VTABLE structure from a PMC, introspect information about that, and be able to call the various VTABLE interface functions there without having to calculate pointer offsets. This might make a good use of composite ops, where we can still do the pointer derefs in Lorito, but hide the nonsense behind some PASM-levle composite ops. We already have most of these ops already, but I think we're going to want to rework some of them. We definitely need to radically reduce the number of VTABLE interface functions, or else interacting with all of them from Lorito will be extremely ungainly.

So far as we are talking about a major redesign of the system, maybe it's time to start considering an idea chromatic has been talking about with making all vtables into methods so we can share a common lookup mechanism, common dispatch mechanism, common MMD behaviors, common inheritance behaviors, etc. That's not a bad idea, but we either need dramatic improvements to PCC performance, or we need to create a PCC fast-path for calls which are made without constructing a call object and without doing too many marshalling operations to get arguments where they need to be.

In fact, on a tangentially-related topic, I definitely think PCC should have a fast-path for functions which do not participate in MMD and which have fixed argument lists containing only positional args (no :optional, no :slurpy, no :named, no :call_sig, etc). This is, I think, an extremely common case and there are plenty of optimizations to be had here if we want them.

To round out the low-level opset, we need ops to move data between registers, between registers and memory, and maybe even ops to move data between memory locations directly, without moving them to a register first.


That's my conception for Lorito: probably 64 ops or less, a capability to add composite ops, treating PASM as a series of common composite ops, and the ability to write low-level code interchangably in C or Lorito. I think this plan for it provides a great development path without too many abrupt stops or course changes. We are quickly going to find ourselves in desperate need of a robust, modern JIT. We are going to be able to move to a proper precise GC since all the Lorito code will be using Parrot's registers to store working values, not relying on stack space which will need to be traced.

I am highly interested in hearing what other people have to say about Lorito, and what changes Parrot should be considering so long as we are writing this blank check for a massive refactor.