Blog Closed

This blog has moved to Github. This page will not be updated and is not open for comments. Please go to the new site for updated content.

Sunday, April 11, 2010

Online Parrot Developer Summit

For anybody who hasn't heard the news, the Parrot online developer summit is starting in about 10 minutes from the time I am writing this blog post. By the time this post circulates through various aggregators and feed readers, it will be start time.

Hope to see lots of people there today! Definitely make sure to stop by the IRC chatroom, even if you only want to lurk and not participate.

MediaWiki PeerReview Extension

Yesterday I put the code public for the new PeerReview extension I've been developing. Actually, I've only been completing and putting the finishing touches on the extension after a friend and colleague, Jason, wrote up the initial versions.

The request was simple. We want a way for students in a class using MediaWiki to be able to take ownership of a page and be graded based on it's contents. There is a certain disconnect here with the whole open-content philosophy, but this is only for the narrowly-defined classroom setting where we have a central authority (teacher) and a group of editors with particular assignments. This kind of extension would obviously not be used in an open wiki like Wikibooks, but it could definitely find a happy home in a private wiki used for a class or closed group.

The PeerReview extension has several components. The first, and most obvious, is a review form that appears at the top of reviewable pages. When a user logs in and goes to a reviewable page, the review form appears and the user can enter a review. The review consists of a single grade (letter or number, as appears in the DB) and a short comment.

The next component is the idea of a page "owner". A new special page and some database extensions allow a person with sufficient authority (a teacher or administrator) to assign ownership of a page to a student or a group of students. In this way reviews left on a page can be associated with the students who have ownership of that page.

The third, and currently final component of the extension is a simple dashboard-like interface where a user can view lists of reviews that they have given and received.

The PeerReview extension is functionally, in the sense that everything that is visible is operational and tested. However, it is still a little bit slim on features right now, and there is a large amount of new development needed to really bring this to the next level of utility and usability. Specifically:
  1. A mechanism to associate students together into class groups, with one or more teachers
  2. An interface for teachers to monitor the reviews given to/from students in their class, including a utility to aggregate reviews into average scores for each student
  3. An improved API with more flexibility and more configurability
  4. Improved aesthetics throughout
I am continuing development on this as I am able, but I would definitely like to solicit feedback and maybe even find people interested in contributing to the project.

Saturday, April 10, 2010

Parrot in C#

Here's a little bit of proof-of-concept code that is going to interest a few people:
using System;
using System.Text;
using System.Runtime.InteropServices;

public static class Program {
    private static extern System.IntPtr Parrot_new(System.IntPtr parent);

    private static extern void Parrot_destroy(System.IntPtr interp);

    [DllImport("parrot", CharSet=CharSet.Ansi)]
    private static extern System.IntPtr Parrot_new_string(System.IntPtr interp,
        string text, int length, System.IntPtr whatever, int whatever2);

    [DllImport("parrot", CharSet=CharSet.Ansi)]
    private static extern System.IntPtr Parrot_compile_string(System.IntPtr interp,
        System.IntPtr compiler_name, string code, StringBuilder errmsg);

    [DllImport("parrot", CharSet=CharSet.Ansi)]
    private static extern void Parrot_ext_call(System.IntPtr interp,
        System.IntPtr sub, string sig);

    public static void Main(string[] args) {
        Console.WriteLine("Hello World from C#");

        System.IntPtr interp = Parrot_new(System.IntPtr.Zero);
        System.IntPtr pir = Parrot_new_string(interp, "PIR", 3, System.IntPtr.Zero, 0);
        System.IntPtr sub = Parrot_compile_string(interp, pir, @"
            .sub 'foo'
                say 'Hello from PIR'
            new StringBuilder(256)

        Parrot_ext_call(interp, sub, "->");

        Console.WriteLine("Hello again from C#");
I've been kicking around the idea of starting a project to do full bindings between Parrot and .NET. I'm not sure that I'm ready to take on a new project, especially not one of that magnitude, but it's fun to think about.

I've got some more code examples for this that I'm putting together. Some of them can be pretty messy, especially when we talk about marshalling data between Parrot's various structures and C# code. However, I think it should be very possible to wrap all that messiness inside a series of nice, clean abstractions. It would be especially nice if we could get some scripts to parse through the embedding and extending function lists and generate the C# extern declarations automatically. With those, we can write some nice wrapper classes around the raw functions.

I'm sure I'll have more to write about this in the days ahead.

Friday, April 9, 2010

GSoC: The Contenders

The GSoC applications deadline came and went today at 19:00 UTC, and we have received quite an impressive crop of proposals from prospective students. I don't know how many slots Google will give to the Perl Foundation, and of those I don't know how many will be devoted to Parrot. However, even if only a few of the proposals we received get final approval it will be a great year for Parrot. Some students did submit multiple applications, and at least one student that I am aware of also submitted applications to other organizations, so even under the best circumstances we won't be getting all of the projects below approved and started.

Anyway, I would like to briefly describe the proposals we have received. I can't go into too much detail because we're entering a period of feedback and revisions, but I can give the gist of what has been proposed.

GMP Bindings for Parrot - Robert Kuo

Robert, better known on IRC by the handle "bubaflub" has been a Parrot contributor for a while now. He has submitted two proposals to Parrot, both of which look quite interesting. His first proposal, proper bindings for the venerable GMP library, is related closely to some of the rantings and ravings I've made previously on this very blog.

There are two premises behind this project:
  1. The BigInt and BigNum libraries don't really belong in the Parrot core distribution
  2. The GMP library has a large amount of power and utility that these two PMC types don't even begin to expose
Another big benefit here is that GMP bindings can be quite intensive for a new runtime, so if they are done properly in Parrot they could be made immediately available to all languages running on Parrot for free.

A PAST Optimization Framework for Parrot - Tyler Curtis

Tyler, "tcurtis" on IRC, is a Parrot newcomer. He originally showed some interest in my Immutable Strings project idea, but has since been steered in the direction of adding an optimization framework for PAST. Personally I'm happy he made the switch.

This is another project that I've discussed at length on my blog, and I'm very happy somebody has taken up the challenge of it. A robust and extendable optimization framework would allow other people to quickly and easily write new optimizations. The framework is much more important in this regard than any individual optimization is. Of course, the proposal is also sort of open-ended: Once the framework is in place, Tyler can devote his energy to adding new optimization stages to it from an endless list of possibilities. For this reason, this is one of those projects where we won't be asking "when will he finish?" , but instead we will ask "how much will he complete before time is up?".

The real power here is adding the optimization framework to PAST, which many (though not all!) of our compiler projects use. In this way, optimizations added once can be immediately used by several compilers. Write once, make faster everywhere. It's quite a compelling project for the performance-weary among us, and it looks like chromatic is quite keen to mentor this one if it gets accepted.

Hybrid Threads for Parrot - Nat Tuck

Threads are another huge issue in Parrot. Our current threading system, while technically "existing" is so messy, bug-ridden, and  lousy as to be essentially unusable by our various projects. I've said on a number of occasions that Parrot can't really call itself "Production Ready" as our 2.0 release tagline was supposed to be until it has the scalability and performance improvements that a robust threading system provides.

Nat Tuck, "Chandon" on IRC, was another person originally interested in Immutable Strings, but has since submitted his application to revamp the threading system in Parrot. It's hard for me to say any other project has the same potential for reward, but it's also hard to say that any other project matches this one in terms of complexity. It's ambitious, to say the least.

Threads are an extremely esoteric area of infrastructure development, similar to GC. When they work, they just work and nobody even thinks about them. Your software just "scales" the way people need it to. Plus, anybody can write threading code, but it can be extremely difficult to do it correctly. That's why Parrot getting it's part right is of utmost importance.

Nat's general idea is to use a small number of OS threads to host a large number of discrete tasks and virtual "green" threads transparently. He's also looking into fixing some of the bugs in cloning interpreters that stymie the current threading system and would make implementation of the new system more difficult.

Improvements to the NCI System and LLVM Stack Frame Builder - John Harrison

John Harrison, also known as IRC regular "ash_", has submitted an interesting and extremely pertinent project proposal for an improved NCI system. The title mentions LLVM specifically, but he's also looking at variants using libFFI. He wants to build a pluggable system with multiple available dynamic frame-builders, and Ahead-Of-Time compliation of frames to save runtime effort.

This project isn't as large or ambitious as some of the others, but in return comes higher probability that the system will be polished and thoroughly tested for robustness by the end of the summer. Massive improvements to the NCI system would also provide major benefits to projects writing library wrappers, like some of the other GSoC projects are doing.

Chance of success for this project is high, and there are clear benefits to Parrot in having this feature.

Implementing an Instrumentation Tool for Parrot VM - Muhd Khairul Syamil Hashim

This project came completely out of left field and is the only one that I never suggested on my blog ahead of time. It's also the only project proposal that came to us with working proof-of-concept code, and the results of that small example already have some of the Parrot developers imagining of all sorts of fancy long-term uses.

Muhd, "Khairul" on IRC, is planning a system of hooks and callbacks in Parrot that would allow one interpreter to attach to another interpreter at runtime and harvest information about the system as it executes. By adding the appropriate hooks and writing the correct handling routines, we could end up with all sorts of great tools, such as profilers, analyzers, debuggers, etc. And the best part is that we gain the ability to implement all these fun tools in PIR or even another HLL, not in C like some current tools use.

You don't have to think about it for long to understand the benefits of this project: Identifying code hotspots and bottlenecks, analyzing PMC usage patterns, tracking and mapping control flow, and the list goes on. We may even get to a point when we can replace Parrot's current debugging, tracing, and profiling runcores with this single tool and a variety of different PIR programs. Quite fun to think about.

NFG and Single-Representation for the Parrot VM - Daniel Arbelo Arrocha

Daniel, everbody's favorite "darbelo" from last year's GSoC fame, has proposed two projects this year. The first is an attempt to implement NFG, a new normalization form for Unicode text strings. The benefits for NFG are a perfect one-to-one mapping between a character and it's underlying representation. This in turn can lead to some significant simplifications in the code of Parrot, and some proposed speedups as well.

NFG is one of the few points in the Parrot design spec that has never even been prototyped in working code, that I am aware of. Everything else has at least been given prototype treatment, but NFG remains nowhere to be seen in the actual codebase.

Daniel's proposal is to implement NFG in a testbed so that it's performance can be compared to the current string system. If it is a benefit (and we all hope it is!) we can merge it into trunk and start a new era of strings in Parrot. If not, we can continue development on it until it becomes better, or even scrap it entirely.

Parrot on RTEMS - Daniel Arbelo Arrocha
Parrot on RTEMS - Robert Kuo

We had two submissions from prospective students with the exact same name. Both darbelo and bubflub have been inspired by dukeleto's work in the area and have proposed to work on the RTEMS port of Parrot. A major portion of this work will be improvements to the cross-compilation capability of our current configuration and build system, improvements which will have many benefits outside RTEMS. In addition, these two are proposing to help improve the operation of Parrot on RTEMS.

As far as I am aware, this submissions were both duplicated to the RTEMS people as well, so even if one of these proposals get accepted it's possible that the student will actually be working with an RTEMS mentor instead of a Parrot one.

In both cases, I think the alternate proposals for both students look a little more desirable (darbelo's NFG proposal, and bubaflub's GMP proposal), but I've never really used RTEMS so I'm probably biased. I wouldn't definitely not do any crying if one of these "Parrot on RTEMS" projects got accepted, however.

GSoC Deadline Approaching

darbelo: less than 9450 seconds to 'SOCKHOP' deadline. Everyone panic!
moritz panics 

Thursday, April 8, 2010

Writing Ops in PIR

I put together an interesting little code snippet a few days ago that, when finished, would allow writing ops in PIR. I am probably not going to complete work on this idea myself (at least, not any time soon), so I want to put the partially-written code up here for other people to look at and maybe run with.


opcode_t *
Parrot_pir_op_lib_wrapper_func(opcode_t *cur_opcode, PARROT_INTERP)
    // TODO: Set dest to the address of the next instruction, after all arguments.
    //       need to parse the number of args from the op signature
    opcode_t * dest = cur_opcode + 3;
    INTVAL opnum = (INTVAL)(*cur_opcode);
    char * const opname = interp->op_info_table[opnum]->name;
    STRING * name = string_make(interp, opname, 8, "ascii",
    PMC * const root_ns = interp->root_namespace;
    PMC * const ns = Parrot_get_namespace_keyed(interp, root_ns, CONST_STRING(interp, "pir_op_lib"));
    PMC * sub = VTABLE_get_pmc_keyed_str(interp, ns, name);

    // TODO: Need to convert the op signature into a PCC signature here
    PMC * call_sig = Parrot_pcc_build_sig_object_from_op(interp, PMCNULL, signature,
    Parrot_pcc_set_signature(interp, CURRENT_CONTEXT(interp), call_sig);
    interp->current_cont   = NEED_CONTINUATION;

    dest = VTABLE_invoke(interp, sub, dest);
    goto ADDRESS(dest);


op register_op_sub(IN STR, IN STR, IN PMC) {
    PMC * const root_ns = interp->root_namespace;
    PMC * const sub = $3;
    PMC * const key = Parrot_pmc_new(interp, enum_class_ResizableStringArray);
    STRING * const opname = $1;
    STRING * const sig = $2;
    PMC * const ns = Parrot_get_namespace_keyed(interp, root_ns, CONST_STRING(interp, "pir_op_lib"));
    INTVAL argcount = 3; // TODO: Count number of args in signature
    STRING * const separator = CONST_STRING(interp, "_");
    STRING * longname = Parrot_str_append(interp, opname, separator);
    longname = Parrot_str_append(interp, longname, sig);

    VTABLE_set_string_keyed_int(interp, key, 0, );

    VTABLE_set_pmc_keyed_str(interp, ns, $2, sub);
    op_info_t * new_op = (op_info_t *)malloc(sizeof(op_info_t));
    new_op->name = Parrot_str_to_cstring(interp, opname);
    new_op->full_name = Parrot_str_to_cstring(interp, longname);
    new_op->jump = 0;
    new_op->opcount = argcount + 1;

    // TODO: Initialize ->type based on what's in signature
    new_op->type = (arg_type_t *)malloc(sizeof(arg_type_t) * PARROT_MAX_ARGS);

    // TODO: Initialize based on the signature
    new_op->dirs = (arg_dir_t *)malloc(sizeof(arg_dir_t) * PARROT_MAX_ARGS);

    // TODO: what is this?
    new_op->labels = (char *)malloc(sizeof(char) * PARROT_MAX_ARGS);

    // TODO: Insert the new op into the optable.

This code, intended to become a dynop library one day, introduces a single new op: register_op_sub. register_op_sub takes three arguments: the string name of the new op, the string signature of the new op, and the Sub PMC for the new op. An entry is prepared for the interpreter's op_lib table (I haven't found a good way to insert that record yet), and the sub is stored in the "pir_op_lib" namespace. The body of the new op is the function Parrot_pir_op_lib_wrapper. This function body looks up the name of the current op in the op_lib table, then looks up the Sub PMC in the namespace, and will attempt to invoke that sub.

I haven't worked out all the mechanics of everything, obviously. But once a few more questions get answered and a modest amount of code gets written, this could turn into a very useful and very cool extension library for Parrot.

Wednesday, April 7, 2010

Parrot Virtual Developer Meeting

Allison sent a message to the list today to schedule a virtual developer meeting for all Parrot developers. The meeting will be this Sunday, 11 April, at 7:30 UTC.

All Parrot devs should try to attend. All prospective Parrot devs and well-wishers (GSoC students?) should either try to watch or read the logs afterwards. It will definitely be extremely interesting stuff.

I hope to see lots of people there, and I hope we get some good planning work accomplished!

GSoC Proposal Deadline: Friday


The deadline for proposals is this Friday at 19:00 UTC. Please submit your proposals soon, even if they are not complete: you will get feedback and opportunity to revise and update the proposals. If you have multiple projects and cannot decide which to submit, submit all of them and we will weed them out later.

Whatever you do, make sure to submit your applications before the deadline or you will be disqualified!!!

Tuesday, April 6, 2010

Problems with C#

I've been using C# at my day job for some time now. When I first started there the place was a little bit disorganized and different projects were using different languages. I did some work in VisualBasic.NET, Perl, C++, and even a little Tcl. Luckily as the place grew we put together some standards, and now all our PC-side and server-side development is done in C#.

Yes, I am deeply involved in Parrot, the dynamic language VM. Yes, C# is hardly a dynamic language (even with some of the new "dynamic" features that have been added in newer revisions). It seems little bit weird that I would be so enamoured with C#, but I am. In fact, I would venture to say that for Windows systems it is my development language of choice. I've heard decent things about Mono on Linux but never used it myself. On Linux, I think my development language of choice may still be C (or NQP!).

However, despite my general happiness with the language there are some problems I have with it. Here are my nits with C#:


I feel like constructors aren't given a fair shake, and I have a few issues with their use. For instance, their use with generics is positively anemic:
public void Foo<T>() where T : new() {
   T bar = new T(new Baz());
The constaint "Where T : new()" only mandates that the type T has any constructor, but in my Foo method I specifically require it to have a constructor that takes a Baz. Generics are supposed to provide typesafety for cases like this, but it's extremely easy to compile a program with this kind of construct and not have any compile-time indication that the types are incompatible.

And while we're at it, interfaces should be able to specify a list of required constructors. I'm constantly baffled by this omission, I run into problems from it on a regular basis, and I can't imagine why it wouldn't be possible.

Static Methods in Interfaces

Speaking of interfaces, for the life of me I can't understand why interfaces cannot be used to specify static class methods and properties. I've seen dozens of half-hearted "explanations" for why we don't have this ability, and none of them compel me at all.

Let me give an example from code I was working on this very day. I'm putting together an MDI application which can host a heterogenous collection of child forms. Different form types have different properties: Some are multiplicious. Some are "singletons", where only one child window of that type can open at a time. Some windows should start maximized, and some minimized. Some should open at program startup. Some of this information I need to have from the child form before I instantiate it, so it would be really nice to have a static method on the type that I can call to get this info. Basically, I want to do this:
private method ShowMDIChild<TForm>() where TForm : Form {
  if (IsSingleton<TForm>() && this.SingletonCache.ContainsKey(typeof(TForm)))
  TForm form = new TForm(...);
The "..." above really is a variable list of arguments, since different types of child forms may want different argument lists to be created. Sure, I could have a large common list of arguments and force all my forms to implement the same constructor with some of the arguments treated as unused shims, but thats a huge hassle (and I can't specify that constructor in a common interface).

One thing I can do, which is particularly ugly in my mind, is this:
public interface ISingletonMDIChild {}


private method ShowMDIChild<TForm>() {
  if (typeof(TForm).GetInterface(typeof(ISingletonMDIChild).FullName) == null) {
And I think we can all agree that this is a hideous mess. Plus, it would require me to write up all sorts of stub interfaces and treat them like boolean class flags. I could use attributes, but the code to reflect over the class and iterate over attributes is even uglier (and much slower). We also can't always inherit from an abstract parent class because of C#'s restriction to single inheritance. What I want, in a nutshell, is a way to gather a standard set of metadata about classes of a related type in a clean way. I can't think of a better way to do this than static interface methods. I haven't heard any compelling arguments about why we can't do this, but I have heard a lot of half-hearted justifications for it.

Multiple Inheritance.

Yes, it's messy. Sure the implementation would be a little bit messy and the coder would probably have to specify the resolution order manually. I don't care. I find so many cases in my day-to-day coding when judicious use of multiple inheritance would be a perfect solution to a problem, and the alternatives are hackish, ugly, and very difficult to maintain.

Type Switches

Consider the case of a tree control which contains a number of objects of different types. Each TreeNode in the tree contains a reference to that object in the TreeNode.Tag property. When I right-click on different types of nodes I want to see different context menus pop up, depending on the type of the object at that node. That context menu should call methods on the parent form when clicked.

One thing I could do is this:
ContextMenu menu = currNode.Tag.GetContextMenu();
But this solution requires the object to build the menu programmatically. To call methods on the parent form I need to do something ugly like pass a variable list of delegates to the object as callbacks, or pass a reference to the Form and make all my methods public, or something like that. Now consider the case where some of the nodes in the list don't have context menus, and some of them call a default version, and we quickly end up with very messy code. What I would really like to do is something like this:
switch (currNode.Tag is) {
  case ClassFoo:
case ClassBar:
I guess I could do:
switch (currNode.Tag.GetType().FullName) {
  case "MyNamespace.ClassFoo":
But that's horrible and unmaintainable. Plus, it doesn't respect inheritance. I could do this:
if (currNode.Tag is ClassFoo) {
} else if (currNode.Tag is ClassBar) {
But that's horrible too. On the bright side this one does respect inheritance, but then again it is harder to maintain.

An ability to do a switch statement, which lends itself naturally to multiple alternatives, with object type would be a huge benefit.

Switch Case Fallthrough

And on the subject of the venerable switch statement, it kills me a little bit inside every time I try to implicitly fall through from one case to another and I get a compiler error about it. One forum post I saw actually suggested this gem to workaround the restriction:
switch (whatever) {
  case 1:
  goto _2;
  case 2:
Gag me with a stick, and deliver me from evil.

This is a pretty small list of wishes and, except for the multiple inheritance thing, don't represent huge changes to the language. If anything I think they will make a few tasks much easier. You can complain about my contrived examples if you want.