Limitations of D for Academic Research

D could have major appeal to academic researchers needing to do numerical computing. The alternative languages I have in mind are R, Python, Matlab/Octave, and Julia. D has some self-imposed limitations that make it a harder sell to academic researchers than should be the case.

Enterprise adoption is the clear goal of the D Language Foundation. Academic research requires a completely different feature set: it should be easy to learn, straightforward to write and read the code, produce correct results by default (even if that comes at the expense of speed), and be fast enough for the things the researcher needs to do. Speed has definitely become more important recently due to large datasets and the heavier use of computationally-intensive methods such as Bayesian inference. That’s clearly an advantage for D.

The experienced D user might be thinking that D is the perfect language for that type of thing. They’d kind of be right, but there are valid reasons it hasn’t seen massive adoption in that sector. Here are four, in no particular order:

• Limited operator overloading. Walter Bright’s position is that operator overloading can be abused. Naturally, as with every programming feature ever, that’s true. There are some serious downsides to declaring operator overloading bad practice. It only makes sense to restrict operator overloading if you’re sure the base language provides enough operators to do what needs doing. The worst thing you can possible do is say “those are the operators I need, so that’s what’s available”.
  • A prominent example is matrix multiplication. The first mistake is thinking function calls are sufficient. This is pretty ugly: matmul(m1, m2) as is m2.matmul(m2). It’s really ugly when there are several mathematical operations being done at the same time. You need matrix multiplication and element-by-element matrix multiplication. Matlab and Julia provide * for the former and .* for the latter. With R, it’s * for element-by-element and %*% for regular. D does not provide sensible operators to represent both types of matrix multiplication. In this case, restricting operator overloading is not preventing abuse, it’s mandating ugly code with a lower probability of being correct. That’s not an indicator of good language design.
  • Another issue is sequences, whether we’re talking about 2..5 or [2..5]. iota(2, 5) wouldn’t be the end of the world if it were part of the language. It’s a PITA to have to do an import for something so fundamental.
  • A third example is grabbing all rows or columns of a matrix/array. Better syntax would be x[,3..7], x[:,3..7], x[..,3..7], or something similar. This one’s not terrible. You can currently use x[0..$,3..7]. It’s not as nice, but it’s not the ugliest or most inconvenient thing either.
  • A final example is the inability to overload the comparison operators. This one hits me particularly hard because of the data I work with. An annual data series has dates that are naturally integers. A monthly data series has dates that are naturally two integers. I use a struct to represent dates. That allows writing things like date - 3 to be three months earlier than the given date. The restricted operator overloading means I cannot do comparisons of this type: date >= [2012,4]. Instead I have to write date.notBefore([2012,4]), which is far from elegant.
• dub.json needs to disappear. [I’ve written more about this here]. This one’s awful for academic researchers. Create a project to do a few calculations? Create a configuration file in JSON? To calculate descriptive statistics? Dub should be used to install packages and hold information about the packages (dependencies and such) but there’s no reason for users to write a dub.json file. Make -i the default. Look in the current directory and in the package installation directory/directories.
• Windows support for calling dynamic libraries is messy. Import libraries have given me nothing but trouble. Manually importing functions from DLLs on Windows has worked well. What doesn’t make much sense is why I have to do that. I’ve written functions that handle the boilerplate. With ImportC, you should be able to feed the compiler some C function declarations and have it create the necessary bindings for you. Even better would be using Dub to install a package that creates the bindings for you and stores the necessary metadata in the configuration file. Someone using R would never have to mess with any of this - it’s all handled by the package management system to the point that most Windows users have never heard of a DLL. There’s no reason we can’t have the same experience.
• The hassle of creating dynamic libraries called by other languages. The compiler is very easily able to make this easy - I know, because I’ve already done it with metaprogramming. What I have in mind is a set of extern(*) declarations: extern(R), extern(Matlab), etc. All you need to do is write a wrapper function that converts the arguments of a D function into types that can be understood by the other language. When I say I’ve done it, I mean I wrote a library that takes a D function with signature double foo(double[] x, int[] y) and creates the function extern(C) SEXP foo(SEXP x, SEXP y). The .so or .dll can be loaded by R and called like any other R function. This would be a killer feature for D.

I don’t view these as particularly difficult to implement:

• Change 1 would require adding new operators to the language.
• Change 2 would require scanning the file for import statements and constructing an appropriate compilation command.
• Change 3 would require more thought, but I’ve already been doing most of this using D’s metaprogramming.
• Change 4 is something I’ve already done for R. All that’s needed is to convert extern(R) declarations to @extern_R and add a mixin statement. Other languages would require figuring out interop.

True minimal theme