Always The Critic

Equilibrium coding challenge - a Scala solution

Inspired by a random tweet about their added Scala support, I tried out the Codility sample test. I rather liked my solution and I think it's a perfect example of some of the niceties of Scala, in a small way, so here it is:

def solution(a: Array[Int]): Int = {
    // Partial sums are Longs to avoid Int overflow.
    val sums = a.scanLeft(0L)(_ + _).tail

    def equilibrium(index: Int) = leftSum(index) == rightSum(index)
    def leftSum(index: Int) = if (index == 0) 0 else sums(index - 1)
    def rightSum(index: Int) = sums.last - sums(index)

    (0 until a.length) find (equilibrium(_)) getOrElse(-1)
}

The use of scanLeft to build up all the partial sums is particularly handy. Having done that it's very easy to run through all the indexes until we find one that satisfies us. Note that the find method returns an Option[A] so we use getOrElse to return -1 if no solution was found (as per the requirements).

Windows 8 = CQRS

Allow me to wax philosphical for a moment with an observation about where computers and their operating systems are heading.

In the world of software development CQRS = Command Query Responsibility Segregation, which in its simplest sense recognises that it's sometimes better to use a different mechanism for reading data than it is for writing it. See Martin Fowler's exposition of the concept if you want to know more, but this post isn't actually about software development at all!

I reckon that we're at a critical juncture in the evolution of personal computing devices and that the CQRS principle is necessarily coming to the fore to save the human race.

Tablet computers are taking the world by storm, in case you hadn't noticed. Apple could barely make enough iPad Minis for me to be able to get my wife one for Xmas, though I did manage it at the very last minute, and shortly thereafter bagged one for myself too. Frankly it's bloody brilliant, but I use it predominantly for consuming rather than creating and I'm far from alone. This is partly because the human populace is inexorably dumbing down towards being fat blobs with brains wired directly into the 'net, consuming inane banter, amusing picture of cats and the latest celebrity news, 140 characters at a time. But that aside, it's just not very pleasant to write large quantities of text, manipulate images or perform other expansive creative works by prodding a tiny screen. Or even a big screen.

To write software, construct lengthy blog posts (ahem), edit movies, sequence the human genome or design great buildings requires a proper computer! On that basis I posit that there will always be a place for desktops and laptops, or indeed whatever replaces them but which necessarily has a non-trivial input mechanism. I genuinely worry that the market for serious computers will be increasingly neglected by the manufacturers, refocussing as they are on the mass consumer market, inevitably leading to the downfall of humankind. Perhaps I exaggerate - at least I hope so.

Now I've never used Windows 8, indeed I shudder at having to use Windows 7 on a daily basis at work, but I understand it represents something of a chimera. It is best known for its shiny, touchy, slidey 'Metro' UI, beckoning your greasy fingers to caress its tiles. However it also allows you to fall back into the more staid world of traditional Windows where presumably you can get some proper work done, as long as you have a keyboard and a pointing device other than your finger. I understand critics are conflicted about this hybrid approach, but it's CQRS writ large and may therefore be the way forwards. One way or another, at least some people will need to create great works. I do hope to be one of them, and to have the equipment to be able to do it.

Detail consultant

I see irritating problems. Every tangible thing I use, every service I consume, I can't help but notice all the little things that could have been a bit better. And it really frustrates me when these are details that seem so very simple to fix.

For instance, when I buy a train ticket from FCC's machines at St Pancras I can't actually see the text on the card payment screen unless I squat uncomfortably, because it's low down and inset out of sight. I know it's low down so disabled/short people can use it, but at least provide a direct line of sight for the average sized person standing in front of the machine - i.e. 95% of your customers!

I recently bought a D-Link powerline networking kit on Amazon, which is a wonderful product by the way, that worked straight out of the box with zero configuration. But the Amazon page did not state that there were two ethernet cables in the box - though I suspected there might be and was explicitly looking for this information. Worse than that, it provided a handy link to buy the product along with two separately supplied ethernet cables. So I went with that (better safe than sorry) but lo and behold: two cables in the box and now I have two extras I paid for and don't need. I feel I have been poorly served by Amazon.

Often when travelling by train, announcements are made that are inaudible due to the low volume (compared to background noise) or muffled by unhelpful acoustics. But these are sometimes really very important announcements and there's precious little point making them if they can't be heard.

Whenever I visit the hairdresser they ask me what I want doing, and I struggle to articulate it, especially when it comes to the intricacies of the back of my head. Why don't they take photos and notes (in special hairdresser shorthand) so that they can sort me out consistently every time, even when it's somebody new cutting my hair. As far as I'm concerned this is a no-brainer.

There are many little disappointments just like this that I spot on a daily basis, with physical and virtual products and services and I'm increasingly wondering whether there's a way to turn this into a job: detail consultant. Available for hire by companies small and large, I would tell them what could be improved to give their customers a delightful rather than frustrating experience. It's not rocket science but apparently there's a gap for this sort of common sense supplied from an external viewpoint. Please form an orderly queue!

In fact I'm surprised that larger organisations don't have full time employee roles with exactly this mandate: roving throughout the company and in the field spotting and fixing these subtle issues. In a competitive world where bad publicity is only a Tweet away, this seems like a very good use of resources to my naive mind. If you have any pride at all in what you're doing that is.

Apache Bench (ab) stalls after 16000 requests, due to ephemeral ports

I've been doing some trivial benchmarking of Play 2 with ab (Apache Bench) just to get an idea of its raw capabilities for serving simple requests - and because it's what I always do when picking up a new framework so I know what I'm dealing with. In doing so I ran into a bit of a puzzler that had me thinking Play 2 was bugged - but my spidey sense soon kicked in and told me it was more likely to be an OS or ab issue. I had done approximately the following, using Play 2.0.1 on OS X 10.7.3, and I'm pretty certain you'll see the same results if you do this on a Mac:

> play new hello  [select option 1 - basic Scala app]
> cd hello
> play start
> ab -c 50 -n 16000 http://localhost:9000/  [Runs fine - about 3700rps]
> ab -c 50 -n 16000 http://localhost:9000/  [Gives up with timeout]
> ab -c 50 -n 16000 http://localhost:9000/  [Runs fine - about 3700rps]
> ab -c 50 -n 16000 http://localhost:9000/  [Gives up with timeout]

It took me a bit of experimentation to establish that it's about 16000 requests that work fine, followed by timeouts, in a reliable pattern. That's a suspicious number, being near enough a power of 2, which is what clued me into it being an OS limit that I was running into. I ran the same ab test (with the same result) against the built in Apache https serving a static file, confirming that Play 2 probably wasn't to blame.

Sure enough, a quick Google turns up the goods. My OS was running out of the approximately 16000 ephemeral ports available and having to wait for them to be released before it could reuse them. So not Play 2 or ab's fault at all. Actually in some senses it is Play 2's fault for being so fast that I've run into this limit.

I'm not going to go into the details of what ephemeral ports really are, as others have done that perfectly well, and there is a good StackOverflow answer with some key ways to work around the problem by modifying parameters of the OS' network stack - but be careful and make sure you understand what you're doing.

However, one very simple way to workaround the issue is to simply pass the -k option to ab, to use HTTP keepalive (assuming the server you're testing supports it). Note that this changes the nature of your test though, as you're no longer really simulating large numbers of separate connections - but for basic sanity check testing it may help. For the record `ab -c 50 -n 100000 -k http://localhost:9000/` benchmarked Play 2 at about 7000 requests per second on my 2.4GHz Core Duo MacBook.

Getting my feet wet with Scala

The hype around the Scala programming language just got too much recently and I decided to give it a go. I'm two thirds of the way through Programming in Scala, a massive but very good tome and I've been experimenting with the language and tools, though only a little bit so far. I have also just received the newly published Scala for the Impatient (a deliberately much more compact book) and will be working through that as well.

To put it mildly, Scala is not for the faint hearted, or anyone who just wants to get some stuff done ASAP. There is a big, steep learning curve and the relative immaturity of the language and the small community means you'd better be used to the pains of the bleeding edge. I thought Ruby was hard work, but Scala is frankly more so in my experience so far.

However the language is somewhat addictive - or perhaps I just like the challenge of learning stuff that takes a bit of grokking. It's a big language, with many very clever facilities and features. It's mind-boggling to start with but I think it's starting to sink in. Of course there's the functional paradigm to understand and master, but I did plenty of that at university so it doesn't frighten me.

Enough people have written about the frustrations and wonders of writing code with Scala, so I think I will reflect on a few miscellaneous findings, mostly of a practical bent.

Compiler speed

The Scala compiler is surprisingly slow and it seems to be one of the main bitching points amongst people trying to get up to speed with the language. But seriously, when you're used to instantaneous results with Grails, or even plain old Java (especially with JRebel) then waiting several seconds even for a trivial app is most upsetting. But I'm trying to be Zen about it.

The mailing lists tend to be full of griping about the compiler's speed, usually met with promises that it's getting faster, but the Scala compiler is so much more complicated than the Java compiler that it seems unlikely to do its work in the blink of an eye anytime soon. See Martin Odersky's explanation of why it's slow on Stack Overflow.

Eclipse support

Eclipse supposedly has good Scala support via the Scala-IDE plugin, but I found the out of box experience so terrible that I have given up. Having created a "new Scala project" I still had to manually add the Scala libraries to the project and manually create run configurations, which would very often still not appear in the lists of ways to run the project. I just want to hit "Go" and for my app to run - it should be trivially easy to make this happen. In my experience the best open source projects are the ones that deliver a delightful out-of-box experience and this has disappointed me on that front.

IntelliJ IDEA support

Luckily the Community (i.e. free) edition of IDEA supports Scala. Even though I hadn't used IntelliJ previously, I figured out how to download their Scala plugin and create a Scala app within just a couple of minutes and it was a very smooth experience compared to installing the Scala support into Eclipse (don't get me started).

However compilation of a Hello World app took 7 seconds every time I modified the single file! A bit of Googling led me to discover FSC - the Fast Scala Compiler - which is part of the standard Scala toolset. Support for FSC is built-in to IDEA but that support is turned off by default. Once I turned it on for my project things hotted up and compilation took just 2 seconds. That's still lamentable compared to most other languages, but just about tolerable for now. I have no idea how things go for a big project, though I get the impression from mailing lists that waiting tens of seconds or even minutes for a compile is fairly commonplace. We shall see.

Play 2.0

One of the reasons I decided to try Scala in the first place was the fuss over the Play 2.0 framework, which has just recently been released in its first 'complete' form. I have only messed with a couple of tutorials so far and frankly it's bewildering and strange coming from frameworks like Spring MVC, Grails, and Ramaze (a small Ruby framework).

There is much wailing and gnashing of teeth on the Play mailing list from people upset about its radical new direction, and its emphasis on Scala compared to Play 1. Maybe it's just the pain of change, but there's no doubt Play 2 is hard work to get to grips with.

I get the impression that a lot of people are missing the point though - it's a case of horses for courses. Play 2 is rather exotically architected using cunning non-blocking approaches that require it to abandon the classic Java Servlet container entirely. It also requires you to write obtuse and sometimes verbose code (compared to Grails say - though it's usually less verbose than Java) and partly because of that non-blocking architecture. Reading and understanding the Play 2 docs on Action Composition may very well require a PhD, but Action Composition is a technique that must be used to achieve relatively common ends. It's all a bit overwhelming for newbies.

This clever shenanigans enables it to handle 40,000 requests per second (albeit very very simple requests) using less than 20MB RAM. I saw Guillame Bort demonstrate this at QCon and it's certainly impressive. The point though is that it's all architected for the sort of new-breed web app that's dealing with connected rich-clients rapidly pushing and pulling data. If you want to create a few CRUD pages for a small admin team to look after a database then I very much doubt that Play 2 is for you.

Grails custom exception mapping

Updated 17/3/2011 with a solution to Spring Security redirecting.

I had a bit of a fight against Grails to get security exceptions handled the way I wanted, but having figured it out I thought I'd write it up. It's really very simple - I had just had the wrong end of the stick.

I want to be able to throw an org.springframework.security.access.AccessDeniedException from anywhere in my code and have that appear to the user as a custom error page of my design, but with a 403 response code (that's 403 Forbidden, rather than 500 Internal Server Error) and maintaining the originally requested URL rather than redirecting to an error URL as Spring Security sometimes prefers to do (because then you can't refresh to retry).

My main confusion arose from the way Grails' UrlMappings.groovy handles custom exception mapping. It turns out that you need to do it like this:

// Handling specific exceptions requires a 500 code on the left for the
// mapping to pick them up, but we can send back another code in the
// controller that sends back the response.
"500"(controller:"error", action:"error403", exception:AccessDeniedException)

The only surprising thing is that you need a "500" code at the left rather than "403". The reasoning seems to be that this is the incoming error code to the mapping function, and all exceptions are deemed to represent a 500 error code at this point, i.e. they represent an internal server error, which on reflection isn't entirely unreasonable. To then send the error back to the user I have an ErrorController and an associated view in /views/error/error403.gsp.

@Secured(['permitAll'])
class ErrorController {
    def error403 = {
        // Ensure that the correct response code gets sent. If we don't do
        // this, it may send a 500 (internal server error) response because
        // of the way we had to configure the UrlMappings for handling specific
        // exception types with 500.
        return response.sendError(javax.servlet.http.HttpServletResponse.SC_FORBIDDEN)
    }
}

I also figured out how to stop Spring redirecting to a new URL when it decided to deny access. I'd rather the originally requested URL remain in the address bar. The trick is to put the following in Config.groovy.

grails.plugins.springsecurity.adh.errorPage = null

The major remaining annoyance is that these exceptions get reported in the logs whereas I'd rather they weren't (they represent the system working correctly) but I imagine I have to reconfigure logging to resolve that.

Grails dependency hell

I just upgraded to latest Spock and Geb releases (0.5 and 0.5.1 respectively) but have had to undergo a strange sort of hell of odd errors before getting back on track. Turned out none of these were due to any actual code incompatibilities and eventually (after a couple of hours of fruitless head scratching) I got past them by running "grails clean" (which got me to some new errors) and finally by deleting ~/ivy2/cache. The latter step sadly seems to be required quite often when adding or upgrading plugins.

Personally I'm bitterly disappointed that Ivy (or Maven, if that's your poison), the so-called solutions to dependency hell, seems to introduce its own dependency nightmare more often than not. Of course every time you delete that Ivy cache folder you have to wait fifteen minutes the next time you run the app, whilst it downloads hundreds of jars. If you're on the train and not connected to t'Internet, you're really stuck.

I'm sure one day I'll grok what's really going on with this stuff, but for now I feel it ought to just work, but it really doesn't, and many people must waste a lot of time (or give up entirely) as a result.

STS 2.5 RC1 "Refreshing external folders" error

I feverishly installed the latest Springsource Tool Suite release today - 2.5 RC1. Unfortunately it fairly swiftly started giving me the error shown below as I worked on my Grails project. It was popping up really frequently and would be a show-stopper if I couldn't fix it.

It became apparent that it does this every time I saved a Groovy file that required my running Grails app to reload. It seems that STS didn't like the files in the 'target' folder changing as it recompiled, especially since I have  auto-refresh turned on in STS. I worked around this by setting up a rule for my project to exclude the target folder.

Right-click on the project, selected "Properties..." then Resource > Resource Filters. Add a filter to exclude the folder "target".

Groovy and Grails thoughts

I've been slowly banging away at a small Grails app on the train to and from work, to teach myself Groovy the language and Grails the web framework. I already had a good familiarity with Java, Spring and Hibernate so it is mostly a case of learning what Groovy and Grails add on top of those. I've still got a long way to go, but I thought it worth jotting down some thoughts in case they help somebody else work their way into this particular world.

Overall, it's been fairly pleasant and I'm optimistic that it's probably as good a way to go about your average web app as any I've found before, and I've tried plenty over the past decade, including a few fairly obscure ones. I should point out that I don't believe in silver bullets any more. There are always going to be frustrations along the way when using any technology for a non-trivial project involving real-world customer requirements, but it's instructive to see just how many of those frustrations there are, and how hard they are to get past. I'm pleased to say that each time I've got a bit stuck with Grails I've been able to dig myself out fairly swiftly with the help of the existing documentation, other web resources, the #grails channel on freenode IRC and the grails-user mailing list. Furthermore, the principle of least surprise (POLS) is alive and well, meaning that when I hope that I'd find a particular feature to solve my problem du jour, it's usually there for me to find.

It's not all sunshine and roses though. For instance when I upgraded to Grails 1.3.3 my unit tests started failing with a strange and unhelpful stack trace emanating deep from within the guts of the plugin framework. Turns out this is a bug introduced in 1.3.3 that plenty of people have been caught out by. I've had to go back to 1.3.2 for now. It's a shade surprising and disappointing that a release could contain a bug like this, and that it's not been fixed and re-released yet. I worry that people trying Grails out for the first time with 1.3.3 will have a bad experience and give up early because of issues like this.

I had a struggle to find half decent tools. I've been a huge fan of Netbeans over the past several years and it claims to have great Groovy and Grails support, but I found it to be woeful I'm afraid. Common GSP tags completely confused the GSP editor (it claims the syntax is wrong) so I struggle to believe that those claiming it has this great support have actually tried developing a non-toy Grails app with it. I've settled on SpringSource Tool Suite (a bastardised Eclipse) which isn't bad - though as of this writing you definitely want to get the 2.3.3 M2 version for the latest Groovy and Grails support that's not yet in the stable release.

Grails GSP HTML escaping confusion

I'm currently getting to grips with Groovy and Grails. One major frustration that took a while to figure out was HTML escaping in GSP pages. Here's the simple lowdown, including at the end the important bit that I had struggled to realise, which is how to not escape specific strings when global HTML escaping is turned on.

Manual escaping

Out of the box (i.e. with a vanilla Grails app with default config) you need to explicitly escape any dangerous strings with the encodeAsHTML() function that Grails makes available on all Strings: ${dangerString.encodeAsHTML()}. This is a bit verbose, but at least it's very clear, and because it's just a method on String it's available everywhere in your app, not just in GSPs.

Auto-escaping with default codec

If you modify Config.groovy to contain grails.views.default.codec = "html" (which is there by default and set to "none") then it automatically calls encodeAsHTML() for you whenever you use ${} in GSPs. This is clearly quite a handy option and a much safer way of configuring things as it lessens the likelihood of slipping up and leaving a hole in your app.

Overriding auto-escaping per item

So far this is all exactly as per the Grails docs (which go into much more detail on codecs and what's really going on, including creating your own) but the crucial bit they fail to mention is what to do if you've turned on the global html codec, but have situations where you don't want escaping. The answer is to simply use the alternative JSP style interpolation syntax <%=mySafeHTMLString%> since the codec is only applied to ${}.

Overriding auto-escaping per page

You can also set the codec on a per-page basis, overriding that set in Config.groovy, with <%@page defaultCodec="html" %> or <%@page defaultCodec="none" %> as appropriate.