Stormin’ F#

Apache Storm is a scalable ‘stream computing’ platform that is fast gaining popularity. Hadoop and Storm can share the same cluster and the two complement each other well for different computing needs – batch for Hadoop and near-real-time for Storm.

Storm provides a macro architecture for executing ‘big data’ stream processing ‘topologies’. For example, one easily increase the parallelism of any node in the Storm topology to suit the performance requirements.

For streaming analytics, however, Storm does not offer much help out of the box. Often one has to write the needed analytic logic from scratch. Wouldn’t it be nice if one could use something like Reactive Extensions (Rx) within Storm components?

Luckily Nathan Marz – the original author of Storm – chose to enable Storm with multi-language support. While Storm itself is written in Clojure and Java, it implements a (relatively simple?) language-independent protocol that can be used with basically any mainstream language.

FsStorm is an attempt to allow F# (and .Net Rx) to be used for defining and running Storm topologies.

Storm and Rx are big topics in and of themselves. If your are interesting in leveraging FsStorm, it would be best to first understand Storm and Rx using the official and supplemental documentation (blog posts, videos, etc.). It took me a while to get a decent enough grasp of Storm to start putting together FsStorm. Be sure to view one of Nathan’s videos on Storm.

Given sufficient Storm knowledge, take a look at the sample project FsSimpleTest. It includes a topology a spout and a bolt. The topology is described with StormDSL (which IMHO makes defining topologies much simpler than in Java).

In order to run FsSimpleTest first setup a Storm cluster under Windows (for now). Download the repo and compile FsSimpleTest. To submit the topology to Storm, use the Submit.fsx script, included in the project but before you run the script make sure the ‘binDir’ variable is set correctly for your environment. Also the “jar” command – which comes with Java JDK – should be in the path (it is used to package the built components into a Jar file which is uploaded to the Storm cluster).

You should be able to see the running topology in Storm UI – the browser-based console.

While the repo does not include a sample that uses Rx (yet) I have been successfully running a 3-component Storm topology that leverages Fsharp.Control.Reactive (the Rx wrapper for F#).

FsStorm does not utilize all of the capabilities offered by Storm. In particular Distributed RPC and Transactional topologies are not supported.

This is a very early but promising start for stream computing with F#. Contributors are welcomed!

Android Wear Starter Project – F# + Xamarin

Wearable computing is undeniably a hot topic currently, what with all the major mobile players (Apple, Google and Microsoft) having entered the market. There is also a plethora of smaller players such as Timex, Nike, Pebble, etc.

Cheap, ubiquitous wearable computers – along with Internet-of-things – may be the next big wave in technology. As they say, “possibilities are endless…”.

As this blog is dedicated to F# and functional programming, here is a starter barebones Android Wear project for Xamarin – written entirely in F# : https://github.com/fwaris/WearableStarter

Please go through the repository documentation (“read.me” file). It will give you an overview of the Android Wear app deployment process. There you will also find a list of potential ‘gotchas’ to avoid and tips on debugging and troubleshooting.

VL – a F# DSL for the Layout of IOS Controls

Starting with IOS 6 Apple added Auto Layout – a language for specifying constraints on the layout of IOS controls. With Auto Layout one can express the height, width, sequence, spacing, margins, anchoring / docking, etc. of controls, as constraints.

Auto Layout constraints can be expressed in an untyped way as strings or one can use the underlying API. While feasible for Objective-C code, these approaches become awkward for other languages such as F#.

The VL DSL is meant to easily create IOS layouts in code in a strongly typed manner. The DSL makes use of F# Algebraic Data Types (Discriminated Unions), pattern matching and operator overloading. (Note that at the time of writing this post Xamarin has announced Xamarin.Forms which may make VL redundant).

Examples:
Auto Layout :       H:| -10-[ctrl1]-[ctrl2]-10-|
VL                :       H [ !- 10. ; @! ctrl1 ; sp ; @! ctrl2;  !- 10. ]

The two constraints are equivalent; they layout two controls horizontally with 10 point margins at the left and right edges and a standard space in between.

Continue reading

Xamarin F# – Experience Report

For the past few months I have been working on an Android application that collects automotive CAN bus data – ultimately for analysis with BigData technologies such as Hadoop. I have already described some aspects of this project here and here. Thus far close to 2 Terabytes of data has been already collected.

In this post I would like to recount my experience of using F# with Xamarin for a real-world mobile application.

Continue reading

OpenXC based Android App

Recently launched an OpenXC (http://www.openxcplatform.com) based app for internal use. The app connects to the OpenXC dongle and collects CAN data for analytics. It is interesting to note that daily capture is in 100 megabyte range (depending on the vehicle and how long it is driven). The app is based on the Xamarin platform and written entirely in F# (also see this post). Here are some screen shots:

Message Rate

Main

Message Frequency Histogram

Stats

Talking to your car – with OpenXC, Android, Xamarin & F#

Virtually all vehicles today operate an internal network called the CAN bus. Different modules (or ECUs – electronic control units) in the car communicate with each other via this network (e.g. engine, transmission, dashboard, etc.). In this post I reference a CAN bus reader built with F# and Xamarin.Android based on the OpenXC platform (http://openxcplatform.com/).

Update: Also see this post.

Continue reading