Category Archives: Software

State of JVM languages

Java is still the top dog on JVM, but there are plenty of alternatives for those programmers who a looking for a change, but how valid are these options? It’s hard to make a case for learning a fringe language apart from personal enjoyment, but do these alternate languages offer valid career paths. To do so, what you need is a critical mass of developers using it to motivate tool makers to create proper tools for those languages. When there are users, then libraries will follow.

I selected tree of the most talked jvm languages apart from Java which are: Scala, Clojure and Kotlin.

Scala is the oldest of these three being released already in 2004. Clojure followed in 2007 and Kotlin being the most recent of these languages being unveiled in 2011 and reaching version 1 early 2016.
 

New Interest

Past 12 Months:

Screen Shot 2017-10-25 at 10.40.59 PM
Blue: Scala tutorial, yellow: Kotlin tutorial, Red: Clojure tutorial

Link to most recent graph.

Past 5 years:

Screen Shot 2017-10-25 at 10.43.49 PM
Link

Scala seems still the most interesting to newcomers. Kotlin popularity clearly spiked mid 2017, but the hype has slowed down a bit since.

Job Market

How useful are these languages in the job market.

LinkedIn Job Search:

software engineer scala Showing 5,540 results
software engineer clojure => Showing 684 results
software engineer kotlin => Showing 586 results

engineer scala => Showing 7,701 results
engineer clojure => Showing 778 results
engineer kotlin => Showing 433 results

data scala => Showing 10,076 results
data clojure => Showing 758 results
data kotlin => Showing 254 results

Based on LinkedIn Worldwide job search Scala is mentioned in roughly 10 times more job adds than Clojure. Kotlin seems to be catching on quite quickly, apparently being officially supported by Android drives adoption. I would be surprised if it did not take over Clojure in popularity in the next 6 months.

Scala benefits from growing data science/engineering market as it’s one of the most important languages in that domain alongside Python. Quite a few data processing tools(Spark, Kafka) are written in scala making it the most natural fit for

Salaries

How well do these salaries pay:
It’s hard to find reliable data on how jobs in given languages pay. Googling around I found this article:
https://gooroo.io/GoorooTHINK/Article/16300/Programming-languages–salaries-and-demand-May-2015/18672#.WfEECxOCzXE
Looks like clojure pays pretty well and clearly better than scala. Both however pay clearly better than numerous java or javascript jobs. Jobs that require functional language knowledge are still fairly few, but if you manage to land one, you will be pretty well compensated.

Advertisements

Why Scala?

scala

Recently we started a new project and I’m happy to say we had quite a lot of freedom to choose the tech stack we wanted to implemented it.

Technically the project did not seem too difficult, basically just aggregating data from few different web service api:s. So it looked like a good chance to take a small chance and try something new. We decided to go with Scala and Play Framework and here is why:

Reactive Model

I had previously done some smaller project using node.js and event driven programming. I definitely think that “reactive” is the way to go and makes sense to learn to do it properly. The thing I was missing was a proper type system which leads to…

Static Typing

Scala’s type system is extremely powerful and type inference allows some compact and concise code. Read more here:

Functional

After (too ) many years of mostly Java development it was definitely time for something more powerful. Scala is pretty much as functional as a language can be. Some purist may argue that it’s not purely functional like Haskell, but in the real world situations Scala is as functional as they come.

Specifically Pattern Matching deserves to be mentioned as one of my favourite features.If you manage to specify most of your data model in case classes, life gets a lot easier.

Cake Pattern

Cake pattern is seems to be the go-to way of wiring Scala apps together. It basically allows you to do modular design and dependency injection without using any library. It does include a bit of boilerplate, but I think that the advantages of using statically typed, compile time checked dependency injection is better than using any separate library even with the price of a little bit of boilerplate. Read more about cake pattern here:

Java Interoperability

If Java has one strength, it’s the plenitude of well tested libraries. Using Java lib’s is trivially easy from Scala.

Maturity

Scala recently turned 10 years old and the language is definitely mature enough. It still evolves, but latest stable releases are worthy of their name and stable.

Play Framework has reach version 2.3.7 and accompanying Activator makes starting projects very easy. Activator has pretty decent template mechanism and you got bunch of templates to choose from when you start a new project.

Sbt the Scala Build Tool has evolved like Scala. It’s regularly updated and has a working plugin system. It comes with a nice REPL. It might not look fancy, but it get’s the job done.

When it comes to IDE:s you got basically two fine choices: Eclipse based Scala IDE and IntelliJ. I personally found IntelliJ:s scala & play plugin to work better and eventually settled on that. The only downside is that play plugin requires the registered (paid) version.

Performance

Scala compiles to java bytecode so the performance is just as good. Static typing allows the compiler to optimize better. Just have a quick look at these benchmarks: http://benchmarksgame.alioth.debian.org/u32/compare.php?lang=scala&lang2=clojure
I chose to compare Scala to another modern jvm language with dynamic typing. Of course this is just a little sample, but Scala is across the board faster. It good to keep in mind that usually performance should be one of the last criterias when selecting the language, but it’s nice to know that when push comes to shove, Scala will deliver. There is a reason why internet giants like Twitter and LinkedIn chose Scala.

Summary

There you have, our reasoning for choosing Scala. After about 4 months into the project it still looks like a good choice. Don’t get me wrong, it has not been a walk in the park and we’ve had some difficulties and problems, but that’s the topic of an upcoming post.

JSON values to typed Id:s in Scala & Play

I recently published a post about how to deal with JSON objects using Scala’s case classes and Play Framework. To keep up with the theme here is another post about the same topic, but this time it’s specifically about types.

JSON format does have types, but as they are not visible in the actual content and how they are mapped to data types at the receiving end depends a lot on developers first look at the incoming data. Majority of content seems to be in String format and that seems like a safe choice, after all you can fairly safely represent a number as a String as long as you don’t process the data in any meaningful form. You would not be so lucky doing it the other way around.

This logic seems pretty valid especially if you are not certain about the format of the data. Maybe the value just happened to be a number this time, but it might include a letter next time and then using anything but String would result in a runtime exception and we definitely don’t want that. Going all-in with String does have some unfortunate side effects and some problems might be creeping into your code.

def doIt(someId: String, someOtherId: String, foo: String, bar: String)

It’s really easy to mix up the parameter order when you have methods like his and what’s the point of having static typing if you deal mostly with strings anyway. In the worst case it will “kind of” work, but the results are wrong. Moreover code like this is a pain to refactor.

So instead of a mess like this wouldn’t it be nice to deal with properly typed values instead? (From here on I concentrate more on id -values, if you need to pass on many values you probably have other design flaws as well)

def doIt(someId: SomeId, someOtherId: SomeOtherId, foo: String, bar: String)

Now we have also regained the ability to trust the developer’s best friend – the compiler. Of course there is nothing new or fancy about wrapping values to classes, but what turned out to be tricky was to maintain the handy JSON-parsing that comes with Play Framework and case classes without too much boilerplate. So, how do we actually achieve this?

First solution: Implicit conversion to typed Id -classes

Well first of all we implemented a proper base trait representing any typed id case class which we will declare later on.

trait BaseId[V] {  val value: V  }

Based on that basic trait we can then implement another trait for every value type we want to support.

trait StringBaseId extends BaseId[String]
trait NumberBaseId extends BaseId[BigDecimal]

What we now need is an implicit conversion of the JavaScript’s primitive type to an instance of a typed id implementation. We do this using an implicit class for every primitive type we support. For example String -based ids we implement like this:

implicit class StringTypedIdFormat[I <: BaseId[String]](factory: Factory[String, I]) 
    extends Format[I] {
  def reads(json: JsValue): JsResult[I] = json match {
    case JsString(value) => JsSuccess(factory(value))
    case _ => JsError(s"Unexpected JSON value $json")
  }
  def writes(id: I): JsValue = JsString(id.value)
}

The provided factory will be used to instantiate a concrete implementation based on a String. The type factory is declared as follows:

type Factory[V, I <: BaseId[V]] = V => I

In the next step we declare concrete id -classes for each id type we need. Staying consistent to our requested method signature of the previous example we write:

case class SomeId(value: String) extends StringBaseId
case class SomeOtherId(value: BigDecimal) extends NumberBaseId

Both case classes extend from our base corresponding base traits. The case class representing the JSON’s would look like this:

case class SomeObject(id:SomeId, name:String)
case class SomeOtherObject(id:SomeOtherId, name:String, value:Number)

To get an implicit conversion between JSON and those two case classes we must provide implicit read and write -functions or JsonCombinator formats like (https://www.playframework.com/documentation/2.3.x/ScalaJsonCombinators)

implicit val someObjectFormat: Format[SomeObject] = 
  Json.format[SomeObject]
implicit val someOtherObjectFormat: Format[SomeOtherObject] = 
  Json.format[SomeOtherObject]

This format wouldn’t yet work because we still need an implicit conversion between JSON and the typed id case classes. We can finally provide those format using the helper classes previously declare. For the two ids we declare:

implicit val someIdFormat: Format[SomeId] = 
  new StringTypedIdFormat[SomeId](SomeId.apply _)
implicit val someOtherIdFormat: Format[SomeOtherId] = 
  new NumberTypedIdFormat[SomeOtherId](SomeOtherId.apply _)

The default apply method of the id case classes can be used directly as the declared factory method to generate the concrete instance of the case class. We can then implicitly convert between JSON primitive type id values and our typed id case classes in Scala.

A simple test to demonstrate the conversion:

val someObjectAsJson: JsValue = Json.parse("""
  {
    "id":"111",
    "name": "someName"
  }
""")

"Parsing generic id object" should {
   "SomeId will be parsed correctly" in {
     val test = someObjectAsJson.as[SomeObject]
     test.id === SomeId("111", “someName”)
   }
}

We can further simplify the id format declaration by adding an additional method to the Scala’s Json object:

object TypedId {

  //implicit convertion to extended json object
  implicit def fromJson(json: Json.type) = TypedId

  //extended format function
  def idformat[I <: StringBaseId](fact: Factory[String, I]) = 
    new StringTypedIdFormat[I](fact)
  def idformat[I <: NumberBaseId](fact: Factory[BigDecimal, I]) = 
    new NumberTypedIdFormat[I](fact)
}

Now we can replace the format declaration with the following simpler version:

implicit val someIdFormat: Format[SomeId] = 
  Json.idformat[SomeId](SomeId.apply _)
implicit val someOtherIdFormat: Format[SomeOtherId] = 
  Json.idformat[SomeOtherId](SomeOtherId.apply _)

We still need to declare the factory method because we can’t instantiate a typed class at runtime.

We are not completely happy with this solution because with the current solution we would have to declare a concrete case class as well as an implicit format for every id class we create. We are looking for a more generic way to declare such id’s and it could look something like this:

def doIt(someId: StringId[SomeObject], someOtherId: NumberId[SomeOtherObject], foo: String, bar: String)

Bye the way:
All the base implementations can be found on GitHub.

Mike Toggweiler, a partner @ Tegonal co-authored this post.

Parsing json with over 22 fields with case classes

We recently started to develop a new product using play framework in the back end and angular js for the client side. The back end is rather light and mainly consist of aggregating data from different web services. We do have a local database and for that we chose MongoDB. So what we have is many sources of data and pretty much all of them provide data in json format. Needless to say json parsing has to work flawlessly.

For the most part that worked great. With play you can create a case class with all the same fields as the corresponding json and then you just create a format for it in the companion class. This is easy and elegant and works great… until…

…until you notice that one of the web services you are calling has more than 22 fields and is a problem because Scala does not allow more than 22 fields in a case class. Normally that’s fine and usually that is a red flag about poor design, but sometimes the web services you use have more than 22 fields in the response objects.

case class FooBar(
   field01: String,
   field02: String,
   field03: String,
// ... many fields ...
   field22: String,
   field23: String)

// compilation errors

object FooBar{
   implicit val foobarFormat: Format[FooBar] = Json.format[FooBar]
}

Luckily there is pretty neat way around this and it might even improve your design. You can use nested case classes and all you need to do is implement a wrapper writer and reader for that class, but first you can create case classes that represent subset of the fields of the response.

case class Foo(
   field01: String,
   field02: String,
// ... many fields ...
   field09: String,
   field10: String)

object Foo{
   implicit val fooFormat: Format[Foo] = Json.format[Foo]
}

case class Bar(
   field11: String,
   field12: String,
// ... many fields ...
   field21: String,
   field22: String,
   field23: String)

object Bar{
   implicit val barFormat: Format[Bar] = Json.format[Bar]
}

So now we have smaller classes that contains subset of the original fields and the only thing missing is a “wrapper” class the represents the complete class. This is rather simple and in this case it has two fields, namely the two classes I just defined.

case class FooBar(
   foo: Foo,
   bar: Bar)

object FooBar {
   implicit val foobarReads: Reads[FooBar] = (
      (JsPath).read[Foo] and
      (JsPath).read[Bar])(FooBar.apply _)

   implicit val foobarWrites: Writes[FooBar] = (
      (JsPath).write[Foo] and
      (JsPath).write[Bar])(unlift(FooBar.unapply))
}

Now in scala you can access fields quite neatly using the dot -notion, but the json is serialized back to the original format. This parsing can be easily tested.

val foobarJson: JsValue = Json.parse("""
  { "field1":"value1", 
    "field2":"value2", 
    ...
    "field22":"value22", 
    "field23":"value23"}
  """)

"Parsing json to nested case classes" should {
  "work just fine" in {
    val foobar = foobarJsonAsString.as[FooBar]
    foobar.foo.field1 === "value1"
// ...
    foobar.bar.field23 === "value23"
  }
}

Hunting for talent

No matter what line of business you have, you always want to have good people. People who can make things happen, who are willing push for solution and questions that you didn’t even think in the first place.

I don’t have fixed list of things I look for, I rely heavily on my intuition how I feel about a candidate,  but there are some things I can list and that is what I will try to do here.

Attitude

I want people that I want to work with. I want people who can contribute to a good atmosphere where people feel comfortable to talk out, provide feedback and ask questions. They should be more of a catalyst than a handbrake for communication. This social matchup is essential.

Secondly the right kind of people tend to do stuff on their own, they contribute to the community. They have github accounts, they participate in user group meetups, they don’t limit themselves to one language. They read books and blogs, they follow interesting people on Twitter. They have opinions and are not afraid to share the. They are curious. Curiosity is a virtue for software professional.

Technical skills and understanding

What separates talent from the rest and this is rather clear even in more junior candidates is that they want to understand how something works. It’s generally not enough to read tutorials and make slight modifications. Good people tend to know know also how things work beneath all the magic of Django or Rails etc.

How do I do it

First would be just talk where I try to scan how well the candidate is up to speed where things are in the software world and how well he/she understand the concepts. I always try to crack a joke or two to relieve the atmosphere. An interview can be a nerve wracking experience especially for younger people. If I can get the candidate to relax I’ll get a better read what they are all about. Before going on to the exercises I make sure the topic what is considered “good code” is raised.

Having the candidate bring over a code he has made and having a chat about is always good. Specially if it’s rather old then questions like “what would you do differently?” or “what do you think about your code now?” make for good conversation.

Then I give the candidate a small piece of fairly poorly written java code and set the question in a way that an intern or trainee would ask for feedback for that method. Good candidates tend to find bunch of things wrong quickly and the best ones also know the difference what is clearly wrong and what is more of a matter of opinion.

It’s time to walk the walk

What happens depends largely on my feel of the candidate so far. With juniors I go for a pair programming exercise where I sit with them and do something like a simple coding kata to see what it would be like to work with them. How they react to feedback and suggestions and especially what kind of suggestions they have.

For more senior candidates  I give them a existing piece of (crappy) software and ask them to implement a feature. The software is of course poorly tested bug ridden mess. Good engineers quickly see what a mess and illogical piece of crap the software is and don’t waste time refactoring most of the crap before trying to implement a then trivial new feature.

If I’m not quite sure or how it went I still do the pair programming stuff I usually do with the more junior candidates. Sometimes the refactoring exercise does take a bit too long and the setup might not be familiar to some candidates (wrong keyboard, wrong IDE etc), but when we work on a small project together I can easily guide and help through the most common simple downfalls like shortcuts etc.

I tend to make some controversial comments or rather stupid design choices to see if the candidate picks them up. The good ones always eventually do, it might take some pushing or going quite extreme, but they finally do 🙂

For a while it really seemed that these exercises were too hard, but finally we got some good candidates that actually breezed through these and made them like just as easy as I thought they would be.

Summary

As you can certainly imagine this is a rather time consuming process and takes a lot of time. I don’t always have time for everything and almost always these topics are scattered into many small sessions.

Generation Kill Software Project

While watching rerun’s of HBO:s acclaimed miniseries Generation Kill it occurred to le how shockingly easy it is to draw parallels between dysfunctional army management and modern day software engineering.

Further the way from action the decisions are made, worse they are and worse the outcome. In war you endanger lives of soldiers and civilians and unfortunate casualties include livelihood of locals and lives of children and women.

Junior soldiers carry out orders with sometimes disastrous results. Seniors know better and from time to time question orders, but are left with dilemma – carry them out or refuse and possibly face disciplinary actions. It’s a decision they have to live with for the rest of their lives.

Software engineering is not much different, but the outcome is of course less dramatic. Bad decisions made without proper understanding of the situation at hand still lead to unwanted and sometimes catastrophic results.  Only difference is that in software engineering casualties include motivation, professionalism, team spirit and in worst cases – careers.

Time to Bury the Construction Metaphor

Using the construction engineering metaphor is probably the most common and well known way of explaining or describing software engineering. Almost every aspiring software professionals dreams of having a fancy title like architect at some point of their career. Unfortunately it’s misleading, creates a lot of confusion and just wrong in so many levels.

People (“even software people”) still use construction metaphor to explain why we need fixed price contracts etc. They seem to forget the fact that even if construction engineering has about 6000 years of history data to help in estimation, big construction efforts still are finished late and the total costs grow way beyond original budget. When you talk about construction, the design cost are negligible compared to actual construction costs and it makes a lot of sense to do the up front planning as well as possible. Changing foundations after building few floors is not possible. Can you imagine building a bridge half way through and then deciding that you need to make it wider to accommodate rails for trains.

Software is different. It’s becoming the norm to talk about emergent design, which means that software design is not done up front, but it emerges little by little. I know this one phrase explanation does not cut it and there are people who give actual talks about emergent design, but it’s out of scope for this blog post.
In software engineering it’s very common to start building something and end up with something completely different. Needs change, understanding evolves and requirements change.

Only constant is change

 The famous quote by Heraclitus is very true at least considering software development. In software development construction costs are non issue and it’s worth building as much as possible. Sometimes you just end up doing the wrong thing and that can’t really be helped.

I didn’t fail, I found 10 000 ways that don’t work

That is very agile mindset really shines. Sometimes you just can’t prepare and you end up building the wrong thing. In those cases failing fast is a lot better than failing slow. Taking a cue from Edison, what I really want to say is that you want is make software that don’t work fast enough so that you still have time to build the right thing. Eventually you’ll get it right.

In software you want feedback. It something doesn’t work – great, now we know, we have some information. Now it is up to us to decide what we want to do about it. Faster the feedback loop the better, but you still you need to validate the feedback. Keep in mind that comments from actual end-users are the most important kind. How can you add value, if you don’t know who the people gaining that value are.

So with that being said, it time to give our respects to old and dated metaphor. May you rust in peace, you just don’t have a place in modern day software development.