Spock Notes

I have started using the Groovy testing framework called Spock.

It looks like a nice testing framework, but it also seems a bit disorganized. The code is on github, but the main site seems to be on Google Code. They are working on some new documentation, which is incomplete, but has a link to some old documentation. On the old Google Code site, Spock Basics and Getting Started are good places to get started. Someone on the mailing list referred me to some javadocs, some groovydocs, and another set of javadocs. I am not too clear why there are two sets of javadocs. Someday I will get the code from github, and see what I can build  myself.

First off, I am not much of a Spock expert. I know I am not using all of the things Spock has to offer. I started porting a DNS server from Java to Groovy, and I decided to start by porting the tests from Java to Groovy. (At this point, very little of the actual server code has been ported. After seeing a lot of byte arrays in the tests, I wonder if writing a DNS server in Groovy is a good idea.) The tests were in JUnit, and after porting them to Groovy, I decided to port them to Spock.

You can find my Groovy Spock tests at the DNS Groovy project here. I have both Spock and JUnit tests in that directory.

I ported the JUnit tests to Spock tests. I extended a couple of classes. I extended org.spockframework.util.ObjectUtil as MyGroovyUtil, and org.spockframework.util.Assert as MyGroovyAssert. I add them to the tests like this:

The classes that I am extending are abstract. But after extending, I was able to use their methods. Here are a few examples of the methods I used and how I used them to port the test from JUnit:
  • assertEquals becomes mgu.equals
  • assertNull(XXX) becomes mgu.equals(null, XXX)
  • assertTrue becomes mga.assertTrue
  • assertFalse(x == y) becomes mga.that( x != y)
  • assertFalse(XX.YY) becomes mga.that(!XX.YY)
  • assertNotSame(m_h, h2) becomes !mgu.equals(m_h, h2)
  • assertSame(x, y) becomes mgu.equals(x, y)

I am not an expert with Spock. I took some code by Venkat S to test the java.util.concurrent.locks interface. I tried porting them from JUnit to Spock, but I was not able to get some of them to work with Spock. Perhaps I will try again in the future. You can find the original JUnit tests in this directory, and my attempts at porting them to Spock are in this directory. Venkat’s code is in this directory. He has a method that adds some values to a map, and he refactors it so that the actual update is in a method called by the put method between the locking and unlocking.


Thoughts On Groovy and Grails

I have been going to the Austin Groovy and Grails group. It seems like Groovy is gaining some momentum. It was gaining some momentum, and then for a couple of years it lost some steam, but now it’s coming back.

I have heard some rumbles that Java shops have gone with Rails since they not find some Grails devs. I wonder if there is anything to these anecdotes. It seems like it would be cheaper for Java devs to learn Grails than Rails. If you go with Rails, not only do you have to learn a new language, but a whole new ecosystem.

I have also heard from a lot of people whose companies are using JRuby. These companies have a huge investment in Java code, and they do not want to throw that away, but they want a more dynamic, agile language that is more amenable to TDD. Which makes me wonder why they did not choose Grails. The reasons they go with JRuby is the whole reason Groovy and Grails were invented.

I have worked with JRuby, and frankly it is more trouble than it is worth. There was a presentation about it recently at Austin on Rails, and the speaker showed a slide with some JRuby code. He said, “This code looks odd to a Java developer. It also looks odd to a Ruby developer.” I thought to myself that is a good reason to not use JRuby. Outside of Ruboto, I think JRuby is a solution looking for a problem.

Another issue with JRuby is you lose the ability to use all the gems written in C. There goes a lot of the power of Ruby.

One reason people give for going to JRuby is that the JVM handles concurrency better than MRI. That is true, but for me that does not change the fact that Groovy is easier to work with on the JVM is than Ruby since that is the environment that it was built for. I just do not find JRuby as easy to work with as MRI Ruby or Groovy/Grails. The whole point of Ruby was to be easier for developers, and I think JRuby takes that away. If MRI has a weakness, then fix it, or just drop Ruby entirely. Don’t try to fit a square peg into a round hole just so you can still tell yourself you’re a Ruby developer. It’s just not worth it in my opinion.

I think that if Elixir can get off the ground, then a lot of Ruby developers will go with that. I am not an Erlang developer, so for all I know Elixir may be another square peg/round hole issue.

Done With Venkat S’s Book

I am finally all through Programming Concurrency on the JVM by Venkat Subramaniam.

I am looking at some code he wrote to test classes in the java.util.concurrent.lock package. I might go through that.

I have a feeling that a lot of what I went through might be obsolete soon. There was a lot of code that could be done more concisely with lambdas, which are coming soon in Java 8. I think you might be able to do extend the ReadWriteLock class and have it take some code that you want to make thread-safe. In one of the times I saw him, he said that locks take care of any issues you would have with the synchronized keyword.

The problem is that you really need to use try/catch/finally blocks, and you should use the tryLock() method, and use a timeout. It can be a lot of code.  You might want to change one variable, but instead of having one line in your setter, you might need about 6. But if you could send that one line to a LockLambda (or whatever you want to call it), it would be better. You might need two, which granted is still more than one, but the LockLambda could handle all the try/catch/finally/get the lock/set the timeout/release the lock work for you, and you might be able to re-use the same LockLambda class.

I might blog about Akka, although I am not too thrilled with it. I do not like Scala, and some of the Akka APIs call Scala APIs. I liked GPars better. Although there were some things in GPars that are not worth the effort to use in Java, you can pretty much use just the GPars APIs. You do not need to look at any Groovy if you do not want to. I think some things I did not like with Akka will be easier once lambdas come out. But I still do not like Scala.

I may still look at the Play Framework.


2013-07-14 Concurrency Update

I have been going through Venkat’s book, but I think I might bail on going any further.

The book came out in 2011, and there have been a LOT of changes to Akka in that time. They stopped using Multiverse for STM, and started using ScalaSTM. There were a lot of changes from version 1 to version 2 (starting actor systems, creating actors, sending messages, pretty much everything). There was a change in the API as I was going through the book.

Some of the method calls require inline Runnables or Callables, and that gets kind of awkward. I think I will move on to other things and come back when there is another edition of the book. Or I might wait until Java 8 comes out. I think the closures will make things easier.

Plus, since Akka is written in Scala, a lot of things are unwieldy in Java. I really do not like Scala. It has the syntactical elegance of Perl, with the simplicity and focus of C++. It is like religion or cigarettes: Its advocates tell me that if I gave it a chance I would see how wonderful it is. I did, and it’s not.

That said, I have no issue with the idea of using something written in Scala. I might take a look at the Play Framework. That is not as odd as it seems. I do not like working with C code. But a LOT of important stuff is written in C: The JVM, MRI, databases, the Linux kernel. If I refused to work with anything written in C, I would need to find a new line of work.

One language I think I will take a closer look at is Clojure. At first I was skeptical about it: It’s just another Lisp, and Lisp has never caught on, so why bother? But I have kept an eye on it, and I think this might be The Lisp That Sticks.

Plus, I have noticed something interesting about Lisp. A lot of people will start out developing in one language, and then they will become dissatisfied, and move to another. And then another. And then another. But it seems like when people find Lisp, they stop moving around. They seem to find what they are looking for.

In the next few months there are some Clojure meetups that will go over some Clojure web frameworks. I might look at them before that.

When Ken Kousen was here for the Groovy meetup, he said that he had a boss who kept saying, “Oh, we could do that in Lisp 20 years ago.” He thought it was really annoying. He said it was more annoying when he realized his boss was right.

Upcoming Meetups

There are a couple of user group meetups this week that I had a part in organizing.

On Tuesday, Roger Brinkley, the host of the Java Spotlight podcast, will speak at Austin JUG. He has spoken at CJUG a few times. I put him in touch with the leader of the Austin JUG. I did not know that Roger would be in San Antonio this week. So it was a pretty nice coincidence that he happened to be in Texas this week.

Then on Thursday, Ken Kousen will speak at the Austin Groovy and Grails group. I am on the list for the Houston JUG. He spoke there a few months ago. I went to his website, and he was going to speak in Houston and Dallas in the same week. I just emailed him out of the blue, and I told him there is a Groovy/Grails group in Austin. I told him he should speak in Austin someday. He emailed me back, and said it was not possible at that time, but he would like to do it someday. I put him in touch with the organizers, and he will be here on Thursday.

Concurrency Without Pain in Pure Java

Here are the notes I took when Venkat Subramaniam was here for Austin JUG. I will mostly present them as I typed them.

Why Concurrency? Why do we care?

We now have mutli-core processors. Things that work well on one core may not work well on multiple cores.

The Java memory model controls how info goes back and forth from main memory and working memory.

His arguments in his classes are all final.

In the account class. Should the getBalance() be synchronized?

“synchronized” gets a lock and crosses the memory barrier, working and main memory.

Accessing a volatile variable, wait, join, a few other calls where you cross the memory barrier. We could compromise the invariant, like during a constructor.

So if you do not make getBalance synchronized, you might get an old value.

The code could have deadlock, or livelock (you wait, but nothing waits for you).

When you call synchronized, you might wait forever. There is no time out. So you use the Lock interface.

So for ReentrantLock you could call tryLock, or throw the Exception, then you have to unlock it.

You can order the accounts by ID to removed threat of dead lock.

95% of Java code is broken for concurrency. (Is he including code for Groovy, Scala, Clojure?)

Synchronize and suffer model.

Solutions: Software Transactional Memory (from Clojure) and Actors (Scala)

STM: We modify a variable (data) in our code. Shared mutability is bad. A few solutions: Avoid mutability. STM says let’s have a special type of variable: Managed mutable variable. It will only change in certain circumstances: a transaction.

Identity: Values are immutable. The stock price at 1:00 is different than at 1:10. But the 1:00 price will never change.

The identity is “google stock price”, points to a different value at different times.

We must protect the identity.

The problem in Java is that many times the problems happen without warning or error message. In STM, you get an error message.

This changes it in a transaction. If there are conflicts, the transaction is aborted and retried. Up to 100,000 times.

Can we do it in Java? He used ref and dosync, so can we use them in Java?

clojure.lang.Ref. Look at his code, see what class he uses for Ref.

The set method will use Lambda expression in Java 8. For now, we could do a Callable. LockingTransaction.runInTransaction

He is using an inline Callable.call()

What if you try to put more in the to account than is the from amount? It will throw an exception.

When to use STM? When you have multiple readers, multiple writers, and infrequent write collision. Not too many things are good for high-write collision.

You should not modify unmanaged variables in a transaction. Do not write to a file, print out messages, do not send email.

What about a long-running task

Another solution to shared mutability: isolated mutability.

Problem with encapsulation: Many threads can call the methods that change your private variables.

It is like voice mail: Many people can leave him a message, he gets a lot of messages, he goes through them one at a time. Actors have built-in message queue. Asynchronous messages.

The Actor runs on a separate thread.

When do actors make sense?

The fields are immutable. The messages are not, so they should be immutable.

You can have variables in an Actor, since only one thread will touch an Actor.

When does Actor make sense: Single threaded tasks sequentially

It punishes readers as much as writers. In STM, readers are never blocked. Writers are punished.

Purpose of Actor is not to deal with shared mutability, you want to prevent concurrent access, you want to serialize.

New concurrency API in Java 8.

You need Java 8 with Lambdas.

Look at java.util.streams.*

:: is a method reference

can be done as

Java is getting more functional

Synchronize and shared mutability can be avoided

JDK 5 concurrency is the assembly code of concurrency

Still Going Through Venkat’s Book

I am still going through Venkat S’s book.

I had gotten through the chapter on STM. A lot of the examples for STM dealt with some old versions of APIs. The first time I just read through them pretty slowly. I was able to follow. But then I did a bit more googling, and I was able to find later versions of the STM libraries he was using.

I had already started the chapter on actors. I decided to go back and work on the STM examples. It is taking a bit of time.

Venkat S Was In Austin

I still am going through Programming Concurrency on the JVM by Venkat Subramaniam.

I have hit a bit of a snag. I am on the chapter covering Actors. He uses the Akka library, but he uses an old version of it. A lot of the packages have changed, and I could not find some of the classes in the documentation. I would like to say that it is nice that they now have javadoc in addition to scaladoc. Frankly, I am not too thrilled with Scala. The Java code that calls Clojure STM classes still works.

Also: Venkat S was here a couple of days ago to speak at the Austin JUG. He gave the same concurrency talk I saw a couple of years ago at No Fluff Just Stuff. I took some notes, and I will post them in a few days. Some of the Java 8 stuff looks interesting.


Fork/Join and More Java 5 Concurrency APIs

As I mentioned before, I am going through Programming Concurrency on the JVM by Venkat Subramaniam.

Chapter 3 was about design. He discussed immutability, shared mutability, and isolated mutability.

In Chapter 4 he went over some of the concurrency APIs that were introduced in Java 5.

The Fork/Join framework was not as much of a beast as I thought (although I am not claiming to know the whole thing). It is a lot like dealing with Callables, Futures and ExecutorServices and thread pools. First, you create a ForkJoinPool. Then you make a class that extends  a child class of ForkJoinTask. There are two, and they each have a method called “compute” that actually does the work, sort of like Callable.call(). RecursiveAction.compute() returns void, and RecursiveTask.compute() returns an object.

Your grandchild class of ForkJoinTask might instantiate a few copies of itself in the compute() method. You build up a java.util.Collection of them, and call invoke() or invokeAll() to start the calculations. I guess this is sort of like ExecutorService.submit(). Then you call ForkJoinTask.join to bring the results together. This is all kicked off by ForkJoinPool.invoke() (which returns an object) or ForkJoinPool.invokeAll() (which returns a List of Future objects). ForkJoinPool implements the ExecutorService interface, and ForkJoinTask implements the Future interface. So in a way it is like Callables and Futures, although I am not too clear right now when you would use one versus the other.

He deals briefly with CountDownLatch and CyclicBarrier. He states that there are better ways to do concurrency than Threads and locking, so I did not spend a whole lot of time on these. “CyclicBarrier” sounds like something out of science fiction.

He touched on AtomicLong, and that it is a good way to make a long variable thread-safe. There are also AtomicInteger and AtomicBoolean classes. I wonder why there is no AtomicFloat or AtomicDouble. The API for these says that these should not be used as replacements for the java.lang.Integer and java.lang.Long classes. Yet if you look at the Really Big Tutorial, they kind of do exactly that.

There was also a section on concurrent collections. He pointed out that they have better performance that older collection classes in addition to being thread safe, although he did not spend a whole lot of time on them.

There was a section on the classes in the java.util.concurrent.locks package. It looks like it is a good substitute for the synchronized keyword. although it is pretty verbose. I made a few classes that set and manipulate a few variables, with one using “synchronized” and two others using locks. One used a separate RentrantLock for each variable, and the other used one for the whole class. The main class just goes through a loop, calls some methods and sees how long it took. The class using the “synchronized” keyword was faster. I know that the Lock objects are better at preventing deadlocks, but it does seem like they sacrificed some speed. Perhaps I need to find a better comparison than the one I made.

Dr S. will be here next week for the Austin JUG meeting. Hopefully by then I will be through chapter 6, Software Transactional Memory. It is the longest chapter in the book. I will have to set up some projects for Clojure and Scala. I am doing the Java stuff with Gradle, and I think I am learning as much about Gradle as I am about concurrency.

A Look At Java’s Callable Interface Part 002

As I mentioned before, I am going through Programming Concurrency on the JVM by Venkat Subramaniam. I am looking at the Callable interface. He uses it a few times in chapter 2. I will go over some code I posted last time.

The two files I will go over are CallableWorker and ForwardCallableRunner. There are a few things in the code that are not relevant to concurrency, but still have use. I put in the ForwardCallableRunner.setIterations which will set the number of iterations a loop will run by using a system property, or set a default value if none is provided. I also put in some logging statements and calls to Thread.sleep() to get a better idea of when things are called.

The class that does the concurrent magic is CallableWorker. It implements the java.util.concurrent.Callable interface. It is part of the concurrency APIs that were introduced in JDK 5. It is similar to the java.lang.Runnable interface that has been part of the JDK since the very beginning. The main difference is that Callable can return a value and Runnable cannot. Developers would have to create additional methods to check if their Runnable.run() method had completed. Dr S says that we should use the newer APIs and not bother with the older stuff.

The only method defined by Callable is the call() method. It takes no arguments, and returns an object. If you are returning a Java primitive (like an int or double), you must use one of the wrapper classes in java.lang, like java.lang.Double.

I added a few parameters to the constructor. A lot of examples online that I saw (including the original that I got this from by Lars Vogella) do not use a lot of data. They just increment a variable in a loop. do some simple math, and return the result. If you want to add data to your class to work on, you could either add it via the constructor, or another method, or create the Callable inline, as Dr S does in his examples. I would guess that adding it in the constructor would be better for thread safety. But ulitimately, I just do some simple summing, like most other examples.

The class that creates and uses the Callables is ForwardCallableRunner. I also made a BackwardCallableRunner. They both create some Callables, which sum some numbers in loops. In ForwardCallableRunner, the numbers send to the constructors for CallableWorker increase in the loop, and in BackwardCallableRunner they decrease.

The first thing you need to do when using a Callable is to create a thread pool. The JDK 5 API gives you some classes to do that.

Use java.util.concurrent.Executors to create a thread pool. I use the method Executors.newFixedThreadPool with an integer argument for the size of the thread pool. This will return an instance of java.util.concurrent.ExecutorService.

Executors.newFixedThreadPool is overloaded. You could also call it with an addition argument, which is a class implementing the java.util.concurrent.ThreadFactory interface. There are also a couple of methods that make thread pools of dynamic size: Executors.newCachedThreadPool. You can use the Executors.newFixedThreadPool without specifying a ThreadFactory. There are a couple of other libraries that do implement or return an instance of java.util.concurrent.ThreadFactory: BasicThreadFactory in Apache Commons Lang, and  ThreadFactoryBuilder in the Google Guava library. I have not tried them, so I do not know what they give you beyond what the default implementation gives you.

The next step is to create an instance of  java.util.concurrent.Future, or a list of them. It must be of the same type as the Callable you created before.

The next part of the example is in a loop. In each iteration, I created a Callable, and I sent it as an argument in a call to the method java.util.concurrent.ExecutorService.submit. This will return a java.util.concurrent.Future, which will hold the result of our Callable. The Callable.call method starts when the ExecutorService.submit method is called. (I inferred this from the examples, but I wanted some verification; this is what led me to put in the logging statements.) I then add the Future that was just created to my list of Future objects.

Lastly, I create a for loop to cycle through the Future objects. This shows the next step in our Callable/Future lifecycle. To get the return value of the Callable.call method, you call Future.get. The documentation says that it will wait if the result is not ready yet. This is why I made an example that counts upwards through the number of iterations and one that starts at the iteration number and counts to 0. Since my CallableWorker.call method also does some sleeping, the class with the highest number should take longer. I wanted to see if the for loop with the call to Future.get would actually wait if the next Callable.call was not done yet. Since I got the same result in both classes, I assume that it did.

The last step is to call ExecutorService.shutdown().

Most of the examples are like the one that I worked with: They call ExecutorService.submit for each Callable. Dr S creates a java.util.List of Callable objects, and starts them off by calling ExecutorService.invokeAll. He also creates his Callable objects inline, and does not have a separate Something.java file for them.

So, to summarize, here are the steps to using Callable:

1. Create a class that implements the java.util.Callable interface. (This step could come later.)

2. Create a thread pool by calling java.util.concurrent.Executors.newFixedThreadPool to return an instance of java.util.concurrent.ExecutorService

3. Create a list with type of java.util.concurrent.Future.

4. Call ExecutorService.submit, sending it a Callable object. This will create a Future object, which you add to your list of Future objects.

5. Get the result of all of your Callable.call methods by iterating through your list of Future objects, and calling Future.get on each one.

6. Clean up by calling ExecutorService.shutdown.

Dr S does this slightly differently:

1. Create a list of inline objects that implement the java.util.Callable interface.

2. Create a thread pool by calling java.util.concurrent.Executors.newFixedThreadPool to return an instance of java.util.concurrent.ExecutorService

3. Create a list with type of java.util.concurrent.Future, and instantiate it with a call to ExecutorService.invokeAll. The paramter of ExecutorService.invokeAll is the list of Callable objects created in step 1.

4. Get the result of all of your Callable.call methods by iterating through your list of Future objects, and calling Future.get on each one.

5. Clean up by calling ExecutorService.shutdown.

His way has fewer steps, and fewer lines of code. But honestly I prefer the way most examples do it. I think it is clearer. Doing too much inline looks a bit noisy and crowded to me.