Clojure And Gradle Update

I am putting twitter-retriever-gradle on hold. I spent a couple of hours trying to send JVM system properties to the clojureRun task, and I couldn’t figure it out.

I raised an issue on the github repo for the Clojure Gradle plug-in. They made a suggestion, and it did not work out. I was using a few libraries that could use system properties for configuration. That is not the only way to do things, but with Gradle my other option was to have multiple copies of the same file all over the repo. I would rather not do that.

I might ask on the Gradle forum. But for now, I think I will move on to something else.

I think Gradle works well for Groovy and Java. It seems a bit harder to use for Clojure. Back to Purely Functional.

You’re welcome.

Clojure With Gradle: twitter-retriever-gradle

I took my twitter-retriever project and reworked it to use Gradle. It is as twitter-retriever-gradle.

To a certain degree, this may just be about satisfying my curiosity. But then again, you might find yourself at a company that uses a lot of JVM languages and everyone knows gradle. Plus I think that a lot of people in the Clojure community are not too thrilled about having to deploy one monster uberjar. With Gradle, you can keep your jars separate.

Before I go any further, let’s go over some issues. I was not able to get the tests to work. The directory structure is different than leiningen, and I had to make a copy of a file. I was not able to use environ for some reason, so I had to hard-code the database info (a lot of Clojure libraries assume you are using lein or boot; to be fair, Clojure people usually are). I will try again; I might just go with cprop [Note 1].

This requires the nebula-clojure-plugin. The Nebula plugins are Gradle plugins made by Netflix. This plugin does not have a lot of documentation. I had to search the source code to see how to do some things. It adds the tasks “clojureRun”, “clojureRepl” and “clojureTest”. You can still call “clean” and “build” as before. I also enabled the application plugin, so “distZip” works.

First thing you have to do is move the files in src/clojure to src/main/clojure.

You can run it two different ways. One is directly with Gradle:

I think you have everything after the –fn in a single quote.

You can also use the application plugin. You specify the main class in build.gradle. I found out you can also put in a few command line args as well. For Clojure, your main function will not change, so you can add that along with clojure.main:

Then run “gradle distZip”, and it will create a tar and a zip file in build/distributions. Expand these, and run the script in the “bin” directory. You can add further command line args when you run it:

For database access, I am using conman, which under its hood uses hugsql. You put your SQL statements in an SQL file which is read in. I put the file on the classpath at “twitter_retriever/sql/statements.sql” (which in this case was $PROJECT/src/main/clojure/twitter_retriever/sql/statements.sql). Gradle could find that when I ran it with the “gradle” command on the command line. But to run it from the expanded zip file that was produced by the distZip command, that did not work. I had to make a copy of the file at $PROJECT/resources/twitter_retriever/sql/statements.sql. Each location only worked for one run method, so I need two copies. I will see if I can get this to work with one file. cprop might help here.

I could not get the tests to run. (First off, the tests must go into src/test/clojure.) It kept having an issue finding statements.sql on the classpath. I tried putting it into src/test/resources/twitter_retriever/sql, but that did not help. Again, cprop might take care of this.

I did not try running a REPL with “gradle clojureRepl”. I was able to use a REPL and do a few things by calling M-x cider-jack-in in an emacs file. When I did, a little option popped up at the bottom asking me if I was using lein or gradle.

So the experiment was not a complete success, but there is some potential.

You’re welcome.

Note 1: I am leaning towards cprop for the regular twitter-retriever. Like the crop README says: What if you have 100 args? That’s 100 properties to put in the environment.

2018-09-06: Notes On Leiningen

I started another tutorial on Purely Functional. One thing I learned is you should put a “:main” option in your lein project.clj.

When you start a Clojure REPL, by default you will go into the “user” namespace. From here you could call “require” on any namespace in your project.clj or any namespace that comes with Clojure, and use their functions. But to go to another namespace and have access to everything, you need to specify a “:main” option.

One of his lessons is about using Clojure to work with JSON, using a repo called AnApiOfIceAndFire, after the book series “Game Of Boobs” is based on. His project is a-game-of-json. He puts the JSON into some maps in the a-game-of-json.core namespace using the core Clojure function defonce. I went into that namespace, and I was unable to access that map. I couldn’t even call core functions like “+”.

Here is an example session:

This was not a problem in the “Introduction To Clojure” lesson, so I looked at that repo’s project.clj.

Here are the last two items in the project.clj file’s map:

After some editing, I figured out that the way to get access to all the functions is to add the “:main” key/value pair. After that, everything seemed fine.

I did this with twitter-retriever, and I was able to call core functions in all the namespaces I defined. I was even able to access clj-time.local, and call a few core functions in that namespace. I used “in-ns” to make a new namespace, but I was not able to call any of the core functions. Maybe my config is a bit off.

But “user” was available to me without any problems.

You’re welcome.

2018-08-29 Update: Anonymous Functions and Kafka

I am still going through the tutorials on Purely Functional. It is taking longer than I thought. I might start going through Luminus at the same time as well. I think that building something while going through the lessons are a better way to learn.

In one of the lessons on Purely Functional, there is an interesting use of anonymous functions.

In version 3.17 of core.clj, we saw this in the function add:

We see a call to “cond”. It uses the type of the ingredient to decide which function to call.

In version 3.18 of core.clj, he adds a map with anonymous functions:

In version 3.19 of core.clj, he gets the functions out of the map and sets them equal to a symbol, replacing the cond:

I will also look into Apache Kafka. I went to the Apache Kafka meetup recently. I think it will be important going forward.

You’re welcome.

Notes On A Drupal Meetup And Estimates

Recently I went to the Austin Drupal Meetup. The topic was “What is this estimation thing? Why “no estimation” is not as crazy as it sounds“. The speaker was Michael Godeck.

Before the presentation, I did talk to someone about Drupal and Gutenberg. He said that right now Drupal uses the CKEditor. It recently got re-written from the ground up, and the new version has not been integrated with Drupal yet. He said that the Drupal community looked at Gutenberg, and saw that there was a lot of WordPress-specific code, so most people decided that Gutenberg was not the direction they wanted to go in.

However, some people kept going, and have ported Gutenberg to Drupal, or re-worked it to work with Drupal. He said that they will demo their results at the next DrupalEurope in September. I think he was referring to this presentation. So, Gutenberg in Drupal is not a sure thing. But if the demo in Europe shows promise, it might happen.

The main presentation was about estimations. He will be presenting at DrupalEurope.

One issue with estimation is that estimates are sometimes interpreted by managers as promises.

He talked about On the Theory of Scales of Measurement by S.S. Stevens which covers different types of measurement. One type is ordinal, which allows ranking, but no meaningful calculation. You can rank movies using a star-based system; usually people go up to four. We know that a four-star is better than a three-star, and a three-star is better than a two-star. But is the difference between a four-star and a three-star the same as the difference between a three-star and a two-star?

Another type is interval, which does allow you to know the differences between quantities. Think of things like time, temperature and distance.

He said one issue with some agile estimation strategies is that the numbers people give are really ordinal, but they are treated as interval. People think that all uses of numbers are the same, but that is not the case. The agile community still seems to be struggling to this day with the question as to whether or not a point is equal to some unit of time.

He also spoke of confidence intervals. It may be time to brush up on statistics.

He also mentioned Thomas Bayes and Claude Shannon.

Thoughts On WordPress

A week or so ago I went to the Austin WordPress Meetup which featured a presentation about Gutenberg.

Gutenberg is the upcoming editor for WordPress. I admit I did not quite understand everything that was said. But it means there are some big changes coming up. It will allow WordPress to interface with many different types of systems that it cannot talk to today. At one point, the presenter theorized that in a few years WordPress may not use themes as we understand them today. This could put some people out of work; I think there are several companies that sell WordPress themes.

Gutenberg will be written with React. I do not like JS and try to avoid it as much as I can (too much churn and reinvention of wheels), but this might help solidify React’s ranking. As WP people love to point out, WP powers 20-25% of all web sites. The presenter said that Drupal will be moving to Gutenberg as well (although I later found out this is not a sure thing).

I have toyed with the idea of porting WP to another language, even though it would be a massive undertaking. A lot of developers do not like PHP. When I was getting into Ruby and Rails, I joined the mailing lists for local Ruby/Rails groups in about a dozen cities. About once a month, someone in one of the cities would ask about any blog or CMS frameworks written in Ruby. They had made a Rails app for a client, and now the client wanted a blog or CMS. A few replies would mention Radiant or Refinery, but it seems like most of the time the original poster would just go with WordPress. Once someone just posted the Most Interesting Man In the World meme. I did have a go at porting WordPress to Rails.

Eric Normand posted a video titled What Clojure needs to grow – a boring web framework and boring data science.  He said Clojure needs a default web framework. At one point he talks about WordPress. It made me think again about porting WordPress to another language. I have not had problems with WordPress, but some sites have, including Naked Capitalism.

It would also be nice to port the WordPress database from MySQL to Postgres. There is pgloader (written in Common Lisp). There is also pg_chameleon, which is written in Python. But the issue is that some people might not like a new system if it does not look and work just like WordPress, kind of like people do not like other office suites. Plus things might change a lot with Gutenberg.

That said, I think if “ClojurePress” becomes a thing, there are a few things it should do out of the box. There are some things that are pretty common that are handled by plugins. Sometimes multiple plugins, sometimes plugins that have not been updated in a while. Maybe some of these are things that only I want, but here is my list:

  • HTML Tables
  • Static HTML Export
  • Contact Page
  • Twitter intregration (posting AND archiving)
  • Facebook (Granted, Facebook could block this; see here and here)
  • Deleting Revisions
  • Move Comments

 

Venkat On Improving Code

The speaker at the Austin JUG last week was Venkat Subramaniam (https://twitter.com/venkat_s, http://www.agiledeveloper.com/). He is to Java what Sandi Metz is to Ruby: If he is speaking in your town, you don’t care what the specific topic is. You know you need to go because you will learn something.

The topic was Twelve Ways to Make Code Suck Less. I will post some notes I took at the presentation. He gave the list counting down from twelve to one.

Why should we care about code quality? You cannot be agile if your code sucks. Code we write is how we tell our colleagues how we feel about them. Quality code is code that can be understood quickly.

  • 12. Schedule time to lower technical debt. Technical debt can take the form of not upgrading a framework (or JDK) to a more current version. Bad code is not technical debt; it is sabotage. You should make a list of your technical debt and ask yourself if you are adding to it, or reducing it. Do not schedule 100% of your developers’ time for adding new features. You need some slack time; you need to spend some time learning.
  • 11. Favor high cohesion. Highly cohesive code does one thing and does it well. It reduces the frequency/reasons for change. High cohesion leads to low complexity. He mentioned Cyclomatic complexity. TL/DR: Be a dessert topping OR a floor wax.
    (I once again point you to the github page for Boot, with this little gem: “The Modern CLJS tutorials are an excellent introduction to Boot and ClojureScript. Pretty much everything you need to know about Boot to get a project off the ground is covered there. Check it out!” What if I want to learn about Boot but do not want to spend any time on ClojureScript? Why not have a tutorial that is just about Boot?)
  • 10. Favor loose coupling. Remove coupling if possible. Code with high coupling is hard to test, hard to extend. Having a lot of mocks in a test is a bad sign.
  • 9. Program with intention. He mentioned Kent Beck’s Design Rules:
    – Passes the tests
    – Reveals intention
    – No duplication
    – Fewest elements
    See also http://wiki.c2.com/?XpSimplicityRules They are in order, so passing the tests takes priority over intention. (So why does Dr Subramaniam go in reverse order?) We should program deliberately.
  • 8. Avoid primitive obsession. Do not write general code. Imperative code has accidental complexity.
    If you write a for-loop, you might wonder if you need less-than, or less-than-or-equal-to. That is a sign. We do imperative because it is familiar. He gave an example of a method with a for-loop. He noted you would need extra variables to hold the result and the count. He contrasted it with an example of functional code using Java lambdas. Functional code is about a series of transformations, not mutations. You can read the functional example in one pass, while with the imperative you need to go up and down while reading it.
    Imperative code is easy to write, but harder to read, while functional is easier to read. Good code is a story, not a puzzle. Lambdas are glue code, and sometimes two lines might be too many.
    Lambdas still take some work to understand. You need to learn the semantics. The functional style is a combination of declarative code and higher-order functions.
  • 7. Prefer clear code to clever code. Clever code makes you feel smart. Do not focus on performance too early.
  • 6. Apply William Zinsser‘s principles on writing: Simplicity, clarity, brevity, humanity.
    He recommended “On Writing Well” by William Zinsser. He said when his son asked to borrow his copy, he refused. Instead he bought his some another copy, and his son has re-read it at least three times.
  • 5. Comment the why, not the what. A lot of developers make a lot of comments because a lot of companies and professors mandate comments. Do not comment to cover up bad code. Write expressive code that is self-documenting. Many times a comment is like explaining a joke. He recommended the book The Design of Everyday Things.
  • 4. Avoid long methods – Apply SLAP (Single Level of Abstraction Principle) Long methods are hard to understand, lack cohesion, tend to become longer, and are harder to test. How to break up methods? Think about what layer of abstraction you are at.
    He said it could be a bad idea to have a hard and fast rule about how many lines a method should be. I know Sandi Metz has a rule that methods should not be any longer than five lines. But is putting a rigid number such a good idea? I am probably not as smart as Sandi Metz, and frankly, you probably are not either. If you dogmatically insist on a hard number, you will probably just get a lot of methods that call methods that call methods, etc. Maybe we should stop using the word “Rule” when “Guideline” would be more appropriate.
    One example he gave is that if someone asks you what you did over the weekend, you might say, “I went to such-and-such city on Saturday, and stayed home and did laundry on Sunday”. If they ask for more details, you could give them. Or you would talk about how you woke up, had breakfast, got in the car, got gas, with route you took, etc, etc, etc. They might not want that level of detail off the bat.
    You might have a method that gets info from a database table, gets info from a web service, combines/transforms them somehow, and writes the result to another table. And has all the try/catch blocks for all the exceptions. That would be a good candidate for breaking it up.
    He related a story about someone who attended one of his lectures who was hired by a company to refactor a method that was 40,000 lines long. After a year, this person got it down to 30,000 lines.
    I stopped to talk to someone after the presentation, and then walked to my car. I think I overheard at least three conversations about this particular point.
  • 3. Give good, meaningful names. Your variable names represent your level of abstraction.
    He gave an example of bad variable names from a company that he could not name. It was software used to control oil refineries. One license is $1 million. He would not name the company, but if one license is $1 million, that narrows it down.
  • 2. Do tactical code reviews. Do code reviews often, and make them incremental. If you put them off, people will just allow code to pass without understanding it or really looking at it.
  • 1. Reduce state and state mutation. Do not make a field in a class (or method) unless you really need it. He related a story about an exhibit of Picasso with twenty versions of a painting, instead of just showing the final result. If Picasso could iterate, so can we. (I have a note asking how do functional languages deal with databases.)

Here are some pages on his site that I found to be informative:

You’re welcome.

2018-06-24 Update

There was not a lot of Clojure this week. There will be more in next week’s update.

I am taking a couple of weeks off from work. I have to take a lot of time off. I have accrued so much that if I do not take a lot in the next few months I will lose some.

I will study some Clojure, perhaps make progress on Simply Scheme. I also plan on writing something on why Lisp interests me, and perhaps why I hate Sharepoint. I will also visit some family in another state that we are not from.

You’re welcome.

2018-06-17 Update

I am still going through the tutorials on Purely Functional. There is nothing new to me in the intro videos, but I am getting practice with the REPL and emacs.

The project that I am on is downsizing. I might leave my employer and try my luck in the big, bad world. At one point I told my manager that I was interested in Lisp and Clojure. I probably mentioned that Lisp is one of the oldest languages and also not widely used. At a recent meeting, he said that I was interested in “legacy” technology. I plan on writing a post about why I am interested in Lisp and Clojure. While Lisp is one of the first programming languages invented, I don’t think it is appropriate to call it “legacy”. Lisp can adapt to any paradigm. “Legacy” has a negative connotation.

Part of the appeal of Lisp is that it seems like languages in the C family are becoming more like Lisp and Smalltalk over time. I have written this on this blog a few times. I mentioned it to some people once in Chicago, and they thought it was an interesting insight.

You’re welcome.

2018-06-03 Update

I am working through the tutorials on Purely Functional.

I got a bit hung up right away. I was trying to figure out how to reload files while working in the REPL as I updated the files. I followed this suggestion on StackOverflow by some guy named Dirk. At first, Dirk’s quirk did not work. Was Dirk a jerk? His burst was not voted first. SO at its worst? It made me so weak I could not speak. My REPL I could not tweak, putting an end to my learning streak. I do not want to sound pissed, but I do not wish to be dissed by Lisp. I am too tired for this.

Then I realized that while the files were reloaded, the new functions were not included in any aliases I used when I brought in the namespace with “require”. I could use the new functions, but I had to type out the ENTIRE namespace. O, the huge manatee! I did some googling, and I don’t think anyone mentions that.

You’re welcome.


2018-06-12_23.46.42 Update:

Just so it is easy to copy and paste, here are the commands:

Or you could do this:

You might have a function called “refresh”, but you will probably not have a namespace “repl”.

Do this after you follow the directions in the StackOverflow answer. It is better to bring in the clojure.tools namespace in your profile than in every stinkin’ project.