2018-09-30 Update: Clojure Web and AI

I am still going through Purely Functional. It is interesting that there are so many ways to define a function in Clojure. I wonder if all functional languages are like that.

I have decided to spend time looking into making web apps in Clojure. I know that AI is big, and it would be nice for Clojure to be a force in that space, but things seem to have changed since I blogged about it in May.

The guy who wrote Guildsman is no longer updating it. I don’t know what is going on with dl4clj. (Granted, I haven’t emailed the guy.) The Deeplearning4J site said that was the official Clojure library for Deeplearning4J. It looks like they are now part of the Eclipse Project, and they totally redid their site. And in my opinion, not very well. There are lots of things that look like links on their FAQ page, but nothing happens when you click them. They have a gitter chat. I will try looking into that sometime. The search on gitter is not that great. I think you have to go day-by-day. Here is the link for the archive on 2018-09-26. Cortex still looks like it is or will be a ghost town. If only someone could make all this stuff less complicated.

I will get more into web apps. I know a lot of people think that mobile is the future, but I think web apps will be around for a while. When the iPad came out in 2010, all the Apple fanboys said that within five years we would all be using iPads for everything all the time. But that did not happen. At my job, we all use laptops. When I go to tech meetups, people still use laptops. Gamers are still into desktops.

Sidenote: One thing that I find odd and frustrating is that while 15-20 years ago a lot of technology people thought they were so smart and sophisticated for not trusting Microsoft (or IBM, or Oracle), yet ever since the iPhone came out, those same people won’t say the sun is out until Apple says so. Yet tech people used to make fun of the suits for not thinking the sun is out unless MS said so. Was I wrong to believe in open source?

For one thing, web browsers are on more devices. Phones and laptops/desktops can all use web apps, but laptops and desktops cannot use mobile apps. Plus I have a hard time typing on phones, and dealing with passwords on phones is very frustrating for me. I prefer dealing with passwords in a password manager or a spreadsheet or an encrypted file. I prefer keeping track of logins and passwords that way rather than deal with a bunch of apps on a phone.

And EVERYBODY makes an app. My bank. My insurance company. The company that processes my rent payments. Which I could make using the app from my credit card company. There is an app for all those stupid scooters all over Austin. Every mom-and-pop Indian place here has an app. My gym has an app. Everybody that Robert Llewellyn talks to on “Fully Charged” has an app. I know my phone is old, but I don’t have a lot of space. I would rather use that space for pictures than yet another app that I might only use one time.

Laptops have bigger screens, bigger keyboards, more hard drive space, more memory; since when did Americans want less?

So there was a thread on the Clojure Google group recently about web apps. A lot of libraries were mentioned. I will go with Luminus first. He walks you through what libraries you need, and you can pick and choose which ones you want to use. Others that were mentioned: Pedestal, Yada, Integrant, and Duct.

I know a lot of Clojure people (and Lispers in general; should we call them Lispters?) don’t like frameworks. They prefer libraries. I never liked the answer “go use libraries.” Nobody ever tells you what libraries to use. I think a lot of people want to be able to produce something quickly that they can see, and frameworks do that. People always like to contrast the “stringing together libraries” approach with Rails. I think that while something like Rails has a lot of “magic” and can do a lot of things for you, magic was not the only reason Rails took off.

What I think helped Rails is that it gave people a default answer. Back then, there was PHP: not a language anybody really loves, and you have to do everything yourself every damn time. “How do you make an app in PHP? From the ground up. Every stinking time.” And you have Java: Lots of frameworks, some of them viable and widely used, some niche. How do you know you are making a good choice and your framework won’t be abandoned later? “How do I make a web app in Java? Pick a framework. We got plenty: A, B, C, D, etc, etc. And then pick a servlet engine. We got a lot of those too!”

“How do I make a web app with Ruby? Use Rails. Go here and do this.” The shortest and most useful answer. Granted, there are/were other frameworks in Ruby, so if you don’t like Rails you have options. But I think having a default answer is the way to go. Choice is great when you understand the choices.

Some people want to make an app, and then go under the hood.

You’re welcome.

Clojure And Gradle Update

I am putting twitter-retriever-gradle on hold. I spent a couple of hours trying to send JVM system properties to the clojureRun task, and I couldn’t figure it out.

I raised an issue on the github repo for the Clojure Gradle plug-in. They made a suggestion, and it did not work out. I was using a few libraries that could use system properties for configuration. That is not the only way to do things, but with Gradle my other option was to have multiple copies of the same file all over the repo. I would rather not do that.

I might ask on the Gradle forum. But for now, I think I will move on to something else.

I think Gradle works well for Groovy and Java. It seems a bit harder to use for Clojure. Back to Purely Functional.

You’re welcome.

Clojure With Gradle: twitter-retriever-gradle

I took my twitter-retriever project and reworked it to use Gradle. It is as twitter-retriever-gradle.

To a certain degree, this may just be about satisfying my curiosity. But then again, you might find yourself at a company that uses a lot of JVM languages and everyone knows gradle. Plus I think that a lot of people in the Clojure community are not too thrilled about having to deploy one monster uberjar. With Gradle, you can keep your jars separate.

Before I go any further, let’s go over some issues. I was not able to get the tests to work. The directory structure is different than leiningen, and I had to make a copy of a file. I was not able to use environ for some reason, so I had to hard-code the database info (a lot of Clojure libraries assume you are using lein or boot; to be fair, Clojure people usually are). I will try again; I might just go with cprop [Note 1].

This requires the nebula-clojure-plugin. The Nebula plugins are Gradle plugins made by Netflix. This plugin does not have a lot of documentation. I had to search the source code to see how to do some things. It adds the tasks “clojureRun”, “clojureRepl” and “clojureTest”. You can still call “clean” and “build” as before. I also enabled the application plugin, so “distZip” works.

First thing you have to do is move the files in src/clojure to src/main/clojure.

You can run it two different ways. One is directly with Gradle:

I think you have everything after the –fn in a single quote.

You can also use the application plugin. You specify the main class in build.gradle. I found out you can also put in a few command line args as well. For Clojure, your main function will not change, so you can add that along with clojure.main:

Then run “gradle distZip”, and it will create a tar and a zip file in build/distributions. Expand these, and run the script in the “bin” directory. You can add further command line args when you run it:

For database access, I am using conman, which under its hood uses hugsql. You put your SQL statements in an SQL file which is read in. I put the file on the classpath at “twitter_retriever/sql/statements.sql” (which in this case was $PROJECT/src/main/clojure/twitter_retriever/sql/statements.sql). Gradle could find that when I ran it with the “gradle” command on the command line. But to run it from the expanded zip file that was produced by the distZip command, that did not work. I had to make a copy of the file at $PROJECT/resources/twitter_retriever/sql/statements.sql. Each location only worked for one run method, so I need two copies. I will see if I can get this to work with one file. cprop might help here.

I could not get the tests to run. (First off, the tests must go into src/test/clojure.) It kept having an issue finding statements.sql on the classpath. I tried putting it into src/test/resources/twitter_retriever/sql, but that did not help. Again, cprop might take care of this.

I did not try running a REPL with “gradle clojureRepl”. I was able to use a REPL and do a few things by calling M-x cider-jack-in in an emacs file. When I did, a little option popped up at the bottom asking me if I was using lein or gradle.

So the experiment was not a complete success, but there is some potential.

You’re welcome.

Note 1: I am leaning towards cprop for the regular twitter-retriever. Like the crop README says: What if you have 100 args? That’s 100 properties to put in the environment.

2018-09-06: Notes On Leiningen

I started another tutorial on Purely Functional. One thing I learned is you should put a “:main” option in your lein project.clj.

When you start a Clojure REPL, by default you will go into the “user” namespace. From here you could call “require” on any namespace in your project.clj or any namespace that comes with Clojure, and use their functions. But to go to another namespace and have access to everything, you need to specify a “:main” option.

One of his lessons is about using Clojure to work with JSON, using a repo called AnApiOfIceAndFire, after the book series “Game Of Boobs” is based on. His project is a-game-of-json. He puts the JSON into some maps in the a-game-of-json.core namespace using the core Clojure function defonce. I went into that namespace, and I was unable to access that map. I couldn’t even call core functions like “+”.

Here is an example session:

This was not a problem in the “Introduction To Clojure” lesson, so I looked at that repo’s project.clj.

Here are the last two items in the project.clj file’s map:

After some editing, I figured out that the way to get access to all the functions is to add the “:main” key/value pair. After that, everything seemed fine.

I did this with twitter-retriever, and I was able to call core functions in all the namespaces I defined. I was even able to access clj-time.local, and call a few core functions in that namespace. I used “in-ns” to make a new namespace, but I was not able to call any of the core functions. Maybe my config is a bit off.

But “user” was available to me without any problems.

You’re welcome.

2018-08-29 Update: Anonymous Functions and Kafka

I am still going through the tutorials on Purely Functional. It is taking longer than I thought. I might start going through Luminus at the same time as well. I think that building something while going through the lessons are a better way to learn.

In one of the lessons on Purely Functional, there is an interesting use of anonymous functions.

In version 3.17 of core.clj, we saw this in the function add:

We see a call to “cond”. It uses the type of the ingredient to decide which function to call.

In version 3.18 of core.clj, he adds a map with anonymous functions:

In version 3.19 of core.clj, he gets the functions out of the map and sets them equal to a symbol, replacing the cond:

I will also look into Apache Kafka. I went to the Apache Kafka meetup recently. I think it will be important going forward.

You’re welcome.

Notes On A Drupal Meetup And Estimates

Recently I went to the Austin Drupal Meetup. The topic was “What is this estimation thing? Why “no estimation” is not as crazy as it sounds“. The speaker was Michael Godeck.

Before the presentation, I did talk to someone about Drupal and Gutenberg. He said that right now Drupal uses the CKEditor. It recently got re-written from the ground up, and the new version has not been integrated with Drupal yet. He said that the Drupal community looked at Gutenberg, and saw that there was a lot of WordPress-specific code, so most people decided that Gutenberg was not the direction they wanted to go in.

However, some people kept going, and have ported Gutenberg to Drupal, or re-worked it to work with Drupal. He said that they will demo their results at the next DrupalEurope in September. I think he was referring to this presentation. So, Gutenberg in Drupal is not a sure thing. But if the demo in Europe shows promise, it might happen.

The main presentation was about estimations. He will be presenting at DrupalEurope.

One issue with estimation is that estimates are sometimes interpreted by managers as promises.

He talked about On the Theory of Scales of Measurement by S.S. Stevens which covers different types of measurement. One type is ordinal, which allows ranking, but no meaningful calculation. You can rank movies using a star-based system; usually people go up to four. We know that a four-star is better than a three-star, and a three-star is better than a two-star. But is the difference between a four-star and a three-star the same as the difference between a three-star and a two-star?

Another type is interval, which does allow you to know the differences between quantities. Think of things like time, temperature and distance.

He said one issue with some agile estimation strategies is that the numbers people give are really ordinal, but they are treated as interval. People think that all uses of numbers are the same, but that is not the case. The agile community still seems to be struggling to this day with the question as to whether or not a point is equal to some unit of time.

He also spoke of confidence intervals. It may be time to brush up on statistics.

He also mentioned Thomas Bayes and Claude Shannon.

Thoughts On WordPress

A week or so ago I went to the Austin WordPress Meetup which featured a presentation about Gutenberg.

Gutenberg is the upcoming editor for WordPress. I admit I did not quite understand everything that was said. But it means there are some big changes coming up. It will allow WordPress to interface with many different types of systems that it cannot talk to today. At one point, the presenter theorized that in a few years WordPress may not use themes as we understand them today. This could put some people out of work; I think there are several companies that sell WordPress themes.

Gutenberg will be written with React. I do not like JS and try to avoid it as much as I can (too much churn and reinvention of wheels), but this might help solidify React’s ranking. As WP people love to point out, WP powers 20-25% of all web sites. The presenter said that Drupal will be moving to Gutenberg as well (although I later found out this is not a sure thing).

I have toyed with the idea of porting WP to another language, even though it would be a massive undertaking. A lot of developers do not like PHP. When I was getting into Ruby and Rails, I joined the mailing lists for local Ruby/Rails groups in about a dozen cities. About once a month, someone in one of the cities would ask about any blog or CMS frameworks written in Ruby. They had made a Rails app for a client, and now the client wanted a blog or CMS. A few replies would mention Radiant or Refinery, but it seems like most of the time the original poster would just go with WordPress. Once someone just posted the Most Interesting Man In the World meme. I did have a go at porting WordPress to Rails.

Eric Normand posted a video titled What Clojure needs to grow – a boring web framework and boring data science.  He said Clojure needs a default web framework. At one point he talks about WordPress. It made me think again about porting WordPress to another language. I have not had problems with WordPress, but some sites have, including Naked Capitalism.

It would also be nice to port the WordPress database from MySQL to Postgres. There is pgloader (written in Common Lisp). There is also pg_chameleon, which is written in Python. But the issue is that some people might not like a new system if it does not look and work just like WordPress, kind of like people do not like other office suites. Plus things might change a lot with Gutenberg.

That said, I think if “ClojurePress” becomes a thing, there are a few things it should do out of the box. There are some things that are pretty common that are handled by plugins. Sometimes multiple plugins, sometimes plugins that have not been updated in a while. Maybe some of these are things that only I want, but here is my list:

  • HTML Tables
  • Static HTML Export
  • Contact Page
  • Twitter intregration (posting AND archiving)
  • Facebook (Granted, Facebook could block this; see here and here)
  • Deleting Revisions
  • Move Comments

 

Reasons I Do Not Like Sharepoint

Here are some reasons I do not like Sharepoint.

First off, I think the underlying concept is wrong. Putting office files in on the web is just stupid.
The web started out at HTML files; even dynamic sites make HTML files. Trying to make the web into a viewer for Office files adds a layer of friction. It is also vendor lock-in. The web was intended to be open. And again, MS is trying to close it.

Speaking of files adding friction to the web: I also do not like PDFs on the web, especially multi-column PDFs.

Why are people putting stuff in Word files anyway? (Spreadsheets are another story.) Why not just use some kind of wiki? Or Atlassian Confluence? Or Drupal? Or Intranet DASHBOARD? Or Jive? Or MindTouch? WYSIWYG editors are not that hard to learn. Why pay for MS Word when you are not using all of its features? And yes, I know SP has a wiki. I have never tried it, and I have no desire to. It is probably terrible. SP is about trying to make proprietary Office files relevant to an open web-centered world. Making a decent wiki is SP would expose the absurdity of SP’s existence.

If you really need to print, I am sure there are other systems that can convert content to PDFs. There are systems that have single sign-on capabilities.

If you use files, why are you using an online system? The impedance mismatch causes problems. People still download the files and email them to each other and get out sync. And even though there is versioning, people will manually add different versions of files. I have seen files in folders like this: file.doc, file_002.doc, file_DECEMBER.doc
I think this is because SP tries to do some many things.

Another reason is almost everyone hates it. At least, almost everyone I have asked about it at companies where I have worked that have used SP have hated it. Maybe everybody tells the big shots they love it and everybody is lying to me, but I really do not have any decision-making authority. Why would people lie to me?

There are many developers who take a course that uses Lisp or Scheme, or get a job using Lisp, and they will rave about it years or even decades later. They will say that Lisp/Scheme changed their perception unlike anything else. That they learned more by using Lisp than they learned using anything else. Nobody really holds MS technologies in the same esteem. A lot of non-technical people love MS products because they are generally unaware of any alternative. We should use technologies that we admire, not simply out of ignorance.

We are always told on the technology side that we have to “learn to understand the needs of the business.” Why don’t they have to learn anything about technology? Or what is good technology?

The only people who seem to like SP are managers and SP consultants. I asked someone at my current employer why we used SP, and he told me that we got a good deal from our cloud provider. Not that we need it. Or that it is good at its intended purpose (whatever that is).

I have found that many things that are justified based on cost and not quality are usually not very good. When I was at BofA, we had to use some computer-based training. Everybody hated it, except our finance guy. When he was asked to justify it at a meeting, he said it was “cost-effective.” If the users hate it and nobody learns anything from it, how is it effective?

Another reason is that it is from MS. It’s the same old bait and switch. They sell it as something easy, and you don’t need an admin. But when you hit a wall, or want to do something advanced, then MS says you need an admin.

Some people have this weird blindness about MS. If you present them with another spreadsheet or word processor, they will reject it because it is not exactly like MS Office. But if MS changes the interface (like from a toolbar to a ribbon), people just go along with it.

Another reason is that the search capabilities in SP are terrible. This is important for something used to store documents. I have always gotten better search results on blogs and wikis. Content on wikis can be versioned, so why put stuff into SP? I think most businesses could replace SP with a wiki. I think a lot of wikis can do single-sign on, permissions, whatever. Bookmarks would not change. A lot of people love to move SP folders.

SP is a complex product sold as a simple product. It is a desert topping and a floor wax. If SP does not fit your business, you have two choices: change your life to fit SP, or replace SP. Desktop PCs, laptops, cel phones induced people to change their behavior and workflow. SP is not amazing enough for people to want to change for it.

At my current employer, search is so bad that a committee was started to come up with solutions. I went to one meeting, and I realized it was a waste of time. Someone suggested a SP site with links to all the other SP sites. For some reason, dropping SP was not considered. I do not get the hold SP has on some people. Yes, migrating would be expensive. But using it is expensive. And painful.

complex product sold as a simple product
impedance mismatch
vendor lock-in
bad search

You’re welcome.

Venkat On Improving Code

The speaker at the Austin JUG last week was Venkat Subramaniam (https://twitter.com/venkat_s, http://www.agiledeveloper.com/). He is to Java what Sandi Metz is to Ruby: If he is speaking in your town, you don’t care what the specific topic is. You know you need to go because you will learn something.

The topic was Twelve Ways to Make Code Suck Less. I will post some notes I took at the presentation. He gave the list counting down from twelve to one.

Why should we care about code quality? You cannot be agile if your code sucks. Code we write is how we tell our colleagues how we feel about them. Quality code is code that can be understood quickly.

  • 12. Schedule time to lower technical debt. Technical debt can take the form of not upgrading a framework (or JDK) to a more current version. Bad code is not technical debt; it is sabotage. You should make a list of your technical debt and ask yourself if you are adding to it, or reducing it. Do not schedule 100% of your developers’ time for adding new features. You need some slack time; you need to spend some time learning.
  • 11. Favor high cohesion. Highly cohesive code does one thing and does it well. It reduces the frequency/reasons for change. High cohesion leads to low complexity. He mentioned Cyclomatic complexity. TL/DR: Be a dessert topping OR a floor wax.
    (I once again point you to the github page for Boot, with this little gem: “The Modern CLJS tutorials are an excellent introduction to Boot and ClojureScript. Pretty much everything you need to know about Boot to get a project off the ground is covered there. Check it out!” What if I want to learn about Boot but do not want to spend any time on ClojureScript? Why not have a tutorial that is just about Boot?)
  • 10. Favor loose coupling. Remove coupling if possible. Code with high coupling is hard to test, hard to extend. Having a lot of mocks in a test is a bad sign.
  • 9. Program with intention. He mentioned Kent Beck’s Design Rules:
    – Passes the tests
    – Reveals intention
    – No duplication
    – Fewest elements
    See also http://wiki.c2.com/?XpSimplicityRules They are in order, so passing the tests takes priority over intention. (So why does Dr Subramaniam go in reverse order?) We should program deliberately.
  • 8. Avoid primitive obsession. Do not write general code. Imperative code has accidental complexity.
    If you write a for-loop, you might wonder if you need less-than, or less-than-or-equal-to. That is a sign. We do imperative because it is familiar. He gave an example of a method with a for-loop. He noted you would need extra variables to hold the result and the count. He contrasted it with an example of functional code using Java lambdas. Functional code is about a series of transformations, not mutations. You can read the functional example in one pass, while with the imperative you need to go up and down while reading it.
    Imperative code is easy to write, but harder to read, while functional is easier to read. Good code is a story, not a puzzle. Lambdas are glue code, and sometimes two lines might be too many.
    Lambdas still take some work to understand. You need to learn the semantics. The functional style is a combination of declarative code and higher-order functions.
  • 7. Prefer clear code to clever code. Clever code makes you feel smart. Do not focus on performance too early.
  • 6. Apply William Zinsser‘s principles on writing: Simplicity, clarity, brevity, humanity.
    He recommended “On Writing Well” by William Zinsser. He said when his son asked to borrow his copy, he refused. Instead he bought his some another copy, and his son has re-read it at least three times.
  • 5. Comment the why, not the what. A lot of developers make a lot of comments because a lot of companies and professors mandate comments. Do not comment to cover up bad code. Write expressive code that is self-documenting. Many times a comment is like explaining a joke. He recommended the book The Design of Everyday Things.
  • 4. Avoid long methods – Apply SLAP (Single Level of Abstraction Principle) Long methods are hard to understand, lack cohesion, tend to become longer, and are harder to test. How to break up methods? Think about what layer of abstraction you are at.
    He said it could be a bad idea to have a hard and fast rule about how many lines a method should be. I know Sandi Metz has a rule that methods should not be any longer than five lines. But is putting a rigid number such a good idea? I am probably not as smart as Sandi Metz, and frankly, you probably are not either. If you dogmatically insist on a hard number, you will probably just get a lot of methods that call methods that call methods, etc. Maybe we should stop using the word “Rule” when “Guideline” would be more appropriate.
    One example he gave is that if someone asks you what you did over the weekend, you might say, “I went to such-and-such city on Saturday, and stayed home and did laundry on Sunday”. If they ask for more details, you could give them. Or you would talk about how you woke up, had breakfast, got in the car, got gas, with route you took, etc, etc, etc. They might not want that level of detail off the bat.
    You might have a method that gets info from a database table, gets info from a web service, combines/transforms them somehow, and writes the result to another table. And has all the try/catch blocks for all the exceptions. That would be a good candidate for breaking it up.
    He related a story about someone who attended one of his lectures who was hired by a company to refactor a method that was 40,000 lines long. After a year, this person got it down to 30,000 lines.
    I stopped to talk to someone after the presentation, and then walked to my car. I think I overheard at least three conversations about this particular point.
  • 3. Give good, meaningful names. Your variable names represent your level of abstraction.
    He gave an example of bad variable names from a company that he could not name. It was software used to control oil refineries. One license is $1 million. He would not name the company, but if one license is $1 million, that narrows it down.
  • 2. Do tactical code reviews. Do code reviews often, and make them incremental. If you put them off, people will just allow code to pass without understanding it or really looking at it.
  • 1. Reduce state and state mutation. Do not make a field in a class (or method) unless you really need it. He related a story about an exhibit of Picasso with twenty versions of a painting, instead of just showing the final result. If Picasso could iterate, so can we. (I have a note asking how do functional languages deal with databases.)

Here are some pages on his site that I found to be informative:

You’re welcome.

2018-06-24 Update

There was not a lot of Clojure this week. There will be more in next week’s update.

I am taking a couple of weeks off from work. I have to take a lot of time off. I have accrued so much that if I do not take a lot in the next few months I will lose some.

I will study some Clojure, perhaps make progress on Simply Scheme. I also plan on writing something on why Lisp interests me, and perhaps why I hate Sharepoint. I will also visit some family in another state that we are not from.

You’re welcome.