Category Archives: Clojure

2018-09-30 Update: Clojure Web and AI

I am still going through Purely Functional. It is interesting that there are so many ways to define a function in Clojure. I wonder if all functional languages are like that.

I have decided to spend time looking into making web apps in Clojure. I know that AI is big, and it would be nice for Clojure to be a force in that space, but things seem to have changed since I blogged about it in May.

The guy who wrote Guildsman is no longer updating it. I don’t know what is going on with dl4clj. (Granted, I haven’t emailed the guy.) The Deeplearning4J site said that was the official Clojure library for Deeplearning4J. It looks like they are now part of the Eclipse Project, and they totally redid their site. And in my opinion, not very well. There are lots of things that look like links on their FAQ page, but nothing happens when you click them. They have a gitter chat. I will try looking into that sometime. The search on gitter is not that great. I think you have to go day-by-day. Here is the link for the archive on 2018-09-26. Cortex still looks like it is or will be a ghost town. If only someone could make all this stuff less complicated.

I will get more into web apps. I know a lot of people think that mobile is the future, but I think web apps will be around for a while. When the iPad came out in 2010, all the Apple fanboys said that within five years we would all be using iPads for everything all the time. But that did not happen. At my job, we all use laptops. When I go to tech meetups, people still use laptops. Gamers are still into desktops.

Sidenote: One thing that I find odd and frustrating is that while 15-20 years ago a lot of technology people thought they were so smart and sophisticated for not trusting Microsoft (or IBM, or Oracle), yet ever since the iPhone came out, those same people won’t say the sun is out until Apple says so. Yet tech people used to make fun of the suits for not thinking the sun is out unless MS said so. Was I wrong to believe in open source?

For one thing, web browsers are on more devices. Phones and laptops/desktops can all use web apps, but laptops and desktops cannot use mobile apps. Plus I have a hard time typing on phones, and dealing with passwords on phones is very frustrating for me. I prefer dealing with passwords in a password manager or a spreadsheet or an encrypted file. I prefer keeping track of logins and passwords that way rather than deal with a bunch of apps on a phone.

And EVERYBODY makes an app. My bank. My insurance company. The company that processes my rent payments. Which I could make using the app from my credit card company. There is an app for all those stupid scooters all over Austin. Every mom-and-pop Indian place here has an app. My gym has an app. Everybody that Robert Llewellyn talks to on “Fully Charged” has an app. I know my phone is old, but I don’t have a lot of space. I would rather use that space for pictures than yet another app that I might only use one time.

Laptops have bigger screens, bigger keyboards, more hard drive space, more memory; since when did Americans want less?

So there was a thread on the Clojure Google group recently about web apps. A lot of libraries were mentioned. I will go with Luminus first. He walks you through what libraries you need, and you can pick and choose which ones you want to use. Others that were mentioned: Pedestal, Yada, Integrant, and Duct.

I know a lot of Clojure people (and Lispers in general; should we call them Lispters?) don’t like frameworks. They prefer libraries. I never liked the answer “go use libraries.” Nobody ever tells you what libraries to use. I think a lot of people want to be able to produce something quickly that they can see, and frameworks do that. People always like to contrast the “stringing together libraries” approach with Rails. I think that while something like Rails has a lot of “magic” and can do a lot of things for you, magic was not the only reason Rails took off.

What I think helped Rails is that it gave people a default answer. Back then, there was PHP: not a language anybody really loves, and you have to do everything yourself every damn time. “How do you make an app in PHP? From the ground up. Every stinking time.” And you have Java: Lots of frameworks, some of them viable and widely used, some niche. How do you know you are making a good choice and your framework won’t be abandoned later? “How do I make a web app in Java? Pick a framework. We got plenty: A, B, C, D, etc, etc. And then pick a servlet engine. We got a lot of those too!”

“How do I make a web app with Ruby? Use Rails. Go here and do this.” The shortest and most useful answer. Granted, there are/were other frameworks in Ruby, so if you don’t like Rails you have options. But I think having a default answer is the way to go. Choice is great when you understand the choices.

Some people want to make an app, and then go under the hood.

You’re welcome.

Clojure And Gradle Update

I am putting twitter-retriever-gradle on hold. I spent a couple of hours trying to send JVM system properties to the clojureRun task, and I couldn’t figure it out.

I raised an issue on the github repo for the Clojure Gradle plug-in. They made a suggestion, and it did not work out. I was using a few libraries that could use system properties for configuration. That is not the only way to do things, but with Gradle my other option was to have multiple copies of the same file all over the repo. I would rather not do that.

I might ask on the Gradle forum. But for now, I think I will move on to something else.

I think Gradle works well for Groovy and Java. It seems a bit harder to use for Clojure. Back to Purely Functional.

You’re welcome.

Clojure With Gradle: twitter-retriever-gradle

I took my twitter-retriever project and reworked it to use Gradle. It is as twitter-retriever-gradle.

To a certain degree, this may just be about satisfying my curiosity. But then again, you might find yourself at a company that uses a lot of JVM languages and everyone knows gradle. Plus I think that a lot of people in the Clojure community are not too thrilled about having to deploy one monster uberjar. With Gradle, you can keep your jars separate.

Before I go any further, let’s go over some issues. I was not able to get the tests to work. The directory structure is different than leiningen, and I had to make a copy of a file. I was not able to use environ for some reason, so I had to hard-code the database info (a lot of Clojure libraries assume you are using lein or boot; to be fair, Clojure people usually are). I will try again; I might just go with cprop [Note 1].

This requires the nebula-clojure-plugin. The Nebula plugins are Gradle plugins made by Netflix. This plugin does not have a lot of documentation. I had to search the source code to see how to do some things. It adds the tasks “clojureRun”, “clojureRepl” and “clojureTest”. You can still call “clean” and “build” as before. I also enabled the application plugin, so “distZip” works.

First thing you have to do is move the files in src/clojure to src/main/clojure.

You can run it two different ways. One is directly with Gradle:

I think you have everything after the –fn in a single quote.

You can also use the application plugin. You specify the main class in build.gradle. I found out you can also put in a few command line args as well. For Clojure, your main function will not change, so you can add that along with clojure.main:

Then run “gradle distZip”, and it will create a tar and a zip file in build/distributions. Expand these, and run the script in the “bin” directory. You can add further command line args when you run it:

For database access, I am using conman, which under its hood uses hugsql. You put your SQL statements in an SQL file which is read in. I put the file on the classpath at “twitter_retriever/sql/statements.sql” (which in this case was $PROJECT/src/main/clojure/twitter_retriever/sql/statements.sql). Gradle could find that when I ran it with the “gradle” command on the command line. But to run it from the expanded zip file that was produced by the distZip command, that did not work. I had to make a copy of the file at $PROJECT/resources/twitter_retriever/sql/statements.sql. Each location only worked for one run method, so I need two copies. I will see if I can get this to work with one file. cprop might help here.

I could not get the tests to run. (First off, the tests must go into src/test/clojure.) It kept having an issue finding statements.sql on the classpath. I tried putting it into src/test/resources/twitter_retriever/sql, but that did not help. Again, cprop might take care of this.

I did not try running a REPL with “gradle clojureRepl”. I was able to use a REPL and do a few things by calling M-x cider-jack-in in an emacs file. When I did, a little option popped up at the bottom asking me if I was using lein or gradle.

So the experiment was not a complete success, but there is some potential.

You’re welcome.

Note 1: I am leaning towards cprop for the regular twitter-retriever. Like the crop README says: What if you have 100 args? That’s 100 properties to put in the environment.

2018-09-06: Notes On Leiningen

I started another tutorial on Purely Functional. One thing I learned is you should put a “:main” option in your lein project.clj.

When you start a Clojure REPL, by default you will go into the “user” namespace. From here you could call “require” on any namespace in your project.clj or any namespace that comes with Clojure, and use their functions. But to go to another namespace and have access to everything, you need to specify a “:main” option.

One of his lessons is about using Clojure to work with JSON, using a repo called AnApiOfIceAndFire, after the book series “Game Of Boobs” is based on. His project is a-game-of-json. He puts the JSON into some maps in the a-game-of-json.core namespace using the core Clojure function defonce. I went into that namespace, and I was unable to access that map. I couldn’t even call core functions like “+”.

Here is an example session:

This was not a problem in the “Introduction To Clojure” lesson, so I looked at that repo’s project.clj.

Here are the last two items in the project.clj file’s map:

After some editing, I figured out that the way to get access to all the functions is to add the “:main” key/value pair. After that, everything seemed fine.

I did this with twitter-retriever, and I was able to call core functions in all the namespaces I defined. I was even able to access clj-time.local, and call a few core functions in that namespace. I used “in-ns” to make a new namespace, but I was not able to call any of the core functions. Maybe my config is a bit off.

But “user” was available to me without any problems.

You’re welcome.

2018-08-29 Update: Anonymous Functions and Kafka

I am still going through the tutorials on Purely Functional. It is taking longer than I thought. I might start going through Luminus at the same time as well. I think that building something while going through the lessons are a better way to learn.

In one of the lessons on Purely Functional, there is an interesting use of anonymous functions.

In version 3.17 of core.clj, we saw this in the function add:

We see a call to “cond”. It uses the type of the ingredient to decide which function to call.

In version 3.18 of core.clj, he adds a map with anonymous functions:

In version 3.19 of core.clj, he gets the functions out of the map and sets them equal to a symbol, replacing the cond:

I will also look into Apache Kafka. I went to the Apache Kafka meetup recently. I think it will be important going forward.

You’re welcome.

2018-06-03 Update

I am working through the tutorials on Purely Functional.

I got a bit hung up right away. I was trying to figure out how to reload files while working in the REPL as I updated the files. I followed this suggestion on StackOverflow by some guy named Dirk. At first, Dirk’s quirk did not work. Was Dirk a jerk? His burst was not voted first. SO at its worst? It made me so weak I could not speak. My REPL I could not tweak, putting an end to my learning streak. I do not want to sound pissed, but I do not wish to be dissed by Lisp. I am too tired for this.

Then I realized that while the files were reloaded, the new functions were not included in any aliases I used when I brought in the namespace with “require”. I could use the new functions, but I had to type out the ENTIRE namespace. O, the huge manatee! I did some googling, and I don’t think anyone mentions that.

You’re welcome.


2018-06-12_23.46.42 Update:

Just so it is easy to copy and paste, here are the commands:

Or you could do this:

You might have a function called “refresh”, but you will probably not have a namespace “repl”.

Do this after you follow the directions in the StackOverflow answer. It is better to bring in the clojure.tools namespace in your profile than in every stinkin’ project.

 

2018-05-28 Update: Clojure AI

I am still working through Clojure For the Brave and True.

I am on the exercises at the end of chapter 5, but I might skip a couple of them. I don’t do too well with these “re-implement function x” exercises. I will also start going through the videos on Purely Functional. I think he is raising the price, and my membership will renew at the end of the year.

I am looking into some Big Data/Deep Learning libraries for Clojure. This was inspired by the last Austin Clojure meetup: There were only four of us and we had a hard time thinking of topics. I tried searching for Clojure meetups around the country for topic ideas, and frankly the pickings were kind of slim.

Frankly, sometimes the technology industry in Austin and in general is kind of a drag. If you don’t want to do out-of-date Java or spend your life on the JavaScript-no-CoffeeScript-no-TypeScript-no-now-its-something-else-Script roller coaster, the pickings can be pretty slim. Sometimes I think about taking a year or so off and going though Structure And Interpretation of Computer Programs or How To Design Programs and become a smarter person (not just better with some particular language or API). The issue is that while a lot of people will say things like “we aren’t looking for people with experience in the technology we use; we just want to hire smart people”, they turn around and only hire people with experience in the technology they use.

Anyway, the consensus in the Clojure community is that Clojure needs to be a bigger player in these spaces. There is a LOT of Python in AI. Being a JVM language, Clojure will have wrappers around a lot of the Java libraries I wrote about in Thoughts On Native, GPU, Groovy, Java and Clojure (even though there was not a lot of Clojure in that post).

I know that Big Data and AI are different things. I was thinking about looking at Sparkling to work with Spark (which I hope I can do on my laptop; do you need big machines to work with Big Data libraries?). This weekend I started looking at some of the videos on the Clojure TV channel on YouTube from Clojure Conj 2017. I did not go, but there seemed to be a LOT of videos about AI/Deep Learning (yes, I am using those terms interchangeably even though a lot of people do not).

There was Deep Learning Needs Clojure by Carin Meier, author of Living Clojure. She wasted the first seven minutes on some stupid joke about Skynet, which is a lot for a thirty minute presentation. I am glad I did not pay to see that. After that it gets better. The talk was pretty general. She mentioned some Clojure libraries, Graal VM, and at about 25:00 talks about how to get into Deep Learning.

Declarative Deep Learning In Clojure by Will Hoyt talked about Deeplearning4j. He says that matrices and linear algebra pretty much IS deep learning. There is some neuroscience in this presentation. He also talks about how the Clojure code is easier to deal with than Java and builders. I do not think he ever posts a link to dl4clj, which according to the Deeplearning4j site is the official port.

The Tensors Must Flow by William Piel is about his library Guildsman, which is a new Clojure interface to Google’s TensorFlow. There are already two Clojure projects that allow access to TensorFlow (clojure-tensorflow and tensorflow-clj). They do some Java interop around the TensorFlow Java API. (You can see Google’s not-quite-Javadoc here.) He wanted something that was more idiomatic for Clojure programmers. TensorFlow is written in Python (what else?), which Google then ports to C++ and other languages. But like most Python AI libs, it seems like it is just a wrapper around CPU or GPU code.

I understood the talk when I watched it. Really, I did. From what I remember, TensorFlow uses something called gradients in its process. I think a gradient implementation is an operation in the process. bpiel says the best way to contribute to Guildsman is to actually contribute C++ gradients to TensorFlow itself. He said in the talk he wanted to be done with Guildsman before the Conj in October. It is almost June, and he is still working on it.

The last one about Deep Learning was Building Machine Learning Models with Clojure and Cortex by Joyce Xu. She talked about a Clojure library called Cortex. This library is closer to Uncomplicate, in that while it does interface with a GPU  library, it is not a wrapper around a Java library in the middle.

The traffic on the Cortex mailing list seems to have dropped off, it’s not at version 1 and there seems to be a drop-off in contributions since January.

I do wish the speakers spent a bit more time talking about the implementation details of these libraries. Hosted languages (like Java or Python) do not do a lot of their AI crunching directly. They usually call a native library to calculate on either the CPU or the GPU. And for the GPU, some can do either CUDA (in other words, NVidia) or OpenCL (every other video card). Some libraries have multiple options, like Uncomplicate or Deeplearning4j. TensorFlow can use a CPU (they have instructions on getting the JNI file here) or GPU withy NVidia only. I have not tried Guildsman, so I do not know how he handles things or if he requires an NVidia GPU. I also have no idea how Cortex handles it. Their instructions tell you to get some SDK from NVidia. Perhaps they default to a CPU if there is no NVidia GPU.

I bought my laptop used, and the one I used before this one is at least six years old. I think the older one had an Intel video card, but I could not find any SDK for that version of the video chip. I think my current laptop may also be too old. (Dealing with the Intel Math Kernel is a LOT easier than wading through their OpenCL pages.) The only reason I can think of to buy an Apple laptop is to not deal with this. It is a bit frustrating. The whole point of using a language like Java or Ruby or Python is that I do not want to deal with hardware details.

Anyway, besides all that, I still have a few ideas for a few web apps to do in Clojure.


2018-05-29_22.14.49 update:

I looked at the Cortex mailing list, and apparently you can run the tests just using the CPU:

It would be great if they put that in the README.

You’re welcome.

2018-05-20 Update

I am still going through Clojure For the Brave and True. He does some pretty mind-bending things in chapter 5 when he walks you through his pegthing program.

Check this out:

In the call to “reduce”, he is sending it a collection of functions called “connect-right”, “connect-down-left” and “connect-down-right”. Then in the function in the reduce, there is a call to “connection-creation-fn”, which is passed to the anonymous function. So he is not actually calling a function called “connection-creation-fn”; that is a placeholder for the functions in the array being passed to “reduce”.

Granted, I knew before this what “reduce” is. But in this chapter he has a lot of functions calling functions. Granted, he does say that reducing over a collection of functions is not something you will do or see very often. Even though I have been looking at Clojure for a while, and this chapter wasn’t really anything new, it was a reminder how different functional programming can be.

You’re welcome.

2018-05-14 Update

I am still going through the the code review of twitter-retriever by Eric Normand of Purely Functional. And going through Clojure For the Brave and True.  Sometimes I come home from work and I just don’t feel like doing anything.

I am seeing a lot of nice core functions in Clojure For the Brave and True. It is helping me in my quest to post examples of as much of the core functions as I can.

I am still watching the code review in bits and pieces. I guess I don’t like being criticized, even though he is very supportive (not only of members of his site, but the Clojure community in general) and not being too critical, and I have incorporated a few of the things he has said. I had to do some funky things with tweet IDs to work Twitter timelines that seemed a bit confusing and might not be idiomatic Clojure; Twitter has a guide for that here. That said, I was able to incorporate some suggestions and reduced the lines of code, and I was able to remove one of the SQL calls and use some functions to get the same result.

He did mention that a lot of the defs should be in a let block. I showed this to the Austin Clojure group, and they had the same suggestion.

I still have a lot of println statements in there. I can run this on the command line or call some of the functions in the REPL, but testing some of this stuff is a bit beyond me. I have no idea how to mock/stub database calls or external calls in Clojure.

You’re welcome.

2018-05-06 Update

I have watched part of the twitter retriever code review. I did make some changes based on what he said.

I am also going through the videos on Purely Functional. I am also working on Clojure For the Brave and True. I am on chapter 4.

I started changing my workouts. I wonder if that is taking energy away from Clojure. There is a lot to learn. I have been working on and off on a post about why I am interested in Clojure. I don’t think I have written a post yet about why Clojure and Lisp in general interests me.

You’re welcome.