What To Do With Myself

I am not clear what I should use my free time doing. Right now I am still going through the Struts 2 tutorials, and I am also upgrading a few web apps that I made a few years ago.

I tried to recompile an old app with Maven, and it took a long time because I had to track down the latest versions of all the dependencies. I am also struggling with testing it as well. I am having a hard time setting things up so that the tests hit a different database than the “production” app. I could just edit the hibernate.cfg.xml file in src/main/resources, but I would prefer to do things the “right” way.

I tried using src/test/resources/hibernate.cfg.xml, but that did not work. I know that editing the prod file is not quite correct, but I figure I could spend more time trying to work on that, or I could write some tests.

It is a bit frustrating that there are  a LOT of sample apps that do not have tests included, and also do not have test configurations as well.

I know some people say that tests should never touch the database, but personally I think that is kind of dumb. All this app does is change the database. I am not against mocking and stubbing, and if you are calling external services you need it. But I do not understand the stance that some fundamentalists take that your tests should NEVER touch the database. Changing databases is what a lot of applications do. If all you are doing is using mocks, then you are not testing your app. You are testing some artificial thing that is not really your app.

Git, Folders, Maven and Modules

I am going through the Struts 2 tutorials. I decided to put it up on Github as I go through it. You can download each section as a separate Maven app, or you can keep updating the same app as you go through it. So far I have been able to update the same app.

I put the app in a folder inside my git repository. At first I was not able to get the folder holding the Maven app to show up on Github. The folder was not a blue link, just a greyed out dead end. I did some Googling, and the issue was that it was being loaded into git as a module. All the pages I saw told me to do something with my .gitmodule file. None of this was helpful since there was no .gitmodule file. Git had decided to treat that folder like a module. Honestly, I am not really clear what a submodule is in git. The pages I saw told me to undo something that I could not undo, since I did not do it.

So I copied the Maven contents up to the main directory, and delete the folder. Then I made a new folder with a random name. I was able to push that to github. Then I changed the name back to the original name. I think Maven and the servlet engine needed the original name. After I did that, I was still able to see the Maven app in the folder, with the original folder name.

I was pretty frustrated during all of this. so there are a lot of comments that look like a blind man’s tweets: “jasdopfhlh”. I just wanted to get this done, and it worked.

I have a couple of repos with Gradle apps in subdirectories, and git did not “help” me with those apps. I guess git or github is aware of Maven, but not Gradle.

 

Ruby Tip: Instances

So I was working on a Rails project, helping someone upgrade from Rails 2 on Ruby 1.8.7 to Rails 3 on Ruby 1.9. There have been a few changes in the APIs. At one point, I was getting some exceptions. I needed to see what variables were available to an object. There were ifs and mixins and supers, so it was not always clear what I could call in a “puts” statement and what I could not.

So here is how you print out a list of the instance variables, and get a value:

 

Making A Difference Right Away

A while back I attended a GeekFest talk by Ashish Dixit  (so far the only person that I have met who I know is from Nepal) on “Making a Difference Right Away.” My notes are in outline form, but this will be a mix of bullet points and complete sentences, and perhaps a mix of past, present and future tenses.

Premise:
– had limited experience
– enthusiastic
– got a great opportunity

Exciting yet stressful
This is not just for new developers. The core issues are the same for new codebases and new domains.

Why is he talking about it? He got a lot of help from the community. He asked for help. There has been a lot of talk about hiring, but very little talk from the perspective of a new developer.

What he has learned:
– pairing is caring: it’s a good way to get up to speed on a codebase
– bugs are your friends: they are challenges to explore a codebase
– fresh set of eyes: people can become habituated to things
– make it work before you make it better
– flying solo
– asking for help is still a struggle: timeboxing is a good idea, go for 30-60 minutes
– automate, automate, automate: set up an environment with a script, “rake bootstrap”
– communication is king: try not to give too muuch background info
– pay it forward
– other strategies: side projects, meetups and conferences

NMatrix Compilation Instructions

Here are some compilation and installation instructions for Ubuntu that I wrote for the NMatrix gem which is part of the SciRuby project.

Make a working directory for this project: /home/$USER/ruby/sciruby

ATLAS needs CPU throttling to be disabled. On Ubuntu, install the package indicator-cpufreq. It will be an applet on the menu bar. Select “performance”.

Download the LAPACK tar file from http://www.netlib.org/lapack. Save it and unzip, but do not untar it. You do not actually need to compile LAPACK to compile ATLAS, but you do need the tar file.

To get ATLAS, go to http://math-atlas.sourceforge.net/ and bunzip2 the file in the working directory.

For some reason, you cannot run “configure” in the untarred directory. Create a new directory /home/$USER/ruby/sciruby/outputatlas.
Run the ATLAS configure command from there:

Then run “make” It takes about 20 minutes.

This will create a few *.a files. As root, make a directory /usr/local/atlas, copy the *.a files there, and create links as .so files

Go to https://github.com/SciRuby/nmatrix, get the github repo, and follow the directions.

When you run the command “gem install pkg/nmatrix-0.0.2.gem”, it will say:

It does take a while, and there is not a lot of feedback to the screen.

After that completes, run “rspec” or “rake spec”.

Notes:
When I ran rspec, I got some errors, but I posted them to the list, and they are all expected and/or being looked at.

The SciPy page has a section for Ubuntu. It says that I should go into the directory I created to build ATLAS, then go into lib, and run “make shared”. This would be /home/$USER/ruby/sciruby/outputatlas/lib This will create the .so files. I tried this, and I was not able to get it to work. I am googling to find out the answer, but so far no success. But by creating links to the .a files I got it to work. I will look into it, and hopefully get some feedback from the SciRuby list, but so far this seems to work.


More notes/commentary (will probably not appear on official SciRuby site):

It depends on some math libraries written in C and Fortran. The Linux installation instructions on the SciRuby site tell people to go to the SciPy site. I think this is bad for a couple of reasons. First off, the SciRuby project is a competitor to SciPy. It is not as advanced or ready for prime time, but if it really wants to compete it will need to handle things on its own. Plus I think some of the instructions on the SciPy site are out of date.

Ruby is a high-level language, so it is a bit odd to be dealing with C and Fortran. I like to deal with high-level languages to avoid having to deal with low-level details. But if you want to do some serious math, C and Fortran are part of the deal. SciPy is really just a wrapper around the C and Fortran libraries.

 

Working With Nokogiri

I recently worked a bit with Nokogiri to parse some XML. I decided to parse the XML behind the map for the Craftsmanship Manifesto.  The map is here  and the XML behind the map can be found here. I put this on Github, and you can find it here.

I tried parsing the XML the textbook way. bin/run.first.parser.sh calls lib/first_parser.rb. It’s a mess. It seems like you have to call the Element and the Text classes to get an element. At least that is what I remember and what I can gather from the code. I have a lot of comments in there. I always have comments in code that is just for exploration. But it seems like I have to call two classes to get one element. Just wrong.

I then looked into using XPath. bin/run.show.parser.sh calls lib/show_parser.rb, which is the example on the Nokogiri site. I was able to parse it with bin/run.first.path.parser.sh, which calls lib/first_path_parser.rb and bin/run.path.parser.sh which calls lib/path_parser.rb I had a problem with namespaces. I first tried  doc.remove_namespaces! but I did not like the idea of disabling namespaces. There was no namespace for the document, so I just prepended “xmlns:” to all the element names and I got it to work.

Eventually I decided to try JSON. I found out about a gem called crack which can convert an XML document to JSON. It is in bin/run.crack.is.whack.sh, which calls lib/crack_is_whack.rb. I was able to try it out with bin/run.first.json.attempt.sh, which calls lib/first_json_attempt.rb. It parses a small version of the file. The whole map is parsed and output to csv with bin/run.json.parser.sh, which calls lib/json_parser.rb

JSON is a lot easier than XML.

Image from  Aurora Consurgens, a 15th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

2012-09-04 Thoughts on Clojure

I am looking at Clojure more these days. Right now I am going through the Clojure lectures on Pluralsight. I came for the concurrency. I stayed for the elegance.

From what I have heard, it handles concurrency better than most other languages. At Lone Star Ruby Conf, a few presenters talked about how Ruby will handle the coming “concurrency freight train”. Robert Martin said that we should look at a purely functional language. I think that was a hint to look at Clojure and avoid Scala.

I am looking at a great site called Clojure Docs, which I heard about from Colin Jones. It has better documentation than the official Clojure site. The docs at cloure.org for ->> are not useful at all. And good luck googling that.

A few weeks ago I attended a presentation at Geekfest about logic programming in Clojure. One of the examples the presenter gave was about chess. He programmed the rules of chess in his application. From there, you could arrange some pieces, and the applicaiton could tell you if that particular arrangement could have resulted from a legitimate game of chess, and what the moves would have been, and also what could happen going forward.

Someone asked me why you couldn’t do that with a rule engine, and honestly I did not have an answer.

But still, I can see how logic programming could be useful for scientific research. You could say that some material could handle X pounds of pressure per square inch. Good stuff.

Image from World Digital Library, assumed allowed under Fair Use. Image from the Ashburnham Pentateuch, or Tours Pentateuch, a Latin manuscript of the first five books of the Old Testament from the 6th century or 7th century. Its place of origin is unknown.

Notes From The Little Redis Book

I am working on a small Ruby app that uses Redis. It won’t really do much. I am just doing it to work with Redis a bit. I am using the Redis gem.

For my reference, here are some commands from The Little Redis Book. (I think this is allowed under the license.)

https://github.com/karlseguin/the-little-redis-book/blob/master/en/redis.md
set users:leto “{name: leto, planet: dune, likes: [spice]}”
get users:leto
set users:leto “{name: leto, planet: dune, likes: [spice]}”
> strlen users:leto
(integer) 42

> getrange users:leto 27 40
“likes: [spice]”

> append users:leto ” OVER 9000!!”
(integer) 54
> incr stats:page:about
(integer) 1
> incr stats:page:about
(integer) 2

> incrby ratings:video:12333 5
(integer) 5
> incrby ratings:video:12333 3
(integer) 8
hset users:goku powerlevel 9000
hget users:goku powerlevel

hmset users:goku race saiyan age 737
hmget users:goku race powerlevel
hgetall users:goku
hkeys users:goku
hdel users:goku age

lpush newusers goku
ltrim newusers 0 50

keys = redis.lrange(‘newusers’, 0, 10)
redis.mget(*keys.map {|u| “users:#{u}”})

sadd friends:leto ghanima paul chani jessica
sadd friends:duncan paul jessica alia

sismember friends:leto jessica
sismember friends:leto vladimir

sinter friends:leto friends:duncan

sinterstore friends:leto_duncan friends:leto friends:duncan

zadd friends:duncan 70 ghanima 95 paul 95 chani 75 jessica 1 vladimir

zcount friends:duncan 90 100

zrevrank friends:duncan chani

Has a good summary of Big O notation

Bad command:
set users:leto@dune.gov “{id: 9001, email: ‘leto@dune.gov’, …}”
set users:9001 “{id: 9001, email: ‘leto@dune.gov’, …}”

better:
set users:9001 “{id: 9001, email: leto@dune.gov, …}”
hset users:lookup:email leto@dune.gov 9001

get users:9001

id = redis.hget(‘users:lookup:email’, ‘leto@dune.gov’)
user = redis.get(“users:#{id}”)

sadd friends:leto ghanima paul chani jessica

sadd friends_of:chani leto paul

keys = redis.lrange(‘newusers’, 0, 10)
redis.mget(*keys.map {|u| “users:#{u}”})

sadd friends:vladimir piter
sadd friends:paul jessica leto “leto II” chani

Ruby:
redis.pipelined do
9001.times do
redis.incr(‘powerlevel’)
end
end

A transaction:
multi
hincrby groups:1percent balance -9000000000
hincrby groups:99percent balance 9000000000
exec

In Ruby:
redis.multi()
current = redis.get(‘powerlevel’)
redis.set(‘powerlevel’, current + 1)
redis.exec()

Or better yet:
redis.watch(‘powerlevel’)
current = redis.get(‘powerlevel’)
redis.multi()
redis.set(‘powerlevel’, current + 1)
redis.exec()

Bad idea in production:
keys bug:1233:*

Better:
hset bugs:1233 1 “{id:1, account: 1233, subject: ‘…’}”
hset bugs:1233 2 “{id:2, account: 1233, subject: ‘…’}”

expire pages:about 30
expireat pages:about 1356933600

ttl pages:about
persist pages:about

setex pages:about 30 ‘<h1>about us</h1>….’

It has pub/sub. Can it be used instead of/in place of something like Rabbit or ZeroMQ?
subscribe warnings

publish warnings “it’s over 9000!”

config set slowlog-log-slower-than 0

slowlog get
slowlog get 10

rpush users:leto:guesses 5 9 10 2 4 10 19 2
sort users:leto:guesses

sadd friends:ghanima leto paul chani jessica alia duncan
sort friends:ghanima limit 0 3 desc alpha

sadd watch:leto 12339 1382 338 9338

set severity:12339 3
set severity:1382 2
set severity:338 5
set severity:9338 4

sort watch:leto by severity:* desc

A group:
hset bug:12339 severity 3
hset bug:12339 priority 1
hset bug:12339 details “{id: 12339, ….}”

hset bug:1382 severity 2
hset bug:1382 priority 2
hset bug:1382 details “{id: 1382, ….}”

hset bug:338 severity 5
hset bug:338 priority 3
hset bug:338 details “{id: 338, ….}”

hset bug:9338 severity 4
hset bug:9338 priority 2
hset bug:9338 details “{id: 9338, ….}”
end of group

sort watch:leto by bug:*->priority get bug:*->details

sort watch:leto by bug:*->priority get bug:*->details store watch_by_priority:leto

config get *log*

rename-command CONFIG 5ec4db169f9d4dddacbfb0c26ea7e5ef
rename-command FLUSHALL 1041285018a942a4922cbf76623b741e

“type $KEY_NAME” is a good command as well

Image from the Rheinau Psalter, a 13th century manuscript housed at Central Library of Zurich. Image from e-Codices. This image is assumed to be allowed under Fair Use.

Looking at Clojure Koans

I worked on the Clojure koans.

The sixth file had this function:

Then one of the problems was like this;

And the answer was this:

I admit that I copied and pasted it.

I am having a hard time figuring this one out. If you call square, you would do it like this:

But in the answer, “square” is at the end. It seems like to get inline functions with higher-order functions you have to alternate between reading left-to-right and right-to-left. So I will try to parse it myself.

Here is the function:

((fn [f] (f 5)) square)

So let’s get started.

So first we have the inline function:

(fn [f] (f 5))

So we are declaring a function that takes an argument of 5. And so “square” is the argument. I guess that even though the name of the function is a string, that if the string matches a function name then Clojure will realize it’s a function.

So let’s see what might happen with this code runs (keep in mind I have not actually looked at the Clojure source code or tried to run any sort of debugger; I am not sure if Clojure has or needs a debugger).

Here is the function again:

((fn [f] (f 5)) square)

So let’s get started.

Let’s put the argument into the definition:

(fn [square] (f 5)) 

Now let’s put the argument into the function body:

(square 5)

Somehow this exercise is not as enlightening as I thought it would be. I don’t know if this sequence could be used by someone to figure out the koan.

To paraphrase Barbi: Clojure is hard.

 

Frustrations With Environment Variables on Ubuntu

I am making a Rails app that I will run on Heroku. I will use Omniauth and Twitter for authorization, following one of the RailsApps templates.

The Twitter key and secret are kept in the config/initializers/omniauth.rb file. Instead of not tracking that file with git, I decided to use environment variables:

I wanted to put them in a file that I could run before I ran my app locally. It looks like the only file that can load environment variables is .bashrc. I do not want to clutter my .bashrc file with variables that I only use for one app. I can set it at the console:

In my script, I tried every possible combination: With and without the word “export”, with and without quotes around the value, with and without a dollar sign in front of the key. I accepted that I should just put the commands in a text file, and then when I need to use the variables I just “more” the file, and then copy and paste the variables into the console. It took me an hour to finally think of that.