2018-04-08 Update

I think I am done for the time being with the Twitter-retriever.

As of now, it can get the last 3200 tweets of a user (excluding retweets) and capture any tweets going forward. It prints them out in HTML to a file with each tweet in enclosing “li” tags.

I tried to get the WordPress REST API working. I installed WordPress on my laptop, but I had a hard time getting anything to work with the POST command. GET was easy. It is the authentication that stumped me. I got a couple to work with curl, but no POST commands to work with clj-http. I might try again later, but for now I am done. It seemed to be a rabbit-hole of google searches, adding yet another plugin, editing the .htaccess file, etc, etc. For now, what I have is good enough.

I might look into an alternative to environ, but for the time being I think it will do.

I don’t know what I will work on next. I might get back to Simply Scheme. I might go back to coding examples for the Clojure API. I might go through some lessons at Purely Functional. Maybe TensorFlow (like with guildsman or clojure-tensorflow or tensorflow-clj). Maybe Luminus. As of right now, I have not decided.

You’re welcome.

2018-04-01 Update

I am still working on the twitter retriever.

I am almost ready to start looking at the wordpress API. I think I will start a local instance so I can work with the REST API and not clutter up this site.

I also looked a bit more at the Clojure library environ. What I like about it is that it gets all the environment variables at start time, and any old time you want you just call the env function, and it works. No fuss, no getting files in every namespace. But I am not so sure if it would work in a production environment. There are a lot of pages from people saying that putting usernames and passwords in the environment is a bad idea, and I am leaning that way too. I have not been able to get my app to use environment variables in .lein-env, profiles.clj or .boot-env when I run it with “lein run”.

I also found out that (after some trial and error) that I cannot compile my app with “lein uberjar”. I use environ for my database creds, and it looks for them during the uberjar process. I was not able to use the profiles.clj or .boot-env for this either. I had to put the database variables into the environment when I compile. (So maybe there is not security risk during production after all.)  Here is what I did in bash:

That is not a production database, so no big deal if the creds are here. But that seemed to do the trick. I was able to use the uberjar in another window without any problems (from what I remember).

Also: Eric Normand from Purely Functional got back to me. I told him I was nowhere near done. Partially due to not spending as much time as I should, partially parsing the infamous error messages, partially due to find and trying out libraries as I get this up and running. He wants to do a code review. I think I will clean up a few things before I say yes.

Before I do, I would like to copy and paste in here a few functions for the sake of posterity:

Those should parse a string with a timestamp like “Tue Feb 11 15:03:45 +0000 2014” and convert it to “2014-02-11_15:03:45”. I then found clj-time.

I also think I found a couple of function calls you can try in the REPL to interactively work on regular expressions:

I know those are pretty wide, but I found it helpful.

Here is just a little let:

 

You’re welcome.

2018-03-25 Update

I am still working on the Twitter retriever. There is not much too else to tell right now. I am just working on some tests.

You’re welcome.

Addendum:

I got this message in the tests:

I had a var called “result-str”, and I had misstyped “result-string” in the comparison. Usually that is what the cause when you get a stack trace with “clojure.lang.Var$Unbound” in it somewhere.

I do agree that sometimes the stack traces can be hard to read.

2018-03-18 Update

The “due date” for my Clojure project is tomorrow. I am not done.

Part of it is because I did not get started until a week into it. And I have been using “Organic-Oriented Development” (a new form of OOD): I did not plan a whole lot out ahead of time. I have just been trying different APIs and libraries to see what they do. I looked at propertied, but I don’t think I will wind up using it. I may use environ, but I have not decided.

I am working on some of the tests now. The author of the Twitter API library tells you to set his API keys as environment variables and run the tests against one of his Twitter accounts. I am not a TDD-purist, but I think that is not the best way to go. Running a database for tests I have no problem with, but needing something external to be live is a big no-no to me. On the other hand, I don’t think I know enough about any Clojure testing APIs to say what is a better way to do it. Maybe the author is correct.

I have not looked at the WordPress JSON API. I think for the time being I will just write the tweets in HTML format to a file, and copy and paste.

I have also started getting back into meditation, so perhaps that will help me to be more disciplined.

I did mention this to one of the organizers of the Austin Clojure group. I did not volunteer, but I should probably have something working by then in case they need someone to present.

You’re welcome.

2018-03-04 Update

I finally got started on the Twitter retriever app in Clojure. I did not do much the first week.

I am able to get tweets for a couple of accounts. I have to make database tables. I think for the most part I will just follow the structure of what Twitter sends back.

I was dealing with a few libraries to deal with configuration. Important stuff. For the database properties and twitter keys, I am using envron. It uses environment variables for configuration. One of the ideas in the 12 Factor App pattern is to put your configuration in the environment. I did some googling, and not everybody thinks that is always a good idea. Plus, what if you have a LOT of configuration?

In non-prod, environ can take variables/fields from a profiles.clj file if you are using Leiningen. So at first I thought environ sidestepped all the security concerns. But for prod, the Github page says to just use environment variables. For now I will go with this as-is.

What I like about environ is that it should be pretty easy to use for the database configuration. I might use cprop. I will look at it later.

I might stick with propertied for some configuration.

At least now I know a few more libraries.

You’re welcome.

2018-02-25 Update

On the Groovy mail server: I am still unable to send to ProtonMail. I might just shelve it for the time being. I would like to get back to it in the future, but I would like to spend more time on Clojure and the Lisp family.

I will get around to installing iRedMail soon. I can’t run an old version of Apache James forever. I think iRedMail uses Postfix under the hood, which is what ProtonMail uses. Perhaps I will get it all working them.

I had my call with Eric Normand of Purely Functional. At one point we touched on ClojureScript. I have been under the impression that you have to deal with JavaScript to use ClojureScript. As the guy who wrote the Pollen DSL for Racket put it, Lisp makes me smarter, and pretty much any language ending in *Script seems to kill brain cells on contact. There seems to be a lot of wheel re-invention in JavaScript. I always get the feeling that a lot of people in the JS community think that there is nothing to software besides JS. Has any other language been called “stupid” by its creator? And it all seems unstable.

Back in June 2015, Bloomberg wrote this:

JavaScript is fast-moving right now. Too much of what you know today will be useless in six months. Every hard-fought factoid about the absolute best and most principled way to use the language will be fetid zoo garbage by the end of the year. And some sniveling, bearded man-toddler will be looking slightly to your right with his pale, buzzword-infected eyes and awkwardly mumbling, “Yeah, no, wow, it says you have a lot of Gulp and Angular, but I’m guessing you don’t use Fleejob or Grimmex with the Snurt extensions? (Long sigh.) I’m just not sure if you’re gonna like working here.”

Maybe it’s better now, but they thought they were smarter than the rest of us while making their ecosystem a total morass.

Anyway, Mr Normand said he used JavaScript several years ago, and he had to use it again recently, and the libraries and ecosystem was totally different. But in four years, ClojureScript has not really changed.

I mentioned that I do not like 4Clojure. I tried it a few times, and I keep finding myself in a position where I am randomly scanning the Clojure Cheat Sheet looking for a function that might help at that moment. He said a side project is a good way to learn a language. He asked me if there is a problem I would like to solve, and I thought of one.

I am starting on a project to publish tweets to a webpage. I used to use a plug-in called twitter-tools that once a week would take all my tweets and make a WordPress post. It is old, and does not work with more recent versions of WordPress. (I think the developer passed away from cancer a few years ago.) I looked recently and I could not find anything else that has the same functionality.

There are a lot of plugins and widgets that will put your twitter stream on the side. In fact, I think Twitter also has a script for that. The other major category is sending a tweet when you make a blog post.

Granted, you can log into twitter and download your archive, but it would be nice to have something automated.

We will see what I can get done in three weeks.

You’re welcome.

The editorializing is by me, not Mr Normand.

2018-02-18 Update

I spent a bit more time on the Groovy Mail Server. I am still unable to send to ProtonMail, which uses Postfix. I opened up a ticket and sent them a log from tshark, but they were not able to find anything wrong. A lot of people use Postfix. Not being able to send to Postfix might be a show stopper.

I will install iRedMail; the purpose is two-fold: 1. Try to get Groovy Mail Server working with Postfix (and later TLS), and 2. And just to try it out in general. I am currently using Apache James. The version I am using is getting old, and it is time to look at other options. I might go with a newer version, but I figured I should try something else too. Eventually the version of James I am using might not work on a future JDK.

I posted a command that you can use to generate passphrases on a Linux laptop. I also added a page about the core Clojure API. Slowly but surely I am making progress. I might look more at Simply Scheme, but I plan on focusing more on Clojure.

You’re welcome.

Command Line Passphrase Generation

I figured out another way to calculate passphrases, this time using the command line.

To cut to the chase, here is the command:

I found out about a command called “shuf” which according to its man page can “generate random permutations”. You can run this against a word list. Many Unix-based systems have a word list.

On my system, the word list is at /usr/share/dict/words, which is a link to /etc/dictionaries-common/words, which itself is a link to /usr/share/dict/american-english. You can also find one online here.

After you run the list against “shuf”, you need to put them all on one line. You can do this with the “tr” command (see man page here). You can use it to “translate or delete characters”. I want to replace the carriage return-new line combination with a space. In the command, we can represent that combination with either ‘\r\n’ or [:cntrl:]. I prefer [:cntrl:] because the only way to represent the space is with a space between single quotes, and I think using one less pair cuts down on ambiguity (“tr [:cntrl:] ‘ ‘” as opposed to “tr ‘\r\n’ ‘ ‘”).

Even though the word list is “american-english”, it still has some characters that we do not use, like vowels with umlauts or the letter “A” with a circle over it (I guess that is a “super-umlaut”). There are also a lot of words in there with an apostrophe followed by an “s”, words with upper-case letters and words with less than 4 characters. Taking out all of that takes us from a file with 100K words to one with about 26K words.

Here is the full grep sequence I used:

You’re welcome.

2018-02-11 Update

This week there was not much Clojure. I have the call with the Purely Functional guy next Monday. He has a conference this Thursday and Friday. I am not attending. I think a few people from Austin Clojure are going.

I spent more time on the Groovy Mail Server. I just don’t know when to quit. I cannot send mail to Proton Mail. I think they are using Postfix. Every time I try to send an RCPT command, they return “550 5.5.1 Protocol error”. If I am reading the Postfix source code correctly, that can happen if you send an invalid command (RCPT is valid) or if you send a new line either by itself or without a carriage return. I do not think I am sending a command in the wrong form. For one thing, just about every other server I try accepts my mail.

I also had problems with Outlook.com. I was getting a socket exception. From what I gathered from googling, they were ending the socket after the QUIT command. I was not having this problem with Outlook.com mails before. I added a try/catch block, and now I save the emails.

I don’t know how much longer I will continue the Groovy Mail Server. Part of me wants to complete it. On the other hand, since Clojure and Lisp are the ultimate direction I want to go in, maybe I should put it aside. On the first hand, it is always good to keep up with a few extra languages. I would also like to get through Simply Scheme and then SICP, because that would make me a really smart person.

If I really wanted to finish my mail server, I would probably have to get another VPS account and set up something like mail-in-a-box for TLS. I might need a few attempts to get TLS right, and I don’t want everybody banning my emails while I figure it out. There also might be an issue with TLS. I don’t know a whole lot about setting up certificates or how to set up a certificate keystore in Java or how to use keytool. I have Let’s Encrypt certs, but those only last for a few months. I don’t know if you can re-import a cert into a Java app while the app is running. Sometimes it seems like there is not a lot of overlap between security and development.

You’re welcome.