The other day, I needed to quickly analyse a data set that came in form of a large CSV file. I wanted to collect a particular column of that table and collect all entries categorised by a key in another column.
A simplified version of the table could look like this:
Key
Value1
Interesting_Value
Other_Value
foo
17
23.5
X
bar
21
1.75
Q
foo
42
12.6
B
baz
27
17.8
F
bar
49
47.2
K
I strived for something like this:
result = {
foo: [23.5, 12.6],
bar: [1.75, 47.2],
baz: [17.8],
}
Iterating over the rows is easy, and getting to the columns is no problem either: The CSV gem is well documented and supports this easily.
A nice way to accumulate data is Enumerable#each_with_object. Since I wanted the result to be grouped by a key value, I’d pass a Hash as the initial argument.
Step 1: each_with_object({})
However, since I’ve planned to append values for changing keys, the default value needed to be an Array, not the default of nil.
Step 2: each_with_object(Hash.new([])
This, however, returns the same empty Array, when a key isn’t found, but I wanted a new empty Array:
Step 3: each_with_object(Hash.new { [] })
This executs the block every time a default values is needed (i.e. the given key isn’t yet in the Hash).
The next step is to append the value found in a row to the (potentially new and empty) Array for the given key.
I thought it would work this way:
data_table.each_with_object( Hash.new { [] }) do |row, acc|
acc[row['Key']] << row['Interesting_Value']
end
But, no, the result of this code is an empty Hash! It needs to be the <<= operator to work, as shown in the snippet of a pry session:
It seems to me, that the Hash lookup with the given default value [] returns an Array, and the append operator << does in fact append the passed object to that Array, but then the result of that does not end up as a (new) value fo the given Hash key. In contrast, the <<= operator does assign the result of the append operation.
While working on the e-books I published on LeanPub, I have developed a number of useful approaches to get fast(er) feedback on how the book looks. Two earlier blog posts describe some of this:
While this provides a nice feedback cycle, sometimes I like to generate a new preview without pushing to the GitHub repository I am using to share the book content with LeanPub. This happens, when I change settings on the Leanpub site that affect the generation of the book (the title image, font faces and sizes are set via the book’s pages on LeanPub, not a configuration file in the repository).
While I could click thought the UI and navigate to the page where I can generate a new preview, I prefer using a command line tool from my local machine: Rake
To use the LeanPub API, an API key is needed. The link to the API documentation above has information where to get that key. To easily use this API key, I store it in an environment variable LEANPUB_API_KEY.
The Rake Task Definition
In the repository of my book I have a Rakefile containing the task definitions. Here’s the one to trigger the generation of a preview on Leanpub:
require 'rest-client'
namespace :leanpub do
LEANPUB_BASE_URL = "https://leanpub.com/<book_id_on_leanpub>"
namespace :preview do
desc "Generate new Preview on LeanPub"
task :generate do |t|
what_to_generate = t.name.split(':')[1]
url = "#{LEANPUB_BASE_URL}/#{what_to_generate}.json"
begin
RestClient.post url, api_key: ENV['LEANPUB_API_KEY']
rescue RestClient::Exception => e
puts "Got error #{e.message} in", caller.first
exit 1
end
puts "Generation of preview was triggered"
end
end
end
I can now easily generate a preview, without having to leave the IDE I’m using to write the book (or the command line) using this:
$ rake leanpub:preview:generate
Generation of preview was triggered
The Rakefile will likely change, for example to also support publishing a new version of the current ebook, or to make it more flexible in order to handle different ebooks.
Update (8. Jan 2021): It turns out that this is really useful: LeanPub provides a web hook to generate a _sub set_ of the book that also generates the PDF (but not the ebook and mobi file). This saves some time from pushing to GitHub to being able to review the generated file. I now use this web hook most of the time.
I now only generate for full book in all formats when I want to check that the book looks good enough to be published. Since this happens less regularly than pushing to the repository, I use the Rake task.
At the “London Tester Gathering Workshops 2015” (also see #LTGWorkshops on Twitter) I offer a workshop “Fast Feedback Loops & Fun with Ruby”. For this, I wanted to give attendees a handout, to make applying the stuff covered easier and it was planned to be a list of brief recipes.
Suffice to say that the handout grew (and it’s entirely possible that this is ‘feature creep’ in action). In fact, it grew to the point that I decided to turn it into an ebook. The book is not done yet, there’s some copy editing to do.
My workshop at the London Tester Gathering Workshops 2015 is announced now! They’re offering an early bird rate until the 18th February, by the way. Find the abstract on the conference page or just read ahead. 🙂
Fast Feedback Loops & Fun with Ruby
Ruby is “a Programmer’s best friend”. Let’s use Ruby to get feedback – including getting feedback automatically – when working on projects. Whether it’s about transforming source code into test results (a.k.a. running automated tests) or generating image files from raw data, Ruby can be used to automate these tasks. Furthermore, it can also be used to automate actually running these tasks, e.g. upon saving a file to disk. Does that sound like a good idea? This session is for you.
I regularly bump into tasks that are…
tedious, if done manually
not done often enough, unless automated
still not done often enough, unless running them is automated, too.
In the workshop we’ll combine some Ruby tools to remedy this situation. In particular the workshop will cover:
Writing a simple Ruby program that does something useful, e.g. turn a markdown file into HTML
Wrapping that in a Rake task
Automate running the task
Knowing how to do this is useful, not only for projects using Ruby as their primary language, but can be handy in all projects.
What is expected:
Some Ruby knowledge; you don’t have to be an expert or anything like that.
A notebook (or tablet) with an internet connection & Ruby installed.
Cool if you’re using RVM, rbenv, chruby or similar
Mac OS X, BSD; Linux & friends are fine, Windows may be a bit problematic.
I’m sure we’ll have a couple of exiting days talking about software testing. And not only talking but also some hands-on stuff using Ruby for fun and (fast) feedback.
Before I publish more information about my workshop, I’d like it to…
After two very exciting and enjoyable days, the Barcelona Ruby Conference (2014 edition) is over, so here are a few things I’ve learned about and/or liked. I’ll mention these in no particular order as long as the memory is still fresh.
Barcelone Ruby Conference 2014
Over one of the lunches I discussed ways to improve coding skills online with some people at my table. For those interested in mathematical puzzles I suggested to have a look at Project Euler, at site that offers interesting and sometimes very hard to solved problems. Someone recommended rubeque, where you can “hone your Ruby skills by solving small Ruby problems while competing against other Rubyists.” Then there is Ruby Koans, where you can learn Ruby by making test cases pass.
Erik Michaels-Ober showed how to write fast Ruby. His examples showed that the faster code can also be the code that’s easier to understand. I hope that he’ll put his slides online sometime, because a) they contain all the examples he presented and b) they were very nicely illustrated.
Leon Gersing‘s presentation “Keep Software Weird” was inspiring, funny and, well, Zen. A quote from his talk: “Code is a living representation of who you are, right now.”
I also really enjoyed talking to the people form KeepFocus (including, but not Limited to Jakob, Palle and Jan).
Thank you to everyone involved in making this conference so enjoyable and exciting!
