The dumbest gift I've ever given

Like most good ideas, this one started in the shower. It was about 4 weeks until my friend Dylan's birthday and I wanted to get him a gift, but not just any gift. There were a few qualities it needed to have.

• Stupid
• Embarrassing
• Thoughtful
• Not something he can throw away without feeling guilty

So whist indulging myself in the pleasure of shampooing one's hair I was pondering as to what possible gift would satisfy these requirements, my thinking was to somehow incorporate Dylan himself into the gift.

We had previously worked together and he was always one to "prank" coworkers, adding GIFs of them to slack, printed t-shirts of their faces that kind of deal, another thing he loves to do is talk shit on Twitter.

And that's when I got the idea, why not get all his tweets printed, but not just on my cheap Canon inkjet it needed to be more substantial, more permanent, I didn't just want them printed I wanted them printed & bound! a first edition hardback copy of his timeline.

And this is the tale of how I did that.

How do you print a book?

Now I've decided on the gift I'm immediately faced with some glaring problems.

1. How do I obtain a copy of all his tweets?
2. How do I format that as a book?
3. How do I print a book?

I was pretty confident the first 2 could be solved with some good old fashioned software but the third was entirely new to me, I started googling.

My first idea was to just find some local print-shop and see if they can do it. They can't. The most they can do for me is a stapled together booklet, not good enough.

So after a bit more searching I find a few "publishers" that print books for self-publishing authors. Of the "publishers" that I found most of them had minimum order quantities in the 100s which was no good as I wanted a single book, the ones that would print a single book were expensive.

After more searching I got another quote, this seemed more reasonable to me. I was so far down this rabbit hole that the joke gift was now worth $60.

I found other services that would provide cheap printed "proofs" but most of these had restrictions on formats or sizes, lots of others also have this expectation that you're a serious individual publishing a serious book with things like ISBN numbers and all sorts of other fees and requirements attached.

Anyway, problem number 3 solved!

Terms and Conditions

So, I think I know how to print a book but now I actually need to "write the book". I had the assumption that I could produce the book as a PDF and all my research into book printing confirmed that assumption was correct but I couldn't just "print" his Twitter profile to a PDF. That was not my vision.

I needed to get a copy of all Dylan's tweets, ideally in a format that allows me to actually do something with them, that something being make a PDF.

Usually, with this kind of problem you go looking for an API to access the data, Twitter has an API but much like some of the book publishers, there are many asterisks appended onto the list of its capabilities which when inspected, brings you to the realization that you cant do what you want to do unless you pay someone exorbitant amounts of money.

Modern problems require modern solutions

-- Dave Chappelle

Exit Twitter API.

Enter TWINT.

TWINT is a pretty cool project, one I wouldn't have been aware of if it wasn't for this escapade. It's basically a web-scraper specifically designed to grab every last ounce of information out of publically accessible Twitter, It was exactly what I needed. It could pull every tweet Dylan had ever posted with all the metadata and output it as JSON.

After a few minutes of casting some incantations with the terminal, I was now in possession of a 5.37MB JSON file detailing every tweet Dylan had ever posted.

6,557 JSON objects each one looking something like this.

Copycopy code to clipboard
1{
2  "id": 1226732665098920000,
3  "conversation_id": "1226732665098919936",
4  "created_at": 1581310833000,
5  "date": "2020-02-10",
6  "time": "05:00:33",
7  "timezone": "UTC",
8  "user_id": 14075725,
9  "username": "icerfish",
10  "name": "WarSynth",
11  "place": "",
12  "tweet": "It's honestly just wonderful that Parasite won best picture (among other awards) and that everyone is just fucking jazzed about it.",
13  "mentions": [],
14  "urls": [],
15  "photos": [],
16  "replies_count": 0,
17  "retweets_count": 0,
18  "likes_count": 2,
19  "hashtags": [],
20  "cashtags": [],
21  "link": "https://twitter.com/icerfish/status/1226732665098919936",
22  "retweet": false,
23  "quote_url": "",
24  "video": 0,
25  "near": "",
26  "geo": "",
27  "source": "",
28  "user_rt_id": "",
29  "user_rt": "",
30  "retweet_id": "",
31  "reply_to": [
32    {
33      "user_id": "14075725",
34      "username": "icerfish"
35    }
36  ],
37  "retweet_date": "",
38  "translate": "",
39  "trans_src": "",
40  "trans_dest": ""
41}

Typesetting

Printer acquired, now to get on with actually producing a PDF out of all this data. I had done some work in the past generating PDFs programmatically, specifically creating PDFs of certificates as part of a backend for an insurance product. The approach I used for that was to generate the documents with HTML and convert that to a PDF using a headless browser.

I started with this approach but ultimately decided I needed something better, I can't remember exactly what went wrong but I seem to remember it being something about controlling where page breaks were and margins.

Then I turned to React, there's a library for rendering PDF's with React that looked promising react-pdf. No doubt I could have done what I needed with this but dealing with Webpack and all the other baggage that comes with JS was not what I wanted to deal with

I briefly also looked at pdfkit which is another library for creating PDFs in Node.JS

After messing around with JS libraries for a while and not making much progress or reaching roadblocks, I started looking at solutions using Go rather than JS

There were 2 options I found

• UniPDF
• GoFPDF

The former had all the features I needed but comes with a price tag of $3000. That was way out of my price range so I settled on the latter.

I got to work generating the book's interior pages.

I had never considered all of the complexities in doing the layout for a book before, one of the biggest revelations was that you need to account for the spine of the book with your margins and that obviously, the spine is on alternating sides of the page.

So to manage this, I need to change the margins of the page after each page break, which means I need to control the page breaks manually. The way I did this is horrific.

Copycopy code to clipboard
1_, fus := pdf.GetFontSize()
2pw, ph := pdf.GetPageSize()
3tl := len(pdf.SplitLines([]byte(tweet.Tweet), pw))
4imgPxHeight := CalculateImageHeights(pdf, images, 0.6)
5if pdf.GetY()+(float64(tl+1)*fus)+23+(pdf.PointToUnitConvert(imgPxHeight)) > (ph - PAGE_BREAK_MARGIN) {
6  AddPage(pdf)
7}

I have no idea what any of these magic numbers mean anymore, but it tries to calculate how much space drawing an individual tweet in the book will take and if it exceeds a set boundary then it adds a page break.

This is somewhat counter-intuitive compared to a regular book because some tweets have images and I wanted the book to be in chronological order it ends up in this situation where some pages have a single tweet printed on them because the subsequent tweet has a huge image attached to it. It was good enough.

Twitter has emoji's and unfortunately, they don't just get rendered into the text cells magically, So I need to find all the emojis in a piece of text and calculate their co-ordinates and render them as images individually. I decided that doing this would be easier if I split each tweet into "words" and manually place each word on the page. This process looks like this

Copycopy code to clipboard
1_, us := pdf.GetFontSize()
2words := strings.Split(tweet.Tweet, " ")
3wordMap := make(map[string]struct{})
4for idx, w := range words {
5  emojis := emoji.FindAll(w)
6  for _, e := range emojis {
7    em := e.Match.(emoji.Emoji)
8    DrawEmoji(pdf, strings.ToLower(em.Key))
9    pdf.SetX(pdf.GetX() + us)
10  }
11  mentionExp := regexp.MustCompile(`^\@[a-zA-Z0-9_]{1,15}$`)
12  if mentionExp.Match([]byte(CleanText(w))) {
13    pdf.SetFont(SERIF_FONT, "B", 12)
14  } else {
15    pdf.SetFont(SERIF_FONT, "", 12)
16  }
17  pdf.Write(5, CleanText(w))
18  if idx < len(words)-1 {
19    pdf.Write(5, " ")
20  }
21}
22pdf.Ln(10)

This breaks a tweet down into words, iterates over them and writes each word onto the PDF, the GoFPDF library keeps track of your cursor position as you write onto the document which makes this process less painful than it would be if it didn't.

There are some utility functions in here to make drawing emojis easier and to clean out nasty characters from the text if they aren't present in the font file. Detecting emojis was made easy thanks to this lib

Twitter also has images and videos, luckily the data dump I have contains links to these assets, so I download all of them. Twitter also generates thumbnails for videos so I use those as well. This amounts to 496MB of data.

I'm printing this book in mono rather than color so I want to convert all these images to black & white. The GIFT (Go Image Filtering Toolkit) library makes this simple.

Copycopy code to clipboard
1g := gift.New(gift.Sigmoid(0.5, 7), gift.Grayscale())
2dst := image.NewGray(g.Bounds((*img).Bounds()))
3g.Draw(dst, *img)
4return dst

I experimented with a few different chains of modifiers & effects and this seemed to work out the best for a variety of images. Preserving a decent amount of detail whilst having a good level of contrast.

Making this worth my while

At this point I had a somewhat passable PDF for printing, I messed around with it a bit more adding finishing touches, deciding on fonts, adding page numbers and title pages.

But I had now spent quite a number of hours getting this all working and I had committed to spending $60 on getting this printed. This is the point when I start thinking "Is this software worth anything to anyone else" and "How could I make this into a product". I get sidetracked and start researching print-on-demand services for books.

That's when I come across Lulu xPress. They have an API that allows you to upload a PDF and get a printed book in the mail. Well shit! this is what I was looking for!

The cherry on top, they are half the price of the previous printer I had found, this book was now only going to cost me $25

Now I'm building a book printing service 🤦

Wrapping up

My book needs a cover, I design one in Photoshop and call it a day.

If I want to sell these books though, I need to be able to generate these covers. Making them by hand in Photoshop is not what I want to spend my time doing.

I start writing more PDF generation code to generate book covers.

Delivery

I now have a PDF for the interior of the book and another for the cover. I submit these files to the Lulu API, pay the invoice and, wait.

A week or so later, a package arrives.

I was relieved to find the book had printed correctly, I was dreading having to debug a process that requires shipping a 400+ page book of your output to your door.

There is something very satisfying having a physical object as the output of your code, it's a feeling that's hard to describe but I'd say it's very similar to the feeling you get when starting to learn to program and you compile & run your first "Hello World" program.

Hours of effort and problem solving culminating in this book arriving at my door, its contents bearing so little resemblance to the code horrors that produced it.

The software that produced this book has so many bugs and nasty edge cases. But the book itself, well the only bugs it has are the ones that live in the dust collecting on top of it.

Birthday

Due to the COVID-19 pandemic, presenting this gift to Dylan was delayed by a few months until the lockdown restrictions in the UK had been eased and we could meet outdoors again.

Here's the book being given to its rightful owner, its author.

He later told me

This is probably the most thoughtful gift I've ever received and I hate it.

Based on my initial requirements, I think that's a good review. And he's given me permission to post this.

Your own book

I'm still working on getting the software sorted out so that anyone can get one of these books for their own Twitter accounts, when it's ready I'll post about it on here.