Automating My Now Page
This post outlines how I automated my now page(s) but I should start with what a now page is. From nownownow.com:
a website with a link that says “now” goes to a page that tells you what this person is focused on at this point in their life. For short, we call it a "now page"
I had heard of now pages a long time ago and always meant to add one to my site but just never got round to it. A few weeks ago omg.lol launched their now page feature which made me think about this again (and I want to keep both my personal site and the omg.lol now page up to date and in sync). One problem with a page that should, in theory, be updated regularly is I will definitely forget to do it. Automation to the rescue! Or more specifically a veritable grab-bag of python and node scripts, web scraping, and APIs.
If you just want to see the pages here are the links:
Overview
For each service I'm pulling data from I'm doing mostly the same thing:
- Fetch some data from somewhere
- Format that data to make is easier to work with
- Add it to
data.json
I then call now/generate.js
which goes through each key in data.json
and outputs it to two markdown files: now-omg.txt
(for omg.lol) and now-web.txt
(for my website). The web version skips the omg.lol-specific syntax and intro section.
For omg.lol, I wanted to append icons to each line so they display as the bullet on my omg.lol now page. I grabbed icons for each section from Font Awesome and apply them as I loop through item in a section:
const text = `Some text from an item ${getIcon('making', 1)}`
const getIcon = (key, index) => {
const icons = {
making: ['microphone-lines', 'laptop-code', 'terminal', 'code-pull-request', 'bug'],
reading: ['book', 'book-bookmark', 'book-open', 'book-open-reader', 'bookmark'],
music: ['headphones', 'radio', 'guitar', 'compact-disc', 'drum', 'sliders', 'volume-high'],
podcast: ['headphones', 'microphone-lines', 'comments', 'tower-broadcast', 'podcast'],
}
return icons[key][index]
}
Getting the data
Music
This was a nice easy one: show the albums, tracks, and artists I've been listening to the past seven days. Last.fm have a decent, if slightly oddly designed, API to get this data.
Not so easy is that Last.fm no longer return artist or track images. To get around this I do a lookup with the MusicBrainz which gives me a link to AllMusic which I then scrape and grab the main image with Cheerio.
I also use text-to-image
to generate images for artists and albums where I'm unable to get one either from Last.fm or AllMusic:
await textToImage.generate(`${album.name} by ${album.artist.name}`, {
maxWidth: 200,
customHeight: 200,
textColor: 'white',
textAlign: 'center',
margin: 20,
bgColor: 'black',
verticalAlign: 'center'
})
Github, Books, and Statuses
These are all straight-forward API calls with a slightly manually process of tagging projects on GitHub with now
- I don't want every recent project showing up and GitHub doesn't have an API for star lists yet.
Games
Like the missing artist images, this involved scraping a page because PSN doesn't have an API. I scrape my PSN Profiles and use Cheerio to grab the latest games I've got trophies for. It's not a perfect system, but it's likely the current game I'm playing will be the top of that list.
Overcast
This export isn't a secret, nor are the scripts to download it but don't hammer this endpoint. I'm downloading once a week. I have emailed Marco to ask if he'd implement an RSS feed of listen history so this isn't required.
This one was trickier but doable. Overcast offers an "All Data" export but you need to be logged in to access it. And it's in OPML format. So this is a two step process:
- Use this python script to login and download the export
- Run this node script to convert the OPML to JSON and then to a valid JSON feed
One thing I noticed is that Relay's member show feeds show the episode and show URL as relay.fm/membership
so I had to handle this myself. Example here with Connected Pro:
const isConnectedPro = podcast.title === 'Connected Pro'
let episodeUrl = episode.url
if (isConnectedPro)
{
const episodeNumber = episode.title.split(':')[0]
episodeUrl = `https://relay.fm/connected/${episodeNumber}`
}
From this data, I take the last 30 episodes, group them by show, and rank them by episode count which then becomes the podcasts section of the now page.
Other
For the "about" and "tv" sections, I wanted to be able to update these from anywhere so I opted to dump some yaml in an omg.lol pastebin and fetch it from there.
Syncing the changes
Each data source doesn't need to be updated every hour or day, so I split them out into a makefile
for jobs to run on a Monday (music, books, games), a Friday (Overcast), and daily (about, tv, github projects). (I could have done these all as separate cron jobs but it was easier to split them in a make file and just have three scheduled jobs instead).
Each time a job runs, it updates the data.json
file so I can run generate.js
and update.js
at any time throughout the week to update my now pages. The update
script posts the contents of web-omg.txt
to the omg.lol API and sends a webhook to Forge to deploy my main site.
All of the code for this is on GitHub if you want to do something similar.
@robb Nice! I was just thinking about if/how to share `/now`s across multiple places.
@robb ICYDK: MusicBrainz and Internet Archive run https://coverartarchive.org. You can throw MBZ release ids at it to get artwork urls back. May be preferable to scraping
Cover Art Archive@edwellbrook I did not, this is very useful - I knew I should have asked you about this! Thanks!
@robb drop me a message any time!
@robb This is just awesome! The more now pages—and associated tools—can gain traction, the better.
@edwellbrook Interestingly the album I initially had the problem with isn't on cover art archive either. I notice there's basically no good images of it on Google either.
@robb submit it to help the next person? 😄
@robb it can be hit or miss though. quality varies a little too
This is great, thanks for sharing!
@robb this is great, nice work!
@dominickjay Thanks!
@robb also, unless I’m wrong with the use case and how your doing the lookup, isn’t the track artwork available under track.getInfo? Thought I remembered having it working myself at some point, so if the docs are outdated then I stand corrected! https://www.last.fm/api/show/track.getInfo
API Docs | Last.fm@dominickjay It does return images but they’re all placeholder images sadly.
@dominickjay I stand corrected, `track.getInfo` does sometimes give artwork but it seems very inconsistent.
@robb Thank you for giving me the kick to pull my Last.fm listening to my now page… Have been meaning to do that for a long long time!
https://jamesdoc.com/now/
James Doc - Now@jamesdoc Yes! This looks great!
@robb ah yeah that’s interesting. Oh well, at least you’ve got a good solution for getting round it!
@robb i really love this. i wish there was a better way to automate tracking games I've played across platforms though!
@robb this is awesome. I love the idea and the design of the page itself, but the automation of it is genius! Such a cool way to keep it up to date.
@robb ooh overcast. I’ve wanted that and never even thought to look at that export. Good stuff here!
@robb @sophie @jaredwhite @j9t @matthiasott that’s very cool!