Lately, I’ve been working on a little project. It had kind of morphed in the year I had been working on it.
After migrating this site off of its bare-metal server sitting in my office and onto its beautiful VPS home over at DigitalOcean, I was left sitting with a small potatoes Linux box that was just collecting dust.
I then had a conversation with Bifuteki, a popular streamer of various fighting game tournaments this side of the Mississippi (I think), I got the idea to repurpose the server into something else: an RTMP server!
Now, attempting such a task was definitely not going to work because using a Intel Celeron CPU released prior to 2014 (MajorTower was built in 2014 and I was building something cheap) wasn’t going to encode anything anytime soon.
So, I sat on that issue for a few months…
During that time, other ideas started kicking around. The RTMP idea had taken a back seat to the Steam Cache. I figured if I built a system with enough storage, I could install all my Steam (and other PC) games onto one system and download them with the quickness from within the home.
So, I bought some parts and set off to building…
Unfortunately, that didn’t work out too well. While all the software was in place, the performance I was seeing just wasn’t there. It turned out, while there weren’t any official specs to get this project up and running, it was deeply implied that you would need some beefy hardware to get it off the ground.
So, I decided to pivot once again. Seeing as I built the thing to handle a whole lot of data, I figured I’d turn it into a file server so I can securely share things with my wife without needing to use a flash drive, which we have yet to do.
So, until then, I decided to turn it into a Plex server. While I don’t have many movies or TV shows stored anywhere to take advantage of those systems, I noticed that I was downloading and watching a ton of content from Giant Bomb. They allow you to download their video content directly from the site, so I began saving videos I had been downloading to Plex so I could watch them from wherever I was in the house without the need of having to broadcast it to a device from my laptop or be restricted to my room so I can watch it from my PC on my mounted TV.
That led me to using their API to start polling their site to see when new content was available and began downloading it directly to the Plex server so I wouldn’t need to do a download and transfer from any other device. All I would need to do is fire up the Plex app, and the videos were already there waiting for me.
That brought me to another idea. At work, I was having an issue with watching YouTube videos in my downtime. Specifically those that loosely pertained to gaming as I watch a lot of technical videos for PC hardware. You know, LTT, JayzTwoCents, but especially Digital Foundry. While some I could take it or leave it, I most certainly wanted to keep up with game performance analysis whenever I could. So, I hatched up a plan to get them in a similar manner to the Giant Bomb videos!
This was actually two fold. First, I needed a system to figure out when a new video available. To somewhat determine that, I found an API endpoint that’s used by streamers to highlight when their latest video on YouTube thanks to CrunchPrank. He created an endpoint that provides information for chatbots to use in custom commands. The one I used provides the name and URL for the latest YouTube video of a specific channel.
https://api.crunchprank.net/youtube/latest_video?user=USERhttps://api.crunchprank.net/youtube/latest_video?id=USERIDThe top endpoint will search based on a channel name while the bottom will search based on the channel ID. I’d like to say you can choose one or the other, but isn’t that easy as some channels do not work with just the USER so you’d have to track down the ID in order to get it to work. Here are a couple of examples from my script:
https://api.crunchprank.net/youtube/latest_video?user=Jayztwocentshttps://api.crunchprank.net/youtube/latest_video?id=UCm22FAXZMw1BaWeFszZxUKw
(ID of Kitboga)By simply curling these endpoints, you the name and URL of the latest video on the channel.
$ curl https://api.crunchprank.net/youtube/latest_video?id=UCm22FAXZMw1BaWeFszZxUKw
52 Hackers Were Found On My Network (by scammers) - https://youtu.be/0IISRrrfrxMNow, I needed a way to get the YouTube videos onto the server. Enter YouTube-DL. It is a command line application written in Python that downloads videos from YouTube along with several other sites. All you need to do is run the following:
$ youtube-dl <youtube video URL>So, now, we have a method to download YouTube videos and know which ones we may not have seen before, it’s time to marry the bits together to automate.
$ youtube-dl --download-archive downloaded.txt `curl https://api.crunchprank.net/youtube/latest_video?user=<youtuber> | egrep -o 'https?://[^ ]+'`Now let’s break this down:
youtube-dl:Used to download the video--download-archive downloaded.txt:Generates a file that saves video origin and ID. This keeps the command from redownloading the same videocurl... :Acquires the latest video information| egrep -o 'https?://[^]+' :Deletes everything from the curl output BEFORE the YouTube video URL.
Now that I have a command, I replicated it for all the channels that I wanted to download videos from, placed them in a Bash script, and set it loose.
It’s something that is very simple to use and can actually be used on macOS and Windows (with WSL) as well!
However, there are some pitfalls with this that I will mention later…