Dillon Smoker Lanier

For Christmas I asked my Grandma for a Raspberry Pi. Six months later it had gathered a healthy coat of dust, and I felt like I was disappointing Grandma. This woman isn’t your normal older person. She worked on programming NASA space missions, only using assembly code, but nowadays is just as technologically illiterate as anyone else’s grandma.

Still, her wall and display full of recognition for her work on NASA missions has always inspired me. I asked her for a Raspberry Pi because I wanted to learn more about computers and knew she would be happy to enable that.

Pi in the Sky

My aspirations for the Raspberry Pi are simple: run it 24/7 to gather data, monitor websites, run trading algorithms, host websites, and create my own network of security devices out in the woods.

How feasible is this? I’m going to need a lot of Pis.

It’s great to dream big, but that isn’t going to get you there. I’m a believer that small tangible steps are how someone achieves a large goal – and I finally found my first step with the Pi.

Monitor the BOJ Website for New Reports

Duh.

This process is already somewhat automated for me, and simply moving the operation onto the Pi and making small changes is a great launch.

Currently, I pull new BOJ reports by booting up Windows Command Prompt, and run a script called ‘data_updater.py’, which checks the BOJ JGB reporting page for any new reports I haven’t downloaded yet, and then automatically downloads them for me.

With the Pi, I can use the portion of ‘data_updater.py’ that checks for a new report, and then add a few lines of code to email myself an alert when one appears. Tack on a Cron job to automatically run this monitor script however often I like.

Getting Familiar With the Pi

First things first, I had to freshen up on Linux. I want to run the Pi headless (no monitor), which means lots of keyboard commands in Terminal.

The first hurdle I hit was accessing the Pi after booting it up. SSH is a protocol that allows one computer to securely connect to another via the command line. This is how I planned to access the Pi from my main computer.

Old SSH keys and a forgotten password meant skimming through at least 10 stack overflow posts before I was able to setup SSH with public/ private keys.

Once I had connections figured out I spent some time poking around – where is Python? How do I create/delete/edit files? What is the Linux directory structure like?

I made the bold decision to edit and write all new files using Vi(m), an old school, unintuitive, powerful command line editor. Using Vi I wrote up a basic Python script that connects to the BOJ JGB reporting page, grabs the header text, and prints it.

Thankfully Pi is well supported and documented online, and getting Selenium and all the dependencies setup wasn’t too hard.

With my memory of Linux refreshed, a test script, and secure connection to the Pi setup, I could now move on to figuring out how to automatically run the script.

Cron Jobs

Cron jobs – One of those things that is super simple once you learn it, and can be a complete pain to figure out the first time.

This was my first time.

I’ll save you the boring details of my tribulations. All there is to say is, there’s a reason systems admins posting on obscure forums aren’t teachers.

This is the line of code I ended up with in my cron tab:

* * * * * /home/admin/Documents/scripts/helloselenium.py >> cron.out

Every minute, ‘helloselenium.py’ gets run, and the output is appended to the ‘cron.out’ file.

One step closer to Pi in the sky.

Automated Emails

Having never used Python to send an email, I was prepared for this to be more difficult than it seemed, and have been proven right.

Python has a ‘Secure Mail Transfer Protocol’ (SMTP) package that is supposed to make this process smoother. But anytime ports and hostnames are involved… I know there’s going to be a lot of tinkering and at least one frustrating evening involved.

I did make life a little harder than it has to be. Most examples online are using gmail as the host to send the email – I want to use a Webmail email address I created on this websites cpanel account. I even gave it a fancy name: raspberrypi@dsmokerlanier.com.

It took me a good amount of tries to get a connection to the webmail host – tinkering with things like file permissions, ports, and the hostname – but in the end we got there.

The Meat

Once I figured out the emails + cron the only thing left was combining them into a fully functional script. While I initially intended to write everything directly on the Pi using Vi, this proved annoyingly inefficient.

The process of opening, editing, debugging, saving & closing, and test running was much slower this way.

I capitulated and moved over to Windows. The hardest bug to solve involved the shebang!

In short, if you add a shebang (#!) at the top of a file, followed by a path to where Python is installed, you can execute a script simply by specifying the path. By adding ‘#!/usr/bin/python’ to the top of my script I could execute the code by simply entering the path to the the file in the command line (or cron job), ‘home/admin/Documents/scripts/cronjob-script.py’.

However, you can’t add any comments before the shebang. I added comments… which led to some error messages I struggled to decipher for a while.

Once the script was complete it worked like this:

• Go to the page with links to reports by year. Follow the link for the current year.
• Once on the current year reports page, grab the first link in the table using the XPATH, as this will be the most recently released report.
• Compare most recent report date (which can be found from the link) to each entry in a text file of report dates.
  • If the report date isn’t in the text file, there is a new report. In this case we send a notification email and add the report date to the text file.
  • If the report date is in the text file, it means we have already sent a notification email about that report. In this case nothing happens.

The cronjob runs the script every 30 minutes, and most of the time nothing happens as new reports are released approximately every 10 days.

In the End

While this isn’t the most useful script, it does give my dusty Raspberry Pi meaning (and me something to talk to grandma about).

This framework could also easily be expanded. Monitoring the web for events can be quite informative, and possibly profitable.

For example, scalpers employ similar methods to scoop up newly released products faster than humans can. Other more scrupulous uses could be automatically updating datasets with newly released information or downloading press releases/ news articles and monitoring them for removal/ revisions.

As things stand I will get an email alert within 30 minutes of each new BOJ JGB press release, which will allow me to run ‘data_updater.py’ on my Windows machine to pull the new data and store/ format it.

Until next time!

Comments

Join the discussion on 'Chasing Pi'