Never miss a Magazine article — build your own RSS notification system

Python is a great programming language to quickly build applications that make our life easier. In this article we will learn how to use Python to build a RSS notification system, the goal being to have fun learning Python using Fedora. If you are looking for a complete RSS notifier application, there are a few already packaged in Fedora.

Fedora and Python – getting started

Python 3.6 is available by default in Fedora, that includes Python’s extensive standard library. The standard library provides a collection of modules which make some tasks simpler for us. For example, in our case we will use the sqlite3 module to create, add and read data from a database. In the case where a particular problem we are trying to solve is not covered by the standard library, the chance is that someone has already developed a module for everyone to use. The best place to search for such modules is the Python Package Index known as PyPI. In our example we are going to use the feedparser to parse an RSS feed.

Since feedparser is not in the standard library, we have to install it in our system. Luckily for us there is an rpm package in Fedora, so the installation of feedparser is as simple as:

$ sudo dnf install python3-feedparser

We now have everything we need to start coding our application.

Storing the feed data

We need to store data from the articles that have already been published so that we send a notification only for new articles. The data we want to store will give us a unique way to identify an article. Therefore we will store the title and the publication date of the article.

So let’s create our database using python sqlite3 module and a simple SQL query. We are also adding the modules we are going to use later (feedparser, smtplib and email).

Creating the Database

#!/usr/bin/python3
import sqlite3
import smtplib 
from email.mime.text import MIMEText

import feedparser

db_connection = sqlite3.connect('/var/tmp/magazine_rss.sqlite')
db = db_connection.cursor()
db.execute('CREATE TABLE IF NOT EXISTS magazine (title TEXT, date TEXT)')

These few lines of code create a new sqlite database stored in a file called ‘magazine_rss.sqlite’, and then create a new table within the database called ‘magazine’. This table has two columns – ‘title’ and ‘date’ – that can store data of the type TEXT, which means that the value of each column will be a text string.

Checking the Database for old articles

Since we only want to add new articles to our database we need a function that will check if the article we get from the RSS feed is already in our database or not. We will use it to decide if we should send an email notification (new article) or not (old article). Ok let’s code this function.

def article_is_not_db(article_title, article_date):
    """ Check if a given pair of article title and date
    is in the database.
    Args:
        article_title (str): The title of an article
        article_date  (str): The publication date of an article
    Return:
        True if the article is not in the database
        False if the article is already present in the database
    """
    db.execute("SELECT * from magazine WHERE title=? AND date=?", (article_title, article_date))
    if not db.fetchall():
        return True
    else:
        return False

The main part of this function is the SQL query we execute to search through the database.  We are using a SELECT instruction to define which column of our magazine table we will run the query on. We are using the * symbol to select all columns ( title and date). Then we ask to select only the rows of the table WHERE the article_title and article_date string are equal to the value of the title and date column.

To finish, we have a simple logic that will return True if the query did not return any results and False if the query found an article in database matching our title, date pair.

Adding a new article to the Database

Now we can code the function to add a new article to the database.

def add_article_to_db(article_title, article_date):
    """ Add a new article title and date to the database
    Args:
        article_title (str): The title of an article
        article_date (str): The publication date of an article
    """
    db.execute("INSERT INTO magazine VALUES (?,?)", (article_title, article_date))
    db_connection.commit()

This function is straight forward, we are using a SQL query to INSERT a new row INTO the magazine table with the VALUES of the article_title and article_date. Then we commit the change to make it persistent.

That’s all we need from the database’s point of view, let’s look at the notification system and how we can use python to send emails.

Sending an email notification

Let’s create a function to send an email using the python standard library module smtplib. We are also using the email module from the standard library to format our email message.

def send_notification(article_title, article_url):
    """ Add a new article title and date to the database

    Args:
        article_title (str): The title of an article
        article_url (str): The url to access the article
    """

    smtp_server = smtplib.SMTP('smtp.gmail.com', 587)
    smtp_server.ehlo()
    smtp_server.starttls()
    smtp_server.login('your_email@gmail.com', '123your_password')
    msg = MIMEText(f'\nHi there is a new Fedora Magazine article : {article_title}. \nYou can read it here {article_url}')
    msg['Subject'] = 'New Fedora Magazine Article Available'
    msg['From'] = 'your_email@gmail.com'
    msg['To'] = 'destination_email@gmail.com'
    smtp_server.send_message(msg)
    smtp_server.quit()

In this example I am using the Google mail smtp server to send an email, but this will work with any email services that provides you with a SMTP server. Most of this function is boilerplate needed to configure the access to the smtp server. You will need to update the code with your email address and credentials.

If you are using 2 Factor Authentication with your gmail account you can setup a password app that will give you a unique password to use for this application. Check out this help page.

Reading Fedora Magazine RSS feed

We now have functions to store an article in the database and send an email notification, let’s create a function that parses the Fedora Magazine RSS feed and extract the articles’ data.

def read_article_feed():
    """ Get articles from RSS feed """
    feed = feedparser.parse('https://fedoramagazine.org/feed/')
    for article in feed['entries']:
        if article_is_not_db(article['title'], article['published']):
            send_notification(article['title'], article['link'])
            add_article_to_db(article['title'], article['published'])

if __name__ == '__main__':
    read_article_feed()
    db_connection.close()

Here we are making use of the feedparser.parse function. The function returns a dictionary representation of the RSS feed, for the full reference of the representation you can consult feedparser ‘s documentation.

The RSS feed parser will return the last 10 articles as entries and then we extract the following information: the title, the link and the date the article was published. As a result, we can now use the functions we have previously defined to check if the article is not in the database, then send a notification email and finally, add the article to our database.

The last if statement is used to execute our read_article_feed function and  then close the database connection when we execute our script.

Running our script

Finally, to run our script we need to give the correct permission to the file. Next, we make use of the cron utility to automatically execute our script every hour (1 minute past the hour). cron is a job scheduler that we can use to run a task at a fixed time.

$ chmod a+x my_rss_notifier.py
$ sudo cp my_rss_notifier.py /etc/cron.hourly

To keep this tutorial simple, we are using the cron.hourly directory to execute the script every hours, I you wish to learn more about cron and how to configure the crontab, please read cron’s wikipedia page.

Conclusion

In this tutorial we have learned how to use Python to create a simple sqlite database, parse an RSS feed and send emails. I hope that this showed you how you can easily build your own application using Python and Fedora.

The script is available on github here.

For Developers Using Software

7 Comments

  1. Really nice post 🙂

    Beware of the SQL injection attack 😉

    “Usually your SQL operations will need to use values from Python variables. You shouldn’t assemble your query using Python’s string operations because doing so is insecure; it makes your program vulnerable to an SQL injection attack (see https://xkcd.com/327/ for humorous example of what can go wrong).

    Instead, use the DB-API’s parameter substitution. Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor’s execute() method.”
    https://docs.python.org/3.6/library/sqlite3.html

    Here is the proposed fix: https://github.com/cverna/rss_feed_notifier/pull/1

  2. And if you continue on this way a bit further you get to rss2email :). Install it from a Fedora package.

  3. Nice, surprised you didn’t teach how to create a pop-up with the notification, instead of emailing. After all wouldn’t that be more secure? Storing your password (even just your email one, as opposed to your user/root password) in a plain text file like a Python script isn’t exactly secure.

  4. José Monteiro

    Great. Exactly like it must be: good code used in a practical situation. I would like very much to see more of this. How to validate user data? Could you write a series of articles, for instance, building an inventory control?

  5. steve allen

    Really enjoyed this article. Short, simple and useful.

    I made one minor change. I changed the code to send one email rather than multiple. My email was getting swamped with messages:

    def send_notification(message_text):
    “”” Add a new article title and date to the database
    Args:
    article_title (str): The title of an article
    article_url (str): The url to access the article
    “””
    smtp_server = smtplib.SMTP(‘smtp.gmail.com’, 587)
    smtp_server.ehlo()
    smtp_server.starttls()
    smtp_server.login(‘your_email@gmail.com’, ‘123your_password’)
    msg = MIMEText( message_text,’html’ )
    msg[‘Subject’] = ‘New Fedora Magazine Article(s) Available’
    msg[‘From’] = ‘your_email@gmail.com’
    msg[‘To’] = ‘destination_email@gmail.com’
    smtp_server.send_message(msg)
    smtp_server.quit()

    def read_article_feed():
    “”” Get articles from RSS feed “””
    feed = feedparser.parse(‘https://fedoramag.wpengine.com/feed/’)
    article_count = 0
    msg_body = ”

    for article in feed['entries']:
        if article_is_not_db(article['title'], article['published']):
            article_count = article_count + 1
            msg_body = msg_body + '<p><a href="' + article['link'] + '">' + article['title'] + '</a></p>'
            add_article_to_db(article['title'], article['published'])
    if article_count > 0:
        send_notification(msg_body)

Comments are Closed

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions