Google Reader Backup Viewer

If you created a full backup for your Google Reader account while it was still possible, you probably need a way browse the data. Fortunately, Mihai Parparita created Zombie Reader, a tool that resurects the Google Reader interface and transforms into a viewer for your data.

"Reader is a canonical single page application: once the initial HTML, JavaScript, CSS, etc. payload is delivered, all other data is loaded via relatively straightforward HTTP calls that return JSON (this made adding basic offline support relatively easy back in 2007). Therefore if I served the archived data in the same JSON format, then I should be able to browse it using Reader's own JavaScript and CSS," says Mihai.

Go to readerisdead.com, download the updated archive and use the instructions from the page. You still need Python 2.7 and some basic command-line skills. For Windows, you could use the instructions from my previous post and replace:

c:\python27\python reader_archive\reader_archive.py --output=download

with

c:\python27\python zombie_reader\zombie_reader.py download

Zombie Reader uses the Google Reader interface and your local data. It's like a Google Reader snapshot that preserved all your subscriptions, all the items you've read or marked as read, all your starred items, your tags and much more. Obviously, many features don't work (search, trends, subscribe), the application doesn't save your state and it doesn't show the latest posts from your subscriptions.

It does show your first read item, first starred item and the first shared item, as well as the people you followed before Reader's social features were removed. The "sort by oldest" feature is no longer limited to the last 30 days, it now sorts all your feed items.

"A side effect is that I now have a self-contained Reader installation that I'll be able to refer to years from now, when my son asks me how I spent my mid-20s," says Mihai. "It also satisfies my own nostalgia kicks, like knowing what my first read item was. In theory I could also use this approach to build a proxy that exposes Reader's API backed by (say) NewsBlur's, and thus keep using the Reader UI to read current feeds. Beyond the technical issues (e.g. impedance mismatches, since NewsBlur doesn't store read or starred state as tags, or has per item tags in general) that seems like an overly backwards-facing option." I'm sure that someone will build the proxy.

moreless

Google and Feeds

For some reason, a day before Google Reader was closed, Google Developers blog published an article about PubSubHubbbub, feeds and the Google Feeds API. PubSubHubbub (PuSH) is a protocol for sending notifications when a feed is updated. "Using the PuSH protocol, servers can subscribe to an almost arbitrarily large number of feeds and receive updates as they occur." Google encourages publishers to submit their feeds to a public PuSH hub like Google's hub and Superfeedr's Open PubSubHubbub Hub.

The most interesting part of the post is this one: "Google directly hosts many feed producers (e.g. Blogger is one of the largest feed sources on the web) and is a feed consumer too (e.g. many webmasters use feeds to tell our Search system about changes on their sites). Our PuSH hub offers easy access to hundreds of millions of Google-hosted feeds, as well as hundreds of millions of other feeds available via the PuSH ecosystem and through active polling." And something else: "We are planning some improvements to the Feed API, as part of our ongoing infrastructure work."

This means that Google's feed processing backend will continue to exist, but will focus on the Google Feeds API and Google Search index updates. Google Feeds API will also continue to exist and third-party feed readers could use it. All of this was an important part of the Google Reader backend, but Google Reader also had to manage subscriptions, labels, the read/unread state, create a search index for each user.

Hopefully, this also means that Google's services will continue to offer feeds. Maybe, at some point, even Google+ could add support for feeds.

Update: Google Feedfetcher is still running and shows almost the same number of subscribers. Here's a screenshot from FeedBurner:

"Feedfetcher is how Google grabs RSS or Atom feeds when users choose to add them to their Google homepage or Google Reader. Feedfetcher collects and periodically refreshes these user-initiated feeds, but does not index them in Blog Search or Google's other search services (feeds appear in our search results only if they've been crawled by Googlebot)." - from Google's help center.

Tomek Wasiak noticed Google Feedfetcher in his site's logs today. He subscribed to the site's feed in Google Reader and he's the only subscriber.

moreless

Google Alerts Drops RSS Feeds

If you read the post about the well-connected Google Reader, you probably anticipated this chain reaction. All the services that integrated with Google Reader will remove their features, now that Reader is gone.

Google Alerts no longer supports RSS delivery and recommends users to switch to email delivery. "Google Reader is no longer available. To continue receiving Google Alerts, go to http://www.google.com/alerts/manage and change your alerts to email delivery," says Google.

You're probably wondering: what's the connection between Google Alerts feeds and Google Reader? Google Alerts feeds, which were added back in 2008, had public URLs and you could use any feed reader to subscribe to them. The trouble is that the feeds were generated by Google Reader and were actually a Google Reader feature, just like the web page monitoring feature.

Unfortunately, Google doesn't offer feeds for search results and the Web Search API has been deprecated and it was too limited to create a service like Google Alerts. The only obvious alternative is to screenscrape Google results.

Talkwalker Alerts looks almost like Google Alerts and offers both email alerts and feeds. The service is free for up to 100 alerts.

moreless

Google Reader's Final Message

If you open Google Reader, you'll see this message:

Thank you for stopping by.

Google Reader has been discontinued. We want to thank all our loyal fans. We understand you may not agree with this decision, but we hope you'll come to love these alternatives as much as you loved Reader.

Sincerely,

The Google Reader team

Frequently-asked questions

What will happen to my Google Reader data?

All Google Reader subscription data (eg. lists of people that you follow, items you have starred, notes you have created, etc.) will be systematically deleted from Google servers. You can download a copy of your Google Reader data via Google Takeout until 12PM PST July 15, 2013.

Will there be any way to retrieve my subscription data from Google in the future?

No -- all subscription data will be permanently, and irrevocably deleted. Google will not be able to recover any Google Reader subscription data for any user after July 15, 2013.

Why was Google Reader discontinued?

Please refer to our blog post for more information.

I don't understand why Google removes Reader data so soon and why the message uses words like "systematically deleted", "permanently, and irrevocably deleted". It's like they try to get rid of everything that's related to Reader. I can still download Google Buzz data from Google Takeout. A similar message was posted for Google Health, but users had one year to download their data.

moreless

The End of This Internet

"Congratulations, you've reached the end of this internet. Look for another?" That's the message you see when you use the Google Reader bookmarklet and you read all the posts from your subscriptions. It's a joke that's quite bitter, now that the end of Google Reader is really close.

Google Reader was more than just a feed reader, it was probably the most well-connected Google service. It was integrated with so many Google services and now all of them will be a little less useful. The missing features, which relied on Google Reader, will show how important this service really was.

iGoogle's canvas view for feeds was powered by Google Reader and now it's no longer available, just like the Reader gadget.

moreless

Full Google Reader Backup

"Reader is dead," says Mihai Parparita, one of the former Google Reader engineers. You still have 2-3 days to use Google Reader, but the best thing you can do is to export your data.

Google Takeout lets you export some of your Reader data, but not everything: your subscriptions, your notes, starred items, shared items, liked items, the list of followers and the people you were following, the items shared by the people you were following. Mihai Parparita wrote some Python scripts that download everything from your Google Reader accounts, including the entire content of the posts from your subscriptions. You need Python 2.7, some basic command-line skills and a lot of free storage: my backup has more than 5 GB for about 250 feeds (vs 125 MB for the uncompressed Takeout backup).

Here are some tips for running the script in Windows 7/8:

- install Python from here (Python 2.7.5 Windows Installer)
- download Mihai's zip file and extract the files
- open the folder in Windows Explorer and you should see a list of subfolders like "base", "bin", "feed_archive".

- Shift + right-click below the folders and select "open command-line window here".
- copy this code, paste it in the command-line window and press Enter (I assumed that Python's folder is c:\python27):

set PYTHONPATH=%cd%
c:\python27\python reader_archive\reader_archive.py --output=download

- a web page will open in your favorite browser and you'll need to click "Accept", copy the authorization code and paste it in the command-line window.
- wait until the script downloads all the files.

Mihai also started to write a script that lets you browse your archive. It's a work in progress, probably because the script for downloading your data is more important right now.

There's also a script for downloading a feed's archive. "Google Reader has (for the most part) a copy of all blog posts and other feed items published since its launch in late 2005 (assuming that at least one Reader user subscribed to the feed). This makes it an invaluable resource for sites that disappear, can serve as a backup mechanism and enables tools to be created." My post from 2007 provides another way to download the history of a feed. You can also upload your OPML file to this site, which preserves hitorical feed data.

"I don't fault Google for providing only partial data via Takeout. Exporting all 612,599 read items in my account (and a few hundred thousand more from subscriptions, recommendations, etc.) results in almost 4 GB of data. Even if I'm in the 99th percentile for Reader users (I've got the badge to prove it), providing hundreds of megabytes of data per user would not be feasible. I'm actually happy that Takeout support happened at all, since my understanding is that it was all during 20% time," says Mihai Parparita, who spent 5 years working on Google Reader.

If you're curious to know which Reader alternative gets a thumb up from Mihai, his answer is "a toss-up between NewsBlur and Digg Reader."

moreless

The Feed Reading Playground Is Now Open

Google used this many times last year: "the playground is open". That's one of the most important things about Android: it's open source and any company can use it. One of the definitions of "open" is "accessible to all, unrestricted as to participants".

The playground for feed readers is now open. Google Reader's demise levels the playing field in the feed reading world. Now there are more competitors, there's more innovation, there are new platforms for distributing feeds, new interfaces. Suddenly, feed readers are a hot topic. Until now, Google Reader was the dominant service and platform, but Google didn't know what to do with it. Google Reader stopped adding new features back in 2010 and it was in maintenance mode ever since then. Few dared to challenge its position and most new feed readers were only Google Reader clients.

Google Reader's disappearance is Google's best idea for saving RSS. From stagnant to vibrant in 3 months - Google Reader's demise helped RSS more than the last 3 years of silence. The playground is now open.

And for those who are sad that Reader will soon be gone - don't worry, there's a bit of Google Reader in any feed reader that has launched in the past 3-4 years or will launch in the near future. Maybe it's time to get a better Reader, even if not from Google...

moreless

A Google Reader Puzzle From 2007

Back in 2007, Google accidentally made public an internal video about Google Reader. The video included a lot of useful information about Google Reader, the kind of details you'll never find in a Google post or presentation.

I'm trying to solve the puzzle and find the number of Google Reader users from 2007. Here are the hints:

1. At that time, Google Reader crawled 8 million feeds.

2. Two thirds of the feeds had only one subscriber, one third of the feeds had more than one subscriber.

3. Google Reader used 10TB for storing all the raw data.

4. The rate of user growth = the rate of growth for the number of feeds.

5. The index size grew 4% every week.

6. 70% of the Google Reader traffic came from Firefox (at that time, Firefox's market share was about 15%).

7. Gmail and orkut were the only Google applications that had a bigger number of pageviews/user than Google Reader.

8. The main Google Blog had 100,000 subscribers and this number includes all the subscribers from iGoogle, Reader and orkut.

So, at that time, only 2.6 million feeds indexed by Google Reader had more than one subscriber. Probably many of the feeds that had only one subscriber were used by other Google services powered by Google Reader's backend (orkut, Blogger widgets, Google Spreadsheets, Ajax API) or they were Google News/Blog Search feeds. It makes sense to assume that the number of Google Reader users was lower than the number of feeds with more than one subscriber.

Here's a chart from 2010 that shows Google Reader's user growth:



And something else from 2010: "the average Reader user reads about 105 items a day".

It's probably worth distinguishing Reader as a service from Reader as a platform.

moreless

The Most Well-Connected Google Service

Many people think that Google Reader could've been more successful if Google promoted it more. The truth is that Google Reader has been the service that connected to the biggest number of Google services. No other Google benefited from so many service integrations.

Here's an incomplete list. Start counting:

1. a link to Reader was displayed in Gmail's inbox when there was no mail

2. for many years, Google Reader could be found in the main navigation bar, next to Gmail, Calendar and Google Docs

3. Google Reader was the first Google service that worked offline and the first Google service that used Google Gears, back in 2007



4. Google Reader integrated with Google Social Search, a feature that allowed you to restrict results to the pages written by your friends or people you follow.



5. iGoogle integrated with Google Reader when it started to add support for canvas view. Maximize a feed gadget and you get the Google Reader interface.



6. Blogger's Following feature was powered by Google Reader. "The blogs you follow in Blogger have been added as subscriptions in Google Reader. Subscriptions can be managed in Reader without affecting your following list in Blogger."



7. Listen, Google's podcast manager app for Android, used Google Reader to store subscriptions.

8. Google Alerts integrated with Google Reader, so you could subscribe to feeds instead of receiving email notofcations.

9. Back in 2008, Google's mobile transcoder displayed the site's feeds at the top of the page and linked to Google Reader.

10. Google Reader was the only Google product with an interface optimized for Nintendo Wii.

11. Google Currents, launched in 2011, allowed you to import your Google Reader subscriptions.

12. Google Buzz's commenting feature was integrated with Google Reader.



13. When Bloglines was discontinued in 2010, Google Reader's team encouraged users to switch to Reader. The blog post includes a graph of Reader users over time.

14. The Google Groups redesign from 2010 was inspired by Google Reader.

15. Google bought FeedBurner to monetize Google Reader and launched AdSense for Feeds.

16. Back in 2007, Google Reader made shared items available to Google Talk contacts and many people complained about this.

17. Google's Power Readers feature from 2008 allowed you to "track the news sites and blogs Barack Obama and John McCain read" using Google Reader. It was a clever way to promote Google Reader. This feature was expanded to "include journalists, techies, fashion critics, foodies and more".

18. Google Toolbar for IE allowed you to subscribe to feeds using iGoogle or Google Reader.

19. Google Blog Search still has this link below the list of search results page: "Subscribe to a blog search feed for [query] in Google Reader". Google News had a similar link.

20. Google Spreadsheets has a special function for importing feeds that was initially called GoogleReader. Now it's called ImportFeed.

21. Google TV Queue uses Google Reader to manage subscriptions.

22. Google Reader was Google's infrastructure for feeds and the technology was used by iGoogle, orkut, Gmail's web clips, Blogger widgets, Google Spreadsheets and the Ajax API.

I'm sure you can find other examples of services that integrated with Google Reader. I didn't include all the third-party apps and services that used the unofficial Google Reader API. As you can see, Reader was an important part of the Google ecosystem and Google did promote the service in many ways.

moreless

From Google Reader to Feedly

There's just one week and a half and Google Reader will be history. If you're using the service, it's a good idea to export your data and switch to a different service. You can choose from Feedly, The Old Reader, MultiPLX, NewsBlur, Feedspot, Netvibes. I haven't decided which one I'll use, but Feedly is a strong contender.

Feedly is probably the service that will benefit the most from Google Reader's demise. It grew from 4 million users to 12 million users in only 3 months and that's impressive. The service was just a Google Reader client, an alternate interface for Google Reader that gained a foothold on mobile.

Feedly has recently started to migrate users from Google's backend to its own backend, while preserving most of their data. Feedly Cloud was built in record time and it's now a scalable infrastructure for Feedly that can also be used by other apps that were powered by the unofficial Google Reader API. There are 9 apps that use it, including gReader, Newsify and Sprout Social. If you don't like the mobile apps or the browser extensions, there's now a Feedly web app that's optimized for the desktop and replaces extensions. It's hard to morph from a client to a platforms in a few months.

I'll miss Google Reader, like many other power users. Unfortunately for us, Google is not the right company for niche services. Google wants to create products that are used by hundreds of million of users and Google Reader wasn't one of them. Feedly and other similar services will have to find a business model for something that's no longer cool, no longer supported by many browsers, no longer supported by Twitter (other sites to follow). For many people, social sites offer better value than feed readers and not even Google could change that.

"As a culture we have moved into a realm where the consumption of news is a near-constant process. Users with smartphones and tablets are consuming news in bits and bites throughout the course of the day — replacing the old standard behaviors of news consumption over breakfast along with a leisurely read at the end of the day. (...) Google is looking at pervasive means to surface news across products to address each user's interest with the right information at the right time via the most appropriate means," said Richard Gingras, Senior Director, News & Social Products at Google.

So what will you use instead of Google Reader?

moreless

Google Reader Data Points

It's hard to estimate the number of Google Reader users, but here are some data points:

- the most popular feed has more than 24 million subscribers (CNN):

- the second most popular feed has 6.6 million subscribers (Engadget):

- the third most popular feed has 1.7 million subscribers (NY Times)

- Google's official blog had 100,000 subscribers in 2007 and now it has about 353,000 subscribers

- JoelOnSoftware.com had 42,000 subscribers in 2008 and now it has 148,000 subscribers.

- according to FeedBurner, 87% of the subscribers to this blog's feed use Google Reader or iGoogle. Google Reader says that this blog has 115,035 subscribers, while the total number of subscribers is 144,173. Here are the FeedBurner stats (the green lines show the number of subscribers):

Here's the Google Trends chart for [google reader]:

moreless

No More Google Reader

Google announced that Google Reader will be discontinued on July 1st. It's a sad news, but it was inevitable. Google Reader has always been "on the chopping block" because it never got enough traction.

Everything started with a feed parser built by Chris Wetherell that turned into a feed reader, helped by Ben Darnell, Laurence Gonsalves, and Mihai Parparita. The product was launched in 2005 as a Google Labs project and it was significantly improved one year later, when the Google Reader team launched a completely new version. Over the years, Google Reader integrated with iGoogle, added social features and handled feed serving for all Google products. Back in 2007, Google Reader crawled 8 million feeds and 70% of the traffic was from Firefox users.

In 2011, Google removed Reader's social features and replaced them with a Google +1 button. It was the beginning of the end for Reader, who lost all the engineers from the original team. Google Reader is in maintenance mode ever since then.

While feeds are no longer important for many users and browsers start to drop support for reading feeds, social networks make newsfeeds popular and mobile apps like Flipboard simplify reading the news. Feeds are now a behind-the-scenes technology and full-fledged feed readers seem outdated.

"We know Reader has a devoted following who will be very sad to see it go. We're sad too. There are two simple reasons for this: usage of Google Reader has declined, and as a company we're pouring all of our energy into fewer products. We think that kind of focus will make for a better user experience," says Google's Alan Green.

It's hard to find a replacement for Google Reader, since Google Reader was the most popular feed reader and the competition couldn't keep up with it. You can still find some web-based feed readers, but none of them is as good as Google Reader. Congratulations to everyone who worked on the Reader team and thanks to all the people who subscribed to this blog in Google Reader.

Here's Google Reader's team from 2007:

moreless