Home    About    Archive    Feed

10 lessons I learnt from my side projects

A Smooth Sea Never Made a Skilful Sailor.

I’ve had a side project or two for as far as I could program. It’s on rare occasions that I would find myself completely free of anything to do, be it at work or on my own free time. Therefore I would like to share my knowledge on what I learnt about my latest project, Swipr, which can be found on the app store.

Without further ado, here are 10 things I learnt working on this project.

  1. Treat side projects like real, money generating projects, have a plan, set goals, break down your main goal into smaller achievable tasks. My goal was to reach 10k downloads/users, but before that I had to create a prototype, code the app, submit to the app store and promote it. I totally failed on the last one, but I still reached my goal —7 months later.
  2. Listen to your users from the word go, listen to their grievances, get a prototype out to your friends and listen to what they have to say. I shipped my app, got a tonne of e-mails, and did absolutely nothing with them, I got 1 star reviews as a result`. Work with your users to make your product better.
  3. Be proud of your project, ask yourself this, if you were invited to an interview at Facebook, would you be able to talk about your project with so much passion that it got you the job? Its not a chore, its a labour of love. My lack of passion showed through this app, the end result was a miserable project, a shadow of what it could have been. I never loved my project, it was buggy, and remained buggy, and apart from this blog, I never really talked it. I also never really used the app, it was never created for me.
  4. Which brings me to: Build a solution for yourself, or your loved ones, or your friends, basically build a solution that will help those close to you. Side projects should be fun, remember you will be the product owner, the PM, the developer, QA and everything else between, you should be a professional in the problem domain, and the reward for your hard work is an easy life for those around you.
  5. Start simple, I mean really really simple, I am talking one button on the screen simple. Remember why you are building the solution, fancy animations make an app better, but a working app is sweeter. Launch, iterate, rinse + repeat. Launch on v.0.7, fix it, get to v.1.0 and then let the whole world know about it.
  6. Early on in the project you will make a lot of assumptions about the problem and the users, don’t let this stifle your creativity, don’t have a one mind track, sometimes v.2.0 will be a different beast from what you started building, but remember, this is only possible if you start small.
  7. The app store has about a million and a half apps, thats a thousand thousand apps homeboy! a lot of zeros, a few heroes. As much as I encourage you to ship a simple app, please make it worth the users time. If you are re-doing an existing app, make sure it is better than the ones before, unless you come up with an app to mint money or cure cancer, quality is a must for you to break the ranks. Weirdly enough, a simple app could be more appealing than overly engineered ones, take a look at flappy bird, ‘nuff said.
  8. Integrate analytics, you need to know your users, what they are doing, and why. Is your app getting more traction in Kathmandu than in London? Maybe your next update should include localisation? Are your users leaving the app without visiting the main attraction hidden 10 screens down a navigation stack? Bring it to the main screen, make it more visible.
  9. Hacking is sweet, but please get permission first. When hacking existing technologies, make sure you have permission from the owners. If not, expect repercussions. Unfortunately Tinder does not appreciate my efforts and they’ve effectively asked me to close shop (more on this later) but am glad I was not making a living out of this app, otherwise it would have been painful.
  10. Lastly, but certainly not least, ship the damn thing. Whats good in a project that no one can use? Good engineers ship.

Remember, have fun, there is no point working weekends and evenings if you are not enjoying it, is the project tiresome and stressful? ditch it, get a new one. Open source the old project and maybe some interested party will fork and develop it. If its not fun, it ain’t worth doing.

How I hacked Tinder (and the app store)

As I wrote in my previous post, I created an app called Swipr, it’s a simple tinder hack for those obsessive right swipers out there. It’s been 5 days since the app went live and this is my summary so far.

Why did I do it?

What made me do it you ask? I went out and asked a few questions, I already knew the Tinder API was exposed, so I knew what was possible. Apparently a few of the people I asked if they would fancy the exploit quite liked the idea. The Tinder API was/is easy to exploit, more on this on another post. For the tech savvy, you can review this Gist

Also, I just wanted a side project for September. :)

How long did I spend on development.

Short answer, not long enough. I had the first version of the app up and running in one evening. It looked something like the image below

As you can see, not much, but it was the only MVP I needed. It achieved the purpose without any of the gimmick.

I spent an extra weekend working on the second version of the app that was a bit more easy on the eyes, this version was used for Beta testing, after feedback from my users (Yes, all two of you!) I ended up with the following.

Quite clean and simple. And the final product was not really different from this.

This all took about 3 days of dev, and about an hour each evening after work for about another week.

Ship It

I had already decided that this app will ship in two weeks regardless of its state. And I wasn’t about to compromise that date. So I shipped, bugs and all.

The app got reject twice, the facebook login was a bit tricky to get right at first. I fixed this and the app went through and hit the store on Monday night, 29th September.

Marketing.

What marketing? So far, its been this blog, a post on Reddit giving away free promo codes (Look here if any are left) and a post on Hacker News that hardly made an impact, thats all the ‘marketing’ I did. Oh, and this landing page that’s had about 5 views since inception.

The results? Well, its hard to explain this, but lets look at my rank on Saturday morning.

Yes, that number 14 on top paid Lifestyle app. (Now slipped to #30). You must be thinking am rolling in Lizzie’s at the moment, hardly the case, But at least I’ve managed to recoup my Apple Dev account charges, and maybe get an extra Mojito over the weekend.

Just In: The app got to #1 on the App store in Senegal and Bahamas…..Which means absolutely nothing unfortunately. Can someone please explain to me how the app store ranking works, is it broken?

There are hardly any reviews or ratings.

Did I hack the Itunes store? Hell no, but I don’t know how I manage to rank so high in the first place.

Conclusion

Did I ship a buggy app? Damn right I did, and am not very proud of that, but that’s what updates are for, even Apple (iOS 8.0.1 anyone?) ships buggy software, so please don’t judge.

This app was an experiment, I have been trying to test the market before commencing development, I did not chose a large test group, if anything, the people I based the app’s interest on where very biased. These were my friends and colleagues.

So why did I try out this strategy? Ever heard of the saying ‘Old ways won’t open new doors’? Enough said.

This experiment is nearly done, I’ll probably ship a few updates before leaving the app to die a slow death or remove it entirely from the store. Should I remind you that this app has been live for only 5 days?

Got some feedback? Mail me a dev at edwinb dot co dot uk or get me on twitter

Shipping Swipr

Swipr is the new app I have been working on. It’s also the quickest I’ve shipped an app (~3 weeks + apple review), it also happens to be the product am least proud of, why is that you ask? Swipr is entirely based on hacking an existing product and provides no foreseeable benefit to the greater good of humanity, but eff that, all you want is a date right? well then, long story short, it’s a Tinder hack. If you are a chronic swiper on Tinder, then just give your fingers a rest and let Swipr take the lead.

The app took about 10 days to develop, got rejected twice by apple (entirely my fault) and I also had my 9-5 job. Its now been on the store for a couple of days and the downloads are steady, but not overly exciting, I have not tweeted or told anyone about it so far, and this blog right here is my bit of ‘marketting’.

This project is more of an experiment, I am developing an app not for my need but for a larger market, all my other side projects have been projects I perfonally found useful, not so much this one. I’ll post the progress and stats as time goes

In the mean time, give me your money (only £0.69)

New project for September

I have always been facinated by programming so much so that I’ve made a career out of it. The one thing that I do to keep my motivational levels high is taking on side projects, any good dev will admit to almost always having a side project going on, and I am no exception.

This project I am working on is a bit old actually. I’ve had to rescue it from the infamous hard-disk of death, better known as my backup drive where once a project enters, it rarely ever ships.

The project I am working on is a Tube Status app for London, I know what you are thinking, not one of those again, we’ve got a bunch of this shit already we don’t need another one. This particular one has a twist though, like no other app on the store, this one is anything but polite. Take a peek at the screen shots below.

Also, I am collecting e-mails to judge interest, if I reach a certain magic number (haven’t decided what it is yet) and I shall only embark on the development if its viable, my time could be better spent on app that more people will find useful. So here is the link. signup

Building a NewsCrawler: pt1

#Introduction

This tutorial/walk through assumes that you are already familiar with python and the terminal. I would also strongly advice anyone remotely interested in understanding scrapy to have a look at their documentation and preferably follow the quick tutorial and setup instruction outlined.

The example in the documentation is a simple spider. Basic spiders are good if we intend on retrieving information from a list of pages that we currently know the URLs of. This is no good in our scenario where a lot of pages need to be scraped. This is where the crawler comes in, crawlers provide a convenient mechanism to follow links, you can refer to the documentation here to refresh your memory.

###Pick a website

We are going to experiment with a Kenyan news website. Please feel free to substitue this website with one that suits your taste. Daily Nation For those of you keen enough, you will realise that I have an app for this news source on the App Store

####Install scrapy Follow the Installation Guide here, but the gist of it all is basically.

pip install Scrapy

####Create a new project

**Have you installed scrapy yet? I struggled to get it to work on osx, if you get libxml issue, just know you are in for a long ride **

scrapy startproject hermes

This will create a project called hermes in the directory you ran it in tutorial in this case. Explore this directory, its pretty boring at first.

	tutorial/ 
	    scrapy.cfg  	#config file for deployment
	    tutorial/    	#project dir/module
	        __init__.py
	        items.py		#Model
	        pipelines.py	
	        settings.py		#project settings
	        spiders/		#spider directory
	            __init__.py
	            ...

Scrapy did the burden of work for us. There are only two files we have to be concerned with at the moment, thats the items.py and the spiders. The spider directory will hold all the spiders.

###Define the model.

In scrapy, the models are defined in the items.py file. All items have a Field() type. We create our model named NationmediaItem our items class looks like the following.

	from scrapy.item import Item, Field

	class NationmediaItem(Item):
		title = Field()
		link = Field()
		summary = Field()
		content = Field()
		author = Field()	

###Defining our Spider Spiders are user defined and they are usually quite specific to a particular website, or a group of websites with the same structure.

A spider in its most basic form will have a list or URLS to download, rules that define how links are followed and a method to parse the content.

A spider must subclass scrapy.spider.Spider and define the following:

  • name: Unique identifier for the spider.
  • start_urls: a list of URLS where crawling will start from.
  • parse(): call back method called when the content has been downloaded, a Response object returned and is passed to the method as an argument.

Navigate into the spider directory and create an empty python file. Call this file nmdspider.py.

Our class looks like the following:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from nationmedia.items import NationmediaItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import Selector

class MainParser():
    def __init__(self, response, tags):
        self.response = response
        self.tags = tags

    def scrapNationOnline(self):
        hxs = Selector(self.response)
        item = NationmediaItem()
        item ["link"] = self.response.url
        item ["title"]  = ''.join(hxs.xpath('//div[@class=\"story-view\"]/header/h1/text()').extract()) #this worked by taking the page title, contains tags we dont need/want ''.join(hxs.css('title::text').extract())
        item ["summary"] = ''.join(hxs.xpath('//div/ul/li/text()').extract())
        item ["content"] = '\n'.join(hxs.xpath('//article/section/div/p/text()').extract())

        #retrieve the author. There are numerous formatting issue with this tag

        author = hxs.xpath('//section[@class=\"author\"]/strong/text()').extract()
        if not author:
            author = hxs.xpath('//section[@class=\"author\"]/text()').extract()
        item ["author"] = ''.join(author)

        return item

class PoliticsSpider(CrawlSpider):
    """Scrapes Polics News"""
    name = "politics"
    allowed_domains = ["nation.co.ke"]
    start_urls = ["http://www.nation.co.ke/news/politics/-/1064/1064/-/3hxffhz/-/index.html"]
    rules = (
        Rule(SgmlLinkExtractor(allow=('news\/politics\/',), deny=('\/view\/asFeed\/', '\/-\/1064\/1064\/-\/3hxffhz\/-\/index.html')), callback='parse_page', follow=True),
    )
    def parse_page(self, response):
        mainParser = MainParser(response, ["News", "Politics"])
        item = mainParser.scrapNationOnline()
        return item

Take a look at the code from the repository here Its going to be clearer than the mess above.

###Time to run the crawler

scrapy crawl politics

Its as simple as that. To run the spider and save the results in a json file run

scrapy crawl politics -o politics.json -t json

###Output

The output should be a json file of all the scrapped articles. In this case from the politics section of nation.co.ke

Some Explanation

As you can see from the code, we are using xpath to extract information from the html response, this is the part that makes scrapping such a fragile task, if the website you are scrapping updates its html files, then you have to do the same here.

#####PoliticsSpider [Class:] This is our main class, if you are going to have multiple crawlers running, this is the class to duplicate. To use this class, you have to set a few parameters.

  1. Name
  2. Allowed Domains
  3. Start_urls
  4. rules.

All these are self explanatory apart from the last one. These are rules** to follow while crawling links originating from the Start_urls.

Allow: In our example, we are only accepting URLS containing news/politics, as we sure that any article with that URL is a politics item.

Restricted_xpaths: Restricts the link to a certain xpath, we are currently not implementing this.

Deny: We also deny /view/asFeed as this is the extension found in the RSS feed page, there are plenty of other rules we can apply, but these two seem to be of utmost importance.

Callback: This is the method we call back to after parsing is complete, please DO NOT name your method ‘parse’ as this is already in use internally by scrapy, I learnt this the hard way.

Follow: Instructs the program to continue following links as long as they exist, this is how we recursively crawl entire websites.

######Parse_page [Method]

This method is called once we have retrieved the contents of a page. The page is still in its original format, in this case HTML, it will need to be structured.

We delegate the actual parsing to the MainParse class, just trying to observe some OOP principles. To execute the actual parsing, we make the following call. item = mainParser.scrapNationOnline(). It returns an Item object, this item object defines a news article.

###Conclusion.

That is all that you need to parse a website, I would suggest you read the documentaion for Scrapy as I have left quite a lot unanswered here.

Website scraping is fun, but don’t be a douche bag, don’t run 100 scrapers 24/7 on someone else’s site. Also don’t steal information and claim it as your own, this really pisses some people off.

In the next web scraping tutorial, we will look at how we deploy our scraper to the cloud and automate the entire process.