Where it starts to run into problems is when working on web-apps that make a lot of ajax calls or load media.

Ajax calls are especially important as the built-in ‘runserver’, which runs in a single thread, guarentees that the order the ajax calls is made is the order they get returned. In the real world, we know this doesn’t happen, and any good developer should plan for this.

Introducing django-concurrent-server (a fork of an existing project which I have updated). Now even better than the original – you can use PIP to install it!

django-concurrent-server

This multi-threaded development server not only increases the performance of your development box, but will also allow you to test in less pristine conditions.

It works essentially exactly the same as Django’s own ‘runserver’ command – taking the same arguments and everything.

(secretproj)ash@dev:~/sandbox/secret-project/source$ python manage.py runcserver 0.0.0.0:8000
Validating models...
0 errors found

Django version 1.2.3, using settings 'source.settings'
Concurrent Development server is running at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

Users can push and pull from other users or from a central repository.

The first thing you need to do is install git on both your clients and server. I am using Ubuntu, so you will have to figure out the equivalent commands and package names.

(client)$ sudo apt-get install git git-core

(server)$ sudo apt-get install git git-core

Now that we have git on our server and client, we can start creating the git repository we want under source control.

The first thing we will do is create a git project. This can either be an existing code base you are already working on, or you can start a project from scratch. For the purposes of this example, we will start a new project.

(client)$ mkdir project
(client)$ cd project
(client)$ git init

Git should reply with the message ‘Initialized empty Git repository in .git/‘. Make a note of the .git directory created in our project – we will need it shortly.

I ran into my first issue here – the version of git I am using doesn’t seem to like to clone empty repositories. In order to allow me to clone (as you will see further down the page), we must have an initial commit.

(client)$ touch README
(client)$ git add README
(client)$ git commit -m "Initial commit." README

We want this new project to be available on a central Git repository (centralized decentralized version control system – the irony is not lost on me). The easiest way is to just scp the .git directory to the server and place them in the directory you plan on containing your repositories. In my case, I created a git directory off of / on the server.

(server)$ sudo mkdir /git

(client)$ git clone --bare project project.git
(client)$ scp -r project.git user@server:/git/

Essentially, we are finished adding the repository to the central server. We can now access the git repo from all clients so long as those users have ssh access to the central server. By default, we can access the repo from the clients using the ssh interface.

(client)$ git pull user@server:/git/project.git master

And we can push changes to the repo in a similar fashion.

(client)$ git push user@server:/git/project.git master

This is about as simple a set up as you can have. The Git server is really quite dumb. The real magic happens on the client side (which is left for another post when I have done a bit more research).

By default, this git server is private. Only users that have shell access to the server have access to push and pull changes. Push access requires private access, but for projects which you want public access, you can set up a git-daemon to serve git clone requests from the public. Rather than run it as a persistent process, you can quite simply tie it into xinetd with this simple configuration (remember, I am using Ubuntu).

(server)$ touch /git/project.git/git-daemon-export-ok
(server)$ sudo apt-get install xinetd
(server)$ sudo vi /etc/xinetd.d/git

service git
{
    disable = no
    type = UNLISTED
    port = 9418
    socket_type = stream
    wait = no
    user = nobody
    server = /usr/bin/git-daemon
    server_args = --inetd --export-all --base-path=/git
}

(server)$ sudo /etc/init.d/xinetd restart

This now gives us a public git repository url we can distribute to others.

(client)$ git clone git://server/project.git

Just remember that you still need to push changes to the repo via ssh.

UPDATE

Talked to one of the guys in #git on irc.freenode.net and he informed me that the usual practice is to create the git repo on the central server first, then clone it to the client and push changes from the client.

The API, or Application Programming Interface, in simple terms, is a set of libraries which can be called which provide linked or shared functionality. In the past, this API was available as binaries or source code which would be compiled into, or released with (in the form of shared libraries) your program.

With the increased popularity of specialized web software, the original idea of the API has evolved (and some argue bastardized). The original, mainstream, bloated and cumbersome SOAP and XMLRPC web API technologies are out in favor of the new hotness – RESTful API’s.

RESTful API

REST, or Representational state transfer, is the style of software architecture which the word wide web was based on. Like all world wide web traffic, RESTful API’s are inherently stateless. They receive a request via a GET or POST, process it, then return a response. The next request is not affected by the previous, nor will future requests be affected by the requests processed in the past.

The great thing about the REST architecture is that it’s horizontally scalable. Assuming an infinite pipe, the web servers can be clustered such that the architecture can be horizontally scaled with money. You can fix scaling issues by throwing hardware at it.

Servers can be added to the cluster and scaled horizontally due to the RESTful nature of the HTTP protocol.

RESTful API as the Application

Although websites are still the main way users interact with internet-based applications, many web applications don’t rely on a web browser at all! An emerging trend is to allow users the ability to access remote resources via an API which provides data for any number of applications – web based or not. Twitter for example, is increasingly accessed via third-party applications such as Twhirl using their RESTful interface.

With the new popularity of web applications that don’t use browsers, I propose changing how these web applications are created. Instead of developing websites followed by the API, or even the API in tandem, why not create the API as the application in and of itself, then write the web application using the RESTful API? Lets think of the website as been just another application making use of remote resources.

By taking advantage of the nature of RESTful API’s we should in theory be able to create a system that scales horizontally. The added bonus is you won’t waste your time creating a website which may not be used by the end users in the future. Mobile and location based applications by their very nature will be using website access less and less.

Another advantage is that you maintain a clear separation of resources throughout your application. It forces developers to decouple the business layer from the display layer. Its very nature stops logic code from creeping into the display (as is the case with numerous PHP web applications).

It also provides many options and avenues when dealing with application management in startups and established businesses. By separating the display in this way, it makes it much easier to farm out display work to third party companies which can focus 100% on that task, decreasing your time to market.

At the recent DevHouseWaterloo meet-up, I proposed this plan to a group of software professionals and tech-savvy students, and they seemed to think this was an excellent way to create new internet based applications. I have been taking this approach with my latest project, and so far I am quite pleased. The biggest advantage I have found is that I can focus on the application. I know I am not a very good website designer nor do I pretend to be. I am much more interested in writing the core software and farming out the work to someone who can give the user interface the care it deserves.

What are the rest of the communities thoughts on this approach to software development? Do you have examples of other companies that take this approach? Are there any reasons not to pursue this development plan?

PostRank is a scoring system developed by AideRSS to rank any kind of online content, such as RSS feed items, blog posts, articles, or news stories. PostRank is based on social engagement, which refers to how interesting or relevant people have found an item or category to be.

Public-facing RESTful API’s are becoming more popular as a way to allow access to services by third-parties. AideRSS recently decoupled their PostRank service and offered it as a stand-alone service allowing any developer to use PostRank within their application. I am in the process of updating this website/blog and I am planning to make use of the PostRank service not only as a tool to measure my engagement with the public, but increase the usefulness of my posts.

What is PostRank Python API and what does it do?

This is an api accessor for python that allows a developer to quickly access all of the services offered by PostRank. It is loosely modeled after Mike Verdone’s Python Twitter Tools in that it is simple to use, and it uses the same arguments as those documented on the PostRank API page.

What are the requirements?

• Python 2.5 (haven’t tested with <2.4)
• simplejson python library (if using json output)

How do I use it?

Using the PostRank Python API is very easy. The first step should be to look at the API documentation provided by the PostRank website.

We can see what options the api needs in order to satisfy the request. We just need to specify the resource that we want to use, then we can add all the arguments we want passed along to PostRank.

If you have selected JSON as your format, the api will return to you a simplejson object which can be accessed just like a multi-dimensional array. When using JSON, it is extremely simple to get at the information provided by PostRank.

There is a bit of a trick when using the ‘postrank‘ resource but it certainly isn’t difficult to use. It works the exact same way – the PostRank API says it wants an array of urls (url[]) so we will pass in an array of urls as an argument (same thing for the feed_id[]).

Following the same pattern as before, we just pass in the resource we want to use and the API arguments.

There really isn’t much more to it. For now, you can find the python library here: http://newthink.net/~ash/software/postrankapi/ When my new website is done, I plan to find a more permanent place for this (and future software). Next step, creating an egg and integrating with ez_install.

Since I had some time this weekend, I decided it was about time I rolled some RPM’s so I could run Rails. Of course, it being over two years since I was last rolling RPM’s for work at Pason, it took me a bit to get back in the swing of things.

I followed the guide found via the Ruby on Rails wiki, and modified it a bit for my server. The two main pieces of software that needed RPM’s were fastCGI and mod_fcgid (slightly different than mod_fastcgid).

Below you can download the RPM’s and the SRPM’s though I will be the first to acknowledge that I cheated a bit. I should have modified the Makefile on each of these programs to make use of an install_dir directive, but since I was building on a clean CentOS4.6 vmware OS, I decided not to go through the hassle. If someone really wants to do it properly, have at it – that’s what SRPMS are for.

RPMS:
mod_fcgid-1.09-1.i386.rpm
fcgi-2.4.0-1.i386.rpm

SRPMS:
mod_fcgid-1.09-1.src.rpm
fcgi-2.4.0-1.src.rpm

First off, I’ll spare you the suspense. I did NOT get offered a job at Google… but I did have a lot of fun!

I was first contacted back in December by a Google talent recruiter. Mike had been in the middle of his Google interviews, so it was an exciting time.

Was I interested? Definitely! There are few in the software industry that wouldn’t jump at a chance to work for Google – one might call them the Mecca of the software industry (but only if it doesn’t offend anyone; if it does, let’s just call it a pretty neat place to work).

I first received an email from the recruiter asking if I was interested, and where I wanted to look for a position. She told me that she thought there was a good fit for me for a Java position in Mountain View, but I asked her about looking for a job in Waterloo. Was it a mistake? Maybe, but she did mention that I could look into a position in Mountain View if the Waterloo position didn’t work out.

Over the Christmas holidays, the recruited emailed me to let me know that the Waterloo position was a C++ position, and that perhaps I would have a better chance finding interest in Mountain View, but the next day she emailed me back to let me know that the Waterloo offices were interested.

I was excited. One of my goals, new years resolutions if you will, was to step out of my comfort zone with regards to software development. It seems that every script I needed to write was being written in Perl and Bash script and every piece of software I was writing professionally was in Java. I wanted get out of the habbit of relying on Java/Perl/Bash. This seemed like a good opportunity as C++ is one of many languages I wanted to work on this year.

Well, I guess what you really want to know is what sorts of things I was asked on my interviews. Well, before that, there are a lot of things that need to be reviewed; first and foremost, asymptotic notation and complexity. The interview questions consisted of asking me to come up with a semi-high level algorithm and then to explain the asymptotic complexity of each part as I went along, and then what the final complexity of the algorithm would be.

The next thing that should be reviewed is your basic sorting algorithms and their respective complexities. I made sure to know heap sort, bubble sort, selection sort, binary sort, merge sort, quick sort and insertion sort. Also important, you should know the memory requirements that each algorithm uses to run (in big-O notation). I was asked in my first interview to not only minimize the runtime, but also the memory footprint (ie. insertion sort although is O(n^2) in runtime, it is O(1) with regards to memory).

The next thing I reviewed was memory structures. Hash’s, binary trees, n-tier trees, etc and the properties that each one possesses. If the interviewer asks you about a binary tree, make sure you know the properties that make a binary tree special so that you can leverage it in your solution.

Now, on to the interviews.

The first interview started with a quick introduction from Steve about where he worked within Google (Google NYC) and what he worked on there (Google maps). Then it was right down to business. The first questions (from my memory, so don’t blame me if it isn’t 100% correct) was as follows:

Given a binary tree, and two leaves, can you find the lowest common node of those leaves?

I won’t give you the answer here. I’ll let you work it out for a bit and perhaps if there is enough discussion in the comments, I’ll leave the answer there. When giving my answer to the interviewer, I think I could have been a lot clearer. I was nervous; very nervous. I really didn’t know what to expect, even though Mike prepared me the night before. The answer I gave was right (as far as a solution and its complexity) however, about 10 minutes after completing my interview, I figured out a way to better leverage the binary tree structure which would have also given a much clearer answer. Also, I was asked to minimize both the runtime and memory usage in my solution.

The next interview I had was about 2 weeks later and it was with Chris (also from the Google NYC offices) but he worked for Google’s blog search. This one started a little different; he asked about what I had worked on at my previous companies and more specifically what I felt the coolest project was. I had so many to choose from at Pason, but I went with the biggest and more complex, the DataStream project. After that, he asked me my ideal problem to solve was if I could leverage all that is Google. This really caught me off guard. I should have known it was going to come; I even read about someone who was asked a similar questions. I spent so much time preparing for the technical part of the interview, I forgot about the lighter part. I wish I could have done it over. My answer was a lame scheduling mash-up for Google maps. I really needed to think bigger… hell, my friends and I have talked for years about cool stuff that we could do if only we had a company that was as big as Google.

I felt like an idiot, but what can you do. No sense dwelling on that. There were two technical questions that comprised this part of the interview. The first was:

If you were designing the Vector class in Java, how would you code the add(Object) method?

Deceptive; not hard per se, but there are a few things that need to be worried about when solving this question.

The next question was the algorithm question:

If you were given an unordered set of plane tickets which represented legs in a trip, can you figure out the order in which to use the tickets?

This was a really fun problem to work on, and again, I’ll save the answer for the comments. Remember that they want you to come up with the lowest runtime algorithm that you can.

Three days after I completed this interview, I got my letter letting me know that they were not going to be offering me a position. Was I disappointed? Sure. I have to admit that I thought about how neat it would be to work at Google, but it wasn’t all bad. It helped me fulfill another one of my new years resolutions – to get back into studying the “science” of computer science, and although stressful, the whole experience was a lot of fun.

Instead, we will add subversion to xinetd. The benefit of this is that subversion is only running when we need access to the repository rather than running all the time. By default, subversion is not set up in xinetd, so the following is how I set it up:

First make sure that xinetd is installed.

# yum install xinetd

Next we want to add the following to our /etc/xinetd.d/svnserve

# default: off
# description: svnserve is the server part of Subversion.
service svn
{
  disable = no
  port = 3690
  socket_type = stream
  protocol = tcp
  wait = no
  user = root
  server = /usr/bin/svnserve
  server_args = -i -r /svn
}

Finally, lets restart xinetd:

/etc/init.d/xinetd restart

Now xinetd will start the subversion repository server every time a subversion request is made.

Here was my process.

# yum install subversion

Ok, that was pretty easy. I now have subversion installed on my system; both from a client and user’s perspective. Next I need to create a subversion repository location on my disk (/svn can be replaced with any location on your server).

# svnadmin create /svn

In /svn/conf/passwd add the following (it is in plaintext, but I am sure there is an encrypted solution):

[users]
achristopher = some secret, yet plaintext password

In the /svn/conf/svnserve.conf, you should set the following:

[general]
anon-access = read
auth-access = write

realm = Test Repo

There, our subversion repository has been set up. Now all we have to do is start the subversion daemon.

# svnserv -r /svn -d

We can test the subversion repo:

# mkdir /tmp/test

# svnlook tree /svn
/
svn/
test/

We have just added a directory to our subversion repository. Now lets checkout this project, and get to work:

# rm -Rf /tmp/test
# svn checkout svn://localhost/svn/test
Checked out revision 1.


I cheated a little there, because I had entered my password before I created this posting, so you can expect to see something like the following:

Authentication realm: Test Repo
Password for 'achristopher':

Just enter your password (the one entered in plain text above) and continue working.

To follow: How to set subversion to work with xinetd (it isnt that exciting, but definitely new post worthy).

I am quite familiar with PHP, so I set out to try to do the same things in rails as I would in PHP. Everything seemed to transfer over without effort… except file upload inputs. These kept me busy for quite some time, and had me wanting to pull out my hair.

I tried to make use of the community; exploring the wiki, reading through the rails cookbook, seeking out blogs that addressed the issue, visiting #rubyonrails on irc.freenode.net on a regular basis. Infact, the IRC community was referring me to the wiki article (HowtoUploadFiles). No matter how I tried, the upload just wouldnt work. I had narrowed the problem down to the multipart => “true” definition, but that was about as far as I could get without help. Luckily, I know a few rails nunjas (gurus) who helped me figure out what to do.

As it turns out, the rails wiki was incorrect. Since the project is opensource, it was easy enough for me to fix the documentation (perhaps the documentation was for a previous version of rails, and hadn’t been updated), but it was still a frustrating week (I was actually only working on it maybe and hour a night, but still) especially since it was such a simple task I was trying to complete.

Hopefully there are no more hiccups like this as I go forward. I am already thinking about a cool new idea which I am excited to work on once I complete this test application.

The book covers Rails development from the beginners level to the fairly advanced. It offers an easy tutorial which one can follow along, and gives a decent amount of information about most of the features they have available.

One of the biggest features that the Rails framework offers is migration. Although it is a fairly small part of Rails, it is a technology which I know is useful to many companies. At Pason, we created our own technology that basically does the same task as migration. It would have been nice to just use the Rails’ migration instead of developing our own.

The biggest issue I am having now, is getting back into the Python-style of coding. Ruby and Python are fairly similar… and very much different from Java and C++ which I am more familiar with. There are also a few speedbumps I am noticing when working on my first Rails project, but I hope to overcome them soon.

While this may not seem like a huge deal to most users, what this means is that you can’t connect to any other mail server when attempting to send emails because Rogers has blocked access. Jenna can’t send emails using her McMaster email from a Rogers account.

When I initially looked at setting up Internet, this was the first question I asked Rogers. They assured me that traffic shaping was the only limitation they put on the account. The customer service agent assured me that their literature even mentions this. Their literature is wrong.

As a way around this problem, you are going to want to set up a SMARTHOST on a linux box within the home network which is attached to the Rogers connection.

I did some testing and determined that Rogers doesn’t block all traffic over port 25, but only traffic that isn’t going to a Rogers controlled mail server.


9:44 ash@galactica:[~]> telnet smtp.broadband.rogers.com 25
Trying 206.190.36.18...
Connected to smtp.broadband.rogers.com (206.190.36.18).
Escape character is '^]'.
220 smtp107.rog.mail.re2.yahoo.com ESMTP


What we can do now, is set up a SMARTHOST in sendmail to basically forward emails from the server to Rogers SMTP server, then on to it’s destination. The following assumes you are the root user.

In your /etc/mail/sendmail.mc file, we need to add the following:

define(`SMART_HOST',`smtp.broadband.rogers.com')


This defines what server we are going to use as a SMARTHOST. The other problem we are going to run into is that unlike other ISP’s which allow you to send via the SMARTHOST based on what your domain is (basically if you are connected on an ISP’s network, you can send using their server) Rogers requires that users authenticate.

To do this, we need to add the following to the /etc/mail/access file:


AuthInfo:smtp.broadband.rogers.com "U:<username>" "P:<password>" "M:PLAIN"


I then changed the permissions to the file since I had my plaintext password in it:

chmod 660 /etc/mail/access

Next, we will regenerate the sendmail.cf and access.db files.


m4 /etc/mail/sendmail.mc > /etc.mail/sendmail.cf
makemap hash /etc/mail/access < /etc/mail/access


Now, restart sendmail, and you should be ready to go.

What you can do now, is set up all of your mail clients to point to the linux server as your outgoing mail server and it will forward all of your emails to Rogers and then on to it’s destination.

Basically, the problem comes from having the POST information still resident after an HTTP POST. If the user were to refresh their browser, that POST information will be resent, resulting in a double post. There are many ways to solve this problem, but the one I will outline (in PHP) seemed like the best for my situation.

Let us assume we have a text input and a button and that our form.php file will process our form as well as present it to the user (yes, I know this isn’t a good idea, but it is just an example).

<form method=”post” action=”form.php”>
    <input name=”mytext” />
    <input name=”submitBtn” type=”submit” />
</form>

When we load the page, we are going to check whether the $_POST variable is set. If it is, then we are going to do something with it.

session_start();

if(isset($_POST['mytext']))
{
    saveToDatabase($_POST['mytext']);
}

Now that we are doing something with this $_POST['mytext'] variable, we can see the double-post problem materialize. Since the HTTP POST is still populated, when the user presses refresh on their browser, the $_POST['mytext'] will be saved to the database twice.

What we need to do is get rid of the $_POST variable after we are done using it.

At the beginning of the form.php file I am going to create a server-side session to store any post information we get. Then I have a check to see whether the $_POST array has been populated.

If the $_POST has been populated, we are going to store the post information in the server-side session we have started and we send a blank header back out (which removes the HTTP POST).

if (isset($_POST['myText']))
{
    $_SESSION['myText'] = $_POST['myText'];
    header(“Location: form.php”);
}

Now, we have gotten rid of the POST which removes the double-post problem, and have stored the information from the $_POST for use. We can then access the POST information stored in the session. Once we have copied the POST information from the session, we will unset the session so that the information doesn’t stay resident.

if (isset($_SESSION['myText']))
{
    $myEnteredText = $_SESSION['myText'];
    unset($_SESSION['myText']);
}

Now you can use the POST information without running the risk of a double post.

If you would like to see this tutorial in action, create a form.php file on your webserver, and paste the following:

<?
    session_start();

    if (isset($_SESSION['myText']))
    {
        $myEnteredText = $_SESSION['myText'];
        unset($_SESSION['myText']);
    }

    if (isset($_POST['myText']))
    {
        $_SESSION['myText'] = $_POST['myText'];
        header(“Location: form.php”);
    }
?>

<html>
<head>

</head>
<body>

<form action=”form.php” method=”POST”>

<?
    if (isset($myEnteredText))
    {
        echo “<p>The message you entered was \”".$myEnteredText.”\”</p>”;
    }
?>

<input name=”myText” type=”text” />
<input name=”submitBtn” type=”submit” value=”ok” />

</form>
</body>

</html>

Now, I know not everyone has a Linux machine just lying around, but I do, and maybe in the future it might be enough to convince you to build your own file-server for your home (for media perhaps?).

Before starting, you need to make sure you have the ffmpeg binary installed on your Linux box. As of this posting, I have ffmpeg-0.4.9-0.3.20051207.2.el4.rf installed on my CentOS4 machine. I will leave it as an exercise to you to figure out how to install it if you don’t have it.

ash@galactica:[/tmp]> ffmpeg -vcodec xvid -b 500 -qmin 1 -qmax 7 -bufsize 4096 -g 300 -acodec aac -ab 96 -i doctor.who.102.the.end.of.the.world.avi -s 320×240 -aspect 16:9 doctor.who.102.the.end.of.the.world.mp4

Here is a run down of what the switches mean and how they affect your converted video:

-vcodec — this is the video codec you want the converted video to use. For iPod use, I suggest xvid. It is basically the same codec as mpeg4 (except open source) and you don’t run into buffer-underflow problems as often.

-b — this is the video bitrate in kbit/s (higher bitrate means higher quality).

-qmin & -qmax — this controls the quality when dealing with VBR (variable bit rate). Basically a range to keep the quality between.

-acodec — the audio codec to use. When converting to iPod video, this must be aac since that is the only format the iPod supports.

-ab — this is the audio bitrate in kbit/s. 96 kbit/s offers decent audio quality for iPod use.

-i — input file (obviously?).

-s — size of the output video. The iPod supports 320×240, 480×480 and with the newest firmware, now supports 640×480. If you want to retain the source aspect ratio, make sure you calculate the new size to follow the sources aspect ratio.

-aspect — aspect ratio for the converted video. This is especially important if your input video is 16:9 and you want to keep the ratio and replace the extra space with black bars. The aspect ratio is usually 4:3 or 16:9.

Now,after you have run this command, your video should begin to be converted. The time it takes to convert depends greatly on the hardware being used. I often offload this task to a P-III 800 machine, and it seems to be able to encode a video in the same time it would take to watch it. Again, the nice thing about offloading this task is that you can now use your main system for other fun tasks.

Might I suggest maybe using your main machine to whip up a quick script to encode a list of videos, perhaps at night when spare cpu cycles are pleniful? The nice thing about batching up video conversions like this, is that it is also compatible with mac’s running OSX. Sure, you could use the mac-GUI program to do it, but that is going to require a whole lot more human interaction to convert 10 shows.

For mor information of supported codecs, see the FFMPEG Documentation