Category Archives: Technology

MarkUs, Squad, How’s / Refactor My Code, Belated Happy Holidays, and Oh Yeah – I’m Not Dead

Belated happy holidays! My last post was over a month ago, and so my blog has a nice layer of web-dust on it right now.  Well, here I am to ease your mind.  I’m still alive!

But that almost wasn’t true.

I won’t bore you with the details – I’ll just give you the facts, and let you fill in the blanks.

  1. My girlfriend Em, her sister Cassie, and myself, were up in Collingwood on New Years Day, enjoying a relaxing day at a Norwegian spa (the outdoor baths were amazing – how awesome is it to be in a boiling hot tub, while simultaneously, your hair is so frozen that it’s snapping off in your hands?)
  2. The roads that night were treacherous.  Snowy, un-plowed, and dark.  I had borrowed my Mom’s car for the trip, and we took it realllllly slow.
  3. After a tortoise-paced two hour ride back to Em’s place in Newmarket, and then another two hour drive from Newmarket to my home in Grimsby the next day, I was getting pretty sick of winter driving.  On top of that, the brakes seemed to be acting funny.  I found myself sliding a lot, and there didn’t seem to be a lot of resistance when I put my foot down.
  4. The next day, my Mom takes the car to go to work.  She doesn’t even leave the drive-way.  The brakes hadn’t been acting funny:  the brakes hadn’t been acting at all.  Turns out we had a leaky brake-line for the entire trip…
  5. Guts of the story:  I think we drove home from Collingwood with about 35% brake power in one of the worst snow storms I’ve ever driven in.

Breakfast tasted especially good for us that morning.

Anyhow, now where was I?  Oh yeah…

MarkUs

MarkUs 0.6 got kicked out a week or so ago.  The MarkUs Team kicked the crap out of a bunch of tickets over the holidays, and I think we ended up with a pretty solid release.  MarkUs is being used again at UofT this semester, and Byron Weber Becker is also piloting it at UWaterloo.  I’ll cautiously say that things seem to be going well for this release.  Great job, MarkUs Team!

I’m TAing the students working on MarkUs for Greg’s UCOSP course again.  We had a fantastic code-sprint this past weekend!  The new team members have already started working on tickets and submitting code to review.  I think we’re on our way into another highly productive semester.

A Few More Web-Based Code Review Tools

Remember that big list of code review tools I put up a while back?  I’ve got a few more to add:

How’s My Code

This is a pretty dead-simple code review tool that came about during a Rails Rumble a few months back.  It has that “big friendly buttons and round corners” web 2.0 thingy going on.  I haven’t gone so far as to actually try it out, but I did watch this web-cast:

Not bad if you just want to get your code out there, and get your team commenting on your changes…

A few things caught my attention:

  • It’s a web service, so you don’t install it…you sign up for it
  • It currently only supports Git.  🙁
  • There doesn’t seem to be any support for contextual per-line commenting…I think it’s just file by file commenting.  I’d love it if I could comment on a single line of code…

Still, if I was working on a project hosted on a Git repo, and I needed a dead-simple code review service, and I needed it quickly, I could probably do a lot worse than this.

Click here to check out How’s My Code

Squad

Remember that time when I wrote about how it might be neat if somebody created a code review tool on top of Google Wave? (or Bespin for that matter – though I didn’t mention it, and should have)

Looks like somebody else was thinking the same thing. And a few months earlier.  I guess it’s not easy to be super cutting-edge.

Anyhow, looks like something Wave-ish (yet simpler, more streamlined) has been developed.  Check out Squad.

I just tried this thing out for free (with ads, features locked, etc), and it was pretty cool.  I could see something like this being very useful for showing new MarkUs team members how to do things.  Actually, I just used it to show a new member of the MarkUs team how to use Shoulda.  Pretty useful.  It sure beats coding through IRC and Pastie.org.

A few things to keep in mind:

  1. Super simple to get going – open up a session, and send someone a generated link, and you’re both coding in no time
  2. One person codes at a time…so while one person edits, the screen is locked for everyone else
  3. Ads on the left are a little annoying
  4. Sports syntax highlighting for a number of languages – though I noticed that Ruby wasn’t one of them.  :/

I can see this becoming second nature, like Pastie.org.

Who knows – I might find more reasons to use Squad as the semester rolls, and MarkUs picks up speed.  I’ll keep you posted.

If you missed the link I put in above, click here to check out Squad

Refactor My Code

This service crowd-sources code review requests, so don’t expect to get deep architectural feedback, because it’ll probably come from strangers who don’t/barely know your code base.

The idea is – slap a piece of code that you’d like refactored up on the site, and then others swoop in with brilliant suggestions (assuming of course, you asked your question properly…check this out…what the…?)

This is the sort of thing that CS instructors probably wouldn’t want their students using too much…it’d then become solve-my-CS-programming-assignment.com.

Still, I think it counts as peer code review.  And it’s way different that anything else I’ve been looking at.  Nice.

Click here to check out Refactor My Code

Anyhow, I just thought I’d mention those.

Fiddling Around with Skype

As I said last week, I’ve been working with a partner (Mohammad Jalali) on a project for our networks course.

The idea:  given an arbitrary IP and port number, we want to find a way of determining whether or not there is an FTP server, an HTTP server, or a Skype node on the other side.  FTP and HTTP are trivial – those protocols essentially announce themselves to the world.

Skype clients, on the other hand, act a little more strangely.  Skype goes out of its way to hide its traffic – from straight-up encryption, down to making their client executable really hard to reverse engineer.  Because of this, Skype has been an interesting challenge to the hacker community.

Anyhow, my partner and I have learned a few interesting things about Skype – and in particular, we’ve found a reliable way to determine whether or not Skype is running behind an arbitrary IP and port.  Cool.

Fact 1:  Skype pretends to be an HTTP server

I’m serious, it does.  Using Wireshark, we noticed that both UDP and TCP packets were being sent to one particular port.  Pretty funny behavior…so, we took a closer look.  And this is what we found.  Pop open your Skype client, connect to the network, then use nmap to find the ports that Skype is using:


$>nmap localhost -p10000-50000

Starting Nmap 5.00 ( http://nmap.org ) at 2009-12-01 20:33 EST
Interesting ports on localhost (127.0.0.1):
Not shown: 39999 closed ports
PORT      STATE SERVICE
48915/tcp open  unknown

Ok, cool – there’s something at 48915, and it looks like it accepts TCP connections.  Pop open Telnet, connect to it, and feed it an HTTP request:


$>telnet localhost 48915
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
HTTP/1.0 404 Not Found
Connection closed by foreign host.

Ok, we got an HTTP response – looks like there’s an HTTP server back there, right?

Wrong.  Reconnect, and send it some garbage:


$>telnet localhost 48915
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
thisissomegarbagetextthatisnotanHTTPrequest
��Nun��2�=���1��N$O/(����
���u.)(yy�g��$
ș�oT�b렑�-z#x�&���[P���\��(yVO���

See all of those funny characters down at the bottom?  That’s what I got back.  In the words of Obi-Wan Kenobi…that’s no HTTP server…it’s a space station (Skype node).

So we’ve learned something here – Skype opens a port, and “spoofs” an HTTP server.  We can easily check for this – just write a script that connects to a port, spews some garbage, and check to see if we got binary garbage back.

It’s so easy, that someone else has already done it.  Remember that nmap tool we used earlier?  Somebody over in that camp wrote a script for the Nmap Scripting Engine that runs this exact analysis on some ip/port.  Don’t believe me?  Read the script yourself! We stumbled upon that script while trying to figure out what Skype was doing with the spoofed HTTP server.
And sure enough:

$>nmap localhost -p48915 --script skype.nse
Starting Nmap 5.00 ( http://nmap.org ) at 2009-12-01 20:45 EST
Interesting ports on localhost (127.0.0.1):
PORT      STATE SERVICE
48915/tcp open  skype2

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

Hmph.  So much for cutting edge, never-been-done research.  Go figure.

Fact 2:  Given some UDP packets, Skype echos back a predictable pattern

For this part, we’re pretty sure no one else has tried this.

While connected to Skype, we recorded some packets with tcpdump.  We wrote a script that loaded up those packets, and could “replay” the packet payloads to an arbitrary IP and port.

So, we played some packets against an IP/port with Skype behind it.  Most of the time, we got TCP packets with RST flags (which is TCP’s way of telling us to “shut yer trap”).  But wayyyy down in the middle, there was a section of UDP packets that actually got a response:

From Misc Blog Images

192.168.0.19 was the computer we were playing the packets from, and 192.168.0.14 was the computer with Skype running on it. See those UDP packets that are getting echoed back?  That’s the interesting part…instead of just shutting us down with RST’s, Skype appears to be saying something back.

So, is there a pattern in all of this?  Actually yes.  We isolated 4 of those UDP packets, and repeatedly fired them at the same IP/Port on the computer running Skype, and we found a pattern.

The pattern:  the first two bytes that are sent in our UDP packets are echo’d back to us in the first two bytes of the UDP packets that come back.

So, for example,  one UDP payload we sent looked like this:

92 40 02 a1 66 65 ea 0d 8c 82 c3 0c 27 cd c5 e7
4e 78 fe a1 50 a6

And we got back:

92 40 17 c0 a8 00 13 74 a0 41 f0

See that common 92 40?  Bingo.  😉

And it’s pretty consistent – if we repeat the same UDP packet, we get (almost) the same response.


92 40 67 c0 a8 00 13 11 00 10 4f

And if we repeat again…


92 40 37 c0 a8 00 13 68 08 43 3a

92, 40, and c0, a8, 00, 13. Nice!  Looks like a fingerprint to me!

Except…

Except, remember, we already found a way of determining whether or not Skype was running behind a given IP/port.  This last finding was just bonus.  My partner and I aren’t sure if our instructor is going to let us stay with this topic, seeing as how it’s pretty much been solved by other people before.  We’ve only got 2 weeks before this project is due, so…if we get another project, let’s hope it’s relatively simple.  Push come to shove, we could always try to fingerprint a different protocol…maybe BitTorrent clients.

Either way, working on this stuff has been pretty cool…and it let me try out some pretty neat tools that are usually reserved for the people with coloured hats (and no, I didn’t mean Red Hat):

  • nmap: port scanner that can also do service/os fingerprinting
  • Scapy: sculpt, gut, spoof, manipulate, and send packets – the power of C, with the simplicity of Python!  We used Scapy as a library while writing our scripts.  Lots of potential with this tool.  Feel like poisoning an ARP cache?  Scapy is for you!
  • Wireshark: a network student’s best friend.

Click here to check out Mohammad’s blog post about this project.

Can Google Wave Bring Something New to Peer Code Review?

If you haven’t already heard about Google Wave, watch this video.

In a nutshell, Google Wave is an attempt at integrating wikis, instant messaging, email and social networking, into a nice, tight, Google-y ball of goodness.

What’s particularly interesting, is that developers can write “gadgets” by coding against the Google Wave API.  Check this out – a collaborative tool for systems modeling built into the Google Wave client.  That’s pretty cool!

Which makes me wonder… could a code review tool be built on top of Google Wave?  And if so, what services, if any, does the Wave protocol offer that might make such an application superior to other web-based code review tools?

What does Google Wave offer that would make such a code review tool special, more effective, and better?

Playing Around with FTP

I’m taking a Computer Networks course this semester, and for my final project, my partner and I are trying to create signatures for FTP, HTTP, and Skype packets.

The big idea:  we want to create some signatures, and then “replay” those signatures against some arbitrary IP and port.  If we get a response, we analyze the response to see if it matches what we expect from the signature.  If it matches, chances are we’ve determined what kind of server is behind that IP/Port.

FTP and HTTP are the trivial ones.  Skype is going to be quite a bit harder.

Anyhow, here is what I’ve found out about FTP…

FTP

FTP runs over a TCP connection, so if you’ve got Telnet, then you’ve got a basic FTP client.  Traditionally, FTP servers run on port 21 – but really you could put one on whichever port you feel like.

So, I’m going to try to futz around with the Mozilla public FTP server, and show you what I get.

First, I’ll connect to the FTP server with Telnet, like so:


mike@faceplant-linux:~$ telnet ftp.mozilla.org 21

Here’s what comes back:


Trying 63.245.208.138...
Connected to dm-ftp01.mozilla.org.
Escape character is '^]'.
220-
220-   ftp.mozilla.org / archive.mozilla.org - files are in /pub/mozilla.org
220-
220-   Notice: This server is the only place to obtain nightly builds and needs to
220-   remain available to developers and testers. High bandwidth servers that
220-   contain the public release files are available at ftp://releases.mozilla.org/
220-   If you need to link to a public release, please link to the release server,
220-   not here. Thanks!
220-
220-   Attempts to download high traffic release files from this server will get a
220-   "550 Permission denied." response.
220

If I type in anything and press RETURN, the server responds with:
530 Please login with USER and PASS.
Since I don’t have an account, I’ll just use the basic anonymous one:

USER anonymous

The server responds back with:

331 Please specify the password.

I don’t have a password, so I’ll just try a blank one…

PASS

and blam, I get a ton of stuff back:

230-
230-   ftp.mozilla.org / archive.mozilla.org - files are in /pub/mozilla.org
230-
230-   Notice: This server is the only place to obtain nightly builds and needs to
230-   remain available to developers and testers. High bandwidth servers that
230-   contain the public release files are available at ftp://releases.mozilla.org/
230-   If you need to link to a public release, please link to the release server,
230-   not here. Thanks!
230-
230-   Attempts to download high traffic release files from this server will get a
230-   "550 Permission denied." response.
230 Login successful.

Hey alright, I’m in!  Er…where exactly am I, though?  I type in PWD, and the server responds with “/”.  So I’m in the root.  Nice.

So what’s in the root directory, anyhow?  I type in LIST.  Here’s what I get back:

425 Use PORT or PASV first.

And here’s where it gets interesting.  This Telnet session I’ve got here is like a control window.  But if I want any actual data from the server, I’m going to need to either open up one of my ports (and do some port-forwarding on my router) to receive it (PORT), or connect to another port that the FTP server can pipe data through (with PASV).

I’d rather not go through all of the trouble of port-forwarding, so I’m going to choose the latter.  I type in PASV.  The server responds with:

227 Entering Passive Mode (63,245,208,138,225,55)

So what does that big string of numbers mean?  The first 4 are the IP address I’m to connect to (63.245.208.138).  The last two tell me what PORT to connect to.  The formula to determine the port number is N1*256 + N2.  N1, in this case, is 225.  N2 is 55.  So 225*256 + 55 is 57655.

So I open another Telnet in a separate window, connect to 63.245.208.138 on port 57655, and get….

nothing.

Yep, just a blank.  I’ve made the connection, but I haven’t asked for any data, so there’s nothing for the connection to say.

However, if I type LIST again in the command window, I get

150 Here comes the directory listing.
226 Directory send OK.

sent into the control window, and

-rw-r--r--    1 ftp      ftp           528 Nov 01  2007 README
-rw-r--r--    1 ftp      ftp           560 Sep 28  2007 index.html
drwxr-xr-x   34 ftp      ftp          4096 Nov 24 23:32 pub
Connection closed by foreign host.

pumped into my data window.  Notice that the connection closed in the data window.  That means that, for every bit of data I want, I either need to redo the whole PASV thing, or supply a PORT that the server can connect to.  Bleh.

Let’s see what else I can do.  I type in “CWD pub” to change to the pub directory.  Using PASV and LIST, I get the following from another data window:

drwxrwxr-x    3 ftp      ftp          4096 Jun 05  2002 OJI
-rw-rw-r--    1 ftp      ftp          1144 Jul 03  2001 README
drwxr-xr-x 5561 ftp      ftp        430080 Nov 24 22:14 addons
drwxr-xr-x    2 ftp      ftp          4096 Jul 05  2005 artwork
drwxr-xr-x    2 ftp      ftp          4096 Jun 13  2008 bouncer
drwxrwxr-x    5 ftp      ftp          4096 Apr 20  2009 calendar
drwxrwxr-x    6 ftp      ftp          4096 Aug 11  2008 camino
drwxr-xr-x   16 ftp      ftp          4096 Oct 16  2006 cck
drwxrwxr-x    3 ftp      ftp          4096 Jul 10  2004 chimera
drwxrwxr-x   12 ftp      ftp          4096 Aug 31  2001 data
drwxrwxr-x    8 ftp      ftp          4096 Jun 19  2007 directory
drwxr-xr-x    4 ftp      ftp          4096 May 17  2005 diskimages
drwxrwxr-x    4 ftp      ftp          4096 Jul 26  2008 extensions
drwxrwxr-x    4 ftp      ftp          4096 May 16  2003 firebird
drwxrwxr-x    5 ftp      ftp          4096 Aug 12  2008 firefox
drwxrwxr-x    3 ftp      ftp          4096 Aug 07  1999 grendel
drwxrwxr-x    5 ftp      ftp          4096 Mar 22  2009 js
drwxrwxr-x    4 ftp      ftp          4096 Oct 22  2004 l10n-kits
drwxrwxr-x    2 ftp      ftp          4096 Nov 24 20:28 labs
-rw-r--r--    1 ftp      ftp       1868178 Sep 17  2003 ls-lR
-rw-rw-r--    1 ftp      ftp        169159 Sep 17  2003 ls-lR.gz
drwxr-sr-x    4 ftp      ftp          4096 Sep 15  2005 minimo
drwxrwsr-x   12 ftp      ftp          4096 Nov 11 06:09 mobile
drwxrwxr-x   15 ftp      ftp          4096 Jan 04  2008 mozilla
lrwxrwxrwx    1 ftp      ftp             1 Sep 15  2006 mozilla.org -> .
drwxrwxr-x    2 ftp      ftp          4096 Aug 25  1998 msgsdk
drwxrwxr-x    5 ftp      ftp          4096 Jul 09  2002 nspr
drwxrwxr-x    4 ftp      ftp          4096 Sep 23  2002 phoenix
drwxrwxr-x    3 ftp      ftp          4096 Aug 03  2000 profiles
drwxrwxr-x    6 ftp      ftp          4096 Aug 12  2008 seamonkey
drwxrwxr-x    5 ftp      ftp          4096 May 04  2006 security
drwxr-xr-x    5 ftp      ftp          4096 Aug 13  2008 static-analysis
drwxrwxr-x    8 ftp      ftp          4096 Sep 24 19:03 thunderbird
drwxrwsr-x    4 ftp      ftp         20480 Nov 19 02:26 webtools
drwxrwxr-x    6 ftp      ftp          4096 Aug 11  2008 xulrunner
drwxr-xr-x    2 ftp      ftp          4096 Sep 12 05:51 zz
Connection closed by foreign host.

Nice.  Alright, now let’s see if I can download one of those files.  I’m going to try to download README.  Using PASV, I create a new data window, and then I type:

RETR README

And, after a little wait, my data window gets:

Welcome to ftp.mozilla.org!
This is the main distribution point of software and developer tools
related to the Mozilla project.  For more information, see our home
page (http://www.mozilla.org/) Go here to download Netscape Communicator:
http://home.netscape.com/download/
A list of ftp.mozilla.org's mirror sites can be found at:
http://www.mozilla.org/mirrors.html
This site contains source code that is subject to the U.S. Export
Administration Regulations and other U.S. law, and may not be exported
or re-exported to certain countries (currently Afghanistan (Taliban
controlled areas), Cuba, Iran, Iraq, Libya, North Korea, Sudan and
Syria) or to persons or entities prohibited from receiving U.S.
exports (including Denied Parties, entities on the Bureau of Export
Administration Entity List, and Specially Designated Nationals).
If you plan to mirror our site read our crypto FAQ. Send mail to
mirrors@mozilla.org to be added to our mirrors list.
http://www.mozilla.org/crypto-faq.html#2-1
We do not guarantee that any source code or executable code
available from the mozilla.org domain is Year 2000 compliant.
Connection closed by foreign host.

Awesome! I think I have enough information to come up with some kind of signature.

Resources

What, you think I figured all this stuff out alone?  No way – I had some help:

Code Spelunking for Students

Last Friday, I went to the FSOSS conference at Seneca@York Campus with Zuzel and Greg.

One of the talks I attended was an open discussion about getting students involved in open source software.

I’m going off of memory there, but I believe one of the speakers at that talk said something like:

Students generally don’t have to deal with large code-bases in their school assignments…1000 lines of code is really nothing.  When students work on an open source project, they get dropped into a massive code-base with only a fork, a spoon, and a compass.  They have to find their way around, and that’s where the real challenge and learning is.  This is a skill that most students just don’t get with normal school assignments.

Again, I’m paraphrasing.

So is this true?  Hm.

During my undergraduate career, I’ve certainly had to explore strange code that someone else has written.  But nothing even close to the size of, say, the Mozilla Firefox code-base.  Or the Chromium code-base.  I mean, these are massive wads of code.  This is not a criticism of my teachers or the UofT CS program by any means – it’s just an observation.

But some students explore these large code-bases on their own in their free time.  During my (admittedly brief) break before summer work began, I started poking around the Firefox code.  I made two discoveries:

  1. The code that I saw was, in my opinion, very well written
  2. I was completely lost, and didn’t know where to start

I still haven’t worked on any software that is nearly as large as Firefox.  Not even close.  MarkUs is a nice chunk of code, but minuscule in comparison.

So just go with me on this for a second.  Let’s assume that a large code base is intimidating and difficult for students to wrap their heads around, and this is one of the main challenges in getting those students to contribute to open source software.

Again, I only have my own experience to back up that claim.  Looking at Firefox, I didn’t know where to start.  I didn’t know where to go.  I didn’t know which way was up.  I was lost.

So how can students get a better grasp on a mountain of code?  A few ideas:

  1. Write tests for the code, starting small and going big.  This is a relatively easy way to play with the code without having to change it. This assumes, of course, that the software has been designed to be easy to test…
  2. Ask someone else.  Go into the appropriate IRC channel and ask around.  This, of course, has it’s own problems.
  3. Read up on the developer documentation.  Let’s just hope it’s up to date and relevant…
  4. Read up on someone else’s experiences exploring the same code base.  Good luck finding those.

Not Quite Blueprints

I’ve always thought of computer software as being like an invisible machine inside my computer.

And to me, the source code is a bit like the description of the blueprints for that invisible machine.  It’s not the top-down crystal-clear cutaways that a blueprint provides…it’s a flat, textual interpretation of those blueprints.  And it takes quite a bit of reading before those descriptions sink in, and the “personality” of the machine becomes clearer.

In his article “Code Spelunking Redux“, George V. Neville-Neil says:

Working in this way is a bit like trying to understand the United States by staring at a street sign in New York City. The ability to look at a high-level representation of the underlying system without the fine details would be perhaps the best tool for the code spelunker. Being able to think of software as a map that can be navigated in different ways—for example, by class relations and call graphs—would make code spelunkers far more productive.

I was thinking a lot about that on my ride home from FSOSS.  When I got home to my computer, I found out that there are some really cool alternative ways of viewing software.  Here are three that I found quite interesting:

Codecity

Imagine that you’re curious about developing on Firefox.  You can wade through the swaths of source code…

or you can stroll through a city that represents the software:

CodeCity

CodeCity

CodeCity is an integrated environment for software analysis, in which software systems are visualized as interactive, navigable 3D cities. The classes are represented as buildings in the city, while the packages are depicted as the districts in which the buildings reside. The visible properties of the city artifacts depict a set of chosen software metrics, as in the polymetric views of CodeCrawler.

Imagine virtually driving around that city, hearing a guided tour through your headphones…you can walk into buildings, check out the different floors…check the plumbing.  Interesting idea.

Cocoviz

Or how about a neighbourhood…

CocoViz

CocoViz

CocoViz address software comprehension by a combination of visualization and audio. It uses common place metaphors (like houses) for an intuitive understanding of software structures and evolution.

For each source code entity, evolution and structural aspects are mapped to such metaphors and annotated with different audio, to represent concepts such as design erosion, code smells or evolution metrics.

The tool is used in the software evolution analysis domain but offers DB-, XML-importer and a plugin architecture to extend its use into other domains.

EvoSpaces

Another attempt at using the architecture/neighbourhood metaphor.  This one does a neat job of displaying execution traces though – check out the video demo.

These are cool ideas.

But are they useful? Are they usable?  Do they work? Could they help students get a firm grasp on a large code-base?  Can they help visualize the evolution of software?

Has anyone actually used any of these?