Category Archives: Firefox

DocShell in a Nutshell – Part 3: Maturation (2005 – 2010)

Whoops

First off, an apology. I’ve fallen behind on these posts, and that’s not good – the iron has cooled, and I was taught to strike it while it was hot. I was hit with classic blogcrastination.

Secondly, another apology – I made a few errors in my last post, and I’d like to correct them:

  1. It’s come to my attention that I played a little fast and loose with the notions of “global history” and “session history”. They’re really two completely different things. Specifically, global history is what populates the AwesomeBar. Global history’s job is to remember every site you visit, regardless of browser window or tab. Session history is a different beast altogether – session history is the history inside the back-forward buttons. Every time you click on a link from one page, and travel to the next, you create a little nugget of session history. And when you click the back button, you move backwards in that session history. That’s the difference between the two – “like chalk and cheese”, as NeilAway said when he brought this to my attention.
  2. I also said that the docshell/ folder was created on Travis’s first landing on October 15th, 1998. This is not true – the docshell/ folder was created several months earlier, in this commit by “kipp”, dated July 18, 1998.

I’ve altered my last post to contain the above information, along with details on what I found in the time of that commit to Travis’s first landing. Maybe go back and give that a quick skim while I wait. Look for the string “correction” to see what I’ve changed.

I also got some confirmation from Travis himself over Twitter regarding my last post:

@mike_conley Looks like general right flow as far as 14 years ago memory can aid. 🙂 Many context points surround…
@mike_conley 1) At that time, Mozilla was still largely in walls of Netscape, so many reviews/ alignment happened in person vs public docs.
@mike_conley 2) XPCOM ideas were new and many parts of system were straddling C++ objects and Interface models.
@mike_conley 3) XUL was also new and boundaries of what rendering belonged in browser shell vs. general rendering we’re [sic] being defined.
@mike_conley 4) JS access to XPCOM was also new driving rethinking of JS control vs embedding control.
@mike_conley There was a massive unwinding of the monolith and (re)defining of what it meant to build a browser inside a rendered chrome.

It’s cool to hear from the guy who really got the ball rolling here. The web is wonderful!

Finally, one last apology – this is a long-ass blog post. I’ve been working on it off and on for about 3 months, and it’s gotten pretty massive. Strap yourself into whatever chair you’re near, grab a thermos, cancel any appointments, and flip on your answering machine. This is going to be a long ride.

Oh come on, it’s not that bad, right? … right?

OK, let’s get back down to it. Where were we?

2005

A frame spoofing bug

Ah, yes – 2005 had just started. This was just a few weeks after a community driven effort put a full-page ad for Firefox in the New York Times. Only a month earlier, a New York Times article highlighted Firefox, and how it was starting to eat into Internet Explorer’s market share.

So what was going on in DocShell? Here are the bits I found most interesting. They’re kinda few and far between, since DocShell appears to have stabilized quite a bit by now. Mostly tiny bugfixes are landed, with the occasional interesting blip showing up. I guess this is a sign of a “mature” section of the codebase.

I found this commit on January 11th, 2005 pretty interesting. This patch fixes bug 103638 (and bug 273699 while it’s at it). What was going on was that if you had two Firefox windows open, both with <frameset>’s, where two <frames> had the same name attribute, it was possible for links clicked in one to sometimes open in the other. Youch! That’s a pretty serious security vulnerability. jst’s patch added a bunch of checks and smarter selection for link targets.

One of those new checks involved adding a new static function called CanAccessItem to nsDocShell.cpp, and having FindItemWithName (an nsDocShell instance method used to find some child nsIDocShellTreeItem with a particular name) take a new parameter of the “original requestor”, and ensuring that whichever nsIDocShellTreeItem we eventually landed on with the name that was requested passes the CanAccessItem test with the original requestor.

DocShell and session history

There are two commits, one on January 20th, 2005, and one on January 30th, 2005, both of which fix different bugs, but are interrelated and I want to talk about them for a second.

The first commit, for bug 277224, fixes a problem where if we change location to an anchor located within a document within a <script> tag, we stop loading the page content because the browser thinks we’re about to start loading a document at a different location. bz fixed the more common case of location change via setting document.location.href in bug 233963. Bug 277224 is interested in the case where document.location.href is modified with the .replace() method.

The solution that bz uses is to add new flags for nsIDocShellLoadInfo, which gives more power in how to stop loading a page. Specifically, it adds a LOAD_FLAGS_STOP_CONTENT flag which allows the caller to stop the rendering of content and all network activity (where the default was just to stop network activity). I believe what happens is that replace() causes an InternalLoad to kick off, and we need content rendering to be stopped in order for this new load to take over properly. That’s my reading on the situation, anyhow. If bz or anybody else examining that patch has another interpretation, please let me know!

So what about the commit on January 30th? Well that one also involves anchors. What was happening was that if we browsed to some page, and then clicked a link that scrolled us to an anchor in that page, clicking back would reload the entire document off the cache again, when we really just need to restore the old scroll position.

The patch to fix this basically detected the case where we were going back from an anchor to a non-anchor but had the same URL, and allowed a scroll in that case.

So how is this related to the commit for bug 277224? Well, what it shows is that at this time, DocShell was responsible for not just knowing how to load a document and subdocuments, but also about the user’s state in that document – specifically, their scroll position. It also more firmly establishes the link between DocShell and Session History – as DocShell traverses pages, it communicates with Session History to let it know about those transitions, and refers to it when traveling backwards and forwards, and when restoring state for those session history entries.

I just thought that was kinda neat to know.

Window pains

On February 8th, 2005, danm landed a patch to fix bug 278143, which was a bug that caused windows opened with window.open to open in a new window if they had no target specified. This wouldn’t normally be a problem, except that this could override a user preference to open those new windows in new tabs instead. So that was bad.

This was simply a matter of adding a check for the null case for the target window name in nsWindowWatcher. No big deal.

The reason I bring this code up, is because I find it familiar – I brushed by it somewhat when I was working on making it possible to open new windows for multi-process Firefox.

Semi-related (because of the “popup” nature of things), is a commit on February 23rd, 2005. This one is for bug 277574, which makes it so that modal HTTP auth prompts focus the tabs that spawn them. This patch works by making sure HTTP auth prompts fire the same DOMWillOpenModalDialog and DOMModalDialogClosed events that tabbrowser listens for to focus tabs.

The copy and the cache

On March 11, 2005, NeilAway landed a commit to add the “Copy Image” command item to the context menu. This was for bug 135300.

What’s interesting here is that “Copy Image Location” was already in the context menu, and in the bug, it looks like there’s some contention over whether or not to keep it. It seems that right around here, the solution they go with is to copy both the image and the image location to the clipboard, and mark each copy with the right “flavours”, so that if you were to paste to a program or field that accepted an “image” flavour, like Photoshop, you’d get the image. If you pasted to a program or field that accepted a “text” flavour, like Notepad, you’d get the image URL.

That’s the solution that was landed, anyhow. Notice that nowadays, Firefox has context menu items that allow users to copy just the image, and just the URL – so at some point, this approach was deemed wanting. I’ll keep my eye out to see if I can find out where that happened, and why. If anybody knows, please comment!

On April 28, 2005, roc landed a commit for bug 240276, which splits up something called “nsGfxScrollFrame” into two things – nsHTMLScrollFrame and nsXULScrollFrame. It seems like, up until this point, layout for both XUL and HTML scrollable frames were handled by the same code. This meant that we were using XUL box-model style layout for HTML, and XUL layout is… well… kind of tricky to work with. This patch helped to further distance our HTML rendering from our XUL rendering. As for how this affected DocShell, the patch removed some scroll calculations from DocShell, where they probably didn’t belong in the first place.

On May 4, 2005, Brian Ryner landed a patch which made it possible to move back and forward across web pages much more quickly. This was for bug 274784, and a key part of a project called “fastback”. When you view a web page, a DocShell is put in charge of requesting network activity to retrieve the document source, and then passing that source onto an appropriate nsIContentViewer. Up until Brian’s patch, it looks like every nsIContentViewer was just getting thrown away after browsing away from a page. His patch made it possible to store a certain number of these nsIContentViewers in the session history of the window, and then retrieve it when we browse back or forward to the associated page. This is a textbook trade-off between speed (the time to instantiate and initialize an nsIContentViewer) and space (stored nsIContentViewers consume memory). And it looks like the trade-off paid-off! We still cache nsIContentViewers to this day. What’s interesting about Brian’s patch is that it exposes an about:config preference1 for setting how many content viewers are allowed to be cached2. As DocShell seems to go hand in hand with session history, it’s not surprising that Brian’s patch touches DocShell code.

about:neterror arrives, Inner and Outer windows appear, and then Session History gets all snuggly with DocShell

On July 14th of 2005, bsmedberg landed a patch to add about:neterror pages, and close a privilege-escalation security vulnerability. Up until this point, network error pages were shown by browsing the DocShell to chrome URLs3, but this allowed certain types of attacks which load iframes resolving to network error pages to potentially gain chrome privileges4.

So instead of going to a chrome URL, the patch causes DocShell to internally load about:neterror5 The great news about this about:neterror page is that it has restricted permissions, so that security hole got plugged.

On July 30th, 2005, jst landed a patch to introduce the notion of inner and outer windows for bug 296639. Inner and outer windows has confused me for a while, but I think I’ve somewhat wrapped my head around it. This document helped.

The idea goes something like this:

The thing that is showing you web content can be considered the outer window – so that could be a browser tab, or an iframe, for example. The inner window is the content being displayed – it’s transitory, and goes away as you browse the web via the outer window.

The outer window then has a notion of all of the inner windows it might contain, and the inner window (via Javascript) gets a handle on the outer window via the window global.

So, for example, if you call window.open, the returned value is an outer window. Methods that you call on that outer window are then forwarded to the inner window.

I hope I got that right. I was originally trying to piece together the meaning of all of it by reading this WHATWG spec describing browsing context, and that was pretty slow going. The MDN page seemed much more clear.

Please comment with corrections if I got any of that wrong.

I’m not entirely sure, but based entirely on instinct and experience, I’m inclined to believe there are interesting security effects of this split. It seems to add a bit more of a membrane6 between web content and the physical window.

Anyhow, jst’s change was pretty monumental. It’s for bug 296639 if you want to read up more about it.

A semi-related change was landed on August 12th by mrbkab, where the entire inner window is stashed in the bfcache (as opposed to what we were doing before, which looks like serialization and deserialization of window state). That was for bug 303267, and sounds related to the fast back and forward caching work that Brian Ryner was working on back in May.

On August 18th of 2005, radha landed the first in the series of patches to session history. Unfortunately, the commit message for this patch doesn’t have a bug number, so I had some trouble tracking down what this work is for. I think this work is for bug 230363, and is actually a copy of interfaces from xpfe/components/shistory/public to docshell/shistory/public. Like I mentioned earlier, DocShell and session history are closely linked, so I suppose it makes sense to put the session history code under docshell/. Later that day, another patch copies the nsISHistoryListener interfaces over as well. Finally, a patch landed to build those interfaces from their new locations, and removes xpfe/components/shistory from Makefile.in. The bug for that last change is bug 305090.

Last bit of 2005

On August 22 of 2005, mrbkap landed a patch that changed how content viewer caching worked. There’s a special page in Firefox called about:blank – if you go to that page right now, you’re going to get a blank page. Some people like to set that as their home page or new tab page, as it is (or should be) very lightweight to load. That page is also special because, from what I can tell, when a new tab or window opens, it’s initially pointing at about:blank before it goes to the requested destination. Before this patch, we used to cache that about:blank content viewer in session history. We didn’t put an entry in the back-forward cache for about:blank though7, so that was a useless cache and a waste of memory. mrbkap’s patch made DocShell check to see if the page it was traveling to was going to re-use the current inner window, and if so, it’d skip caching it. Memory win!

That was the last thing I found interesting in 2005. On to 2006!

Preferences and threads…

On February 7th, 2006, bz landed a patch that made it possible for embedders to override where popup windows get opened.

There are preferences in Firefox that allow you to tweak how web content is able to open new windows8. Those preferences are browser.link.open_newwindow and browser.link.open_newwindow.restriction. If a page is attempting to open a new window, these preferences allow a user to control what actually occurs. The default (in most cases) is to open a new tab instead – but these preferences allow you to open that new window, or to open the content in the same window that the link is executed in. These are the kind of tweaks that power-users love.

Up until this point, only Firefox had these tweaking capabilities. bz’s patch moved that tweaking logic “up the chain”, so to speak, which means that applications that embedded Gecko could also be tweaked like this. Pretty handy.

For the Gecko hackers reading this, this patch also introduced the nsIWindowProvider interface9.

On May 10th, 2006, “darin” landed a patch for bug 326273 that put the nsIThreadManager interface and implementation into the tree. It’s a big commit, and affected many parts of the codebase. nsIThreadManager is, not surprisingly, used to implement multi-threading and thread manipulation in Gecko. From my look at the patch, it looks like it replaces something called nsIEventQueue / nsIEventQueueService. It looks like Gecko already had some facility for multi-threading10, but nsIThreadManager looks like a different model for multi-threading.

For DocShell, this change meant modifying the way that restoring PresShell’s from history would work. Before, DocShell had a RestorePresentationEvent that extended PLEvent, which allowed it to be posted to an nsIEventQueue. Now, instead, we define an inner-class that implements nsRunnable11, and also define a weak pointer to that runnable on a DocShell.

So the way things would go is this: DocShell::RestorePresentation would get called, and this would cancel any pending RestorePresentationRunnable that the DocShell is weak-pointing to. Next, we’d instantiate a new RestorePresentationRunnable that we’d then dispatch to the main thread. This isn’t really different to what we were doing before, but it makes use of the nsIThreadManager and nsRunnable class instead of nsIEventQueue and nsIEventQueueService.

What’s interesting about this patch, DocShell-wise, is that it shows the usage of FavorPerformanceHint, which looks like a way of trading-off UI interactivity with page-to-screen time. Basically, it looks like the FavorPerformanceHint is used when restoring PresShell’s to tell the nsIAppShell, “hey – we want you to favor native events over other events for a small pocket of time so we can get this stuff to the screen ASAP”. If I’m interpreting that right, it’s a tradeoff between total time to execute and responsiveness here. “Do you want it fast, or do you want it smooth?”.

I was probably wrong about the name

In one of my past posts, I made some guesses about why DocShell was called DocShell. I thought:

I think nsDocShell was given the “shell” monicker because it did the job of taking over most of nsWebShell’s duties. However, since nsWebBrowser was now the touch-point between the embedder and embedee… maybe shell makes less sense. I wonder if we missed an opportunity to name nsDocShell something better.

But now that I look at nsIAppShell, and nsIDocShell, and nsIPresShell… I think I’m starting to understand. A while back, when I first started planning these posts, I asked blassey why he thought nsIDocShell was named the way it was, and he said he thought it might be related to the notion of a command shell – like a terminal input. From my understanding, a shell is a command interface with which one can manipulate and control something pretty complex – like the file-system or processes of a computer. I think blassey is right – I think that’s the “Shell” in nsIDocShell. I think the idea is that this interface would be the one to control and manipulate the process of loading and displaying a document. It seems obvious now, but it sure wasn’t when I started looking into this stuff.

DOM Storage (session and global), KungFuDeathGrip, friendlier search…

On May 19th, 2006, jst picked up, finished and landed a patch originally be Enn that implemented DOM Storage for bug 335540.

This patch adds two new methods to nsIDocShell – getSessionStorageForDomain and addSessionStorage. The first method is accessed in a number of cases, but most importantly when some caller reads sessionStorage or globalStorage off of the window object12.

The relationship between nsGlobalWindow and nsDocShell is brought to my attention with this patch. Here’s a fragment from an old chat I had with Ms2ger, smaug and bz, which started when I asked Ms2ger what he’d rename DocShell to.

14:11 (Ms2ger) mconley, I would call it WindowProxy 🙂
14:12 (smaug) outer window? yes, WindowProxy please
14:12 (mconley) Ms2ger: wait, outer window = docshell currently?
14:12 (khuey) what are we doing with WindowProxy?
14:13 (Ms2ger) mconley, well, no, there’s nsDocShell and nsGlobalWindow (with IsOuterWindow() true)
14:13 (Ms2ger) mconley, those are pretty much isomorphic
14:13 (mconley) I see
14:13 (bz) nsDocShell and outer nsGlobalWindow are in a 1-1 relationship
14:14 (bz) The fact that they are two separate objects is sort of a historical accident that we may want to rectify sometime
14:14 (mconley) this sounds like another post to write – how nsDocShell and nsGlobalWindow are related…

So I think nsGlobalWindow (instances of which can either be “inner” or “outer”) when it has IsOuterWindow being true, works in tandem with nsDocShell to “be” the outer window. That’s really imprecise, hand-wavey language. I’ll probably need to tighten this up in a follow-up post once somebody reads this and gives me better words to describe things13.

On May 24, 2006, smaug landed a patch to fix bug 336978. Bug 336978 was a crash caused by loading the following code in an iframe:

<html>
<head></head>
<body>
  <script>
    window.addEventListener("pagehide", doe, true);
    function doe(e) {
      var x = parent.document.getElementsByTagName('iframe')[0];
      x.parentNode.removeChild(x);
    }
    setTimeout(doe2,500);
    function doe2() {
      window.location = 'about:blank';
    }
  </script>
</body>
</html>

What this code does is wait 500ms, and then change the location to about:blank. Changing the location causes the pagehide event to fire while we’re unloading the original page, and when we hear it, we get the host of the iframe to remove the iframe from itself.

smaug’s solution to this bug is for nsDocShell to hold a reference via an nsCOMPtr to the nsIContentViewer for the document while the pagehide event is fired. This ensures that the nsIContentViewer doesn’t get destructed before we’re truly done to it. The name we give this nsCOMPtr is “kungFuDeathGrip”. This isn’t the only place where some hold of an object is maintained with a variable called kungFuDeathGrip – check out dxr for some more uses.

I’d seen kungFuDeathGrip over the years, and I never looked closely at what it was doing. I always thought kungFuDeathGrip was some magical global function that destroyed things unequivocally, but on closer inspection, I’m pretty sure it’s really just a way of saying “this variable’s sole purpose is to hold a reference to this thing until I’m done with it.”

I think the phrase “kung fu” distracted me. I thought it did this:

Black Dynamite layin' the smack down.

Woooooo!

But it’s really more like this:

Spock taking out Kirk with the Vulcan nerve pinch thing.

Kkkg….*gurgle*…ngahh….

On June 15th, 2006, “brettw” landed a patch for bug 245597 to make it so that anything that gets put into the AwesomeBar that isn’t parse-able as a URI automatically turns into a keyword search. That’s great! This made both the search input and the AwesomeBar useful for more users. This change occurred in docshell/base/nsDefaultURIFixup.cpp, which is, as I understand it, the central location for code that turns erroneous URIs into what the user probably intended.

nsIMutationObserver, some new about: pages…

On July 2nd, 2006, Jonas Sicking added nsIMutationObserver to the tree for bug 342062, making it possible to observe changes to the DOM within a subtree. It’s a pretty big patch, but it looks like a good chunk of it is just swapping in usage of nsIMutationObserver to replace old usage of nsIDocumentObserver (which supplied the same observations, but for an entire document instead of a subtree). Note that it’d still be a few years before DOM3 Mutation Events would be exposed for web developers to use, and after a few more years, those events were deprecated in favour of the Mutation Observer API.

On September 15th, 2006, several new Gecko-wide about: pages landed, which means they got put into the redirection map in docshell/base/nsAboutRedirector.cpp. These pages were about:buildconfig (bug 140034), about:about (bug 56061), and about:license (bug 256945). That same day, bz landed a patch to make it so that new about: pages didn’t have to have special rules hardcoded into nsScriptSecurityManager::CanExecuteScripts to execute script even if the user has script disabled. Instead, the nsAboutRedirector mapping was extended to allow a boolean for indicating that an about: page required script execution.

Simplifying DocShell, and then some spoofing and malware protection

The next interesting thing (according to me, anyhow) didn’t occur until the following May 6th, 2007. That day, bz landed a patch for bug 377303 which simplified the structure of things inside the DocShell tree.

Up until that point, there had existed both nsIDocShellTreeItem and nsIDocShellTreeNode had both existed as interfaces for interacting with nodes within a DocShell tree. I’ll quote myself from my previous post:

The (somewhat nebulous) distinction of DocShell “treeItems” and “treeNodes” is made. At this point, the difference between the two is that nsIDocShellTreeItem must be implemented by anything that wishes to be a leaf or middle node of the DocShell tree. The interface itself provides accessors to various attributes on the tree item. nsIDocShellTreeNode, on the other hand, is for manipulating one of these items in the tree – for example, finding, adding or removing children. I’m not entirely sure this distinction is useful, but there you have it.

It looks like enough was finally enough. bz didn’t go so far as to fully merge the two interfaces (though he makes a note in his patch about doing so), but instead made the (arguably more complex) nsIDocShellTreeItem interface inherit from nsIDocShellTreeNode14.

Later, on May 17th, 2007, Mats Palmgren landed a patch for bug 376562 to remove a childOffset attribute from nsIDocShellTreeItem, and instead move a setter for the childOffset to the nsIDocShell interface instead.

Reading this bug comment as well as one of Mats’ comments in the patch, it sounds as if childOffset never really worked as advertised, and was a bit of a foot-gun.15

On June 14th, 2007, bz landed a patch for bug 371360, which prevents onUnload handlers from starting any page loads. Before this, it seems that it was possible for a page do to something like this:

<html>
  <body onunload="location.href = 'http://www.somesite.com';">
    <a href="http://slashdot.org/">http://slashdot.org/</a>
  </body>
</html>

With the result being that you could (potentially) phish a user. For example, suppose you’re a member of MySafeBank, which has a site at mysafebank.com. Suppose you’re at my seemingly innocent site totallyevil.com, and also suppose that I’ve registered a domain at rnysafebank.com (that’s an r and an n, which, if you’re not paying attention, look pretty close to a m). If you’re at my site, and I notice that you’re trying to head to mysafebank.com, I could redirect you to rnysafebank.com, which has a very similar user interface and favicon. Yadda yadda yadda, your bank info is now mine.

So bz stopped that one in its tracks by just preventing a DocShell from attempting any kind of load if we’re in the middle of an unload.

On July 3rd, 2007, johnath16 landed a patch for bug 380932 to add a new mode for about:neterror for pages suspected of serving up malware.

If you haven’t seen that page before, count yourself lucky – you’ve been surfing in safe places! This is what the page looks like (or, used to look like, anyhow):

Minefield reporting a Suspected Attack Site

Old version of Firefox showing a Suspected Attack Site

johnath’s patch allowed an about:neterror page to have a specific CSS class associated with it as part of its URL. This allowed for the dramatic styling in the image above17.

showModalDialog

showModalDialog was a non-standard function that Microsoft introduced in Internet Explorer 418. This allows a web page to create a modal dialog that contains web content. jst landed a patch on July 26th, 2007 to implement it in Firefox as part of bug 194404. This patch made it possible for a DocShell to have a modal dialog be its parent.

showModalDialog has since been marked as deprecated on MDN, and Google Chrome have announced they will no longer support it after May of 2015. Firefox will support it until sometime after Firefox 39 on the release channel19.

about:crashes, Larry, and tab tearing

There’s a long gap in time here where nothing really interesting happens under docshell/.

Finally, on January 24th, 2008, Mossop landed a patch for bug 411490 that exposes about:crashes as a handy way of getting at the list of crash reports that have been collected.

As about:crashes is a Gecko-wide about: page, this meant once again adding an entry to the docshell/base/nsAboutRedirector.cpp map, as had been done with about:buildconfig and about:about.

On April 28th, 2008, gavin landed a patch originally by ehsan that adds a friendlier set of icons for reporting SSL errors for bug 430904.

That icon is Larry. Have you met Larry? This is Larry:

Larry, the SSL dude.

This is Larry.

Larry is the name for a series of icons that were developed to describe how secure your communications are with a particular site. You might recognize him from the airport, because he looks a lot like a customs agent or border patrol.

You can read up on Larry here on johnath’s blog.

On August 7th, 2008, bz landed a patch for bug 113934 to lay the foundation for letting users drag tabs out from a window, or drag tabs between windows. This introduced a new method to nsIFrameLoader, “swapFrameLoaders”. This method is the real key to moving tabs between windows – each <xul:browser> implements nsIFrameLoaderOwner, and the nsIFrameLoader is (yet another) thing that can load web content. This is essentially a brain transplant between two <xul:browser>’s. You can see the real guts of the brain transplant in this method of the patch.

On January 13th, in a semi-related patch, bz landed code for bug 449780 to flush the bfcache20 when swapping frameloaders. Apparently, we were storing information in the bfcache that was simply incorrect after a frameloader swap. The best way to avoid internal confusion in such a case was to just invalidate the cache.

Big gaps…

Lots of big gaps between the next few changes.

On March 18th, 2009, Honza Bambas landed a patch for bug 422526 to implement window.localStorage. localStorage was a replacement for globalStorage21 that persisted across browser restarts (unlike sessionStorage).

Note that both localStorage and sessionStorage were synchronous storage APIs. It’d take until around June 24th of 2010 before an asynchronous storage mechanism became available.

On May 7th, 2009, bz landed a patch for bug 490957 to finally get rid of nsWebShell.cpp. If you recall from my earlier blog post, that was one of Travis Bogard’s goals at the start of this whole adventure. bz’s patch essentially folds the functionality of nsWebShell into nsDocShell. The webshell/ folder remained, but just contained interfaces.

Curiously, a good chunk of nsWebShell’s functionality seemed to revolve around anchor pings, a massively unpopular “feature” that allows a website to get your browser to send a request every time you click on a link. Thankfully, this “feature” is disabled by default in Firefox22. Here’s Jorge Villalobos’s post on anchor pings. Correction (Jan 4th, 2015) – I’ve since changed my tune about anchor pings. See these three comments.

On June 29th, 2009, dbolter landed a patch for bug 467144 so that nsIMutationObserver’s, when they observe an attribute being changed, also get a copy of the old attribute as well as the new one. Actually, to be specific, it adds a callback “AttributeWillChange” to include the old attribute. This fires before the “AttributeChanged” callback.

A day later, bsmedberg landed a massive patch to implement remote tabs. There’s no bug number in the commit message, but this is clearly part of the Electrolysis efforts that were just starting up around this time. Remote tabs means browsers that run in different processes, which is the overall goal of Electrolysis, and (possibly unbeknownst to bsmedberg at the time) a foundational piece for Fennec (Firefox for Android)23.

On October 3rd, 2009, vvuk landed about:memory for bug 515354, a key piece of the war against high-memory consumption in Firefox (a.k.a. MemShrink). This is very similar to the about:crashes page that Mossop landed back in 2008.

And finally…

On January 7th, 2010, smaug landed a patch for bug 534226 to remove support for multiple PresShell’s. PresShell stands for “Presentation Shell”, and as I understand it, is the primary interface to the “frame tree”24.

It looks like, up until this point, Gecko had the ability to have multiple frame trees per content tree. I’m not entirely sure what the point of that was, but the capability was there. smaug’s patch simplifies everything by making sure a document has only a single, primary PresShell. This removes a lot of iteration and management code for those multiple PresShell’s, which is nice.

And last, but not least, on June 30th, 2010, Benjamin Stover landed a patch for bug 556400 which made it so that visits to webpages are recorded asynchronously in Places. It looks like this patch takes I/O off the main-thread, so it gets a big thumbs-up from me.

Essentially, this patch adds new asynchronous write methods to the History service, and then makes nsDocShell use those methods on webpage visits. nsDocShell falls back to the synchronous methods of nsIGlobalHistory2 if, for some reason, it can’t get at the History service and its asynchronous methods.

Phew!

Did you make it? Are you still with me? I know it might feel like this:

qwop

everyone is a winner

but I think we’re making real progress here. I think we’re learning important stuff about the history of Firefox, and changes that have occurred in some of its core functionality over time.

So to sum, that, to me, was the most interesting stuff to happen in and around docshell/ from 2005-2010. There might have been other neat stuff in there, but it didn’t catch my eye when I was browsing commit messages.

There’s still much to do – I have to look at commits from 2011 to 2014. After that, I’m planning on doing a line-by-line code review / walkthrough of nsDocShell.cpp, and then I’d like to try to summarize my findings and any recommendations I’ve put together from my time studying this stuff.

Hold tight!


  1. This pref was browser.sessionhistory.max_viewers, if you’re interested – though that preference appears to have been superseded by browser.sessionhistory.max_total_viewers. The default value for that pref is -1, meaning to adjust the number of allowed cached viewers based on how much memory is available. If you’re looking to reduce how much memory Firefox consumes, it’s possible setting this to some low integer will allow you reverse that trade-off between space and speed. 

  2. I assume per session history 

  3. You wouldn’t notice that you were at a chrome URL though, because DocShell loads this URL internally, while pretending to be at the URL that caused the error. The end result is the user going to http://www.sitethatcausesnetworkerror.com still sees that URL in their AwesomeBar, despite the fact that their web content shows the appropriate network error page hosted at a chrome URL. 

  4. “chrome privileges” means that a web page now essentially has the same permissions that Firefox, the program on your computer, has – meaning it can potentially read and write files, and communicate with anybody on your network. Yikes! 

  5. You can visit this page in Firefox right now and see a generic network error. It’s showing a generic error because it hasn’t the foggiest idea how you’ve arrived at about:neterror, since it wasn’t passed any error information. 

  6. Or the infrastructure to create such a membrane. 

  7. So you couldn’t go back to about:blank in cases I described, where a tab or window was initialized at about:blank before going to a new page. 

  8. I’m actually quite familiar with this stuff because I worked on opening new windows for Electrolysis not too long ago. 

  9. From the header of that interface:

    /**
     * The nsIWindowProvider interface exists so that the window watcher's default
     * behavior of opening a new window can be easly modified.  When the window
     * watcher needs to open a new window, it will first check with the
     * nsIWindowProvider it gets from the parent window.  If there is no provider
     * or the provider does not provide a window, the window watcher will proceed
     * to actually open a new window.
     */

     

  10. The nsIEventQueueService service mentions that it is used to manage event queues for a particular thread, and makes use of nsIThread – so multi-threading must have already been a thing. 

  11. Still called RestorePresentationEvent though… strange that the opportunity wasn’t taken to rename this to RestorePresentationRunnable. 

  12. Those two properties are part of a new nsIDOMStorageWindow interface that nsGlobalWindow implements after this patch. That interface is later removed in bug 670331, and the two accessors are moved directly into nsIDOMWindow instead. 

  13. I have a feeling the real answer lies somewhere in the comments in bug 296639

  14. It wasn’t immediately clear to me why the inheritance didn’t go the other way around – especially since bz himself had a comment in nsIDocShellTreeNode suggesting that arrangement. Look at his first comment in the bug though:

    This would allow consumers to start using just nsIDocShellTreeItem in their code, until we can just merge nsIDocShellTreeNode into nsIDocShellTreeItem.

    Basically, it sounds like nsIDocShellTreeNode was being deprecated, and that callers who used to use nsIDocShellTreeNode should migrate to use nsIDocShellTreeItem instead (which inherits nsIDocShellTreeNode’s methods). Then the two interfaces could be merged. 

  15. Mats’ warning was removed on August 17, 2010 as part of bug 462076. It looks like SetChildOffset is still only ever used when adding the child though, so it’s still probably valid. 

  16. The same johnath who is currently the VP of Firefox! 

  17. Later on in August, dcamp would land this patch as part of bug 384941 which prevents suspected malware sites from even loading, instead of just not displaying them. 

  18. To quote Douglas Adams:

    This has made a lot of people very angry and has been widely regarded as a bad move.

     

  19. And the 38 ESR will continue to be supported until mid-2016. If you maintain a site that uses showModalDialog, you’d best get rid of it. 

  20. The bfcache, or “back-forward cache” is a collection of “frozen” pages that are stored in memory for fast back/forward action – see this page for more detail. 

  21. globalStorage, I believed, allowed all web properties read and write access to the same storage – so clearly it was a good idea to replace it. globalStorage was removed on October 9th, 2011 by Honza as part of bug 687579

  22. But according to this, is enabled by default in both Chrome and Opera. Lovely

  23. Remote tabs are also hugely important for Boot2Gecko / Firefox OS

  24. From this document:

    …the frame tree…is the visual representation of the document. Each frame can be thought of as a rectangular area on the page. The content nodes for XML elements are usually associated with one or more frames which display the element — one frame if the element is rectangular, more if the element is something more complex (like a chunk of bolded text that happens to be word-wrapped)…

    And from the nsIPresShell.h header:

    /**
    * Presentation shell interface. Presentation shells are the
    * controlling point for managing the presentation of a document. The
    * presentation shell holds a live reference to the document, the
    * presentation context, the style manager, the style set and the root
    * frame. <p>
    *
    * When this object is Release’d, it will release the document, the
    * presentation context, the style manager, the style set and the root
    * frame.

     

DocShell in a Nutshell – Part 2: The Wonder Years (1999 – 2004)

When I first announced that I was going to be looking at the roots of DocShell, and how it has changed over the years, I thought I was going to be leafing through a lot of old CVS commits in order to see what went on before the switch to Mercurial.

I thought it’d be so, and indeed it was so. And it was painful. Having worked with DVCS’s like Mercurial and Git so much over the past couple of years, my brain was just not prepared to deal with CVS.

My solution? Take the old CVS tree, and attempt to make a Git repo out of it. Git I can handle.

And so I spent a number of hours trying to use cvs2git to convert my rsync’d mirror of the Mozilla CVS repository into something that I could browse with GitX.

“But why isn’t the CVS history in the Mercurial tree?” I hear you ask. And that’s a good question. It might have to do with the fact that converting the CVS history over is bloody hard – or at least that was my experience. cvs2git has the unfortunate habit of analyzing the entire repository / history and spitting out any errors or corruptions it found at the end.1 This is fine for small repositories, but the Mozilla CVS repo (even back in 1999) was quite substantial, and had quite a bit of history.

So my experience became: run cvs2git, wait 25 minutes, glare at an error message about corruption, scour the web for solutions to the issue, make random stabs at a solution, and repeat.

Not the greatest situation. I did what most people in my position would do, and cast my frustration into the cold, unfeeling void that is Twitter.

But, lo and behold, somebody on the other side of the planet was listening. Unfocused informed me that whoever created the gecko-dev Github mirror somehow managed to type in the black-magic incantations that would import all of the old CVS history into the Git mirror. I simply had to clone gecko-dev, and I was golden.

Thanks Unfocused. 🙂

So I had my tree. I cracked open Gitx, put some tea on, and started pouring through the commits from the initial creation of the docshell folder (October 15, 1999) to the last change in that folder just before the switch over to 2005 (December 15, 2004)2.

The following are my notes as I peered through those commits.

Artist’s rendering of me reading some old commit messages. I’m not claiming to have magic powers.

“First landing”

That’s the message for the first commit when the docshell/ folder was first created by Travis Bogard. Correction (Jan 3rd, 2015) – this is actually not true at all. The commit I’ve linked to here is the first commit for nsDocShell.cpp, but the docshell/ folder was actually created over a year earlier. It looks like the “docshell” monicker was selected before any of Travis’ refactoring had even begun3.

Without even looking at the code, that’s a pretty strange commit just by the message alone. No bug number, no reviewer, no approval, nothing even approximating a vague description of what was going on.

Leafing through these early commits, I was surprised to find that quite common. In fact, I learned that it was about a year after this work started that code review suddenly became mandatory for commits.

So, for these first bits of code, nobody seems to be reviewing it – or at least, nobody is signing off on it in commit messages.

Like I mentioned, the date for this commit is October 15, 1999. If the timeline in this Wikipedia article about the history of the Mozilla Suite is reliable, that puts us somewhere between Milestone 10 and Milestone 11 of the first 1.0 Mozilla Suite release.4

Travis Bogard

Before we continue, who is this intrepid Travis Bogard who is spearheading this embedding initiative and the DocShell / WebShell rewrite?

At the time, according to his LinkedIn page, he worked for America Online (which at this point in time owned Netscape.5) He’d been working for AOL since 1996, working his way up the ranks from lowly intern all the way to Principal Software Engineer.

Travis was the one who originally wrote the wiki page about how painful it was embedding the web engine, and how unwieldy nsWebShell was.6 It was Travis who led the charge to strip away all of the complexity and mess inside of WebShell, and create smaller, more specialized interfaces for the behind the scenes DocShell class, which would carry out most of the work that WebShell had been doing up until that point.7

So, for these first few months, it was Travis who would be doing most of the work on DocShell.

Parallel development

These first few months, Travis puts the pedal to the metal moving things out of WebShell and into DocShell. Remember – the idea was to have a thin, simple nsWebBrowser that embedders could touch, and a fat, complex DocShell that was privately used within Gecko that was accessible via many small, specialized interfaces.

Wholesale replacing or refactoring a major part of the engine is no easy task, however – and since WebShell was core to the very function of the browser (and the mail/news client, and a bunch of other things), there were two copies of WebShell made.

The original WebShell existed in webshell/ under the root directory. The second WebShell, the one that would eventually replace it, existed under docshell/base. The one under docshell/base is the one that Travis was stripping down, but nobody was using it until it was stable. They’d continue using the one under webshell/, until they were satisfied with their implementation by both manual and automated testing.

When they were satisfied, they’d branch off of the main development line, and start removing occurances of WebShell where they didn’t need to be, and replace them with nsWebBrowser or DocShell where appropriate. When they were done that, they’d merge into main line, and celebrate!

At least, that was the plan.

That plan is spelled out here in the Plan of Attack for the redesign. That plan sketches out a rough schedule as well, and targets November 30th, 1999 as the completion point of the project.

This parallel development means that any bugs that get discovered in WebShell during the redesign needs to get fixed in two places – both under webshell/ and docshell/base.

Breaking up is so hard to do

So what was actually changing in the code? In Travis’ first commit, he adds the following interfaces:

along with some build files. Something interesting here is this nsIHTMLDocShell – where it looked like at this point, the plan was to have different DocShell interfaces depending on the type of document being displayed. Later on, we see this idea dissipate.

If DocShell was a person, these are its baby pictures. At this point, nsIDocShell has just two methods: LoadDocument, LoadDocumentVia, and a single nsIDOMDocument attribute for the document.

And here’s the interface for WebShell, which Travis was basing these new interfaces off of. Note that there’s no LoadDocument, or LoadDocumentVia, or an attribute for an nsIDOMDocument. So it seems this wasn’t just a straight-forward breakup into smaller interfaces – this was a rewrite, with new interfaces to replace the functionality of the old one.8

This is consistent with the remarks in this wikipage where it was mentioned that the new DocShell interfaces should have an API for the caller to supply a document, instead of a URI – thus taking the responsibility of calling into the networking library away from DocShell and putting it on the caller.

nsIDocShellEdit seems to be a replacement for some of the functions of the old nsIClipboardCommands methods that WebShell relied upon. Specifically, this interface was concerned with cutting, copying and pasting selections within the document. There is also a method for searching. These methods are all just stubbed out, and don’t do much at this point.

nsIDocShellFile seems to be the interface used for printing and saving documents.

nsIGenericWindow (which I believe is the ancestor of nsIBaseWindow), seems to be an interface that some embedding window must implement in order for the new nsWebBrowser / DocShell to be embedded in it. I think. I’m not too clear on this. At the very least, I think it’s supposed to be a generic interface for windows supplied by the underlying operating system.

nsIGlobalHistory is an interface for, well, browsing history. This was before tabs, so we had just a single, linear global history to maintain, and I guess that’s what this interface was for. Correction (Jan 3rd, 2015) – global history is really just a recording of everywhere you’ve ever been, ever. It looks like I was getting confused between global history (all the places I’ve been) and session history (all the places I’ve been this session).

nsIScrollable is an interface for manipulating the scroll position of a document.

So these magnificent seven new interfaces were the first steps in breaking up WebShell… what was next?

Enter the Container

nsIDocShellContainer was created so that the DocShells could be formed into a tree and enumerated, and so that child DocShells could be named. It was introduced in this commit.

about:face

In this commit, only five days after the first landing, Travis appears to reverse the decision to pass the responsibility of loading the document onto the caller of DocShell. LoadDocument and LoadDocumentVia are replaced by LoadURI and LoadURIVia. Steve Clark (aka “buster”) is also added to the authors list of the nsIDocShell interface. It’s not clear to me why this decision was reversed, but if I had to guess, I’d say it proved to be too much of a burden on the callers to load all of the documents themselves. Perhaps they punted on that goal, and decided to tackle it again later (though I will point out that today’s nsIDocShell still has LoadURI defined in it).

First implementor

The first implementation of nsIDocShell showed up on October 25, 1999. It was nsHTMLDocShell, and with the exception of nsIGlobalHistory, it implemented all of the other interfaces that I listed in Travis’ first landing.

The base implementation

On October 25th, the stubs of a DocShell base implementation showed up in the repository. The idea, I would imagine, is that for each of the document types that Gecko can display, we’d have a DocShell implementation, and each of these DocShell implementations would inherit from this DocShell base class, and only override the things that they need specialized for their particular document type.

Later on, when the idea of having specialized DocShell implementations evaporates, this base class will end up being nsDocShell.cpp.

That same day, most of the methods were removed from the nsHTMLDocShell implementation, and nsHTMLDocShell was made to inherit from nsDocShellBase.

“Does not compile yet”

The message for this commit on October 27th, 1999 is pretty interesting. It reads:

added a bunch of initial implementation. does not compile yet, but that’s ok because docshell isn’t part of the build yet.

So not only are none of these patches being reviewed (as far as I can tell), and are not mapped to any bugs in the bug tracker, but the patches themselves just straight-up do not build. They are not building on tinderbox.

This is in pretty stark contrast to today’s code conventions. While it’s true that we might land code that is not entered for most Nightly users, we usually hide such code behind an about:config pref so that developers can flip it on to test it. And I think it’s pretty rare (if it ever occurs) for us to land code in mozilla-central that’s not immediately put into the build system.

Perhaps the WebShell tests that were part of the Plan of Attack were being written in parallel and just haven’t landed, but I suspect that they haven’t been written at all at this point. I suspect that the team was trying to stand something up and make it partially work, and then write tests for WebShell and try to make them pass for both old WebShell and DocShell. Or maybe just the latter.

These days, I think that’s probably how we’d go about such a major re-architecture / redesign / refactor; we’d write tests for the old component, land code that builds but is only entered via an about:config pref, and then work on porting the tests over to the new component. Once the tests pass for both, flip the pref for our Nightly users and let people test the new stuff. Once it feels stable, take it up the trains. And once it ships and it doesn’t look like anything is critically wrong with the new component, begin the process of removing the old component / tests and getting rid of the about:config pref.

Note that I’m not at all bashing Travis or the other developers who were working on this stuff back then – I’m simply remarking on how far we’ve come in terms of development practices.

Remember AOL keywords?

Tangentially, I’m seeing some patches go by that have to do with hooking up some kind of “Keyword” support to WebShell.

Remember those keywords? This was the pre-Google era where there were only a few simplistic search engines around, and people were still trying to solve discoverability of things on the web. Keywords was, I believe, AOL’s attempt at a solution.

You can read up on AOL Keywords here. I just thought it was interesting to find some Keywords support being written in here.

One DocShell to rule them all

Now that we have decided that there is only one docshell for all content types, we needed to get rid of the base class/ content type implementation. This checkin takes and moves the nsDocShellBase to be nsDocShell. It now holds the nsIHTMLDocShell stuff. This will be going away. nsCDocShell was created to replace the previous nsCHTMLDocShell.

This commit lands on November 12th (almost a month from the first landing), and is the point where the DocShell-implementation-per-document-type plan breaks down. nsDocShellBase gets renamed to nsDocShell, and the nsIHTMLDocShell interface gets moved into nsIDocShell.idl, where a comment above it indicates that the interface will soon go away.

We have nsCDocShell.idl, but this interface will eventually disappear as well.

[PORKJOCKEY]

So, this commit message on November 13th caught my eye:

pork jockey paint fixes. bug=18140, r=kmcclusk,pavlov

What the hell is a “pork jockey”? A quick search around, and I see yet another reference to it in Bugzilla on bug 14928. It seems to be some kind of project… or code name…

I eventually found this ancient wiki page that documents some of the language used in bugs on Bugzilla, and it has an entry for “pork jockey”: a pork jockey bug is a “bug for work needed on infrastructure/architecture”.

I mentioned this in #developers, and dmose (who was hacking on Mozilla code at the time), explained:

16:52 (dmose) mconley: so, porkjockeys
16:52 (mconley) let’s hear it
16:52 (dmose) mconley: at some point long ago, there was some infrastrcture work that needed to happen
16:52 mconley flips on tape recorder
16:52 (dmose) and when people we’re talking about it, it seemed very hard to carry off
16:52 (dmose) somebody said that that would happen on the same day he saw pigs fly
16:53 (mconley) ah hah
16:53 (dmose) so ultimately the group of people in charge of trying to make that happen were…
16:53 (dmose) the porkjockeys
16:53 (dmose) which was the name of the mailing list too

Here’s the e-mail that Brendan Eich sent out to get the Porkjockey’s flying.

Development play-by-play

On November 17th, the nsIGenericWindow interface was removed because it was being implemented in widget/base/nsIBaseWindow.idl.

On November 27th, nsWebShell started to implement nsIBaseWindow, which helped pull a bunch of methods out of the WebShell implementations.

On November 29th, nsWebShell now implements nsIDocShell – so this seems to be the first point where the DocShell work gets brought into code paths that might be hit. This is particularly interesting, because he makes this change to both the WebShell implementation that he has under the docshell/ folder, as well as the WebShell implementation under the webshell/ folder. This means that some of the DocShell code is now actually being used.

On November 30th, Travis lands a patch to remove some old commented out code. The commit message mentions that the nsIDocShellEdit and nsIDocShellFile interfaces introduced in the first landing are now defunct. It doesn’t look like anything is diving in to replace these interfaces straight away, so it looks like he’s just not worrying about it just yet. The defunct interfaces are removed in this commit one day later.

On December 1st, WebShell (both the fork and the “live” version) is made to implement nsIDocShellContainer.

One day later, nsIDocShellTreeNode interface is added to replace nsIDocShellContainer. The interface is almost identical to nsIDocShellContainer, except that it allows the caller to access child DocShells at particular indices as opposed to just returning an enumerator.

December 3rd was a very big day! Highlights include:

Noticing something? A lot of these changes are getting dumped straight into the live version of WebShell (under the webshell/ directory). That’s not really what the Plan of Attack had spelled out, but that’s what appears to be happening. Perhaps all of this stuff was trivial enough that it didn’t warrant waiting for the WebShell fork to switch over.

On December 12th, nsIDocShellTreeOwner is introduced.

On December 15th, buster re-lands the nsIDocShellEdit and nsIDocShellFile interfaces that were removed on November 30th, but they’re called nsIContentViewerEdit and nsIContentViewerFile, respectively. Otherwise, they’re identical.

On December 17th, WebShell becomes a subclass of DocShell. This means that a bunch of things can get removed from WebShell, since they’re being taken care of by the parent DocShell class. This is a pretty significant move in the whole “replacing WebShell” strategy.

Similar work occurs on December 20th, where even more methods inside WebShell start to forward to the base DocShell class.

That’s the last bit of notable work during 1999. These next bits show up in the new year, and provides further proof that we didn’t all blow up during Y2K.

On Feburary 2nd, 2000, a new interface called nsIWebNavigation shows up. This interface is still used to this day to navigate a browser, and to get information about whether it can go “forwards” or “backwards”.

On February 8th, a patch lands that makes nsGlobalWindow deal entirely in DocShells instead of WebShells. nsIScriptGlobalObject also now deals entirely with DocShells. This is a pretty big move, and the patch is sizeable.

On February 11th, more methods are removed from WebShell, since the refactorings and rearchitecture have made them obsolete.

On February 14th, for Valentine’s day, Travis lands a patch to have DocShell implement the nsIWebNavigation interface. Later on, he lands a patch that relinquishes further control from WebShell, and puts the DocShell in control of providing the script environment and providing the nsIScriptGlobalObjectOwner interface. Not much later, he lands a patch that implements the Stop method from the nsIWebNavigation interface for DocShell. It’s not being used yet, but it won’t be long now. Valentine’s day was busy!

On February 24th, more stuff (like the old Stop implementation) gets gutted from WebShell. Some, if not all, of those methods get forwarded to the underlying DocShell, unsurprisingly.

Similar story on February 29th, where a bunch of the scroll methods are gutted from WebShell, and redirected to the underlying DocShell. This one actually has a bug and some reviewers!9 Travis also landed a patch that gets DocShell set up to be able to create its own content viewers for various documents.

March 10th saw Travis gut plenty of methods from WebShell and redirect to DocShell instead. These include Init, SetDocLoaderObserver, GetDocLoaderObserver, SetParent, GetParent, GetChildCount, AddChild, RemoveChild, ChildAt, GetName, SetName, FindChildWithName, SetChromeEventHandler, GetContentViewer, IsBusy, SetDocument, StopBeforeRequestingURL, StopAfterURLAvailable, GetMarginWidth, SetMarginWidth, GetMarginHeight, SetMarginHeight, SetZoom, GetZoom. A few follow-up patches did something similar. That must have been super satisfying.

March 11th, Travis removes the Back, Forward, CanBack and CanForward methods from WebShell. Consumers of those can use the nsIWebNavigation interface on the DocShell instead.

March 30th sees the nsIDocShellLoadInfo interface show up. This interface is for “specifying information used in a nsIDocShell::loadURI call”. I guess this is certainly better than adding a huge amount of arguments to ::loadURI.

During all of this, I’m seeing references to a “new session history” being worked on. I’m not really exploring session history (at this point), so I’m not highlighting those commits, but I do want to point out that a replacement for the old Netscape session history stuff was underway during all of this DocShell business, and the work intersected quite a bit.

On April 16th, Travis lands a commit that takes yet another big chunk out of WebShell in terms of loading documents and navigation. The new session history is now being used instead of the old.

The last 10% is the hardest part

We’re approaching what appears to be the end of the DocShell work. According to his LinkedIn profile, Travis left AOL in May 2000. His last commit to the repository before he left was on April 24th. Big props to Travis for all of the work he put in on this project – by the time he left, WebShell was quite a bit simpler than when he started. I somehow don’t think he reached the end state that he had envisioned when he’d written the original redesign document – the work doesn’t appear to be done. WebShell is still around (in fact, parts of it around were around until only recently!10 ). Still, it was a hell of chunk of work he put in.

And if I’m interpreting the rest of the commits after this correctly, there is a slow but steady drop off in large architectural changes, and a more concerted effort to stabilize DocShell, nsWebBrowser and nsWebShell. I suspect this is because everybody was buckling down trying to ship the first version of the Mozilla Suite (which finally occurred June 5th, 2002 – still more than 2 years down the road).

There are still some notable commits though. I’ll keep listing them off.

On June 22nd, a developer called “rpotts” lands a patch to remove the SetDocument method from DocShell, and to give the implementation / responsibility of setting the document on implementations of nsIContentViewer.

July 5th sees rpotts move the new session history methods from nsIWebNavigation to a new interface called nsIDocShellHistory. It’s starting to feel like the new session history is really heating up.

On July 18th, a developer named Judson Valeski lands a large patch with the commit message “webshell-docshell consolodation changes”. Paraphrasing from the bug, the point of this patch is to move WebShell into the DocShell lib to reduce the memory footprint. This also appears to be a lot of cleanup of the DocShell code. Declarations are moved into header files. The nsDocShellModule is greatly simplified with some macros. It looks like some dead code is removed as well.

On November 9th, a developer named “disttsc” moves the nsIContentViewer interface from the webshell folder to the docshell folder, and converts it from a manually created .h to an .idl. The commit message states that this work is necessary to fix bug 46200, which was filed to remove nsIBrowserInstance (according to that bug, nsIBrowserInstance must die).

That’s probably the last big, notable change to DocShell during the 2000’s.

2001: A DocShell Odyssey

On March 8th, a developer named “Dan M” moves the GetPersistence and SetPersistence methods from nsIWebBrowserChrome to nsIDocShellTreeOwner. He sounds like he didn’t really want to do it, or didn’t want to be pegged with the responsibility of the decision – the commit message states “embedding API review meeting made me do it.” This work was tracked in bug 69918.

On April 16th, Judson Valeski makes it so that the mimetypes that a DocShell can handle are not hardcoded into the implementation. Instead, handlers can be registered via the CategoryManager. This work was tracked in bug 40772.

On April 26th, a developer named Simon Fraser adds an implementation of nsISimpleEnumerator for DocShells. This implementation is called, unsurprisingly, nsDocShellEnumerator. This was for bug 76758. A method for retrieving an enumerator is added one day later in a patch that fixes a number of bugs, all related to the page find feature.

April 27th saw the first of the NSPR logging for DocShell get added to the code by a developer named Chris Waterson. Work for that was tracked in bug 76898.

On May 16th, for bug 79608, Brian Stell landed a getter and setter for the character set for a particular DocShell.

There’s a big gap here, where the majority of the landings are relatively minor bug fixes, cleanup, or only slightly related to DocShell’s purpose, and not worth mentioning.11

And beyond the infinite…

On January 8th, 2002, for bug 113970, Stephen Walker lands a patch that takes yet another big chunk out of WebShell, and added this to the nsIWebShell.h header:

/**
 * WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING !!!!
 *
 * THIS INTERFACE IS DEPRECATED. DO NOT ADD STUFF OR CODE TO IT!!!!
 */

I’m actually surprised it too so long for something like this to get added to the nsIWebShell interface – though perhaps there was a shared understanding that nsIWebShell was shrinking, and such a notice wasn’t really needed.

On January 9th, 2003 (yes, a whole year later – I didn’t find much worth mentioning in the intervening time), I see the first reference to “deCOMtamination”, which is an effort to reduce the amount of XPCOM-style code being used. You can read up more on deCOMtamination here.

On January 13th, 2003, Nisheeth Ranjan lands a patch to use “machine learning” in order to order results in the urlbar autocomplete list. I guess this was the precursor to the frencency algorithm that the AwesomeBar uses today? Interestingly, this code was backed out again on February 21st, 2003 for reasons that aren’t immediately clear – but it looks like, according to this comment, the code was meant to be temporary in order to gather “weights” from participants in the Mozilla 1.3 beta, which could then be hard-coded into the product. The machine-learning bug got closed on June 25th, 2009 due to AwesomeBar and frecency.

On Februrary 11th, 2004, the onbeforeunload event is introduced.

On April 17, 2004, gerv lands the first of several patches to switch the licensing of the code over to the MPL/LGPL/GPL tri-license. That’d be MPL 1.1.

On July 7th, 2004, timeless lands a patch that makes it so that embedders can disable all plugins in a document.

On November 23rd, 2004, the parent and treeOwner attributes for nsIDocShellTreeItem are made scriptable. They are read-only for script.

On December 8th, 2004, bz lands a patch that makes DocShell inherit from DocLoader, and starts a move to eliminate nsIWebShell, nsIWebShellContainer, and nsIDocumentLoader.

And that’s probably the last notable patch related to DocShell in 2004.

Lord of the rings

Reading through all of these commits in the docshell/ and webshell/ folders is a bit like taking a core sample of a very mature tree, and reading its rings. I can see some really important events occurring as I flip through these commits – from the very birth of Mozilla, to the birth of XPCOM and XUL, to porkjockeys, and the start of the embedding efforts… all the way to the splitting out of Thunderbird, deCOMtamination and introduction of the MPL. I really got a sense of the history of the Mozilla project reading through these commits.

I feel like I’m getting a better and better sense of where DocShell came from, and why it does what it does. I hope if you’re reading this, you’re getting that same sense.

Stay tuned for the next bit, where I look at years 2005 to 2010.


  1. Messages like: ERROR: A CVS repository cannot contain both cvsroot/mozilla/browser/base/content/metadata.js,v and cvsroot/mozilla/browser/base/content/Attic/metadata.js,v, for example 

  2. Those commits have hashes 575e116287f99fbe26f54ca3f3dbda377002c5e7 and  60567bb185eb8eea80b9ab08d8695bb27ba74e15 if you want to follow along at home. 

  3. Strangely enough, that docshell folder wasn’t even being built. It was just sitting there, containing a copy of nsWebShell.cpp. That copy mirrored the one under webshell/, and the two copies were kept in sync. Maybe DocShell was always a thing that somebody always wanted to do but never got around to until Travis came along. 

  4. Mozilla Suite 1.0 eventually ships on June 5th, 2002 

  5. According to Wikipedia, Netscape was purchased by AOL on March 17, 1999 

  6. That wiki page was first posted on October 8th, 1999 – exactly a week before the first docshell work had started. 

  7. I should point out that there is no mention of this embedding work in the first roadmap that was published for the Mozilla project. It was only in the second roadmap, published almost a year after the DocShell work began, that embedding was mentioned, albeit briefly. 

  8. More clues as to what WebShell was originally intended for are also in that interface file:

    /**
     * The web shell is a container for implementations of nsIContentViewer.
     * It is a content-viewer-container and also knows how to delegate certain
     * behavior to an nsIWebShellContainer.
     *
     * Web shells can be arranged in a tree.
     *
     * Web shells are also nsIWebShellContainer's because they can contain
     * other web shells.
     */
    

    So that helps clear things up a bit. I think. 

  9. travis reviewed it, and the patch was approved by rickg. Not sure what approval meant back then… was that like a super review? 

  10. This is the last commit to the webshell folder before it got removed on September 30th, 2010. 

  11. Hopefully I didn’t miss anything interesting in there! 

Bugnotes

Over the past few weeks, I’ve been experimenting with taking notes on the bugs I’ve been fixing in Evernote.

I’ve always taken notes on my bugs, but usually in some disposable text file that gets tossed away once the bug is completed.

Evernote gives me more powers, like embedded images, checkboxes, etc. It’s really quite nice, and it lets me export to HTML.

Now that I have these notes, I thought it might be interesting to share them. If I have notes on a bug, here’s what I’m going to aim to do when the bug is closed:

  • Publish my notes on my new Bugnotes site1
  • Comment in the bug linking to my notes
  • Add a “bugnotes” tag to my comment with the link

I’ve just posted my first bugnote. It’s raw, unedited, and probably a little hard to follow if you don’t know what I’m working on. I thought maybe it’d be interesting to see what’s going on in my head as I fix a bug. And who knows, if somebody needs to re-investigate a bug or a related bug down the line, these notes might save some people some time.

Anyhow, here are my bugnotes. And if you’re interested in doing something similar, you can fork it (I’m using Jekyll for static site construction).


  1. I didn’t want to pollute this blog with them (plus, dumping these files into WordPress seemed to be a bit heavy)  

DocShell in a Nutshell – Part 1: Original Intents

I think in order to truly understand what the DocShell currently is, we have to find out where the idea of creating it came from. That means going way, way back to its inception, and figuring out what its original purpose was.

So I’ve gone back, peered through various archived wiki pages, newsgroup and mailing list posts, and I think I’ve figured out that original purpose.1

The original purpose can be, I believe, summed up in a single word: embedding.

Embedding

Back in the late 90’s, sometime after the Mozilla codebase was open-sourced, it became clear to some folks that the web was “going places”. It was the “bees knees”. It was the “cat’s pajamas”. As such, it was likely that more and more desktop applications were going to need to be able to access and render web content.

The thing is, accessing and rendering web content is hard. Really hard. One does not simply write web browsing capabilities into their application from scratch hoping for the best. Heartbreak is in that direction.

Instead, the idea was that pre-existing web engines could be embedded into other applications. For example, Steam, Valve’s game distribution platform, displays a ton of web content in its user interface. All of those Steam store pages? Those are web pages! They’re using an embedded web engine in order to display that stuff.2

So making Gecko easily embeddable was, at the time, a real goal, and a real project.

nsWebShell

The problem was that embedding Gecko was painful. The top-level component that embedders needed to instantiate and communicate with was called “nsWebShell”, and it was a pretty unwieldy. Lots of internal knowledge about the internal workings of Gecko was leaked through the nsWebShell component, and it’s interface changed far too often.

It was also inefficient – the nsWebShell didn’t just represent the top-level “thing that loads web content”. Instances of nsWebShell were also used recursively for subdocuments within those documents – for example, (i)frames within a webpage. These nested nsWebShell’s formed a tree. That’s all well and good, except for the fact that there were things that the nsWebShell loaded or did that only the top-level nsWebShell really needed to load or do. So there was definitely room for some performance improvement.

In order to correct all of these issues, a plan was concocted to retire nsWebShell in favour of several new components and a slew of new interfaces. Two of those new components were nsDocShell and nsWebBrowser.

nsWebBrowser

nsWebBrowser would be the thing that embedders would drop into the applications – it would be the browser, and would do all of the loading / doing of things that only the top-level web browser needed to do.

The interface for nsWebBrowser would be minimal, just exposing enough so that an embedder could drop one into their application with little fuss, point it at a URL, set up some listeners, and watch it dance.

nsDocShell

nsDocShell would be… well, everything else that nsWebBrowser wasn’t. So that dumping ground that was nsWebShell would get dumped into nsDocShell instead. However, a number of new, logically separated interfaces would be created for nsDocShell.

Examples of those interfaces were:

  • nsIDocShell
  • nsIDocShellTreeItem
  • nsIDocShellTreeNode
  • nsIWebNavigation
  • nsIWebProgress
  • nsIBaseWindow
  • nsIScrollable
  • nsITextScroll
  • nsIContentViewerContainer
  • nsIInterfaceRequestor
  • nsIScriptGlobalObjectOwner
  • nsIRefreshURI

So instead of a gigantic, ever changing interface, you had lots of smaller interfaces, many of which could eventually be frozen over time (which is good for embedders).

These interfaces also made it possible to shield embedders from various internals of the nsDocShell component that embedders shouldn’t have to worry about.

Ok, but… what was it?

But I still haven’t answered the question – what was the DocShell at this point? What was it supposed to do now that it was created.

This ancient wiki page spells it out nicely:

This class is responsible for initiating the loading and viewing of a document.

This document also does a good job of describing what a DocShell is and does.

Basically, any time a document is to be viewed, a DocShell needs to be created to view it. We create the DocShell, and then we point that DocShell at the URL, and it does the job of kicking off communications via the network layer, and dealing with the content once it comes back.

So it’s no wonder that it was (and still is!) a dumping ground – when it comes to loading and displaying content, nsDocShell is the central nexus point of communications for all components that make that stuff happen.

I believe that was the original purpose of nsDocShell, anyhow.

And why “shell”?

This is a simple question that has been on my mind since I started this. What does the “shell” mean in nsDocShell?

Y’know, I think it’s actually a fragment left over from the embedding work, and that it really has no meaning anymore. Originally, nsWebShell was the separation point between an embedder and the Gecko web engine – so I think I can understand what “shell” means in that context – it’s the touch-point between the embedder, and the embedee.

I think nsDocShell was given the “shell” monicker because it did the job of taking over most of nsWebShell’s duties. However, since nsWebBrowser was now the touch-point between the embedder and embedee… maybe shell makes less sense. I wonder if we missed an opportunity to name nsDocShell something better.

In some ways, “shell” might make some sense because it is the separation between various documents (the root document, any sibling documents, and child documents)… but I think that’s a bit of a stretch.

But now I’m racking my brain for a better name (even though a rename is certainly not worth it at this point), and I can’t think of one.

What would you rename it, if you had the chance?

What is nsDocShell doing now?

I’m not sure what’s happened to nsDocShell over the years, and that’s the point of the next few posts in this series. I’m going to be going through the commits hitting nsDocShell from 1999 until the present day to see how nsDocShell has changed and evolved.

Hold on to your butts.

Further reading

The above was gleaned from the following sources:


  1. I’m very much prepared to be wrong about any / all of this. I’m making assertions and drawing conclusions by reading and interpreting things that other people have written about DocShell – and if the telephone game is any indication, this indirect analysis can be lossy. If I have misinterpreted, misunderstood, or completely missed the point in any of the above, please don’t hesitate to comment, and I will correct it forthwith. 

  2. They happen to be using WebKit, the same web engine that powers Safari, and (until recently) Chromium. According to this, they’re using the Chromium Embedding Framework to display this web content. There are a number of applications that embed Gecko. Firefox is the primary consumer of Gecko. Thunderbird is another obvious one – when you display HTML email, it’s using the Gecko web engine in order to lay it out and display it. WINE uses Gecko to allow Windows-binaries to browse the web. Making your web engine embeddable, however, has a development cost, and over the years, making Gecko embeddable seems to have become less of a priority. Servo is a next-generation web browser engine from Mozilla Research that aims to be embeddable. 

Alice in DocShell Land

I’ve been reading a book called The Annotated Alice. In this book, the late and great Martin Gardner shows us the stories of Alice’s Adventures in Wonderland and Through the Looking-Glass but supplies copious footnotes to illustrate the puns, wordplay, allusions, logic problems and satire going on beneath the text. Some of these footnotes delve into pure conjecture (there are still people to this day who theorize about various aspects of the stories), and other footnotes show quite clearly that Carrol wrote these stories with a sophisticated wit and whimsy that isn’t immediately obvious at first glance.

And it’s clear that Gardner (and others like him) have spent hours upon hours thinking and theorizing about these stories. A purposeful misspelling gets awarded a two page footnote here, and a mention of a mirror sends us off talking about matter and anti-matter and other matters (ha) of quantum physics.

So much thinking and effort to interpret these stories, and what you get out of it is a fascinating tapestry of ideas and history.

Needless to say, I’ve been finding the whole thing fascinating. It’s a hell of a read.

While reading it, I’ve wondered what it’d be like to apply the same practice to source code. Take some relatively mysterious piece of source code that only a few people feel comfortable with, and explode it out. Go through the source control history, and all of the old bugs, and see where this code came from. What was its purpose to begin with? What is its purpose now? What are the battle scars?

After much thinking, I’ve decided to try this, and I’m going to try it on a piece of Gecko called “DocShell”.

I think I just heard Ms2ger laughing somewhere.

It’s become pretty clear having talked to a few seasoned Mozilla hackers that DocShell is not well understood. The wiki page on it makes that even more clear – it starts:

The goal of this page is to serve as a dumping/organization ground for docshell docs. When someone finds out something, it should be added here in a reasonable way. By the time this gets unwieldy, hopefully we will have enough material for several actual docs on what docshell does and why.

So, I’m going to attempt to figure out what DocShell was supposed to do, and figure out what it currently does. I’m going to dig through source code, old bugs, and old CVS commits, back to the point where Netscape first open-sourced the Mozilla code-base.

It’s not going to be easy. It’s definitely going to be a multiple month, multiple post effort. I’m likely to get things wrong, or only partially correct. I’ll need help in those cases, so please comment.

And I might not succeed in figuring out what DocShell was supposed to do. But I’m pretty confident I can get a grasp on what it currently does.

So in the end, if I’m lucky, we’ll end up with a few things:

  1. A greater shared understanding of DocShell
  2. Materials that can be used to flesh out the DocShell wiki
  3. Better inline documentation for DocShell maybe?

I’ve also asked bz to forward me feedback requests for DocShell patches, so that way I get another angle of attack on understanding the code.

So, deep breath. Here goes. Watch this space.