Rambling about a new Address Book for Thunderbird

Hey all.

I’ve been pretty bad about talking about the address book. Sorry about that. I’ve been pretty busy helping to get Instant Messaging shipped for TB 15.

But with IM polished off, I’ve (once again) started refocusing my attention to the address book.

So where am I?

The Ensemble Project

In short, I’m writing an add-on to experiment with approaches to a new address book. I’m calling it “Thunderbird Ensemble”, because every project needs a cool code name.

The add-on is in its very very early stages. Nothing is usable. Nothing is stable or guaranteed. There’s no UI yet, but there are some tests.

You can track my progress here.

Here are some highlights, implementation details and unstructured thoughts about the add-on, and where I’m going with it:

SQLite Storage Back-end

I’ve chosen to use SQLite (via mozStorage) for the contact database. I was originally toying with using indexedDB, but it’s lack of support for substring searches (which is necessary for things like auto-complete) wasn’t very appealing. I briefly toyed with throwing my own indexing layer on TOP of indexedDB, and then I realized that I was probably reinventing the wheel.

So SQLite it is!

Diffs, Diffs Everywhere

Contact records are a representation of the fields and meta data about a contact. My add-on will be able to calculate diffs between records, and those diffs can be applied on top of records to make changes.

So, for example, suppose I decide to make some changes to a contact – I add some email addresses, remove a phone number, and change a birthday. I’m effectively creating a diff.

Now we apply the diff on top of the original contact, and send the resulting contact record to the datastore for storage.

If the storage happens to be on the network, and the original record we applied the diff on to was out of date, we should get a fresh copy of the contact record, apply the diff, and try again.

Conflict Resolution

With diffs comes conflicts resolution – but conflict resolution is never a pleasant experience, and I’d like the user to not have to deal with that if possible.

My solution is to just be optimistic – if a diff is asking to remove something that isn’t there, great! Mission accomplished! If the diff is adding something that already exists, no problem! We skip!

And for things that get changed, like birthdays, we simply say that the last writer wins.

Maybe that’s naive, but that’s the approach that I’m taking.

What do I have?

I can create contact records, diff them, apply diffs, and “merge” two records together. And I have tests for all of that work.

I’ve got the start of the SQLite backend working (basically, we can set up the database…but that’s it.)

I’m currently working on importing old Thunderbird contacts into Ensemble’s notion of contact records.

After that, I’ll work on getting them written to the database.

Comments? Questions? Complaints? Ask them here. It’s still really early on in the process, but if you find bugs in my code, please file issues on GitHub.

27 thoughts on “Rambling about a new Address Book for Thunderbird”

  1. Mike Ratcliffe

    Awesome, in the future will this allow us to use e.g. google contacts?

    They can be used now but there are huge issues with contacts getting duplicated or lost.

  2. rgloor

    Hi Mike

    Wow. You just made my day!!!

    The address book is the TB number 1 issue for me (and many others).
    The fact that you are back working on it and you are making progress, that is the best news for a long time.

    Here some feedback from the desk of ‘just’ a plain user:

    1.) The Thunderbird Ensemble Project
    ————————————
    Congratulation to your Thunderbird Ensemble Project. I really very, very, very much appreciate it.
    Regarding the add-on approach:
    I hope, that after successful development, it will be merged into the TB core (main app). So any risk of incompatibility with a new release is vanished.

    2.) SQLite
    ———-
    I think that is a great decision.
    If I am not wrong, several data sets are already stored in SQLite DB’s (files) in FF and TB.
    So we also have consistency there.

    3.) Diffs
    ———
    Great. That sounds very logical to me. Now different fields of the same contact can be modified on different storages (local, web, phone, 2nd PC, …) without any problems. Good approach. I like it.

    4.) Conflict Resolution
    ———————–
    Your (basic) logic is fine. While in the past, many/most apps only checked if a contact as a whole was changed. And hence, there was a high chance of conflicts.
    With the diff approach, that has been solved for most (change) cases. They are simple (on the record/field level) and don’t generate conflicts:

    For the now very rare event of a conflict, I wished, there were some user configurable settings (preferences):

    a) newer wins
    b) duplicate entries (field, record)

    c) don’t change that contact (leave the difference)

    …but for all conflict cases, create a conflict log.
    The conflict log should be easy accessible. (eg through the menu and a button, which could be prepared but left to the user to place it in the icon bar or not. But has to be available when modifying the icon bars.)
    The conflict log should include:
    – only entries when conflicts occure
    – date of conflict
    – storage places (local and google, local and PC 2, …)
    – which contact is affected
    – show the (previous) different field contents
    – if changed, show what changed

    Sample:
    **** Address Book Sync Conflict: 2012_08_23-18:56 ****
    – Mike Sample:
    — Home Phone:
    — Thunderbird local: +41 41 622 23 24
    — Google Contacts: +41 41 655 55 55
    — changed to +41 41 655 55 55 (or: not changed)
    — URL:
    — Thunderbird local: http://www.mikesample.com
    — Google Contacts: http://www.mike-sample.com
    — changed to http://www.mikesample.com
    ****

    And there should be an easy accessible button to delete the log.
    So one could solve all conflicts and if everything is fine, delete the log.
    Hence only unhandled conflicts appear.

    Additionally, the conflicts could show in a pop-up, similar to the reminder pop-up from Lightning.
    Then one could select an individual conflict entry, solve it and delete just this one (which would then be removed from the log).

    With the diff’s approach, there should be only a very rare occurance of conflicts (on the record/field level).
    But those might be needed to deal with.

    The above mentioned thoughts represent just some ideas, which might help you to solve it.

    Oh, there is one more wish:

    5.) CardDAV Sync
    —————-
    Please implement the CardDAV sync support. (Beside the other important standards.)

    Again, thank you very much for your work on this important issue.

    Best regards from Switzerland,
    Rolf

  3. Pingback: Thunderbird bekommt ein neues Adressbuch - soeren-hentzschel.at

  8. Vasco

    I do have a question. Is this aiming at eventually being incorporated into the main codebase when it’s ready, or will it just end up like Lightning and TB Conversations as a perpetual add-on?

  9. Mike

    @Robert Kaiser

    The Contacts API’s purpose is to allow access to the device or system address book – so I’m hoping to use that to communicate with the OSX Address Book on OSX, EDS on Linux, and Windows Contacts equivalents on Windows.

  10. Mike

    @James Napolitano

    Like I said to Robert, the Contacts API back-end is specifically to communicate with the device address book, and has no notion of external address books.

    My design plans to communicate with the OS’s address book via the Contacts API, as well as communicating with other contact services (LDAP, Google Contacts, etc). As such, I can’t really depend on the Gaia Contacts back-end (plus, I wouldn’t want to be constrained, nor constrain them).

  11. Mike

    @Vasco

    My goal is to merge this project into the main codebase once the project has stabilized and we’ve got at least an equivalent feature set to the current address book.

  13. john

    I need some features from thunderbird contacs so to get rid of ms outlook.

    1. be able to put categories in contacts and sort them according to those
    2. be able to have folders and subfolders of contacts.

    Yes I need folders even if categories exist.
    In folders I may put old contacts, business contacts etc.
    In categories I may put the type of business

    is there any free alternative to outlook that can sync with android and google contacts? I dont know any yet.

    I hope to see these in thunderbird

    thanks a lot

  14. rgloor

    @Mike (Conley)

    Thanks a lot for working on this.
    (Looking at the Git Commit History,) It looks as you are making good progress.

    That is very encouraging for TB users!

    Thanks again.

    Rolf

  15. Alan Jones

    I am very excited about an improved Thunderbird address book. The current address book is good, but lacking in a few key areas. One thing I did not see that may be there is export/import support for various formats so that if one has to move to another system they can easily get the contacts out. Minimum CSV (quoted delimited) but others would be great for exporting both individual and groups/all contacts.

    In the long run my preference would be the the new features not be an add-on but directly integrated for more “direct” support. However I know there are many preferences in this area.

    For Google syncing maybe you could pull from/collaborate with an existing Google Sync tool.
    Not sure this one is as well know….
    http://www.pirules.org/addons/index.php

    Can’t forget about this one.
    https://addons.mozilla.org/en-us/thunderbird/addon/google-contacts/

    I know we all look forward to seeing something soon.

    good luck!!

    Alan

  16. rgloor

    @Mike (Conley)

    Hi Mike.

    I saw a great activity on the git (up to 7 commits per day).
    However, no more commits after August 31.

    Is the plugin finished? I also checked the Thunderbird Add-On website, but couldn’t find anything like that.

    Do you have some further information (roadmap)?
    When do you think, the add-on will be ready for testing?
    (I am so excited.)

    Thanks in advance for your great work and looking forward to your answer.

    Greetings from Switzerland,
    Rolf

  17. Mike

    @john

    Hey, thanks for writing to me, and sorry for the late response.

    I’m interested in the hierarchy you’re looking for in your groupings of contacts.

    You differentiate between folders and categories, whereas I just have a notion of categories / tags.

    What is stopping you from putting some contacts in an “old contacts” category, a “business contacts” category?

    Why can you not have a contact in the “business contact” category, and also have it in the type of business category? A contact should be able to exist in multiple categories.

    Is that an acceptable model? Or is there a requirement that I’m not understanding?

    -Mike

  19. Mike

    @Alan

    The way I’ve been approaching the new address book is to maintain contacts in an intermediate format (a Javascript object, easily serializable to JSON, and highly interoperable with DOM Contact).

    Exporting and importing from various sources is a requirement of my design, and the ability to register connectors to various source types is also a requirement.

    For static files, VCard, CSV and LDIF are on my export radar – both for individual and groups of contacts.

    I’m developing this feature as an add-on simply to make it easier to iterate on the design and (eventually) allow people to try it and give me feedback without having to prepare special builds. The eventual goal is to merge what I create (or a feature equivalent representation) into the core code.

    I’m hoping to have the code in a state where the community can get more involved soon!

    -Mike

  20. Mike

    @Rolf

    I’ve been traveling for a few days, and I’ve recently begun exploring an alternative implementation approach for the address book. Up until now, I’ve been developing my own utilities for manipulating contacts, and had begun writing my own code to save those contacts to the database.

    A colleague of mine has introduced me to Backbone.js and Underscore.js, and I’m currently evaluating them as possible replacements for my hand-rolled approach.

    I hope to have a roadmap, as well as a structure for getting the community involved soon. Soon, as in end of October, if I’m lucky / things go my way. That’s tentative, and please don’t hold me to that.

    -Mike

  21. rgloor

    @Mike

    Thanks a lot for all those updates.

    It is REALLY GOOD, and I really mean REALLY, REALLY GOOD, to sense that this time (with you) there is some serious work done on this TB AB.
    I hope Mozilla let’s you stay on this project and bringing it to the finish line. Like until it is merged into the core code. 😉

    Since I am not a programmer / coder and I can’t help you on this front. But let just my (supportive) words assure you, that we appreciate your work VERY, VERY much.

    And if there is a chance to test the add-on in a save manner (like with a secondary AB, beside the AB with live data, or something alike), then I would be glad to do some testing.

    Good luck with with those new JS code, hoping it takes some monkeys of your shoulder.

    Looking forward to the first beta (test) release.

    Thanks.

    -Rolf

  22. Norma

    @john, can’t you use separate address books for business contacts and old contacts?
    As for syncing, here’s a freeware Windows program that syncs Thunderbird address books (and calendars, etc) with an Android device: MyPhoneExplorer http://fjsoft.at

  23. Job

    I’m very happy to read this, there is hope for the addressbook and contacts in thunderbird. We have had so many issues with it in the past: duplicates, syncronization, difficult to handle a large amount of contacts, and so on. Thanks for your work and contributions.

  25. Kr1ll1n

    So, since you are working on an improved contact manager for Thunderbird, may I make a simple request?

    User selectable phone prefix+area code display formats. Seriously. This needs to happen.

    As an example, in my current address book, I have contacts whose numbers are presented in the following ways;

    8881112222
    (888)-111-2222
    (888) 111-2222
    +1 (888)-111-2222
    +1-(888)-111-2222
    etc.etc…..

    As you can see, this is obnoxious to look at, and while I and many others love and appreciate the FLOSS philosophy, sometimes being able to enforce a particular standard, even if it is a choice of many different standards, is a better solution than to present none at all.

    Thanks, and thank you for all your hard work!

  26. Konstantin

    Hello Mike,

    Do plan any synchronization capabilities of the address book with an external server? This is not an request or something like that, i was just interesting if this is in on our roadmap.

    Regards,

    Konstantin

Comments are closed.