Introducing Ubiquity

An experiment into connecting the Web with language.

It Doesn’t Have to be This Way

You’re writing an email to invite a friend to meet at a local San Francisco restaurant that neither of you has been to.  You’d like to include a map. Today, this involves the disjointed tasks of message composition on a web-mail service, mapping the address on a map site, searching for reviews on the restaurant on a search engine, and finally copying all links into the message being composed.  This familiar sequence is an awful lot of clicking, typing, searching, copying, and pasting in order to do a very simple task.  And you haven’t even really sent a map or useful reviews—only links to them.

This kind of clunky, time-consuming interaction is common on the Web. Mashups help in some cases but they are static, require Web development skills, and are largely site-centric rather than user-centric.

It’s even worse on mobile devices, where limited capability and fidelity makes this onerous or nearly impossible.

Most people do not have an easy way to manage the vast resources of the Web to simplify their task at hand. For the most part they are left trundling between web sites, performing common tasks resulting in frustration and wasted time.

Enter Ubiquity

Today we’re announcing the launch of Ubiquity, a Mozilla Labs experiment into connecting the Web with language in an attempt to find new user interfaces that could make it possible for everyone to do common Web tasks more quickly and easily.

The overall goals of Ubiquity are to explore how best to:

• Empower users to control the web browser with language-based instructions. (With search, users type what they want to find. With Ubiquity, they type what they want to do.)
• Enable on-demand, user-generated mashups with existing open Web APIs. (In other words, allowing everyone–not just Web developers–to remix the Web so it fits their needs, no matter what page they are on, or what they are doing.)
  
• Use Trust networks and social constructs to balance security with ease of extensibility.
  
• Extend the browser functionality easily.
  
  

Learn more about Ubiquity and the capabilities that it could provide for users and developers.

The Initial Prototype

As part of this announcement, we’re also releasing an early experimental prototype to demonstrate some of the concepts of Ubiquity and the possibilities that it opens up. This release is meant as a illustration of a concept and mainly focuses on the platform. The next release will explore interfaces that are closer to features that might make it into Firefox.

Install the prototype and you’ll be presented with a tutorial to get you started.

Ubiquity 0.1

• Lets you map and insert maps anywhere; translate on-page; search amazon, google, wikipedia, yahoo, youtube, etc.; digg and twitter; lookup and insert yelp review; get the weather; syntax highlight any code you find; and a lot more. Ubiquity “command list” to see them all.
  
• Find and install new commands to extend your browser’s vocabulary through a simple subscription mechanism
  
• Read about Ubiquity In Depth, or see a number of the commands in action (with screenshots) in the Ubiquity Tutorial.

All of the code underlying the Ubiquity experiment is being released as open source software under the the GPL/MPL/LGPL tri-license.

This is the goal of what kinds of language-based services Ubiquity hopes to inspire people to create:

This is a screenshot of Ubiquity’s current map functionality:

Influences, References, and Background Resources

• The Graphical Keyboard User Interface by Alex Faaborg
• Why Verbs by Jono DiCarlo
• The Linguistic Command Line by Aza Raskin
• The UI Breakthrough-Command Line Interfaces by Don Norman
  
• Enso by Humanized
  
• Trust-based security, and Herd-based knowledge by Atul Varma
  

For a full list, see the credits page.

Get Involved

Mozilla Labs is a virtual lab where people come together online to create, experiment and play with Web innovations for the public benefit. The Ubiquity experiment is still in its infancy and just getting started. There are many ways to join the team and get involved:

• Download Ubiquity 0.1
• Find Ubiquity Commands
• Get Help, Make Suggestions, or Write Documentation to Help Others
• Write Commands for Ubiquity
  
• Report a Bug
• Discuss Ubiquity
  
• View Source

We’ve also started compiling a suggestion list for possible Ubiquity commands. If you have any suggestions, add them here or get inspired and develop one of them and add them to the command repository.

I’m sure that I’m not the only one, before noticing it was coming from Planet Mozilla, thought:

“What does maps have to do with Ubuntu’s graphical installer?”

Wow. This is simplay amazing. This prototype is already quite useful, I can only dream about what is possile with this concept in the future.

Good job, Aza. Keep it coming.

I’m stunned. This is fantastic thinking; I can;t wait to try it out.

brilliant,
you guys are just doing a terrific job here.. keep up the good work.

I have one question, how will you make this tool open for everyone to write services or plugins for it so it can recognize more services and web sites.

Clicked the Discuss link - got a load of porn. I hate porn. Who is in charge of keeping the discussion group clean?

Very cool. It’s like Humanized.com’s Enso, but for the web.

I’m very impressed Aza with your work !
Actually I’m a Songbird Community Develloper and I want to explore a new way of this extension in Songbird ;)

Absolutely wonderful. This is how to get life done. Nice work, I hope there’s enough developer interest in this to bring this to the top of the popular plug-ins!

genius

If you want people to adapt it, write a book “Mozilla’s ubiquity for dummies”

great ideas!

This is really awesome! Would love to see support for Google Apps for my domain

@Denny et al., Thanks for the kind words. We are super-excited about where this shared exploration of increasing the the connected-ness of the web means. I really cannot take the credit though. That goes to these folks.

In particular, Atul for architecting and thinking about trust networks, Jono for the logic of the parser and fledgling internationalization, Blair for writing many of the commands, and Abi for being the glue (especially in documentation) that keeps things together. This is community-based collaboration.

@Amr, Free Stuff: We have two such tutorials. The first is for users. The second is a step-by-step for all developers on how to extend Ubiquity.

@Lars: Thanks. We’ve fixed it.

Sounds ultra cool, there’s this scheduling service called Presdo that has similar natural language capabilities, but exclusive to event organization, this sound pretty easy to integrate with it.

Just so you know, I’m going to extend this for Presdo, I think it makes a great fit, and I’ve already done firefox extensions before. The Presdo natural language parser is a little more flexible, so hopefully I can inject the whole parser into parts of Ubiquity.

Ubiquity is stunning its worth the simple 5 second install just to use the twitter feature, Great work!

This is shaping up to be a very good program, even in this very early stage.

I’ve noticed the translate feature has a rather large flaw from my limited time with it, however; the program does not recognize letters with umlaut from German/Swedish etc: Ä/ä Ö/ö Ü/ü.

Awesome.

When is it getting speech recognition? :-)

Let me echo the “awesome” and “good job” comments by others…I agree.

There is so much at our fingertips that feel too time consuming to bother accessing, collating and delivering. Like Quicksilver, I can see this quickly becoming ‘table stakes’ to be in the game for a power user.

Kudos! Mozilla Labs team is the undisputed web-apps/browser innovative leader.

Wow! All I can say is Wow! Great job - looking forward to seeing this mature.

Is this kind of like a sophisticated Gnome Do but for Firefox?
http://do.davebsd.com/

This really rocks. I’ve been using it all morning. It’s kind of like the Hyperwords extension that I have been using a lot but I think this will be much better.

Aza, great work - this really blew me away - downloaded and installed :)

Ooh, video tutorial from Aza — I watched just the beginning so far but look forward to viewing the whole thing, and understanding better!

Aza, how was the video made? I like the part in the beginning where the text syncs to your words. :)

Most of these stuff can be achieved by bookmarklets and greasemonkey ?

So what the hell is that gibberish at 0:37? -

ABIMANYU RAJA ATUL VARMA AZA
RASKIN BLAIR MCBRIDE
JONO DICARLO GIALLOPORPORA
CHRIS BEARD DAN MILLS MYK
MELEZ DIETRICH AYALA

Absolutely awesome, I love this thing already

Excellent.
Ubiquity will be soon everywhere ;-)

Great idea and a great concept.

Good Idea.

Super Idea!

amazing, it’s quicksilver or the lesser spotlight for the new computer, the browser. now we need to get this to work with flash. or rather flash to work with everything else.

hmm, for some reason, when i type

email this to chris

‘to chris’ appears at the end of the preview (aka part of the message) and pressing enter does exactly that, leaving the To field empty. am i missing something?

Guess who will envy stuff like this in their apps? Everyone but Mozilla! ;-)

Ubiquity rocks!

brilliant work. a few bugs, but this is an outstanding 0.1!

Not to be a dick, but you guys make a decent simple browser, it starts to get used and mozilla cashes out via google ads. Next thing you know you bloat it up, and start dumping one dumb feature after another into it. I thought the goal of Phoenix, Firebird, Firefox was to take all the junk out of Mozilla (email, irc chat, etc) and just make a streamlined browser. You know you lost me, I’m using Safari now. It isn’t trying become the emacs of web browsers

Amazing… Just amazing. Keep up the great work!

Impressive. It’s like a fantastic combination of YubNub and my favorite FF extensions all rolled up into one delicious package.

I look forward to seeing what people come up with for Ubiquity.

Nice project. Any chance that supports any other languague like spanish?

Absolutely stunning. Can we utilize Awesome Bar for this ?

@Bjorn Tipling’s comment is quite interesting … “trying to become the emacs of web browsers”.

I had to convert all my email accounts to GMAIL because I was unable to keep both my email client and firefox open for more then 2-3 hours before rebooting and getting the white screen. Although, I prefer it now .. is not relevant :)

But, that’s an interesting video and can see some useful applications .. and would also second @free stuff’s suggestion … “write a book “Mozilla’s ubiquity for dummies””

Amazing!! Looking forward to play with it..

Hello,

Amazing utility, Aza.

I’ll report one bug: It makes FastDial add-on to not show its screen.

(where do I file that bug?)

Thanks

Why didn’t I think of this?

Oh, I found the bug site, but I can’t file a new ticket there…

great experiment! nice to see. Normally i used print screens of things you mentioned, like maps or other web services… this is far more easy!!!

i would love to see an integration in Thunderbird!

I love it !
This is amazing for a first alpha =)

This is awesome. I was skeptical at first about the usefulness of something like this, then I actually tried it….. just simply amazing. And the best part: it doesn’t slow down my computer at ALL!

I don’t know how you managed that, to keep it from slowing down the computer, but you must be gods at coding.

wauw, this looks really amazing!
can’t wait to try this out!

Wow,looking for this.
This is absolutely great!

I’ve just installed Ubiquity and give it a try for a few minutes… it’s a pure bliss, just every repeating task I always found completely useless seems to vanish in the air. YEAH!

Thank you very much, it’s like (ShortWave+Twitterific+Gmail+…)^1K

Wow!!!!!!! Excellent. This is exactly the condensation of activities I always perform. I’m looking for houses and cars at the moment, and this tool is saving me tons of time via custom car spec checks, price comparison checks, and the “built in” map command.

This is superlative!

Great stuff guys! Keep up the good work.

I will certainly be following this closely!

Exciting stuff!

The idea is great, but why do I have to type so much? Its so time cosuming …
Life is short bro

what about mobile access?

This is great!

But there is a problem; the email is not an efficient environment to communicate about maps, reviews or any other microformat. Here is a solution (Google example):

http://bonte-pierre.blogspot.com/2008/04/user-interface-for-google-contents-and.html

This could be done by the browser too.

Simply awesome. Will be using this alot!

Heard about Devo?

https://addons.mozilla.org/en-US/firefox/addon/8179

You should *REALLY* cooperate! A combination of this and Devo would be awesome!

Its really, really great but:
has anybody considered that this response-on-typing feature (autocomplete) lets say with wikipedia’s or google’s APIs has a ecological aspect too? millions of users will use these features. on every hit there will be a request across the globe, which costs some energy. (i know there is caching, however,…)
i am a techfreak too and i would love to see this project growing, but i never read about these issues, and i am wondering if these aspects are considered too at mozilla labs.

Wow, this is a terrible idea! How could you have possibly come up with it?

“This kind of clunky, time-consuming interaction is common on the Web.”

Oh wait, I see where you went wrong. Do you see it too? That’s right… this kind of interaction is not even remotely common on the web. Ask mom: “Mom, you know how you are always bugging me about how it’s so hard to create on-demand, user-generated mashups?”

Yes, there’s a minority of web users for whom speed and automation are important, and they already know how to use command line interfaces, so this will work for them. If this is your intended audience, then hats off to you, sir! Otherwise, this is another bad idea that began with “If only people were software developers like me, they would be so much better off!”

UBER COOL!

very nice thing to have

WOnderful…
got to know abt it through Twitter/mba_piyush

Wonderful …love it

cam to know abt it through Twitter

Brilliant work.

I use the Simple Mail extension and this kills it

good job. it is very useful in the future.

I’m surprised to not see YubNub on the inspirations list, since this looks like the logical progression of that.

This looks incredible. You guys at Mozilla are doing great things to improve our online worlds. I can’t wait to try it!

@Natanael L HAHA. I wrote Devo and I’ve been contributing to Ubiquity for a while now. =)

I’ve just gone ahead and installed it to give it a try. The video looks great and I think it’s a great step forward is making the Web more useful for everyone.

Blogged about it here: http://www.u-g-h.com/2008/08/27/natural-language-in-your-browser/

Genius….I’ve never been so motivated to donate my time as a developer as I am now. I guess I have some reading to do :)

Very promising project! Great that it’s open source from the beginning, unlike Quicksilver which is slowly declining due to stability problems and lack of new releases (even though the core has been opened recently). I’m already addicted to Ubiquity, I think it’s already better than Quicksilver when it comes to cooperation with the web.

Tons of bugs but also great potential…

http://compujitsu.blogspot.com/http://compujitsu.blogspot.com/

What can I say? Ground-breaking. This really has a lot of potential, and it’s quite easy to write your own commands too. I bet there will be a lot of them everywhere, in just a few days.
Keep it up guys!

Absolutely great! Great potential, awesome idea!

the logo looks like the Ubisoft logo. also the name!

Thanks for the information

brilliant,Amazing…
you guys are rocking just doing a superb job .. keep it up !!! great work done.

Great……This is going to change the dimension of user experience.
Just 2 thoughts for now:
a) Will it be browser independent…-:)
b) Same user experience has to be taken out of realm of web and should be used for all type of applications.

Any thought of some type of Facebook integration ?

What are you talking about? I add maps to email all the time. Not links, maps. Bring up the map, click on grab or hit the grab area hotkey, sweep out the map, go back to the mail program, hit paste, it’s done.

What, you can’t do that on Windows?

Oh, well, that’s a different problem.

Superb - far far more useful than the desktop version of something similar to this (jeez for the life of me I can’t remember or find the name of the app, but it does a similar thing for the desktop)

Outstanding!!

This looks like a very very nice tool :)
Ofcourse as previously said it is similar to greasemonkey but it is probably easier to use.

Good job Aza - yet another great idea! I think you’re on to something with the whole notion of a web-tool for everything.

Wow, this is amazing! I will definitely try to develop some commands.

I’ve just tried it out. It is absolutely fantastic!!! Hope it can get into the next release of Firefox!

You’re thinking outside of the box, when others are thinking status quo.

It’ sounds great! I’ll test it asap.

Lovely. Lots of potential. Now overtake the IE market already, I can’t wait.

Excellent work! I’ve already started using this and am finding it wonderful.

This looks and sounds amazing although I believe it will be quite hard for some users especially beginners, to understand the key terms used to perform a task. It will be very interesting to see how you form the semantic relationships which define those tasks into an easy to understand and cognitive fashion.

I have used ubiquity for only five minutes and can not go back to not using it. Congratulations on a fantastic tool!

I guess I just don’t see an overall need for this, but can’t the resources being put into this, be put into something more useful? Like fixing more security holes and optimizing speeds even more?

The opening paragraph, explaining the steps to set up a lunch date with someone. Oh no! a few extra clicks! The horror! Links only, not the actual text and pictures, oh no, how ever shall we coordinate, we have to do all this clicking and reading for ourselves.

We’re already becoming exponentially more lazy and expecting computers to do everything for us. We don’t need more ‘advances’ to push this along.

Don’t get me wrong, I love computers, I’m a computer scientist, they are my life. But I don’t expect them to, nor do I want them to, interpret my every need.

http://www.golovelife.com - meet single girls.

This is just crazily amazing!I can’t write anything anymore, I am still in shock!

very interesting. just installed and im gonna play around with it. video of it looked great

So, I am a neophyte, yes, but I am also a Mac user and I when I tried to download I got a message saying I don’t compatibility with my Mac. Is that the case? I’d love to test drive this thing.

Very innovative!

Fantastic! It’s one of those things I wanted a lot in a browser!

It’s the most interesting and the easiest way to extend the browser at all!

As soon as I finish work here, I’ll create my own verbs.

Congrats! Mozilla, as always, is improving the way we browse, and so the way we live.

Brilliant! This is amazing.

nice, but i don’t like to type shit. clik, clik, clik. mouse. use it. put all that shit on the second mouse button, muse scroll > choose click use. yep im lacy.

Brilliant!

Just installed it and now my firefox dies at launch!!! Not nice at all!!!

This is great! Would it be possible to allow users to define their own keys in the next version? I’d rather hit tab to scroll through the commands than the down arrow.

Awesome. I’ll (of course) miss this in all my other apps. I’d use Enso, but my Caps Lock key is currently serving Control key duty (and I like it that way!).

Anyway, I like the zoom command. I’ve never seen a Firefox extension that zooms the *entire* interface.

/me still wish Windows had on-the-fly font DPI switching like Linux (or at least Gnome/GTK) has.

– J.P.

Other than UI implementation, how is this the same/different than IE 8 Activities? Will it support the same schema?

This is a really neat project. I hope it takes off. However, I doubt asking people to type commands is a really understandable solution to the non-computer-nerd segment of the population.

I’d recommend taking a look at Apple Mail’s Data Detectors for inspiration towards a different approach. For example, whenever a date is automatically detected in Mail, hovering over it presents an option menu that allows you to perform related actions such as creating a new iCal event with that date pre-selected.

It should work the same way in a web-browser. Highlighting text should alter the types of available commands in a contextual menu.

Here’s an example of how things might work.

Say you’re reading a message board and somebody wrote, “Hey, did you guys hear that Linkin Park is playing at the Spectrum on Sept. 27th? I love that song of theirs My December. I can’t wait for that!”

Highlighting that bit of text, the system should quickly process it, and give you a menu that presents a few different options.

* a Wiki link for the music artist (the system should be intelligent enough through various grammatical cues and perhaps a quick wiki/google search)
* a link to the ticket purchase site
* plane tickets to Philadelphia (the system should be intelligent enough to realize that the Spectrum is in Philadelphia and you live far enough away to justify plane tickets
* create a new calendar event
* A Map of the region
* An Email this link
* A youtube link to that song’s video.

This approach is definitely not as powerful as a command-line type system, but probably much more usable for the average person.

Implementing this well is obviously a really hard problem to solve though. But I think this ubiquity project is definitely a large step in the right direction. I like this project right now but I really love where it could go within a year or two if this other approach is taken.

@alsomike: I think you’re missing the bigger picture. Yes the examples like tinyurl and twitter are very “early adopter” scenarios, but if you were to add voice recognition into the mix then your looking at the start of the voice enabled internet. Your grandma may not want to do “command-shift” and know the right context when to right click, but I’d bet she’d like it if she turned on the computer and said “Find me some place I can complain about the Internet.” and Google’s created an I’m Feeling Lucky script that instantly pops up internethaters.com. Or she says “Can you look for Mary Jones? I went to grade school with her. I want to apologize for killing her cat 80 years ago.” And a Classmates.com widget find her classmate, finds she’s been active on their site for years and opens a message window that already has prepopulated “I want to apologize for killing your cat.” Your system’s current voice recognition can’t do this because it has no context to put the instructions in and no logic to carry out the commands. This puts both in the hands of the individual websites that know their context and know their own business logic. This is just another step in moving webservices closer to the client with a different interaction other than a Flash or HTML based ui.

My concern is there are already several applications out there like this, and each time the websites that want to capitalize on them they have to rewrite the wheel, there’s no standardized interface. This is another proprietary script you’ll need to maintain and update. While I detest waiting 14 years for the 0.1 spec for this to come out from the W3C, there may need to be some kind of informal coalition created to develop some kind of standardized API for these contextualized webservice clients (and please leave room for voice). I say this because when it gets popular, both Microsoft and Apple will develop their own interfaces into safari and ie.

অসাধারণ। এমনইতো চাই।

Some of the most innovative ideas I’ve seen in the last year. Congratulations!

Woaw, a real shift in web browsing! Bravo!
Waiting for Hotmail integration!

Really good prototype. The email’s a bit funny tho. At first it kept just writing the html instead of rich text, but now it’s working ok. don’t like how it forms the emails sometimes and would be great if it could actually send simple emails straight from the interface without having to open up gmail (like how it does twitter). anyway, good work. can’t wait for more updates

The gayist thing Ive ever seen, don’t you lot have real lives?

also, it only opens up the email pane of gmail when you don’t have gmail already open in another tab

Hey very very cool stuff!!

A question, is it possible to change the translator language?

Thx for than

Not usually a big fan of most concept ideas but I really love this one and can’t wait to be able to use it.

I like vodka