developing met a web apps

8/7/2019 Developing Met a Web Apps

http://slidepdf.com/reader/full/developing-met-a-web-apps 1/164

Developing Metaweb-Enabled WebApplications

Metaweb Technologies, Inc.



Developing Metaweb-Enabled Web ApplicationsMetaweb Technologies, Inc.

Published 2007-03-08Copyright © 2007 Metaweb Technologies, Inc.



Table of Contents1. Introduction .................................................................................................................. 1

1.1. The Metaweb Query API ..................................................................................... 11.2. About this Manual .............................................................................................. 3

2. Metaweb Architecture .................................................................................................... 52.1. The Metaweb Object Model ................................................................................ 5

2.1.1. Common Object Properties ....................................................................... 72.1.2. Names, Keys, and Ids ............................................................................... 82.1.3. Topics ..................................................................................................... 8

2.2. Values ................................................................................................................ 92.2.1. /type/int .................................................................................................. 92.2.2. /type/ f oat ................................................................................................ 92.2.3. /type/boolean ......................................................................................... 102.2.4. /type/id .................................................................................................. 102.2.5. /type/text ............................................................................................... 102.2.6. /type/key ............................................................................................... 10

2.2.7. /type/rawstring ....................................................................................... 112.2.8. /type/uri ................................................................................................ 112.2.9. /type/datetime ........................................................................................ 11

2.3. Types ............................................................................................................... 132.3.1. Core Types ............................................................................................ 142.3.2. Content Types ........................................................................................ 152.3.3. Access Control Types ............................................................................. 16

2.4. Domains .......................................................................................................... 172.5. Namespaces ..................................................................................................... 172.6. Access Control ................................................................................................. 18

3. The Metaweb Query Language ..................................................................................... 193.1. JavaScript Object Notation ................................................................................ 19

3.1.1. JSON Literals: null, true, false ................................................................ 203.1.2. JSON Numbers ...................................................................................... 203.1.3. JSON Strings ......................................................................................... 203.1.4. JSON Arrays ......................................................................................... 213.1.5. JSON Objects ........................................................................................ 22

3.2. MQL Tutorial ................................................................................................... 233.2.1. Our First Query ..................................................................................... 253.2.2. Query/Response Symmetry .................................................................... 263.2.3. Metaweb Object IDs .............................................................................. 263.2.4. Multiple Results and Uniqueness Errors .................................................. 273.2.5. Nested Queries ...................................................................................... 30

3.2.6. Asking Metaweb For Objects .................................................................. 313.2.7. Expanded Values and Default Properties .................................................. 343.2.8. Review: Asking for Values ...................................................................... 353.2.9. Too Much Information ........................................................................... 363.2.10. The id and name Properties ................................................................... 373.2.11. Numeric Constraints ............................................................................. 393.2.12. Textual Constraints: Pattern Matching in Queries ................................... 413.2.13. Limiting Queries .................................................................................. 43

iii



3.2.14. The Sort Directive ................................................................................ 443.2.15. Ordered Collections ............................................................................. 463.2.16. Optional Queries .................................................................................. 483.2.17. Using Fully-Quali ed Property Names .................................................. 493.2.18. Wildcards ............................................................................................ 513.2.19. Expressing AND in Queries .................................................................. 533.2.20. Expressing OR in Queries ..................................................................... 553.2.21. Expressing NOT in Queries .................................................................. 573.2.22. Re f ective Queries ................................................................................ 58

3.3. The MQL Grammar .......................................................................................... 604. Metaweb Read Services ............................................................................................... 63

4.1. Basic mqlread Queries with Perl ........................................................................ 634.1.1. A Better Perl Album Lister ..................................................................... 64

4.2. The mqlread Service ......................................................................................... 674.2.1. mqlread Input ........................................................................................ 674.2.2. mqlread Output ...................................................................................... 684.2.3. Query and Response Envelopes .............................................................. 68

4.3. A Python Album Lister ...................................................................................... 704.4. A Metaweb-enabled PHP Web Application ......................................................... 724.5. Metaweb Queries with JavaScript ...................................................................... 74

4.5.1. Listing Albums and Tracks with JavaScript .............................................. 754.5.2. Client-side MQL Queries with <script> ................................................... 80

4.6. mqlread Errors ................................................................................................. 814.7. mqlread Cursors ............................................................................................... 824.8. Fetching Content with trans ............................................................................... 85

4.8.1. Browsing Recent Content on freebase.com .............................................. 864.9. Example: A Metaweb Type Browser .................................................................. 89

5. The MQL Write Grammar ............................................................................................ 955.1. MQL Write Tutorial .......................................................................................... 95

5.1.1. Creating a Type to Work With ................................................................. 955.1.2. Creating Objects .................................................................................... 975.1.3. Connecting Objects ................................................................................ 995.1.4. Disconnecting Objects .......................................................................... 1025.1.5. Writes and Default Properties ............................................................... 1045.1.6. Creating and Connecting More Objects ................................................. 1065.1.7. Review: Write Directives ...................................................................... 1115.1.8. Working with Sets ................................................................................ 1125.1.9. Bidirectional Links and Reciprocal Properties ........................................ 1145.1.10. Writes and Ordered Collections ........................................................... 1185.1.11. Namespaces ....................................................................................... 121

5.1.12. Properties, Types, and Domains ........................................................... 1275.2. MQL Write Grammar ..................................................................................... 1316. Metaweb Write Services ............................................................................................ 133

6.1. Logging in to Metaweb ................................................................................... 1336.1.1. The Login API ..................................................................................... 135

6.2. Making Write Queries ..................................................................................... 1356.2.1. The mqlwrite Query Envelope ............................................................... 1356.2.2. The Response Envelope ........................................................................ 1366.2.3. A mqlwrite Utility Function .................................................................. 137

Developing Metaweb-Enabled Web Applicationsiv



6.2.4. Example: US State Quarters .................................................................. 1386.2.5. Sending Multiple Queries to mqlwrite ................................................... 140

6.3. Uploading Data to Metaweb ............................................................................ 1406.3.1. An Upload Utility ................................................................................. 1416.3.2. Examples: Uploading Images of State Quarters ...................................... 1426.3.3. Uploading Documents .......................................................................... 143

6.4. Example: Unlinking Objects ............................................................................ 144A. Additional Code ........................................................................................................ 149

A.1. json.js ........................................................................................................... 149A.2. Client-side MQL Queries through a Proxy ....................................................... 151A.3. Example: Auto-completion with mqlread ......................................................... 153

vDeveloping Metaweb-Enabled Web Applications



vi



List of Examples3.1. qedit.html: Code for a Metaweb query editor .............................................................. 234.1. albumlist.pl: submitting MQL queries in Perl ............................................................. 644.2. albumlist2.pl: a better Perl album lister ...................................................................... 654.3. metaweb.py: using mqlread with Python .................................................................... 704.4. albumlist.py: listing albums in Python ........................................................................ 714.5. metaweb.php: using mqlread with PHP ...................................................................... 724.6. albumlist.php: A Metaweb-enabled web application in PHP ........................................ 744.7. albumlist.html: a JavaScript album and track lister ...................................................... 764.8. metaweb.js: Metaweb queries with script tags ............................................................. 804.9. metaweb.py: querying Metaweb with a cursor, in Python ............................................. 834.10. WhatsNew.html: fetching new images and documents from freebase.com ................... 874.11. TypeBrowser.html: a Metaweb type browser ............................................................. 906.1. metaweb.py: logging in to Metaweb with Python ...................................................... 1346.2. metaweb.py: sending a query to mqlwrite ................................................................. 1376.3. quarters.py: writing a data set to Metaweb ................................................................ 139

6.4. metaweb.py: uploading content to Metaweb .............................................................. 1416.5. quarterpix.py: uploading images to Metaweb ............................................................ 1426.6. uploaddoc.py: uploading HTML documents to Metaweb ........................................... 1436.7. unlink.py: unlinking Metaweb objects ...................................................................... 144A.1. json.js: JSON parsing and serialization in JavaScript ................................................ 149A.2. metaweb_proxy.js: Metaweb queries through a proxy ............................................... 151A.3. mqlread.php: a trivial mqlread proxy in PHP ............................................................ 153A.4. completiontest.html: using Metaweb.addValidationAndCompletion() ........................ 153A.5. validateAndComplete.js: form validation and completion with mqlread ..................... 154

vii



Chapter 1. IntroductionFreebase is a vast, free, open online database of structured knowledge, powered and maintainedby Metaweb Technologies ( metaweb.com ). Users can access and contribute to Freebase at ht-tp://www.freebase.com , or through the Metaweb API explained in this manual. If you visit the

freebase.com website, you'll nd that Metaweb has seeded the database with detailed informationabout popular music and movies. Figure 1.1 is a sample page from this site:

Figure 1.1. Browsing knowledge at freebase.com

This manual teaches you how to write Metaweb-enabled programs that interact with Freebase.It assumes that you already know the "what" and "why" of Freebase, and that you have read thedocumentation topics (such as "What is Freebase" and "Freebase Demo") linked from the Freebasehome page http://www.freebase.com/view/ .

Metaweb and Freebase

Metaweb (the company) has developed Metaweb (the technology and API). Freebase (theopen global structured knowledge base) is a high-pro le public instantiation of the Metawebtechnology, but is unlikely to be the only instantiation.

This manual documents general Metaweb services and APIs, and relies on Freebase forexample data and example applications. The services and APIs are applicable beyondFreebase, however, and you'll nd that this manual uses the name "Metaweb" far morethan it does the name "Freebase".

1.1.The Metaweb Query APIMetaweb offers a powerful API for making programmatic queries. This allows you to incorporateknowledge from the Freebase database into your own applications and websites. Let's take thisAPI for a spin. Type the following URL into your web browser's location bar. (Type it all on asingle line: it is broken across two lines here to t on the printed page.)

1



http://www.freebase.com/api/service/mqlread?queries={"albums":{"query":{"type":"/music/artist","name":"The Police","album":[]}}}

There are a lot of braces, quote marks, colons and commas in that URL, but remember that thisis a programmatic API: the query is supposed to be generated by a computer, not pecked out byhuman ngers! Translated into English, this query says:

Find an object in the database whose type is "/music/artist" and whose name is"The Police". Then return its array of albums.

If you got all the punctuation correct, the Metaweb server will respond to this query with a responseof MIME type application/json . The response is plain text, but your browser will probablynot display it to you. Instead, the browser will allow you to save it to a le, which you can thenview from the command line or with any text editor. When you view it, you'll see something likethis:

{"status": "200 OK",

"query": {"album": [],"type": "/music/artist","name": "The Police"

},"messages": [],"result": {

"album": ["Outlandos d'Amour","Reggatta de Blanc","Live in Boston","Zenyatta Mondatta",

"Ghost in the Machine","Synchronicity",],"type": "/music/artist","name": "The Police"

}}

Cookie-Based Authentication

While Metaweb is rolling out and scaling up the freebase.com website, queries like the oneshown above can only be made by users who have registered for a Freebase account.

Metaweb uses Cookie based authentication. This means that if you have logged on atwww.freebase.com , your web browser will have the cookies it needs for the query URLabove to work. But if you enter that URL on a web browser that has never visited Freebasebefore, you'll get a HTTP "401 Unauthorized" error.

Once freebase.com is fully deployed, read queries like this will be open to the world andno cookies will be required.

Developing Metaweb-Enabled Web Applications2



The response has the same braces and quotes that the query did: they provide the structure thatmakes this response easy to parse (for a computer). This response begins with an HTTP statuscode. It repeats the query we made, and then provides the response to our query. Our query in-cluded the text:

"album":[]

In the response, those empty square brackets have been lled in with a long list of album names.(For brevity, a number of live and compilation albums were omitted from the list shown above.)

Making queries from your web browser's location bar is interesting, but it becomes much moreinteresting if we make the queries under programmatic control. Imagine a script running on yourown web server that sends queries to Freebase and formats the results as HTML: this is a Metaweb-enabled web application. It might look like Figure 1.2 .

Figure 1.2. A Metaweb-enabled web application

1.2. About this ManualThe goal of this manual is to explain everything you need to know to create Metaweb-enabledweb applications like the one pictured in Figure 1.2 . We assume that you are a programmer whohas some experience with languages like PHP, Python, JavaScript and Java, and that you under-stand the basics of the HTTP protocol.

The chapters that follow are:

Chapter 2: Metaweb ArchitectureThis chapter explains the Metaweb architecture, including a discussion of Metaweb objects,values, types, domains, namespaces and access control.

3Chapter 1. Introduction



Chapter 3: The Metaweb Query LanguageThis chapter explains the Metaweb Query Language (MQL) that is used to express Metawebqueries. The syntax is quite a bit more powerful and complex than what was shown in thisintroduction.

Chapter 4: Metaweb Read ServicesThis chapter explains how to read data from Metaweb servers. It demonstrates (with examplesin a variety of scripting languages) how to submit MQL queries to the mqlread service, andhow to interpret the response. It also demonstrates how to use the trans service to downloadcontent (such as HTML articles or binary image data) from Metaweb.

This chapter includes the source code for the application that created Figure 1.2 .

Chapter 5: The MQL Write Grammar This chapter explains MQL syntax for writing to the Metaweb database. The MQL writegrammar resembles MQL for read queries, but is different in a number of important ways.

Chapter 6: Metaweb Write Services

This chapter explains how to write to Metaweb servers. It demonstrates (with Python code)the mqlwrite service for sending write queries to Metaweb, and the upload service for upload-ing textual or binary content. In addition, it shows how to use the login service, which is anecessary prerequisite for mqlwrite and upload .




Chapter 2. Metaweb ArchitectureA Metaweb database is a sea of knowledge organized as a graph: a set of nodes and a set of linksor relationships between those nodes. This chapter covers the key features of the Metaweb archi-tecture, and explains how Metaweb types and properties tame this vast graph of knowledge byde ning a manageable object-oriented view of it.

2.1.The Metaweb Object ModelFigure 2.1 illustrates one tiny (hypothetical) piece of the Metaweb graph of nodes and relationships.

Figure 2.1. Nodes and relationships

This portion of the Metaweb graph organizes knowledge about something named "Arnold". Ittells us that Arnold is a Person, Politician, Body Builder, and Actor. It tells us that Arnold'scountry of birth is Austria, his political party is Republican, and that he acted in something named"Terminator" (which is an instance of something known as a "Film"). The relationships in thegraph are bi-directional, so this gure also tells us, for example, that Austria has Arnold as acitizen, the Republican Party has Arnold as a member, and that Terminator has Arnold as a castmember. (Note that this is an example only. An "Arnold Schwarzenegger" node does exist atwww.freebase.com , but it may nor may not have the particular relationships pictured here.

This nodes-and-relationships representation of knowledge is ideal for searching algorithms, butis not ideal for human understanding: we quickly become lost in the maze of links. In order tomake the data structure more understandable to humans, Metaweb allows us to view the graphthrough an object-oriented lens. Rather than thinking about nodes and their relationships to othernodes, this object-oriented view lets us think about objects and their properties:

5



Arnoldsex: malebirthdate: 1947-July-30country of birth: Austriapolitical party: Republicanfilm: Conan the Barbarian

film: Terminatorfilm: Kindergarten Copelected office: Governor of California

In this view, Arnold is an object with a set of properties. Each property has a name and a value.What is missing from the view is any kind of typing. In many object-oriented systems, eachproperty of an object has a known type, and the value of that property must be a member of thattype.

Look back at Figure 2.1 again, and consider the relationships labeled type and instances . Arnoldis an instance of Person, Actor, and Politician. Person, Actor, and Politician are Metaweb types:they are nodes in the Metaweb graph, but they also impose an object-oriented structure on the

graph. Each type de nes a set of properties that its instances are expected to have. Each propertyhas a name and a type. An object in a Metaweb database, therefore is a node in the graph, plusthe type that it should be viewed as:

Arnold as Person Arnold as PoliticianSex sex: male ElectedOffice office: Governor of CADate birthdate: 1947-July-30Country birthplace: Austria

Next, let's consider Arnold as an Actor. Notice that the list of properties above included threeproperties named film . This is perfectly ne for a nodes-and-relationships model, but it doesn't

t our object-oriented model where we expect each property to have a single value. A Metaweb

type may specify whether each of its properties must be unique or not. For the Actor type, we'dneed a non-unique property named film . The type of this property is a set of lms that Arnoldhas acted in:

Arnold as ActorSet of Film film: [Conan the Barbarian,

Kindergarten Cop,Terminator]

Note that the film property is an unordered set of values, not an ordered list of values. If youwanted to display this set of lms to an end user, you would most likely want to arrange theminto alphabetical order, or by release date. You can ask Metaweb to order them for you, or you

can sort them yourself. Some sets, such as the set of tracks on an album have an implicit order,and you can ask Metaweb to return the members of the set in this order. We'll see how to do thisin Chapter 3 .




2.1.1. Common Object PropertiesAll Metaweb objects, regardless of their type or types, de ne the following properties:

name This property is a set of human-readable names for the object, suitable for display

to the end users of Metaweb. Each name is a /type/text value which holds astring and de nes the human language in which it is written. The name propertyis special in two ways:

• An object may have more than one name, but may only have one name perlanguage. That is, it can have only one English name, only one French name,and so on.

• When querying Metaweb, you may treat the name property as if it was a single/type/text value rather than a set of values. Metaweb will automaticallyreturn the object's name (if it has one) in your language of choice.

key This property is a set of fully-quali ed names for the object. These fully-quali-ed names are intended for use by developers and scripts and are not typicallydisplayed to end users. Each member of the set is a /type/key value that spe-ci es a namespace object and a name within the namespace. Metaweb guaranteesthat no two objects will ever have the same fully-quali ed name.

guid Every object in a Metaweb database has a globally unique identi f er or guid .The guid property speci es the unique identi er for an object. A guid is a longstring of hexadecimal digits following the hash character, and might look likethis: #0801010a40005e838000000000019bd2 . No two objects will ever have thesame value of the guid property. This property is read-only.

id The id property is used to uniquely identify an object either by its guid, or bya fully-quali ed name de ned by a key property. If you query the id propertyof an object, Metaweb usually returns the guid of the object to you. For objectsthat are instances of core types, Metaweb returns a fully-quali ed name (suchas /type/text or /lang/en ) instead of a guid. Like guid , the id property isunique: no two objects will ever have the same value for this property. Thisproperty is read-only. You may not set the id property. (If you add a key propertyto an object, however, you will be able to refer to the object through a newfully-quali ed name.)

type This property is the set of types associated with the object. The object can beviewed as an instance of any of these types. Each type is itself a Metaweb object,

of /type/type .

timestamp This read-only property is a single value of /type/datetime that speci es whenthe object was created.

creator This read-only property is a single link to a /type/user object that speci eswhich Metaweb user created the object.

7Chapter 2. Metaweb Architecture



permission This read-only property is a single link to a /type/permission object. A per-mission object speci es which Metaweb usergroups are allowed to alter theobject. See Section 2.6 for more on users, usergroups and permissions.

2.1.2. Names, Keys, and IdsNotice that four of the eight common properties described above have to do with names andidenti ers for Metaweb objects. It is important to understand the difference between human-readable names, fully-quali ed names, and guids.

A Metaweb database contains an object that represents the human language English. The nameproperty of this object speci es its human-readable name: "English". Metaweb objects can haveonly a single name in each language. Our English object might have names "Anglais" and "Ingles"in French and Spanish. It is important to understand that the human-readable name of an objectdoes not uniquely identify it: there may be many other Metaweb objects with the name "English".

Because the name property allows only one name in each language, you cannot use it to specify

nicknames for an object. You cannot, for example, give the English object the name "AmericanEnglish" in addition to "English". As we'll see below, most Metaweb objects that are intendedfor display to end-users are instances of a type called /common/topic . This type de nes a propertynamed alias , which you can use to specify any number of nicknames for an object.

The key property of the English object is completely different than the name property. It speci esthat the object has the name "en" in a particular namespace object. That namespace object has akey property of its own, which speci es that it has the name "lang" in a special root namespaceobject. Metaweb uses the slash character to delimit names, so the English object has the fully-quali ed name "/lang/en". Fully-quali ed names are intended for developers and are often usedin code, so we'll usually write them in code font like this: /lang/en .

The critical thing about fully-quali ed names is that they are unique. Metaweb ensures that notwo objects ever have the same fully-quali ed name at the same time.

Human-readable names and fully-quali ed names are optional; Metaweb objects are not requiredto have either. But every object does have a guid value that identi es it uniquely. A unique guidis assigned to an object when it is created, and it never changes. It is always possible to uniquelyidentify an object by specifying the value of its guid property. The guid of the /lang/en objectis "#9202a8c04000641f8000000000000092"

Guids and fully-quali ed names are both unique identi ers for objects. The id property is f exibleand allows you to use either one. If you want to refer to the English object, you could specify anid property of "#9202a8c04000641f8000000000000092" or "/lang/en".

2.1.3.TopicsObjects that are displayed to users of freebase.com are called topics . These are regular Metawebobjects that are members of the type /common/topic in addition to any of their other, more-spe-ci c types. /common/topic de nes properties that allow descriptions, nicknames, documents andimages to be associated with an object, and the freebase.com client uses these properties to as-semble an informative web page that describes the object or topic.




All topics in Metaweb are also objects. But not all objects are topics. The distinction is that topicsare entries that might be of interest to end users. Objects that are not topics are typically part of the Metaweb infrastructure, and may be of interest to Metaweb developers but not end users.Types, properties, domains and namespaces are not topics, but albums, movies, and restaurantsare.

2.2. ValuesLike many object-oriented programming languages, Metaweb draws a distinction between objects(arbitrary collections of properties) and values (single primitives such as numbers, dates andstrings). Metaweb de nes nine value types. Like all Metaweb types, value types are identi edby type objects. Each type object has a fully-quali ed name such as /type/int (for the valuetype that represents integer values).

Values have a dual nature in Metaweb. Depending on how you ask about them, they may behavelike primitives, or like simple objects. If you query a value as if it were an object, then it behaveslike a simple object with two properties (as we'll see shortly, two of the value types actually includea third property as well):

value this property holds the primitive value

type this property refers to the type object that speci es the type of the value.

If you query a value as a primitive, then just the value of the value property is returned.

The various Metaweb value types are described in the sub-sections that follow. Notice that valuetypes are in the /type domain, and that their names fall under the /type namespace. (We'll seemore about namespaces in Section 2.5 .)

2.2.1. /type/intValues of this type are signed integers. Metaweb uses a 64-bit representation internally, whichmeans that the range of valid values of /type/int is from -9223372036854775808 to9223372036854775807. An integer literal is simply an optional minus sign followed by a sequenceof decimal digits. Metaweb does not support octal or hexadecimal notation for integers, nor doesit allow the use of exponential notation for expressing integers.

2.2.2. /type/floatValues of this type are signed numbers that may include an integer part, a fractional part, and anorder of magnitude (a power of ten by which the integer and fractional parts are multiplied.)Metaweb uses the 64-bit IEEE-754 f oating point representation which supports magnitudesbetween 10 -324 and 10 308 . C and Java programmers may recognize this as the double datatype.Metaweb does not support the special values In nity and NaN, however.

A literal of /type/float consists of an optional minus sign, and optional integer part, and optionaldecimal point and fractional part and an optional exponent. The integer and fractional parts are




simply strings of decimal digits. The exponent begins with the letter e or E, followed by an op-tional minus sign, and one to three digits. The following are all valid /type/float literals:

1.0 # integer and fractional part1 # integer part alone.0 # fractional part alone

-1 # minus sign allowed as first character1E-5 # exponent: 1 × 10 -5 or 0.000015.98e24 # weight of earth in kg: 5.98 × 10 24

There are an in nite number of real numbers, and a 64-bit representation can only describe a nitesubset of them. Any number with 12 or fewer signi cant digits can be stored and retrieved exactlywith no loss of precision. Numbers with more than 12 signi cant digits may have those digitstruncated when they are stored in Metaweb.

2.2.3. /type/booleanThere are only two values for this type; they represent the boolean truth values true and false .Note that Metaweb sometimes uses the absence of a value ( null ) in place of false .

2.2.4. /type/idValues of this type are object identi ers, either guids or fully-quali ed names. The object prop-erties guid and id have values of this type.

2.2.5. /type/textAn instance of /type/text is a string of text plus a value that speci es the human language of

that text. The name property of an object is a set of values of this type./type/text is unusual. Its value property speci es the text itself, but it also has a lang propertythat speci es the language in which the text is written. The lang property refers to an object of type /type/lang . The /lang namespace holds many instances of this type, such as /lang/en forEnglish. We'll say more about /type/lang and the /lang namespace later in this chapter.

The text of a /type/text value must be a string of Unicode characters, encoded using the UTF-8 encoding. The encoded string must not occupy more than 4096 bytes. Longer chunks of text(or binary data) can be stored in Metaweb in the form of a /type/content object, which is de-scribed later.

2.2.6. /type/keyInstances of /type/key represent a fully-quali ed name. The key property of an object is a setof /type/key values. The value property of a /type/key value is the local, or unquali ed partof a fully-quali ed name. Like /type/text , /type/key has a third property. The namespaceproperty of a key refers to the /type/namespace object that quali es the local name. Thenamespace property and the value property combine to produce a fully-quali ed name.




As an example, consider the Metaweb object that represents the value type /type/int . The keyproperty of this object has a value of "int", and a namespace that refers to the /type namespace.The /type namespace is also an object, and its key property has a value of "type" and a namespacethat refers to the root namespace object.

The value property of a key must be a string of ASCII characters, and may include letters,numbers, underscores, hyphens and dollar signs. A key may not begin or end with a hyphen orunderscore. The dollar sign is special: it must be followed by four hexadecimal digits (using lettersA through F, in uppercase), and is used when it is necessary to map Unicode characters intoASCII so that they can be represented in a key. To represent an extended Unicode character (thatdoes not t in four hexadecimal digits), encode that character in UTF-16 using a surrogate pair,and then express the surrogate pair using two dollar-sign escapes.

Keys used as names for domains, types and properties are further restricted: they may not includehyphens or dollar signs, and may not include two underscores in a row.

2.2.7. /type/rawstringA value of /type/rawstring is a string of bytes with no associated language speci cation. Thelength of the string must not exceed 4096 bytes.

Use /type/rawstring instead of /type/text for small amounts of binary data and for textualstrings that are not intended to be human readable.

2.2.8. /type/uriAn instance of /type/uri represents a URI (Uniform Resource Identi er: see RFC 3986). Thevalue property holds the URI text, which should consist entirely of ASCII characters. Any non-ASCII characters, and any characters that are not allowed in URIs should be URI-encoded usinghexadecimal escapes of the form %XXto represent arbitrary bytes.

2.2.9. /type/datetimeAn instance of /type/datetime represents an instant in time. That instant may be as long as ayear or as short as a fraction of a second. The value property is a a string representation of a dateand time formatted according to a subset of the ISO 8601 standard. /type/datetime only supportsdates speci ed using month and day of month. It does not support the ISO 8601 day-of-year,week-of-year and day-of-week representations.

A /type/datetime value that represents the rst millisecond of the 21st century looks like this:

2001-01-01T00:00:00.001Z

Notice the following points about this format:

• Longer intervals of time (years, months, etc.) are speci ed before shorter intervals (minutes,seconds, etc.).




• Years must be speci ed with a full four digits, even when the leading digits are zeros. Negativeyears are allowed, but years with more than four digits are not allowed.

• Months and days must always be speci ed with two digits, starting with 01, even when therst digit is a 0.

• The components of a date are separated from each other with hyphens.

• A date is separated from the time that follows with a capital letter T.

• Times are speci ed using a 24-hour clock. Midnight is hour 00, not hour 24. Hours and minutesmust be speci ed with two digits, even when the rst digit is 0.

• Seconds must be speci ed with two digits, but may also include a decimal point and a fractionalsecond. Metaweb allows up to 9 digits after the decimal point.

• The hours, minutes, and seconds components of a time speci cation are separated from eachother with colons.

• A time may be followed by a timezone speci cation. The capital letter Z is special: it speci esthat the time is in Universal Time, or UTC (formerly known as GMT). Local timezones thatare later than UTC (east of the Greenwich meridian) are expressed as a positive offset of hoursand minutes such as +05:30 for India. Local times earlier than UTC are expressed with a neg-ative offset such as -08:00 for US Paci c time. If no timezone is speci ed, then then the/type/datetime value is assumed to be a local time in an unknown timezone. Specifying atimezone of +00:00 is the same as specifying Z. Specifying -00:00 is the same as omitting thetimezone altogether.

• All characters used in the /type/datetime representation are from the ASCII character set,so date and time values can be treated as strings of 8-bit ASCII characters.

A /type/datetime value can represent time at various granularities, and any of the date or timeelds on the right-hand side can be omitted to produce a value with a larger granularity. For ex-

ample, the seconds eld can be omitted to specify a day, hour, and minute. Or all the time eldsand the day-of-month eld can be omitted to specify just a year and a month. Also, the date eldscan be omitted to specify a time that is independent of date. A timezone may not be appended toa date alone: there must be at least an hour eld speci ed before a timezone.

Here are some example /type/datetime values that demonstrate the allowed formats:

2001 # The year 20012001-01 # January 2001

2001-01-01 # January 1st 20012001-01-01T01Z # 1 hour past midnight (UTC), January 1st 20012000-12-31T23:59Z # 1 minute before midnight (UTC) December 31st, 20002000-12-31T23:59:59Z # 1 second before midnight (UTC) December 31st, 20002000-12-31T23:59:59.9Z # .1 second before midnight (UTC) December 31st, 200000:00:00Z # Midnight, UTC12:15 # Quarter past noon, local time17-05:00 # Happy hour, Boston (US Eastern Standard Time)




2.3. TypesTypes that are not value types are object types . Metaweb pre-de nes a number of object types,organized into domains of related types. Metaweb users are allowed (and encouraged) to de nenew object types as needed. Pre-de ned object types can be categorized into the core types thatare part of the Metaweb infrastructure, common types that are used commonly throughout Metaweb,and domain-speci c types. The core types are all part of the /type domain (which they sharewith the value types), and the common types are all part of the /common domain. freebase.comde nes many domain-speci c types, such as the music-related types /music/artist , /music/albumand /music/track . Figure 2.2 illustrates these type categories.

+--/type/id|+--/type/int|+--/type/float|+--/type/boolean|

+--Value Types--+--/type/text| || +--/type/rawstring| | +--/restaurant domain| +--/type/uri || | +--/location domain| +--/type/datetime || | +--/film domain +-/music/track

Types-+ +--/type/key | || +--/music domain--+-/music/album

| | || +--Freebase Types-----+--/book domain +-/music/artist| | || | +--etc.| |+--Object Types-+--Core Types (/type domain)

|+--Common Types (/common domain)|+--User-defined Types

Figure 2.2. Categories of Metaweb types

The sub-sections that follow introduce the most important core and common types. You do notneed to understand these types in detail in order to make productive use of Metaweb. Still,knowing what these basic types are is a helpful orientation to the system.




and fully-quali ed names for the property. In addition, properties speci c to /type/propertyspecify:

• The expected type of the value of the property

• Whether the property is unique . A unique property may only have a single value (or may have

no value). A property that is not unique has a set of zero or more values.

• The reciprocal property, if there is one.

• The type of which this property is a part.

The notion of a reciprocal property deserves more explanation. Recall that all links in Metawebare bi-directional. This means that any time a property of type A refers to an object of type BMetaweb automatically has a link from that object of type B back to the originating object of type A. Type B can take advantage of this bi-directionality and include a property that links back to objects of type A. As a concrete example, consider the properties property of /type/type :it speci es the set of properties for a type. Its reciprocal is the schema property of /type/property ,

which speci es the type object (or "schema") of which the property is a part. You'll nd furtherexploration of reciprocal properties in Chapter 5 .

2.3.1.4. /type/domain

A domain represents a set of related types, and also serves as a namespace for those types. Foraccess control purposes, each domain object refers to one or more usergroup objects that "own"the domain. Only members of the speci ed usergroups are allowed to add new types to the domainor to edit types within the domain.

2.3.1.5. /type/namespace

This type represents a namespace, and is used by the value type /type/key . It de nes the keysproperty which is a set of /type/key values that specify the names in the namespace.

2.3.2. Content TypesThe following types from the /type and /common domains are important content-related types:

/type/contentLarge chunks of content, such as HTML documents and graphical images are not stored inregular Metaweb nodes. Instead, these large objects (sometimes called lobs ) are kept in aseparate store. A /type/content object is the bridge between the Metaweb object store andthe Metaweb content store. A /type/content object represents an entry in the content store,and the guid of the /type/content object is used as an index for retrieving the content.

In addition to providing access to the content store, /type/content de nes important prop-erties. The media_type property speci es the MIME type of the content. For textual content,the text_encoding and language properties specify the encoding and language of the text.The length property speci es the size (in bytes) of the content. The source property refersto a /type/content_import object that speci es the source of the content.




Chapter 4 shows how to download content from Metaweb, and Chapter 6 demonstrates howto upload content.

/type/content_importThis type describes the source of imported content. Its properties include the URI or lenamefrom which the content was obtained, the user who imported the content, and a timestampthat speci es when the content was imported.

/type/media_typeInstances of this type represent a MIME media type such as "text/html" or "image/png". In-stances are given fully-quali ed names within the /media_type namespace, and can be spe-ci ed with ids like /media_type/text/html or /media_type/image/png .

/type/text_encodingInstances of this type represent standard text encodings, such as ASCII and Unicode UTF-8.Instances are given fully-quali ed names within the /media_type/text_encoding namespace,and can be speci ed with ids such as /media_type/text_encoding/ascii .

/type/langThis type represents a human language. It is used by /type/content objects and also by

/type/text values. Pre-de ned instances of this type are given fully-quali ed names withinthe /lang namespace, and can be speci ed with ids like /lang/en and /lang/fr .

/common/topicAs described earlier in this chapter, Metaweb objects that are intended for display to endusers are called "topics". Such objects typically have some appropriate domain-speci c type,such as /music/artist or /food/restaurant , but are also instances of the type /common/top-ic . This type de nes properties that allow documents and images to be associated with thetopic. Another property allows a set of URLs to be associated with the topic. Also, becauseobjects can only have a single name in any given language, /common/topic has an aliasproperty that allows any number of nicknames to be speci ed for the topic.

/common/documentThis type represents a document of some sort. /common/topic uses this type to associatedocuments with topics. The most important property is content , which speci es the single/type/content object that refers to the document content. Other properties of /common/doc-ument provide meta-information about the document, such as authors, publication date, andso on.

/common/image/type/content objects that represent images are typically co-typed with this type. /common/im-age de nes a size property that speci es the pixel dimensions of the image.

2.3.3. Access Control TypesThe following types are part of the Metaweb access control framework.




/type/userEach registered Metaweb user is represented with an object of /type/user . User objects havefully-quali ed names in the /user namespace. If your username is joe_developer , then your/type/user object is /user/joe_developer .

/type/usergroup

This type represents a set of users.

/type/permissionThis type is the key to Metaweb access control. Its properties specify the set of objects thatrequire this permission for modi cations, and also the set of usergroups that have the permis-sion. See Section 2.6 for further details.

2.4. DomainsA domain is a Metaweb object of /type/domain . It represents a collection of related types. We'vealready seen a number of types from the /type and /common domains. freebase.com pre-de nestypes in a number of general domains, and Chapter 3 and Chapter 4 feature many examples usingthe Freebase /music domain. The set of Freebase domains is expected to grow, but at the timeof this writing, it includes:

/business /food /measurement_unit/education /language /music/film /location

As you might guess from the names of these domains, domain objects are also instances of /type/namespace , and the types contained by domains are members of both the domain and thenamespace.

Every Metaweb user who registers for an account has their own domain. If your Metaweb usernameis fred, then your domain is /user/fred/default_domain . When you use the freebase.com clientto de ne a new type named Beer, it is given the id /user/fred/default_domain/beer . If yourtype becomes an important and commonly used one, it may be promoted by Metaweb adminis-trators to a top-level domain. In this case, your type might be given a new fully-quali ed namelike /zymurgy/beer .

2.5. NamespacesNamespaces are a critical part of the Metaweb infrastructure because they allow us to refer toimportant objects, such as types, with simple mnemonic names rather than opaque guids. It would

be very inconvenient to query Metaweb i f we had to wr i te"#9202a8c04000641f8000000000000565" instead of "/common/topic", for example.

We've already learned about a number of important namespaces including /type , /user , /lang ,and /media_type . In addition to these, each domain and user object is also a namespace. Also,there is the root namespace, whose id is simply / .




A number of important namespaces are populated with pre-de ned objects using names de nedby international standards. The languages in the /lang namespace use language codes (such as"en" for English and "fr" for French) de ned by ISO 639. The media types in /media_type arede ned by IANA and listed at http://www.iana.org/assignments/media-types/ . And the text en-codings in /media_type/text_encoding use names de ned by IANA at http://www.iana.org/as-signments/character-sets .

2.6. Access ControlMetaweb is completely open for reading. Anyone who can connect to Metaweb servers can readdata from them. When adding or editing data, however access control comes into play. We'vealready seen that the types /type/user , /type/usergroup , and /type/permission are used foraccess control.

Metaweb's access control model is quite simple. Every object has a permission property thatrefers to a /type/permission object. The permission object speci es a set of usergroups whosemembers have permission to modify the object. If a user is a member of one or more of the spe-ci ed groups, then that user can edit the object. Otherwise, the user is not allowed to.

This simple access control model is, by default, also very open. In order to allow and encouragefree collaboration most Metaweb objects have a permission object that gives edit permission toall Metaweb users. If Metaweb user Fred creates a new object, his friend Jill can freely edit thatobject. Any other Metaweb user can edit the object as well, and there is no way for Fred to restrictthe permission on his object.

The primary exception to this open access control model is type objects. Having a stable typesystem is very important. Each domain has a usergroup associated with it, and only members of that usergroup can create new types in the domain or alter existing types in the domain. Eachuser account has an associated domain. Fred's domain is /user/fred/default_domain . Thisdomain has an associated usergroup. Initially, Fred is the only member of this group. He is allowedto add to the usergroup, and if he adds his friend Jill, then she is permitted to create new typesin Fred's domain.

Other key parts of the Metaweb infrastructure also have restrictive access control, of course.Ordinary users are not allowed to insert objects into the /lang namespace or the /type domain,for example.




Chapter 3.The Metaweb Query LanguageThis chapter explains the Metaweb Query Language, or MQL, which is used to express Metawebqueries. This chapter begins and ends with formal rules of MQL syntax, but the middle is an ex-tended tutorial that teaches MQL by example. You are expected and encouraged to run queriesand to experiment with your own queries, using a "query editor" program that submits yourqueries to Metaweb and displays the results.

This chapter teaches you to write MQL queries, but does not explain how to issue those queriesto and retrieve responses from Metaweb servers : that is the topic of Chapter 4 . Also, this chapterdoes not cover updates, or writes, to Metaweb. Updates are expressed using a variant of MQLthat is covered in Chapter 5 .

3.1. JavaScript Object NotationThe Metaweb queries and responses we saw in Chapter 1 contained a lot of punctuation: curlybraces, quotation marks, colons, and commas. Before we study more queries, it is important tounderstand this punctuation. Metaweb queries and responses use a plain-text data interchangeformat known as JavaScript Object Notation or, more commonly, JSON. If you are a JavaScriptprogrammer, then this format will be familiar to you since it is a subset of the JavaScript language.1 If you are not a JavaScript programmer, the format is easy-to-learn, and does not require theuse of the JavaScript language.

JSON is formally described in RFC 4627 ( http://www.ietf.org/rfc/rfc4627.txt ), and is also docu-mented at http://json.org . The JSON website includes pointers to code, in a variety of programminglanguages, for serializing data structures into JSON format and for parsing JSON text into datastructures. 2

Figure 3.1. JSON Values

1 You should read this section even if you already know JavaScript. JSON is only a subset of JavaScript, and its syntax is stricter thanJavaScript syntax.2 The JSON syntax diagrams that appear below are also from the JSON website, where they have been placed in the public domain.

19



A JSON-formatted string is a serialized form of an array or object. The array or object may containnumbers, strings, other arrays and objects, and the literal values null , true , and false . TheseJSON values are illustrated in Figure 3.1 and explained in the sub-sections that follow:

3.1.1. JSON Literals: null, true, falseJSON supports three literal values. null is a JSON value representing "no value". The literalstrue and false a represent the two possible Boolean values.

3.1.2. JSON NumbersA JSON number consists of an optional minus sign followed by an integer part followed by anoptional decimal point and fractional part followed by an optional exponent. This format is thesame as the format described for /type/float in Chapter 2 . All numbers use decimal digits: octaland hexadecimal notation are not supported.

3.1.3. JSON StringsA JSON string is much like a string in Java or JavaScript: zero or more Unicode characters 3

between double quotation marks. See Figure 3.2 .

Figure 3.2. JSON string syntax

3 JSON itself supports 32-bit, 16-bit and 8-bit encodings of Unicode text. Metaweb, however, requires the 8-bit UTF-8 encoding.




A backslash is special: it is an escape character and is interpreted along with the character orcharacters that follow:

CharacterEscape

A quotation mark that does not terminate the string\"

A single backslash character that is not an escape\\A forward slash character. Although it is legal to escape the forward slash character, itis never necessary to do so.

\/

The Backspace character\b

The Formfeed character\f

The Newline character\n

The Carriage Return character\r

The Tab character\t

The Unicode character whose encoding is the four hexadecimal digits XXXX . To encode

extended Unicode codepoints that do not t in four hex digits, use two \u escapes toencode a UTF-16 surrogate pair.

\u XXXX

3.1.4. JSON ArraysAn array is a comma-separated list of JSON values enclosed in square brackets. See Figure 3.3

Arrays may contain any JSON values, including objects and other arrays. The elements of a JSONarray need not have the same type (though in MQL they always do). The following JSON arraymight be returned in response to a MQL query:

["Outlandos d'Amour", "Reggatta de Blanc", "Zenyatta Mondatta"]

Figure 3.3. JSON array syntax

A JSON array with no elements consists of just the square brackets: [] . Empty arrays often appearin MQL queries.

21Chapter 3.The Metaweb Query Language

3.2. MQL TutorialThis section is a tutorial that teaches Metaweb queries by example, and uses freebase.com as asource of interesting data to query. Try to follow along as you read it by trying out the queriespresented. To do this, you need a simple way to submit a query to freebase.com and view theresult. You can do this with the Freebase query editor at http://www.freebase.com/view/queryed-itor/ , or you can create your own simple query editor: save the code from Example 3.1 to a local

le, and view it in your web browser. Figure 3.5 shows the resulting UI, displaying a simplequery and response. (Remember that during the roll-out period, freebase.com queries requirecookie-based authentication. So the query editor of Example 3.1 will only work in web browsersthat have previously logged on to freebase.com and have authentication credentials stored in acookie.

Figure 3.5. A simple Metaweb query editor

Example 3.1. qedit.html: Code for a Metaweb query editor

<html><head><title>Metaweb Query Editor</title><style> /* CSS styles for nice output */#q, #r { width: 400px; height: 300px; border-width:0px;padding:5px;}th {background-color:black; color:white; font:bold 12pt sans-serif;}td.border,th {border:solid black 3px;}input { margin: 5px; font-weight: bold; }table { border-collapse: collapse;}</style></head><body>

<form action="http://www.freebase.com/api/service/mqlread"method="get"target="r">

<input type="hidden" name="callback" value=" ">

<table><tr><th>Query</th><td></td><th>Result</th></tr> <tr>

<td class="border"><textarea name="queries" id="q">

{"qname": {

"query": [{enter query here

}]}

}</textarea></td>

<td valign="top"><input type="submit" value="Send"><br> <input type="reset" value="Erase"></td> 

<td class="border"><iframe name="r" id="r"></iframe></td> 

</tr></table>

</body></html>

A Note about Query Envelopes

A MQL query is a JSON object. In order to get a Metaweb server to execute the query,however, you must nest it inside two more JSON objects, which are known as the "envelope".We'll explain envelopes in detail in Section 4.2.3 when we're explaining how to send MQLqueries to Metaweb. For now, however, you just need to know enough about envelopes sothat you can try out queries in query editors.

In order to execute a query in the query editor of Example 3.1 , you must put this text beforeyour query:

{"qname":{"query":

The text "qname" is actually an arbitrary name for the query; you can use any name youwant here. The text "query" must be entered verbatim. Whitespace is optional in JSON, soyou can of course insert spaces and newlines into that text to make it look nice. Since theenvelope is a JSON object, you must provide the matching closing braces. This means thatyou must follow your query with:

}}



The Freebase query editor at http://www.freebase.com/view/queryeditor/ is a little easier.When you use it, you omit the outer part of the envelope, and just begin the query with{"query": and end it with } .

Metaweb's response to MQL queries are also JSON objects, and like the queries are wrapped

within envelope objects. If you use the query editor in Example 3.1 , you'll see the completeresponse envelope object, and you'll nd the MQL query result as the value of a propertynamed result in an object that is itself the value of a property named qname (or whateverquery name you choose in the query envelope). If you use the Freebase query editor, youhave the option of viewing just the result or the complete response envelope. In this chapter,we omit the response envelopes except when we need to look at error messages that appearin the envelope.

3.2.1. Our First QueryLet's begin by revisiting the simple query from Chapter 1 . We would like to know what albums

The Police have recorded:

{"type" : "/music/artist","name" : "The Police","album" : []

}

Once we extract the result from its envelope, we're left with the following JSON object (someof the album names are omitted here for brevity):

{"type": "/music/artist","name": "The Police","album": [

"Outlandos d'Amour","Reggatta de Blanc","Zenyatta Mondatta","Ghost in the Machine","Synchronicity",

]}

To query Metaweb we tell it what we already know by specifying properties and their values:

"type" : "/music/artist","name" : "The Police",

And then we tell it what we want to know by specifying properties without values:

"album" : []

Sending an empty array in a MQL query tells Metaweb that we'd like to have the array lled in.




Singular or Plural?

Note that the property we query in the example above is named "album" and not "albums",even though bands like The Police may well have more than one album. This re f ects theunderlying nature of Metaweb: the database object that represents The Police has many

links of type "album" that refer to the objects that represent those albums. The type /mu-sic/artist aggregates these many links into a single set of albums, but retains the singularname "album" because that is the name of the underlying link type.

Also, as we'll see soon when you want to obtain information (such as a list of tracks) aboutone particular album, you specify a single value (instead of an array) for the album property.In this case, the singular name makes a lot of sense.

Although property names are typically singular, there are exceptions to this rule, andsometimes you'll see a plural property name.

3.2.2. Query/Response SymmetryLet's look one more time at the simple "albums by The Police" query and response from above.This time the query and response are presented side-by-side to emphasize that the query and re-sponse objects have the same properties, but the response object has values lled in:

ResultQuery

{"type": "/music/artist","name": "The Police","album": [

"Outlandos d'Amour","Live in Boston","Reggatta de Blanc","Zenyatta Mondatta","Ghost in the Machine","Synchronicity","Every Breath You Take: The Singles","Greatest Hits"

]}


}

This symmetry of queries and responses is a fundamental and elegant part of MQL. We'll use

this two-column query/response format throughout the chapter.

3.2.3. Metaweb Object IDsRecall that all Metaweb objects have a unique identi er in their id property. Here's how we ndthe id for The Police:




ResultQuery

{"type" : "/music/artist",


"name" : "The Police","name" : "The Police","id" : "#1f800000000006df1b",

}"id" : null

}

This query includes the same name and type as the last query. But instead of specifying an emptyarray of albums, it speci es a null id. The null value is our query: this is what we want Metawebto ll in. The response looks just like the query, but the null is replaced with an id string.

Freebase IDs in this Tutorial

The ids shown in the online HTML-formatted version of this tutorial are valid freebase.comguids, and should remain valid in perpetuity. In hardcopy and PDF versions of the tutorial,guids have been shortened to allow queries and responses to t side-by-side in the two-

column format shown above. To use these printed guids online, restore them to validity byinserting 9202a8c0400064 between the # and the digit that follows it.

3.2.4. Multiple Results and Uniqueness ErrorsNow that we know the id of the object, let's turn our query around and ask about name and typeof the object with that id:

{"id": "#1f800000000006df1b","name" : null,

"type" : null}

We're telling Metaweb what we have (the id) and asking for the values (name and type) that wedon't have. When we submit this query, though, it doesn't work. The response envelope lookslike this:

{"status": "200 OK","qname": {

"status": "/mql/status/error","messages": [

{ "status": "/mql/status/result_error","info": {

"count": 2,"result": [

"#1f80000000011ae833","#1f8000000000000565"

]},




"path": "type","query": {

"error_inside": "type","type": null,"id": "#1f800000000006df1b","name": null

},"message": "Unique query may have at most one result. Got 2","type": "/mql/error"

}]

}}

The various status properties tell us that something is wrong with the query. The messages[0]object provides details. Its message property gives us an error message, and its info objectprovides details to go with the message. The query.error_inside and path properties tell usthat the error is associated with the type property in our query.

What we learn from this response is that Metaweb could not respond to our query because weasked for a single type and it found two types. Let's try the query again. Now we're requesting asingle name and an array of types for this uniquely speci ed object. This query works:

ResultQuery

{"id":"#1f800000000006df1b","name" : "The Police","type" : [

"/common/topic","/music/artist"

]}

{"id":"#1f800000000006df1b","name" : null,"type" : []

}

The Metaweb object we asked about has the name "The Police" and it is a member of two types:/common/topic and /music/artist . Recall from Chapter 2 that /common/topic is a very generictype. Just about every Metaweb object that represents something an end user would have an interestin is a member of this type. The lesson to draw here is that objects almost always have more thanone type, and any queries on the type property should use arrays. In general, it is always safe touse [] in place of null in your queries. If there is only one result the array returned in the responsewill simply have a single element. When you know that there can only be one result, however,it is usually more convenient and ef cient to use null .

Single Values and Sets of Values

There is a fundamental asymmetry to MQL: when we query the type of an object, we getan array of types. But when we look up an object by type, we specify only one type. Metawebobjects have a set of types, not one single type. So when we specify the type of an objectin a MQL query, all we are saying is that the object has at least one "type" link with that




value. Thus, writing "type":"/music/artist" in a query does not say "the type is /mu-sic/artist", but "the set of types includes /music/artist". Put another way, we can say that aquery provides constraints, and that the response provides values for the unconstrainedproperties of the query.

Uniqueness errors are a common pitfall for developers crafting Metaweb queries. Recall that/type/property allows certain properties to be speci ed as unique. id is unique: no object canhave more than one id. The name property behaves as if it is unique (but is only unique per lan-guage). As we've seen, however, the type property, is not unique: objects can (and most objectsdo) have more than one type. If a property is not guaranteed to be unique, then you should alwaysuse square brackets when querying its value.

The id property is unique in another way. As we've seen, no object can have more than one id.More importantly, however, no two objects share the same id. Therefore, if a query includes anid , you can be con dent that no more than one object will match. Therefore, a query like thisone is correct:

{"id": "#1f800000000006df1b","name" : null,"type" : []

}

Recall that an object can have only one name in any given language, and that the name propertybehaves like a unique property even though it is not really. For this reason, it is always safe toquery name with null , as we do above, rather than [] .

On the other hand, the query that we started this tutorial with is risky:


}

This query worked for us: Freebase only knows about one musical artist named "The Police".Note, however, that there is no guarantee that this will always be the case. There is nothing toprevent someone from adding another band named "The Police" to freebase.com . If such an ad-dition were made, our query would suddenly fail.

Depending on the design of your application, a uniqueness failure in this situation might actually

be exactly what you want. If you get two results when you expected one, then perhaps the rightthing to do is fail and display an error message to the user. On the other hand, you could writeyour query more cautiously, using square brackets, so that multiple results can be returned:

[{"type" : "/music/artist","name" : "The Police","album" : []

}]




In this case, you should be sure to check the number of results returned and take appropriate action(such as asking the user to choose) if you get more than one.

3.2.5. Nested Queries

Let's nd out more about our favorite band. What are the names of the tracks on the album Syn-chronicity ?

ResultQuery



"name" : "The Police","name" : "The Police","album" : {"album" : {

"name" : "Synchronicity","name" : "Synchronicity","track" : ["track" : []

"Synchronicity I",}} "Walking in Your Footsteps",

"O My God","Mother","Miss Gradenko","Synchronicity II","Every Breath You Take","King of Pain","Wrapped Around Your Finger","Tea in the Sahara","Murder by Numbers"

]}

}

The interesting thing about this query is that it includes a nested query. We're asking for an arrayof tracks from an album named "Synchronicity" recorded by a band named "The Police".

There are other ways to obtain the same information. Here's another query that gets us the samedata:

{"type" : "/music/album","name" : "Synchronicity","artist" : "The Police","track" : []

}

Rather than identifying the band rst, and then querying an album recorded by that band, thisquery goes straight to the album, which it identi es by name and by artist. (It assumes this isenough to uniquely identify a single album and avoid uniqueness errors!)




3.2.6. Asking Metaweb For ObjectsIn our queries so far, we've used null and [] to ask Metaweb to ll in a single value or an arrayof values. There are other ways to ask for information as well. Recall the following query:

{ "id" : "#1f800000000006df1b","name" : null,"type" : []

}

It asks for the name and types of a unique object. Both the name, and the individual elements of the type array are returned as strings. Recall, however, that the name of an object is of /type/textand that types are of /type/type . /type/text is a value type in the Metaweb object model, butwe can treat values as objects if we want to. Let's modify the query to use {} and [{}] insteadof null and [] . {} asks for a single value, expanded as an object, and [{}] asks for an array of values expanded into objects:

{"id": "#1f800000000006df1b","name" : {},"type" : [{}]

}

This query fails with a uniqueness error. The object we're querying has more than one name. Thename property behaves specially when queried with null : it returns the value of the name in thedefault language. It only works to query name with {} if there is only one name, with no transla-tions. To make the query work, we ask for both the name and type with [{}] :

ResultQuery

{"id":"#1f800000000006df1b",

{"id": "#1f800000000006df1b",

"name":[{"name" : [{}],"lang":"/lang/fr","type" : [{}]

} "type":"/type/text","value":"The Police"

},{"lang":"/lang/en","type":"/type/text","value":"The Police"

},{

"lang":"/lang/es","type":"/type/text","value":"The Police"

}],"type":[{

"id":"/music/artist","name":"Musical Artist","type":["/type/type","/freebase/type_profile"]

},{




ResultQuery

"id":"/common/topic","name":"Topic","type":["/type/type","/freebase/type_profile"]

}]}

We learn from this query that the name of the speci ed object is "The Police" in each of severaldifferent languages (some languages have been omitted here). We also learn that the object is of type /common/topic and /music/artist and that these types have common names (as opposedto the formal ids that we use in queries) "Topic" and "Musical artist".

Let's use this query technique to learn more about the tracks on the album Synchronicity. (Theresult is truncated for brevity.)

ResultQuery

{"type": "/music/album",

{"type" : "/music/album",

"name": "Synchronicity","name" : "Synchronicity","artist": "The Police""artist" : "The Police","track": [{"track" : [{}]

} "type": [ "/music/track" ],"name": "Synchronicity I","id": "#1f8000000001275dbb"

},{"type": [ "/music/track" ],"name": "Walking in Your Footsteps","id": "#1f8000000001275dc2"

},{"type": [ "/music/track" ],"name": "O My God","id": "#1f8000000001275dc9"

}]}

This query doesn't actually tell us much about the tracks themselves. We already know the typeof the tracks. The id might be useful in future queries, but it doesn't tell us anything about thetrack. The name is useful, but we could have obtained that without using curly braces, just byquerying "track":[] .

When you ask Metaweb to ll in empty curly braces for you, it returns all the properties if thevalue is a value type. The name property of an object is of /type/text , and querying it with {}returns all of its properties. If the property is an object type instead of a value type, then Metawebreturns only the name , type and id properties (all of which are de ned by /type/object and arecommon to all Metaweb objects). That is, instead of using [{}] , we could write out the queryexplicitly like this:




{"type" : "/music/album","name" : "Synchronicity","artist" : "The Police","track" : [{

"name" : null,

"id" : null,"type" : []

}]}

What if we want to know absolutely everything freebase.com knows about the tracks on Synchron-icity ? We write the query using a wildcard: 4

{"type" : "/music/album","name" : "Synchronicity","artist" : "The Police",

"track" : [{"*":null}]}

"*" is a wildcard property name. It means "all property names". (Note that it is different from []which means "all property values") The type /music/track de nes a number of its own properties,and the expansion of the "*" wildcard also includes the universal properties de ned by/type/object . Here, for example, is what freebase.com knows about the song "Walking in YourFootsteps":

{"name":"Walking in Your Footsteps","type":["/music/track"],"id":"#1f8000000001275dc2","guid":"#1f8000000001275dc2","creator":"/user/mwcl_musicbrainz","key":["a2313ee6-ccce-4ced-bc3c-af7d4b06f09f","TRACK179899"],"permission":"/boot/all_permission","timestamp":"2006-12-10T00:39:58.0931Z","album":["Synchronicity"],"length":[216.8],"lyricist":[],"lyrics":[],"song":[],"artist":null,"composer":[],"acquire_webpage":[]

}

If {} gives us too little useful information, And {"*":null} gives us more than we really need,then we must re ne our query to express exactly what it is we would like to know. Here's howwe ask for just the name and length of each of the tracks:

4 We'll return to the topic of wildcards later in this tutorial.




ResultQuery

{"type": "/music/album",


"name": "Synchronicity","name" : "Synchronicity","artist": "The Police","artist" : "The Police","track": ["track" : [{

{"name":"Synchronicity I", "length":203.533},"name":null,{"name":"Walking in Your Footsteps","length":null

"length":216.8},}]} {"name":"O My God", "length":242.226},

{"name":"Mother", "length":185.6},{"name":"Miss Gradenko", "length":120.0},{"name":"Synchronicity II", "length":305.066},{"name":"Every Breath You Take",

"length":254.066},{"name":"King of Pain", "length":299.066},{"name":"Wrapped Around Your Finger",

"length":313.733},{"name":"Tea in the Sahara", "length":255.44},{"name":"Murder by Numbers", "length":273.693}

]}

3.2.7. Expanded Values and Default PropertiesIn this tutorial we've said that we query the value of a property p with "p":null and "expand"that value into an object with "p":{} . This is helpful terminology, but it is actually the oppositeof what is really going on. Everything in Metaweb is an object (or, in the case of literal values,can be viewed as an object). When you use curly braces, objects are naturally expressed as objects.

When you use null , however, objects are compressed: instead of returning the complete object,Metaweb returns only the value of the object's default property. If the object is of value type, thisdefault property is always the value property and is expressed as a string, number, or booleanliteral. If the object is not an instance of a value type, then the default property is either name orid , both of which are expressed using string literals. Object types in the /type domain use id astheir default property. All others object types use name .

Default properties are not only used when you ask Metaweb to ll in a null or a [] for you. Theyare also used when you express the information you already have. Consider the following query:


"name" : "Synchronicity","artist" : "The Police","track" : []

}

This query could also be expressed more verbosely like this:





"name" : {"value":"Synchronicity", "lang":"/lang/en"},"artist" : {"type":"/music/artist", "name":"The Police"},"track" : []

}

The verbose form of the query illustrates the fact that the succinct form relies on default properties.

The name property is of /type/text , whose default property is value . The artist property is of type /music/artist , whose default property is name .

3.2.8. Review: Asking for ValuesIf you want to ask Metaweb to return a value, use one of the terms listed in Table 3.1 on the right-hand side of a property name:

Table 3.1. Asking for Values

MeaningTerm

If the property is of value type, return the value property. If the property is of object type, return the name or id property. The default_property property of /type/type speci es which property is returned for every object type.

null

Like null , but return an array of values instead of a single value.[]

If the property is of value type, return an object that represents the value. Thisobject will have type and value properties. If the property is /type/text , the

{}

returned object will also have a lang property, and if it is of /type/key , it willhave a namespace property.

If the property is of object type, return an object that includes its name , id , andtype properties. In this case, the term {} is equivalent to:{"name":null,"id":null,"type":[]} .

Like {} , but return an array of objects instead of a single one.[{}]

A query of this form returns an object and all of its properties. The meaning of "all of its properties" requires some explanation, however. Suppose Metaweb

{"*":null}

sees the query "p":{"*":null} . It looks up the property p and determines thatits expected type is t . Then it looks up the type t , and determines what propertiesthat type de nes. Then it expands the wildcard query so that each property of t(plus the properties of /type/object ) is queried with either null or [] dependingon whether it is unique or not.

Like {"*":null} , but return an array of objects instead of a single one.[{"*":null}]

A query of this form is like {"*":null} , except that when the wildcard is expan-

ded, each property is queried with {} or [{}] instead of null or [] .

{"*":{}}

Like {"*":{}} , but return an array of objects instead of a single one.[{"*":{}}]




3.2.9. Too Much InformationSuppose we want to nd the answers to the following questions: Which artists have recordedsongs named Too Much Information ? How long are the recordings, and on what albums werethey released?

Here's a simple query to answer this question, along with the freebase.com response:

ResultQuery

[{"type" : "/music/track",

[{"type":"/music/track",

"name" : "Too Much Information","name":"Too Much Information","artist" : "The Police","artist":null,"album" : "Message in a Box (disc 3)","album":null,"length" : 222.733"length":null

}] },{"type" : "/music/track","name" : "Too Much Information","artist" : "The Police","album" : "Ghost in the Machine","length" : 222.733

}]

You should have no trouble understanding this query. It requests an array of tracks with thespeci ed name, and asks Metaweb to ll in the artist, album, and length of each track. But thereare other ways to ask for this information. The above track-centric query is simple, but returnsan unordered and unstructured list of tracks. If multiple artists have recorded the same song, wemight like the result to be organized by artist. Here's how to write an artist-centric version of thequery, along with the more structured response from freebase.com :

ResultQuery

[{"type" : "/music/artist",

[{"type":"/music/artist",

"name" : "The Police","name":null,"album" : [{"album": [{

"name" : "Ghost in the Machine","name":null,"track" : [{"track": [{

"name" : "Too Much Information","name":"Too Much Information","length" : 222.733"length": null

}]}]}, {}]

}] "name" : "Message in a Box (disc 3)","track" : [{

"name" : "Too Much Information","length" : 222.733

}]}]

}]




Take a look at that query again. It involves three different objects: an album, and artist, and atrack. We can't tell Metaweb anything interesting about the album (such as a name or id): justthat it contains the song we're interested in. We can't tell Metaweb anything about the artist objecteither: just that they recorded an album that includes the song. Despite the seeming vaguenessof this query, Metaweb has no trouble nding the answer we want.

At rst glance, it seems as if the only information we're providing to Metaweb with this queryis the track name. But notice that we also explicitly specify the type of the outermost object:we've said that we want an object of type /music/artist . This is critical, because types haveproperties, and properties specify the type of their values. Since we've speci ed that the outermostobject is /music/artist , Metaweb knows that the middle object is a /music/album (because thatis the type of the /music/artist/album property) and that the inner object is a /music/track(because that is the type of the /music/album/track property). 5

We've answered our question about the song Too Much Information with a track-centric queryand an artist-centric query. For completeness, here is the album-centric query that returns thesame information:

[{"type":"/music/album","name":null,"artist":null,"track": [{

"name":"Too Much Information","length": null

}]}]

3.2.10. The id and name PropertiesThe id and name properties of every Metaweb object have special behavior that is important tounderstand. Some of this behavior was explained in Chapter 2 , but it bears repeating here.

The critical thing about id is that it is unique: every object's id is different. For objects, such astypes, that are organized into namespaces, the id is a fully-quali ed name such as "/music/artist".For other objects, the id is a guid: a unique, but meaningless, string of hexadecimal digits. Notethat although ids are represented with JSON strings, the id property of /type/object is of /type/id rather than /type/text or /type/rawstring .

In addition to its guarantees of uniqueness, the id property has some special behavior. Speci cally,the id property cannot be constrained with pattern-matching or comparison operators, and cannot

be used as a sort key. (We'll learn about operators and sorting later in this tutorial.)The special thing about the name property is that it behaves like a unique property (you can safelyquery it with null instead of [] , for example) but it is not truly unique. Any Metaweb object canhave multiple names, but may have only one name in any given language. That is, the nameproperty is unique on a per-language basis. When you query the name of an object, Metaweb

5 Yes, properties have fully-quali ed names that include the name of the type of which they are a part. We'll see example queries usingfully-quali ed names later in this tutorial.




returns its name (if it has one) in your preferred language. (The desired language is speci ed asa parameter to the mqlread query service, which is the topic of Chapter 4 .)

To demonstrate the special behavior of the name property, we must choose a topic that hastranslations into other languages. Let's nd the freebase.com topic named "Anarchism":

ResultQuery{

"type": "/common/topic",{

"type" : "/common/topic","name": "Anarchism","name" : "Anarchism","id": "#1f8000000000003b60"

}"id":null

}

Now, let's take this object identi ed by id, and ask for its name:

ResultQuery

{ "id":"#1f8000000000003b60",{ "id":"#1f8000000000003b60","name":"Anarchism"

}"name":null

}

This simply returns the English name we started with: "Anarchism". Let's ask for all names:

ResultQuery

{"id":"#1f8000000000003b60",

{"id":"#1f8000000000003b60",

"name":["Anarchism"]

}

"name":[]

}

This query just returns the unique English name in an array. So let's try again and ask for allnames, along with the languages in which they are encoded:




ResultQuery

{"id" : "#1f8000000000003b60","name" : [

{"lang":"/lang/en","type":"/type/text","value":"Anarchism"},

{"lang":"/lang/es","type":"/type/text","value":"Anarquismo"},

{"lang":"/lang/fr","type":"/type/text","value":"Anarchisme"},

{"lang":"/lang/it","type":"/type/text","value":"Anarchismo"},

{"lang":"/lang/de","type":"/type/text","value":"Anarchismus"},

]}

{"id":"#1f8000000000003b60","name":[{}]

}

Bingo! We nd that this object has names in English (en), Spanish (es), French (fr), Italian (it),and German (de)

Here's how we can ask for a name of the object in a speci c language other than our preferredlanguage:

ResultQuery

{"id" : "#1f8000000000003b60",

{"id":"#1f8000000000003b60",

"name": {"name":{"value": "Anarchisme","value":null,"lang": "/lang/fr""lang":"/lang/fr"

}}}}

3.2.11. Numeric ConstraintsWe know how to ask "what are the names and lengths of the tracks on the album Synchronicityby The Police?". The query looks like this:

{"type":"/music/album","name":"Synchronicity",

"artist":"The Police","track":[{"name":null, "length":null}]

}

Metaweb also allows us to ask "What are the names and lengths of the long songs on the album?"The query below includes a numeric constraint on the length property, and the freebase.comresponse only includes the two songs on the album that are longer than 300 seconds:




ResultQuery


{"type":"/music/album",

"name" : "Synchronicity","name":"Synchronicity","artist" : "The Police","artist":"The Police","track" : [{"track":[{

"name" : "Synchronicity II","name":null,"length" : 305.066"length":null,

}, {"length>":300"name" : "Wrapped Around Your Finger",}]

} "length" : 313.733}]

}

The line "length>":300 in the query expresses a constraint to Metaweb: it speci es that the track must be longer than 300 seconds. In addition to >, you can also use < for less-than, and <= and>= for less-than-or-equal and greater-than-or-equal. Note, however, that no spaces are allowedbefore or after these punctuation characters.

This constraint syntax looks quite odd at rst. It is a result of the limitations of the JSON format:everything must be expressed with property names, colons, and values. We would like to be ableto express a constraint like:

"length" <= 300

But that is not legal JSON syntax, so we express it instead like this:

"length<=" : 300

You can include more than one numeric constraint on the same property, restricting the value toa range. Here's how we ask for songs that are at least three minutes long, but less than four:

{"type":"/music/album","name":"Synchronicity","artist":"The Police","track":[{

"name":null,"length":null,"length>=":180,"length<":240

}]}

If you include a constraint on a property, you must also ask Metaweb to return the value of thatproperty. You cannot, for example ask: "List all songs longer than 5 minutes, but don't bother totell me exactly how long they are."




Numbers are not the only type that can be constrained with these operators. Here, for example,is a query that constrains a /type/datetime property to obtain a list of albums released in 1999:

[{"type":"/music/album","name":null,

"artist":null,"release_date":null,"release_date>=":"1999-01-01","release_date<=":"1999-12-31"

}]

3.2.12. Textual Constraints: Pattern Matching inQueries

Metaweb queries can also place constraints on textual values. To do this use the pattern matchingoperator ~=. 6 To try this out, let's nd some short songs about love:

[{"type":"/music/track","artist":null,"name":null,"name~=":"love","length":null,"length<":120

}]

Here's a query for songs about love recorded by bands whose name begins with "The":

[{ "type":"/music/track","artist":null,"artist~=":"^The","name":null,"name~=":"love"

}]

Results include If You Love Somebody, Set Them Free by The Police and I'm Sick of Love by TheWhite Stripes.

Notice that the constraint on the artist property in the query above uses the ^ character to specifythat the word The must appear at the beginning of the artist's name. If you're familiar with regularexpressions, this might make you think that Metaweb supports pattern matching with regularexpressions. In fact, Metaweb's matching syntax is closer to that used by internet search engines.Table 3.2 summarizes MQL pattern matching syntax. Note that all searches are case-insensitive.

6 If you've done programming with languages like Perl or Ruby, this syntax should look familiar. If you're not already familiar withit, think of "~=" as meaning "approximately equal" or "like".




Table 3.2. MQL Pattern Matching Syntax

MatchesPattern

Matches any string that contains the word "love". Does not match strings containing"glove" or "lover".

love

Matches any string containing a word that begins with "love", such as "love", "lover"or "lovely". Does not match "glove".

love*

Matches any string containing a word that ends with "love", such as "love" or "glove".*love

Matches any string that contains "love", such as "love", "glove", "lover" and "glover".*love*

Matches any single word*

Matches any string that contains the phrase "love you". Does not match strings thatcontain "glove you", "love your", "you love", "love hate you" or "loveyou".

love you

Matches the beginning of a string. For example, ^the matches any string that beginswith the word "the", and ^the* matches any string that begins with word that beginswith "the", such as "they" or "there".

^

Matches the end of a string. For example, hits$ matches any string that ends withthe word "hits", and *love$ matches "Sunshine of your Love" and "Smell the Glove".$

A hyphen or other punctuation matches an optional space. For example, bi-direc-tional matches "bi directional", "bi-directional", or "bidirectional".

-

Use a backslash to escape any punctuation character that you want to match literally.\' matches any string that contains an apostrophe, for example. Note, however, thatJSON string literals require backslashes like this to be doubled. If you type a JSONquery "by hand" or use string manipulation techniques to create a query, be sure todouble the backslashes. If you use a JSON serializer to create the query, it shoulddouble the backslashes for you.

\

Here's a query to nd all bands whose name is two words long and begins with the word The(such as The Police, and The Clash):

[{"type" : "/music/artist","name" : null,"name~=" : "^The *$"

}]

What bands have three-word names that begin with "the" and end with a plural (e.g. The BeachBoys, The Doobie Brothers)?

[{"type" : "/music/artist","name" : null,"name~=" : "^The * *s$"

}]

In addition to matching text with ~=, string constraints can also be applied with the <, >, <=, and>= operators, which compare strings in case-insensitive, Unicode-aware alphabetical order. Forexample, to nd bands whose name begins with one of the letters A through F, use this query:




[{"type" : "/music/artist","name" : null,"name>=" : "A","name<" : "G"

}]

Note that it is not legal to constrain the id property, with either the pattern-matching operator orthe greater-than or less-than operators.

3.2.13. Limiting QueriesEvery Metaweb query for a set of values is implicitly limited to 100 values -- to reduce resourceconsumption and bandwidth usage, Metaweb does not return more values than this unless youexplicitly ask for more. If you ran the query above for bands whose name begins with the lettersA through F, you ran up against this limit. To change the number of desired results to a larger,or a smaller, number, use the limit directive. Here, for example, is a query that returns the names

of up to 2000 bands:[{

"type":"/music/artist","name":null,"limit":2000

}]

limit is not a property name: it is a reserved word in MQL. No type may have a property named"limit". Limits can be useful to prune the result tree of values you aren't really interested in. Thefollowing query, for example, asks "What bands have names that begin with "The" and have re-corded songs longer than 8 minutes? I'm only interested in the band name, so just give me oneof the long songs, not the full list."

[{"type" : "/music/artist","name" : null,"name~=" : "^The","track": [{

"name":null,"length":null,"length>":480,"limit":1

}]}]

Note that we use a limit of one in the above. Specifying a limit of zero means "don't limit theresults: return everything you've got". Although MQL allows you to ask for an unlimited numberof results, Metaweb does not guarantee that you'll always get an answer. Complicated querieswith a large number of results may time out before Metaweb can complete the result.




Since the limit directive must appear within curly braces, limiting a query sometimes requiresyou to transform a simple query into a slightly more complex one. Consider this query to list allalbums by The Police:


"name" : "The Police","album" : []

}

If we want to limit the result to ve albums, we must rewrite the query as follows:

{"type" : "/music/artist","name" : "The Police","album" : [{"name":null, "limit":5}]

}

3.2.14. The Sort DirectiveUse the sort directive if you'd like the Metaweb server to sort the results of your query beforereturning them. For example, to ask for the names of the tracks on an album in alphabetical order,sort them by name:

// Tracks on the album Synchronicity, in alphabetical order{

"type":"/music/album","name":"Synchronicity","artist":"The Police","track": [{

"name":null,"sort":"name"

}]}

As you can see, the sort directive simply speci es the name of the property by which the sortis to be done. To order these same tracks from shortest to longest, use "length" as the sort key:

// Tracks on the album Synchronicity, from shortest to longest{

"type":"/music/album","name":"Synchronicity",

"artist":"The Police","track": [{"name":null,"length":null,"sort":"length"

}]}




Note that the query above includes "length":null . If you want to use a property as a sort key,you must query that property.

To reverse this order, precede the name of the sort key by a minus sign:

// Tracks on the album Synchronicity, from longest to shortest

{"type":"/music/album","name":"Synchronicity","artist":"The Police","track": [{

"name":null,"length":null,"sort":"-length"

}]}

The sorts shown above are convenient, but could easily be duplicated on the client side. That is,

you could request unordered results from Metaweb and sort them yourself. One situation in whichthe sort directive cannot be duplicated on the client is when it interacts with the limit directive.Result sets are truncated to the speci ed limit after the sort is applied. Use sort and limit togetherin queries like this:

// What is the longest track on Synchronicity?{

"type":"/music/album","name":"Synchronicity","artist":"The Police","track": {

"name":null,"length":null,"sort":"-length","limit":1

}}

(Note that explicitly specifying a limit of 1 means that we can safely omit the square bracketsfrom the query.)

Sorting need not be limited to a single sort key. To specify more than one key, use an array onthe right-hand side of the sort directive:

// List all tracks by The Police, sorted by album name and track name[{

"type":"/music/track","artist":"The Police","name":null,"album":null,"sort":["album","name"]

}]




If your query includes sub-queries, then the properties of those sub-queries can also be used assort keys. The query below is a variation on the one above that uses this kind of hierarchically-named sort key:

// List all tracks by The Police, on albums released before 1980,// sorted by album name and track name

[{"type":"/music/track","artist":"The Police","name":null,"album":{

"name":null,"release_date":null,"release_date<":"1980"

},"sort":["album.name","name"]

}]

Here is an example that uses the sort directive in two places:// List all albums by The Police, along with the name of their longest track.// Order the albums from longest longest track to shortest longest track.[{

"type":"/music/album","artist":"The Police","name":null,"track":{

"name":null,"length":null,"sort":"-length","limit":1

},"sort":"-track.length"

}]

3.2.15. Ordered CollectionsIf you do not include a sort directive in a query then Metaweb returns unordered results to you.In practice, with the current implementation, values are returned in the order that they were addedto the database. For ordered data, such as the list of tracks on an album, this insertion order isoften non-random. But there is no guarantee that this will always be the case. If you don't ask forthe data to be sorted, you should treat the result as an unordered set of values rather than anordered list. 7

Some data, such as the tracks on an album, have a natural order. If you want results to be sortedaccording to this natural ordering, use "sort":"index" . (Or, to reverse the natural ordering, use"sort":"-index" .

7 Metaweb's ordered collections are sometimes described as lists, but this term is inaccurate because lists are allowed to have duplicateelements. Metaweb's ordered collections are still fundamentally sets, and duplicates are not allowed.




// Return the tracks on the album Synchronicity in the order that they appear{

"type":"/music/album","artist":"The Police","name":"Synchronicity","track":[{

"name":null,"index":null,"sort":"index"

}]}

Since we've used "index" as a sort key, we must query the value of "index" as well. index is akeyword in MQL and is not a true property of any object. It can be queried, however, and whenyou do this, Metaweb returns a non-negative integer. It is important to understand that the notionof order does not apply to objects in Metaweb, but to the relationships between objects. It is thelink between the album "Synchronicity" and the track "Mother" that has an index of 3, not thetrack itself. This becomes clear when you consider the case of a track that appears on more than

one album. If "Mother" also appears on an album named "Greatest Hits" it is likely to have adifferent index on that album. 8

Since index is not a true property, there are a lot of things you cannot do with it. You cannotconstrain the index with property names index> or index< . MQL read queries may use index asa sort key, and they may query the index with "index":null , but may not use the keyword inany other way. You cannot write "index":1 to ask for the second item in a set, for example. (Theindex keyword can be used in other ways in write queries, however, and we'll learn about thatin Chapter 5 ).

The index keyword can be used in conjunction with the limit directive. Consider the followingquery, which ask for the last two tracks on Synchronicity:

ResultQuery

{"type": "/music/album","artist": "The Police","name": "Synchronicity","track": [{

"index": 1,"name": "Murder by Numbers"

},{"index": 0,"name": "Tea in the Sahara"

}]}

{"type":"/music/album","artist":"The Police","name":"Synchronicity","track":[{

"name":null,"index":null,"sort":"-index","limit":2

}]}

8 In Metaweb's schema tracks only appear on a single album: if multiple albums have a track by the same name, each one is a uniqueobject. So the example given here could not actually happen.




3.2.15.1. Indexes are Relative

The query above correctly returns the names of the nal two tracks on the album Synchronicity.Look carefully, however at the index values it returns: the last track is given an index of 1 andthe penultimate track an index of 0. This is not a bug: this query simply reveals the true nature

of ordered collections in Metaweb. Metaweb does not include an absolute index for each link.The implementation is able to say whether any link is greater-than or less-than another, but itcannot tell you the absolute position of that link within the complete set of links.

The number that Metaweb returns as the value of the index property is a synthetic one, generatedby Metaweb as a simple way to express the order of elements. If Metaweb returns an arrayholding n elements, then it generates index values for those elements that range from 0 to n-1.For example, if you ask for the last two tracks on an album, the resulting values have indexes 0and 1. If you ask for tracks that are shorter than 2 minutes and Metaweb nds three of them, thenit will assign them index values of 0, 1, and 2. If you want to know the track number for the trackson a particular album, you must query the complete set of tracks. Then add one to the index valueto get the track number. If you want to know the track numbers of the short songs, you must

query the complete set of tracks, and search for the short songs yourself.

3.2.16. Optional QueriesIn addition to the limit and sort directives, MQL also includes an optional directive. If partof your query is not required to match, add "optional":true to it. For example, we can use theoptional directive to ask the question: "What bands have names that begin with "The", and dothey have a Greatest Hits album?". The query looks like this:

[{"type" : "/music/artist","name" : null,"name~=" : "^The","album":[{

"name":null,"name~=": "greatest hits","optional":true

}]}]

Without the optional directive, the query would only return bands whose name begins with Thewho have released a Greatest Hits album. With the optional directive, we get all bands whosename begins with The, and additionally, we get the name of any albums they have released thatinclude the phrase "greatest hits".

Optional queries can be nested inside optional queries. The following query is an extension tothe one above. It further asks for the names of tracks longer than 5 minutes, if any exist, on theGreatest Hits album, if it exists.

[{"type" : "/music/artist","name" : null,




"name~=" : "^The","album":[{

"name":null,"name~=": "greatest hits","optional":true,"track": [{

"length":null,"length>":300,"name":null,"index":null,"sort":"index","optional":true

}]}]

}]

Note that it is legal, but never necessary or useful, to add "optional":false to a query. Also, itis never useful to use the optional directive in the top-level of a query. Queries are implicitly

optional at that level: if Metaweb can't nd a match, it returns an empty result.

3.2.17. Using Fully-Qualified Property NamesRecall from the beginning of this tutorial that most objects in Metaweb have two or more types:

ResultQuery

{"id":"#1f800000000006df1b",

{"id":"#1f800000000006df1b",

"name":"The Police","name":null,"type":["type":[]

} "/common/topic","/music/artist"

]}

What do you do if you want to query one property, such as a list of albums from one type, andanother property, such as a list of images, from a second type? MQL addresses this issue by al-lowing you to specify a fully-quali ed property name that includes the name of the type to whichit belongs. So here is how we ask for the albums by and pictures of, The Police:

{"type":"/music/artist","name":"The Police","album":[],"/common/topic/image":[{}]

}

The rst line of this query speci es that the object to be matched should be of type /music/artist .The second line speci es the name of the object. name and type are properties of /type/object ,and are shared by all objects in the database. These property names (along with id , guid , key ,




timestamp , creator , and permission ) can always be used without quali cation (although youcan qualify them with /type/object if you want to). Other types are not allowed to de neproperties whose names con f ict with these.

The third line of the query asks for a property named album . This property is not de ned by/type/object , but it is de ned by /music/artist , and the query has already declared that theobject will be an instance of that type. The fourth line asks for a property named image. This isnot de ned by /type/object nor by /music/artist , and so we must qualify it with the nameof its type so that Metaweb can understand it.

For symmetry, and to be explicit, you can rewrite the query to fully-qualify both properties of interest:

{"type":"/music/artist","name":"The Police","/music/artist/album":[],"/common/topic/image":[{}]

}

If you do this, you might be tempted to drop the initial type speci cation, since the album propertyis now fully-quali ed:

[{"name":"The Police","/music/artist/album":[],"/common/topic/image":[]

}]

Notice that we've put the toplevel query in square brackets now. This query will return any object

whose name is The Police, even if it has noalbum

orimage

properties, and even if it is an instanceof neither /music/artist nor /common/topic .

Note that quali ed property names use / as a delimiter and nested sort keys use . as a delimiter.If your query uses quali ed property names and sorts by those names, you may end up usingboth delimiter characters. The following query is a variation on one shown earlier, in which twoof the properties have been (unnecessarily) quali ed. Note the lengthy sort key:

// Police songs from albums released before 1990, sorted by album name[{

"type":"/music/track","artist":"The Police","name":null,"/music/track/album":{

"/type/object/name":null,"release_date":null,"release_date<":"1990"

},"sort":"/music/track/album./type/object/name"

}]




3.2.18. WildcardsWe saw wildcards earlier in this tutorial in the forms {"*":null} and {"*":{}} . But they aresomewhat more versatile than that. Consider the following query:

ResultQuery{

"id":"#1f8000000002f9e349",{

"id":"#1f8000000002f9e349","guid":"#1f8000000002f9e349","*":null

} "name": "Synchronicity","type": ["/music/album", "/common/topic"],"key": [

"1299f319-8ff4-44fb-8440-7fb990972864","RELEASE3178"

],"creator": "/user/mwcl_musicbrainz","permission": "/boot/all_permission","timestamp": "2006-11-30T13:42:18.0194Z"

}

This query identi es a unique object by ID, and then uses a wildcard to ask for all of its properties.Since no type has been speci ed, the wildcard is expanded with all the properties of /type/object ,and the result is as shown above.

Note that some of the properties expand to a single value, and others to arrays. Thus the syntax"*":null really means "*":null-or-[] . We could instead write the query using "*":[] . In thiscase, all of the property are returned as arrays, even unique properties.

Now let's modify the query to specify a type other than the default of /type/object :

{"type":"/music/album","name":"Synchronicity","*":null

}

In this query, the * wildcard expands differently. Since we have speci ed that the object is of type /music/album , Metaweb looks up the properties of that type and queries each one with anull or [] , depending on whether the property is unique or not. It does this in addition to alsoquerying the common object properties shown in the query result above.

Note that if a property is explicitly listed in a query, a wildcard expansion will not overwrite it.Consider this:

{"type":"/music/album","name":"Synchronicity","track":[{}],"*":null

}




This query explicitly asks for an array of tracks, as objects rather than just as track names. Theexpansion of the wildcard would normally include "track":[] , but in this case that propertywould con f ict with the explicitly speci ed one and will be left out of the expansion.

Wildcards can also be used in a second, more aggressive, form. "*":{} expands to query eachproperty with {} or [{}] instead of null or [] . Similarly, "*":[{}] expands to query eachproperty, even unique properties, with [{}] . Let's repeat the query with which we began thissection, using "*":{} instead. With this query, each of the properties of /type/object is expandedinto a complete object, and the result is much longer. The long response is reproduced here in itsentirety because it serves as a useful review of the structure of some of the most fundamentalMetaweb data types:

ResultQuery

{"id": "#1f8000000002f9e349",

{"id":"#1f8000000002f9e349",

"guid": {"*":{}} "type": "/type/id",

"value": "#1f8000000002f9e349",},"name": {

"lang": "/lang/en","type": "/type/text","value": "Synchronicity"

},"type": [{

"type": ["/type/type"],"id": "/music/album","name": "Record album"

},{"type": ["/type/type"],"id": "/common/topic","name": "Topic"

}],"key": [{

"type": "/type/key","namespace":

"/user/metaweb/datasource/MusicBrainz","value":

"1299f319-8ff4-44fb-8440-7fb990972864"}, {

"type": "/type/key","namespace":

"/user/metaweb/datasource/MusicBrainz/name",

"value": "RELEASE3178"}],"creator": {

"type": ["/type/user"],"id": "/user/mwcl_musicbrainz","name": null

},"permission": {

"type": ["/type/permission"],




ResultQuery

"id": "/boot/all_permission","name": "Global Write Permission"

},"timestamp": {

"type": "/type/datetime","value": "2006-11-30T13:42:18.0194Z"

}}

3.2.19. Expressing AND in QueriesMQL queries use JSON properties to express constraints. Those constraints are implicitly ANDedtogether. Consider:

[{"type":"/music/artist","name":null,"name~=":"^The","album":"Greatest Hits"

}]

This query says: tell me the names of objects which have type "/music/artist" AND which havea name that begins with "The" AND which have an album named "Greatest Hits".

Suppose we want to nd the names of all bands who have an album named "Greatest Hits" ANDan album named "Super Hits". We might try this query:

[{

"type":"/music/artist","name":null,"album":["Greatest Hits","Super Hits"] // Invalid MQL

}]

But this is not legal MQL. And if it was, it would probably mean nd an artist who has recordedexactly two albums, with names "Greatest Hits" and "Super Hits". A musical artist object mayhave multiple album links to album objects. We want to constrain our query so that all result objectshave links to two speci c album names. Here's a natural way to express this query:


"name":null,"album":"Greatest Hits","album":"Super Hits" // Invalid JSON

}]

This query makes sense in the Metaweb object model: nd objects that have one "album" link to an album named "Greatest Hits" and another "album" link to an album named "Super Hits".Unfortunately, this query is not valid JSON: since it includes the same property name twice, itcannot be parsed into object form.




MQL's solution to this dilemma is to allow an arbitrary identi er and colon to pre x any propertyname. The pre x and colon are ignored: they serve simply as a workaround to the JSON limitation

just described. With this trick we can rewrite the query above like this:

ResultQuery

[{"type": "/music/artist",


"name": "Alice Cooper","name":null,"a:album": "Greatest Hits","a:album":"Greatest Hits","b:album": "Super Hits""b:album":"Super Hits",

},{"limit":2}] "type": "/music/artist",

"name": "Dan Fogelberg","a:album": "Greatest Hits","b:album": "Super Hits"

}]

Note that the arbitrary pre xes we choose for the query are repeated in the result objects. Thepre xes are arbitrary, but they must be valid identi ers which means they cannot contain punc-tuation characters and must not begin with a digit.

This property pre xing scheme is not limited to sets of two properties. And pre xed propertiescan include operator suf xes. Let's nd bands that have lots of hits and have recorded Christmasalbums:

[{"type":"/music/artist","name":null,"a:album":"Greatest Hits","b:album":"Super Hits","c:album~=":"christmas","c:album":[]

}]

Another use of property pre xes is to constrain a property and also query the property at the sametime. Let's nd bands that have released a Greatest Hits album, and also ask for the names of allthe albums they have released:

[{"type":"/music/artist","name":null,

"album":[],"includes:album":"Greatest Hits",

}]

Note that although property pre xes are arbitrary, we can choose identi ers that add meaning toour queries.

At the beginning of this tutorial, we wrote a query to determine the types of the object that rep-resents The Police. In order to do this, we rst asked for the id of The Police, and then used the




resulting guid to uniquely identify the object so we could ask for its types. Property pre xes makethis easier:

ResultQuery

{"constraint:type": "/music/artist",

{"constraint:type":"/music/artist",

"name": "The Police","name":"The Police","query:type": ["query:type":[]

} "/music/artist","/common/topic"

],}

As an interesting aside, let's return to the query with which we started this section. We want tond bands that have released "Greatest Hits" and "Super Hits" albums. There is actually a way

to do this without property pre xes. It relies on the fact that Metaweb relationships are alwaysbi-directional and that MQL queries can be "turned inside out":

[{"type":"/music/artist","name":null,"album":[{

"name":"Greatest Hits","artist":{

"album":"Super Hits"}

}]}]

Translated into English, this query says: "give me the names of all bands that have released analbum named "Greatest Hits", the artist of which has released an album named "Super Hits". Thealbum property of a band object refers to an album object. And the artist property of the albumobject refers back to the band object. We can use this fact to further constrain the artist. Thistechnique (some would say "hack") is worth understanding because it illustrates one of the deepproperties of Metaweb objects.

3.2.20. Expressing OR in QueriesMQL has no general-purpose way to express an OR relationship. Suppose we want a list of bandswho have released an album named "Greatest Hits" OR and album named "Super Hits". The ~=

pattern matching operator does not have an OR syntax, and there is no way to specify that an albumname match either "Greatest Hits" OR "Super Hits". The only way to do this is to make twoqueries and combine their results: 9


9 It is possible to send two distinct queries in a single envelope with a single HTTP request. We'll learn how to do this in Chapter 4 .




"album":"Greatest Hits"}]


"album":"Super Hits"}]

Combining the results of two queries is fairly straightforward. The only tricky issue is avoidingduplicates. If a band appears in the results of both queries, for example, you would want to takecare that it did not appear twice in the combined result.

Combining property pre xes with a pattern matching operator and the optional directive, wecan achieve something vaguely like an OR operation:


"name":null,"album":[],"album~=":"hits","great:album":[{

"name":"Greatest Hits","optional":true

}],"superb:album":[{

"name":"Super Hits","optional":true

}]}]

This query returns all bands that have any albums whose name includes the word "hits". It returnsthe names of those albums, and includes optional sub-queries for the particular names we're in-terested in.

3.2.20.1.The |= Constraint

We began the discussion of expressing OR in queries by saying that MQL had no general-purposesyntax for expressing OR. There is, however a specialized syntax for expressing OR, and it isuseful in a number of situations. |= can be used at the end of a property as a constraint like > or~=. The value of such a constrained property should be a JSON array of strings. The constraintsays "match any one of the values in this array". (That is: match the rst value OR the second

value OR the third value...)

The reason that this is not a general-purpose way to express OR in MQL is that it only workswhen the strings in the array are object ids or guids. The meaning and use of the |= constraintbecomes much clearer with some examples. One straightforward use is to run the same queryover multiple objects that are speci ed by id. The following query asks for the properties of threetypes:




[{"id|=":["/type/type", "/type/property", "/type/key"],"id":null,"/type/type/properties":[]

}]

This next example asks for the ids of GIF or PNG (but not JPEG) images:

[{"type":"/common/image","id":null,"/type/content/media_type":null,"/type/content/media_type|=":[

"/media_type/image/gif","/media_type/image/png"

]}]

Finally, here is an example that uses both the |= constraint to express an OR and uses a propertypre x to express AND. It asks for the French and Spanish translations of the country name"England":

ResultQuery

{"type":"/location/country",

{"type":"/location/country",

"english:name":"England","english:name": "England","foreign:name":[{"foreign:name": [{

"value":"Angleterre","value":null,"lang":"/lang/fr""lang":null,

},{"lang|=":["/lang/fr","/lang/es"]

"value":"Inglaterra",}]} "lang":"/lang/es"

}]}

3.2.21. Expressing NOT in QueriesMetaweb has no syntax to perform logical NOT operations. In general, with huge universe of knowledge, the NOT of a result may be a very, very large set of objects. There is not a way towrite a single query that says: list all bands who have a Greatest Hits album but do not have analbum that includes the word "Best". To do this, you'd rst query the bands with a Greatest Hitsalbum. Then you'd query the bands who have "album~=":"best" , and then you'd subtract theresults of the second query from the rst query.

As another example, suppose you wanted to know what bands had an album named "GreatestHits", but wanted to exclude all country music. You could do one query for Greatest Hits albums,and then do another for all country music bands (using the /music/artist/genre property), andthen subtract the second result from the rst. This is not particularly ef cient, since there areprobably a whole lot of country music bands. Better would be a single query for albums named




"Greatest Hits" that also asks for the genre of the album (with /music/album/genre ). Then, parsethe JSON result, and post-process it yourself to remove albums whose genre is "country".

3.2.22. Reflective Queries

If you've enjoyed making queries against Freebase's repository of musical knowledge, you mightalso enjoy querying the underpinnings of the Metaweb infrastructure. Types, properties andnamespaces are all Metaweb objects, and they can all be queried just like other objects. Here, forexample is how we nd the properties of /type/object : 10

ResultQuery

{"type": "/type/type",

{"type":"/type/type",

"id": "/type/object","id":"/type/object","properties": ["properties":[]

} "/type/object/id","/type/object/guid",

"/type/object/type","/type/object/name","/type/object/key","/type/object/timestamp","/type/object/permission","/type/object/creator"

]}

And let's ask about the name property:

ResultQuery{

"type": "/type/property",{

"type":"/type/property","id": "/type/object/name","id":"/type/object/name","guid": "#1f80000000000000ca","*":null

} "name": "name","key": ["display_name", "name"],"expected_type": "/type/text","unique": true,"schema": "/type/object","master_property": null,"reverse_property": [],"creator": "/user/root","permission": "/boot/root_permission","timestamp": "2006-11-30T12:43:53.0081Z"

}

10 As a matter of convention, property names are usually singular, even when they are non-unique properties and multiple values areexpected. /type/type/properties is an exception to this rule.




This kind of re f ective query is not only useful for exploring the Metaweb infrastructure, but canbe helpful in understanding the schemas of types you actually want to use. Suppose you knowthat the type /music/album has a property named track , and you want to know what type a track is so that you can query tracks directly. This is actually a very easy query, if you understand howtypes and properties work:

ResultQuery{

"id":"/music/album/track",{

"id":"/music/album/track","/type/property/expected_type":"/music/track"

}"/type/property/expected_type":null

}

Note that we omitted a type speci cation from this query and instead simply used the fully-quali ed name of the expected_type property.

If you were planning to write a program that made many music-related queries, you might rstwant to explore all of Freebase's music-related types. But where do you get a list? You query thedomain: 11

ResultQuery

{"type": "/type/domain",

{"type":"/type/domain",

"id": "/music","id":"/music","types": ["types":[]

} "/music/group_membership","/music/recording_contribution","/music/track","/music/album","/music/album_release_type","/music/artist","/music/genre","/music/group_member","/music/instrument","/music/performance_role","/music/record_label","/music/voice"

]}

11 /type/domain/types is another plural property name.




3.3.The MQL GrammarThis chapter began with an explanation of the JSON grammar. Although MQL uses JSON, it hasadditional grammar rules layered on top of those imposed by JSON. The extended tutorial aboveexplained MQL by example. This section documents MQL grammar more formally. In thegrammar that follows, bold text indicates terminals that must appear literally and italics are usedfor the terms de ned by the grammar. Angle brackets indicate terms that are not formally de ned,such as <positive number> Spaces in the grammar are optional, and may usually be replacedby zero or more whitespace characters. In places where no whitespace is allowed, a . is used toindicate concatenation.

Let's begin at the top level. A query is a comma-separated list of one or more pairs , enclosedin curly braces, which are optionally enclosed in square braces:

query:: { pairs } | [{ pairs }]pairs:: pair ( , pair )*

A pair may be a property, a wildcard, a comparison, or a directive:

pair:: property | wildcard | comparison | directive

A property begins with a property name in quotation marks followed by a colon and a propertyvalue. The property value may be a nested query, a JSON literal value or an "empty" value suchas null or [] . As a special case, the index query "index": null is also allowed.

property:: " . property-name . " : property-value |"index" : null

property-value:: query | literal | empty

literal:: <JSON-string> | <JSON-number> | true | false

empty:: null | [ ] | { } | [ { } ]

A property-name is either a simple-name or a qualified-name or a prefix and a colon followed(without spaces) by a simple-name or quali f ed-name .

A simple-name is an identi er that is not a reserved word. A prefix is the same thing. A quali-fied-name consists of a slash and one or more identi ers followed by slashes, all followed by asimple-name . Finally, an identi er is a string of ASCII characters that begins with a letter andconsists of letters, numbers and underscores. Additionally, an identi er may not end with anunderscore or contain two underscores in a row. Reserved words include sort , limit , optionaland index , and also include directives used by the MQL write grammar (see Chapter 5 ) and anumber of other identi ers that are reserved for possible use in the future:

property-name:: simple-name |qualified-name | prefix . : . simple-name | prefix . : . qualified-name

simple-name:: identifier <but not reserved-word >




qualified-name:: / . ( identifier . / )+ . simple-nameprefix:: identifier <but not reserved-word >identifier:: <ASCII string matching /^[A-Za-z](_?[A-Za-z0-9])*$/>reserved-word:: all | any | as | attribute | class

connect | count | create | cursor | datatypedefault | delete | destroy | else | for

function | future | if | in | indexinsert | is | left | limit | link

macro | meta | mql | offset | optional pagesize | property | read | relationship | replace

return | right | scope | select | selfsort | sql | super | this | typeguidupdate | var | while | write | xml

The remaining kinds of pair that can appear in a query are wildcard , comparison and directive .A wildcard is an asterisk in quotes (to indicate a wildcard property name) followed by a colonand an empty query (a null , [] , {} or [{}] ):

wildcard:: "*" : empty

A comparison is quoted name followed by a colon and a value, where the name includes an op-erator and the value is a string or a number:

comparison:: " . comparison-name . " : comparison-valuecomparison-name:: property-name . comparison-operator comparison-value:: <JSON-string> | <JSON-number>comparison-operator:: < | <= | > | >= | ~=

A directive is a limit, optional or sort directive:

directive:: "limit" : <positive number> |"optional" : true | "optional" : false |sort-directive

The sort directive syntax requires further explanation. A sort directive is the keyword "sort"followed by a colon and a sort key or an array of sort keys. A sort key is the keyword "index"or one or more property names separated with . characters:

sort-directive:: "sort" : sort-keyssort-keys:: sort-key | [ ( sort-key , )* sort-key ]sort-key:: "index" |

" . property-name ( . property-name )* "

The MQL grammar for updating Metaweb has some additional rules. We'll learn about those inChapter 5 .




62



Chapter 4. Metaweb Read ServicesChapter 3 explained how to express Metaweb queries using MQL. This chapter explains how todeliver those queries to Metaweb servers and retrieve their response using the mqlread service.It also explains how to retrieve chunks of data (such as images and HTML documents) using thetrans service. The chapter includes many example applications, written in Perl, Python, PHP,and JavaScript.

4.1. Basic mqlread Queries with PerlMetaweb's services are all implemented on top of the HTTP protocol. Submitting a MQL queryand retrieving the response, therefore, is simply a matter of constructing the appropriate URLand fetching its content via an HTTP request.

The basic URL for submitting MQL queries to freebase.com is:

http://www.freebase.com/api/service/mqlread

To submit a query to the mqlread service, place the query inside an "envelope" object. Next, useJSON to serialize the query object. Then URL encode the JSON string and pre x it with ?quer-ies= . Finally, append the whole thing to the URL above, and retrieve the content of the resultingURL with an HTTP GET request.

The metaweb-user Cookie

While freebase.com is being rolled out and scaled up, the mqlread service at that site requiresHTTP cookie headers for authentication. If you are an early user at freebase.com , this issue

will affect you. Some of the examples in this chapter demonstrate how to correctly log into freebase.com , obtain credentials, and submit them to the mqlread service using HTTPcookies. Others, like the one that follows, simply ask you to hardcode your cookie datainto the script.

To nd the authentication data you need, rst visit the freebase.com home page and makesure you are logged in. This will ensure that your browser has stored the necessary cookies.Obtaining cookie data is a browser-speci c task. If you are using Firefox, pull down theEdit menu and select Preferences . Click the Show Cookies... button in the dialog thatappears. This will open a second dialog. Enter "freebase.com" into the Search box, andthen scroll through the resulting list of cookies until you nd the one named metaweb-user .Highlight this entry in the list to view the content of the cookie. It will probably something

like this:

A|u_docs|g_#1f8000000001209013|4.xwItasvXuoVOQiXg3Sm04b

This is the string you'll need to enter into Example 4.1 and Example 4.6 . (You have to useyour own, though: the cookie shown here is not actually valid.)

63



Example 4.1 is a command-line utility that list the albums released by any band you specify. Ituses the Metaweb API to retrieve data from freebase.com . It is written in Perl, and demonstrateshow to nest an MQL query within an envelope and send that envelope to to the mqlread service.(The structure of the envelope object will be explained in Section 4.2.3 .) It sends hard-codedauthentication credentials to mqlread using HTTP cookies. Until freebase.com has fully openedits services to the world, you'll have to insert your own cookie data into this script to make itwork.

Example 4.1. albumlist.pl: submitting MQL queries in Perl

#!/usr/bin/perluse URI::Escape; # This module provides the uri_escape function used below

# Build the Metaweb query, using string manipulation# CAUTION: the use of string manipulation here makes this script vulnerable# to MQL injection attacks when the command-line argument includes JSON.$band = $ARGV[0]; # This is the band or musician whose albums are to be listed$query='{"type":"/music/artist","name":"' . $band . '","album":[]}';

# Now place the query in JSON envelope objects, and URL encode the envelopes$envelope = '{"qname":{"query":' . $query . '}}';$escaped = uri_escape($envelope);

# Construct the URL that represents the query$baseurl='http://www.freebase.com/api/service/mqlread'; # Base URL for queries$url = $baseurl . "?queries=" . $escaped;

# During Freebase's roll-out, authentication data must be encoded in a cookie# Enter your cookie data below.$auth = 'metaweb-user=###Enter Your cookie data here###';

# Use the command-line utility curl to supply the cookies and fetch the# content of the URL.$result = ̀ curl -s --cookie \'$auth\' $url`;

# Use regular expressions to extract the album list from the HTTP response$result =~ s/^.*"album"\s*:\s*\[\s*([^\]]*)\].*$/$1/s;$result =~ s/[ \t]*"[ \t,]*//g;

# Finally, display the list of albumsprint "$result\n";

4.1.1. A Better Perl Album ListerThe rst thing to notice about Example 4.1 is that it does not use a JSON serializer or parser: aJSON-encoded MQL query is constructed with string concatenation and the desired results areextracted with regular expressions. These shortcuts keep the example simple and allow us to focuson how the mqlread URL is built and its content fetched. More sophisticated applications, however,use a JSON encoder to serialize the query and a JSON decoder to parse the result. Buildingqueries with string manipulation can be reasonable in the simplest applications (though caution




is required to avoid MQL injection attacks), but attempting to extract results with regular expres-sions is brittle and not a technique to emulate in your own code!

Example 4.2 is a higher-level version of Example 4.1 . It uses a JSON serializer and parser andalso a higher-level API for URL manipulation. To use it, you must have the JSON.pm module(which you can nd at http://search.cpan.org ) installed. This version of the program also uses asomewhat more sophisticated query to sort albums by their release date, and also does errorchecking and error reporting in case anything goes wrong with the query. Finally, this versionof the program demonstrates how to log in to Metaweb to obtain authentication credentials. Insteadof hardcoding your cookie data into the script, you must instead hardcode your Freebase usernameand password. Although this example uses the Metaweb login service, that service is not formallydocumented until Chapter 6 .

Note that Example 4.2 places the query in the inner and outer envelope objects before JSONserialization. Note also that the mqlread service returns the query results in its own two-layerresponse envelope object. The outer object includes a property with the same name as the outerobject of the query envelope. The inner object of the response has a property named result . Thevalue of the result property is the result of the MQL query.

Example 4.2. albumlist2.pl: a better Perl album lister

#!/usr/bin/perl -wuse strict; # Don't allow sloppy syntaxuse JSON; # JSON encoding and decodinguse URI::Escape; # URI encodinguse LWP::UserAgent; # High-level HTTP API

# Some constants for this scriptmy $SERVER = 'http://www.freebase.com'; # The Metaweb servermy $QUERYURL = $SERVER . '/api/service/mqlread'; # Path to mqlread service

my $LOGINURL = $SERVER . '/api/account/login'; # Path to login servicemy $USERNAME = 'user'; # Enter your Freebase username namemy $PASSWORD = 'pass'; # Enter your Freebase password here

# Create the HTTP "user agent" we'll use to send the querymy $ua = LWP::UserAgent->new;

# Login to Metaweb to get authentication credentials for our UA object.# This will add an authentication cookie to subsequent HTTP requests.# The login() subroutine is defined below.&login($ua, $USERNAME, $PASSWORD);

# What band did the user ask about?my $band = $ARGV[0];

# Construct a Metaweb query as a Perl data structuremy $query = {

type => "/music/artist", # We're looking for a bandname => $band, # This is the name of the bandalbum => [{ # Return some albums

name => undef, # undef is Perl's null

65Chapter 4. Metaweb Read Services



sort => "release_date", # sort by release daterelease_date => undef # return release date, too

}]};

# Put the query in an envelope object

my $envelope = { # This is the outer envelope objectalbumquery => { # "albumquery" is an arbitrary name for inner envelope

query => $query # The "query" property of inner envelope holds query} # End of inner envelope

}; # End of outer envelope

# Convert the envelope object from Perl hash to JSON string, and URI encode itmy $json = JSON->new(); # Create JSON parser/serializermy $encoded = $json->objToJson($envelope); # Serialize object to stringmy $escaped = uri_escape($encoded); # URI encode the string

# Build the complete query url

my $url = $QUERYURL . "?queries=" . $escaped;

# Send request to the server and get the responsemy $response = $ua->get($url);

if ($response->is_success) { # If we get HTTP 200 OKmy $responsetext = $response->content; # Get result as JSON textmy $outer=$json->jsonToObj($responsetext); # Parse text to a Perl hashmy $inner = $outer->{albumquery}; # Open outer envelope

if ($inner->{status} ne "/mql/status/ok") { # If the query was not okaymy $err = $inner->{messages}[0]; # Get the error message obj

die $err->{status}.': '.$err->{message}; # and exit with error message}

my $result = $inner->{result}; # Open inner envelopemy $albums = $result->{album}; # Get albums from resultfor my $album (@$albums) { # Loop through albums

print "$album->{name}"; # Print the name of eachif ($album->{release_date}) { # Print release date

print " [" . substr($album->{release_date},0,4) . "]";}print "\n"; # Add a newline

}

}else { # If query faileddie "Server returned error code " . $response->code . "\n";

}

# This subroutine calls the Metaweb login service to obtain authentication# credentials. It asks the UA to send those credentials as cookies in# all future requests.sub login {




4.2.2. mqlread Outputmqlread returns an HTTP response with a Content-Type header of application/json (ortext/plain if the callback parameter was speci ed). The body of the response is a JSON-seri-alized envelope object that holds a MQL result object (or objects if multiple queries were submit-

ted). The format of mqlread response envelopes is speci ed in Section 4.2.3

4.2.3. Query and Response EnvelopesMQL queries must be nested inside two JSON objects before being sent to mqlread . Similarly,the MQL response sent my mqlread is nested inside in two layers of JSON objects. These wrapperobjects are known as envelopes . To understand the envelope metaphor, imagine that the internetis actually run by the postal service...

Suppose that we have two MQL queries that we want mqlread to execute. We write the rstquery on a piece of paper, fold it up and place it in an envelope. (This is the "inner query envelopeobject"). We name this query "q0" and write those letters on the envelope. Next we write thesecond query on another piece of paper. We put that paper in another envelope, and write "q1"on that envelope. Finally, we place both envelopes within a cardboard box (the outer query en-velope object) and mail the box off to http://www.freebase.com/api/service/mqlread .

The mqlread service opens the box and opens the envelopes it contains. It executes the twoqueries and writes the results on two pieces of paper. It places the results of the rst query in anenvelope (the inner response envelope) and writes "q0" on that envelope. It does the same forthe results of the second query, and writes "q1" on that envelope. Then it puts the two envelopesin a box (the outer response envelope) and mails the box back to us.

The envelopes and cardboard boxes in our postal metaphor are just JSON objects, of course.

Here's the two-layer query envelope described above looks like in JSON:{ # This is the outer envelope object

"q0": { # This is the first inner envelope. The name "q0" is arbitrary"query": { # The first MQL query goes here}

},"q1": { # This is the second inner envelope

"query": [{ # Second MQL query goes here. Note that this one is in []}]

}}

The response envelope has the same structure as the query envelope. In JSON it might look likethis:

{ # This is the outer envelope object"q0": { # This is the first inner envelope. The name "q0" is arbitrary

"status":"/mql/status/ok" # The query was successful"result": { # The result of the first MQL query goes here}




},"q1": { # This is the second inner envelope

"status":"/mql/status/ok" # The query was successful"result":[{ # The result of the second MQL query goes here.}]

}

}

The outer and inner query and response envelopes are described more formally below:

4.2.3.1.The Outer Query Envelope

The outer query envelope is a JSON object with one or more properties. The value of eachproperty must be an inner envelope object. The name of each property de nes a name for thequery that is included in the inner envelope. The response envelope sent by mqlread includes aproperty by the same name.

4.2.3.2.The Inner Query EnvelopeThe inner query envelope is a JSON object that must have a property named query . The valueof this property is a MQL query.

The inner query envelope may include other properties that provide additional information abouthow the query should be executed. At the time of this writing the only such property is cursorwhich is documented in Section 4.7 .

4.2.3.3.The Outer Response Envelope

The outer response envelope has the same properties as the outer query envelope. Each of these

properties names a query, and its value is the inner response envelope for that query.

4.2.3.4.The Inner Response Envelope

Each inner response envelope object has a status property. (Note: the mqlread implementationmay also include a status property in the outer response envelope. This outer status propertyis not the same and should not be used.) If the query was successful, then the status property is"/mql/status/ok" . In this case, the inner response envelope also has a property named result ,and the value of this property is the MQL result of the query.

If, on the other hand, something was wrong with the query, then the status property will be"/mql/status/error" , and the inner response envelope will have a messages property whosevalue is a JSON array of message objects that provide details about the error or errors. mqlread error messages are documented in Section 4.6 .




args = urllib.urlencode({'queries':simplejson.dumps(env)})# Build the URL and create a Request object for iturl = 'http://%s%s?%s' % (host, readservice, args)req = urllib2.Request(url)

# Send our authentication credentials, if any, as a cookie.

# The need for mqlread authentication is a temporary restriction.if credentials:

req.add_header('Cookie', credentials)

# Now upen the URL and and parse its JSON contentf = urllib2.urlopen(req) # Open the URLresponse = simplejson.load(f) # Parse JSON response to an objectinner = response['qname'] # Open outer envelope; get inner envelope

# If anything was wrong with the invocation, mqlread will return an HTTP# error, and the code above with raise urllib2.HTTPError.# If anything was wrong with the query, we won't get an HTTP error, but

# will get an error status code in the response envelope. In this case# we raise our own MQLError exception.if inner['status'] != '/mql/status/ok':

error = inner['messages'][0]raise MQLError('%s: %s' % (error['status'], error['message']))

# If there was no error, then just return the result from the envelopereturn inner['result'];

# If anything goes wrong when talking to a Metaweb service, we raise MQLError.class MQLError(Exception):

def __init__(self, value): # This is the exception constructor method

self.value = valuedef __str__(self): # Convert error object to a stringreturn repr(self.value)

With the metaweb.read() function de ned, we can now write our album listing code in Python.Example 4.4 shows how we do this. Note that this example uses the metaweb.login() functionfor authentication. The implementation of this function is in Chapter 6 .

Example 4.4. albumlist.py: listing albums in Python

import sysimport metaweb # Defines the metaweb.read() and login() functions

# Compose our MQL query using a Python data structureband = sys.argv[1] # The band we wantquery = { 'type': '/music/artist', # Our MQL query in Python

'name': band,'album': [{ 'name': None, # None is Python's null

'release_date': None,'sort': 'release_date' }]}




# Login to get authentication credentials# Insert your Freebase username and password here.credentials = metaweb.login("username", "password");

# Submit the query using metaweb.read() and check for valid resultsresult = metaweb.read(query, credentials)

if not result or not result['album']: sys.exit('Unknown band')

# A utility function to get year from a MQL datetime valuedef getYear(date):

if not date: return ''return "[%s]" % date[0:4]

# Now output the resultsfor album in result['album']:

print "%s %s" % (album['name'], getYear(album['release_date']))

4.4. A Metaweb-enabled PHP Web Applica-tionIn this section, we'll demonstrate how to create an online version of our album-lister application.We'll use the the server-side scripting language PHP to create the web application that was shownin Figure 1.2 of Chapter 1 . Example 4.5 is a PHP le that de nes a class named Metaweb . Thisclass has a single method, named read that behaves just like the metaweb.read() function de nedin Example 4.3 .

The code in Example 4.5 is commented and you should be able to follow it even if you are notfamiliar with PHP. One point to note is that in PHP the data structure known as an array works

as both a sequential array and as an associative array. That is, JSON objects and JSON arraysare both arrays in PHP. Example 4.5 depends on an external module for JSON serialization andparsing. The module used here is from http: //pear.php.net .

Example 4.5. metaweb.php: using mqlread with PHP

<?php/*

* The Metaweb class defines a read() method for invoking the Metaweb* mqlread service on freebase.com. read() takes a MQL query (as a PHP* array) and freebase.com authentication credentials. It sends* that query to the mqlread service and retrieves the response. It parses

* the response to a PHP array, and extracts the query result from the* response envelopes and returns it. If the query fails, it returns* null (without providing useful diagnostics).*/

require "JSON.php"; // A JSON encoder/decoder from http://pear.php.net

class Metaweb {var $json; // Holds the JSON encoder/decoder objectvar $URL = "http://www.freebase.com/api/service/mqlread";




the user can enter the name of a band. When the form is submitted, it lists the albums by thatband.

Like Example 4.1 , this example requires you to hard-code the value of your freebase.com authen-tication cookie into the script. See the instructions earlier in this chapter for nding the value of the metaweb-user cookie.

Example 4.6. albumlist.php: A Metaweb-enabled web application in PHP

<html><body><form>Band: <input type="text" name="band"><input type="submit"></form><?php$band = $_GET["band"]; // What band is specified in the URL?if ($band) { // Only list albums if a band has been specified

require "metaweb.php"; // Import Metaweb utility code$metaweb = new Metaweb(); // Create a Metaweb object

// Build a MQL request for the list of albums by the band$query = array("type" => "/music/artist", // We want an musical artist"name" => $band, // This is its name"album" => array()); // Fill in this empty albums array!

// Insert your own freebase.com cookie data into the string below$credentials = 'metaweb-user=### Put your cookie data here ### ';

// Submit the query using the utility function defined earlier$result = $metaweb->read($query, $credentials);

// This is the array of albums we want

$albums = $result["album"];

// Display the albums on the web pageecho "<hr><h1>Albums by " . $band . "</h1>";foreach ($albums as $album) echo $album . "<br>";

}?></body></html>

4.5. Metaweb Queries with JavaScriptSince the MQL syntax is based on JSON, Metaweb queries are most gracefully expressed inJavaScript. We haven't seen a JavaScript-based Metaweb application so far for one importantreason: the same-origin policy . The same-origin policy is a sweeping (but necessary) securityrestriction in JavaScript that says that code embedded in a document that was served by serverA can only interact with content that is also served by server A. This restriction applies to theXMLHttpRequest object which is what is typically used to fetch the contents of a URL. A webapplication hosted at www.freebase.com can use XMLHttpRequest to submit MQL queries to the




mqlread service on that same server, but this is not allowed for web applications hosted on anyother server.

There are two workarounds to this restriction. The rst, and most obvious, is to run a proxy scripton your own site that behaves like the mqlread service but simply forwards your query to free-base.com.

The second workaround relies on the fact that a query result, in JSON format, is valid JavaScriptcode. This means that a mqlread URL can be used as the value of the src attribute of a <script>tag. When the server returns its result, the <script> tag evaluates the JSON text as JavaScriptcode. Evaluating the JSON text creates the JavaScript object we want, but to make this schemework, the script then has to be able to do something with that object. The solution is to add anotherURL parameter to the mqlread service. If the URL for your query includes a callback= parameter,then mqlread will take the value of that parameter to be the name of a JavaScript function. Then,instead of simply returning a JSON text, it will return the speci ed function name, an open par-enthesis, the JSON text and a close parenthesis. When used this way with a <script> tag, theJSON text is evaluated, and the object that results is passed to the speci ed function (which youhave de ned previously).

We use the <script> technique in this chapter: it is simple, elegant and in common use acrossthe internet. If you prefer a proxy and XMLHttpRequest -based approach, you can nd sampleproxy code in Appendix A .

One thing you'll notice about our JavaScript examples here (and in in Appendix A ) is that theyare asynchronous: when you submit a query you do not get the result immediately. Instead, thecallback function you specify is invoked when the result is available. This asynchronous program-ming model is common in client-side JavaScript, but is substantially different from the synchronousmodel demonstrated in Example 4.5 and other examples.

JavaScript Web Apps and Cookies

The JavaScript-based web applications shown in this chapter do not attempt to automaticallyobtain authentication credentials. They simply assume that you have visited and logged into www.freebase.com before you attempt to run them. If you do this, then your browserwill have the authentication cookies it needs, and it will automatically pass those cookiesto mqlread when <script> tags invoke it.

4.5.1. Listing Albums and Tracks with JavaScriptLet's jump right in. Example 4.7 is a JavaScript-based album and track lister application, pictured

in Figure 4.1 . Notice that this code depends on two external modules. json.js is a le that de nesJavaScript functions for parsing and serializing JSON. The code for this module is not shownhere; you can nd it as Example A.1 in Appendix A . The second module of external code ismetaweb.js . This module, whose code listed in Example 4.8 , de nes the utility functionMetaweb.read() that submits a MQL query, through a <script> tag, to the mqlread service.


Figure 4.1. Listing albums and tracks

Example 4.7 lists the albums by a speci ed band, and also displays the tracks on an album whenthe user clicks on the name of the album. There are several features worth noting in this example.First, notice that this code uses the Metaweb.read() function to send queries to the mqlread service.We'll see how Metaweb.read() is implemented in the next section. Second, note that the codedisplays a "Loading..." message to the user while the queries are pending and also displays anappropriate message if a query fails. Finally, Example 4.7 demonstrates two different ways toinsert Metaweb query results into an HTML document. When the album query returns, the albumlist is populated using DOM methods to build each text node and <div> tag. When the track queryreturns, on the other hand, the list of tracks is built as a string of HTML text and is inserted intothe document by setting the innerHTML property of the container element.

In addition to querying album and track names, the queries in this example also ask for albumrelease date and track length. The example includes utility functions to massage this data, extractinga year from a /type/datetime string and converting a track length in seconds into the more fa-miliar mm:ss format.

Example 4.7. albumlist.html: a JavaScript album and track lister

<html><head><script src="json.js"></script>



// Add an event handler to display tracks when an album is clickeddiv.onclick = makeHandler(band, albums[i].name);

}

// This function returns a function. We do it this way to create// a closure that captures the band and album names.

function makeHandler(band, album) {return function(e) { listtracks(band, album); }

}}

// A utility function to return the year portion Metaweb /type/datetimefunction getYear(date) {

if (!date) return null;if (date.length == 4) return date;if (date.match(/^\d{4}-/)) return date.substring(0,4);return null;

}

}

/* Display the tracks on the specified album by the specified band */function listtracks(band, albumname) {

// Begin by displaying a Loading... messagevar tracklist = document.getElementById("tracklist");tracklist.innerHTML = "<h2>" + albumname + "</h2><p>Loading...";tracklist.style.visibility = "visible";

// This is the MQL query we will issuevar query = {

type: "/music/album",

name: albumname,artist: band,// Get track names and lengths, sorted by indextrack: [{name:null, length:null, index:null, sort:"index"}]

};

// Issue the query, invoke the nested function when the response arrivesMetaweb.read(query, function(result) {

if (result && result.track) { // If result is definedvar tracks = result.track; // array of tracks// Build an array of track names + lengthsvar listitems = []

for(var i = 0; i < tracks.length; i++) {var n = tracks[i].name + " (" +toMinutesAndSeconds(tracks[i].length)+")";

listitems.push(n);}

// Display the track list by setting innerHTMLtracklist.innerHTML = "<h2>" + albumname + "</h2>" +

"<ol><li>" + listitems.join("<li>") + "</ol>";}


else {// If empty result display error messagetracklist.innerHTML = "<h2>" + albumname + "</h2>" +

"<p>No track list is available.";}

});

// Convert track length in seconds to minutes:seconds formatfunction toMinutesAndSeconds(seconds) {

var minutes = Math.floor(seconds/60);var seconds = Math.floor(seconds-(minutes*60));if (seconds <= 9) seconds = "0" + seconds;return minutes + ":" + seconds;

}};</script><style>

#albumlist { width:50%; padding: 5px; }#tracklist {

width: 45%; float:right; visibility:hidden;padding: 5px; border: solid black 2px; margin-right: 10px;background-color: #8a8;

}#tracklist h2 { font: bold 16pt sans-serif; text-align: center;}#tracklist p { text-align: center; font: italic bold 12pt sans-serif; }#tracklist li { font-style: italic;}div.album { font: bold 12pt sans-serif; margin: 2px;}div.album:hover {text-decoration: underline;}</style>

</head><body><form onsubmit="listalbums(this.band.value); return false;"><b>Enter the name of a band: </b><input type="text" name="band"><input type="submit" value="List Albums"></form><hr><h1 id="title"></h1> 

<div id="tracklist"></div> <div id="albumlist"></div> </div></body></html>



4.5.2. Client-side MQL Queries with <script>In this section, we develop the Metaweb.read() utility function used by Example 4.7 . The codein Example 4.8 is short but somewhat complicated. The key to understanding it is to realize thateach call to Metaweb.read() de nes a function with a name like Metaweb._3() (the number is

different on each invocation). This function does the work of processing the response from theMetaweb server. In order to get this function invoked, Metaweb.read() adds a callback parameterto the mqlread query URL, like this:

&callback=Metaweb._3

When the mqlread service is invoked with this callback parameter, it does not return the resultas a pure JSON object. Instead it returns JavaScript code. The code is simply a function invocationof the function named by the parameter. The invocation includes a JSON object as the single ar-gument to the function:

Metaweb._3(/* JSON object goes here */)

Since JSON is a subset of the JavaScript object and array literal syntax, any JSON object is avalid function argument. By simply wrapping a function invocation around the JSON object,we've converted the mqlread response into a form suitable for use with a <script> tag.

Note that Metaweb.read() uses JSON.serialize() to serialize the query object into JSON form.This utility function is de ned in Example A.1 in Appendix A . The corresponding JSON.parse()function is not required, however, since the JavaScript interpreter that processes the <script>tag serves as our JSON parser.

Example 4.8. metaweb.js: Metaweb queries with script tags

/*** metaweb.js:** This file implements a Metaweb.read() utility function using a <script>* tag to generate the HTTP request and the URL callback parameter to* route the response to a specified JavaScript function.**/

var Metaweb = {}; // Define our namespaceMetaweb.HOST = "http://www.freebase.com"; // The Metaweb serverMetaweb.QUERY_SERVICE = "/api/service/mqlread"; // The service on that serverMetaweb.counter = 0; // For unique function names

// Send query q to Metaweb, and pass the result asynchronously to function fMetaweb.read = function(q, f) {

// Define a unique function namevar callbackName = "_" + Metaweb.counter++

// Create a function by that name in the Metaweb namespace.// This function expects to be passed the outer query envelope.// If the query fails, this function throws an exception. Since it// is invoked asynchronously, we can't catch the exception, but it serves




// to report the error to the JavaScript console.Metaweb[callbackName] = function(outerEnvelope) {

var innerEnvelope = outerEnvelope.qname; // Open outer envelope// Make sure the query was successful.if (innerEnvelope.status != "/mql/status/ok") { // Check for errors

var error = innerEnvelope.messages[0] // Get error message

throw error.status + ": " + error.message // And throw it!}var result = innerEnvelope.result; // Get result from inner envelopedocument.body.removeChild(script); // Clean up <script> tagdelete Metaweb[callbackName]; // Delete this functionf(result); // Pass result to user function

};

// Put the query in inner and outer envelopesenvelope = {qname: {query: q}}

// Serialize and encode the query object

var querytext = encodeURIComponent(JSON.serialize(envelope));

// Build the URL using encoded query text and the callback namevar url = Metaweb.HOST + Metaweb.QUERY_SERVICE +

"?queries=" + querytext + "&callback=Metaweb." + callbackName

// Create a script tag, set its src attribute and add it to the document// This triggers the HTTP request and submits the queryvar script = document.createElement("script");script.src = urldocument.body.appendChild(script);

};

You'll nd a proxy-based implementation of this same Metaweb.read() function in Example A.2in Appendix A .

4.6. mqlread ErrorsA number of things can go wrong when using mqlread . If you invoke it with incorrect URLparameters or supply a query envelope that is not valid JSON, mqlread will response with anHTTP "400 Bad Request" error. The body of the response will be a JSON object that providesdetails about the error. It might look like this:

{"status": "400 Bad request","messages": [

{"text": "JSON parse error: Expecting property name at line 1 column 1","type": "/service/error/invalid_value","field": "queries","value": "{<}\r\n","level": "error"




}]

}

This kind of error can occur if you are cutting-and-pasting raw mqlread URLs or if you are enteringMQL queries as JSON text into a query editor application. When you write scripts that use JSON

serializers and invoke mqlread using tested code, this kind of error should not occur. Errors arestill possible, however: a MQL query can be invalid, even if it is expressed using well-formedJSON and passed to mqlread using the correct URL parameters.

If you submit an invalid MQL query, mqlread returns an HTTP error code of "200 OK", but thestatus property of the inner response envelope is "/mql/status/error" . The inner responseenvelope also includes a property named messages instead of a property named result . Thevalue of the messages property is an array (usually of length 1) of message objects each of whichhas the following properties:

type A string that indicates what kind of message this is. For error messages, this is always"/mql/error".

status An identi er that names the speci c kind of error. /mql/status/parse_error and/mql/status/type_error are typical values.

Note that the status property of a message object is distinct from, and more inform-ative than, the status property of the inner response envelope.

message A human-readable description of the error

info An object that provides additional details about the error. For type errors, for example,the properties of this object specify the value and type that appeared in the queryand the type that was expected.

query A copy of the query object with the addition of a special error_inside property,to indicate where error occurs. For parse errors, this property is omitted, since thequery couldn't be property parsed.

path A string that speci es the "path" of property names from the root of the MQL queryto the the location of the error. If the error is in the outermost object of the query,then this property is just an empty string. For parse errors, this property is omitted.

4.7. mqlread Cursors

When a MQL query is to be submitted to mqlread , it is placed inside an inner query envelopeobject, as the value of a property named query . Often, this is the only property of the inner queryenvelope. But a property named cursor is also allowed.

Use a cursor when you want to retrieve results in batches from a large result set. Start by includingthis property in your inner query envelope:

cursor: true




When you do this, mqlread will include a cursor property in the inner envelope of its response.If the value of the response cursor property is false , then mqlread has returned the completeset of query results to you. If the cursor property is not false , then it will be a string containingopaque data. Take the value of this cursor property, insert it back into your inner query envelope,and send the query back to mqlread . mqlread will send you the next batch of results and willagain include a cursor property in the inner response envelope. Repeat this process until thecursor property of the response is false .

Example 4.9 is a metaweb.readall() function written in Python. It works like the metaweb.read()function of Example 4.3 , but uses a cursor to iterate through a large result set, making multiplequeries and concatenating the results into a single array before returning them. (Note that thisfunction doesn't allow any kind of parallelism: it does not allow the rst batch of results to beprocessed while the second batch is being fetched, for example. So if you're using a limit directiveand cursors to improve response time, this readall() methods is not appropriate.)

Example 4.9. metaweb.py: querying Metaweb with a cursor, in Python

import urllib # URL encoding

import urllib2 # Higher-level URL content fetchingimport simplejson # JSON serialization and parsing

host = 'www.freebase.com' # The Metaweb hostreadservice = '/api/service/mqlread' # Path to mqlread service

# Submit the MQL query q and return the result as a Python object# This function behaves like read() above, but uses cursors so that# it works even for very large result setsdef readall(q, credentials=None):

# This is the start of the mqlread URL.# We just need to append the envelope to it

urlprefix = 'http://%s%s?queries=' % (host, readservice)

# The query and most of the envelope are constant. We just need to append# the encoded cursor value and some closing braces to this prefix stringjsonq = simplejson.dumps(q);envelopeprefix = urllib.quote_plus('{"q0":{"query":'+jsonq+',"cursor":')

cursor = 'true' # This is the initial value of the cursorresults = [] # We accumulate results in this array

# Loop until mqlread tells us there are no more resultswhile cursor:

# append the cursor and the closing braces to the envelopeenvelope = envelopeprefix + urllib.quote_plus(cursor + '}}')# append the envelope to the URLurl = urlprefix + envelope

# Begin an HTTP request for the URLreq = urllib2.Request(url)

# Send our authentication credentials, if any, as a cookie.




# The need for mqlread authentication is a temporary restriction.if credentials:

req.add_header('Cookie', credentials)

# Read and parse the URL contentsf = urllib2.urlopen(req) # Open URL

response = simplejson.load(f) # Parse JSON responseinner = response['q0'] # Get inner envelope from outer

# Raise a MQLError if there were errorsif inner['status'] != '/mql/status/ok':

error = inner['messages'][0]raise MQLError('%s: %s' % (error['status'], error['message']))

# Append this batch of results to the main array of results.results.extend(inner['result']);

# Finally, get the new value of the cursor for the next iteration

cursor = inner['cursor']if cursor: # If it is not false, put it

cursor = '"' + cursor + '"' # in quotes as a JSON string

# Now that we're done with the loop, return the results arrayreturn results

It is important to understand that cursors only work when multiple results are expected at thetop-level of the query. The cursor property is part of the mqlread envelope syntax, not part of the MQL query language, and it cannot be applied to sub-queries of a query. Another way to saythis is that it only makes sense to include "cursor":true in an envelope if the rst characterfollowing "query": in the envelope is [ . The query must be expressed as an array in order for a

cursor to be meaningful.

Consider the code in Example 4.4 . It contains this query:

query = { 'type': '/music/artist', # Our MQL query in Python'name': band,'album': [{ 'name': None, # None is Python's null

'release_date': None,'sort': 'release_date' }]}

This is a perfectly valid query, and works just ne in Example 4.4 . But suppose we wanted toport that script to use the metaweb.readall() function de ned above. To do this, we'd also haveto alter the query so that the array of albums was at the top level of the query:

query = [{'type': '/music/album','artist': band,'name': None,'release_date': None,'sort': 'release_date','limit': 10}]




Note that we've added an explicit limit directive to this modi ed query. In general, it makessense to specify an explicit limit when using cursors.

4.8. Fetching Content with transAs explained in Chapter 2 , Metaweb is really two databases in one. One database is the graph of nodes and relationships. The second is the content store that holds chunks of data such as HTMLdocuments and graphical images. We use mqlread service to retrieve data from the graph, andwe use the trans service to retrieve content from the content store.

The trans service is so named because in addition to fetching the requested data, it can alsotranslate it for you. For example, it can "translate" an image to thumbnail size.

The trans service is HTTP based, just as mqlread is. Content is retrieved by specifying the desiredtranslation and the content id, with a URL of this form:

http://www.freebase.com/api/trans/ translation / guid

Here, for example, is an actual trans URL at freebase.com :

http://www.freebase.com/api/trans/raw/%239202a8c04000641f8000000003c1978c

The translation portion of a trans URL must be one of the following:

raw Use raw to request that no translation is to be done on the data: it should bereturned as is. (Note, however that HTML content is not completely raw: itis "sanitized" by stripping executable content such as JavaScript.)

image_thumb Use image_thumb to request a thumbnail-sized version of an image.

blurb Use blurb to request an excerpt from the beginning of a document. Thisprovides a kind of a preview, of the kind you might see in a list of searchresults.

The path component that follows the translation is the URL-encoded version of a Metaweb guid.%23 is the encoding of the # character, and the letters and digits that follow are the hexadecimaldigits of the guid. The guid passed to trans must identify an object of type /type/content ,/common/image or /common/document . These three types are closely related:

/type/content A /type/content object is the representation in the Metaweb graphof an entry in the Metaweb content store.

/common/image When an image is added to the content store, the /type/content objectfor the image is co-typed /common/image , in order to add a sizeproperty that supplies the image dimensions. For images, therefore,the guid of the /type/content and /common/image objects are thesame.

/common/document When document content is added to the content store, a /type/contentobject is created to represent the entry in the content store. A separate




/common/document object is also created. The content property of thedocument object refers to the content object. Other properties of the/common/document object provide additional meta-information aboutthe document.

/common/document objects can also represent Wikipedia documentcontent (which is not stored in the Metaweb content store). Documentsthat represent Wikipedia entries have content properties of null .

Given the guid of a document object, the trans service returns thecontent of both Wikipedia and non-Wikipedia documents. For non-Wikipedia documents, you can use either the guid of the /common/doc-ument object or of the /type/content object it refers to.

The trans service does not support a callback parameter as the mqlread service does, so youcannot use it with <script> tags. If you implement a proxy on your own web server, then youcan invoke the trans service indirectly to retrieve content with XMLHttpRequest , however.

It is usually easier, however, to use the trans service with <img> and <iframe> tags. To retrieveand display an image, simply use a trans URL as the src attribute of an <img> tag. And to retrieveand display the HTML content of a document, use a trans URL as the src attribute of an <iframe> .

Like mqlread , the trans service requires cookie-based authentication during the freebase.comroll-out period. The examples in this chapter assume that you are using the trans service in a webbrowser that has visited and logged on to www.freebase.com .

HTML Content, <iframe> tags, and Security

When you display Metaweb content in an <iframe> , the origin of the framed content isdifferent from the origin of your web application. This means that the same-origin security-rules apply. The user of your web application can see the framed Metaweb content, butJavaScript code in your application cannot access this content.

Metaweb strips executable code from HTML before returning it to you, but if any JavaScriptcode were somehow to make it past Metaweb's sanitizer, that code would also be subjectto the same-origin policy and would be unable to interact with your web application content.In this way, using an <iframe> for content fetched with trans gives you an extra layer of security.

4.8.1. Browsing Recent Content on freebase.comExample 4.10 is a JavaScript-based example that demonstrates the use of the trans service, andthe raw , image_thumb , and blurb translations. It uses mqlread to nd the ten images and tendocuments most recently added to freebase.com . It then generates <img> and <iframe> tags withtrans URLs to display thumbnails for the images and blurbs for the documents. It also generateshyperlinks so that the thumbnails and blurbs are linked to full-sized versions of the images anddocuments. (These links open new windows to display the image or document.)




Example 4.10 does not use the Metaweb.read() utility function developed earlier in the chapter.Instead, it de nes a variant of that function called sendQueries() . This sendQueries() functionsends multiple queries, in multiple inner envelopes bundled together into a single outer envelope.This means that the example can ask for recent images and recent documents in a single invocationof mqlread . If the example had used Metaweb.read() , it would have had to invoke mqlread twice.Remember that you must log on to www.freebase.com before using this example.

Example 4.10. WhatsNew.html: fetching new images and documents from freebase.com

<head><script src="json.js"></script><script>// These are a few important constantsvar HOST = "http://www.freebase.com";var READ = "/api/service/mqlread";var RAW = "/api/trans/raw/";var THUMB = "/api/trans/image_thumb/";var BLURB = "/api/trans/blurb/";

/*** Send the queries named in the outer envelope object to Metaweb,* and pass the outer response envelope to the function f. This is a* variant of the Metaweb.read() function that runs multiple queries.*/

function sendQueries(queryEnvelope, f) {// Define a unique function namevar callbackName = "_" + sendQueries.counter++

// Create a function by that name, using sendQueries as a namespace.// This function expects to be passed the response to the query

sendQueries[callbackName] = function(responseEnvelope) {document.body.removeChild(script); // Remove <script> tagdelete sendQueries[callbackName]; // Delete this functionf(responseEnvelope); // Pass response to user function

};

// Serialize and encode the query objectvar queries = encodeURIComponent(JSON.serialize(queryEnvelope));

// Build the URL using encoded query text and the callback namevar url = HOST + READ + "?queries=" + queries +

"&callback=sendQueries." + callbackName

// Create a script tag, set its src attribute and add it to the document// This triggers the HTTP request and submits the queryvar script = document.createElement("script");script.src = urldocument.body.appendChild(script);

};sendQueries.counter = 0; // Initialize the counter




// How many images and how many documents do we display?var N = 10; // This is the defaultif (window.location.search.substring(0,3) == "?n=") // URL argument overrides

N = parseInt(window.location.search.substring(3));

// These are the queries we issue to find the n newest images and documents

var queries = {images: {

query: [{type:"/common/image", id:null, // Return image idstimestamp:null, sort:"-timestamp", // Most recent firstlimit:N, // Only N of them"/type/content/media_type":null, // Check image type, too"/type/content/media_type|=":[ // We only want images that are:

"/media_type/image/gif", // GIF or"/media_type/image/png", // PNG or"/media_type/image/jpeg" // JPEG

]

}]},docs: {

query: [{type:"/common/document", id:null, // Return document idstimestamp:null, sort:"-timestamp", // Most recent firstlimit:N // Only N of them

}]}

};

// When the document has loaded, send the queries above to freebase.com.

// Then call the function below with the resultswindow.onload = function() { sendQueries(queries, displayResults) }

// This function gets called with our query resultsfunction displayResults(response) {

// First, display image thumbnailsvar images = response.images.result; // Array of imagesvar container=document.getElementById("newimages"); // Where they gofor(var i = 0; i < images.length; i++) { // Loop through them

var id = encodeURIComponent(images[i].id); // Image id in URL form

var thumbnail = document.createElement("img"); // Create <img> tag

thumbnail.src = HOST + THUMB + id; // url for image thumbnailthumbnail.title = images[i].timestamp; // timestamp as tooltip

var link = document.createElement("a"); // Hyperlink for imagelink.href = HOST + RAW + id; // to a full-size imagelink.target = "_new"; // displayed in a new window

link.appendChild(thumbnail); // Put thumbnail inside linkcontainer.appendChild(link); // Put link inside container


}

// Next display document blurbsvar docs = response.docs.result; // Array of documentscontainer = document.getElementById("newdocs"); // Where they gofor(var i = 0; i < docs.length; i++) { // Loop through them

var id = encodeURIComponent(docs[i].id); // Doc id in URL formvar blurb = document.createElement("iframe"); // Create an iframeblurb.src = HOST + BLURB + id; // to hold doc blurbvar link = document.createElement("a"); // Hyperlinklink.href = HOST + RAW + id; // To full documentlink.target = "_new"; // In a new windowlink.innerHTML = docs[i].timestamp; // Timestamp as link textvar listitem = document.createElement("li"); // Create list itemlistitem.appendChild(blurb); // Put blurb in itemlistitem.appendChild(link); // Put link in itemcontainer.appendChild(listitem); // Put item in container

}

}</script><style> /* Make it all look good with a stylesheet */img { margin: 5px;}iframe { width: 70%; height: 75px; vertical-align: top;}li a { vertical-align: bottom; }h2 { margin-bottom: 5px; }</style></head><body><h2>The Newest Images</h2><i>Click thumbnail for full-size image</i><div id="newimages"></div>

<h2>The Newest Documents</h2><i>Click timestamp for full document</i><ol id="newdocs"></ol></body>

4.9. Example: A Metaweb Type BrowserThis chapter concludes with one nal 1 example. Example 4.11 is a JavaScript-based web applic-ation for browsing Metaweb types. Figure 4.2 shows a sample page that displays informationabout the type /type/type . Clicking on the id of another type (or typing a type id in the upperright) displays information about that type. You may actually nd this type browser quite usefulfor exploring Metaweb system types and the types in other domains. Remember, though, that

you must log on to www.freebase.com before using the example.

1 If you want more, Section A.3 is a JavaScript-based example that demonstrates Metaweb-powered autocompletion for HTML textelds.

Figure 4.2. A Metaweb type browser

This example is notable because it uses a more complicated query than the other queries in thischapter. Example 4.11 uses the result data to generate a page of information about the speci edtype. This example is also notable because its HTML output is more complex than previous ex-

amples. The code is well-commented, and if you've understood previous JavaScript examples,you should not have trouble following this one.

Example 4.11. TypeBrowser.html: a Metaweb type browser

<html><head><script language="javascript" src="json.js"></script><script language="javascript" src="metaweb.js"></script><script language="javascript">// This is the query we need to get information about a type.// Note that we have to fill in the type we're interested in// before sending this query.var query = {

type:"/type/type", // The type of our type is /type/type :-)id:null, // The type we're asking about. Filled in below.name:null, // What is the human-readable type name?// Objects with documentation are co-typed /freebase/documented_object// Here we ask for a short description of the type"/freebase/documented_object/tip":null,



properties:[{ // What properties does this type have?optional:true,name:null,key:[],expected_type: {name:null, id:null},unique:null

}],expected_by:[{ // What properties are of this type?

optional:true,name:null,key:[],schema: {name:null, id:null}

}],instance:[{ // What are some instances of this type?

optional:true,id:null,name:null

}]

};

// When we're first loaded, display /common/topic, or the type// specified by the ?t= argument in the URLwindow.onload = function() {

var type = "/common/topic";var search = window.location.search;if (search && search.indexOf("?t=") == 0)

type = decodeURIComponent(search.substring(3));queryType(type);

}

// Query the specified type. Call displayType() when the results arrivefunction queryType(type) {query.id = type; // Specify the type in the query aboveMetaweb.read(query, // Issue the query

displayType); // Pass result object to displayType}

// Generate a page of information based on our query results.function displayType(result) {

// DOMStream is a helper class defined below// We use it here to output HTML text to the placeholder elementvar out = new DOMStream("placeholder");

out.clear()

// If we didn't get any results then the input was invalidif (!result) {

out.write("No such type");out.flush();return;

}




// Now begin generating information about the typeout.write("<h1>", result.id, "</h1>"); // Titleout.write("<h2>Name</h2>", result.name); // Sectionout.write("<h2>Description</h2>"); // Another sectionvar tip = result["/common/documented_object/tip"];if (tip) out.write(tip);

else out.write("No description available");

// Display a table of propertiesout.write("<h2>Properties</h2>")if (result.properties.length == 0) out.write("No properties");else {

out.write('<table border=1><tr>','<th>Property Name</th>','<th>Property Key</th>','<th>Property Type</th></tr>');

for(var i = 0; i < result.properties.length; i++) {

out.write('<tr><td>', result.properties[i].name,'</td><td>', result.properties[i].key.join(", "),'</td><td>');

if (result.properties[i].unique) out.write("unique ");displayTypeLink(out, result.properties[i].expected_type.id,

result.properties[i].expected_type.name);out.write('</td></tr>');

}out.write("</table>");

}

// Display the properties of other types that use this type

out.write("<h2>Used by</h2>")if (result.expected_by.length == 0)out.write("There are no Properties of this type.");

else {out.write('<table border=1><tr>',

'<th>Type</th>','<th>Property Key</th>','<th>Property Name</th>','</tr>');

for(var i = 0; i < result.expected_by.length; i++) {out.write('<tr><td>');

displayTypeLink(out, result.expected_by[i].schema.id,result.expected_by[i].schema.name);out.write('</td><td>', result.expected_by[i].key.join(", "),

'</td><td>', result.expected_by[i].name,'</td><tr>');

}out.write("</table>");

}




// Output a list of the names of instances of this typeout.write("<h2>Instances</h2>");if (result.instance.length == 0) out.write("No instances");else {

for(var i = 0; i < result.instance.length; i++)

out.write(result.instance[i].name, ", ");}

// Calling flush makes the output visible on the pageout.flush();

}

// Output a link to a type. Use the type id as the link text, and// make the type name available as a tooltipfunction displayTypeLink(out, id, name) {

out.write('<a title="', name, '" onclick="queryType(\'', id, '\')">',id, '</a>');

}

// This little DOMStream class writes HTML into the element we specifyfunction DOMStream(id) { // Constructor function

this.elt = document.getElementById(id);this.buffer = [];

}DOMStream.prototype.clear = function() { // Erase element content

this.elt.innerHTML = "";};DOMStream.prototype.write = function() { // Buffer up all arguments

this.buffer.push.apply(this.buffer, arguments);

};DOMStream.prototype.flush = function() { // Output all text to the elementthis.elt.innerHTML += this.buffer.join("");this.buffer.length = 0;

};</script>

<style>/* Some CSS styles to make everything look good */body {

font-family: Arial, Helvetica, sans-serif; /* We like sans-serif */margin-left: .5in; /* Indent everything... */

}h1, h2 { margin-left: -.25in; } /* ...except headings */h2 { margin-bottom: 5px; margin-top:10px; }/* Make tables look nice */table { border-collapse: collapse; width: 95%;}th { background-color: #aaa;}td { background-color: #ddd; padding: 1px 5px 1px 5px; }

/* Our <a> tags don't have hrefs, so we need to style them ourselves */


a { color: #00a; }a:hover { text-decoration:underline; cursor:pointer;}

/* Make the input field look nice */form.inputform {

float:right; border: solid black 2px; background-color: #aba;

margin: 15px 30px 0px 0px; padding: 10px;}</style></head><body><form class='inputform' onsubmit="queryType(this.t.value); return false;">Enter type id: <input name="t"></form><div id="placeholder"></div></body></html>



Chapter 5. The MQL Write GrammarInsertions, deletions and updates to a Metaweb database are expressed in a variant of the MetawebQuery Language documented in Chapter 3 . The variant used for writing to Metaweb is knownas the MQL write grammar, and is the subject of this chapter. The chapter begins with a longtutorial introduction to MQL writes, and then speci es the MQL write grammar more formally.Write queries are submitted to Metaweb via the mqlwrite service, which is covered in Chapter 6 .

5.1. MQL Write TutorialMQL writes are represented as JSON objects, just as MQL reads are. A number of features of the MQL read grammar only make sense for reads and are not allowed in MQL writes. Theseinclude the use of [] to query an array of values, and the use of the sort , limit and optionaldirectives. MQL write grammar supports two directives that are not allowed for reads. The createdirective is used to create a new object in the database, and the connect directive is used to create

a link between two objects. (As we'll see in the tutorial, however, theconnect

directive issometimes implicit and need not be speci ed explicitly).

Using this Tutorial

The best way to follow this tutorial is to try out the queries as you read about them. Whileyou learn how to make MQL writes, please use the freebase.com sandbox server at ht-tp://sandbox.freebase.com . This server is intended for experimentation. The sandbox hostsa replica of freebase.com , and this replica is re-created approximately once a week. Thismeans that any writes you perform (or mistakes you make!) on the sandbox will not persistlonger than a week.

Anyone can read data from Freebase, but before you can execute write queries, you mustregister for an account and login. If you already have an account at www.freebase.com , itmay already have been replicated on the sandbox server. If not, you can create a new accountfor yourself on the sandbox. Follow the links from the http://sandbox.freebase.com/ homepage to register. (If you used an invitation code when you registered at www.free-base.com , just reuse it if you need to register on the sandbox.freebase.com .)

Once you are logged on to the sandbox, you can execute MQL write queries using the theFreebase query editor at http://sandbox.freebase.com/view/queryeditor/ . Enter queries fromthis tutorial, click the write button, and view the results. Just as with reads, you must placeyour MQL write queries in an "envelope" object. So instead of entering queries as they arewritten in this chapter, you must pre x them with {"query": and end them with an extraclosing } .

5.1.1. Creating a Type to Work WithBefore we do any explicit MQL writes, let's begin by creating a simple type to work with. Bycreating and using your own type, you guarantee that the writes you try while working throughthis tutorial won't interact with writes being issued by other developers who may be working on

95



the tutorial at the same time. As you know, Metaweb types are de ned by regular Metaweb objectsin the database. This means that types are created like any other objects, with MQL queries. De-

ning a type with raw MQL is dif cult and error prone, however, so just about everyone de nestypes using the freebase.com client.

The type we're creating will represent musical notes, and we'll call it "note". In order to createit, follow the "My Freebase" link from the freebase.com home page. On the My Freebase page,click on "Types Created", and enter the name "Note". Figure 5.1 illustrates.

Figure 5.1. Creating a new type on freebase.com

That's all you need to do for now. We'll add some properties to this type later, but now we justneed the type itself. If you click on the name of the newly created type (note that the freebase.comclient capitalizes the name for you) and look at the URL that it takes you to, you'll see that thename of the new type is /user/username/default_domain/note , where "username" is the user-name you logged in with. In the tutorial that follows, you'll see the username docs , but you shouldsubstitute your own name throughout.




5.1.2. Creating ObjectsLet's begin with a very simple write query:

ResultWrite

{"create":"created",

{"create":"unless_exists",

"type":"type":"/user/docs/default_domain/note","/user/docs/default_domain/note",

"name":"A","name":"A","id":"#1f8000000000037ffc"

}"id":null

}

The rst line of the query says that we want to create a new object, unless a matching objectalready exists. The second line speci es the type of the object we're creating (remember to sub-stitute your own user name for "docs" here). The third line speci es a value for the name property

of the new object. The fourth line of the write query is a request for the id of the newly createdobject. Asking for an id is the only way you are allowed to use null in a write query. You maynot use null or [] for any other property.

Now let's look at the response to the write query. The rst line is the create property, but itsvalue has changed from unless_exists to created . This tells us that the object we speci ed didnot already exist, and Metaweb has created it for us. The second and third lines simply repeatthe type and name properties that we passed in. They don't provide any new information, butmaintain the MQL invariant that responses have the same properties as queries. Finally, the fourthline returns the id of the newly created object.

Metaweb IDs in this TutorialThe guids used in this tutorial were created on the sandbox server, and are no longer valid,so you should not try to query these objects directly. Instead, substitute your own user nameinto the write queries, and create your own objects, with their own guids, as you followalong with this tutorial. If you are reading a printed or PDF version of this chapter, notethat the guids have been shortened so that they t more easily in two-column format.

Now let's see what happens if we run exactly the same query again:

ResultWrite

{"create":"existed",


"type":"type":"/user/docs/default_domain/note","/user/docs/default_domain/note",

"name":"A","name":"A","id":"#1f8000000000037ffc"

}"id":null

}

97Chapter 5.The MQL Write Grammar



We're asking that an object be created unless it already exists. And this time it does already exist.So Metaweb returns the existed as the value of the create property, and returns the id of thealready existing object. Note that this id is the same as the one we've already seen.

Now let's force Metaweb to create another new test object for us:

ResultWrite{

"create":"created",{

"create":"unconditional","type":"type":

"/user/docs/default_domain/note","/user/docs/default_domain/note","name":"A","name":"A","id":"#1f800000000003800f"

}"id":null

}

In this query, we've changed the value of the create directive to unconditional . As its nameimplies, this value tells Metaweb to create a new object no matter what. Since a new object iscreated unconditionally, the value of the create property in the response will always be created .You can see that a new object was created by comparing the id returned by this query to thosereturned by the previous two queries.

When to use create:unconditional

It is almost never necessary or correct to use "create":"unconditional" in a MQL writequery. Most writes use "create":"unless_exists" , and many others use "create":"un-less_connected" , which has not been introduced yet. Using "unconditional" leads to du-plicate objects, and as we'll see below, this can get you into trouble!

We now have two note objects with the name "A". What happens if we run the original unless_ex-ists write again?

{"status": "400 Bad request","messages": [

{"query": {

"create": "unless_exists","type": "/user/docs/default_domain/note","name": "A","error_inside": ".",

"id": null},"text": "Need a unique result to attach here, not 2","args": {

"count": 2,"guids": [

"#1f8000000000037ffc","#1f800000000003800f"




]},"type": "/service/error/result","level": "error"

}],

"result": {"q": null

}}

The query fails this time, and returns the JSON object shown above. The "create":"unless_ex-ists" directive works only if there are 0 or 1 instances of the object. If there is no object thatmatches, it creates one. If there is one object that matches, it returns it. But if there are more thanone, it has no way to choose which one to return, and fails with an error message. Note that thequery fails even if we omit "id":null . The lesson here is that if you plan to use unless_exists ,you should use it consistently so you never end up with more than one instance of an object.

5.1.3. Connecting ObjectsSo far we've created two distinct objects with identical types and names. Let's now rename oneso we can tell them apart by name. Recall that an object is named by linking it to a primitivevalue of /type/text . We want to update the name link to refer to a different value:

ResultWrite

{"id":"#1f800000000003800f",

{"id":"#1f800000000003800f",

"name":{"name":{"connect":"updated","connect":"update",

"value":"B","value":"B","lang":"/lang/en""lang":"/lang/en"

}}

}}

The rst line of the query identi es, by id, the object we want to modify. The second and thirdlines specify that want to update the name property of that object so that it refers to the /type/textvalue speci ed by the 4th and 5th lines. (Recall that /type/text is a primitive value that consistsof a string of text and a language identi er for that text. MQL write queries require you to specifyboth the value and lang properties when manipulating a name.)

The response looks just like the query except that the value of the connect property has changedto updated . This tells us that the update we requested has been performed.

What happens if we run exactly the same write query again?




ResultWrite

{"id":"#1f800000000003800f",

{"id":"#1f800000000003800f",

"name":{"name":{"connect":"present","connect":"update","value":"B","value":"B","lang":"/lang/en""lang":"/lang/en"

}}

}}

We're asking to make a change that has already been made, and Metaweb lets us know this bysetting the connect property of the response to present .

We now have two newly-created objects with the same type and different names. We changedthe name of the second object by updating a /type/text value. /type/text is a primitive typein Metaweb, so this isn't quite the same thing as a link between two different objects in the data-base. Now, let's modify the rst object (the note A) so that it is a /common/topic in addition tobeing a note:

ResultWrite

{"id":"#1f8000000000037ffc",

{"id":"#1f8000000000037ffc",

"type":{"type":{"connect":"inserted","connect":"insert","id":"/common/topic""id":"/common/topic"

}}

}}

The rst line of the query speci es the object to be modi ed. (Normally, we'd identify the objectby name and type, but we can't specify the type and add a type in the same query. The name "A"is probably not unique by itself, so we specify the object we want to modify by id.) The secondand third lines specify that we want to insert a new connection between this object and anotherobject, and that this new connection should use the type property. The fourth line speci es, byid, the object that is being connected to.

Note that the value of the connect directive is insert instead of update , which is what we usedabove. The difference between the two is simple. Use "connect":"update" for properties thathave a unique value (and for the name property, which is unique on a per-language basis). Use"connect":"insert" for properties, such as type , that can have more than one value. You are

also allowed to use "connect":"insert" with unique properties if there is not already a valuefor that property.

The response object sets the value of the connect directive to inserted , telling us that the insertionwas successful. Our note named "A" is now also a /common/topic . If you visit your "My Freebase"page, the note object should now be visible under the "Topics Created" heading. This is the mainreason to use the /common/topic type on the objects you create: it allows them to work well withthe freebase.com client.




What happens if we run the same query again?

ResultWrite

{"id":"#1f8000000000037ffc",

{"id":"#1f8000000000037ffc",

"type":{"type":{"connect":"present","connect":"insert","id":"/common/topic""id":"/common/topic"

}}

}}

We're asking to insert /common/topic into a set of types that already includes /common/topic ,and we get the response present . It tells us that this value is already in the set and that nothinghas changed. (Non-unique properties in Metaweb are like sets: they do not allow duplicates.)

Let's do a quick read query to con rm that our object is a member of two types:

ResultRead

{"id":"#1f8000000000037ffc",

{"id":"#1f8000000000037ffc",

"type":["type":[]} "/user/docs/default_domain/note",

"/common/topic"]

}

So we see that our object is, in fact, a note and a topic.




5.1.4. Disconnecting ObjectsWe've seen that Metaweb allows us to connect objects with "connect":"insert" or "con-nect":"update" . To disconnect objects, use "connect":"delete" . Let's alter the object thatrepresents the note A again, to remove /common/topic from its set of types:

ResultWrite

{"id":"#1f8000000000037ffc",

{"id":"#1f8000000000037ffc",

"type":{"type":{"connect":"deleted","connect":"delete","id":"/common/topic""id":"/common/topic"

},}

}}

This query looks just like the query we used to add the type, except that we've changed "insert"

to "delete". And Metaweb's response looks just like the response to the insertion, except that"inserted" has changed to "deleted". You can verify that the object is no longer a /common/topicby visiting "My Freebase" on sandbox.freebase.com and noting that it no longer appears in the"Topics Created" list.

At this point, you probably have a pretty good idea what will happen if we re-run the query:

ResultWrite

{"id":"#1f8000000000037ffc",

{"id":"#1f8000000000037ffc",

"type":{"type":{"connect":"absent","connect":"delete","id":"/common/topic""id":"/common/topic"

}}

}}

We asked Metaweb to remove /common/topic from a set that did not contain /common/topic ,so it returned "absent" to indicate that nothing has been changed.

The MQL write grammar has no syntax for deleting objects themselves. The closest thing to de-leting an object is to delete all connections from that object to others. If an object has no type,no name, and no other properties of interest, then it becomes effectively unreachable, and is almostas good as gone. Note, however, that Metaweb maintains a modi cation history for each object.When you view an object in the freebase.com client, you'll see a "History" link at the bottom of each page. Clicking this link allows you to view the change history for the object, and allowsyou to undo changes, including deletions.

When an object has had all its links deleted, it can still be queried by guid or creator (Metawebdoes not allow these read-only properties to be deleted.) In practice, however, unreachable objectswill only be found by determined searchers, and their continued existence is very unlikely to affect




the results of future queries. Unreachable objects may at some point be purged from a Metawebdatabase, but their guids will never be reused.

Let's use this unlinking technique to "delete" the two note objects we've created:

ResultWrite

[{"id":"#1f8000000000037ffc",

[{"id":"#1f8000000000037ffc",

"type":{"type":{"connect":"deleted","connect":"delete","id":"/user/docs/default_domain/note""id":"/user/docs/default_domain/note"

},},"name":{"name":{

"connect":"deleted","connect":"delete","value":"A","value":"A","lang":"/lang/en""lang":"/lang/en"

}}},{},{

"id":"#1f800000000003800f","id":"#1f800000000003800f","type":{"type":{

"connect":"deleted","connect":"delete","id":"/user/docs/default_domain/note""id":"/user/docs/default_domain/note"


"connect":"deleted","connect":"delete","value":"B","value":"B","lang":"/lang/en""lang":"/lang/en"

}}]

}}]

Note that this write query is really two separate queries, included within square brackets. Themqlwrite service (the topic of Chapter 6 ) accepts submissions of multiple writes at once. Notethat names are deleted with "connect":"delete" , even though they are unique and were originallycreated with "connect":"update" . You must specify the lang property explicitly when deletinga name.

As a nal test, let's query the rst of these objects (by id) and nd out what little information itstill carries:

ResultRead

{"id":"#1f8000000000037ffc",

{"id":"#1f8000000000037ffc",

"guid":"#1f8000000000037ffc","*":null} "name":null,

"type":[],"key":[],"creator":"/user/docs","permission":"/boot/all_permission","timestamp":"2006-11-08T20:00:02.0000Z"

}




As expected, the name and types of the object are gone. All that remains are its id, creator, creationtimestamp, and permissions.

Example 6.7 in Chapter 6 is a command-line script for unlinking Metaweb objects in this way.Its default behavior is to delete all objects you have created. You may nd this script useful towipe your slate clean while experimenting with MQL writes.

Multiple Queries and Atomicity

When you submit multiple top-level write queries to Metaweb at the same time, it is naturalto ask whether they are executed in order, and whether a query can depend on an objectcreated by a previous query. The answer to both questions is no. The reason is a good one,however: when multiple queries are submitted at the same time, they are executed atomically:all are executed or none are executed.

In order to implement this atomic behavior, the Metaweb server rst tests each query todetermine whether it will succeed. It does this without actually executing the query. If all

queries pass the test, then all are executed. Note, however, that this means that each querymust be able to succeed before any other queries have been run. Therefore, the queriesmust be completely independent of each other. And since they are independent, there isreally no way to tell what order Metaweb executes them in.

5.1.5. Writes and Default PropertiesTake a look again at the MQL write queries we use to create and "delete" Note objects. First, thecreation:

ResultWrite



"type":"/user/docs/default_domain/note","type":"/user/docs/default_domain/note","name":"C#","name":"C#","id":"#1f800000000104befe"

}"id":null

}

Now contrast this with the query that "deletes" the object by unlinking its type and name:

ResultWrite

{

"id":"#1f800000000104befe",

{

"id":"#1f800000000104befe","type":{"type":{

"connect":"deleted","connect":"delete","id":"/user/docs/default_domain/note""id":"/user/docs/default_domain/note"


"connect":"deleted","connect":"delete","value":"C#","value":"C#","lang":"/lang/en""lang":"/lang/en"




ResultWrite

}}

}}

The creation query is much more compact because we are able to specify the type as a single idand the name as a single string. In the deletion query, we must specify the expanded objects.There are three factors that interact to make the creation query shorter. First, recall from Chapter 3that every type has a default property . For value types such as /type/text (the type of the nameproperty) the default property is value . For core types in the /type domain, the default propertyis id . For all other types, the default property is name . So in the creation query,"type":"/user/docs/default_domain/note" is shorthand (but see the caution below!) for:

"type": { "id":"/user/docs/default_domain/note" }

The second factor that makes the creation query so compact is the fact that when you specify adefault property rather than a full object in a MQL write query, Metaweb assumes an implicit"connect":"insert" . So writing "type":"/user/docs/default_domain/note" is kind of (butnot exactly: see the caution that follows) like writing:

"type": {"connect":"insert","id":"/user/docs/default_domain/note"

}

The third factor that makes the creation query compact is that the language of /type/text valuesis automatically set to the default of English, or to your preferred language as speci ed by aparameter to the mqlwrite service. (See Chapter 6 for details.)

All three factors come into play when we write "name":"C#" . "C#" becomes the value of the

default property, which is the value . An implicit "connect":"insert" is added. And a langproperty is added to specify /lang/en , or whatever language we are using. So "name":"C#" ex-pands to (but see the caution!):

"name": {"connect":"insert","value":"C#","lang":"/lang/en"

}




Caution: unless_exists with Expanded Objects

From the explanation above, you might assume that the compact creation query with whichwe began this section could be equivalently (but less compactly written) as:

{"create":"unless_exists","id":null,"name": "C#","type": {

"connect":"insert","id":"/user/docs/default_domain/note"

}}

If the queries used "create":"unconditional" then they would be the same. But themeaning of unless_exists is different for the two queries. The original compact querycould be translated as If you can f nd a Note object named "C#", return its id. Otherwise,create a new Note object, name it "C#", and return its id.

But this variant that expands the type property is different in a subtle but important way.It tells Metaweb: f nd or create an object named "C#", and then add Note to its set of types .The difference between the two queries is critical if there is already an object (of type/programming/language , perhaps) with the name "C#".

Here's another way to think about this. When the type is speci ed by id, this is a constrainton the query. Metaweb must nd an object that matches, or must construct one. When thetype is speci ed in a sub-query with an explicit connect directive the sub-query is not aconstraint, and does not affect the results of the unless_exists search.

5.1.6. Creating and Connecting More ObjectsLet's try some more advanced examples. Before we start, though, we need to add a property toour Note type. Here's how you do it ( Figure 5.2 illustrates):

• Visit your "My Freebase" page on sandbox.freebase.com and click on the Note type underTypes Created .

• Click on the "Edit Type" link.

• Click on the Add a New Property button, enter the property name "next" into the text eldthat appears, and click the Save button .

• Freebase now allows you to enter details about the property:

• Enter the type name "Note" into the Expected Type eld (you may see a drop-down listcontaining your version of the Note type and many other developer's versions. Select theone that is followed by your username in parentheses.




• Click the Restrict to one value checkbox to indicate that the property may have only asingle value.

Figure 5.2. Adding a property on freebase.com

The next property we've just added to our note type allows us to link one note to another in achain or a ring. We'll use this property to link each note to its perfect fth--the note that is 7semitones higher (usually, this is 5 white keys on a piano keyboard, which is probably why it iscalled a fth.) If we start with the note C, we nd that it's fth is the note G. Before we start usingthe next property to represent fths, however, let's run a simple query that will give us a convenientshortcut:

ResultWrite

{"id":"/user/docs/default_domain/note",

{"id":"/user/docs/default_domain/note",

"key":{"key":{

"namespace":"/user/docs","connect":"insert", "connect":"inserted","namespace":"/user/docs","value":"note""value":"note"

}}

}}

This query speci es our Note type object by id, and then adds a new /type/key value to its keyproperty. What we've done is to make /user/docs/note a synonym for /user/docs/default_do-




main/note . You may nd this a helpful shortcut as you type in the example queries that follow.We'll explore namespaces again later in this tutorial.

Now, let's create Note objects to represent the notes C and G. Note that the following query istwo independent queries in an array:

ResultWrite[{

"create":"created",[{

"create":"unless_exists","id":"#1f80000000000384b0","id":null,"type":"/user/docs/note","type":"/user/docs/note","name":"C""name":"C"

},{},{"create":"created","create":"unless_exists","id":"#1f80000000000384b4","id":null,"type":"/user/docs/note","type":"/user/docs/note","name":"G"

}]"name":"G"

}]

We've asked Metaweb to create two Note objects, with names C and G, and to return their ids tous. Now, let's insert the link that indicates that G is the fth of C:

ResultWrite

{"id":"#1f80000000000384b0",

{"id":"#1f80000000000384b0",

"/user/docs/note/next":{"/user/docs/note/next":{"connect":"inserted","connect":"update","id":"#1f80000000000384b4""id":"#1f80000000000384b4"

}}}}

This compact query identi es both note objects by id and connects them with a connect directive.Since we de ned the next property to be unique, it uses "connect":"update" instead of "con-nect":"insert" . Note that since this query never speci es the type of the objects, we must usea fully-quali ed property name for the next property. You can verify that this query did whatwe intended using the freebase.com client. Visit My Freebase on sandbox.freebase.com , andclick on the Note type. On the page for the Note type, you should see a list of instances of thattype. Click on the one named "C", and you'll see that it includes a hyperlink to the note G labeled"Next".

The linking technique shown above is straightforward and easy to understand. It uses one queryto create (or look up) the two objects to be linked. Then it uses a second simple query to connectthe two objects. It is usually possible, however, to combine the creation and linking into a singlequery. The following query, for example, sets the next property of the note G to a newly-creatednote named D:




ResultWrite

"next":{"create":"unless_connected",

"next":{"create":"created",

"type":"/user/docs/note", "type":"/user/docs/note","name":"B flat" "name":"B flat"

}}

}}

This query creates a new note E f at, and connects it to B f at. Notice, however, that in the nestedclause of the query, we used a different form of the create directive: "create":"unless_connec-ted" . And in the response we have a "create":"created" . If you examine the list of Note in-stances in the freebase.com client, you'll see that there are now two of them named "B f at". If you use unless_connected , then Metaweb looks for a matching object that is already connected.If it cannot nd one, it creates a new one and connects it. In this case, there was an existing Noteobject named B f at, but it was not already connected, so the query created a new one. If we re-run the query, however, it simply returns "create":"existed" because the object and the con-

nection exist.

Note that unless_connected only makes sense in nested clauses. If we change the outermostunless_exists in the query above to unless_connected , Metaweb complains: Can't use 'create':'unless_connected' at the root of the query .

When to use unless_connected

"create":"unless_connected" directive is relatively infrequently used. Use it when objectsmust be unique within their "parent". One example of this is in the /music domain, where/music/track objects are unique for each /music/album . When a band releases an albumwith a hit song on it, that song is likely to end up being released again on compilation al-bums, live albums, cover albums, and so on. In the freebase.com /music domain, however,each song object is associated with only one album. With a schema like this, unless_con-nected is more useful than unless_exists .

Let's clean up the extra B f at object we created:

ResultWrite

{"type":"/user/docs/note",


"name":"E flat","name":"E flat","next":{"next":{

"connect":"deleted","connect":"delete","type":{"type":{

"connect":"deleted","connect":"delete","id":"/user/docs/note""id":"/user/docs/note"


"connect":"deleted","connect":"delete","value":"B flat","value":"B flat",




ResultWrite

"lang":"/lang/en"}

"lang":"/lang/en"}

}}

}}

Note that the query above does two things. It disconnects the name and type of the extra B f atobject, and also disconnects that object from E f at. Now all we have to do is connect E f at tothe valid B f at object. This should be easy for you now:

ResultWrite



"name":"E flat","name":"E flat","next":{"next":{

"type":"/user/docs/note","connect":"insert",

"connect":"inserted","type":"/user/docs/note","name":"B flat""name":"B flat"

}}

}}

5.1.7. Review: Write DirectivesAt this stage of the tutorial, you've seen all the variations of the create and connect directives.Let's do a quick review before diving in to some more advanced examples.

The create directive comes in three forms:

"create":"unconditional" Always create the speci ed object. It is almost never ne-cessary or appropriate to use this form of the create dir-ective.

"create":"unless_exists" Look for the object in the database and create a new oneif a match cannot be found.

"create":"unless_connected" Look for a matching object that already exists and isalready connected to the parent query. If no such objectexists, create and connect a new one.

The possible responses to a create directive are the following:

"create":"created" Indicates that a new object has been created. This is always theresponse for unconditional directives, but may also be returnedby unless_exists and unless_connected directives.




"create":"existed" Indicates that a pre-existing match was found and no object wascreated. This may be returned by unless_exists or unless_con-nected directives.

"create":"connected" Indicates that the object already existed but a connection hasbeen made. This response is only possible for unless_exists

directives that are nested within a parent query.

The three forms of the connect directive are:

"connect":"insert" Use this form to attach a value or object to a non-unique property.It can also be used to attach the rst value or object to a uniqueproperty.

"connect":"update" Use this form to attach a value or object to a unique property, repla-cing any value or object that was previously connected.

"connect":"delete" Use this form to detach a value or object from a property. It works

for unique and non-unique properties.

There are ve possible responses to a connect query:

"connect":"inserted" Indicates that an insert directive was successful.

"connect":"updated" Indicates that an update directive was successful.

"connect":"deleted" Indicates that a delete directive was successful.

"connect":"present" Indicates that an insert or update directive was unsuccessfulbecause the speci ed connection was already present.

"connect":"absent" Indicates that a delete directive was not successful because theconnection to be deleted did not exist.

5.1.8. Working with SetsThe most interesting examples we've explored so far have used the next property of our Notetype. We de ned this property to be unique--so that it can have only one value. There are somefeatures of the MQL write grammar that only become apparent when used on non-unique prop-erties, however. Let's de ne a Chord type and give it a non-unique property named note whichlinks to Note objects. (By convention, we use a singular property name, even though we expecteach Chord object to refer to multiple Note objects.) Create this type and its property on sand-

box.freebase.com by repeating the steps we followed to de ne the Note type and its nextproperty. Just change the names to Chord and note , and don't check the "Restrict to one value"box.

If you appreciated not having to type default_domain/ in the examples above, you can use thesame shortcut for the new Chord type:




and this allows us to connect more than one value to a property. In the case of the type property,our query speci es two types by their id . As we discussed earlier, types can be speci ed by idbecause id is the default property of /type/type . When types are speci ed this way, "con-nect":"insert" is assumed. The reason that we specify /common/topic in addition to the Chordtype is that the freebase.com client uses topics as its organizing metaphor. Objects of type /com-mon/topic simply work better in the client. For example, /common/topic objects you create arelisted under the heading "Topics Created" on your My Freebase page.

Multiple Types and Unqualified Property Names

When we specify more than one type for an object, we use a JSON array. But the Metawebobject model represents the types as an unordered set, so the order in which we specifythem should not matter. In fact, however, it does. The last type in the array of types is usedto qualify any unquali ed property names that are not /type/object properties.

In the query above, if we had speci ed /user/docs/chord rst, and /common/topic second,then Metaweb would have assumed that the unquali ed note property meant /common/top-

ic/note , and this would have caused an error since there is no such property. If you don'twant to rely on the order of the types, you can just be explicit and use the fully-quali ednames of all properties, such as /user/docs/chord/note .

5.1.9. Bidirectional Links and Reciprocal PropertiesOne of the fundamental aspects of Metaweb is that all links between nodes are bi-directional.Our CEG Chord node has links to the nodes that represent the notes C, E, and G. Those links arebi-directional, which means that the C, E, and G nodes are linked to the CEG Chord node. Thelinks are there, but our Note type doesn't de ne a appropriate property that exposes those linksin the object-oriented view of the database.

Fortunately, the freebase.com client makes it very easy to de ne such a property:

• Go to your My Freebase page on sandbox.freebase.com home page, and click on your Notetype under User's Types .

• Click on the "View Schema" link on the page for your Note type.

• Look near the bottom of the schema page (you may have to scroll down) for the headingSuggested Properties . You should see something like what is shown in Figure 5.3

• This tells you is that the type Chord has a property named Note. 1 The client is suggesting that

you add a reciprocal property to expose the other direction of the link. The link is already there:all that is required is that you give this property a name so that you can refer to it.

• Since the Chord property that refers to Notes is named note , it seems sensible to name theNote property that refers to Chords chord . Click the Edit button or double-click the double-click to edit text message. Then type in "chord" and hit Enter or click Save .

1 The freebase.com client capitalizes type and property names: these are the "human-readable" forms: their ids are still lowercase/user/docs/chord and /user/docs/chord/note ).




Figure 5.3. Adding a reciprocal property

You have now created the property /user/docs/note/chord , which is the reciprocal propertyof /user/docs/chord/note . Since we now have a pair of properties, we can take advantage of the bi-directional nature of the links between chords and notes.

Let's experiment with this. First, we'll query the Chord CEG to nd out what notes it contains:

ResultRead

{"type":"/user/docs/chord",


"name":"CEG","name":"CEG","note":["C","G","E"]

}"note":[]

}

This result is unsurprising, given that the /user/docs/chord/note property is the one we de nedoriginally. Now let's turn the query around and try out the reciprocal /user/docs/note/chordproperty we've just added. What chords is the note C a part of?

ResultRead



"name":"C","name":"C","chord":["CEG"]

}"chord":[]

}

The note C "knows" that it is part of the chord CEG even though we never set its chord property.Setting a property automatically causes its reciprocal property to be set as well. Because linksare bi-directional in Metaweb, this is all automatic.

Now let's create a new chord:

ResultWrite



"type":["/common/topic","type":["/common/topic","/user/docs/chord"],"/user/docs/chord"],

"name":"BFG"}

"name":"BFG"}




We've created a chord named BFG, but we haven't added the notes B, F and G to it. To furtherdemonstrate reciprocal properties, we'll do the reverse, and add the chord to the notes:

ResultWrite

[{"create":"created",

[{"create":"unless_exists",

"type":"/user/docs/note","type":"/user/docs/note","name":"B","name":"B","chord":{"chord": {

"connect":"inserted","connect":"insert","type":"/user/docs/chord","type":"/user/docs/chord","name":"BFG""name":"BFG"

}}},{},{

"create":"existed","create":"unless_exists","type":"/user/docs/note","type":"/user/docs/note","name":"F","name":"F","chord":{"chord": {


}}},{},{

"create":"existed","create":"unless_exists","type":"/user/docs/note","type":"/user/docs/note","name":"G","name":"G","chord":{"chord": {


}}]

}}]

This query connects the BFG chord to the chord property of the notes B, F, and G. (It also createsthe note B, which didn't exist yet.) Now let's ask BFG what notes it contains:

ResultRead



"name":"BFG","name":"BFG","note":["B","F","G"]

}"note":[]

}

Once again, we've demonstrated that we can set a property of an object by setting the reciprocalproperty to refer to that object.




5.1.9.1. How Reciprocal Properties Work

Now let's explore the properties /user/docs/chord/note and /user/docs/note/chord to ndout what makes them the reciprocal of each other. Recall that properties are themselves Metawebobjects. First we'll query the note property of Chord:

ResultRead

{"type":"/type/property",


"id":"/user/docs/chord/note","id":"/user/docs/chord/note","expected_type":"/user/docs/note","expected_type":null,"master_property":null,"master_property":null,"reverse_property":"reverse_property":null

} "/user/docs/default_domain/note/chord"}

We nd that the note property of Chord has an expected_type of Note, as expected. More inter-estingly, though, we nd a property named reverse_property that refers to the chord propertyof the Note type. 2 So let's query that property now:

ResultRead



"id":"/user/docs/note/chord","id":"/user/docs/note/chord","expected_type":"/user/docs/chord","expected_type":null,"master_property":"master_property":null,

"/user/docs/default_domain/chord/note","reverse_property":null} "reverse_property":null

}

This property has a reverse_property of null , but has a property named master_property thatrefers back to the rst property we looked at.

Reciprocal properties are linked to each other via the master_property and reverse_propertyproperties. When one property is set, its reciprocal, if it has one, is automatically set. The recipro-city is symmetrical: the terms "master" and "reverse" imply a directionality or hierarchy, but theproperty labeled "master" has no special status or preference over the property labeled "reverse".(When types are created with the freebase.com clients, the property created rst is the masterproperty.) 3

2 Note that default_domain has crept back into the result. We've created shortcuts for our types in easier-to-type namespaces, buttheir original location was under /user/docs/default_domain .3 The master_property and reverse_property properties are themselves reciprocal properties. (Verifying this with a MQLquery is left as an exercise for the reader.) This means that if you set p.reverse_property to q then q.master_property isautomatically set to p .




5.1.10. Writes and Ordered CollectionsIf a Metaweb property has not been declared a unique property, it may have a set of values. Aswe saw in Chapter 3 , these sets may be ordered , and MQL read queries can access this orderwith the index keyword. This section shows how to de ne an ordering with a MQL write query.

Not surprisingly, this is also uses the index keyword.

In order to demonstrate how to create an ordered collection, we'll need a suitable type. Chordsdon't work: the notes of a chord are played simultaneously, and no order is required. A brokenchord (or arpeggio ) is a chord in which the notes are played sequentially. Since there is a sequence,there is an order. Use the freebase.com client to de ne a new type named "Arpeggio" in thesandbox. Give it a property named "note" whose expected type is Note. Arpeggio is actually justlike Chord: only the names are different. To save yourself typing, copy the type from/user/docs/default_domain/arpeggio (use your own username) to /user/docs/arpeggio , justas we did for the Note and Chord types:

{

"id":"/user/docs/default_domain/arpeggio","key":{

"connect":"insert","namespace":"/user/docs","value":"arpeggio"

}}

Now with our type de ned, let's create our rst ordered collection:

ResultWrite

{

"create":"created",

{

"create":"unless_exists","type":"/user/docs/arpeggio","type":"/user/docs/arpeggio","name":"broken CEG","name":"broken CEG","note":[{"note": [{

"index":0,"index":0,"type":"/user/docs/note","type":"/user/docs/note","name":"C""name":"C"

},{},{"index":1,"index":1,"type":"/user/docs/note","type":"/user/docs/note","name":"E""name":"E"

},{},{

"index":2,"index":2,"type":"/user/docs/note","type":"/user/docs/note","name":"G""name":"G"

}]}

}]}

Creating an arpeggio is much like creating a chord. Two things stand out about this query, however.First, each note has an index associated with it, and there are no connect directives. The index




property speci es the ordering, and does an implicit "connect":"insert" . (If the object wasalready inserted, then the index would simply re-order it without attempting to re-insert it.) If wewere creating the Note objects at the same time as we were inserting them into this Arpeggioobject, we would have to include both the create directive and the index property.

There are some strict rules that govern the use of the index property:

• The index property may not appear within a top-level query. Indexes don't apply to objectsbut to the links between objects. The index property is used in sub-queries to specify the orderof the links between the parent object and the children.

• If there are n sibling sub-queries that specify an index, the values speci ed must include everyinteger from 0 to n-1. You must always start with zero. You may not include duplicate indexes,and you may not skip an index. It is not required that every element of a sub-query array havean index. Metaweb collections can be partially ordered and partially unordered.

This second rule may seem surprisingly strict, but remember that despite the name "index", thevalues we specify with the index property are not array indexes. The numbers are merely a simple

way to specify a series of less than and greater than relationships. The requirement that indexesalways run from 0 through n-1 means that there is really no way to insert an element at a givenlocation with an ordered collection, and no way to move an element from one spot to another.

Suppose, for example, that we want to insert the notes B and F into our CEG arpeggio at the be-ginning so the arpeggio consists of the ve sequential notes BFCEG:

ResultWrite

{"type":"/user/docs/arpeggio",

{"type":"/user/docs/arpeggio",

"name":"broken CEG","name":"broken CEG","note":[{"note": [{

"create":"connected","create":"unless_exists","index":0,"index":0,"type":"/user/docs/note","type":"/user/docs/note","name":"B""name":"B"

},{},{"create":"connected","create":"unless_exists","index":1,"index":1,"type":"/user/docs/note","type":"/user/docs/note","name":"F""name":"F"

}]}

}]}

This query demonstrates that the index property can be used along with a create directive. Thenotes already exist, but are not connected so we get "create":"connected" in the response.

What has this query actually done? We can use a read query to ask for the notes of the arpeggioin order and see if we've accomplished what we wanted to. But before we do, let's consider whatinformation we've given Metaweb. Previously, we told Metaweb that C comes before E and Ecomes before G. Now, we've also told Metaweb that B comes before F. But this isn't enough.




Metaweb does not know anything about the relationship between BF and CEG. There are manypossible orderings that meet the criteria we've speci ed. BFCEG is one possible ordering, but sois CEGBF, and so is CBEFG!

Let's see what we actually get:

ResultRead{

"type":"/user/docs/arpeggio",{

"type":"/user/docs/arpeggio","name":"broken CEG","name":"broken CEG","note":["note":[{

{"index":0, "name":"B"},"index":null,{"index":1, "name":"F"},"name":null,{"index":2, "name":"C"},"sort":"index"{"index":3, "name":"E"},}]

} {"index":4, "name":"G"}]

}

Metaweb actually returns the notes in the order we wanted! The fact that Metaweb inserts newelements at the beginning of an ordered collection is an implementation detail, however, and isnot behavior that is guaranteed. In practice, if you want to de ne particular ordering for the ele-ments of a collection, you must write a single query that enumerates each of those elements andgives them all an index. Any subsequent insertions or shuf f es require you to submit a new querythat again lists all elements and de nes their order. 4

Here, therefore, is how we should have written the query above:

{"type":"/user/docs/arpeggio","name":"broken CEG","note": [{

"create":"unless_exists","index":0,"type":"/user/docs/note","name":"B"

},{"create":"unless_exists","index":1,"type":"/user/docs/note","name":"F"

},{ "index":2, "type":"/user/docs/note", "name":"C" },{ "index":3, "type":"/user/docs/note", "name":"E" },

4 What if, when we'd inserted B and F, we'd also included C, with index 2 in the query. Then Metaweb would know that B is less thanF and F is less than C. And it already knows that C is less than E and E is less than G. Wouldn't that de ne a complete ordering? Thissounds plausible, but, in fact, after such a query Metaweb would have an ordering for the subset BFC and another ordering for thesubset EG, and it still would not know the relationship between those two subsets.




{ "index":4, "type":"/user/docs/note", "name":"G" }]}

This query inserts the two new notes and re-iterates the order of the three existing ones, to specifya complete ordering of ve notes with indexes 0 through 4.

Metaweb's ordered collections are not arrays and do not behave like arrays. The requirement thatyou re-specify the complete ordering even for simple insertions demonstrates this. And it alsomakes it clear that ordering is only practical for relatively small and static collections. It makesgood sense to de ne an ordering for the tracks on an album, for example. It makes less sense tode ne an ordering for the albums recorded by a band, however, because this set of albums maychange with time and a database maintainer would be required to respecify the complete disco-graphy each time a new album was added. And it makes no sense at all to try to specify an orderingfor bands (by specifying an index for each instance property of the /music/artist type): thereare simply too many of them.

Arpeggios and Duplicate Notes

If you are a musician, you probably know that broken chords often repeat a note. In practice,we'd want to represent arpeggios like EGCE, where the note E appears twice. In Metaweb,the value of a property is a set, and sets do not allow duplicates, even when they are ordered.That is, ordered collections in Metaweb are still sets, not lists, and they do not allow duplic-ates. In order to represent an arpeggio EGCE, therefore, we'd have to create two separatenote objects named E. But having two objects that both represent the note E is problematic:the next and chord properties of the Note type are premised on the assumption that therewill only be one Note instance for each note.

There are two lessons to be learned here:

•Ordered collections are still sets, and do not allow duplicates.

• Designing good Metaweb schemas for knowledge representation is hard to do.

5.1.11. NamespacesIn several places throughout this tutorial we've placed types into new namespaces simply to makeour queries a little easier to enter into the query editor. In this section we'll explore namespacesin more detail.

We begin with a review of material from Chapter 2 . First, remember that fully-quali ed namesand namespaces don't have anything to do with the name property of an object. The name propertyde nes a human-readable display name for an object. Fully-quali ed names are unique and canbe used as an alternative to the object guid.

Fully-quali ed names are de ned by the value type /type/key . Every object has a key propertythat holds a set of /type/key values. If you want an object to have a fully-quali ed name, inserta key into its key property. The value property of the key speci es the object's unquali ed orlocal name. And the namespace property of the key speci es the object that de nes the namespace.




Any object can be a namespace: the only requirement is that the object must itself have a key. Inthis way we get a chain of /type/key/value properties that continues until we nd a/type/key/namespace property that refers to the special root namespace object.

The type /type/namespace exists, and de nes the the property /type/namespace/keys , whichis the reciprocal of /type/key/namespace . Objects that are used as namespaces are usually giventhe type /type/namespace , but this is not required.

The reason that namespaces are useful is that namespaces allow us to use fully-quali ed namesto uniquely identify objects. If an object is given a key, then we can use its unique fully-quali edname as the value of the id property. Identifying objects with a human-readable id is simplerthan using a long guid, and is more reliable than using the name and type properties together.

With that review of namespaces, let's try to put some of the note objects we've created into anamespace. We'll use the /user/docs/note type object as our namespace:

ResultWrite

[{"type":"/user/docs/note",

[{"type":"/user/docs/note",

"name":"C","name":"C","key":{"key":{

"connect":"inserted","connect":"insert","namespace":"/user/docs/note","namespace":"/user/docs/note","value":"C""value":"C"

}}},{},{

"type":"/user/docs/note","type":"/user/docs/note","name":"E","name":"E","key":{"key":{

"connect":"inserted","connect":"insert","namespace":"/user/docs/note","namespace":"/user/docs/note","value":"E""value":"E"

}}},{},{

"type":"/user/docs/note","type":"/user/docs/note","name":"G","name":"G","key":{"key":{

"connect":"inserted","connect":"insert","namespace":"/user/docs/note","namespace":"/user/docs/note","value":"G""value":"G"

}}]

}}]

This query gives the notes C, E, and G keys named "C", "E", and "G" within the namespace/user/docs/note . That is, it de nes fully-quali ed names for these notes /user/docs/note/C ,/user/docs/note/E , and /user/docs/note/G . Now that these notes have unique ids, it becomes(somewhat) easier to use them in queries. Here's how we might create a chord:




ResultWrite

{"create":"existed",


"type":"/user/docs/chord","type":"/user/docs/chord","name":"CEG","name":"CEG","note":[{"note":[{

"connect":"unchanged","connect":"insert","id":"/user/docs/note/C""id":"/user/docs/note/C"

},{},{"connect":"unchanged","connect":"insert","id":"/user/docs/note/E""id":"/user/docs/note/E"

},{},{"connect":"unchanged","connect":"insert","id":"/user/docs/note/G""id":"/user/docs/note/G"

}]}

}]}

This query replaces the name and type properties of each note with a single id property. It doesn'tactually do anything, since we have already created the CEG chord. We've seen that we can usea note's fully-quali ed name as the value of its id property. What if we query the id of a note?

ResultRead



"name":"CEG","name":"CEG","note":[{"id":"#1f800000000104bd02"},"note":[{"id":null}]

} {"id":"#1f800000000104beae"},{"id":"#1f800000000104beec"}]

}

We get the guids of the notes rather than the fully-quali ed names we've just de ned. Core types,such as /type/type and /type/property , that use id as their default property return a fully-quali ed name instead of their guid in queries like this.

5.1.11.1.The /type/namespace/keys Property

We've seen that we can put objects into a namespace by setting the key property of the object. Itis also possible to work with namespaces using the reciprocal property /type/namespace/keys .We've been using /user/docs/note as a namespace. This next query asks what keys it holds:

ResultRead

{"id":"/user/docs/note",


"/type/namespace/keys":["/type/namespace/keys":[]} "next","chord","arpeggio","C","E","G"

]}




The namespace holds the names of the three notes we added, and also the names of the threeproperties we de ned for the type. Let's repeat the query and ask for more detail:

ResultRead



"/type/namespace/keys":[{"/type/namespace/keys":[{}]} "type":"/type/key",

"namespace":"/user/docs/default_domain/note/next",

"value":"next"},{

"type":"/type/key","namespace":

"/user/docs/default_domain/note/chord","value":"chord"

},{"type":"/type/key","namespace":

"/user/docs/default_domain/note/arpeggio","value":"arpeggio"


"/user/docs/default_domain/note/C","value":"C"


"/user/docs/default_domain/note/E","value":"E"


"/user/docs/default_domain/note/G","value":"G"

}]}

The values of the /type/namespace/keys property are /type/key values that have value andnamespace properties. You'll notice that default_domain has crept back into the object ids in thequery results. This is interesting, but not terribly important. We'll investigate it later in this section.

There is one very important point to notice about these query results. When a key value is usedwith /type/object/key , the namespace property is the id of the namespace object (such as/user/docs/note ) that holds the key. But when a key value is used with /type/namespace/keys ,the namespace property is the id of the object (such as /user/docs/note/C ) contained by thenamespace. This is important to understand, so we'll state it another way: suppose that an objecto has a fully-quali ed name in the namespace n. If we query the key property of o, we'll nd a/type/key object whose namespace property refers to n. And if we query the




/type/namespace/keys property of n, we'll nd a /type/key object whose namespace propertyrefers to o.

If you wanted to create a Metaweb namespace browser application, you could repeat the queryabove, starting with the id of the root namespace "/" . The namespace properties of each of thereturned keys specify the ids of all objects in the root namespace. If you recursively query eachof these ids, you'll nd the complete set of Metaweb objects with fully-quali ed names.

It is also possible to add objects to namespaces using the /type/namespace/keys property insteadof /type/object/key . The following query creates a new Note object named "G f at" and assignsit the fully-quali ed name /user/docs/note/G_flat :

ResultWrite



"/type/namespace/keys":{"/type/namespace/keys":{"connect":"connected","connect":"insert","value":"G_flat","value":"G_flat""namespace":{"namespace":{

"create":"created","create":"unless_exists","name":"G flat","name":"G flat","type":"/user/docs/note""type":"/user/docs/note"

}}}

}}

}

5.1.11.2. Fully-Qualified Names and Uniqueness

In this section we explore the uniqueness of fully-quali ed names. First, recall that earlier in the

tutorial we de ned shortcut names for types. We've been using the name /user/docs/note fora type that was originally de ned as /user/docs/default_domain/note . If the type has twofully-quali ed names, and we're using that type as a namespace, then each of the notes we insertedinto that namespace should also have two names. /user/docs/default_domain/note/G shouldbe the same thing as /user/docs/note/G :

ResultRead

{"id":"/user/docs/default_domain/note/G",

{"id":"/user/docs/default_domain/note/G",

"/user/docs/note/chord":["CEG","BFG"]}

"/user/docs/note/chord":[]}

So a single object can have more than one fully-quali ed name. But can a fully-quali ed namerefer to more than one object? Let's try to give the note F the same key that we assigned to G:

{"type":"/user/docs/note","name":"F","key":{

"connect":"insert",




"namespace":"/user/docs/note","value":"G"

}}

This query fails, although there is nothing obviously wrong with it. Metaweb simply will not allow

the fully-quali ed name /user/docs/note/G to refer to two different note objects. If you wantto make /user/docs/note/G refer to the note F, you must rst make sure that the note does notrefer to the note G. This takes two queries. First, we must remove the fully-quali ed name forthe note G:

ResultWrite

{"id":"/user/docs/note/G",

{"id":"/user/docs/note/G",

"key":{"key":{"connect":"deleted","connect":"delete","namespace":"/user/docs/note","namespace":"/user/docs/note","value":"G""value":"G"

}}

}}

And then we can assign that fully-quali ed name to the note F:

ResultWrite



"name":"F","name":"F","key":{"key":{

"connect":"inserted","connect":"insert",

"namespace":"/user/docs/note","namespace":"/user/docs/note","value":"G""value":"G"

}}

}}

Now if we were to ask for the name of the note /user/docs/note/G , we'd get "F". Making afully-quali ed name refer to another object is simpler if we use the /type/namespace/keysproperty instead. Here's how we could make /user/docs/note/G refer to the note G again:

ResultWrite

{

"id":"/user/docs/note",

{

"id":"/user/docs/note", "/type/namespace/keys":{"/type/namespace/keys": {"value":"G","value":"G","namespace":{"namespace":{

"connect":"updated","connect":"update","type":"/user/docs/note","type":"/user/docs/note","name":"G""name":"G"

}}




ResultWrite

}}

}}

This query locates the /type/key object that de nes the name /user/docs/note/G , and updatesthe namespace property of that key, so that the name points to a different object. Note that youshould not typically have to alter namespaces like this. Objects that have fully-quali ed namesshould typically be constants.

Finally, notice that changing the object to which a fully-quali ed name refers (as we did above)is a completely different operation than changing the fully-quali ed name of an object. If wewanted to refer to the note G by the name /user/docs/note/Gnatural instead of /user/docs/note/G , we could do this:

ResultWrite

{

"name":"G",

{

"name":"G", "type":"/user/docs/note","type":"/user/docs/note","key":[{"key":[{

"connect":"deleted","connect":"delete","namespace":"/user/docs/note","namespace":"/user/docs/note","value":"G""value":"G"

},{},{"connect":"inserted","connect":"insert","namespace":"/user/docs/note","namespace":"/user/docs/note","value":"Gnatural""value":"Gnatural"

}]}

}]}

5.1.12. Properties, Types, and DomainsIn this nal section of the tutorial, we dig a deeper into Metaweb internals. Creating types andproperties is almost always best done using the freebase.com client: there are a lot of details toget right, and correct setup of types and properties is critical for correct functioning. In general,it is better to be safe and let the client create types and properties for you.

On the other hand, types, properties, and domains are Metaweb objects just like any others, andthey can be created and manipulated with MQL write queries. There is some educational valuein seeing how this is done, and there are a few things that we can do with MQL that we cannot

do through the client.

5.1.12.1. Creating Self-Referential Reciprocal Properties

The rst property we de ned in this tutorial was /user/docs/note/next , and we used it tomodel the cycle of fths, so that the next property of the note C referred to G, and so on. Wenever created a reciprocal property for next , but it seems logical that the note G should have aprevious property that refers to C.




At the time of this writing, the freebase.com client does not allow us to create reciprocal propertieswhere both ends of the link refer the same type, but we can do it with MQL:

ResultWrite



"/type/type/properties":{"/type/type/properties":{"create":"created","create":"unless_exists","type":"/type/property","type":"/type/property","name":"Previous","name":"Previous","key":{"key":{

"namespace":"/user/docs/note","connect":"insert","connect":"inserted","namespace":"/user/docs/note","value":"previous""value":"previous"

},},"expected_type":"expected_type":

"/user/docs/Note","/user/docs/Note","unique":true,"unique":true,"master_property":"master_property":

"/user/docs/note/next""/user/docs/note/next"}

}}

}

The rst line of the query identi es the type to which we're adding the property. The second,third and fourth lines specify that we're creating a new /type/property object and connectingit to the properties property of our type. The 5th line gives the property the human-readablename "Previous" and the following four lines de ne a key so that the property has the fully-quali ed name /user/docs/note/previous . The expected_type property speci es that thenewly created property should link to other Note objects. The unique property speci es that eachNote can have only a single value for the previous property. And, nally, the master_propertyproperty speci es that this new property is the reciprocal of /user/docs/note/next .

After executing this query, you can test it by querying the previous property of the note G. Youcan also use the freebase.com client to browse the note objects you've created and follow theirnext and previous properties back and forth.

5.1.12.2. Creating a Domain

Now that we've de ned several music-related types, we may feel that they don't really belongunder either /user/docs/default_domain or /user/docs . Let's use MQL to create a new domain

and give it the fully-quali ed name /user/docs/music . (At the time of this writing, there is noway to create a custom domain with the freebase.com client.)

Domain objects have two properties: types speci es the set of types that are part of the domain.owners speci es one or more usergroups that own the domain and have permission to createtypes in it. Creating a domain is not enough: we must also set its ownership. We'll give this newdomain the same owner as /user/docs/default_domain . So before we create the domain, let's

nd out the owner of the existing one:




ResultRead

{"id":"/user/docs/default_domain",

{"id":"/user/docs/default_domain",

"/type/domain/owners":["/type/domain/owners":[],"#1f80000000010499ee","type":[]

} ],"type":["/type/domain"]

}

In addition to querying the owner of the domain, we also queried its types. Notice that it is notco-typed with /type/namespace even though it is used as a namespace.

Now, we can create the new domain. Remember to use your own username in place of "docs",and to substitute the id you obtained in the query above for the one shown here.

ResultWrite

{ "create":"created",{ "create":"unconditional","type":"/type/domain","type":"/type/domain","owners":"#1f80000000010499ee","owners":"#1f80000000010499ee","key":{"key":{

"connect":"inserted","connect":"insert","namespace":"/user/docs","namespace":"/user/docs","value":"music""value":"music"

}}

}}

Note that this is an example of a query in which "create":"unconditional" is actually required.

The constraints in this query match an existing domain object, and we need to force the creationof a new domain. (Since the key sub-query has a connect directive, it is not a constraint.)

The only thing we can do with a domain is add types to it. That is the subject of the next section.

5.1.12.3. Creating Types

Let's add simple note and chord types to our newly created music domain. Creating Metawebtypes is tricky. There are a number of properties that must be set correctly, and in general, a typecannot be created in a single query: there are multiple steps that must be followed to link everythingup correctly. Our approach with MQL follows the basic type creation strategy on the client: createthe type rst, and then add properties to it. We'll create the note and chord type objects at thesame time:

ResultWrite

[{"create":"created",

[{"create":"unless_exists",

"type":"/type/type","type":"/type/type","name":"Note","name":"Note","key":{"key":{




ResultWrite

"connect":"insert","namespace":"/user/docs/music",

"connect":"inserted","namespace":"/user/docs/music",

"value":"note" "value":"note"}, },"domain":"/user/docs/music" "domain":"/user/docs/music"

},{ },{"create":"unless_exists", "create":"created","type":"/type/type", "type":"/type/type","name":"Chord", "name":"Chord","key":{ "key":{

"connect":"insert", "connect":"inserted","namespace":"/user/docs/music", "namespace":"/user/docs/music","value":"chord" "value":"chord"

}, },"domain":"/user/docs/music"

}]"domain":"/user/docs/music"

}]

Note that these queries specify a fully-quali ed name for the newly created type objects, but alsospecify their domain. /user/docs/music serves as both the namespace and the domain for thenew types, and it is important that both are speci ed.

The next step is to add properties. To keep this example simple, we will add a note property tothe chord type, and then (in a separate query) add the reciprocal property to the note type. Here'show we de ne the note property:

ResultWrite

{

"id":"/user/docs/music/chord"

{

"id":"/user/docs/music/chord", "/type/type/properties":{"/type/type/properties":{"create":"created","create":"unless_connected","type":"/type/property","type":"/type/property","name":"Notes","name":"Notes","expected_type":"expected_type":

"/user/docs/music/note","/user/docs/music/note","key":{"key":{

"connect":"inserted","connect":"insert","namespace":"namespace":

"/user/docs/music/chord","/user/docs/music/chord","value":"note""value":"note"

}} }}

}}

Now we set up the reciprocal property in the note type:




ResultWrite

{"id":"/user/docs/music/note",

{"id":"/user/docs/music/note",

"/type/type/properties":{"/type/type/properties":{"create":"created","create":"unless_connected","type":"/type/property","type":"/type/property","name":"Chords","name":"Chords","expected_type":"expected_type":

"/user/docs/music/chord","/user/docs/music/chord","master_property":"master_property":

"/user/docs/music/chord/note","/user/docs/music/chord/note","key":{"key":{

"connect":"inserted","connect":"insert","namespace":"namespace":

"/user/docs/music/note","/user/docs/music/note","value":"chord""value":"chord"

}}}

}}

}

With these types and properties de ned, we can now create instances:

ResultWrite



"type":"/user/docs/music/chord","type":"/user/docs/music/chord","name":"CG","name":"CG","note":[{"note":[{

"create":"created","create":"unless_exists", "type":"/user/docs/music/note","type":"/user/docs/music/note","name":"C""name":"C"

},{},{"create":"created","create":"unless_exists","type":"/user/docs/music/note","type":"/user/docs/music/note","name":"G""name":"G"

}]}

}]}

5.2. MQL Write GrammarThis section describes the MQL write grammar more formally. It uses the same notation as, andborrows some de nitions from MQL read grammar of Section 3.3 .

A write-query is either a single-query or a query-list . A query-list is a comma-separatedlist of single-queries within square brackets. And a single-query is a comma-separated listof pairs within curly braces:




write-query:: single-query | query-listquery-list:: [ single-query ( , single-query )* ]single-query:: { pair ( , pair )* }

A pair can be a property , a creation , a connection , an index-specification , or an id-query :

pair:: property |creation |connection |index-specification |id-query

A property pair is a property-name in quotation marks followed by a colon and a property-value . A property-name is the same as it is in the MQL read grammar. See Section 3.3 for details.A property-value is either a nested write-query , or a JSON string, a JSON number, or booleanliteral:

property:: " . property-name . " : property-valueproperty-value:: write-query | <JSON string> | <JSON number> | true | false

A creation directive is the keyword "create" in quotation marks followed by a colon and one of the quoted strings "unconditional", "unless_exists" or "unless_connected".

creation:: "create" : creation-kind creation-kind:: "unconditional" | "unless_exists" | "unless_connected"

A connection directive is similar to a creation directive. It consists of the keyword "connect"in quotation marks followed by a colon and one of the strings "insert", "delete", or "update":

connection:: "connect" : connection-kind connection-kind:: "insert" | "delete" | "update"

An index-specification is the keyword "index" in quotation marks, followed by a colon anda non-negative integer:

index-specification:: "index" : <non-negative integer>

Finally, an id-query is simply the identi er "id" in quotation marks, followed by a colon andthe keyword null, without quotation marks. Alternatively, "/type/object/id" can be used in placeof "id"

id-query:: "id" : null |"/type/object/id" : null




Chapter 6. Metaweb Write ServicesThis chapter explains, and demonstrates with examples, how to deliver MQL write queries to theMetaweb mqlwrite service. As a necessary prerequisite, it shows how to log in to Metaweb toobtain authentication credentials, and how to present those credentials in subsequent writes. Thechapter also demonstrates how to upload content, such as images and HTML documents to theMetaweb content store.

The examples in this chapter are all written in Python. They all communicate with the Metawebservices at sandbox.freebase.com . And they are all designed to be run as command-line utilities.Note that this differs from Chapter 4 which placed more emphasis on server-side code for webapplications. Many Metaweb-enabled web applications will be read-only, but the Python codeshown in this chapter should be straightforward to port for use in server-side scripts if you needit.

6.1. Logging in to MetawebNo registration or authentication is required to use the mqlread service, but writing to Metawebrequires a log in rst. Before we can cover the mqlwrite service, therefore, we must explain thelogin service.

Since Metaweb services are HTTP-based, Metaweb authentication is cookie based. You log inby making an HTTP POST request to the URL http://www.freebase.com/api/account/login ,passing your username and password as URL-encoded form parameters. If login is successful,Metaweb returns one or more HTTP cookies in the response headers. These cookies contain yourauthentication credentials, and you must pass these back to Metaweb in the HTTP request headersof all subsequent write and upload requests.

The names and values of the authentication cookies are an implementation detail rather than aspeci cation detail, and are subject to change. To ensure success, your code must accept allcookies returned by the login service, and must present all of them to the mqlwrite service. If youwrite your applications using a suitably high-level HTTP library, cookie handling may be per-formed automatically for you. For this chapter, however, we explicitly handle the cookies at alower level.

The cookies returned by the login service are persistent, which means that you do not have tolog into the freebase.com client each time you visit the site. Nevertheless, when writing scriptsthat use Metaweb services, the best practice is to assume that your cookies have expired and login each time the script is invoked.

Example 6.1 shows Python code that de nes a metaweb.login() utility function. This functionsends a username and password to the login service, and returns cookies that can be treated asopaque authentication credentials. If login fails, the function raises a metaweb.MQLError exception.This login utility function is part of a larger metaweb.py module. Other examples in this chapterare also part of the module. We rst saw metaweb.py in Chapter 4 , where Example 4.3 de neda metaweb.read() utility function. Example 4.3 also includes the de nition of the MQLError ex-ception class used in this example. Example 6.1 (and all our other Python examples) also dependson the simplejson module, which is available from http://cheeseshop.python.org/pypi/simplejson .

133



Example 6.1. metaweb.py: logging in to Metaweb with Python

import httplib # Low-level HTTP networkingimport urllib # URL encodingimport simplejson # JSON serialization and parsing

host = 'sandbox.freebase.com' # The Metaweb hostloginservice = '/api/account/login' # Path to login service

# Submit the specified username and password to the Metaweb login service.# Return opaque authentication credentials on success.# Raise MQLError on failure.def login(username, password):

# Establish a connection to the server and make a request.# Note that we use the low-level httplib library instead of urllib2.# This allows us to manage cookies explicitly.conn = httplib.HTTPConnection(host)conn.request('POST', # POST the request

loginservice, # The URL path /api/account/login# The body of the request: encoded username/passwordurllib.urlencode({'username':username, 'password':password}),# This header specifies how the body of the post is encoded.{'Content-type': 'application/x-www-form-urlencoded'})

# Get the response from the serverresponse = conn.getresponse()

if response.status == 200: # We get HTTP 200 OK even if login fails# Parse response body and raise a MQLError if login failedbody = simplejson.loads(response.read())

if body['status'] != '200 OK':raise MQLError(body['messages'][0]['text'])

# Otherwise return cookies to serve as authentication credentials.# The set-cookie header holds one or more cookie specifications,# separated by commas. Each specification is a name, an equal# sign, a value, and one or more trailing clauses that consist# of a semicolon and some metadata. We don't care about the# metadata. We just want to return a comma-separated list of# name=value pairs.cookies = response.getheader('set-cookie').split(',')return ';'.join([c[0:c.index(';')] for c in cookies])

else: # This should never happenraise MQLError('HTTP Error: %d %s' % (response.status,response.reason))




6.1.1. The Login APIAs you can see from Example 6.1 , the login service expects URL-encoded username and passwordparameters, in the body of an HTTP POST request to the path /api/account/login . The login servicealways returns an HTTP status code of "200 OK", even when login fails. The body of the response

is a JSON formatted object, and the status property of this object speci es whether the loginsucceeded or failed. If the status property is "200 OK", then login succeeded, and the responsewill include a "set-cookie" header that contains authentication credentials.

If the login fails, then the status property of the JSON object will be something other then "200OK". In this case, the messages property is an array (typically with just one element) of messageobjects. The text property of the rst element of this array includes an error message that describeswhat went wrong. (Typically, this is "Invalid username or password".)

6.2. Making Write QueriesChapter 5 demonstrated, in great detail, how to express write queries in MQL. Now we explainhow to send those queries to Metaweb and retrieve the result. The Metaweb write service ismqlwrite . The path to this service is /api/service/mqlwrite , and it responds to HTTP POSTrequests only. Wrap your query in an envelope object (described below) and submit the envelopeas the value of the q parameter in the body of the request. Also, include your authentication cre-dentials in the HTTP Cookie header of the request. The mqlwrite service performs your query,wraps the result in a response envelope object, and returns the JSON serialization of that objectas the body of its HTTP response.

X-Metaweb-Request

The Metaweb mqlwrite service (and also the upload service documented later in this chapter)require a custom HTTP request header, named X-Metaweb-Request to be present in all re-quests. The value of the header can be anything.

The requirement that the custom header be present is a security measure to prevent XSS(cross-site scripting) attacks. It has a profound implication for the services that require it:these services cannot be invoked via HTML form submission, since there is no way to tella web browser to add a custom header like this when POSTing a form.

The query and response envelopes are described in the sub-sections that follow, and those explan-ations are followed by example code that performs writes.

6.2.1. The mqlwrite Query EnvelopeThe value of the q parameter should be the JSON serialization of an envelope object. This objectsimply serves to give your query a name, by which it will be referred to in the response object.Consider the following query:


135Chapter 6. Metaweb Write Services



"type":"/common/topic","name":"my test object"

}

To submit it to mqlwrite you would rst envelop it like this:

{"query":{

"create":"unless_exists","type":"/common/topic","name":"my test object"

}}

The name "query" is arbitrary, but it will be reused in the response envelope.

6.2.2. The Response Envelopemqlwrite is an HTTP service. It returns an HTTP response with a Content-Type header of ap-plication/json . The body of the response is a JSON-serialized object. Only part of this objectis the result of the submitted query or queries: other properties provide meta-information. Thefollowing are the properties of the response envelope:

result The value of this property is an object that holds the results of the submittedqueries. If the query envelope used the property name query for the query, thenthe response envelope will make the results of the query available as result.query .If a query fails, this property will be set to null .

status For successful queries, this property will have the value "200 OK".

queries The value of this property is a copy of the query envelope that was submitted.

messages This property is an array of error messages. For successful queries it is empty. Forunsuccessful queries, it contains one or more message objects. Each message objecthas the following properties:

text A human-readable description of the error

args An object that provides additional details about the error. For example,if a query uses "create":"unless_exists" and it cannot complete be-cause two matching objects exist, then the text property might be "Needa unique result to attach here, not 2", and the args object will specify

the guids of the two matching objects.

query A copy of the query object (not the query envelope, but the query itself)that contains the error, with the addition of a special error_insideproperty, to indicate where error occurs.

level A string that indicates the severity of the error. A typical value is "error".




type A string that indicates the kind of error that occurred. A typical value is"/service/error/type" to indicate a type error.

6.2.3. A mqlwrite Utility Function

Example 6.2 is another piece of our metaweb.py module. It de nes a metaweb.write() functionthat submits a MQL write query to the mqlwrite service, and returns the result. The query thatis submitted and the result that is returned are both Python dictionary objects. The utility functiontakes case of JSON serialization and parsing and also seals the query in a query envelope andopens the response envelope to extract the query result. Note that metaweb.write() expects au-thentication credentials as its second argument. Use it with the metaweb.login() function incode like this:

import metaweb # Use the metaweb modulequery = {'create':'unless_exists', 'id':None, # This is our query

'type':'/common/topic', 'name':'my test object'}credentials = metaweb.login("name", "pass") # Login first

result = metaweb.write(query, credentials) # Execute queryprint "%s: %s" % (result['create'], result['id']) # Print result details

The code in Example 6.2 uses the metaweb.MQLError exception class. That class was de ned inExample 4.3 .

Example 6.2. metaweb.py: sending a query to mqlwrite

import urllib # URL encodingimport urllib2 # Higher-level URL content fetchingimport simplejson # JSON serialization and parsing

host = 'sandbox.freebase.com' # The Metaweb hostwriteservice = '/api/service/mqlwrite' # Path to mqlwrite service

def write(query, credentials):# We're requesting this URLreq = urllib2.Request('http://%s%s' % (host, writeservice))# Send our authentication credentials as a cookiereq.add_header('Cookie', credentials)# The body of the POST request is encoded URL parametersreq.add_header('Content-type', 'application/x-www-form-urlencoded')# This custom header is required and guards against XSS attacksreq.add_header('X-Metaweb-Request', 'True')# Wrap the query object in a query envelopeenvelope = {'query': query}# JSON encode the envelopeencoded = simplejson.dumps(envelope)# Use the encoded envelope as the value of the q parameter in the body# of the request. Specifying a body automatically makes this a POST.req.add_data(urllib.urlencode({'q':encoded}))try:

f = urllib2.urlopen(req) # Submit the request




response = simplejson.load(f) # Parse HTTP response as JSONif (response['status'] == '200 OK'): # Check for valid result

return response['result']['query'] # Open envelope, return resultelse: # Otherwise raise an exception

raise MQLError(response['messages'][0]['text'])except urllib2.HTTPError, e: # If anything goes wrong

if e.code == 400: # For code 400:body = simplejson.load(e) # parse the responseraise MQLError(body['messages'][0]['text']) # to get an error message

else: # For any other error code, report the code and the response bodyraise MQLError('HTTP Error %d:\n%s' % (e.code, e.read()))

6.2.4. Example: US State QuartersExample 6.1 and Example 6.2 are helpful utility functions, but they are just utilities, not real-world examples of how you might write to Metaweb. For the sake of example, let's suppose thatyou are a coin collector and you think that freebase.com should include information about each

of the coins issued by the US Mint under its 50 State Quarters program. First, visit the US Mint'swebsite 1 to nd out when each state quarter was released, how many were minted for each state,and when each state became a state.

Next, create a Metaweb type to model this data. Login to the freebase.com client and create anew type named "US State Quarter". (Review the procedure for creating types and adding prop-erties in Chapter 5 , if necessary.) If your Freebase username is "fred", this will create a type withid /user/fred/default_domain/us_state_quarter .

Next, give your type four properties:

• A property named "State", of /type/text to specify the name of the state. (The freebase.comclient speci es types by name rather than by id, so use the type name "Text" in the client).

• A property named "Release", of /type/datetime , to specify the date on which the quarter wasreleased into circulation. (Use the type name "Date/Time").

• A property named "Mintage", of /type/int , to specify how many quarters were minted. (Usethe type name "Integer".)

• A property named "Statehood", of /type/datetime to specify when the state gained statehood.

Be sure to make each of these properties unique by clicking the "Restrict to one value" checkbox.

Next, you need to get your data into manageable form. Extract data from the US Mint site, andarrange it in a plain text le named quarters.txt that looks like the following:

Delaware,1999-01-04,1787-12-07,774824000Pennsylvania,1999-03-08,1787-12-12,707332000New Jersey,1999-05-17,1787-12-18,662228000Georgia,1999-07-19,1788-01-02,939932000

1 Speci cally the page http://www.usmint.gov/mint_programs/50sq_program/index.cfm?action=schedule




Connecticut,1999-10-12,1788-01-09,1346624000Massachusetts,2000-01-03,1788-02-06,1163784000

Each line in this le is the data for a single quarter. Fields are separated by commas. The rsteld is the name of the state. The second and third elds are the release date and statehood date

for that state. And the fourth eld is the mintage for that state quarter.

With our type created, and the data in this format, we can now write a simple script to uploadthe data to freebase.com . Example 6.3 shows Python code to do this. Note that you need to insertyour own Freebase username and password into the script to make it work for you.

Example 6.3. quarters.py: writing a data set to Metaweb

import metaweb # Use the metaweb module

USERNAME = 'username' # Put your Freebase username and password herePASSWORD = 'password'

# The ID for our US State Quarter type depends on our usernameTYPEID = '/user/' + USERNAME + '/default_domain/us_state_quarter'

# Make sure we can log in before we go any furthercredentials = metaweb.login(USERNAME, PASSWORD)

# We will be creating multiple quarters in a single MQL query.# We start with an empty array and add MQL writes to it in the loop belowquery = []

# Open our file of quarter dataf = open("quarters.txt", "r")

# Loop through lines of the filefor line in f:

# Break each line into fieldsfields = line.strip().split(',')

# This query creates a single quarterq = {'create':'unless_exists', # Create a new object

'id':None, # And return its id'type':["/common/topic", TYPEID], # Make it a topic and a quarter'name': fields[0] + ' State Quarter', # The object's name'state': fields[0], # State name'release': fields[1], # Release date'statehood': fields[2], # Statehood date'mintage': int(fields[3])} # How many minted

# Add this write to the array of writesquery.append(q);

# Close the data filef.close()




# Now send our one big query to Metaweb and get the resultresult = metaweb.write(query, credentials)

# Display the id of the Metaweb object for each statefor r in result:

print "%s %s: %s" % (r['create'], r['state'], r['id'])

6.2.5. Sending Multiple Queries to mqlwriteAs we saw in Chapter 5 , the MQL write grammar allows multiple independent writes to be spe-ci ed in a single query as elements of a JSON array. The mqlwrite service also allows multiplenamed queries to be placed in an envelope. Suppose we want to create two objects in a singlecall to mqlwrite . Here are two envelopes that can accomplish that:

2 Queries in 1 Envelope2 Writes in 1 Query

{

"query1":{

{

"query":[{"create":"unless_exists","create":"unless_exists","type":"/common/topic","type":"/common/topic","name":"my test object #3""name":"my test object #1"

},},{"query2":{"create":"unless_exists",

"create":"unless_exists","type":"/common/topic","type":"/common/topic","name":"my test object #2""name":"my test object #4"}]

} }}

When you include multiple writes in a single query, the writes are executed atomically: they allsucceed or they all fail. As a result, they are not allowed to depend on each other, and there is noway to tell what order they are executed in.

If you submit multiple queries in a single envelope, they are not atomic. Each one succeeds orfails on its own. JSON envelopes are unordered collections of properties, and there is no guaranteethat the queries will be executed in the order in which they are written. For this reason, queriesthat share an envelope should not depend on each other.

The fact that mqlwrite can accept multiple queries explains why an envelope object is requiredin the rst place. It is needed so that each query has a name that can be used for query results. If

multiple named queries are submitted in an envelope, then the response envelope contains multiplesresult. If the queries are named q1 and q2 , then the responses are available as result.q1 andresult.q2 .

6.3. Uploading Data to MetawebThe unique feature of Metaweb is the way that it stores relationships between objects. But, likeany database, it can also store large chunks of data, such as long HTML documents or binary




image les. In Chapter 4 we learned about the trans service for retrieving content. Here, we'lllearn how to upload content to be stored in Metaweb. The service for uploading content is namedupload , and has the URL path /api/service/upload . The upload service responds only to HTTPPOST requests. The data to be uploaded is passed in the body of the request. The MIME type of the content, as well as the encoding of textual content is speci ed in the HTTP Content-Typerequest header.

6.3.1. An Upload UtilityExample 6.4 shows the Python implementation of a metaweb.upload() utility. Pass it a string of content (Python strings can be binary data or textual data), a MIME type for the content, and theauthentication credentials returned when you logged in. upload() creates a new /type/contentobject to hold your content and returns the guid of that object to you. This guid can be used toretrieve the content with the /api/trans/raw service (see Chapter 4 ).

No Duplicates

The upload service does not always create a new /type/content object for the content youupload. All content is checksummed when it is uploaded, and these checksums are used todetect duplicate uploads. If the content you are uploading already exists in the Metawebcontent store, the existing /type/content object is used. The blob_id property of /type/content holds the checksum value.

Example 6.4. metaweb.py: uploading content to Metaweb

import urllib2 # Higher-level URL content fetchingimport simplejson # JSON serialization and parsing

host = 'sandbox.freebase.com' # The Metaweb hostuploadservice = '/api/service/upload' # Path to upload service

# Upload the specified content (and give it the specified type).# Return the guid of the /type/content object that represents it.# The returned guid can be used to retrieve the content with /api/trans/raw.def upload(content, type, credentials):

# This is the URL we POST content tourl = 'http://%s%s'%(host,uploadservice)# Build the HTTP requestreq = urllib2.Request(url, content) # URL and content to POSTreq.add_header('Content-Type', type) # Content type header

req.add_header('Cookie', credentials) # Authentication headerreq.add_header('X-Metaweb-Request', 'True') # Guard against XSS attackstry:

f = urllib2.urlopen(req) # POST the requestresponse = simplejson.load(f) # Parse the responsereturn response['result']['id'] # Extract and return content id

except urllib2.HTTPError, e:if e.code == 400: # For code 400:

body = simplejson.load(e) # parse the response




raise MQLError(body['messages'][0]['text']) # to get error messageelse: # For any other error, report code and response body

raise MQLError('HTTP Error %d:\n%s' % (e.code, e.read()))

6.3.2. Examples: Uploading Images of State QuartersIn order to demonstrate the metaweb.upload() utility of Example 6.4 let's continue with our USState Quarter example. The US Mint has images of each of the state quarters on its website. Let'supload those images to freebase.com , and make them visible through the /common/topic/imageproperty of each quarter object. Example 6.5 shows how to do this.

In order to understand this code, you have to know that the upload service handles images spe-cially. When you upload images, the /type/content object is also given the type /common/image .Furthermore, the upload service determines the size of the image and creates an appropriate/measurement_unit/rectangle_size object for the /common/image/size property. Becauseyour uploaded /type/content is co-typed as /common/image , you can link directly to imagecontent from /common/topic/image .

Example 6.5. quarterpix.py: uploading images to Metaweb

import metaweb # Our metaweb utilitiesimport urllib2 # For downloading images from the mint server

USERNAME = 'username' # Put your Freebase username and password herePASSWORD = 'password'

# The ID for our US State Quarter type depends on our usernameTYPEID = '/user/' + USERNAME + '/default_domain/us_state_quarter'

# Make sure we can log in before we go any furthercredentials = metaweb.login(USERNAME, PASSWORD)

# All the images files are beneath this URLimagedir = 'http://www.usmint.gov/images/mint_programs/50sq_program/states/'

# This dictionary maps state name to image name.images = { 'Delaware': 'DE_winner.gif',

'Pennsylvania': 'PA_winner.gif','New Jersey': 'NJ_winner.gif','Georgia': 'GA_winner.gif','Connecticut': 'CT_winner.gif','Massachusetts': 'MA_winner.gif'}

# Loop through the statesfor state,filename in images.items():

# First, download the image from the Mint's websiteimage = urllib2.urlopen(imagedir + filename)type = image.info()['Content-Type']content = image.read()




# Now upload it to Metawebid = metaweb.upload(content, type, credentials)

# Define a write query to link the quarter object to the uploaded imagequery = { 'type': TYPEID,

'state':state,

'/common/topic/image': { 'id':id, 'connect':'insert' }}

# Submit the query and get the resultresult = metaweb.write(query, credentials)

# Output the resultprint "%s: %s %s" %(state, result['/common/topic/image']['connect'], id)

Once you have run the code in Example 6.5 , use the freebase.com client to view your state quartertopics. You'll see that there are now images on the page.

6.3.3. Uploading DocumentsIt is fairly easy to upload an image and make it visible to users of freebase.com . It is a littletrickier to do the same for textual content. When you upload a document, a /type/content objectis created for that document. This allows the content to be retrieved with /api/trans/raw , but itdoesn't allow it to be viewed in any natural way in the client. To accomplish that, you must createa /common/document object to reference the content, and a /common/topic object to referencethe document. Example 6.6 shows how you can do this.

Example 6.6. uploaddoc.py: uploading HTML documents to Metaweb

import sys, re, metaweb

# Read the content of the file specified on the command line# It must be an HTML file with a <title>filename = sys.argv[1]try:

f = open(filename)doc = f.read()f.close()

except Exception, e:sys.exit(e)

# Search through the document for a titletry: title = re.search("(?i)<title>(.*)</title>", doc).group(1)except: sys.exit("Document has no title")

# Log in to Metaweb. Put your own username and password herecredentials = metaweb.login('username','password')

# Upload the document content to Metaweb.# Note that we hardcode the text/html content type.content_id = metaweb.upload(doc, "text/html", credentials)




# Submit a MQL write query to create a /common/topic and /common/document# for the uploaded contentresult = metaweb.write({'create':'unless_exists',

'type':'/common/topic','id':None,

'name':title,'article' : { 'create':'unless_exists',

'type':'/common/document','id':None,'content':content_id }},

credentials)

# Tell the user what we didprint "Uploaded %s: %s\n\tcontent: %s\n\tdocument: %s %s\n\ttopic: %s %s" % (

filename, title, content_id,result['article']['create'], result['article']['id'],result['create'], result['id'])

This Python program expects the name of an HTML le as a command-line argument. It readsthe le and determines the document title by searching for a <title> tag. It uploads the documenttext with the metaweb.upload() method of Example 6.4 . Then it submits a MQL write query tocreate a /common/topic that refers to a /common/document that refers to the uploaded content. Ituses the document title as the name of /common/topic object.

6.4. Example: Unlinking ObjectsAs discussed in Chapter 5 , Metaweb does not allow objects to be deleted. The closest we cancome to deleting an object is deleting all the links between that objects and others. When important

links such as the object's name and type are deleted, the object effectively ceases to have anyuseful identity. Example 6.7 is an object unlinking utility, which attempts to unlink all objectsyou have created (or a subset with the speci ed type and/or name) in the database. For each objectto be deleted, it rst queries the set of types of the object. Then it queries the value of all propertiesde ned by all of those types. It turns the results of this read query into a write query, by addinga "connect":"delete" directive to each property. Finally, it executes this write query to unlink all properties of the object.

Example 6.7 performs a number of read queries as well as writes. To do this, it uses themetaweb.read() utility de ned in Example 4.3 of Chapter 4 .

Example 6.7 is a relatively complex example. It demonstrates the mqlwrite service, of course,

but also demonstrates low-level reads and writes of types and properties. If you can make senseof this example, you have a solid understanding of the Metaweb architecture.

Example 6.7. unlink.py: unlinking Metaweb objects

import sys, getopt # Command-line parsingimport metaweb # login, read, and write utilities

# A cache that maps type ids to arrays of property information




typecache = {}

# Primitive types other than /type/text and /type/key, which are# handled speciallyprimitives = set(["/type/boolean", "/type/datetime", "/type/float",

"/type/id", "/type/int", "/type/rawstring", "/type/uri"])

# Return information about the properties of type t.# Use a cache to avoid repeated queriesdef getPropertiesOfType(t):

if (t not in typecache):props = metaweb.read({'type':'/type/type',

'id':t,'properties':[{'id':None,

'expected_type':None}]})typecache[t] = props['properties']

return typecache[t]

## Return a MQL write query that will unlink the object with the specified id#def makeUnlinkQuery(id):

# Find all types of the objecttypes = metaweb.read({ 'id':id, 'type':[] })['type']

# Make a list of all properties of all types of the objectprops = []for t in types:

props.extend(getPropertiesOfType(t))

# We start off with queries for the well-known properties of /type/objectq = { 'id': id,

'name': [{'value':None, 'lang':None, 'optional':True}],'type': [{'id':None, 'optional':True}],'key': [{'value':None, 'namespace':None, 'optional':True}]}

# Add properties to this query to ask for the id or value of all# other properties of all types. Note that the way we ask for the value# of a property depends on the expected type of the propertyfor p in props:

propid = p['id']

proptype = p['expected_type']# We now need to add propid to the query.# The value of the property depends on its expected type.if proptype == '/type/text':

q[propid] = [{ 'value':None, 'lang':None, 'optional':True }]elif proptype == '/type/key':

q[propid] = [{ 'value':None, 'namespace':None, 'optional':True }]elif proptype in primitives:

q[propid] = [{ 'value':None, 'optional':True }]




else:q[propid] = [{ 'id': None, 'optional':True }]

# Get the result of this queryr = metaweb.read(q)

# The query is structured in such a way that the result is almost ready# to be reused as a write query. We loop through the properties of the# result, and for any that are non-empty, we copy them to a query object# adding "connect":"delete" to transform it into a MQL write queryq = {}for p,v in r.iteritems():

if p == 'id': # leave the id of the query aloneq[p] = vcontinue

# if the property is not id, then the value is an array# if the array is empty, then skip this property; don't copy to queryif len(v) == 0: continue

# otherwise, iterate through the elements of the array# and add the connect:delete directive to each onefor elt in v: elt['connect'] = 'delete'# and copy to the queryq[p] = v

# Return the MQL write query that we can use to unlink the objectreturn q

def main():

# Use the getopt module to parse the command line argumentstry:opts, args = getopt.getopt(sys.argv[1:],

"u:p:n:t:",["user=","password=","name=","type=","debug"])

except getopt.error, msg:print msgsys.exit(0)

# Now process each of the parsed argumentsusername, password, name, type, debug = (None, None, None, None, None)for o,a in opts:

if (o in ('-u', '--user')): username = aelif (o in ('-p', '--password')): password = aelif (o in ('-n', '--name')): name = aelif (o in ('-t', '--type')): type = aelif (o == '--debug'): debug = True

if username is None or password is None:sys.exit("username and password must be specified")




148



Appendix A. Additional CodeThis appendix collects interesting or useful examples that are too long, or would be too much of a digression to appear in the elsewhere in this manual.

A.1. json.jsExample A.1 is the json.js module that de nes the JSON.parse() and JSON.serialize()functions used in the JavaScript-based examples of Chapter 4 .

Example A.1. json.js: JSON parsing and serialization in JavaScript

/*** json.js:* This file defines functions JSON.parse() and JSON.serialize()* for decoding and encoding JavaScript objects and arrays from and to

* application/json format.** The JSON.parse() function is a safe parser: it uses eval() for* efficiency but first ensures that its argument contains only legal* JSON literals rather than unrestricted JavaScript code.** This code is derived from the code at http://www.json.org/json.js* which was written and placed in the public domain by Douglas Crockford.**/

// This object holds our parse and serialize functionsvar JSON = {};

// The parse function is short but the validation code is complex.// See http://www.ietf.org/rfc/rfc4627.txtJSON.parse = function(s) {

try {return !(/[^,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]/.test(

s.replace(/"(\\.|[^"\\])*"/g, ''))) &&eval('(' + s + ')');

}catch (e) {

return false;}

};

// Our JSON.serialize() function requires a number of helper functions.// They are all defined within this anonymous function so that they remain// private and do not pollute the global namespace.(function () {

var m = { // A character conversion map'\b': '\\b', '\t': '\\t', '\n': '\\n', '\f': '\\f','\r': '\\r', '"' : '\\"', '\\': '\\\\'

},

149

a[a.length] = ',';}a.push(s.string(i), ':', v);b = true;

}}

}a[a.length] = '}';return a.join('');

}return 'null';

}};

// Export our serialize function outside of this anonymous functionJSON.serialize = function(o) { return s.object(o); };

})(); // Invoke the anonymous function once to define JSON.serialize()

A.2. Client-side MQL Queries through aProxy

Chapter 4 includes a JavaScript Metaweb.read() utility function whose script-based implement-ation is shown in Example 4.8 . In this section we present a proxy-based implementation of thesame function.

The proxy-based implementation of Example A.2 behaves identically to the script-based imple-mentation in Example 4.8 . We could convert the test application of Example 4.7 to use this newimplementation simply by changing this line:

<script src="metaweb.js"></script> 

to this:

<script src="metaweb_proxy.js"></script> 

This implementation of Metaweb.read() uses an XMLHttpRequest object to submit the HTTPrequest to the proxy service. ( XMLHttpRequest allows HTTP requests to be scripted with JavaScript.The API for this object is not described here: it is a common feature of all Ajax-based web ap-plication frameworks and should be documented in any modern JavaScript reference.) It uses theJSON.serialize() function to convert the query object to a string, and uses JSON.parse() to

convert the response text back into object form. The implementation of these JSON functions isin Example A.1 .

Example A.2. metaweb_proxy.js: Metaweb queries through a proxy

/*** metaweb_proxy.js:** This file implements a Metaweb.read() utility function using XMLHttpRequest

151Appendix A. Additional Code



* and a server-side proxy script named mqlread.php* For simplicity, this code requires a native XMLHttpRequest object,* which means that it does not work in Internet Explorer prior to IE7.**/

var Metaweb = {}; // Define our namespaceMetaweb.QUERY_PROXY = "mqlread.php"; // The relative URL of the proxy service

// Send query q to Metaweb, and pass the result to function f.// If hasEnvelope is omitted or false, then this function wraps an// envelope around the query.Metaweb.read = function(q, f) {

// Put the query in inner and outer envelopesenvelope = {qname: {query: q}}// Serialize the envelope to a JSON stringvar serialized = JSON.serialize(envelope);// URL encode the serialized queryvar encoded = encodeURIComponent(serialized);// Build the query URL

var url = Metaweb.QUERY_PROXY + "?queries=" + encoded

// Use XMLHttpRequest to submit the request to the proxyvar request = new XMLHttpRequest();// When the response arrives, call this functionrequest.onreadystatechange = function() {

// If the request is done and was successfulif (request.readyState == 4 && request.status == 200) {

// Parse the JSON text of response to an envelope objectvar outerEnvelope = JSON.parse(request.responseText);// Get inner envelope from outer envelopevar innerEnvelope = outerEnvelope.qname;

// Make sure the query was successfulif (innerEnvelope.status == "/mql/status/ok") {// Take the result object out of the response envelopevar result = innerEnvelope.result;// And pass that object to the user's functionf(result);

}}

}// Now send the request to the proxyrequest.open("GET", url);request.send(null);

};

This Metaweb.read() implementation is not complete without the mqlread.php proxy script thatit relies on. Example A.3 shows a very simple implementation of such a proxy: it is a PHP scriptthat simply forwards the URL parameter query to the mqlread service at Metaweb and relies onthe default behavior of the PHP curl_exec() function which directs the response from the for-warded query to the output stream of the script. Notice that this trivially simple script is not afully adequate proxy. For a production web application, you would want to use a fully- f eshedout proxy.




Example A.3. mqlread.php: a trivial mqlread proxy in PHP

<?php$q = str_replace("\\\"", "\"", $_GET["queries"]);$url = "http://www.freebase.com/api/service/mqlread?queries=" . urlencode($q);$request = curl_init($url);

curl_setopt($request, CURLOPT_COOKIE, "###freebase.com cookie data here###");curl_exec($request);curl_close($request);?>

A.3. Example: Auto-completion with mqlreadThis section presents a JavaScript example that demonstrates how to use mqlread to performauto-completion and validation of form input. For the sake of code brevity, the auto-completiondemonstrated here is a fairly simple kind: if there is a single possible completion for the user'sinput, the code below will complete that input. It is also possible, though not demonstrated here,

to use a dropdown menu to list all possible completions of the user's input. (See, for example,http://developer.yahoo.com/yui/autocomplete/ for a JavaScript UI that allows the user to chooseamong multiple completions). Metaweb servers also support a specialized (and optimized) auto-completion service, which is used by the freebase.com client but is not documented here.

The auto-completion and validation code is in a function called Metaweb.addValidationAndCom-pletion() . Before showing the implementation of this function, however, we'll show how it isused. Example A.4 is a simple HTML page that displays two HTML input elds. One eld expectsthe name of a country, and the other the name of a rock band. The page includes a script thatcalls Metaweb.addValidationAndCompletion() to set up appropriate event handlers for those

elds. The page also includes a CSS stylesheet that de nes styles for error messages and back-ground colors for input elds that contain invalid or incomplete input.

Example A.4. completiontest.html: using Metaweb.addValidationAndCompletion()

<script src="json.js"></script><script src="metaweb.js"></script><script src="validateAndComplete.js"></script><script>window.onload = function() {

// When the document loads, set up autocompletion for our text fieldsMetaweb.addValidationAndCompletion(document.getElementById("country"),

"/location/country",document.getElementById("countrymsg"));

// Note that we add an additional constraint hereMetaweb.addValidationAndCompletion(document.getElementById("band"),

"/music/artist",document.getElementById("bandmessage")/*{genre:"Rock"}*/);

}</script><style>/* Styles for incomplete and invalid input */


input.invalid { background-color: #f88; } /* invalid input is red */input.incomplete {background-color: #ff8; } /* incomplete input is yellow */span.message { font-style: italic; }</style><body>

Enter a country name (or unique prefix):<input id="country"><span id="countrymsg" class="message"></span><br>Enter the name of a rock band (or unique prefix):<input id="band"><span id="bandmessage" class="message"></span></body>

Example A.5 is the implementation of Metaweb.addValidationAndCompletion() . It begins witha long comment that explains how it works and how it is used. Note that the MQL query used inthis code uses the ~= pattern-matching operator, with ^ and * to nd objects in the graph whosename begins with the user's input. If this query returns no results, then the user's input is invalid.If the query returns exactly one result, then it can be used for auto-completion. And if the query

returns more than one result (none of which match the user's input exactly) then the input cannotbe completed. If the user's input is invalid or incomplete, the function sets the CSS class of thetext eld to give the user feedback, and also optionally displays an error message.

Example A.5. validateAndComplete.js: form validation and completion with mqlread

/*** Add an onchange event handler to the specified textfield object* to validate the user's input and autocomplete it if necessary.* The type argument is the Metaweb type, such as "/music/artist"* for which autocompletion should be done.*

* The optional message argument specifies a document element into* which error messages (such as "invalid input" or "incomplete input"* should be displayed) The optional constraints argument is an object* that contains additional MQL properties that should be added to the* query. This can be use to further constrain the autocompletion* beyond simple type-based autocompletion.** If the callback handler determines that the user's input is invalid or* incomplete, it sets the cssClass property to "invalid" or "incomplete"* (overwriting any other class values specified by that property).* You can define these CSS classes to set background colors or otherwise* highlight fields that require the user's attention.

** Validity and autocompletion is done using the ~= pattern matching* operator to ask for results of the specified type that begin with* the specified string. The query requests that results are sorted by* name and that only the first two are returned. If no results are* returned, this means that the user's input is invalid. If exactly* one result is returned, then the user's input is unique and* autocompletion is performed. If two results are returned, then the* user's input is not unique, but may still be valid, if the input



* matches the first result (because the results are sorted), and that* result is a prefix of the second. If two results are returned and* the first result is not the same as the user's input, then the* input is incomplete.** This function sets the onchange and class attributes of the text

* field element and should not be used with HTML elements that set* these attributes themselves. Also, this function alters the* visibility and content of the optional message element.*/

Metaweb.addValidationAndCompletion = function(textfield, type,message, constraints)

{// Ensure that the message element, if any, is hiddenif (message) message.style.visibility = "hidden";

// And add an event handler to the text field.textfield.onchange = function() {

// Get the user's input and convert to lowercase.// Metaweb does case-insensitive pattern matching.var input = this.value.toLowerCase();

// This is the MQL query we use for autocompletionvar query = [{

type: type, // Find objects of this typename: null, // We want to know object name"name~=": "^" + input + "*", // ^ and * make this a prefix matchsort: "name", // Shortest completions firstlimit: 2 // We only care about the first 2

}];

// Add any additional constraints to the queryif (constraints) for (c in constraints) query[0][c] = constraints[c];

// Now submit the query and pass the results to the nested functionMetaweb.read(query, function(results) {

// If there are no results, input is invalidif (results.length == 0) {

// Set invalid class and display invalid messagetextfield.className = "invalid";if (message) {

message.innerHTML = "invalid input";

message.style.visibility = "visible";}}// If there is one result or if the first result// matches exactly, input is valid.else if (results.length == 1 ||

results[0].name.toLowerCase() == input) {// Autocomplete the value.// Use capitalization Metaweb returns to us




textfield.value = results[0].name;// Clear class and messagetextfield.className = "";if (message) {

message.innerHTML = "";message.style.visibility = "hidden";

}}// If there is more than one result and no match// then the user's input is incompleteelse {

// Set incomplete class and messagetextfield.className = "incomplete";if (message) {

message.innerHTML="incomplete input";message.style.visibility = "visible";

}}

});}

}