Archive

Archive for the ‘Bruce's Posts’ Category

Working with Freebase – Part 2

February 14, 2013 Leave a comment

In part 1 I demonstrated a method to generate a Freebase MQL query from an annotated class that is intended to mimic JPA’s ORM behavior. Next I’ll show how to use the Spring RestTemplate to execute the query and create an array of objects from the results. The following code fragment illustrates the steps.

RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(new MappingJacksonHttpMessageConverter());

URI uri = new URI("https", "www.googleapis.com", "/freebase/v1/mqlread/", Util.createQuery(variables), null);
ResponseEntity entity = restTemplate.getForEntity(uri, PersonsResponse.class);
PersonsResponse response = entity.getBody();
Person[] persons = response.getResult();

The first step is the create a RestTemplate and then add a JSON message converter (MappingJacksonHttpMessageConverter) to it. Freebase returns the its results in JSON format. An array of results will look like this …

{
"code":          "/api/status/ok",
"result": [
{
"type": "/people/person",
"id": "/en/dan_milbrath",
"gender": {
"type": "/people/gender",
"id": "/en/male",
"name": "Male"
},
"name": "Dan Milbrath"
},
{
"type": "/people/person",
"id": "/en/david_safavian",
"gender": {
"type": "/people/gender",
"id": "/en/male",
"name": "Male"
},
"name": "David Safavian"
},
{
"type": "/people/person",
"id": "/en/robert_cook",
"gender": {
"type": "/people/gender",
"id": "/en/male",
"name": "Male"
},
"name": "Robert Cook"
}
],
"status":        "200 OK",
"transaction_id": "cache;cache02.p01.sjc1:8101;2013-02-14T07:31:07Z;0064"
}

ResponseEntity holds the code, status, and transaction_id properties. Typing it to an appropriate class allows it to hold the result returned in the JSON format. In this case it is PersonsResponse which is a subclass of MQLMultipleResultResponse typed to Person.

class PersonsResponse extends MQLMultipleResultResponse<Person>  {
}

public class MQLMultipleResultResponse<T> {
	private String cursor;
	private T[] result;

	public T[] getResult() {
		return result;
	}
	public void setResult(T[] result) {
		this.result = result;
	}
	public String getCursor() {
		return cursor;
	}
	public void setCursor(String cursor) {
		this.cursor = cursor;
	}
}

The code is available at GitHub.

Categories: Bruce's Posts, Java, Spring

Working with Freebase – Part 1

February 14, 2013 Leave a comment

Freebase provides a remote read service API for accessing Freebase database using the Metaweb query language (MQL). Using an HTTP endpoint an application can send a MQL query and received a result set in the response. The result set is returned in the JSON (JavaScript Object Notation) format.

In my case I wanted to query the database for quotes. For example, return quotations from the database. The following MQL returns all quotations associated with a person in the database.

[{
"type": "/people/person",
"id": null,
"name": null,
"/people/person/quotations": [{
"type": "/media_common/quotation",
"id": null,
"name": null,
}]
}]

This produces a results similar to this:

{
"code":          "/api/status/ok",
"result": [
{
"/people/person/quotations": [
{
"id":   "/en/first_thing_we_do_lets_kill_all_the_lawyers",
"name": "First thing we do, let's kill all the lawyers.",
"type": "/media_common/quotation"
},
{
"id":   "/en/o_brave_new_world_that_has_such_people_int",
"name": "...O brave new world, That has such people in't!",
"type": "/media_common/quotation"
}
],
"id":   "/en/william_shakespeare",
"name": "William Shakespeare",
"type": "/people/person"
},
{
"/people/person/quotations": [
{
"id":   "/m/02hylj7",
"name": "...the shifty, hangdog look which announces that an Englishman is about to talk French.",
"type": "/media_common/quotation"
},
{
"id":   "/en/routine_is_the_death_to_heroism",
"name": "Routine is the death to heroism.",
"type": "/media_common/quotation"
}
],
"id":   "/en/p_g_wodehouse",
"name": "P. G. Wodehouse",
"type": "/people/person"
}
],
"status":        "200 OK",
"transaction_id": "cache;cache02.p01.sjc1:8101;2013-02-14T07:31:07Z;0064"
}

In addition to executing the query I want the code to follow an ORM (Object Relational Mapping) style pattern. The code should know how to build the query, execute it, and return the results as objects. I used JPA annotations as a model for creating the mappings.

I created two annotations:

  • FBEntity – Describes which Freebase resource type the class is associated with.
  • FBProperty – Describes which Freebase property the method or field is associated with.

Using those annotations I create the following classes to represent a person and gender.

Person.java

@FBEntity(type = "/people/person")
public class Person {

	@FBProperty(property_value = "/people/person")
	private String type;
	@FBProperty
	private String id;
	@FBProperty
	private String name;
	@FBProperty
	private Gender gender;

	public String getType() {
		return type;
	}
	public void setType(String type) {
		this.type = type;
	}
	public String getId() {
		return id;
	}
	public void setId(String id) {
		this.id = id;
	}
	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}
	public Gender getGender() {
		return gender;
	}
	public void setGender(Gender gender) {
		this.gender = gender;
	}
}

Gender.java

@FBEntity(type = "/people/gender")
public class Gender {

	@FBProperty(property_value = "/people/gender")
	private String type;
	@FBProperty
	private String id;
	@FBProperty
	private String name;

	public String getType() {
		return type;
	}
	public void setType(String type) {
		this.type = type;
	}
	public String getId() {
		return id;
	}

	public void setId(String id) {
		this.id = id;
	}
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}
}

A query builder class uses the annotations to create a MQL query. This is where all the heavy lifting occurs. The builder will parse the class and for each field/method with an annotation it will generate the necessary MQL. It accepts special directives such as limit which are associated with a class. These directives can be used to control the behavior of the query.  It also accepts an instance of the object being queried and use it’s values to refine the query.

MQLQueryBuilder.java

public class MQLQueryBuilder {
	static private final String OPEN_SQUARE = "[";
	static private final String CLOSE_SQUARE = "]";
	static private final String OPEN_BRACKET = "{";
	static private final String CLOSE_BRACKET = "}";
	static private final String EMPTY = "";
	static private final String SPACE = " ";
	static private final String QUOTE = "\"";
	static private final String COMMA = ",";
	static private final String NEW_LINE = "\n";

	static public int PROPERTY_COMPACT	= 0;
	static public int PROPERTY_PRETTY	= 1;

	private String space = EMPTY;
	private String new_line = EMPTY;

	public MQLQueryBuilder(int property) {
		if (property == PROPERTY_COMPACT) {
			space = EMPTY;
			new_line = EMPTY;
		}
		else if (property == PROPERTY_PRETTY) {
			space = SPACE;
			new_line = NEW_LINE;
		}
	}

	public MQLQueryBuilder() {
		this(PROPERTY_COMPACT);
	}

	private String indent(int tab) {
		StringBuffer buffer = new StringBuffer();
		for (int i = 0; i < tab * 2; i++) {
			buffer.append(space);
		}
		return buffer.toString();
	}

	private String quote(String string) {
		return new StringBuffer().append(QUOTE).append(string).append(QUOTE).toString();
	}

	private Object getValueFromObject(Class<?> clazz, Object object, Field field) {
		//System.out.printf("clazz: %s  object: %s  field: %s\n", clazz, object, field);
		Object value = null;
		String name = "get".concat(field.getName()).toLowerCase();

		Method method = null;

		for (Method m : clazz.getMethods()) {
			if (m.getName().toLowerCase().equals(name)) {
				try {
					method = clazz.getMethod(m.getName());
				} catch (NoSuchMethodException e) {
					e.printStackTrace();
				} catch (SecurityException e) {
					e.printStackTrace();
				}
				break;
			}
		}

		if (method != null) {
			try {
				value = method.invoke(object);
			} catch (IllegalAccessException e) {
				e.printStackTrace();
			} catch (IllegalArgumentException e) {
				e.printStackTrace();
			} catch (InvocationTargetException e) {
				e.printStackTrace();
			}
		}

		return value;
	}

	private String createQuery(Class<?> clazz, Map<Class<?>, MQLProperty[]> map, Object object, Stack<Class<?>> stack, int tab) {
		StringBuffer sb = new StringBuffer();
		String indent = indent(tab);

		if (clazz.isArray()) {
			sb.append(String.format("%s", OPEN_SQUARE));
			sb.append(createQuery(clazz.getComponentType(), map, object, stack, tab + 1));
			sb.append(String.format("%s", CLOSE_SQUARE));
		}
		else {
			FBEntity fbEntity = clazz.getAnnotation(FBEntity.class);

			if (fbEntity != null) {
				stack.push(clazz);
				boolean next = false;
				sb.append(String.format("%s%s", OPEN_BRACKET, new_line));

				for (Field field : clazz.getDeclaredFields()) {
					Class<?> type = field.getType();
					if (!stack.contains(type.isArray() ? type.getComponentType() : type)) {
						FBProperty fbProperty = field.getAnnotation(FBProperty.class);
						if (fbProperty != null) {
							String name = quote((fbProperty.property_name().length() != 0) ? fbProperty.property_name() : field.getName());
							String value = null;

							if (type.isArray()) {
								value = createQuery(type, map, object != null ? getValueFromObject(clazz, object, field) : null, stack, tab + 1);
							}
							else {
								fbEntity = type.getAnnotation(FBEntity.class);
								if (fbEntity != null) {
									value = createQuery(type, map, object != null ? getValueFromObject(clazz, object, field) : null, stack, tab + 1);
								}
								else {
									if (object != null) {
										Object o = getValueFromObject(clazz, object, field);
										if (o != null) {
											value = quote(o.toString());
										}
									}

									if (value == null) {
										value = fbProperty.property_value().length() != 0 ? quote(fbProperty.property_value()) : null;
									}
								}
							}

							sb.append(String.format("%s%s%s:%s%s", next ? (COMMA + new_line) : EMPTY, indent, name, space, value));
							next = true;
						}
					}
				}

				if (map != null) {
					MQLProperty[] directives = map.get(clazz);
					if (directives != null) {
						for (MQLProperty directive : directives) {
							String name = quote(directive.getName());
							String value = directive.getValue() != null ? directive.getValue() instanceof String ? quote(directive.getValue().toString()) : directive.getValue().toString() : null;
							sb.append(String.format("%s%s%s:%s%s", next ? (COMMA + new_line) : EMPTY, indent, name, space, value));
							next = true;
						}
					}
				}

				sb.append(String.format("%s%s%s", new_line, indent, CLOSE_BRACKET));
				stack.pop();
			}
		}

		return sb.toString();
	}

	public String createQuery(Class<?> clazz, Map<Class<?>, MQLProperty[]> map, Object object) {
		return createQuery(clazz, map, object, new Stack<Class<?>>(), 0);
	}

	public String createQuery(Class<?> clazz, Map<Class<?>, MQLProperty[]> map) {
		return createQuery(clazz, map, null);
	}

	public String createQuery(Class<?> clazz) {
		return createQuery(clazz, null);
	}
}

The following test program will produces two queries. One will return all persons and the other only males. The limit directive limits the query to just three results. Passing an instance of Person with the Gender set to “Male” will refine the results to only males.

PersonTestOne.java

public class PersonTestOne {

	public static void main(String[] args) {
		Map<Class<?>, MQLProperty[]> map = new HashMap<Class<?>, MQLProperty[]>();

		MQLProperty[] directives = new MQLProperty[] {
				new MQLProperty("limit", new Integer(3))
		};
		map.put(Person.class, directives);

		MQLQueryBuilder builder = new MQLQueryBuilder(
				MQLQueryBuilder.PROPERTY_PRETTY);
		String mql = builder.createQuery(Person[].class, map);
		System.out.println(mql);

		Gender gender = new Gender();
		gender.setName("Male");
		Person person = new Person();
		person.setGender(gender);
		mql = builder.createQuery(Person[].class, map, person);
		System.out.println(mql);
	}
}

Running the program generates the following MQL.

[{
 "type": "/people/person",
 "id": null,
 "name": null,
 "gender": {
 "type": "/people/gender",
   "id": null,
   "name": null
   },
 "limit": 3
 }]

[{
 "type": "/people/person",
 "id": null,
 "name": null,
 "gender": {
 "type": "/people/gender",
   "id": null,
   "name": "Male"
   },
 "limit": 3
 }]

The next step is to execute the query in a manner that populates an array of objects with the results.

Categories: Bruce's Posts

In Search of Quotes

February 14, 2013 2 comments

Wikiquote

My search continues for a source of interesting quotations that I can incorporate in my mobile application. Wikimedia’s Wikiquote appeared to be an excellent source. It holds thousands of quotes. Wikiquote describes itself as a free compendium of quotations that is being written collaboratively by the readers.  The trick is how to access those quotes with an API.

Wikiquote is powered by MediaWiki the software that runs various Wikimedia sites such as Wikipedia. MediaWiki provides a web service API to access its pages. But while the page contents is easily accessed it is not structured in way that allows it to be easily parsed into discrete data elements. In other words, extract quotes from any given Wikiquote page isn’t straight forward. I could write a parser but I suspected that something like this had already been done. This suspicion lead me to  DBpedia.

DBpediaDBpedia describes itself as …

… a crowd-sourced community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to make sophisticated queries against Wikipedia, and to link other data sets on the Web to Wikipedia data.

DBpedia regularly extracts data from Wikipedia and stores it using a Resource Description Framework (RDF) model for data interchange. Those resources can be remotely queried using SPARQL a query language to RDF.  DBpedia’s ontology contains a quotation property. Unfortunately when I started querying DBpedia for resources that had quotes very few were returned. Apparently Wikiquote is not one of the Wikimedia sites that DBpedia sources from. So while DBpedia looked promising it turned out to be a dead end.

Freebase

More searching lead me to Freebase. Freebase is very similar to DBpedia. It is an open collection of structured data that can be accessed using a remote API. Here too data is pulled from a variety of sources such as Wikipedia and stored as a  graph model comprised of nodes (data objects) and relationships between nodes. This model can be queried with Freebase’s proprietary Metaweb Query Language (MQL).

For example, the follow query will return quotations for Albert Einstein.

[{
  "type": "/people/person",
  "id": null,
  "name": "Albert Einstein",
  "gender": {
    "type": "/people/gender",
    "id": null,
    "name": null
    },
  "/people/person/quotations": [{
      "type": "/media_common/quotation",
      "id": null,
      "name": null,
      "subjects": [],
      "limit": 5
      }]
  }]

Here are the results …

<em id="__mceDel">{
  "code":          "/api/status/ok",
  "result": [{
    "/people/person/quotations": [
      {
        "id":       "/en/imagination_is_more_important_than_knowledge",
        "name":     "Imagination is more important than knowledge.",
        "subjects": [],
        "type":     "/media_common/quotation"
      },
      {
        "id":       "/m/02kpjn_",
        "name":     "Great spirits have always encountered violent opposition from mediocre minds.",
        "subjects": [],
        "type":     "/media_common/quotation"
      },
      {
        "id":       "/m/02nrfj2",
        "name":     "Not everything that counts can be counted, and not everything that can be counted counts.",
        "subjects": [],
        "type":     "/media_common/quotation"
      },
      {
        "id":   "/quotationsbook/quote/21171",
        "name": "If men as individuals surrender to the call of their elementary instincts, avoiding pain and seeking satisfaction only for their own selves, the result for them all taken together must be a state of insecurity, of fear, and of promiscuous misery.",
        "subjects": [
          "Instinct"
        ],
        "type": "/media_common/quotation"
      },
      {
        "id":   "/quotationsbook/quote/23603",
        "name": "The ideals which have always shone before me and filled me with the joy of living are goodness, beauty, and truth.",
        "subjects": [
          "Life and Living"
        ],
        "type": "/media_common/quotation"
      }
    ],
    "gender": {
      "id":   "/en/male",
      "name": "Male",
      "type": "/people/gender"
    },
    "id":   "/en/albert_einstein",
    "name": "Albert Einstein",
    "type": "/people/person"
  }],
  "status":        "200 OK",
  "transaction_id": "cache;cache02.p01.sjc1:8101;2013-02-14T06:45:22Z;0064"
}

With Freebase I now had an online source for thousands of interesting quotes. The next step was how best to use them.

Categories: Bruce's Posts, Java