Archive

Archive for February, 2013

Working with Freebase – Part 3

February 21, 2013 Leave a comment

My next step with Freebase was to generalize the way I queried it. I decided to model my API after the Java Persistence API (JPA) which provides a POJO persistence model for object-relational mapping. In this case its an object-structured data mapping. The Freebase graph is a set of nodes and a set of links or relationships between those nodes. My API will provide a straight forward way to retrieve those nodes and relationships in the form of a POJO. This post will describe my initial efforts.

In its most primitive form a Freebase node has a unique identifier, creation time stamp, and creator. Properties specify a relationship. That can be a relationship with another node or primitive value. Unique properties have only one relationship to a node/value. Non-unique properties can have many.  There are also universal properties (name, key, type, permission, mid). Primitive values include int, float, boolean, text, rawstring, uri, datetime, key, and id.

JPA describes an entity as …

An entity is a lightweight persistence domain object. Typically an entity represents a table in a relational database, and each entity instance corresponds to a row in that table. The primary programming artifact of an entity is the entity class, although entities can use helper classes.

The persistent state of an entity is represented either through persistent fields or persistent properties. These fields or properties use object/relational mapping annotations to map the entities and entity relationships to the relational data in the underlying data store.

I will reuse this concept of entity to represent a Freebase node. For example, the following class represents a node of type /people/person.

@FBEntity(type = "/people/person")
public class Person implements Serializable {

	@FBProperty(property_value = "/people/person")
	private String type;
	@FBProperty
	private String id;
	@FBProperty
	private String name;

	public String getType() {
		return type;
	}

	public void setType(String type) {
		this.type = type;
	}

	public String getId() {
		return id;
	}

	public void setId(String id) {
		this.id = id;
	}

	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}
}

Consider the simplest of queries – select all from. In SQL this would look like SELECT * FROM name. MQL does not have a true equivalent.  The following MQL would select type, id, name from node type /people/person.

[{
 "type": "/people/person",
 "id": null,
 "name": null
}]

The code for executing this query looks like this.


public class QueryTestOne {
	private static final String personQuery =
			"[{"						+
				  "\"type\": \"/people/person\","	+
				  "\"id\": null,"			+
				  "\"name\": null"			+
				  "}]";

	public static  void dump(List list) {
		if (list != null) {
			for (T result : list) {
				System.out.printf("result: %s\n", result);
			}
		}
	}

	public static void test1(EntityManager em, String query) {
		Query q = em.createQuery(query);
		List<?> list = q.getResultList();
		dump(list);
	}

	public static void test2(EntityManager em, String query, Class<?> clazz) {
		Query q = em.createQuery(query, clazz);
		List<?> list = q.getResultList();
		dump(list);
	}

	public static void main(String[] args) {
		EntityManagerFactory emf = Persistence.createEntityManagerFactory("https", "www.googleapis.com", "/freebase/v1/mqlread/", null);
		EntityManager em = emf.createEntityManager();

		test1(em, personQuery);
		test2(em, personQuery, Person.class);
	}
}

The Freebase read API will return the results in a JSON response envelop that uses the following general form.

{
  "result": [{ MQL Response Here }],
  "status": "200 OK",
  "code": "/api/status/ok",
  "transaction_id":[opaque string value]
}

result contains the body of results and can either be {} (singleton) or [] array. My API will always use an array. I created  a class MQLMultipleResultResponse to hold the response envelop. For now I’m ignoring the other items such as status.

public class MQLMultipleResultResponse {
	private T[] result;

	public T[] getResult() {
		return result;
	}
	public void setResult(T[] result) {
		this.result = result;
	}
}

The query can be executed with or without a type. If the type is absent the underlying code will use Object. The type defines what kind of of object array will be returned. For example, if Person.class is specified an array of Person will be returned.

The underlying code executes the following …

URI uri = new URI(entityManager.getScheme(), entityManager.getAuthority(), entityManager.getPath(), Util.createQuery(variables), null);

ClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
ClientHttpRequest request = requestFactory.createRequest(uri, HttpMethod.GET);
ClientHttpResponse response = request.execute();
ObjectMapper objectMapper = new ObjectMapper();
MQLMultipleResultResponse body = objectMapper.readValue(response.getBody(), typeReference);
results = body.getResult();

Most of this is straight forward HTTP client and JSON calls. The use of objectMapper.readValue() is somewhat unique. Since MQLMultipleResultResponse uses a generic I initially ran into problems with the following …

objectMapper.readValue(response.getBody(), MQLMultipleResultResponse.class);

objectMapper.readValue() cannot know what type the result body will be so it defaults to a Map. The following isn’t legal …

objectMapper.readValue(response.getBody(), MQLMultipleResultResponse.class);

Fortunately Jackson provides an alternate. Instead of passing the Class you can pass a TypeReference. The following code creates the necessary TypeReference …

TypeReference<MQLMultipleResultResponse> typeReference =
  new TypeReference<MQLMultipleResultResponse>() {
    public Type getType() {
      return ParameterizedTypeImpl.make(MQLMultipleResultResponse.class, new Type[] { resultClass }, null    );
  }
};

The key is providing the correct Type. In this case the subclass ParameterizedType. ParameterizedTypeImpl implements ParameterizedType. Java contains a version of ParameterizedTypeImpl but it’s inaccessible. Implementing ParameterizedType is just complicated enough that I decided to just copy ParameterizedTypeImpl’s source code into my project. It comes with a helpful make() method.

With all this in place the test client generates the following results. In one case an array of objects and in the other an array of Person.

result: {name=Dan Milbrath, type=/people/person, id=/en/dan_milbrath}
result: {name=David Safavian, type=/people/person, id=/en/david_safavian}
result: {name=Robert Cook, type=/people/person, id=/en/robert_cook}

result: org.bwgz.freebase.model.Person@2a444: getName: Dan Milbrath; getId: /en/dan_milbrath; getType: /people/person; getClass: class org.bwgz.freebase.model.Person; }
result: org.bwgz.freebase.model.Person@c126b3: getName: David Safavian; getId: /en/david_safavian; getType: /people/person; getClass: class org.bwgz.freebase.model.Person; }
result: org.bwgz.freebase.model.Person@1c01b97: getName: Robert Cook; getId: /en/robert_cook; getType: /people/person; getClass: class org.bwgz.freebase.model.Person; }
Categories: Freebase, Java, JPA, Structured Data

Working with Freebase – Part 2

February 14, 2013 Leave a comment

In part 1 I demonstrated a method to generate a Freebase MQL query from an annotated class that is intended to mimic JPA’s ORM behavior. Next I’ll show how to use the Spring RestTemplate to execute the query and create an array of objects from the results. The following code fragment illustrates the steps.

RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(new MappingJacksonHttpMessageConverter());

URI uri = new URI("https", "www.googleapis.com", "/freebase/v1/mqlread/", Util.createQuery(variables), null);
ResponseEntity entity = restTemplate.getForEntity(uri, PersonsResponse.class);
PersonsResponse response = entity.getBody();
Person[] persons = response.getResult();

The first step is the create a RestTemplate and then add a JSON message converter (MappingJacksonHttpMessageConverter) to it. Freebase returns the its results in JSON format. An array of results will look like this …

{
"code":          "/api/status/ok",
"result": [
{
"type": "/people/person",
"id": "/en/dan_milbrath",
"gender": {
"type": "/people/gender",
"id": "/en/male",
"name": "Male"
},
"name": "Dan Milbrath"
},
{
"type": "/people/person",
"id": "/en/david_safavian",
"gender": {
"type": "/people/gender",
"id": "/en/male",
"name": "Male"
},
"name": "David Safavian"
},
{
"type": "/people/person",
"id": "/en/robert_cook",
"gender": {
"type": "/people/gender",
"id": "/en/male",
"name": "Male"
},
"name": "Robert Cook"
}
],
"status":        "200 OK",
"transaction_id": "cache;cache02.p01.sjc1:8101;2013-02-14T07:31:07Z;0064"
}

ResponseEntity holds the code, status, and transaction_id properties. Typing it to an appropriate class allows it to hold the result returned in the JSON format. In this case it is PersonsResponse which is a subclass of MQLMultipleResultResponse typed to Person.

class PersonsResponse extends MQLMultipleResultResponse<Person>  {
}

public class MQLMultipleResultResponse<T> {
	private String cursor;
	private T[] result;

	public T[] getResult() {
		return result;
	}
	public void setResult(T[] result) {
		this.result = result;
	}
	public String getCursor() {
		return cursor;
	}
	public void setCursor(String cursor) {
		this.cursor = cursor;
	}
}

The code is available at GitHub.

Categories: Bruce's Posts, Java, Spring

Working with Freebase – Part 1

February 14, 2013 Leave a comment

Freebase provides a remote read service API for accessing Freebase database using the Metaweb query language (MQL). Using an HTTP endpoint an application can send a MQL query and received a result set in the response. The result set is returned in the JSON (JavaScript Object Notation) format.

In my case I wanted to query the database for quotes. For example, return quotations from the database. The following MQL returns all quotations associated with a person in the database.

[{
"type": "/people/person",
"id": null,
"name": null,
"/people/person/quotations": [{
"type": "/media_common/quotation",
"id": null,
"name": null,
}]
}]

This produces a results similar to this:

{
"code":          "/api/status/ok",
"result": [
{
"/people/person/quotations": [
{
"id":   "/en/first_thing_we_do_lets_kill_all_the_lawyers",
"name": "First thing we do, let's kill all the lawyers.",
"type": "/media_common/quotation"
},
{
"id":   "/en/o_brave_new_world_that_has_such_people_int",
"name": "...O brave new world, That has such people in't!",
"type": "/media_common/quotation"
}
],
"id":   "/en/william_shakespeare",
"name": "William Shakespeare",
"type": "/people/person"
},
{
"/people/person/quotations": [
{
"id":   "/m/02hylj7",
"name": "...the shifty, hangdog look which announces that an Englishman is about to talk French.",
"type": "/media_common/quotation"
},
{
"id":   "/en/routine_is_the_death_to_heroism",
"name": "Routine is the death to heroism.",
"type": "/media_common/quotation"
}
],
"id":   "/en/p_g_wodehouse",
"name": "P. G. Wodehouse",
"type": "/people/person"
}
],
"status":        "200 OK",
"transaction_id": "cache;cache02.p01.sjc1:8101;2013-02-14T07:31:07Z;0064"
}

In addition to executing the query I want the code to follow an ORM (Object Relational Mapping) style pattern. The code should know how to build the query, execute it, and return the results as objects. I used JPA annotations as a model for creating the mappings.

I created two annotations:

  • FBEntity – Describes which Freebase resource type the class is associated with.
  • FBProperty – Describes which Freebase property the method or field is associated with.

Using those annotations I create the following classes to represent a person and gender.

Person.java

@FBEntity(type = "/people/person")
public class Person {

	@FBProperty(property_value = "/people/person")
	private String type;
	@FBProperty
	private String id;
	@FBProperty
	private String name;
	@FBProperty
	private Gender gender;

	public String getType() {
		return type;
	}
	public void setType(String type) {
		this.type = type;
	}
	public String getId() {
		return id;
	}
	public void setId(String id) {
		this.id = id;
	}
	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}
	public Gender getGender() {
		return gender;
	}
	public void setGender(Gender gender) {
		this.gender = gender;
	}
}

Gender.java

@FBEntity(type = "/people/gender")
public class Gender {

	@FBProperty(property_value = "/people/gender")
	private String type;
	@FBProperty
	private String id;
	@FBProperty
	private String name;

	public String getType() {
		return type;
	}
	public void setType(String type) {
		this.type = type;
	}
	public String getId() {
		return id;
	}

	public void setId(String id) {
		this.id = id;
	}
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}
}

A query builder class uses the annotations to create a MQL query. This is where all the heavy lifting occurs. The builder will parse the class and for each field/method with an annotation it will generate the necessary MQL. It accepts special directives such as limit which are associated with a class. These directives can be used to control the behavior of the query.  It also accepts an instance of the object being queried and use it’s values to refine the query.

MQLQueryBuilder.java

public class MQLQueryBuilder {
	static private final String OPEN_SQUARE = "[";
	static private final String CLOSE_SQUARE = "]";
	static private final String OPEN_BRACKET = "{";
	static private final String CLOSE_BRACKET = "}";
	static private final String EMPTY = "";
	static private final String SPACE = " ";
	static private final String QUOTE = "\"";
	static private final String COMMA = ",";
	static private final String NEW_LINE = "\n";

	static public int PROPERTY_COMPACT	= 0;
	static public int PROPERTY_PRETTY	= 1;

	private String space = EMPTY;
	private String new_line = EMPTY;

	public MQLQueryBuilder(int property) {
		if (property == PROPERTY_COMPACT) {
			space = EMPTY;
			new_line = EMPTY;
		}
		else if (property == PROPERTY_PRETTY) {
			space = SPACE;
			new_line = NEW_LINE;
		}
	}

	public MQLQueryBuilder() {
		this(PROPERTY_COMPACT);
	}

	private String indent(int tab) {
		StringBuffer buffer = new StringBuffer();
		for (int i = 0; i < tab * 2; i++) {
			buffer.append(space);
		}
		return buffer.toString();
	}

	private String quote(String string) {
		return new StringBuffer().append(QUOTE).append(string).append(QUOTE).toString();
	}

	private Object getValueFromObject(Class<?> clazz, Object object, Field field) {
		//System.out.printf("clazz: %s  object: %s  field: %s\n", clazz, object, field);
		Object value = null;
		String name = "get".concat(field.getName()).toLowerCase();

		Method method = null;

		for (Method m : clazz.getMethods()) {
			if (m.getName().toLowerCase().equals(name)) {
				try {
					method = clazz.getMethod(m.getName());
				} catch (NoSuchMethodException e) {
					e.printStackTrace();
				} catch (SecurityException e) {
					e.printStackTrace();
				}
				break;
			}
		}

		if (method != null) {
			try {
				value = method.invoke(object);
			} catch (IllegalAccessException e) {
				e.printStackTrace();
			} catch (IllegalArgumentException e) {
				e.printStackTrace();
			} catch (InvocationTargetException e) {
				e.printStackTrace();
			}
		}

		return value;
	}

	private String createQuery(Class<?> clazz, Map<Class<?>, MQLProperty[]> map, Object object, Stack<Class<?>> stack, int tab) {
		StringBuffer sb = new StringBuffer();
		String indent = indent(tab);

		if (clazz.isArray()) {
			sb.append(String.format("%s", OPEN_SQUARE));
			sb.append(createQuery(clazz.getComponentType(), map, object, stack, tab + 1));
			sb.append(String.format("%s", CLOSE_SQUARE));
		}
		else {
			FBEntity fbEntity = clazz.getAnnotation(FBEntity.class);

			if (fbEntity != null) {
				stack.push(clazz);
				boolean next = false;
				sb.append(String.format("%s%s", OPEN_BRACKET, new_line));

				for (Field field : clazz.getDeclaredFields()) {
					Class<?> type = field.getType();
					if (!stack.contains(type.isArray() ? type.getComponentType() : type)) {
						FBProperty fbProperty = field.getAnnotation(FBProperty.class);
						if (fbProperty != null) {
							String name = quote((fbProperty.property_name().length() != 0) ? fbProperty.property_name() : field.getName());
							String value = null;

							if (type.isArray()) {
								value = createQuery(type, map, object != null ? getValueFromObject(clazz, object, field) : null, stack, tab + 1);
							}
							else {
								fbEntity = type.getAnnotation(FBEntity.class);
								if (fbEntity != null) {
									value = createQuery(type, map, object != null ? getValueFromObject(clazz, object, field) : null, stack, tab + 1);
								}
								else {
									if (object != null) {
										Object o = getValueFromObject(clazz, object, field);
										if (o != null) {
											value = quote(o.toString());
										}
									}

									if (value == null) {
										value = fbProperty.property_value().length() != 0 ? quote(fbProperty.property_value()) : null;
									}
								}
							}

							sb.append(String.format("%s%s%s:%s%s", next ? (COMMA + new_line) : EMPTY, indent, name, space, value));
							next = true;
						}
					}
				}

				if (map != null) {
					MQLProperty[] directives = map.get(clazz);
					if (directives != null) {
						for (MQLProperty directive : directives) {
							String name = quote(directive.getName());
							String value = directive.getValue() != null ? directive.getValue() instanceof String ? quote(directive.getValue().toString()) : directive.getValue().toString() : null;
							sb.append(String.format("%s%s%s:%s%s", next ? (COMMA + new_line) : EMPTY, indent, name, space, value));
							next = true;
						}
					}
				}

				sb.append(String.format("%s%s%s", new_line, indent, CLOSE_BRACKET));
				stack.pop();
			}
		}

		return sb.toString();
	}

	public String createQuery(Class<?> clazz, Map<Class<?>, MQLProperty[]> map, Object object) {
		return createQuery(clazz, map, object, new Stack<Class<?>>(), 0);
	}

	public String createQuery(Class<?> clazz, Map<Class<?>, MQLProperty[]> map) {
		return createQuery(clazz, map, null);
	}

	public String createQuery(Class<?> clazz) {
		return createQuery(clazz, null);
	}
}

The following test program will produces two queries. One will return all persons and the other only males. The limit directive limits the query to just three results. Passing an instance of Person with the Gender set to “Male” will refine the results to only males.

PersonTestOne.java

public class PersonTestOne {

	public static void main(String[] args) {
		Map<Class<?>, MQLProperty[]> map = new HashMap<Class<?>, MQLProperty[]>();

		MQLProperty[] directives = new MQLProperty[] {
				new MQLProperty("limit", new Integer(3))
		};
		map.put(Person.class, directives);

		MQLQueryBuilder builder = new MQLQueryBuilder(
				MQLQueryBuilder.PROPERTY_PRETTY);
		String mql = builder.createQuery(Person[].class, map);
		System.out.println(mql);

		Gender gender = new Gender();
		gender.setName("Male");
		Person person = new Person();
		person.setGender(gender);
		mql = builder.createQuery(Person[].class, map, person);
		System.out.println(mql);
	}
}

Running the program generates the following MQL.

[{
 "type": "/people/person",
 "id": null,
 "name": null,
 "gender": {
 "type": "/people/gender",
   "id": null,
   "name": null
   },
 "limit": 3
 }]

[{
 "type": "/people/person",
 "id": null,
 "name": null,
 "gender": {
 "type": "/people/gender",
   "id": null,
   "name": "Male"
   },
 "limit": 3
 }]

The next step is to execute the query in a manner that populates an array of objects with the results.

Categories: Bruce's Posts