Amii is proud to once again sponsor the Competition on Legal Information Extraction and Entailment (COLIEE). By building a community of practice regarding legal information processing and textual entailment, COLIEE has been able to steadily advance the understanding of the process of applying AI to legal reasoning. Now in its sixth year, COLIEE takes place on June 21, 2019 in Montreal, and is run in association with the International Conference on Artificial Intelligence and Law (ICAIL) 2019.

Origins of COLIEE

How many competitions have begun in response to a failed bar exam? At least one. COLIEE was founded by long-time colleagues Randy Goebel (Amii Fellow and Professor at the University of Alberta), and his colleague Ken Satoh (Professor at the National Institute for Informatics).

“Ken and I had always worked on all kinds of applications of AI to reasoning in any domain. Legal reasoning was an early target for AI […] because it’s a domain in which there’s an attempt to be precise about how you write things in natural language,” says Goebel. “And Ken was so keen on this, that nine years ago […] he got very frustrated with trying to take the next step in applying AI to legal reasoning, so he took a law degree.”

Satoh completed a law degree at the University of Tokyo, while at the same time maintaining his position as a full-time professor at one of the most prestigious computer science departments in Japan. Once completed, he took the Japanese bar exams and failed – five times.

“COLIEE was born out of his frustration at the way the bar exams were administered,” explains Goebel. “He and I decided we could stage a competition. We could use Japanese bar law exams as examples with statute law in Japan and mount a competition to see how people could use AI to answer bar law exam questions.”

Since the competition was first staged, Satoh has passed the Japanese bar exams. Meanwhile, Goebel has accomplished the same, albeit with AI; he led a team which designed a program that passed the exams in 2017. But COLIEE still happens every year, growing in challenge and ambition. This year, Goebel and Satoh are just two of six coordinators, and a total of 18 teams are competing from 13 different countries, from the US to Botswana; Argentina to Germany.

The Tasks at Hand

Since its inception, the challenges have steadily increased in complexity; now, the competition includes two categories classified by legal concepts in statute law and case law. Those two categories each have two tasks: an information retrieval task and an entailment task. Teams have the option of submitting for one task, all four tasks, or any number in between.

The information retrieval tasks challenge a program to take a given test case and retrieve the related statutes or cases. The entailment tasks take it a step further.

“Entailment is really just saying: if this is a statute and this is a question, does the statute entail the question or not? [Editor’s note: the Dictionary.com definition of entail is “to cause or involve by necessity or as a consequence: e.g. a loss entailing no regret”] It’s the legal reasoning required to say, if you are wearing an expensive kimono and a bystander pushes you out of the way of a car that’s about the hit you […] is he liable for damages to the kimono?” says Goebel. “So we want to build computer programs that […] retrieve the appropriate statutes, then they have to find a connection.”

In other words, an entailment task requires a program to perform information retrieval, then create a yes/no argument for the test case based on the returned statutes or cases. In the example above, a program would be required to retrieve statutes related to property damage and liability, then determine whether or not the bystander would be liable for the damaged kimono.

The structure of the competition has accelerated research in this area in a unique way. COLIEE has built a community of people who are tackling similar challenges, and as a result, its participants have naturally developed a vocabulary that has quickened the process of exchanging information and ideas.

“It’s almost to that point at the COLIEE competition workshops […] people talk without even specifying. They say, ‘and here’s our approach to task one’. They don’t describe task one anymore.”

Explaining explainability

Explainability – the ability to determine how a model arrived at an answer – plays a large role in the entailment portion of the competition. Goebel’s lab, the Explainable AI (XAI) Lab out of the University of Alberta, is dedicated to this exact concept. Goebel’s lab is competing in COLIEE, and in this aspect, he has an advantage.

“Our lab has been tackling these two areas of medical reasoning and legal reasoning; we want to drive the science of AI forward. So we have an advantage to all of the people on the planet who, in an ad hoc way, apply learning of any kind to these domains, because we’ve been doing them for longer, and our focus is on making them explainable.”

Many machine learning models are challenged by the inability to explain how they arrive at decisions. For example, a model can take points of data and group them together based on a similarity it noticed, but it will not be able to articulate the similarity. Many refer to this issue in metaphor, calling them “black box” models.

Black box machine learning models can perform brilliant tasks – but in application, they can have serious consequences. If a model is assisting a doctor to recommend treatment, a judge to recommend sentencing, or a hiring manager to choose resumes, it is important to be able to determine why it is arriving at decisions.

“Explainability is as simple as saying: when I tell you something, please explain,” says Goebel. “The background that I have comes from formal philosophy and building systems to create hypotheses about data. That’s what scientists do. In there lies all of the mechanisms you need to do explanation.”


The 6th Competition on Legal Information Extraction and Entailment (COLIEE) takes place on June 21, 2019, and is run in association with ICAIL in Montreal, Quebec. Visit https://sites.ualberta.ca/~rabelo/COLIEE2019/ for more information.
Edmonton Mayor Don Iveson presides over the signing of a Memorandum of Understanding between Hong Kong AI and Amii.

Amii and the Hong Kong AI Lab (HKAI Lab) are pleased to announce an ongoing collaborative relationship between the two AI-focused institutes. The announcement follows from a brief signing ceremony at Inventure$ today between Amii’s CEO, John Shillington, and Timothy Leung, Executive Director of HKAI Lab.

The relationship, which was formalized through the signing of a Memorandum of Understanding, will explore possibilities for collaboration between the two groups and will encourage new connections between innovators in both Alberta and Hong Kong.

“We’re so thrilled to embark on this new collaboration,” says Shillington. “We see many similarities between the mandates and activities of HKAI Lab and Amii. We both have much to learn from one another, and we can’t wait to begin exploring the possibilities of a much stronger connection between our organizations.”

“We are pleased to be in Canada today to strengthen the relationship between two respectable AI centres,” says Leung. “Both Alberta and Hong Kong have recently emerged as global destinations of AI research and commercialization, and through this mutually-beneficial relationship, we hope to strengthen transpacific collaboration not only for Amii and HKAI Lab but also for our clients and other partners.”

Established in May last year, the HKAI Lab aims to advance the frontier of AI with cutting-edge technologies and expertise, and empowering startups to develop and commercialize their new inventions and technology. Through different seminars, workshops and sharing sessions, HKAI Lab dedicates to inspire startups with new ideas and knowledge among academics, scientists and entrepreneurs in the field of AI. The HKAI Lab is a non-profit initiative that is funded by the Alibaba Hong Kong Entrepreneurs Fund and SenseTime, with support from the Hong Kong Science and Technology Parks Corporation and Alibaba Cloud. Alibaba Hong Kong Entrepreneurs Fund is a non-profit initiative launched by Alibaba Group to help Hong Kong entrepreneurs and young people realize their dreams and visions for Hong Kong. And SenseTime is an artificial intelligence unicorn based in Hong Kong.

In the coming months, Amii and the HKAI Lab will concentrate on groundwork activities, including exploring potential for advanced collaboration and joint projects. The two organizations will also share insights and knowledge about their respective initiatives.

Community & Events

Catch us at Inventure$

Inventure$ has arrived! This “unconference,” held from June 5-7 at the Telus Convention Centre in Calgary, joins together entrepreneurs, innovators, investors, researchers and thought leaders to discover and share the latest in innovation. We couldn’t be more excited to be part of what is quickly becoming Alberta’s premier business and technology event.

If you are also in Calgary this week, here is where you can catch the Amii team:

Oil Sands Innovation Summit Panel

When: June 4, 3:20 – 4:20 p.m.
Where: Hyatt Regency (700 Centre St SE)
Can Alberta be a global leader in Oil & Gas AI? What is the implication for our businesses? Find out at this panel featuring Geoff Kliza, Director of Amii Innovates. The Oil Sands Innovation Summit takes place June 3 – 4 in partnership with Inventure$. The event is sold-out.

Calgary AI Meetup

When: June 4, 5:30 – 7:30 p.m.
Where: Room EN C70, Engineering Complex, University of Calgary (2500 University Dr NW)
RSVP Here!
Anna Koop from Amii’s science team will be sharing the latest reinforcement learning research happening at Amii, and she’ll be joined by other members of the Amii team to answer your burning questions about machine learning in this latest edition of the Calgary AI Meetup.

Inventure$ Showcase

Where: June 5, 9 a.m. – 4 p.m.
Where: Calgary Telus Convention Centre (120 9 Ave SE)
Purchase Tickets Here!
Find Amii on the conference floor – we’ll be in our booth answering questions, sharing information and demoing research from our Amii Fellows and Innovation Affiliates:

Bento Arm: Developed at the BLINC Lab under the supervision of Amii’s Patrick Pilarski, the Bento Arm is a platform for training and research applications of AI-enabled prosthetic arms. The Bento Arm will be on display from 9 a.m. – 4 p.m.

DeepStack: Developed by a research team led by Amii’s Michael Bowling, DeepStack is the first AI capable of beating professional poker players at heads-up no-limit Texas hold’em poker. DeepStack will be on display from 9 a.m. – 4 p.m.

SAMDesk: SAMDesk monitors social media to determine when important events are taking place around the world. Visit our booth to see the tool work in real time. SAMDesk will be on site from 9:30 – 11:30 a.m.

Mikata Health: Mikata Health uses AI to help doctors and staff connect assist patients more effectively through chat technology. Mikata Health will be joining us from 11:40 a.m. – 1:40 p.m.

Medo.ai: Medo.ai aims to simplify the use of ultrasound for common and critical conditions. Visit the booth to see how they use AI-augmented ultrasound imaging to support diagnosis. Catch Medo.ai at the Amii booth from 1:50 – 3:50 p.m.

ATB Tech on Tap – Machine Learning Mishaps

When: June 5, 5 – 7 p.m.
Where: ATB Financial (102 8 Ave SW)
RSVP Here!
ATB is hosting a very special Inventure$ edition of Tech on Tap happy hour! Drink free beer and listen to Tara and Anna from the Amii science and education teams as they share some memorable Machine Learning fails and mishaps.


Can’t make it to these events? Join us at Hello, Amii in Edmonton on June 19 or Hello, Amii in Calgary on June 24!