April is the dreaded tax season, which is a brilliant moment to reassess how well are our governments doing in developing tech solutions to all those confusing forms.
I finally got my hands on my own (second hand) copy of Good Services. In the book, Lou Downe notes that “in the public sector […] there are many services who believe they’re users to be brokers, conveyancers, tax specialists or legal advisors.” Does that match your experience with the filings? Do you need to be an expert or hire one?
I hope you will enjoy this opportunity for a little design review while paying your civic duties. And if this is your first encounter with service design, help yourself with my introduction here.
Now let’s dive into some of what caught my eye as of lately.
GPT keeps on taking the Bar Exam(s)
Quick takeaway: GPT4 may or may not have done so well on the Bar Exam. That does not change anything about our very urgent need to rethink how we train the next generation of lawyers.
In 2023, we saw GPT3.5 fail and GPT4 succeed in taking the Bar Exam, shooting for the 90th percentile.
A new article by Eric Martínez of MIT suggests that the 90th percentile claim may not be as accurate as it first seemed. The machine allegedly made it only somewhere around 68th-ish, and into 48th when compared against those who actually passed. This consideration was also noted in the original paper (footnote 5).
Across the pond, the GPT joined other language models in a similar quest, focusing on the multiple choice part of the Czech bar exam. This time, all the models got reportedly beaten by a proprietary model from a prominent Czech law firm. It is worth noting that the main parts of the exam, both in written and oral (!) sections are highly idiosyncratic, so Czech lawyers are in no danger of being taken over by AI anytime soon.
Ultimately, as I pointed out in 2023 Legal Tech Trends, GPT passing the Bar tells us a lot more about the nature of the exam itself rather than GPT’s aptitude. Or as Martin Bregulla put it: an AI passing the bar exam solves nobody’s problem. Martínez offers a similar thought in the paper as well.
One thing is for certain: if there will be machine upheaval any time soon, it will be probably because GPT will finally get tired of taking the bar exam every other week.
Design(s) for Law
Quick takeaway: There is a 460 pages long opus from a group of super knowledgeable legal design experts.
Design(s) for Law are out and have a very impressive lineup. Full 460 pages filled by the most prominent legal design thinkers, on service design, speculative design, and more.
I am still working on my legal design bookclub review for Range, but once I am done, this is going to be my next read.
Once you are done with the collection, check out also the Legal Design Perspectives from the same editorial team. In particular, Legal Design beyond Design Thinking by Margaret Hagan is a mandatory read in my legal design class.
The state of Generative AI in legal context
Quick takeaway: There is significant funding and general excitement around Generative AI in legal context. Liquid Legal Institute predicts that Generative AI will revolutionize the legal industry in the next 3 years.
Here is a quick digest from the CodeX Generative AI Landscape 2024 and the Liquid Legal Institute’s Report of the survey of 203 law firms.
Each took a slightly different approach – while LLI asked law firm professionals about their thoughts and moves, CodeX analysed startup funding. Long story short: law firms as well as investors are generally hungry for more AI. Not much surprise there.
A lot of the developments are very US centric. This means that work in litigation is on the chopping block and at the top of the funding table. Is it that US law firms may be a little bit more used to tech applications thanks to the long-term history with eDiscovery?
One thing is for certain: I will be forever freaking out about the note that reportedly only 58% of law firms reported doing a pilot before implementing an AI solution and only 50% “monitor the implementation very closely”. I guess that I am in a different camp when it comes to discovery and prototyping of new tech.
There is so much more to unpack, including expert opinions and more data – I just watched Mary O’Caroll do a reaction video on Legal Ops Comp Report and now I want to do some too (although I am by no means close to being such a rockstar).
Even more reading on legal design, technology, and AI
If you would like to read even more, here is some of my reading and writing from the past month:
- Contract Lifecycle Management – are you traditional, process-optimised, or tech-enabled type?
- Alternative legal careers: Discover Legal Designer, Legal Engineer, Legal Ops Specialist, and Legal Project Manager
- I coached a training on Design Thinking and career design for law students – and I encouraged people to hold on to their weird hobbies
- The Czech Company Lawyers Association invited me on their annual conference on a panel on AI, and I was super impressed with how proactive lawyers are when it comes to Generative AI implementation.
- Also, the Czech magazine Lawyers & Business reported on Modern Lawyers and Attorney-at-Code (in Czech)
Did you discover anything interesting this month?
Did you find any other cool legal tech resources that you’d like add to spread the love?
Let me know in the comments!
Baru
