In EP7, host Oladeji Tiamiyu speaks with Memme Onwudiwe about his company’s use of artificial intelligence in preventative dispute resolution and contract management.
“Convergence” is a bi-weekly, limited series of conversations with thought-leaders and practitioners at the intersection of dispute resolution and technology. Host Oladeji Tiamiyu will focus on such topics as the role technology has had in resolving disputes during the pandemic, various ways technological tools have historically been incorporated into dispute resolution, and creative use cases that technology presents for resolving disputes into the future.
Host
Guests
Memme Onwudiwe is part of the founding team at Evisort, Inc., a legal-tech start-up using artificial intelligence in contract management. Evisort grew out of Harvard’s Innovation Lab, and serves a wide variety of customers, including Microsoft and BNY Mellon. He currently serves as Evisort’s Executive Vice President of Legal and Business Intelligence, working with clients to ensure that Evisort’s AI solutions solve their core business problems. Memme founded and chaired Harvard’s legal technology Symposium, an annual conference focused on addressing challenges and opportunities in legal tech. Memme also serves as a board advisor to the Innovation Law Club Africa (ILCA).
Resources
Adobe-Evisort business case study
Transcript
Oladeji 00:01
1234 Welcome to convergence with Oladeji Tiamiyu. So for this episode, I will be chatting with a dear friend Memme Onwudiwe, who is a founding member of Evisort, a legal tech startup using artificial intelligence and contract management. And I know the temptation can be to view contracts as somewhat boring, but contracts and ambiguity around contract clauses, are at the very center of what leads to disputes. So Evisort has been using AI as a tool for managing contracts in a really interesting way that the legal profession should be mindful of. In addition to his work at Evisort, Memme also founded and chairs Harvard’s legal technology symposium, which is an annual conference focused on addressing challenges and opportunities in legal tech broadly. Alright, so let’s get to it. All right, Memme Onwudiwe, you are the executive vice president of legal and business intelligence at Evisort. How you doing?
Memme Onwudiwe 01:17
I’m doing excellent. How are you? Thank you so much for having me.
Oladeji 01:21
Yeah, it’s great to have you on here. I’m doing great, doing great. Now. So I’m aware of Evisort. I actually classify it as a sort of avant garde contract company, and legal tech. And so I actually imagine that you have a special love for contracts. When you were in law school as a student there [Harvard Law School], what was your fondest memory of your contracts class?
Memme 01:49
Oh, that’s, that’s a great question. I had a lot of fond memories from from contracts class. I was with Professor Brewer. I think one thing that I found really interesting was one thing that people typically complain about their contracts classes, is the fact that like much of law school, you really just reading cases, which is a bit ironic. But people go through a whole contracts class and never read a contract. You know, they’re just reading cases about contracts. And, you know, the excerpts of the contract that might be cited in the case, but like, not actually the contracts themselves, and that kind of goes towards how law schools being litigation, you know, a bit focused, right. But I did like on that final exam, one part of it was actually a scenario from from an airline, you know, how someone got kicked off an airline? Well, the airline made the mistake, they double booked, right? So the airline double booked, and they had like, this whole scene pulling this person off a plane right, even though he did nothing wrong. It was their mistake. And so basically, the exam question was that like a press release of that scenario, that was real, and then actually just kind of a copy/paste of both that airlines customer policy, and the kind of rules from the, you know, the the national kind of transportation, TSA service and, and how they say that airlines have to treat people, you basically had to kind of do your own interpretation of how they both state things based off that scenario. And I found that interesting, because you got to read the actual contracts and actually analyze a document itself.
Oladeji 03:25
Yeah, that’s, that’s really cool. I feel like airlines are in a particularly, it’s a good hypo, I would say on an exam because it’s almost predictive of where airlines have been for the past 20 months with the pandemic where they’ve had to some airlines have had to remove passengers because they refuse to wear masks, or they didn’t comply with certain national or company specific rules and regulations. And so what to do when there is, from the airline’s perspective, you know, like a material breach is, is not the clearest thing, at least when you’re a customer because you just buy the ticket, you don’t really read the fine print all that much.
Memme 04:15
Mmm hmmm, that makes a lot of sense. And I feel yeah, basically, I mean, with a lot of these adhesion contracts, even if you didn’t read it, no one reads the term negotiates. You could read it, you wouldn’t really, you know, do anything you basically it kind of at those whims, right. It’s interesting, especially when they differ, right, when there’s some planes, you know, where, you know, you have to have a seat that’s empty in between the, other you know, whether it’s not and it’s kind of up to the consumer to decide, kind of, you know, kind of weigh those differences, right. So it can get interesting.
Oladeji 04:46
Yeah, yeah. Well, it sounds like Professor Brewer asked a really great hypothetical and relevance for the current moments in this exam a couple years a few years ago. So I guess I’m wondering, it also raises the issue of data management, right? These large corporations, they have so many customers, so many different stakeholders at play. And sometimes they make mistakes. And with that hypothetical is basically like double booking and not knowing it until the final moment when it was too late. So Evisort is, my understanding, trying to solve this challenge with contract management,Evi with about greater data analysis with managing contracts. And your CEO, Jerry Ting, he told me a couple years ago, when Evisort was still at Harvard’s Innovation Lab, he mentioned that Evisort just stands for evidence sorter, which I thought was a really cool, a good combination. And so maybe you could speak to like the importance of contract management in this specific moment, and the importance of sorting evidence through the approach that Evisort is taking?
Memme 06:16
Yeah, no, that’sa that’s a really good, that’s a great anedcote. I will say the name was a little bit more litigation focused before, when we initially as a brainchild, or was more on the litigation side, but more it’s still on now, it’s more in the contract side, on the transactional side of the house. But to your point, it really is about kind of sorting through data. And it’s basically about turning contracts, you know, into data, because as you just said, there’s a lot of important information, you know, at the micro level, that is within those contracts. And so when these different situations come up, it’s not just legal, who needs to know, information within contracts. So it’s actually business units and teams across the organization, that have important information they need to get to within contracts. But legal is the holder of the contracts, not because of the necessity of it to their kind of day to day, but because they’re the ones who can understand them and make sure they’re correct. Rather, the custodians of the contracts, you know, from, from that perspective. What we’re really, you know, getting at when we think about kind of turning contracts into data, right, and really being able to kind of get the key information to the correct folks across the organization, you know, to give you like, another micro example, if I’m in the marketing department, right? And I want to put someone’s logo on our website. And I wonder, can we do that? I mean, today, what I would do is I that’s legal, and then, you know, ruin someone’s day, right, who has to go through, you know, find the different agreements, you know what those people find that exact language and get it to them, right. But if you think about that different information within contracts as data, then you be able to not give the entire contract to that person in marketing, right, but just the relevant pieces to that, right, because that data is construction. So you can pass the relevant information to the relevant people. Sos a lot of the busy work that’s done by legal to retrieve information from contracts and get them to relevant business units can be automated, freeing up legal to do higher level work across the organization. And that’s kind of been our approach to kind of looking at contracts from that data-centric perspective.
Oladeji 08:19
Yeah, yeah, that’s, that’s so interesting. I just to zoom in on turning contracts into data, it makes complete sense because there is so much information in a contract, and it’s not digestible to people who are outside of the legal community. So converting that into data, I’m sure is useful for just like awareness. But it’s also I imagine in terms of data analytics, really powerful when those words are now in a data form. And you can analyze them, you can process them and maybe do some predictive evaluative work. So maybe take a step back, because it’s clear, you have a special love for contracts, I kind of just wanted to explore how you think about contracts from a bit of a theoretical way, because now you’re describing contracts in the within the transactional perspective. And for me, I’ve thought of contracts is really powerful for preventative dispute resolution. Right? So this This podcast is all about dispute resolution. I think like contracts are at the core of disputes, and having clarity, and sufficient understanding and consent from all contracting parties can be really powerful to prevent these contracting parties from entering into a disputes in the future. So that’s kind of how I think about contracts. How, what role do you think contracts plays in from this like, theoretical framing in business management.
Memme 10:04
Yeah, it’s, it’s interesting, right? Because so much of contracts, if we’re gonna go to the theoretical, right, so much of it is really just trust. And so much of it is really writing things that frankly, if any of those things actually enforced, then things have already failed, right? And you kind of go into a contract not wanting to trigger any of its terms, right? Because you’re going in wanting to do something great. And this is just the failsafe that kind of hopefully, gets dusty, right? Like it’s best if a contract, kinda gets dusty, you know, he kind of sits in a drawer, I think that there is much less active enforcement of contracts, then we can kind of assume, much like the law more generally. But that’s a conversation for different day. But when you think about active enforcement of contracts, anything about you know, companies with hundreds of 1000s, millions of contracts, there’s not eyes on that all the time. And like I said before, in most folks best case scenario, it sits in the drawer and gets dusty, right. And so I think that you’ll find a lot of the times it gets in the drawer, and sits in a drawer and gets dusty until events on the ground trigger, you know, situations when you need to go and look at it, right. But there’s not that much of actively going in to the contract ahead of time and looking for opportunities to leverage pre-negotiated things to your advantage, because of the issue I talked about before. It’s like it’s very difficult to and you know, it’s so difficult to that, you know, there’s a whole professional people just to kind of be the ones to handle them. And then anyone else going to them has to kind of go through them, right? And it’s the idea of like, you know, for example, let me give you the idea of rebates, right? We got clients like this all the time, where their lawyers, they go into these negotiations, they spend months, and they negotiate these amazing things like, hey, for every 10,000 quid that we buy, you know, we get x widgets, right? And it’s like, every time we’re scaling, we’re making all this money, right? Then negotiate this amazing language but they never effectuate it because who’s gonna go to page 85 of this 100 page contract to find that table, and look, it’s okay. So when we’re in between this band of, you know, volume of widgets purchased, we’re saving this much, right? So we’ll even create policies that say, hey, we’ll only enforce rebates on contracts that might be over $75,000 in value, because those are the ones was actually the amount we’re gonna make from that rebate is actually worth the effort of going through every contract of that size and reviewing looking for that, right. Which is to say that they’re actively leaving money on the table, right? Just because of like, how difficult it would be to get there. Which is to say that, like much of law, it’s a question of enforcement, right? And, you know, not most contracts might be sitting and getting dusty, but when there’s opportunities to leverage them, you know, folks get it. And I think another step, you know, to your point talking about the now, obviously, COVID is, you know, a situation where, that is a perfect example where, nobody was keeping track of every force majeure clause, They were, you know, it was a throwaway clause in contracts for a lot of people. It would be against human rights to ask your paralegal for no reason just to be copy/pasting every single, you know, force majeure clause of me keeping track of everyone that talks about the pandemic, but when something occurs, you need to have that information and then you kind of scrambling, right? And had you been storing that data, which is what a contract is really, frankly, the most important data at a company right actually would elevate the role of the lawyer if they could look at the contract and this different perspective, right. But if you didn’t look at it like data, that was just a data point that was kind of missing right. If you can bring that data centric view, you won’t be caught off guard by you know, a force majeure being triggered, because of course, your visibility and everything you’ve signed, you signed it right? like I think, I think that I think that we often think that you know, these are perfectly all we know, everything we signed, we know everything in our contracts, when in reality, it’s very much not the case.
Oladeji 14:02
Yeah, that’s, that’s so fascinating. I just so many thoughts, and you touched on the example with like variable rates, and how they can be kept in a closet and get dusty. And I just imagine like, negotiating those variable rates within rebates, like they were done very deliberately. And I’m sure hopefully, it met both contracting parties interests yet, because some of these contracts, especially with larger corporations, are so large, they can easily be forgotten. And so that’s basically even though the contract negotiation satisfied the clients and the contracting parties interests with time, just just the mere scope of the contracts can lead to these key items being forgotten. And I think that raises the question,, and I think this is a problem that Evisort is specifically designed to solve. The question is how do we manage these contracts that have somewhat material terms that are forgotten with time? So that I guess the original intent, the original spirit of the negotiators are fulfilled? How can how does Evisort seek in converting contracts into data, how does Evisort seek to solve that problem of forgetfulness? Or dusty contract clauses?
Memme 15:41
Yeah, I mean, first of all, you need to think about the fact that that occurs everywhere, right? So many companies, you even think about turnover of lawyers, right? You know, as lawyers come and go, it’s not that they’re going in there first, spending the first weeks reading every contract that has been signed to that company, right? Nobody’s doing that. Right. So over time, that’s probably occurring in most companies, you also think a lot of companies, I’ll get their first in house lawyer until kind of years into kind of being companies, right. So there’s already probably hundreds of contracts been signed at that point, that kind of, unless you’re spending your first couple weeks going through meticulously keeping track of data on all of them, you know, I’m, I’m just trying to get to the point that that idea of kind of a key person to describe what you’re asking, like, hey, what is the person who’s been doing all these negotiations leaves, I would posit that that’s kind of probably the standard of like, where most companies are at where there isn’t someone who has that visibility, you know, into all the contracts, I guess, you know, in that frame, it’s more of a reactive, you know, something in the world happens. And it happens on, you know, a paradigm that isn’t covered in the five things I track in all my contracts, right? Let alone I started checking those five things 1000 contracts in, and maybe three of them I started looking into, and we started to be year ago. So even these data sets, you think you have a not perfect data sets and running an analysis over non perfect data set doesn’t get you anything. So at the end of the day, you know, these manual systems lead to I mean, at the end of the day manual review. What’s required is going into every contract, and either copy/pasting that language into somewhere else, where you keeping track of it, or you just keeping spreadsheets and spreadsheets of data, we’re going to a contract management system, where it’s like, oh, great I bought a contract management system. But it’s not managing the contract, when you’re the one that it’s just an Excel with, like a nice color. You’re still the one entering all that data. And you know what I mean, it’s repositories, right? And that’s why when we I mean, because, you know, we started this as students, or even when we saw that as the world, and we’re like, it’s 2016, its about to be 2020, it is ridiculous that I just spent hours typing and effective date and clauses into a contract uploaded to a contract management system, and if I wanted to remember any of that, to type it again, it doesn’t make any sense. [laughs] What’s the point of this system? So yeah, we just thought, you know, it’s very natural to make a contract management system that’s actually managing your contract. You don’t have to tell it this will expire in two years, unless it will forget. You put it in, then it’s like, Oh, this will expire in two years. That’s a list already made. You know, here’s all the key clauses. Because to your point, you might not need the force majeure that day.
Oladeji 18:17
Right.
Memme 18:18
But it’s good to have it. And frankly, even in the event when you’re using AI with the second use, because I mean, you’re always going to be tracking X number of clauses and data points in your contracts. And there’s always going to be a situation that you need x plus one. Let me tell you, I mean, like we do 60 data points out of the box, most people keep track of like five or 10 manually, and like I said, usually incomplete, they started different ones at different times, right? And so, but the real issue comes when its set x plus one, right? And that x plus one, it’s either Hey, we’re gonna start tracking this moving forward, which is great, but just creates an incomplete data set, or we’re gonna have to manually go through every one of our existing agreements, and you know, determine whether or not it has that, you know, information or not. And even when folks are doing that, they’re probably not copy/pasting a clause. They’ll either say, oh, it has the clause doesn’t have it, or it’s standard or it’s not standard. But if something like COVID happens, you need to say does the force majeure clause say pandemic, epidemic, or quarantine. Show me every one that says that. You’re gonna have to do it again, you know, I mean, it’s just it’s a very imperfect system. It’s driven by kind of events, right? And I guess this is a passive system. But you know, how Evisort does it. Evisort is passively stru . . . all we do is we structure unstructured data. We just focus on contracts. And so we’re just passively structuring this data and in contracts as it’s being stored within our system. You think about you know, you think about kind of the difference between like you know, file cabinets and Google right. You know, the file you need to know exactly what folder it’s, in what sub folders, and what sub sub folders, what document it is and what page the information you’re looking for in that document is and then you go for it right? You go to Google you don’t say hey, look exactly here for this. You just say, hey, show me this. Show me that. And it’s already structured the data of the internet. So it’s instantaneously showing exactly what you’re looking for, right. And that’s what we’re doing the contracts, right? You know, we’re structuring the data in them. And so as issues arise that you maybe didn’t conceive of previously, it’s not going to be a fire drill. It’s something that’s on hand. And we enable users to train their own algorithms. So if there is an x plus one situation, you can just show the AI some examples of x plus one and it’s going to do that backwards and forwards. So all the existing agreements and new ones it finds then it’ll start looking for it, right. And that’s how our approach to kind of contracts, you know, kind of addresses the issues you were talking about.
Oladeji 20:37
Gotcha. So basically, there are sometimes these dusty contracts and dusty contract clauses in in our cabinets. And Evisort uses, or seeks to structure unstructured data. Basically, the contracts themselves that are large in number with so much going on within them, Evisort’s contract management tools are bringing structure to all of that data there. So that makes complete sense. And I can see why there’s such a strong value proposition with what you’re providing to the, to the market. And so you mentioned in passing, in what you just shared, that AI is involved in this contract management tools that Evisort of uses. So could you speak to like, how specifically artificial intelligence is being used at Evisort for contract management?
Memme 21:40
I mean, yeah, AI is the core of our system and I would say in the years that we’ve been around, I mean, we have a full CLM. We’re, you know, able to draft contracts, go through approval workflows, pre signature, you know, do the signatures and all the way through storage and alerting and audits and, you know, things down the line, you know, from that perspective. But the first thing we developed, we were literally, four law students at from Harvard, and data scientists from MIT, and a couple computer scientists as well. There wasn’t even a business person around and we were like, what we were looking at, like I said, in the beginning, we thought it was crazy. But once we graduated, you’re going to be paid hundreds of 1000s of dollars for the first couple of years of just copy/pasting indemnities, and we thought we could probably, it’s like 2016, that could definitely be automated, right? This is the first idea we had was the AI of why is it that we’re going to these years of training and intense legal thought through case theory to come out getting, paid a lot, yes, but copy/pasting indemnities in a year, at a time, when we know that’s a process that could be automated, right. And the irony of that is that we ended up copy/pasting lots of indemnity clauses, because that’s how you train the AI. And so the lawyers were doing that, but we were doing it, of course, so no one ever had to do it again. It was kind of like, you know, not kind of a one-off project with a project so that, you know, that kind of would be a manual test that folks wouldn’t have to do. So, we started, we actually had AI before UI. There’s actually a really funny conversation, because like when I [sound quality insufficient] sarted making these algorithms, and we started really becoming very accurate, you know, Jerry was just like, hey, look, let’s start making the platform. This is great. Let’s go the [sound quality insufficient] I go I can’t do that. And that’s when we quickly learned the difference between data science and computer sciences. He could make, you know, algorithms, but he couldn’t make a platform or like a button when your mouse goes down, and it goes bigger. Right? So we have to [sound quality insufficient] separately from the data science team. But [sound quality insufficient] which I think is pretty funny.
Oladeji 23:39
Yeah, that’s that’s so funny. That it mean, like, Evisort overall was developed the, the engineering solutions through artificial intelligence, before the user interface came along that that’s actually really funny. I think that that makes it makes sense. I mean, your startup and you see a very specific problem in in the legal profession and with law firms. And as you mentioned earlier, you kind of are somewhat advancing human rights, because you want to make sure that the paralegal or first year associate isn’t just doing the mind numbing, copy and paste actions with contracts. And so I just want to be clear for people listening. So the way I think about AI and algorithms, because sometimes they’re sometimes treated consistently with one another, and they’re just some small differences. So for me, I think of like an algorithm, it’s just a set of instructions. And there are pre determined outputs that a computer scientist, a programmer writing the algorithm, determines when specific inputs are provided. And then like artificial intelligence is an aggregation of algorithms so that that AI system that has been programmed by the computer scientists, it can perform greater analysis with recognizing the complexities when seeking to generate a specific output. And this is all in comparison, like single algorithms. So basically AI can operate with greater flexibility and handle more complexity relative to a single algorithm. So I just wanted to highlight that for for people listening. And so basically, the inputs for whether it’s an a single algorithm or artificial intelligence, the inputs are really important. And the common fear with people in your industry that are trying to harness the benefits of artificial intelligence is just like the garbage in/garbage out problem. like, how do we get the right inputs so that the outputs that our AI system is generating is not problematic? So how do you how do you think about the garbage in/garbage out problem? And how how does Evisort kind of make sure that the inputs that the AI is analyzing are appropriate inputs?
Memme 26:24
Yeah, I mean, we really just almost over expose it to contracts. I mean, 10s of millions of contracts, we train the AI on from that perspective. But I would say actually, the the best approach is to because I mean, accuracy is always going to be 100% accuracy with algorithms right, isn’t going to be possible because of this predictions, right? Even even what it’s doing is kind of, you know, putting out numbers and [sound quality insufficient] a 90 or 80% level at something like is recognizing something else. But I think what is important is to understand and kind of balance how you approach, you know, any levels of any levels of inaccuracy, right? And so to build it in potentially an over inclusive way, and so that in a way that in a way that if it’s clauses, for example, a real risk would be missing an indemnification clause, versus if there’s 100 clauses, maybe you read five extra that might be on that kind of gray area between indemnification and limitation of liability, right? And then also, if there’s issues that lower quality, right, lower accuracy, right, you know, like, if it’s, you know, low, if the document comes in that a low quality OCR and we can identify, hey, there’s something in this document that lower the accuracy, being able to triage out, right, you know, if there are a percentage where there might require human review, right. And so I think that would be kind of the the kind of best approach for the garbage in/garbage out. But I would say, a lot of the issue that we’re trying to solve is the garbage in/garbage out, which is that for a lot of companies tracking things manually, a lot of that data isn’t reliable, either, because different humans kind of approach different ways of tracking things, or because like I said, it’s very unlikely they’ve been tracking everything from the start, right. And as they choose to track x plus one, a lot of times they’re just starting from that point in time moving forward, right? So when you want to do full analysis looking backward, it’s difficult. And so a lot of times, even if I think a kind of AI approaches are really solid for resolving the garbage in/garbage out. Because of that, even if there is situations where you might require manual review, you can focus that manual review only on those smaller circumstances, the 5% versus having it being 95 or 100%. Like,
Oladeji 28:46
That’s so interesting. Yeah, yeah. So from what you mentioned, Evisort in testing before actually putting it to the market and deploying it with your customers, like the testing takes in hundreds of thousands or millions of contracts, right? There’s a substantial amount, as you said, an overexposure to contracts before it’s actually deployed.
Memme 29:11
Yeah, I’ll just say that he we spent years at the Harvard Innovation Lab doing that. And that’s what we were spending years of the Harvard Innovation Lab like doing, almost as a science project of could we train this and what we did, right, and just like, Oh, well, then we should probably keep doing this because, you know, it actually works. Right. And so yeah, I mean, definitely, I think definitely a different kind of growth journey from other companies as we kind of were a research company, researchers at Harvard Law and MIT, kind of looking at capabilities of AI turn contracts from the data. We actually trademarked earlier this year, going through the process, but when you’re first online trademark [sound quality insufficient] data ties, which I think is a steal.
Oladeji 29:52
Nice. Data ties.
Memme 29:55
I mean, it’s crazy. We snuck in the night before [sound quality insufficient] frankly. [Laughs] But I mean, because that’s what we’re doing. And we I mean, it’s like we don’t just want to digitize your contracts. Digitize contracts are so blase. It’s a scanned PDF, because sometimes you can’t even control F. You need it to be data, frankly.
Oladeji 30:11
And smart move. Smart move, of course, with the data ties trademark now well done. Yeah. So so the overexposure to the contracts came from being with the Innovation Lab. And then like after leaving there, now the deployments are actually being launched. So I guess I was I was just wondering, like, is there a specific threshold? You mentioned 100%, accuracy is impossible. Makes complete sense. But in the testing, I imagine some of your potential customers could be asking questions around the accuracy and whether you have a specific threshold of accuracy before it’s being launched out. So do you have a specific threshold? Or is it more flexible for accuracy percentage?
Memme 30:57
No, we don’t unless it’s over 95 in internal testing, we won’t go. But there folks, frankly, a lot of folks, 95 is very high for a lot of folks, a lot of folks, like [sound quality insufficient] people, when they’re coming in, you’ve got to imagine Oladeji, we’re talking about company, we work with Microsoft, Bank of New York, Mellon, folks with hundreds of thousands, millions of contracts. If you’re saving a human beings having to read 800,000 contracts.
Oladeji 31:23
Yeah
Memme 31:24
Right? And then also you think of it, let me tell you something that I think is also going to be, it’s also going to kind of increase the leverage, the use of these technologies moving forward. And it’s kind of from a data privacy perspective. Also, when you think about the fact that a lot of times, these kinds of processes are being done in kind of other countries, you know, or folks are going to leveraging folks to kind of manually view these contracts was really a data privacy. Facebook can’t tell my favorite color to the folks in Canada. Right? And so for how long will companies be able to send the hundreds of thousands of contracts that, that countries like India, you know, Indonesia elsewhere, right. And so, I think that, you know, when we think about these percentages, you also need to think about the accuracy of a human being, which is under 90%, right? And it’s the same way when you look at AI bias and judging, you start thinking about human bias and judging right? People always talking about AI bias. I could show you lots of human bias. You know, and so I think that I think a lot of times, you know, I love having the accuracy conversations, I think people need to bring the right perspective to think about human plus AI is definitely, you know, that’s way of doing. And I mean, who’s humans, like I just said. It’s very important, but human plus the best, right? And you need to be thinking about it from that perspective, and not be kind of caught up in accuracy numbers, but look at the value number, the value you’re going to be getting from it. And I mean, that’s, that’s like, that rebate example alone is hundreds of millions of dollars being left at the table for some large companies. Right. And so I think that’s the perspective to take.
Oladeji 32:54
Yes, yes. So you’re, yeah, I, I find that really persuasive. Oftentimes, when we think about the trade offs, or the shortcomings and the biases with AI, it’s just the fact the AI is making certain mistakes, or has some biases that it’s operating with. But rarely, or, from my perspective, there’s just an insufficient consideration for like, what are we comparing it to? It’s everything is relative. I’m all about the theory of general relativity. So we have to think about these things in comparison to the alternatives. And you mentioned judging and judicial decision making. And in that camp, it’s basically judges and human decision makers in general have their own biases. And to say that AI is forever flawed, because of a limited set of biases that are strongly influenced by the programmers or just the social context, kind of misses the point around what we’re comparing AI bias to. And the human biases are just as or I would say, incredibly more significant one we’re talking about a individual with concentrated power, who has who has rarely had to, at least in judging rarely had to be judged, rarely had to have been kind of subjugated to the judicial decision making process. They’re usually in the position of power. And so they’re just like, from a system level, there are inherent biases that will come from a judge making the decision when they’ve never or rarely had to have been part of the the party being judged basically. So you’re kind of yeah, I I find that really persuasive and I’m gonna stop rambling on that and go back to what you mentioned around data, data privacy. So I think that’s a really important point. When we when these fortune 500 companies are sending contracts for review to other countries that don’t have as robust data privacy regulations, like the EU, like California, the US, in my opinion has pretty weak data privacy regulations relative to California and the UE’s GDPR approach. But certainly relative to markets that haven’t had to spend a lot of time considering the importance of data privacy, and so Evisort from how you’re explaining, it kind of does have that value add in reducing the need to send all of these contracts to be read in a jurisdiction that has weaker data privacy protections than the US or the EU. So I think that’s really persuasive. Maybe to flip things on its head, Evisort is still a centralized intermediary, right? Like you are taking the data that contracting parties agreed to, and you are consolidating that within your system. And by being an intermediary, I think that there are certain unique risks you’re exposed to, especially with cybersecurity. And so how does Evisort from like a cybersecurity perspective, think about and seek to manage concerns around ransomware and hacks like that?
Memme 36:46
And I mean, I’d say from a cybersecurity perspective, you know, we first I mean, when I talked about us working with the Bank of New York Mellons, and the Microsofts of the world, you can be sure that comes with a lot of penetration testing, and working with that, you know, companies like that. When it comes to, you know, hosting our data, you know, we only work with kind of the the top level kind of providers from that perspective and have a very strong internal computer security and IT security team. And so I really think it’s kind of the focus there that that we have. But just to be very clear, when I’m, I wouldn’t compare risks here to the risks I was talking about before, in that we host our data in the United States, right? And so when you think about issues from a GDPR perspective, it’s about where the data is going from a hosting residency perspective, especially data that has PII, you know, personal identifying information. And so contracts many times can do have personal identifying information, like names, email addresses, and phone numbers. And so a lot of these rules, I’m not saying like, hypotheticals, I’m saying they’ll de facto say that such data with that kind of information can’t be sent to certain places, right. And so I’m just saying it’d be de facto an infraction versus kind of situation of a potential attack or something, just to be kind of clear on that.
Oladeji 38:05
Yeah. Oh, yeah. Yeah, totally. I get that I was, I see the distinction you’re making between data privacy, and the benefits Evisort presents. I was just kind of introducing the cybersecurity lens, since you do have a lot of data that I’m sure your customers are, you mentioned the stress testing piece with Microsoft and BNY Mellon. So that makes complete sense. So right now, there are a couple of important technologies like within the AI genre, and that’s, since you’re reading contracts and converting it into data, you have some companies in your line of work are using like optical character recognition. Others are using ICR. And yeah, I was I was wondering if, if you have any comments around which between OCR and ICR with optical character recognition, which is being used it Evisort because OCR from my understanding has a strong value in just extracting text from scanned documents or JPEGs and then converting it into like machine readable forms, and ICR has, at times been used for like neural network systems. So do you have any thoughts around the OCR/ICR use cases within Evisort?
Memme 39:33
Yeah, yeah. I mean, so we actually we use neural networks within that kind of AI technology. We do leverage mainly OCI. There is actually a case study online. first things we did was build in enhancements for OCI. We actually the case study that is on Adobe’s websites, we’ve been working with Adobe on it for a while now too, in that even that use case I told you about tables is not a typical use case because as you know, a lot of OCRs sometimes engages with tables. Tries to turn it into an image file. And a lot of important data in those tables is then lost, right? We found that, and this goes actually even goes past questions about accuracy, right? To become an AI company, we first had to be an OCR company, so that we could get contracts to a level where AI could actually run an exam [sound quality insufficient] only looking and only reviewing the text that’s been digitally created. And so that text to your garbage in/garbage out question, right? That text is, you know, an issue that’s going to create an issue for us, right, which is why we also kind of flag that for them. You know, even if it does get kind of get past that point. It’s I’d say we’ve leveraged OCR. That being said, but I mean, we’ve still been pushing the envelope on OCI as you can kind of see there. And I definitely take out the look at the case as a good picture of [sound quality insufficient] Innovation Lab, actually. And I think past that, we have a different approach for like, you know, we can identify signatures, right, within signature blocks, right? And so maybe we can read their name, right, we can take the important information that this has been a signed contract, and then have it as a data point of, hey, here’s all your contracts that are signed. Here’s the ones that aren’t signed. Which is a really important use case. We’ve helped companies who are going to buy a company for like millions of dollars, and they have the valuation and they send over their due diligence, and we they run through Evisort. And they say, hey, these are 14 of your sales contracts where they’re not signed by the counterparty. Unless you can find that counterparty [sound quality insufficient] your valuation will drop right.
Oladeji 41:26
Interesting.
Memme 41:27
And that’s the other thing is enforcement, they couldn’t do that, without the visibility into their contracts. So it’s not a question of it’s really a question of enforcement. It’s really a question of visibility, which very little exists. There’s the idea that it exists, because everyone kind of nods. [sound quality insufficient]. They’re not looking back. They don’t know what’s been done.
Oladeji 41:48
Yeah. Wow, that’s, that’s really cool. It’s really cool. How you’re using and expanding on the OCR so that’s super interesting. I want to just maybe switch gears a bit from the technology driving Evisort, and I wanted to switch to how Evisort is impacting the future of education, right. And my understanding is Evisort, every year has been hiring legal fellows that do non traditional work. So how are you engaging with legal fellows, current law students that you hire as legal fellows?
Memme 42:30
Yeah, so yeah, like I said, we started the company as law students, right. And like marketing, sales, you know, finance, you know, leadership is all lost and [sound quality insufficient] an MBA on it, right? And so at the time, and when they designed this, of course, from a business perspective, was Lawson’s doing it. So I mean, we kind of know from that there’s not a job at that company they a law student couldn’t do, because, like, we wouldn’t be a company if that was the case, right? Because we were doing you know, all of those kinds of different kind of jobs. So we actually now have over 20 lawyers working at Evisort and we’re hiring our first legal counsel now. Which is to say those lawyers are working as I mean CEO, COO, or chief of staff. [sound quality insufficient] Account managers you know. Sales people, designers, marketing, right, kind of writing our blog posts, right like all kinds of alternative roles. And so when we look at the 2L fellow program, right which gets active students either during the school year or during their summers opportunities to work at Evisort, like I said, we don’t have a legal team for them to work at you know. They can be on the solutions architects team, right, which of the folks actually you know, developing you know, platforms are folks in the field. Our implementation teams, right? Even marketing, right? Our data science team. Actually, we’re hiring right now. I’ll give a little bit of a plug. Just to work with, you know, Fred’s Malkin on our data science team, right. And he’s a lawyer himself, who’s kind of the innovation work and is working as a data science manager, you know, on our team helping to develop new algorithms. So the the new fellow we’re going to be getting will be working specifically on developing algorithms and working on definitions. Because if you think about it, when you’re turning a contract into data, you’re taking something subjective, but like language and law and contracts and turning it into something objective — numbers. And so the definition of what you’re looking for is very important and there’s actually a lot of the work from a legal perspective in that dynamic, and so they’re going to get a really unique experience you know, working on that during the school year in so kinda you know, for reading contracts, [sound quality insufficient] when you are reading contracts, you see what I’m saying? Interesting opportunities. And actually, I think I’m this year we’ll be bringing on our first [sound quality insufficient] full time worker. I will say, our first ever fellow actually went on to create his own legal tech company, and actually just graduated from Georgetown this last May, and I’m really been honored to lead him throughout it’s a couple of years.
Oladeji 44:54
Yeah, that’s awesome. So Evisort is kind of like an incubation hub. Now it sounds like that. That’s really great. So another question, one of my last questions, is around you, because I know that you do a lot outside of Evisort. And that includes teaching at law schools. And so the academic year is about to start. And I just wanted to figure out what Memme is up to in academia. You have some papers that you’ve written, so could you just speak to what you’ve been doing on on those fronts?
Memme 45:31
[Laughs] yes, yes. Yes, we were talking before this and yesterday, I was a guest lecture at a class at UGA, UGA Law. Actually a really interesting class taught by Brian Mink on legal analytics. He said he’s actually a general counsel at the same time [sound quality insufficient]. And I’ve been as you said, you know, issued to get published in peer reviewed journal on some issues around space law that I was very passionate about. And like I said, we’ve actually, you know, we’ve worked on case studies you know, with Columbia Business School that’s being taught there now and also one with a you know, Professor Scott Westfall that’s being taught in his class at Harvard Law School. Actually just got published last week in an article by from corporate counsel. Actually Louis Firestone you know, the the General Counsel of Louis Vuitton. You know, we were talking about alternative kind of things, basically saying, hey, coming out of is pandemic, everyone’s looking to change. We’ve seen law school operate in a different way, you know, let’s look at how we can optimize law schools ideas, like apprenticeships, kind of having legal tech, or business in house legal, you know, provide more jobs directly. I think that’s the one thing that I’m a little bit passionate about, which is that, we’ve been talking about, especially these last couple years, you know, how big law is becoming a little bit less powerful, as in house kind of flexes its muscle, right? Even this year, with legal tech making billions of dollars, you know, and kind of in kind of investments [sound quality insufficient]. But you’re still seeing that from it comes to opportunities for young lawyers, the standard path is going directly into a law firm, right? Doing a couple years of training, you know, securing the bag and then kind of doing things [sound quality insufficient]. And so how can we, we can’t gripe at big law for not teaching new skills like technology or can I be more business oriented skills, when you’re not providing opportunities for law students inside their careers with business oriented folks in house law, or tech folks, [sound quality insufficient] like being legal tech, and so I think it’s important to, to kind of do that. And that’s one thing that, you know, I’m I think about as well,
Oladeji 47:34
Cool, cool. Yeah. Yeah, that’s awesome. That’s awesome. And I’m really happy and I’m happy for you, and excited to see how some of the thoughts you’re putting on paper and engaging with students will impact the future of the legal profession. So my final question to you is the most important question, and that is what you believe about the future of contract management or contracts generally. And technology, that very few people in your industry believe.
Memme 48:12
Um, I mean, I would say, it’s really just a synthesis of what I’ve been saying kind of this whole time, but contracts are data, and contracts are not only data, but are the most important data at a company, which makes, you know, a general counsel of the company, frankly, the most important data executive at a company because they hold the keys, the information that can drive business and kind of business strategy fully, right. And they had and they can be the ones to provide that visibility to the rest of the company and really elevate the role of the company, but a lot of them kind of, not a lot of them but I just think a lot of the view right now is kind of, you know, either kind of hey focus on reviewing contracts, and you know, reduce risk, right, and then kind of do that day to day. But not as much into the gold mine that is the kind of thousands of contracts you’ve already signed. I think for a lot of companies, they look at like a repository full of 100,000 scanned contracts, and it’s just they’re like, Oh my God, that’s the worst thing ever is, but they should be so lucky. That is a treasure trove. That is 100,000 data points of exactly how your negotiation is going to end. and you can leverage that information to optimize your next negotiation to do it within days and not weeks because you know exactly what they’re gonna do.
Oladeji 49:31
Cool. Cool. Well, thank you so much Memme. I know you have plenty on your plate today so I will let you go, but I just wanted to thank you for the amazing conversation and being a part of it.
Memme 49:42
Nice. It’s been a it’s an absolute pleasure, Oladeji, always great. You know, from when we were section mates back in the day. Happy to be on anytime. Thank you so much.
Oladeji 49:54
Thank you.
