In this blog post, we will discuss how ChatGPT, a large language model that can generate legal documents, performs in terms of quality and accuracy. We will use some examples of the generated documents and compare them with the standards of legal drafting. We will also highlight some of the limitations and challenges of using ChatGPT for legal purposes.
ChatGPT Explained for Lawyers
Assuming that you have been living in a parallel world where ChatGPT was never invented, let me give a brief explaination of what ChatGPT is. It is a Large Language Model (“LLM”) that can produce natural language texts on a few keywords or a prompt. It has been trained on a large corpus of text data from various domains, including legal texts. ChatGPT can generate different types of legal documents, such as contracts, agreements, letters, notices, etc., after taking basic instructions.
However, it is pertinent to point out that ChatGPT is not a substitute for lawyers by any stretch of the imagination. Therefore, you should always verify the accuracy, validity, and relevance of any text generated by ChatGPT before using it in your legal work.
Legal Drafting using ChatGPT
As mentioned above, one way to use ChatGPT is to generate sample texts for standard legal documents, such as contracts, pleadings, applications, or motions and can be helpful to lawyers and legal professionals who need to draft legal documents quickly and efficiently.
The generated drafts can be used as a starting point giving lawyers an idea of how to structure and format their document, as well as what kind of language and arguments to use. However, you should not copy and paste the text generated by ChatGPT without checking it for errors, inconsistencies, or inaccuracies. You should also make sure that the text is tailored to your specific case, jurisdiction, and audience.
To use ChatGPT for this purpose, you can provide a few keywords or a prompt that describes the type and topic of the document you want to generate. For instance, the following is a non-disclosure agreement generated by ChatGPT based on the prompt “Draft a Non Disclosure Agreement”:
Non-Disclosure Agreement Generated by ChatGPT
This Non-Disclosure Agreement (the "Agreement") is made and entered into as of the date of last signature below (the "Effective Date") by and between _____________, a _____________ corporation with its principal place of business at _____________ ("Disclosing Party") and _____________, a _____________ corporation with its principal place of business at _____________ ("Receiving Party").
WHEREAS, Disclosing Party is engaged in the development of software products and services (the "Software");
WHEREAS, Receiving Party is interested in evaluating the Software for potential business collaboration with Disclosing Party;
WHEREAS, in connection with such evaluation, Disclosing Party may disclose to Receiving Party certain confidential and proprietary information relating to the Software and its business;
NOW, THEREFORE, in consideration of the mutual promises and covenants herein, the parties agree as follows:
- Definition of Confidential Information. For purposes of this Agreement, "Confidential Information" means any information disclosed by Disclosing Party to Receiving Party, either directly or indirectly, in writing, orally or by inspection of tangible objects, that is designated as "Confidential", "Proprietary" or some similar designation, or that reasonably should be understood to be confidential given the nature of the information and the circumstances of disclosure. Confidential Information includes, without limitation, the Software, its features, functionality, design, architecture, code, documentation, specifications, test results, performance data, business plans, marketing strategies, customer lists, pricing information and any other information relating to the Software or Disclosing Party's business.
- Non-use and Non-disclosure. Receiving Party agrees not to use any Confidential Information for any purpose except to evaluate and engage in discussions concerning a potential business relationship between the parties. Receiving Party agrees not to disclose any Confidential Information to third parties or to its employees or contractors, except to those employees or contractors who are required to have access to such Confidential Information in order to perform their duties in connection with the potential business relationship between the parties and who are bound by confidentiality obligations at least as restrictive as those contained herein.
- Protection of Confidential Information. Receiving Party agrees to protect the confidentiality of Confidential Information in the same manner that it protects its own confidential information of like kind (but in no event using less than reasonable care). Receiving Party agrees to take all reasonable measures to prevent any unauthorized use, disclosure, reproduction or distribution of Confidential Information.
- No Obligation. Nothing herein shall obligate either party to proceed with any transaction between them, and each party reserves the right, in its sole discretion, to terminate the discussions contemplated by this Agreement concerning the business opportunity.
- No Warranty. ALL CONFIDENTIAL INFORMATION IS PROVIDED "AS IS". NEITHER PARTY MAKES ANY WARRANTIES, EXPRESS OR IMPLIED.
- This Agreement constitutes the entire understanding between the parties concerning the subject matter hereof and supersedes all prior and contemporaneous agreements, whether oral or written.
- No amendment or modification to this Agreement shall be effective unless it is in writing and signed by both parties.
IN WITNESS WHEREOF, the parties hereto have executed this Non-Disclosure Agreement as of the Effective Date.
[Your Company/Organization Name]
[Your Name, Title]
[Recipient's Name, Signature]
Evaluating the Generated Legal Documents by ChatGPT
Non-compliance with Applicable Laws
Even though the legal documents generated by ChatGPT are almost always logical and internally coherent, they may not be compliant with the applicable laws and regulations of the relevant jurisdiction. This is because ChatGPT does not have any knowledge or understanding of the legal system or the specific rules that govern different types of contracts or transactions.
It only relies on the data that it has been trained on, which may not reflect the current or accurate state of the law. As per the OpenAI themselves, ChatGPT had a knowledge cutoff from September 2021. That means it does not have access to any information on events that occurred after that date.
For example, ChatGPT may generate clauses that are illegal, invalid, unenforceable, or contradictory to the existing laws or precedents. It may also generate clauses that are irrelevant, redundant, or contradictory to each other. For example, in the non-disclosure agreement above, ChatGPT generated a clause that states:
"Receiving Party agrees not to disclose any Confidential Information to third parties or to its employees or contractors, except to those employees or contractors who are required to have access to such Confidential Information in order to perform their duties in connection with the potential business relationship between the parties and who are bound by confidentiality obligations at least as restrictive as those contained herein."
This clause may not be compliant with some laws or regulations that require disclosure of certain information to third parties for specific purposes, such as health care privacy laws (HIPAA), securities laws (SEC), consumer protection laws (FTC), etc. It may also be too broad or unreasonable to restrict the disclosure of confidential information to all employees or contractors of the receiving party, regardless of their role or function.
Lack of Customization
ChatGPT can generate legal documents that are generally valid and coherent, but they may not be very specific or tailored to the needs of the parties involved. As you can see, the NDA generated above is quite generic. It does not include any details or preferences that may be relevant to the specific situation or context of the parties involved.
For example, it does not specify what kind of software is being developed or evaluated; what are the terms and conditions of the potential collaboration; what are the consequences of breach or termination; etc. These details may vary depending on the nature and scope of the project and the relationship between the parties.
However, by providing more information and preferences as input to ChatGPT, it can generate drafts that are more customized to meet your needs. For example, if we add some more keywords such as "mobile app", "healthcare", "HIPAA compliance", "exclusive license", "royalty payments", "liquidated damages", etc., ChatGPT may generate a more specific and relevant contract. But even this approach has its limitations because of the inherent limitation of training data.
Clause Wise Modification is Very Cumbersome
At least at the time being, you have to rely on ChatGPT's ability to infer your prompts and generate the legal document accordingly. Once, the draft has been generated, it is very cumbersome to make ChatGPT customize each and every clause with specific instructions.
It is again next to impossible to specify the logical connection or coherence among the various clauses of the contract. You also cannot edit or revise the generated document interactively with ChatGPT. For instance, in the above NDA, ChatGPT initially refers to one party as Disclosing Party" and later on as “Your Company/Organization Name”.
Different From The Usual Language Typically Used In Contracts
ChatGPT can generate legal documents on various different topics, but they may not use the language that is typically used in contracts drafted by actual lawyers. A similar finding was noted by researchers from Singapore in a recent paper when comparing contracts generated by ChatGPT with the ones filed with SEC. (See: Para 6.4 of “Applying Large Language Models for Enhancing Contract Drafting”, available here [Last accessed on August 07, 2023]).
This may be due to the fact that ChatGPT has been trained on data from various sources and not just contracts. Also, the output from ChatGPT might be evaluated on the fact that how easy it is for a layperson to understand, a benchmark on which I guess even the best-drafted contracts would fail today.
It has been observed that ChatGPT may use words or phrases that are too informal, vague, ambiguous, or inconsistent for legal purposes. It may also omit or misplace important terms or clauses that are essential for the validity and enforceability of the contract.
For example, in the non-disclosure agreement above, ChatGPT used certain terms that are not very common in modern contracts. It also omitted the governing law and jurisdiction clause, which is usually included in most contracts to specify which law applies and which courts will have the authority to resolve disputes.
Possibility of Hallucinations
Perhaps the most troubling aspect is ChatGPT's potential to generate content based on false assumptions or hallucinations. It lacks the capability to verify or validate the information it generates, leading to inaccuracies and fictional elements in the documents.
The Way Forward?
Considering the limitations of ChatGPT's legal document generation mentioned above, it is clear that it may not meet the rigorous standards set by legal professionals. In critical legal matters, relying solely on ChatGPT might lead to adverse consequences.
However, fear not, a solution has been crafted to address these concerns. Allow me to introduce Amto AI - the result of relentless efforts by a team of diligent lawyers, developers, and data scientists. At Amto, we take pride in eradicating the very limitations that have plagued other AI-powered systems.
What sets Amto apart, you ask? Let me elucidate:
Amto AI leverages semantic embeddings to access all relevant laws applicable to a specific jurisdiction, ensuring utmost accuracy and compliance.
With Amto, users can input specific information and preferences, paving the way for more customized and precise document generation.
Initiative UI for Clause Customization
Our platform provides an intuitive UI to customize any clause, empowering users to provide specific instructions for each clause, thereby elevating the quality of the content produced.
Fine-tuned on billions of lines of contracts, Amto generates language that aligns with the actual practices used by seasoned legal professionals.
Lawyers in Loop
Finally, at Amto we always keep human lawyers in the loop via our internal team and feedback from legal professionals, to ensure that any improvement in LLM actually moves needles in the direction lawyers want.
In conclusion, while ChatGPT is an impressive language model, it falls short on many fronts when it comes to legal drafting. Legal professionals must exercise caution and choose tools that align with the precision and reliability required in the legal domain.
So, when it comes to drafting legal documents, Amto AI stands as a reliable and dependable ally, ready to meet and exceed the highest standards set by legal experts. Experience the difference today and entrust your legal documentation to Amto AI - your ultimate solution for precision, reliability, and peace of mind.