Risk considerations and guidelines for artificial intelligence.

Artificial Narrow Intelligence is a specific type of artificial intelligence in which a learning algorithm is designed to perform a single task and any knowledge gained from performing that task will not automatically be applied to other tasks. While it’s still very beneficial to users, this is considered “weak AI” and examples include search engines and recommender systems that you see on Netflix or Amazon.

Artificial General Intelligence is the ability of an AI agent to learn, perceive, understand, and function like a human being. These systems will be able to independently build multiple proficiencies and form connections and generalizations across fields, massively cutting down training time. This will make AI systems nearly as capable as humans by replicating our multi-functional capabilities. Examples include chatbots that use natural language processing to analyze what humans are saying and create a response, as well as music AIs.

The development of Artificial Superintelligence will probably mark the peak of AI research. In addition to being able to replicate the multi-faceted intelligence of human beings, ASI will be exceptionally enhanced because of greater memory, faster data processing and analysis, and decision-making capabilities.

Fundamentally, this latest revolution in AI is caused by two technologies: large language models (LLMs) and transformers (GPT stands for “generative pre-trained transformers”).

A large language model (LLM) is a prediction engine that takes a sequence of words (such as “the sky is ”) and tries to predict the most likely sequence to come after that sequence (“blue”). It does this by assigning a probability to likely next sequences and then samples from those to choose one. The process repeats until some stopping criteria are met. Large language models learn these probabilities by training on large corpora of text.

Language models have existed for decades, though traditional language models had many deficiencies. In 2017, however, Google wrote a paper, Attention Is All You Need(1), that introduced Transformer Networks(2), kicking off a massive revolution in natural language processing. Overnight, machines could suddenly do tasks like translate between languages nearly as well as and sometimes better than humans. Transformers introduce a mechanism called “attention,” that allows the model to analyze the entire input all at once, in parallel, choosing which parts are most important and influential. Every output token is influenced by every input token, similar to how a human would read and interpret a sentence. When most companies talk about modern uses of AI, large language models are typically what are being referred to in the current zeitgeist.

This is the implementation of AI as part of a company’s products and services. LLMs in particular command a new place in products for translating natural language requests into actions, creating a new paradigm for user experience. Product AI can identify user intent, reduce time spent by users completing tasks, and perform actions on the user’s behalf. This is the most visible and impactful type of AI for customers and can be a market differentiator.

This application of AI refers to all of the processes a business needs to do behind the scenes to function efficiently. Operational AI automates many aspects of back office operations that traditionally may require human intervention to compete. The major benefit of operational AI is in operational speed and cost savings.

Internal AI refers to all the new AI tools and services a company may allow employees to use as part of their role to make them more productive. These can be generative tools that create images to specification, writing tools to create internal documents, or tools that consume large amounts of documents to produce easy-to-consume summaries. Many existing productivity tools from companies such as Google and Zoom are including AI features in existing products and offering them to customers today. Using these tools can carry risks if employees begin using them without properly vetting them first.

You’ll want to consider the copyright implications of using AI to create ideas or products so you can ensure your ability to protect your innovations. The US Copyright Office and the US Patent and Trademark Office have declined to extend copyright or patent protection to outputs created by AI tools, on the basis that only works created by human authors or inventors are eligible for protection in their most recent decision in Thaler v. Perlmutter upholding the requirement for human authorship for copyrights.(3)

Most recently, the US Copyright Office denied protections for another AI-generated image as it was not the product of human authorship.(4) The Office asked the artist to disclaim the parts of the image that AI generated in order to receive copyright protection because it contained more than a minimal amount of AI-created material.

Generally, without these copyright protections, you may not be able to prevent others from copying or reusing the output generated from your input, or to stop an AI platform from using or disclosing identical outputs.

If you are unable to demonstrate and separate the portions that were created by you vs. generated by AI to persuade the USPTO and the US Copyright Office of human authorship, you may not be able to protect your proprietary materials because there is no copyright protection for works created by non-humans, including machines.

It’s also important to know that outputs created by AI may infringe upon third-party intellectual property rights, both due to the nature of the inputs and the nature of your prompts. Several lawsuits have already been filed against AI platforms, alleging that the use of inputs owned by third parties to train models and generate outputs without permission infringes upon their intellectual property rights and violates other rights and laws.

If you use AI to generate an output that refers to, or is inspired by, identifiable third-party materials (e.g., requesting an output displaying a character designed by an artist, or that mimics another person), the output may infringe upon that third party’s intellectual property rights or privacy.

Deepening these risks, most AI platform terms and conditions typically provide no protection against lawsuits based on output, and, in fact, often place liability entirely on the user. This means you may face liability if you generate and use problematic outputs and have no right of indemnification or other recourse to avoid that liability.

If you use an AI platform to develop code, it is critical to understand that these platforms are typically trained on publicly available source code as their inputs — the majority of which are subject to open source licenses(4). Some of these licenses are “copyleft,” meaning if incorporated into your software, may require that you make your proprietary code available for free based on the same copyleft license. Even “permissive” licenses commonly have attribution or other requirements on distribution. Using AI for code development presents those same risks, but without identifying the applicable licenses for each open source code.

In coding outputs, there may also be bugs, vulnerabilities, or security risks. AI platforms typically disclaim any responsibility for output, which means these platforms may provide source code output that is not reliable and may subject the company to legal and security risks.

AI is still a developing technology and it is far from perfect. For the most part, outputs are often accurate and suitable for their use case. Other times, they may contain errors, be misleading or deceptive, or be trained on data that is inaccurate to begin with.

AI might also “hallucinate” (fabricate something but present it as a fact). Remember “The ChatGPT Lawyer”?(5) AI can also generate output that is discriminatory, unethical, and offensive to local customs or community norms. It also could be biased as the output may be influenced by the underlying training inputs with these traits.

Such defects and biases make it extremely important to note that when using someone else’s AI model, you likely will have no insight into what kind of data was used to train the model. The risk is heightened where output is used in circumstances in which accuracy or fairness is essential, such as human resource decisions (hiring and performance management) or when providing services or products to customers (access to credit or insurance, the provision of healthcare, etc.).

When you submit a prompt on an AI platform, unless you negotiate a contract that says otherwise, the platform may retain rights to reuse that information or to publish the output, and more generally use that data to train their models. Although each platform has different terms and conditions, AI platforms typically do not commit to maintaining the confidentiality or security of the prompts or outputs.

You should also have due diligence for AI service providers. Even if they have the best intentions to accomplish confidentiality and data privacy, architectural and business limitations of AI platforms may mean that the platforms are actually unable to do so. It’s important to evaluate the third-party risk as you would assess any other vendor getting access to your data and incorporate organizational, contractual, and physical safeguards for these types of incidents.

As a best practice, when providing data for prompts, you should assume that all information you provide will be public — think of these as disclosures to a third party. With that in mind, how much would you actually share?

It’s especially important to be extra careful with:

The big takeaway here is that the best way to avoid disclosures and data leaks is to avoid inputting personal information and anything you would need to keep confidential or legally privileged into prompts.

It’s important also to stay on top of emerging AI regulations as AI becomes more widespread and new use cases are tested.

EU AI Act
In April 2021, the European Commission proposed the first EU regulatory framework for AI(8). It says that AI systems that can be used in different applications are analyzed and classified according to the risk they pose to users. The different risk levels will determine the level of regulatory requirements on the different platforms/systems. If approved, these will be the world’s first rules for AI.

US State Law Tracker(10)
Many states have incorporated the usage of AI or the restriction of automated decision making for instances involving personal data into their data privacy regulations.

Colorado's (SB21-169) legislation
Colorado Division of Insurance (“CDI”) has finalized a regulation requiring insurers to implement AI governance and risk management measures that are reasonably designed to prevent unfair discrimination in the use of AI models that use external consumer data and information sources. The regulation restricts insurers from using ‘external consumer data’ and prohibits data, algorithms, or predictive models from unfairly discriminating. It requires insurers to test their systems and demonstrate that they are not biased. The regulation also requires insurers to implement AI governance and risk-management measures that are reasonably designed to prevent unfair discrimination in the use of AI models that use external consumer data and information sources.

Companies using or planning to use AI internally should have an AI policy that takes into account the potential use cases for AI, the risks of using AI, and the company’s risk tolerance given the potential benefits to its business. This policy should include clear guidelines for employees and contractors that outline:

Oftentimes, companies can leverage existing policies and assessments to incorporate AI reviews and guidelines.

These policies are often best developed by a cross-functional team, including legal, compliance, management, IT, engineering, and other internal stakeholders with internal checks and balances to enforce it. Companies should also regularly review new policies and frameworks given the fast-changing legal environment pertaining to AI.

Given the risks identified above, it’s important to apply human oversight and careful review of any AI outputs before they are used. Output should be subject to the standard quality controls of the business, including accuracy, consistency, and security with your company’s standards and policies. AI should be used in a way where there is always a “human audit” of the outputs to prevent automated decision making and other potential risks.

As with other third-party vendors, AI platforms should go through a risk assessment and diligence of their systems, terms, and their public-facing statements regarding data, security, and associated practices, before deciding to use them. Companies should ensure they meet standards and expectations and any other obligations that may be passed down to you by regulators or your contractual agreements. Investigate and consider whether the platform:

A lack of this information may also be informative to your decision-making process.

State privacy laws and regulatory bodies such as the FTC are routinely releasing guidance warning businesses about the harms of AI-powered tools including advertising accurately and transparently to their consumers. The FTC has already brought enforcement actions(11) against companies for using AI systems in a manner that was inconsistent with their privacy obligations and, as part of their settlements, required companies to destroy the underlying algorithms they used to develop their AI systems.

Companies should record how they are using AI, including what outputs are created and where they are used within the business and/or product. Transparent AI policies and procedures can help build consumer trust. If you have clear guidelines regarding the use of AI and maintain records of AI use, you can demonstrate that you are aware of the risks and are taking active steps to mitigate them, while still taking advantage of the technological opportunities that AI provides.

AI systems are designed to operate with a varying amount of autonomy. While the use of AI can provide monumental benefits in the way Brex and others continue to scale and deliver value to customers, the risks posed by AI systems are in many ways unique and may require a more nuanced risk review. Our goal is to enable Brex’s use of these systems in a manner that is both innovative and safe for our customers.

Guardrails we have implemented to enable the business to innovate safely include:

In addition to a technical security review via our Secure Product Lifecycle program and legal review for appropriate contractual protections, our GRC team performs risk assessments when evaluating new AI technologies and asks these questions:

1. Data source: Does the AI rely on querying datasets/sources of information and if so, what are those sources? How do you protect against data poisoning?

a. Why do we ask this?
It’s important to understand exactly what data sources the technology is indexing or leveraging to provide answers and outputs. AI is not magic. It is based on data aggregation and algorithms, and changes to the data source can drastically affect the reliability and accuracy of the outputs. Bias can also be introduced, so it is also important to understand how change management and the prevention of data poisoning are handled for in-scope data sources and systems — to ensure the reliability and integrity of the data being used.

b. What risks are we trying to mitigate?
Legal risks (data privacy and confidentiality)

2. Data lifecycle: What happens when we input data into the AI engine? Where is the data stored? Who has access?

a. Why do we ask this?
Essentially, we want to know the lifecycle of the data as it moves through the AI system — are queries logged/saved? Is output logged/saved? If so, who at the AI vendor has access to the stored information, and what are their data retention policies? Is data access in accordance with the principle of least privilege? Is need-to-know enforced? This helps us understand if the risk of unauthorized access or data exposure is managed.

b. What risks are we trying to mitigate?
Legal risks (data privacy and confidentiality)

3. Data segregation: Is our data segregated from other companies' data?

a. Why do we ask this?
We ask this question to find out if each customer/partner of this AI technology has their own instance of the data set being used against the algorithms. This is preferred to avoid inadvertent mixing of data among customers. It also ensures that our particular company’s policies can be applied, such as opting out of training the model and data retention requirements.

b. What risks are we trying to mitigate?
Legal risks (work product and output defects)

4. Model training: How often is the AI re-trained, and where does the AI reside? How does the retraining occur? Can we opt out of our data being used to train the model?

a. Why do we ask this?
As mentioned in the legal risk section above, AI technologies may contain errors, be misleading or deceptive, or be trained on data that is inaccurate to begin with. It is imperative to take steps to understand how and when the data is trained and how companies can opt out of having their data train the model. Understanding model training can help reduce the risk of bias and distortion in AI outputs, as well as reduce the risk of inadvertently incorporating intellectual property into the model for future querying.

b. What risks are we trying to mitigate?
Legal risks (intellectual property and output defects)

At Brex, we have a cross-functional team that is dedicated to empowering our employees to innovate with the use of AI, but in a way that’s focused on stewardship and safety. The use of AI can provide significant benefits, and our team has the following recommendations for other companies looking to leverage AI:

1. Establish a baseline understanding of what AI, LLM, etc., means in the context of your business, your products/services, and your targeted customer base. Understanding how your company plans to use AI, LLM, etc., and the benefits it provides will be key in understanding the business case and where investments in AI could be made, which will inform potential risk areas.

2. Develop governing policies and procedures. AI usage guideline docs and acceptable use policies can go a long way in keeping your data safe. These guidelines should focus on mitigating risk and should provide employees with general best practices that they need to follow when using AI.

3. Perform internal assessments, document the results and use them to improve your methodology over time.

As you evaluate the potential risks and rewards of leveraging AI to improve your internal and operational processes and customer-facing innovations, we hope that our proven best practices, guidelines, and insights shared above will help you make faster, more confident decisions about which AI applications are right for your company. There is no doubt that AI technology should be embraced. However, we recommend starting by ensuring that you’re making appropriate investments in regulatory and compliance resources and evaluation, so that you can protect your business and your customers above all else while harnessing the transformative power of AI.