What are generative (and discriminative) models?

If the model learns a distribution of the form or , where are the inputs and the outputs/labels, from which you can sample data, then it's a generative model. An example of a generative model: variational autoencoder (VAE).

Bishop also defines generative models in this way (p. 43)

Approaches that explicitly or implicitly model the distribution of inputs as well as outputs are known as generative models, because by sampling from them it is possible to generate synthetic data points in the input space

If it learns a distribution of the form , then it's a discriminative model - many/most classifiers learn this distribution, but you can also derive the conditional given the the joint and prior (that's why above Bishop uses implicitly or explicitly).

Bishop also defines discriminative models in this way (p. 43)

Approaches that model the posterior probabilities directly are called discriminative models

The related Wikipedia article claims that people have not always been using these terms consistently (which is common in machine learning), so one should always keep that in mind.

GPTs are autoregressive

As far as I know, GPTs are autoregressive models. Here is another potentially useful post that explains what autoregressive models are.

My understanding of autoregressive models, at least based on neural networks, is that they are also generative models - the linked articles and even the GPT-2 paper seem to start the descriptions from the assumption that you can factorize some joint distribution like into conditional distributions.

ChatGPT is based on a GPT model, so it's probably considered a generative model too, but there are several steps involved to create this model, so it may not be super clear how to categorise this model.

Moreover, the authors of the transformer, which GPT models are based on, claim that the transformer is an autoregressive model.

Conclusion

It seems to me that many people in ML refer to any model that generates data as a generative model, even if there's no written theoretical formulation of it as a generative model, which doesn't mean that you cannot formulate these models as generative models, i.e. a model that learns some distribution that you can use to sample data from data distribution.

I am currently not familiar enough with the details of the GPT models to say if they have been mathematically formulated as generative models of the form , but they model some distribution of the form , from which you can sample, otherwise, how could you even sample data (words)?

Answer from nbro on Stack Exchange
🌐
Bernard Marr
bernardmarr.com › home › the difference between chatgpt and generative ai
The Difference Between ChatGPT And Generative AI | Bernard Marr
July 5, 2024 - ChatGPT: A specific type of generative AI, ChatGPT is specialized in text generation. Think of it as a specialized tool within the larger AI toolkit, specifically sharpened for generating readable, coherent text.
🌐
Reddit
reddit.com › r/chatgpt › stop confusing “ai” with “generative ai”
r/ChatGPT on Reddit: Stop Confusing “AI” with “Generative AI”
October 13, 2024 - Like, okay, let’s say that artists ... cancers, optimizing engineering solutions, or other technical applications. ... ChatGPT is generative AI trained on unauthorized use of copyrighted material so I’m not sure how you disentangle ...
🌐
McKinsey
mckinsey.com › featured-insights › mckinsey-explainers › what-is-generative-ai
What is ChatGPT, DALL-E, and generative AI? | McKinsey
April 2, 2024 - Generative artificial intelligence (AI) describes algorithms (such as ChatGPT) that can be used to create new content, including audio, code, images, text, simulations, and videos.
🌐
Upwork
upwork.com › resources › articles › chatgpt vs. generative ai: definitions and distinctions
ChatGPT vs. Generative AI: Definitions and Distinctions - Upwork
May 8, 2025 - Yes, ChatGPT is a type of generative AI. Learn how it works, how it compares to other tools, and what makes it unique in the AI space.
Top answer
1 of 3
4

What are generative (and discriminative) models?

If the model learns a distribution of the form or , where are the inputs and the outputs/labels, from which you can sample data, then it's a generative model. An example of a generative model: variational autoencoder (VAE).

Bishop also defines generative models in this way (p. 43)

Approaches that explicitly or implicitly model the distribution of inputs as well as outputs are known as generative models, because by sampling from them it is possible to generate synthetic data points in the input space

If it learns a distribution of the form , then it's a discriminative model - many/most classifiers learn this distribution, but you can also derive the conditional given the the joint and prior (that's why above Bishop uses implicitly or explicitly).

Bishop also defines discriminative models in this way (p. 43)

Approaches that model the posterior probabilities directly are called discriminative models

The related Wikipedia article claims that people have not always been using these terms consistently (which is common in machine learning), so one should always keep that in mind.

GPTs are autoregressive

As far as I know, GPTs are autoregressive models. Here is another potentially useful post that explains what autoregressive models are.

My understanding of autoregressive models, at least based on neural networks, is that they are also generative models - the linked articles and even the GPT-2 paper seem to start the descriptions from the assumption that you can factorize some joint distribution like into conditional distributions.

ChatGPT is based on a GPT model, so it's probably considered a generative model too, but there are several steps involved to create this model, so it may not be super clear how to categorise this model.

Moreover, the authors of the transformer, which GPT models are based on, claim that the transformer is an autoregressive model.

Conclusion

It seems to me that many people in ML refer to any model that generates data as a generative model, even if there's no written theoretical formulation of it as a generative model, which doesn't mean that you cannot formulate these models as generative models, i.e. a model that learns some distribution that you can use to sample data from data distribution.

I am currently not familiar enough with the details of the GPT models to say if they have been mathematically formulated as generative models of the form , but they model some distribution of the form , from which you can sample, otherwise, how could you even sample data (words)?

2 of 3
2

They both refer to the same type of models. However, the second definition is a more 'intuitive' explanation of what generative AI does, while the first is a definition that refers more to what a generative model is.

To generate new data similar to some training data (definition 2), a model needs to learn the training data distributions (definition 1). Only if the model has learned that distribution it can use that distribution to sample (generate) new data from that distribution.

During training, ChatGPT also learned the distribution of the training data that OpenAI provided the model with. After training, the model simply takes in the input and uses the input to sample from the learned distribution to generate an output. So ChatGPT also follows both of your definitions.

🌐
Coursera
coursera.org › coursera articles › data › ai and machine learning › is chatgpt generative ai: understanding its functioning, capabilities, and limitations
Is ChatGPT Generative AI: Understanding Its Functioning, Capabilities, and Limitations | Coursera
April 11, 2025 - Because ChatGPT is built on OpenAI’s neural network specifically designed for natural language processing (NLP) known as generative pre-trained transformer (GPT) architecture—it is generative AI.
🌐
University of British Columbia
guides.library.ubc.ca › c.php
Evaluating Information Sources: Generative AI and ChatGPT - Generative AI and ChatGPT - Research Guides at University of British Columbia
Generative AI is a type of artificial intelligence that can produce new text, images, audio, and video in response to user prompts. It includes tools like ChatGPT, Google’s Bard and Bing’s Chatbot.
🌐
Call Centre Helper
callcentrehelper.com › the-difference-between-chatgpt-llms-and-generative-ai-222433.htm
The Difference Between ChatGPT, LLMs, and Generative AI
Generative AI covers a wide range of creative and functional applications, with ChatGPT being one example among many, and is a broad term for AI models that produce unique and novel outputs.
Find elsewhere
🌐
MIT LIDS
lids.mit.edu › news-and-events › news › explained-generative-ai-how-do-powerful-generative-ai-systems-chatgpt-work-and
Explained: Generative AI - How do powerful generative AI systems like ChatGPT work, and what makes them different from other types of artificial intelligence? | MIT LIDS
November 9, 2023 - In fact, some of those headlines may actually have been written by generative AI, like OpenAI’s ChatGPT, a chatbot that has demonstrated an uncanny ability to produce text that seems to have been written by a human.
🌐
LinkedIn
linkedin.com › pulse › understanding-distinction-generative-ai-vs-chatgpt-ragu
Understanding the Distinction: Generative AI vs. ChatGPT
June 10, 2023 - Generative AI has found applications in diverse areas such as art, design, and content creation. ... ChatGPT, on the other hand, is a specific implementation of generative AI designed explicitly for conversational purposes.
🌐
Wesleyan
libguides.wesleyan.edu › chatgpt
What Is Generative AI? - ChatGPT and other Generative AI - LibGuides at Wesleyan University
Generative AI can be used in innumerable ways from content creation to problem solving. ChatGPT (Chat Generative Pre-trained Transformer) is, according to ChatGPT itself, “a computer program created by OpenAI that can understand and generate text like a human.
🌐
ChatGPT
chatgpt.com › g › g-XNOaitB9h-generative-ai
ChatGPT - Generative AI
ChatGPT is your AI chatbot for everyday use. Chat with the most advanced AI to explore ideas, solve problems, and learn faster.
🌐
Pluralsight
pluralsight.com › blog › ai & data
What are ChatGPT and Generative AI (and how can I use them)? | Online Courses, Learning Paths, and Certifications - Pluralsight
Discriminative AI is normally used for supervised machine learning. ... ChatGPT stands for “Chat Generative Pre-Trained Transformer”, and it’s a generative AI language model that acts in a conversational way.
🌐
Medium
medium.com › data-science-at-microsoft › generative-ai-openai-and-chatgpt-what-are-they-3c80397062c4
Generative AI, OpenAI, and ChatGPT: What are they? | by Francesca Lazzeri | Data Science at Microsoft | Medium
May 30, 2023 - It is a product of OpenAI that allows users to generate text in natural language. ChatGPT is a sibling model to InstructGPT, which is trained to follow an instruction in a prompt and provide a detailed response.
🌐
University of Waterloo
subjectguides.uwaterloo.ca › chatgpt_generative_ai
Introduction - ChatGPT and Generative Artificial Intelligence (AI) - Research guides at University of Waterloo
If you have questions regarding ChatGPT or other generative artificial intelligence software that are not covered in this guide, please consult the FAQ from the Associate Vice-President, Academic's Office. Note: Some assignments do not permit the use of ChatGPT or AI tools.
🌐
Coursera
coursera.org › coursera articles › data › data science › what is chatgpt? how it works, how to use it, and more
What Is ChatGPT? How It Works, How to Use It, and More | Coursera
The intuitive, free tool has already gained popularity as an alternative to traditional search engines and as a tool for AI writing, among other things. The "GPT" in ChatGPT is short for generative pre-trained transformer.
Published   October 13, 2025
Views   214
🌐
Nature
nature.com › news feature › article
What ChatGPT and generative AI mean for science
February 8, 2023 - Abstracts written by ChatGPT fool scientists · How language-generation AIs could transform science · Tools such as ChatGPT threaten transparent science; here are our ground rules for their use · Could AI help you to write your next paper? Publishing · Machine learning ·
🌐
CIO
cio.com › home › brandposts › driving innovation: how to manage the tech hype cycle
ChatGPT, the rise of generative AI | CIO
May 19, 2023 - A transformer is a type of AI deep learning model that was first introduced by Google in a research paper in 2017. Five years later, transformer architecture has evolved to create powerful models such as ChatGPT.
🌐
Reveation Labs
reveation.io › blog › generative-ai-vs-chatgpt
Generative AI vs ChatGPT for Conversational AI Applications
It uses complex algorithms to create ... originality are crucial. On the other hand, ChatGPT is designed to mimic human conversations and relies on extensive pre-training and fine-tuning....