You could exclude only optional model fields that unset by making of union of model fields that are set and those that are not None.
Pydantic provides the following arguments for exporting method model.dict(...):
exclude_unset: whether fields which were not explicitly set when creating the model should be excluded from the returned dictionary; defaultFalse.
exclude_none: whether fields which are equal toNoneshould be excluded from the returned dictionary; defaultFalse
To make union of two dicts we can use the expression a = {**b, **c} (values from c overwrites values from b). Note that since Python 3.9 it could be done just as a = b | c.
from pydantic import BaseModel
from typing import Optional
from pydantic.json import pydantic_encoder
import json
class Foo(BaseModel):
x: int
y: int = 42
z: Optional[int]
def exclude_optional_dict(model: BaseModel):
return {**model.dict(exclude_unset=True), **model.dict(exclude_none=True)}
def exclude_optional_json(model: BaseModel):
return json.dumps(exclude_optional_dict(model), default=pydantic_encoder)
print(exclude_optional_json(Foo(x=3))) # {"x": 3, "y": 42}
print(exclude_optional_json(Foo(x=3, z=None))) # {"x": 3, "z": null, "y": 42}
print(exclude_optional_json(Foo(x=3, z=77))) # {"x": 3, "z": 77, "y": 42}
Update
In order for the approach to work with nested models, we need to do a deep union( or merge) of two dictionaries, like so:
def union(source, destination):
for key, value in source.items():
if isinstance(value, dict):
node = destination.setdefault(key, {})
union(value, node)
else:
destination[key] = value
return destination
def exclude_optional_dict(model: BaseModel):
return union(model.dict(exclude_unset=True), model.dict(exclude_none=True))
class Foo(BaseModel):
x: int
y: int = 42
z: Optional[int]
class Bar(BaseModel):
a: int
b: int = 52
c: Optional[int]
d: Foo
print(exclude_optional_json(Bar(a=4, d=Foo(x=3))))
print(exclude_optional_json(Bar(a=4, c=None, d=Foo(x=3, z=None))))
print(exclude_optional_json(Bar(a=4, c=78, d=Foo(x=3, z=77))))
{"a": 4, "b": 52, "d": {"x": 3, "y": 42}}
{"a": 4, "b": 52, "d": {"x": 3, "y": 42, "z": null}, "c": null}
{"a": 4, "b": 52, "c": 78, "d": {"x": 3, "y": 42, "z": 77}}
Answer from alex_noname on Stack OverflowPydantic v2 exclude parameter in Field and Annotated
python - In JSON created from a pydantic.BaseModel exclude Optional if not set - Stack Overflow
pydantic v2 python validation, how to dump some, not all fields with None value - Stack Overflow
Struggling with Pydantic 'excludes'
I am playing around with Pydantic v2.5 and trying to see how the exclude works when set as a Field option. Let's imagine that I have a User BaseModel class and a Permissions BaseModel class. Both are used in the Config class.
We do not want to print the all User info, hence why I added the exclude in the Permissions class when the user is defined.
from typing import Annotated, Optional
from pydantic import BaseModel, Field
class User(BaseModel):
name: str
surname: str
card_id: str
class Permissions(BaseModel):
user: Annotated[User, Field(exclude=True)]
config_rule: str
class Config:
def __init__(self):
self.user = User(**dict(name="Name", surname="Surname", card_id="12343545"))
self.config_data = Permissions(user=self.user, config_rule="admin")
c = Config()
print(c.config_data.model_dump_json(indent=2))
In this case, the exclude set in the Permissions class seems to be working fine, in fact the print output is:
{
"config_rule": "admin"
}
However, if I change the Permissions class with:
class Permissions(BaseModel):
user: Optional[Annotated[User, Field(exclude=True)]]
config_rule: strThe output is:
{
"user": {
"name": "Name",
"surname": "Surname",
"card_id": "12343545"
},
"config_rule": "admin"
}And I am not sure I understand why. Any thoughts? Thanks.
You could exclude only optional model fields that unset by making of union of model fields that are set and those that are not None.
Pydantic provides the following arguments for exporting method model.dict(...):
exclude_unset: whether fields which were not explicitly set when creating the model should be excluded from the returned dictionary; defaultFalse.
exclude_none: whether fields which are equal toNoneshould be excluded from the returned dictionary; defaultFalse
To make union of two dicts we can use the expression a = {**b, **c} (values from c overwrites values from b). Note that since Python 3.9 it could be done just as a = b | c.
from pydantic import BaseModel
from typing import Optional
from pydantic.json import pydantic_encoder
import json
class Foo(BaseModel):
x: int
y: int = 42
z: Optional[int]
def exclude_optional_dict(model: BaseModel):
return {**model.dict(exclude_unset=True), **model.dict(exclude_none=True)}
def exclude_optional_json(model: BaseModel):
return json.dumps(exclude_optional_dict(model), default=pydantic_encoder)
print(exclude_optional_json(Foo(x=3))) # {"x": 3, "y": 42}
print(exclude_optional_json(Foo(x=3, z=None))) # {"x": 3, "z": null, "y": 42}
print(exclude_optional_json(Foo(x=3, z=77))) # {"x": 3, "z": 77, "y": 42}
Update
In order for the approach to work with nested models, we need to do a deep union( or merge) of two dictionaries, like so:
def union(source, destination):
for key, value in source.items():
if isinstance(value, dict):
node = destination.setdefault(key, {})
union(value, node)
else:
destination[key] = value
return destination
def exclude_optional_dict(model: BaseModel):
return union(model.dict(exclude_unset=True), model.dict(exclude_none=True))
class Foo(BaseModel):
x: int
y: int = 42
z: Optional[int]
class Bar(BaseModel):
a: int
b: int = 52
c: Optional[int]
d: Foo
print(exclude_optional_json(Bar(a=4, d=Foo(x=3))))
print(exclude_optional_json(Bar(a=4, c=None, d=Foo(x=3, z=None))))
print(exclude_optional_json(Bar(a=4, c=78, d=Foo(x=3, z=77))))
{"a": 4, "b": 52, "d": {"x": 3, "y": 42}}
{"a": 4, "b": 52, "d": {"x": 3, "y": 42, "z": null}, "c": null}
{"a": 4, "b": 52, "c": 78, "d": {"x": 3, "y": 42, "z": 77}}
If you're using FastAPI then using exclude_none doesn't seem to work when a response_model is mentioned in the route decorator.
@app.post("/items/", response_model=Item)
async def create_item(item: Item):
return item.dict(exclude_none=True)
Fast api seems to reprocess the dict with the pydantic model
So overriding the dict method in the model itself should work
def Item(BaseModel):
name: str
description: Optional[str]
...
def dict(self, *args, **kwargs) -> Dict[str, Any]:
kwargs.pop('exclude_none', None)
return super().dict(*args, exclude_none=True, **kwargs)
(an actual solution would put this definition in separate subclass of BaseModel for reuse)
Note: just changing the default value of the exclude_none keyword argument is not enough: it seems FastAPI always sends exclude_none=False as an argument.
Source:
https://github.com/tiangolo/fastapi/issues/3314#issuecomment-962932368
I'm building an API that deals with a bunch of related data structures. And I'm currently struggling with some of the intricacies of Pydantic. Here's my problem. I hope someone out there is able to help me out here! :)
Consider a Pydantic schema like this:
class Node(BaseModel):
name: str
uuid: UUID
parent_node_uuid: UUID | None
With that it is possible to represent a hierarchical, tree-like data set. Individual node objects are related to each other through the parent_node_uuid property. Each parent node can have multiple children. But each child can only ever have a single parent. (in other words there is a self-referential one-to-many relationship)
Now, when outputting this data set through the api endpoint, I could simply use the above schema as my response_model. But instead I want to make the data more verbose and instead nest the model, so that the output of one node includes all the information about its parent and child nodes. The naive approach looks like this:
class Node(BaseModel):
name: str
uuid: UUID
parent_node: "Node" | None = None
child_nodes: List["Node"] = []Unfortunately, this does not work as intended. When I try to pass sqlalchemy objects (which do have the required relationships set up) to this Pydantic schema, I'm running into an infinite recursion and python crashes. The reason is that the parent_node includes the main node object inits child_nodes property. Similarly, each child_node of our main node will have the main node set as their parent_node. - it's easy to see how Pydantic gets stuck in an infinite loop here.
There is a solution to one part of this problem:
class Node(BaseModel):
name: str
uuid: UUID
parent_node: "Node" | None = Field(None, exclude={"child_nodes"})
child_nodes: List["Node"] = []Using the exclude option, we're able to remove the child_nodes from the parent. - That's one half of the issue resolved. The above model now works in cases where our main node doesn't have any children, but it has a parent. (more on this in the docs here)
Unfortunately though, this solution does not work with lists. I've tried the following without success:
class Node(BaseModel):
name: str
uuid: UUID
parent_node: "Node" | None = Field(None, exclude={"child_nodes"})
child_nodes: List["Node"] = Field([], exclude={"parent_node"})*Does anyone know how I can get the exclude parameter to work when dealing with a List of models? *
Pydantic will exclude the class variables which begin with an underscore. so if it fits your use case, you can rename your attribues.
class User(UserBase):
_user_id=str
some_other_field=str
....
To exclude a field you can also use exclude in Field:
from pydantic import BaseModel, Field
class Mdl(BaseModel):
val: str = Field(
exclude=True,
title="val"
)
however, the advantage of adding excluded parameters in the Config class seems to be that you can get the list of excluded parameters with
print(Mdl.Config.exclude)
So, I am ware of exclude_none, but it works on per-model-basis, meaning that in the case deeply nested models, you would have to add this to every submodel. In certain situations, especially working with 3rd party provided models, this is not really an option. Is there a workaround to this limitation?