You may use schema (PyPi Link)
schema is a library for validating Python data structures, such as those obtained from config-files, forms, external services or command-line parsing, converted from JSON/YAML (or something else) to Python data-types.
from schema import Schema, And, Use, Optional, SchemaError
def check(conf_schema, conf):
try:
conf_schema.validate(conf)
return True
except SchemaError:
return False
conf_schema = Schema({
'version': And(Use(int)),
'info': {
'conf_one': And(Use(float)),
'conf_two': And(Use(str)),
'conf_three': And(Use(bool)),
Optional('optional_conf'): And(Use(str))
}
})
conf = {
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar'
}
}
print(check(conf_schema, conf))
Answer from Danil Speransky on Stack Overflow
» pip install schema
validation - How to validate structure (or schema) of dictionary in Python? - Stack Overflow
Python Schema Or() - Stack Overflow
Python library for automating data normalisation, schema creation and loading to db
best way to validating csv file in python
Videos
Hey Data Engineers!,
For the past 2 years I've been working on a library to automate the most tedious part of my own work - data loading, normalisation, typing, schema creation, retries, ddl generation, self deployment, schema evolution... basically, as you build better and better pipelines you will want more and more.
The value proposition is to automate the tedious work you do, so you can focus on better things.
So dlt is a library where in the easiest form, you shoot response.json() json at a function and it auto manages the typing normalisation and loading.
In its most complex form, you can do almost anything you can want, from memory management, multithreading, extraction DAGs, etc.
The library is in use with early adopters, and we are now working on expanding our feature set to accommodate the larger community.
Feedback is very welcome and so are requests for features or destinations.
The library is open source and will forever be open source. We will not gate any features for the sake of monetisation - instead we will take a more kafka/confluent approach where the eventual paid offering would be supportive not competing.
Here are our product principles and docs page and our pypi page.
I know lots of you are jaded and fed up with toy technologies - this is not a toy tech, it's purpose made for productivity and sanity.
Edit: Well this blew up! Join our growing slack community on dlthub.com
You may use schema (PyPi Link)
schema is a library for validating Python data structures, such as those obtained from config-files, forms, external services or command-line parsing, converted from JSON/YAML (or something else) to Python data-types.
from schema import Schema, And, Use, Optional, SchemaError
def check(conf_schema, conf):
try:
conf_schema.validate(conf)
return True
except SchemaError:
return False
conf_schema = Schema({
'version': And(Use(int)),
'info': {
'conf_one': And(Use(float)),
'conf_two': And(Use(str)),
'conf_three': And(Use(bool)),
Optional('optional_conf'): And(Use(str))
}
})
conf = {
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar'
}
}
print(check(conf_schema, conf))
Use Pydantic!
Pydantic enforces type hints at runtime, and provides user friendly errors when data is invalid. Define how data should be in pure, canonical python; validate it with pydantic, as simple as that:
from pydantic import BaseModel
class Info(BaseModel):
conf_one: float
conf_two: str
conf_three: bool
class Config:
extra = 'forbid'
class ConfStructure(BaseModel):
version: int
info: Info
If validation fails pydantic will raise an error with a breakdown of what was wrong:
my_conf_wrong = {
'version': 1,
'info': {
'conf_one': 2.5,
'conf_two': 'foo',
'conf_three': False,
'optional_conf': 'bar'
}
}
my_conf_right = {
'version': 10,
'info': {
'conf_one': 14.5,
'conf_two': 'something',
'conf_three': False
}
}
model = ConfStructure(**my_conf_right)
print(model.dict())
# {'version': 10, 'info': {'conf_one': 14.5, 'conf_two': 'something', 'conf_three': False}}
res = ConfStructure(**my_conf_wrong)
# pydantic.error_wrappers.ValidationError: 1 validation error for ConfStructure
# info -> optional_conf
# extra fields not permitted (type=value_error.extra)