Django’s serialization framework provides a mechanism for “translating” Django\nmodels into other formats. Usually these other formats will be text-based and\nused for sending Django data over a wire, but it’s possible for a\nserializer to handle any format (text-based or not).
\n\nAt the highest level, you can serialize data like this:
\nfrom django.core import serializers\n\ndata = serializers.serialize("json", SomeModel.objects.all())\nThe arguments to the serialize function are the format to serialize the data\nto (see Serialization formats) and a\nQuerySet to serialize. (Actually, the second\nargument can be any iterator that yields Django model instances, but it’ll\nalmost always be a QuerySet).
You can also use a serializer object directly:
\nJSONSerializer = serializers.get_serializer("json")\njson_serializer = JSONSerializer()\njson_serializer.serialize(queryset)\ndata = json_serializer.getvalue()\nThis is useful if you want to serialize data directly to a file-like object\n(which includes an HttpResponse):
with open("file.json", "w") as out:\n json_serializer.serialize(SomeModel.objects.all(), stream=out)\nIf you only want a subset of fields to be serialized, you can\nspecify a fields argument to the serializer:
from django.core import serializers\n\ndata = serializers.serialize("json", SomeModel.objects.all(), fields=["name", "size"])\nIn this example, only the name and size attributes of each model will\nbe serialized. The primary key is always serialized as the pk element in the\nresulting output; it never appears in the fields part.
If you have a model that is defined using an abstract base class, you don’t have to do anything special to serialize\nthat model. Call the serializer on the object (or objects) that you want to\nserialize, and the output will be a complete representation of the serialized\nobject.
\nHowever, if you have a model that uses multi-table inheritance, you also need to serialize all of the base classes\nfor the model. This is because only the fields that are locally defined on the\nmodel will be serialized. For example, consider the following models:
\nclass Place(models.Model):\n name = models.CharField(max_length=50)\n\n\nclass Restaurant(Place):\n serves_hot_dogs = models.BooleanField(default=False)\nIf you only serialize the Restaurant model:
\ndata = serializers.serialize("json", Restaurant.objects.all())\nthe fields on the serialized output will only contain the serves_hot_dogs\nattribute. The name attribute of the base class will be ignored.
In order to fully serialize your Restaurant instances, you will need to\nserialize the Place models as well:
all_objects = [*Restaurant.objects.all(), *Place.objects.all()]\ndata = serializers.serialize("json", all_objects)\nDeserializing data is very similar to serializing it:
\nfor obj in serializers.deserialize("json", data):\n do_something_with(obj)\nAs you can see, the deserialize function takes the same format argument as\nserialize, a string or stream of data, and returns an iterator.
However, here it gets slightly complicated. The objects returned by the\ndeserialize iterator aren’t regular Django objects. Instead, they are\nspecial DeserializedObject instances that wrap a created – but unsaved –\nobject and any associated relationship data.
Calling DeserializedObject.save() saves the object to the database.
This ensures that deserializing is a non-destructive operation even if the\ndata in your serialized representation doesn’t match what’s currently in the\ndatabase. Usually, working with these DeserializedObject instances looks\nsomething like:
for deserialized_object in serializers.deserialize("json", data):\n if object_should_be_saved(deserialized_object):\n deserialized_object.save()\nIn other words, the usual use is to examine the deserialized objects to make\nsure that they are “appropriate” for saving before doing so. If you trust your\ndata source you can instead save the object directly and move on.
\nThe Django object itself can be inspected as deserialized_object.object.\nIf fields in the serialized data do not exist on a model, a\nDeserializationError will be raised unless the ignorenonexistent\nargument is passed in as True:
serializers.deserialize("json", data, ignorenonexistent=True)\nDjango supports a number of serialization formats, some of which require you\nto install third-party Python modules:
\nIdentifier | \nInformation | \n
|---|---|
| \nSerializes to and from a simple XML dialect. | \n
| \nSerializes to and from JSON. | \n
| \nSerializes to and from JSONL. | \n
| \nSerializes to YAML (YAML Ain’t a Markup Language). This\nserializer is only available if PyYAML is installed. | \n
The basic XML serialization format looks like this:
\n<?xml version="1.0" encoding="utf-8"?>\n<django-objects version="1.0">\n <object pk="123" model="sessions.session">\n <field type="DateTimeField" name="expire_date">2013-01-16T08:16:59.844560+00:00</field>\n <!-- ... -->\n </object>\n</django-objects>\nThe whole collection of objects that is either serialized or deserialized is\nrepresented by a <django-objects>-tag which contains multiple\n<object>-elements. Each such object has two attributes: “pk” and “model”,\nthe latter being represented by the name of the app (“sessions”) and the\nlowercase name of the model (“session”) separated by a dot.
Each field of the object is serialized as a <field>-element sporting the\nfields “type” and “name”. The text content of the element represents the value\nthat should be stored.
Foreign keys and other relational fields are treated a little bit differently:
\n<object pk="27" model="auth.permission">\n <!-- ... -->\n <field to="contenttypes.contenttype" name="content_type" rel="ManyToOneRel">9</field>\n <!-- ... -->\n</object>\nIn this example we specify that the auth.Permission object with the PK 27\nhas a foreign key to the contenttypes.ContentType instance with the PK 9.
ManyToMany-relations are exported for the model that binds them. For instance,\nthe auth.User model has such a relation to the auth.Permission model:
<object pk="1" model="auth.user">\n <!-- ... -->\n <field to="auth.permission" name="user_permissions" rel="ManyToManyRel">\n <object pk="46"></object>\n <object pk="47"></object>\n </field>\n</object>\nThis example links the given user with the permission models with PKs 46 and 47.
\n\nWhen staying with the same example data as before it would be serialized as\nJSON in the following way:
\n[\n {\n "pk": "4b678b301dfd8a4e0dad910de3ae245b",\n "model": "sessions.session",\n "fields": {\n "expire_date": "2013-01-16T08:16:59.844Z",\n # ...\n },\n }\n]\nThe formatting here is a bit simpler than with XML. The whole collection\nis just represented as an array and the objects are represented by JSON objects\nwith three properties: “pk”, “model” and “fields”. “fields” is again an object\ncontaining each field’s name and value as property and property-value\nrespectively.
\nForeign keys have the PK of the linked object as property value.\nManyToMany-relations are serialized for the model that defines them and are\nrepresented as a list of PKs.
\nBe aware that not all Django output can be passed unmodified to json.\nFor example, if you have some custom type in an object to be serialized, you’ll\nhave to write a custom json encoder for it. Something like this will\nwork:
from django.core.serializers.json import DjangoJSONEncoder\n\n\nclass LazyEncoder(DjangoJSONEncoder):\n def default(self, obj):\n if isinstance(obj, YourCustomType):\n return str(obj)\n return super().default(obj)\nYou can then pass cls=LazyEncoder to the serializers.serialize()\nfunction:
from django.core.serializers import serialize\n\nserialize("json", SomeModel.objects.all(), cls=LazyEncoder)\nAlso note that GeoDjango provides a customized GeoJSON serializer.
\nDjangoJSONEncoderLink to this heading#The JSON serializer uses DjangoJSONEncoder for encoding. A subclass of\nJSONEncoder, it handles these additional types:
datetimeA string of the form YYYY-MM-DDTHH:mm:ss.sssZ or\nYYYY-MM-DDTHH:mm:ss.sss+HH:MM as defined in ECMA-262.
dateA string of the form YYYY-MM-DD as defined in ECMA-262.
timeA string of the form HH:MM:ss.sss as defined in ECMA-262.
timedeltaA string representing a duration as defined in ISO-8601. For example,\ntimedelta(days=1, hours=2, seconds=3.4) is represented as\n'P1DT02H00M03.400000S'.
Decimal, Promise (django.utils.functional.lazy() objects), UUIDA string representation of the object.
\nJSONL stands for JSON Lines. With this format, objects are separated by new\nlines, and each line contains a valid JSON object. JSONL serialized data looks\nlike this:
\n{"pk": "4b678b301dfd8a4e0dad910de3ae245b", "model": "sessions.session", "fields": {...}}\n{"pk": "88bea72c02274f3c9bf1cb2bb8cee4fc", "model": "sessions.session", "fields": {...}}\n{"pk": "9cf0e26691b64147a67e2a9f06ad7a53", "model": "sessions.session", "fields": {...}}\nJSONL can be useful for populating large databases, since the data can be\nprocessed line by line, rather than being loaded into memory all at once.
\nYAML serialization looks quite similar to JSON. The object list is serialized\nas a sequence mappings with the keys “pk”, “model” and “fields”. Each field is\nagain a mapping with the key being name of the field and the value the value:
\n- model: sessions.session\n pk: 4b678b301dfd8a4e0dad910de3ae245b\n fields:\n expire_date: 2013-01-16 08:16:59.844560+00:00\nReferential fields are again represented by the PK or sequence of PKs.
\nIn addition to the default formats, you can create a custom serialization\nformat.
\nFor example, let’s consider a csv serializer and deserializer. First, define a\nSerializer and a Deserializer class. These can override existing\nserialization format classes:
path/to/custom_csv_serializer.py import csv\n\n from django.apps import apps\n from django.core import serializers\n from django.core.serializers.base import DeserializationError\n\n\n class Serializer(serializers.python.Serializer):\n def get_dump_object(self, obj):\n dumped_object = super().get_dump_object(obj)\n row = [dumped_object["model"], str(dumped_object["pk"])]\n row += [str(value) for value in dumped_object["fields"].values()]\n return ",".join(row), dumped_object["model"]\n\n def end_object(self, obj):\n dumped_object_str, model = self.get_dump_object(obj)\n if self.first:\n fields = [field.name for field in apps.get_model(model)._meta.fields]\n header = ",".join(fields)\n self.stream.write(f"model,{header}\\n")\n self.stream.write(f"{dumped_object_str}\\n")\n\n def getvalue(self):\n return super(serializers.python.Serializer, self).getvalue()\n\n\n class Deserializer(serializers.python.Deserializer):\n def __init__(self, stream_or_string, **options):\n if isinstance(stream_or_string, bytes):\n stream_or_string = stream_or_string.decode()\n if isinstance(stream_or_string, str):\n stream_or_string = stream_or_string.splitlines()\n try:\n objects = csv.DictReader(stream_or_string)\n except Exception as exc:\n raise DeserializationError() from exc\n super().__init__(objects, **options)\n\n def _handle_object(self, obj):\n try:\n model_fields = apps.get_model(obj["model"])._meta.fields\n obj["fields"] = {\n field.name: obj[field.name]\n for field in model_fields\n if field.name in obj\n }\n yield from super()._handle_object(obj)\n except (GeneratorExit, DeserializationError):\n raise\n except Exception as exc:\n raise DeserializationError(f"Error deserializing object: {exc}") from exc\nThen add the module containing the serializer definitions to your\nSERIALIZATION_MODULES setting:
SERIALIZATION_MODULES = {\n "csv": "path.to.custom_csv_serializer",\n "json": "django.core.serializers.json",\n}\nThe default serialization strategy for foreign keys and many-to-many relations\nis to serialize the value of the primary key(s) of the objects in the relation.\nThis strategy works well for most objects, but it can cause difficulty in some\ncircumstances.
\nConsider the case of a list of objects that have a foreign key referencing\nContentType. If you’re going to\nserialize an object that refers to a content type, then you need to have a way\nto refer to that content type to begin with. Since ContentType objects are\nautomatically created by Django during the database synchronization process,\nthe primary key of a given content type isn’t easy to predict; it will\ndepend on how and when migrate was executed. This is true for all\nmodels which automatically generate objects, notably including\nPermission,\nGroup, and\nUser.
There is also the matter of convenience. An integer id isn’t always\nthe most convenient way to refer to an object; sometimes, a\nmore natural reference would be helpful.
\nIt is for these reasons that Django provides natural keys. A natural\nkey is a tuple of values that can be used to uniquely identify an\nobject instance without using the primary key value.
\nConsider the following two models:
\nfrom django.db import models\n\n\nclass Person(models.Model):\n first_name = models.CharField(max_length=100)\n last_name = models.CharField(max_length=100)\n\n birthdate = models.DateField()\n\n class Meta:\n constraints = [\n models.UniqueConstraint(\n fields=["first_name", "last_name"],\n name="unique_first_last_name",\n ),\n ]\n\n\nclass Book(models.Model):\n name = models.CharField(max_length=100)\n author = models.ForeignKey(Person, on_delete=models.CASCADE)\nOrdinarily, serialized data for Book would use an integer to refer to\nthe author. For example, in JSON, a Book might be serialized as:
...\n{"pk": 1, "model": "store.book", "fields": {"name": "Mostly Harmless", "author": 42}}\n...\nThis isn’t a particularly natural way to refer to an author. It\nrequires that you know the primary key value for the author; it also\nrequires that this primary key value is stable and predictable.
\nHowever, if we add natural key handling to Person, the fixture becomes\nmuch more humane. To add natural key handling, you define a default\nManager for Person with a get_by_natural_key() method. In the case\nof a Person, a good natural key might be the pair of first and last\nname:
from django.db import models\n\n\nclass PersonManager(models.Manager):\n def get_by_natural_key(self, first_name, last_name):\n return self.get(first_name=first_name, last_name=last_name)\n\n\nclass Person(models.Model):\n first_name = models.CharField(max_length=100)\n last_name = models.CharField(max_length=100)\n birthdate = models.DateField()\n\n objects = PersonManager()\n\n class Meta:\n constraints = [\n models.UniqueConstraint(\n fields=["first_name", "last_name"],\n name="unique_first_last_name",\n ),\n ]\nNow books can use that natural key to refer to Person objects:
...\n{\n "pk": 1,\n "model": "store.book",\n "fields": {"name": "Mostly Harmless", "author": ["Douglas", "Adams"]},\n}\n...\nWhen you try to load this serialized data, Django will use the\nget_by_natural_key() method to resolve ["Douglas", "Adams"]\ninto the primary key of an actual Person object.
Deserialization of objects with no primary key will always check whether the\nmodel’s manager has a get_by_natural_key() method and if so, use it to\npopulate the deserialized object’s primary key.
So how do you get Django to emit a natural key when serializing an object?\nFirstly, you need to add another method – this time to the model itself:
\nclass Person(models.Model):\n first_name = models.CharField(max_length=100)\n last_name = models.CharField(max_length=100)\n birthdate = models.DateField()\n\n objects = PersonManager()\n\n class Meta:\n constraints = [\n models.UniqueConstraint(\n fields=["first_name", "last_name"],\n name="unique_first_last_name",\n ),\n ]\n\n def natural_key(self):\n return (self.first_name, self.last_name)\nThat method should always return a natural key tuple – in this\nexample, (first name, last name). Then, when you call\nserializers.serialize(), you provide use_natural_foreign_keys=True\nor use_natural_primary_keys=True arguments:
>>> serializers.serialize(\n... "json",\n... [book1, book2],\n... indent=2,\n... use_natural_foreign_keys=True,\n... use_natural_primary_keys=True,\n... )\nWhen use_natural_foreign_keys=True is specified, Django will use the\nnatural_key() method to serialize any foreign key reference to objects\nof the type that defines the method.
When use_natural_primary_keys=True is specified, Django will not provide the\nprimary key in the serialized data of this object since it can be calculated\nduring deserialization:
...\n{\n "model": "store.person",\n "fields": {\n "first_name": "Douglas",\n "last_name": "Adams",\n "birth_date": "1952-03-11",\n },\n}\n...\nThis can be useful when you need to load serialized data into an existing\ndatabase and you cannot guarantee that the serialized primary key value is not\nalready in use, and do not need to ensure that deserialized objects retain the\nsame primary keys.
\nIf you are using dumpdata to generate serialized data, use the\ndumpdata --natural-foreign and dumpdata --natural-primary\ncommand line flags to generate natural keys.
Sometimes when you use natural foreign keys you’ll need to deserialize data where\nan object has a foreign key referencing another object that hasn’t yet been\ndeserialized. This is called a “forward reference”.
\nFor instance, suppose you have the following objects in your fixture:
\n...\n{\n "model": "store.book",\n "fields": {"name": "Mostly Harmless", "author": ["Douglas", "Adams"]},\n},\n...\n{"model": "store.person", "fields": {"first_name": "Douglas", "last_name": "Adams"}},\n...\nIn order to handle this situation, you need to pass\nhandle_forward_references=True to serializers.deserialize(). This will\nset the deferred_fields attribute on the DeserializedObject instances.\nYou’ll need to keep track of DeserializedObject instances where this\nattribute isn’t None and later call save_deferred_fields() on them.
Typical usage looks like this:
\nobjs_with_deferred_fields = []\n\nfor obj in serializers.deserialize("json", data, handle_forward_references=True):\n obj.save()\n if obj.deferred_fields is not None:\n objs_with_deferred_fields.append(obj)\n\nfor obj in objs_with_deferred_fields:\n obj.save_deferred_fields()\nFor this to work, the ForeignKey on the referencing model must have\nnull=True.
It’s often possible to avoid explicitly having to handle forward references by\ntaking care with the ordering of objects within a fixture.
\nTo help with this, calls to dumpdata that use the dumpdata\n--natural-foreign option will serialize any model with a natural_key()\nmethod before serializing standard primary key objects.
However, this may not always be enough. If your natural key refers to\nanother object (by using a foreign key or natural key to another object\nas part of a natural key), then you need to be able to ensure that\nthe objects on which a natural key depends occur in the serialized data\nbefore the natural key requires them.
\nTo control this ordering, you can define dependencies on your\nnatural_key() methods. You do this by setting a dependencies\nattribute on the natural_key() method itself.
For example, let’s add a natural key to the Book model from the\nexample above:
class Book(models.Model):\n name = models.CharField(max_length=100)\n author = models.ForeignKey(Person, on_delete=models.CASCADE)\n\n def natural_key(self):\n return (self.name,) + self.author.natural_key()\nThe natural key for a Book is a combination of its name and its\nauthor. This means that Person must be serialized before Book.\nTo define this dependency, we add one extra line:
def natural_key(self):\n return (self.name,) + self.author.natural_key()\n\n\nnatural_key.dependencies = ["example_app.person"]\nThis definition ensures that all Person objects are serialized before\nany Book objects. In turn, any object referencing Book will be\nserialized after both Person and Book have been serialized.