Data Schemas

Data Schemas

ShiftForward DMP accepts custom data schemas for both event and CRM data. In either case, for setting a new one, a data sample of about 100 entries is required. Schemas should be sent and agreed beforehand and data entries should never include private personal user information such as real email addresses, phone numbers or postal addresses. If required, these should be anonymized first using a secure hashing algorithm such as SHA256 with a private salt.

User and device data must be identified by a unique key such as a browser’s user id cookie and domain, user’s email hash, phone hash, device IDFA or Google Ad ID, custom CRM ID or any other usable unique identifier. Attributes can be sent as separate fields or as lists of key-value pairs for dynamic attributes (where both keys and values are not known beforehand).

Examples

Device Ad id with age and gender

Example for data records from mobile devices identified through an IDFA (Apple) or Google Advertising ID (GAID) that include Date of Birth in YYYY/MM/DD format and Gender using M, F or Unknown values:

CSV with header:

“IDFA”,GAID”,”DOB”,”Gender”
“13CA035B-7G0F-4989-A9CE-999EAB23605A”,””,”1983/04/12″,”M”
“6D92078A-8246-4BA4-AE5B-76104861E7DC”,””,”1965/12/21″,”M”
“AEBE52E7-03EE-455A-B3C4-E57283966239″,””,”1995/01/27″,”F”
“”,”26ba0cb3-deac-4203-83d3-dc1c74080af7″,”1975/05/17″,”M”
“”,”4c23fac7-3a36-7408-bc31-fe95abc317af “,”1986/03/01″,”Unknown”

AVRO:

AVRO schema (e.g. adstaxcustomerdata.avsc):

{
"type": "record",
"name": "adstaxcrmdata",
"doc": "Example AdStax Customer CRM data",
"namespace":"io.adstax.clientId",
"fields": [
{"name": "IDFA", "type": ["null","string"]},
{"name": "GAID", "type": ["null","string"]},
{"name": "DOB", "type": ["null","string"]},
{"name": "Gender", "type": "enum", "symbols": ["M","F","Unknown"]},
]}

Data following above schema in JSON format (e.g. customerdatarecords.json):

{"IDFA":"13CA035B-7G0F-4989-A9CE-999EAB23605A","GAID":"","DOB": "1983/04/12", "Gender": "M" }
{"IDFA":"6D92078A-8246-4BA4-AE5B-76104861E7DC","GAID":"","DOB": "1965/12/21", "Gender": "M" }
{"IDFA":"AEBE52E7-03EE-455A-B3C4-E57283966239","GAID":"","DOB": "1995/01/27", "Gender": "F" }
{"IDFA":"","GAID":"26ba0cb3-deac-4203-83d3-dc1c74080af7","DOB": "1975/05/17", "Gender": "M" }
{"IDFA":"","GAID":"4c23fac7-3a36-7408-bc31-fe95abc317af","DOB": "1986/03/01", "Gender": "Unknown" }

As AVRO is a binary format, it is not suitable for showing an example on this document, but the above JSON could be converted to the equivalent AVRO format with the official AVRO command line tools as follows:

java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file adstaxcustomerdata.avsc customerdatarecords.json > adstaxcustomerdata.snappy.avro