DDEX recently created a new and powerful flat-file standard for sales/usage reporting (DSR). DDEX recommends to use this standard. For users requiring the older versions – flat-file or XML-based – should contact the DDEX Secretariat. This section of the DDEX Knowledge base contains:
What is the aim of DDEX flat-file DSR Format?
The aim of DDEX's flat-file DSR format is to enable licensees to report to rights controllers information regarding the level of usage and/or revenue generated from the distribution of music or videos.
The flat-file format released in 2016 addresses the requirements for sales/usage reports to owners and controllers of rights in musical works. DDEX has, however, started looking at requirements for sales/usage reports to owners and controllers of rights in sound recordings and expects to extend the standard in the future.
The new flat-file format replaces the previous XML format, which was in production for close to a decade, as it ultimately proved to be unable to meet the requirements for a format that has to be able to be easily adapted to address the technical challenges of the ever-changing digital music industry.
The approach taken by DDEX when developing the flat-file DSR is very pragmatic as it offers simplicity but can be used (or: extended) to all business models.
Why should a company use the flat-file DDEX DSR Format?
- Already widely adopted: Even if the standard has only been published in 2016, this new standard is already widely adopted by many actors in the digital music industry;
- Simple to use and implement: Experiences from early adopters show that the format is very easy to understand, read and implement;
- Business reporting flexibility: It is possible to combine several offers and/or territories in a single sales/usage report. This allows avoiding to repeat the same information multiple times;
- Reports volume decreasing: The format keeps reports as compact as possible. The flexibility described above combined with an optimized flat-file structure reduces volumes drastically: Up to 100s of times when compared to the old XML format;
- Enrichment of necessary information: Data fields not used in the XML format were removed from the new standard whole others were added (e.g. identifiers for parties);
- Processing efficiency: the old XML format obliged receivers to compute and store first all Release, Resource and Work data before being able to process related sales/usage data. The new standard combines Release, Resource and Work data and sales lines into self-contained “block” that can be processed independently form one another;
- Cheap to implement and to run: One of the consequences of having a simple structure is the reduced cost for development and maintenance. Combined with the reduction of data volumes to be exchanged and processed, this allows a cut in operational costs; and
- Flexibility in terms of evolution of the format: The new format is based on a logic that allows the standard to be easily adapted and exchanged to changes in reporting requirements.
How to get started?
The first thing is to check which profile you need to use. This decision will depend on the business model (or models) you use. Once you have made this decision you can download the appropriate standard from the red box on the right hand side.
In order to encompass all business models, DDEX created different profiles in order to avoid creating generic format that becomes complex in the end. In order to facilitate implementation, these profiles use the different records and value sets and the same basic architecture described in a few generic documents.
In any case, implementers need to have a look at:
- Part 1: Architecture: general technical structure of DDEX DSR Flat-file;
- Part 2: Allowed-value Sets: all value sets used for different fields; and
- Part 8: Record Type Definitions: description of the records used in the different profiles;
- The Part (or Parts) of the flat-file DSR standard that documents the profile (or profiles) relevant to the implementer's operation (see next question); and
- The samples (providing at least one example for each profile).
Which profile should a company use?
Business model addressed
Basic Audio Profile
The aim of the Basic Audio Profile is to enable licensees to report to rights controllers information regarding the level of usage and/or revenue generated from the distribution of music.
The aim of the UGC Profile is to enable licensees to report to rights controllers information regarding the level of usage and/or revenue generated from the usage of music where content may have been uploaded to a licensee's platform by any user which is not necessarily the owner or right controller of the content.
The aim of the Audio-visual Profile is to enable licensees (typically digital service providers) to report to rights controllers (typically audio-visual rights societies) information regarding the level of usage and/or revenue generated from the distribution of audiovisual recordings (excluding musical videos).
Royalty Reporting Profile
The aim of the Royalty Reporting Profile is to provide a standardised mechanism for Licensees to report to Rights Controllers Royalties based on usages of musical Releases that took place in countries where there is a direct contractual relationship between Message Sender and Message Recipient.
Radio Broadcast Profile
The aim of the Radio Broadcast is to enable Licensees to report to Rights Controllers information regarding the level of usage and/or revenue generated from the distribution of Music in the context of Global Radios.
|Profile for Financial Reporting to Record Companies|
The aim of the Profile for Financial Reporting to Record Companies is to report to record companies sales and/or usage information.
The aim of the Masterlist Profile is to allow communicating Release, Resource and Work information before any usage has been generated or before such usage is ready to be communicated.
|Basic Audio Profile for The Mechanical Licensing Collective|
The message defined in this Part provides a mechanism for Licensees (typically Digital Service Providers) to report to The Mechanical Licensing Collective (as established under the US Music Modernization Act) sales/usage information from the exploitation of Products based on electronic Releases containing Sound Recordings which embody Musical Works and/or other Resources. This Profile is a simplified version of the Basic Audio Profile and most aspects that apply to this profile also apply to the Basic Audio Profile for The Mechanical Licensing Collective.
Single-Record Block Variant or Multi-Block Variant?
DDEX has defined some of its profiles in two forms:
- Multi-Record Block Variants (MRBV) and
- Single-Record Block Variants (SRBV).
For most profiles, the MRBV is the default.
In MRBVs the main body of the sales/usage report is made out of "Blocks" which contain multiple Records. The Blocks in the Basic Audio Profile, for instance is made up of one Record the Release that has been transacted, one or more Records for each of the Resources that make up the Release. For each of the Resource Records there may then be none, one or multiple Records describing Musical Works that are used in each of the Resources. Further Records can then be used to describe Sub-Releases and, finally, provide different sales/usage figures. This is depicted on the right.
SRBVs, on the other hand, collapse all of the above information into one single, albeit potentially long, Record. This makes the ingestion process simpler. Also, smaller SRBV files can easier be imported into a spreadsheet application. These benefits do come at the cost of potentially repeating a significant amount of information. MRBVs are therefore notably more efficient than SRBV.
The choice between MRBV and SRBV depends on multiple factors that sender and recipient need to weigh up:
- How large is the report likely to be? Is it a problem if the report is, multiple times larger because of data repetition?
- Is it important to easily import the sale/usage report into a spreadsheet application?
- Will sender and recipient be able to handle richer data structures?
How to concretely implement the standard and start using it?
The steps for successfully implementing the flat-file DSR standard are
- Identification of the Profile or Profiles that need implementing;
- Taking-out an implementation licence and obtaining a DDEX Party ID on dpid.ddex.net (more information on DPIDs is available here);
- Reading the relevant documentation (i.e. Parts 1, 2 and 8 as well as the Part for the relevant Profile(s)) and looking at the appropriate samples (see the red box on the right);
- As a sender of a sales/usage report it is recommended that a DSRF file is tested against the "dsrf validator tool" that can be downloaded from GitHub. While this tool cannot determine in all cases whether a DSRF file is valid, it can help implementers to identify any issues. This can be done only as part of the implementation and/or onboarding phase or as a general test before each sales/usage report is delivered;
- Recipients may find it prudent to use the "dsrf validator tool" before ingesting a received sales/usage report into their system to reduce the likelihood of buggy reports being processed.
Steps 1 and 3 as well as 5 and 6 are recommended to be undertaken with a specific partner in mind and
Delimiters and Special Characters
Clause 6.6.4 of Part 1 states that "to communicate Delimiters in a Cell [...] the Delimiters shall be immediately preceded by a backslash character (Unicode U+005C). Therefore the string A|B would have to be communicated as A\|B.":
Assume that a creation has three contributors: Paul Simon, Art Garfunkel and 24|7. This will need to be communicated as
Note that there are no quotes used to enclose either the complete string or its three components. If one adds a fourth contributor called “Mad Dog” Dolly, one would have
Assume that a creation has three contributors: Paul Simon, Art Garfunkel and 24\7. This will need to be communicated as
Multiple Identifiers for Multiple Parties
Party Names and Party Identifiers
Some records in the DSR and CDM standards allow to report multiple parties, this happens for instance in MW01 records with the need to send multiple writers, this fields would generally be broken down into two cells:
- One cell for the name of the parties (e.g.
- One Cell for the identifiers (e.g.
These two Cells are linked, i.e. the order in which the names and the identifiers are provided, needs to be the same so that the recipient of the message can know which names belongs with which identifier. The first name belongs with the first identifier, the second name belongs with the second identifier and so on. Changing the order of one of the fields without changing the order in the linked field would lead the wrong attribution of names and identifiers. Consequently, if there are only identifiers for a subset of the parties, then an empty space in the right place would need to be left to indicate that there is no identifier for that particular party.
For instance, if a Message Sender had the follow information:
IPI Name Number
the fields ComposerAuthorName and ComposerAuthorPartyId would be provided by using the “secondary delimiter”, the pipe character (|). The identifiers themselves also need to be namespaced using the double colon syntax:
Note that the two adjacent pipe characters indicate that the second name, John Smith, is not accompanied by an identifier.
Multiple Party Identifiers
If the Message Sender wants to report multiple identifiers for any given party, the individual identifiers can be communicated using the escaping mechanism described below.
The examples use the following data:
Process to write multiple identifiers
Step 1 – The content of each identifier needs to be escaped, this is done by adding a single backslash (\) character in front of any backslash, pipe or tab character. This is only necessary if any of the identifiers contain a backslash, pipe or tab character.
Step 2 – Creation of a single identifier string for each party’s identifiers by separating them with pipes. For Jane Smith this would be
Step 3 – The content of this string needs to be escaped by adding a single backslash character in front of any backslash, pipe or tab character. For Jane Smith this would become
Step 4 – All identifier strings can now be combined, separated by an un-escaped pipe character. For Jane Smith and Peter Miller we would have:
Step 5 – The content of this string needs to be escaped by adding a single backslash character in front of any backslash or tab character.
Step 6 – This string can now be combined with the names and placed into the message:
The process of extracting the identifiers is the inverse of the above: (i) “un-escape” the record containing the identifiers and extract the name/identifier cell pair, (ii) split the ID cell into the strings for each party at the pipe character, (iii) take each of these sub strings and “un-escape” them by removing all backslashes, and (iv) split the sub-strings at the pipe character into the individual identifiers:
If there are escaped characters in the identifiers the process of un-escaping is more complex as one additional step of un-escaping these strings is required. Whether this is necessary can be determined by checking if any of the strings resulting from step (iv) contains a backslash.
Revenue and IndirectValue
The DSR standard has a number of cells to communicate monetary values from a DSP to a licensor. Two of these are
Revenue, is for any money that the consumer has paid to get access to the content. This may be a subscription fee or the amount paid for a download. This cell may also contain revenue generated from adverts if such revenues are directly linked to a consumer action such as taking out a subscription.
IndirectValue (it used to be called
IndirectRevenue in an earlier version of the DSR standard), is for any monetary value credited to the DSP’s service that cannot directly be linked to a consumer action. Examples include revenues derived from the sale of tickets to live events, or from the sale of advertisements, sponsorships, brand placements or other ad partnerships targeted to live events. It also includes cases where such generated monetary values are not based on direct revenue (e.g. where a DSP needs to report a (monetary) value that was generated independently from the distribution of music).
IndirectValue must not be used for communicating any other monetary value such as costs, royalties, etc.
The contract between licensee and licensor will typically define which amounts should be reported in which field.
Dealing with uncommon use cases
Most music distribution companies use a common set of distribution methods (e.g. pay as you go downloads, subscription-based streaming, etc.) whose key data points can comfortably be reported to the licensors of the rights in musical works and sound recordings using one of the profiles in the DSR standard. Some of these music distribution companies, however, have entered into licence agreements with licensors that require usage data to be exchanged that goes beyond the commonly used data points contained in the profiles of the DSR standard.
Whilst DDEX makes every attempt to ensure that its standards accommodate as wide a set of requirements as possible, ultimately it is not possible for DDEX’s standards to enable the communication of every possible single item of data that may exist across the music industry value chain. This obviously applies as well to the various profiles within the DSR standard.
A licensee that is required to communicate data points about usage which are not accommodated in one of the profiles of the DSR standard, has three options:
- Wait until DDEX augments the relevant DSR standard profile in a new version with cells for these data points;
- Add cells for the additional data points into an otherwise conformant DSR standard message; or
- Communicate the additional data in a separate proprietary message sent in conjunction with a conformant DSR standard message.
Approach (1) above has two drawbacks:
- It introduces a delay in a licensee’s ability to report data it is contractually obligated to report; and
- The relevant updated profile in the DSR standard would become more and more complex with the addition of more and more cells that may well only ever be used by very few licensees.
DDEX tries to mitigate the delay issue by being as responsive to new requirements as possible. However, DDEX cannot resolve the second issue by meeting all licensees’ requirements because, some data points are “too esoteric” to be suitable for standardisation.
Approach (2) above would mean that the message sent is not conformant with the relevant DSR profile – and thus it is not conformant with the DSR standard. This implies that the licensee with the extra reporting needs, as well as all the licensors receiving such “extended” messages would need to have at least two implementations of the standard, a compliant DSR one, and one that supports the additional cells. Such extensions would need to be agreed bilaterally between licensors and licensees and thus negates the benefit of having a standard in the first place.
Approach (3) would keep the implementations of the standard common and shift all extra data points into a separate message. While somewhat similar to (2) it does mean that all companies can keep their DSR implementations conformant and thus maintain just one implementation. However, they would need to maintain multiple proprietary formats for the extra data at least until DDEX determines that some of the data points are sufficiently common, and adds them into a new version of the relevant DSR profile.
It is for the reasons in the last paragraph that DDEX recommends that if any licensee is required to communicate data that goes beyond the commonly used data points contained in the profiles in the DSR standard they should use approach (3) above and avoid approaches (1) and (2).