Skip to end of metadata
Go to start of metadata

DDEX recently created a new and powerful flat-file standard for sales/usage reporting (DSR). DDEX recommends to use this standard. For users requiring the older versions – flat-file or XML-based – should contact the DDEX Secretariat. This section of the DDEX Knowledge base contains:

What is the aim of DDEX flat-file DSR Format?

The aim of DDEX's flat-file  DSR format is to enable licensees to report to rights controllers information regarding the level of usage and/or revenue generated from the distribution of music or videos.

The flat-file format released in 2016 addresses the requirements for sales/usage reports to owners and controllers of rights in musical works. DDEX has, however, started looking at requirements for sales/usage reports to owners and controllers of rights in sound recordings and expects to extend the standard in the future.

The new flat-file format replaces the previous XML format, which was in production for close to a decade, as it ultimately proved to be unable to meet the requirements for a format that has to be able to be easily adapted to address the technical challenges of the ever-changing digital music industry.

The approach taken by DDEX when developing the flat-file DSR is very pragmatic as it offers simplicity but can be used (or: extended) to all business models.

Why should a company use the flat-file DDEX DSR Format?

  • Already widely adopted: Even if the standard has only been published in 2016, this new standard is already widely adopted by many actors in the digital music industry;
  • Simple to use and implement: Experiences from early adopters show that the format is very easy to understand, read and implement;
  • Business reporting flexibility: It is possible to combine several offers and/or territories in a single sales/usage report. This allows avoiding to repeat the same information multiple times;
  • Reports volume decreasing: The format keeps reports as compact as possible. The flexibility described above combined with an optimized flat-file structure reduces volumes drastically: Up to 100s of times when compared to the old XML format;
  • Enrichment of necessary information: Data fields not used in the XML format were removed from the new standard whole others were added (e.g. identifiers for parties);
  • Processing efficiency: the old XML format obliged receivers to compute and store first all Release, Resource and Work data before being able to process related sales/usage data. The new standard combines Release, Resource and Work data and sales lines into self-contained  “block” that can be processed independently form one another;
  • Cheap to implement and to run: One of the consequences of having a simple structure is the reduced cost for development and maintenance. Combined with the reduction of data volumes to be exchanged and processed, this allows a cut in operational costs; and
  • Flexibility in terms of evolution of the format: The new format is based on a logic that allows the standard to be easily adapted and exchanged to changes in reporting requirements.

How to get started?

The first thing is to check which profile you need to use. This decision will depend on the business model (or models) you use. Once you have made this decision you can download the appropriate standard from the red box on the right hand side.

In order to encompass all business models, DDEX created different profiles in order to avoid creating generic format that becomes complex in the end. In order to facilitate implementation, these profiles use the different records and value sets and the same basic architecture described in a few generic documents.

In any case, implementers need to have a look at:

  1. Part 1: Architecture: general technical structure of DDEX DSR Flat-file;
  2. Part 2: Allowed-value Sets: all value sets used for different fields; and
  3. Part 8: Record Type Definitions:  description of the records used in the different profiles;
  4. The Part (or Parts) of the flat-file DSR standard that documents the profile (or profiles) relevant to the implementer's operation (see next question); and
  5. The samples (providing at least one example for each profile).

Which profile should a company use?

Profile

Business model addressed

Basic Audio Profile

The aim of the Basic Audio Profile is to enable licensees to report to rights controllers information regarding the level of usage and/or revenue generated from the distribution of music.

More information on the Basic Audio Profile can be found here.

UGC Profile

The aim of the UGC Profile is to enable licensees to report to rights controllers information regarding the level of usage and/or revenue generated from the usage of music where content may have been uploaded to a licensee's platform by any user which is not necessarily the owner or right controller of the content.

More information on the UGC Profile – including information on scalability – can be found here.

Audio-visual Profile

The aim of the Audio-visual Profile is to enable licensees (typically digital service providers) to report to rights controllers (typically audio-visual rights societies) information regarding the level of usage and/or revenue generated from the distribution of audiovisual recordings (excluding musical videos).

More information on the Audio-visual Profile can be found here.

Royalty Reporting Profile

The aim of the Royalty Reporting Profile is to provide a standardised mechanism for Licensees to report to Rights Controllers Royalties based on usages of musical Releases that took place in countries where there is a direct contractual relationship between Message Sender and Message Recipient.

More information on the Royalty Reporting Profile can be found here. 

Radio Broadcast Profile

The aim of the Radio Broadcast is to enable Licensees to report to Rights Controllers information regarding the level of usage and/or revenue generated from the distribution of Music in the context of Global Radios.

More information on the Radio Broadcasting Profile can be found here.  

Profile for Financial Reporting to Record Companies

The aim of the Profile for Financial Reporting to Record Companies is to report to record companies sales and/or usage information.

More information on the Profile for Financial Reporting to Record Companies can be found here.  

Masterlist Profile

The aim of the Masterlist Profile is to allow communicating Release, Resource and Work information before any usage has been generated or before such usage is ready to be communicated.

More information on the Masterlist Profile can be found here.

Basic Audio Profile for The Mechanical Licensing Collective

The message defined in this Part provides a mechanism for Licensees (typically Digital Service Providers) to report to The Mechanical Licensing Collective (as established under the US Music Modernization Act) sales/usage information from the exploitation of Products based on electronic Releases containing Sound Recordings which embody Musical Works and/or other Resources. This Profile is a simplified version of the Basic Audio Profile and most aspects that apply to this profile also apply to the Basic Audio Profile for The Mechanical Licensing Collective.

More information on the Basic Audio Profile for The Mechanical Licensing Collective can be found here.

Single-Record Block Variant or Multi-Block Variant? 

DDEX has defined some of its profiles in two forms:

  • Multi-Record Block Variants (MRBV) and 
  • Single-Record Block Variants (SRBV). 

For most profiles, the MRBV is the default.

In MRBVs the main body of the sales/usage report is made out of "Blocks" which contain multiple Records. The Blocks in the Basic Audio Profile, for instance is made up of one Record the Release that has been transacted, one or more Records for each of the Resources that make up the Release. For each of the Resource Records there may then be none, one or multiple Records describing Musical Works that are used in each of the Resources. Further Records can then be used to describe Sub-Releases and, finally, provide different sales/usage figures. This is depicted on the right.

SRBVs, on the other hand, collapse all of the above information into one single, albeit potentially long, Record. This makes the ingestion process simpler. Also, smaller SRBV files can easier be imported into a spreadsheet application. These benefits do come at the cost of potentially repeating a significant amount of information. MRBVs are therefore notably more efficient than SRBV.

The choice between MRBV and SRBV depends on multiple factors that sender and recipient need to weigh up:

  • How large is the report likely to be? Is it a problem if the report is, multiple times larger because of data repetition?
  • Is it important to easily import the sale/usage report into a spreadsheet application?
  • Will sender and recipient be able to handle richer data structures?

How to concretely implement the standard and start using it?

The steps for successfully implementing the flat-file DSR standard are

  1. Identification of the Profile or Profiles that need implementing;
  2. Taking-out an implementation licence and obtaining a DDEX Party ID on dpid.ddex.net (more information on DPIDs is available here);
  3. Reading the relevant documentation (i.e. Parts 1, 2 and 8 as well as the Part for the relevant Profile(s)) and looking at the appropriate samples (see the red box on the right);
  4. Implement;
  5. As a sender of a sales/usage report it is recommended that a DSRF file is tested against the "dsrf validator tool" that can be downloaded from GitHub. While this tool cannot determine in all cases whether a DSRF file is valid, it can help implementers to identify any issues. This can be done only as part of the implementation and/or onboarding phase or as a general test before each sales/usage report is delivered;
  6. Recipients may find it prudent to use the "dsrf validator tool" before ingesting a received sales/usage report into their system to reduce the likelihood of buggy reports being processed.

Steps 1 and 3 as well as 5 and 6 are recommended to be undertaken with a specific partner in mind and

Delimiters and Special Characters

 Click here to expand...

Clause 6.6.4 of Part 1 states that "to communicate Delimiters in a Cell [...] the Delimiters shall be immediately preceded by a backslash character (Unicode U+005C). Therefore the string A|B would have to be communicated as A\|B.":

Assume that a creation has three contributors: Paul Simon, Art Garfunkel and 24|7. This will need to be communicated as

Paul Simon|Art Garfunkel|24\\|7

Note that there are no quotes used to enclose either the complete string or its three components. If one adds a fourth contributor called “Mad Dog” Dolly, one would have  

Paul Simon|Art Garfunkel|24\\|7|"Mad Dog" Dolly

 

Assume that a creation has three contributors: Paul Simon, Art Garfunkel and 24\7. This will need to be communicated as

Paul Simon|Art Garfunkel|24\\\7

Handling long titles and special characters in the DSR, CDM and RDR-R standards

 Click here to expand...

Receiving a message with, say, a sound recording title that exceeds the field length restrictions in a recipient’s systems, can be problematic.

However, a recipient’s system limitations should not force the sender of the message to such a recipient to shorten the content of the string. It is up to the recipient of the message to carry out such shortening to meet the limitations of their own system(s).

The same general rule applies to the handling of special characters, such as glyphs from various languages using non-Roman characters supported by UTF-8. It is up to the recipient of the message to transliterate special character strings if their system cannot handle them.

 

Multiple Identifiers for Multiple Parties 

 Click here to expand...

Party Names and Party Identifiers

Some records in the DSR and CDM standards allow to report multiple parties, this happens for instance in MW01 records with the need to send multiple writers, this fields would generally be broken down into two cells:

  • One cell for the name of the parties (e.g. ComposerAuthorName) and 
  • One Cell for the identifiers (e.g. ComposerAuthorPartyId).

These two Cells are linked, i.e. the order in which the names and the identifiers are provided, needs to be the same so that the recipient of the message can know which names belongs with which identifier. The first name belongs with the first identifier, the second name belongs with the second identifier and so on. Changing the order of one of the fields without changing the order in the linked field would lead the wrong attribution of names and identifiers. Consequently, if there are only identifiers for a subset of the parties, then an empty space in the right place would need to be left to indicate that there is no identifier for that particular party.

For instance, if a Message Sender had the follow information:

 

Name

IPI Name Number

Writer 1

Jane Smith

31823461941

Writer 2

John Smith

<none>

Writer 3

Peter Miller

73519527851

 

the fields ComposerAuthorName and ComposerAuthorPartyId would be provided by using the “secondary delimiter”, the pipe character (|). The identifiers themselves also need to be namespaced using the double colon syntax:

ComposerAuthorName

ComposerAuthorPartyId

Jane Smith|John Smith|Peter Miller

IPI::31823461941||IPI::73519527851

Note that the two adjacent pipe characters indicate that the second name, John Smith, is not accompanied by an identifier.

Multiple Party Identifiers

If the Message Sender wants to report multiple identifiers for any given party, the individual identifiers can be communicated using the escaping mechanism described below.

The examples use the following data:

 

Name

Identifiers

Writer 1

Jane Smith

IPI::31823461941 and ISNI::9876876576546543

Writer 2

Peter Miller

IPI::73519527851 and ISNI::1234234534564567

Process to write multiple identifiers

Step 1 – The content of each identifier needs to be escaped, this is done by adding a single backslash (\) character in front of any backslash, pipe or tab character. This is only necessary if any of the identifiers contain a backslash, pipe or tab character.

Step 2 – Creation of a single identifier string for each party’s identifiers by separating them with pipes. For Jane Smith this would be

IPI::31823461941|ISNI::9876876576546543

Step 3 – The content of this string needs to be escaped by adding a single backslash character in front of any backslash, pipe or tab character. For Jane Smith this would become

IPI::31823461941\|ISNI::9876876576546543

Step 4 – All identifier strings can now be combined, separated by an un-escaped pipe character. For Jane Smith and Peter Miller we would have:

IPI::31823461941\|ISNI::9876876576546543|IPI::73519527851\|ISNI::1234234534564567

Step 5 – The content of this string needs to be escaped by adding a single backslash character in front of any backslash or tab character.

IPI::31823461941\\|ISNI::9876876576546543|IPI::73519527851\\|ISNI::1234234534564567

Step 6 – This string can now be combined with the names and placed into the message: 

ComposerAuthorName

ComposerAuthorPartyId

Jane Smith|Peter Miller

IPI::31823461941\\|ISNI::9876876576546543|IPI::73519527851\\|ISNI::1234234534564567

Extracting Information

The process of extracting the identifiers is the inverse of the above: (i) “un-escape” the record containing the identifiers and extract the name/identifier cell pair, (ii) split the ID cell into the strings for each party at the pipe character, (iii) take each of these sub strings and “un-escape” them by removing all backslashes, and (iv) split the sub-strings at the pipe character into the individual identifiers:

If there are escaped characters in the identifiers the process of un-escaping is more complex as one additional step of un-escaping these strings is required. Whether this is necessary can be determined by checking if any of the strings resulting from step (iv) contains a backslash.

Revenue and IndirectValue

 Click here to expand...

The DSR standard has a number of cells to communicate monetary values from a DSP to a licensor. Two of these are Revenue and IndirectValue

The former, Revenue, is for any money that the consumer has paid to get access to the content. This may be a subscription fee or the amount paid for a download. This cell may also contain revenue generated from adverts if such revenues are directly linked to a consumer action such as taking out a subscription.

The latter, IndirectValue (it used to be called IndirectRevenue in an earlier version of the DSR standard), is for any monetary value credited to the DSP’s service that cannot directly be linked to a consumer action. Examples include revenues derived from the sale of tickets to live events, or from the sale of advertisements, sponsorships, brand placements or other ad partnerships targeted to live events. It also includes cases where such generated monetary values are not based on direct revenue (e.g. where a DSP needs to report a (monetary) value that was generated independently from the distribution of music).

IndirectValue must not be used for communicating any other monetary value such as costs, royalties, etc.

The contract between licensee and licensor will typically define which amounts should be reported in which field.

Currency conversion in DSR and CDM 

 Currency conversion in DSR and CDM
The DSR and CDM standards enable the communication of three types of monetary amounts in multiple currencies:
  • The currency of the transaction (DSR);
  • The currency of reporting (DSR and CDM); and
  • The currency of invoicing (CDM).

 

Both standards enable the communication of two monetary amounts for most crucial financial data points. In addition to these data points the standards also enable the identification of the conversion rate, the source for the conversion rate and the date the conversion rate was published.

For recipients of messages containing these data points, the conversion rate should be used to multiply the source amount to arrive at the target amount. In the DSR standard this means that a transaction amount multiplied by the conversion rate will result in the amount being reported. In the CDM standard a reporting amount multiplied by the conversion rate will result in the amount being invoiced.

These conversion rates are solely for the conversion of the monetary amounts included in the relevant messages. The nature of the status of such conversion rates and the date the conversion rate was published is determined by the bilateral agreement between the licensee and the licensor.

As the application of multiple currency conversions to the same monetary amounts will likely lead to errors, DDEX recommends that a maximum of two currency amounts and one currency conversion rate are used to extrapolate monetary amounts in the use of chains of DSR and CDM messages.


 Dealing with uncommon use cases

 Dealing with uncommon use cases
The music industry is complex. There are many ways in which music is exploited and traded, especially since the methods of distributing music are constantly evolving. This means that the mechanisms for the reporting of such exploitations, DDEX calls them “usages”, are also complex.

Most music distribution companies use a common set of distribution methods (e.g. pay as you go downloads, subscription-based streaming, etc.) whose key data points can comfortably be reported to the licensors of the rights in musical works and sound recordings using one of the profiles in the DSR standard. Some of these music distribution companies, however, have entered into licence agreements with licensors that require usage data to be exchanged that goes beyond the commonly used data points contained in the profiles of the DSR standard. 

Whilst DDEX makes every attempt to ensure that its standards accommodate as wide a set of requirements as possible, ultimately it is not possible for DDEX’s standards to enable the communication of every possible single item of data that may exist across the music industry value chain. This obviously applies as well to the various profiles within the DSR standard.

A licensee that is required to communicate data points about usage which are not accommodated in one of the profiles of the DSR standard, has three options:

  1. Wait until DDEX augments the relevant DSR standard profile in a new version with cells for these data points;
  2. Add cells for the additional data points into an otherwise conformant DSR standard message; or
  3. Communicate the additional data in a separate proprietary message sent in conjunction with a conformant DSR standard message.

Approach (1) above has two drawbacks: 

  • It introduces a delay in a licensee’s ability to report data it is contractually obligated to report; and
  • The relevant updated profile in the DSR standard would become more and more complex with the addition of more and more cells that may well only ever be used by very few licensees.

DDEX tries to mitigate the delay issue by being as responsive to new requirements as possible. However, DDEX cannot resolve the second issue by meeting all licensees’ requirements because, some data points are “too esoteric” to be suitable for standardisation.

Approach (2) above would mean that the message sent is not conformant with the relevant DSR profile – and thus it is not conformant with the DSR standard. This implies that the licensee with the extra reporting needs, as well as all the licensors receiving such “extended” messages would need to have at least two implementations of the standard, a compliant DSR one, and one that supports the additional cells. Such extensions would need to be agreed bilaterally between licensors and licensees and thus negates the benefit of having a standard in the first place.

Approach (3) would keep the implementations of the standard common and shift all extra data points into a separate message. While somewhat similar to (2) it does mean that all companies can keep their DSR implementations conformant and thus maintain just one implementation. However, they would need to maintain multiple proprietary formats for the extra data at least until DDEX determines that some of the data points are sufficiently common, and adds them into a new version of the relevant DSR profile.

It is for the reasons in the last paragraph that DDEX recommends that if any licensee is required to communicate data that goes beyond the commonly used data points contained in the profiles in the DSR standard they should use approach (3) above and avoid approaches (1) and (2).

 What should I do if I have data requirements that are not addressed by DDEX?
DDEX standards are developed by consensus amongst its members and while DDEX standards are typically created comparatively speedily, there are instances where a company’s data requirements are not addressed by a new version of a DDEX standard. Examples include missing tags or cells for data that a company needs to communicate to a business partner or, conversely, cases where a tag or cell is mandatory but where the prospective message sender does not have suitable data to populate that tag or cell.

In such cases it is permitted for parties to bilaterally agree to deviate from the published DDEX standard. The companies that agree on such a deviation should reach out to DDEX to seek to have the deviation adopted as an agreed part of the next version of the relevant standard. Furthermore, once such adoption has been agreed and a new version published companies should endeavour to stop using such deviations as soon as possible.

 LiveStream
Over the last few years, UGC video services have started to allow their content creator users to “go live”, i.e. to record a video and stream it to their “followers” at the same time. Once the live event has finished, the recorded video usually continues to be available as a stream or download to consumers. The same, albeit on a lesser scale, also happens with audio-only content.
When analysing these services, DDEX has determined that such offerings are, in fact, two distinct (but combined) offerings to consumers:
  1. The DSP allows its consumers to stream and enjoy the stream at time of recording and
  2. The DSP allows its consumers to stream or download the recording after the live stream has concluded.

To support the first of these, DDEX has added a new UseType called LiveStream into its Data Dictionary. For the time being this allowed value is only available to the DSR Standard, i.e. for the reporting of the live stream to potential licensors of music embedded in the live stream. The second offering is covered by existing UseTypes.

Note: the term LiveStream was, however, already in DDEX’s Data Dictionary, albeit with a different meaning and for a different purpose. This old term, used to describe a resource based on its content, intended audience, format or technical characteristics has been renamed to LiveEventStream. For backwards compatibility, the allowed value in the RecordingFormat allowed value set remains LiveStream.

 GDPR and UGC
The UGC profile requires that the DSP sends information about UCG resources that consumers have created and uploaded to the DSP’s service for distribution to other consumers. This is communicated in an RU record defined in the UGC profile of the DSR standard and contains, amongst other things, an identifier for the UGC resource that enables the licensor to listen to or to watch the UGC resource as part of the process of identifying the musical works used. 

There are instances, however, where the DSP is required to remove such UGC resources from its service. One example is the “right to forget” provisions of the European General Data Protection Regulation (GDPR). This may require the DSP to not only remove the UGC resource but also the associated identifier from its internal systems.

As a consequence, the DSP may not be able to provide the identifier of a removed UGC resource in the relevant RU record when reporting usages to the licensor.

In such circumstances the DSP should omit that particular UGC resource from the RU record in its sales/usage report to the licensor even if the UGC resource had been made available to consumers during the period of the sales/usage report, before it was taken down. The sales/usage will of course still need to be reported in the relevant SU record.

If, however, this means that there are no UGC resources to be reported in an RU record because, for example, all UGC resources based on a specific sound recording or video have been taken down, the DSP may wish to include at least one item in a RU record with the identifier cell containing the token #deleted#.

  • No labels