TPP known issues

SRCode re-bulks

Under GPIT Futures, TPP offer an IM1 Bulk Extract to extract data from SystmOne. TPP call this their ‘Strategic Reporting’ extract, or just ‘SR’ for short. This is their only mechanism for extracting data on an organisation-level (i.e. all patients at an organisation).

TPP offer other mechanisms to extract data, but these are used on a patient-by-patient basis. Any third-party getting data from SystmOne for entire organisations is using their Strategic Reporting extract (although they may not be using it through IM1 Pairing integration).

The TPP Strategic Reporting extract works by generating the extracts in the SystmOne data centre, then downloading the extract files to a nominated SystmOne Gateway client, which are then available for the extracting organisation to utilise.

For DDS, the specific flow is:

Generate extract (SystmOne data centre) -> Download (to SystmOne Gateway) -> Upload (to DDS) -> Process into DDS

The SystmOne Strategic Reporting extract contains many different files, each representing a different type of clinical concept; for example, a file for repeat medications, or a file for patient addresses. When a SystmOne organisation first starts publishing data to DDS the first Strategic Reporting extract will contain all existing data for all patient records at the organisation - a ‘bulk’ extract. The following day, and each day after that, only the changes from the previous extract are included in the files, making them considerably smaller - ‘delta’ extracts. Extracting, downloading and processing a bulk extract, which may contain more than 10GB of data, is obviously considerably slower and more resource intensive than processing a daily delta, which is typically less than 20MB.

The SRCode file is the largest file in the Strategic Reporting extract, containing all CTV3/Snomed coded items, including numeric values, codes entered during consultations, data entered into new and old-style templates. SRCode typically contains as much data as the other files put together. So if an extracted SystmOne organisation's bulk extract was 10GB, their SRCode file will be 5GB of that.

TPP have a semi-regular issue where they are unable to generate the expected daily deltas of the SRCode files for their Strategic Reporting organisations. Rather than a daily delta of around 20MB of data, the extract delivered to third parties contains a full bulk of the SRCode file, plus the expected daily deltas of the other files. So rather than a 20MB daily extract, a 5GB extract is delivered. Because the SRCode file makes up a huge proportion of the full data set, when this issue occurs, it impacts on the time taken to generate the extracts and the time taken to download them to the SystmOne Gateway (the time taken to upload and process them in to DDS isn't hugely affected). This issue seemingly affects many (if not all) users of the TPP Strategic Reporting system, not just DDS.

We have been informed that TPP did an update to SystmOne (WE 19/09/2020) and as a consequence of this the issue with SRCode re-bulks has occurred again since then.

This issue has happened ten times so far in 2020, on and around the below dates:

• 2020-04-02
• 2020-05-03
• 2020-05-07
• 2020-05-31
• 2020-06-07
• 2020-06-26
• 2020-07-06
• 2020-08-23
• 2020-08-28
• 2020-09-17

Updated: 21 September 2020