TPP known issues

From Discovery Data Service
Revision as of 13:19, 9 November 2020 by JoC (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
Add this page to your watchlist to get email notifications to new changes to this page.


Current issues

TPP strategic reporting extract data not included in rebulk

  1. No delta extract was received for the date range 2020-10-05 1707 to 2020-10-08 1707. Each TPP extract contains a file called SRManifest that states the time period the extract spans. The delta extract for that period had been lost/corrupted on the SystmOne Gateway machine.
  2. The extract for the period 2020-10-09 1707 to 2020-10-10 1258 was received by DDS. The SREvent file contained a record with ID 20890171236 for patient ID 53226527. Patient ID was unknown to DDS, having never been in an SRPatient file, so this record was correctly ignored.
  3. To make up for the lost delta, a rebulk of practice data from the last month was requested.
  4. The rebulk arrived, with its SRManifest stating it contained data up to 2020-10-14 1708, and was processed by DDS. This rebulk contained patient ID 53226527 in the SRPatient and SRPatientRegistration files, with the registration timestamp showing the registration happened at 2020-10-08 1433, which was in the time period of the missing delta. So the absence of the patient was clearly due to the lost daily delta.
  5. The rebulk did not contain SREvent record with ID 20890171236, despite it having being sent well within the last month. Because the rebulk did not contain the SREvent record, it was not processed, at a point when the patient record existed in DDS, so that SREvent record has not been saved to DDS (as a FHIR Encounter). The end result is that DDS is missing this FHIR Encounter for this patient record.

Two possible scenarios are caused by the above rebulk data issue:

  1. Rebulk doesn't contain data that DDS has previously received
    In the above scenario, the SREvent that was missed from the rebulk had previously been received by DDS and had been ignored because the patient was unknown to DDS. Now DDS is aware of the patient, it should be possible to reprocess any previously ignored data and create the missing FHIR Encounter. This is essentially the same process as happens when missing EMIS clinical codes arrive, and it automatically re-queues any previously ignored data. The same mechanism maybe could be adapted for this use, or the data received since the gap could simply be re-queued and the missing data restored (depending on whether we expect this to happen again or not).
  2. Rebulk doesn't contain data that DDS has not previously received
    What if there are other records missing from the re-bulk that DDS had not previously received? We only know the rebulk is missing an SREvent record because that record had been received in one of the deltas that wasn't lost. However, there's no reason to believe that the rebulk isn't also missing SREvent (and SRCode, SRImmunisation etc.) records that were only in the delta that was lost. This means that possibly there are records that have never been sent to DDS.
The TPP gateway is managed by the NWL team. They are aware of the situation and are working with us to resolve this issue as quickly as possible.

Updated: 2 November 2020

TPP Gateway

Due to an ongoing issue with the TPP Gateway and the need to rebulk several configurations, the Discovery Data Service is not receiving daily deltas (or is receiving delayed deltas) for the following areas:

  • Hammersmith and Fulham
  • Ealing South
  • Hounslow Great West
  • Central London

North West London is investigating this issue and doing everything they can to resolve this issue as quickly as possible.

Updated: 19 October 2020

Resolved issues

TPP delta - missing data

A corrupted TPP delta for "TPP_YDDH3_07Y_FHH", which covers HOUNSLOW FELTHAM, was received from the TPP Gateway, resulting in missing data for the following 21 practices:

E85625, E85060, E85126, E85716, E85015, E85059, E85040, E85693, E85058, E85007, E85658, E85746, E85692, E85004, E85113, E85605, E85683, E85001, E85600, E85030, and E85735

The TPP SRCode file has supplied us with a significant amount of the lost data already, but the DDS will still be missing data from the specific date that isn’t in the SRCode file (e.g. new patient registrations, vaccinations, medications). A full rebulk has been requested for each of these practices in order to obtain the missing data and will be processed as soon as it is made available to Discovery. We will provide further updates in due course.

Updated: 19 October 2020

SRCode re-bulks

Under GPIT Futures, TPP offer an IM1 Bulk Extract to extract data from SystmOne. TPP call this their ‘Strategic Reporting’ extract, or just ‘SR’ for short. This is their only mechanism for extracting data on an organisation-level (i.e. all patients at an organisation).

TPP offer other mechanisms to extract data, but these are used on a patient-by-patient basis. Any third-party getting data from SystmOne for entire organisations is using their Strategic Reporting extract (although they may not be using it through IM1 Pairing integration).

The TPP Strategic Reporting extract works by generating the extracts in the SystmOne data centre, then downloading the extract files to a nominated SystmOne Gateway client, which are then available for the extracting organisation to utilise.

For DDS, the specific flow is:

Generate extract (SystmOne data centre) > Download (to SystmOne Gateway) > Upload (to DDS) > Process into DDS

The SystmOne Strategic Reporting extract contains many different files, each representing a different type of clinical concept; for example, a file for repeat medications, or a file for patient addresses. When a SystmOne organisation first starts publishing data to DDS the first Strategic Reporting extract will contain all existing data for all patient records at the organisation - a ‘bulk’ extract. The following day, and each day after that, only the changes from the previous extract are included in the files, making them considerably smaller - ‘delta’ extracts. Extracting, downloading and processing a bulk extract, which may contain more than 10GB of data, is obviously considerably slower and more resource intensive than processing a daily delta, which is typically less than 20MB.

The SRCode file is the largest file in the Strategic Reporting extract, containing all CTV3/Snomed coded items, including numeric values, codes entered during consultations, data entered into new and old-style templates. SRCode typically contains as much data as the other files put together. So if an extracted SystmOne organisation's bulk extract was 10GB, their SRCode file will be 5GB of that.

TPP have a semi-regular issue where they are unable to generate the expected daily deltas of the SRCode files for their Strategic Reporting organisations. Rather than a daily delta of around 20MB of data, the extract delivered to third parties contains a full bulk of the SRCode file, plus the expected daily deltas of the other files. So rather than a 20MB daily extract, a 5GB extract is delivered. Because the SRCode file makes up a huge proportion of the full data set, when this issue occurs, it impacts on the time taken to generate the extracts and the time taken to download them to the SystmOne Gateway (the time taken to upload and process them in to DDS isn't hugely affected). This issue seemingly affects many (if not all) users of the TPP Strategic Reporting system, not just DDS.

We have been informed that TPP did an update to SystmOne (WE 19/09/2020) and as a consequence of this the issue with SRCode rebulks has occurred again since then.

This issue has happened ten times so far in 2020, on and around the below dates:

  • 2020-04-02
  • 2020-05-03
  • 2020-05-07
  • 2020-05-31
  • 2020-06-07
  • 2020-06-26
  • 2020-07-06
  • 2020-08-23
  • 2020-08-28
  • 2020-09-17
The DDS SFTP Reader application now has built-in filtering to detect the large bulk files and remove any previously received identical record.

Updated: 2 November 2020