Menu

QQdata.quality™

The QQdata.quality™ engine complements the QQvalidator, leveraging similar principles, but applied to assess the quality of source (or processed) data and to identify problems requiring:

  • alerts;
  • manual correction interventions in the source information systems;
  • automatic corrections or alerts;
  • manual correction proposals to the Data Quality Stewart.

As such, included in this engine are a configuration space for checks, alerts and corrections/correction proposals, including message types and recipient identification logics, DQ problem identification system, alerting system, lock until dataset remediation and post-correction release, the system to identify situations requiring manual correction and to collect correction decisions, plus a set of monitors providing at any time a summarized situation over the current status and history of alerts and DQ corrections, regardless of the resolution path applied (alert with upstream correction / automatic correction / manual correction).

The QQdata.quality™ app contains check alert buttons, the green button shows that the data has passed the check and is correct, and the red button shows that the data has not passed the check and some or all of the checks have failed.

Below, we present some of the interfaces of QQdata.quality™, in the context of an export-import process of invoices issued and customers registered in an Operational ERP to the accounting notes of a financial-accounting ERP.

The system allows:

  • the definition of several waves (rules) of data quality evaluation, each associated to a table;
  • general filters, valid on the respective table;
  • a specific granularity;
  • as well as sub-rules for the identification of undesirable situations, specific proposed solutions and communication-alert channels, appropriate to each situation.

An example of a very necessary customer validation rule in such a process: it is not allowed that a customer does not have in the ERP nomenclature source of information not filled in the fields Country, City and County, according to the current legal-accounting requirements (see SAF-T and eFactura, at least).

The rule has several sub-rules for checking and action, as follows:

  • If Country is not filled in, we autocomplete with Ro;
  • If Locality is not filled in, we send an email to the original person responsible (the initiator of the data entry into the ERP), but also to the central Data Quality Assurance team, or to the branch/store managers responsible for sales data quality;
  • And if the county is not filled in, we pre-fill, as a proposal to the responsible Data Stewart (local or central), the county of the location as the county of the customer, and the customer fills in directly in the Qlik™ interface his final decision.

Note: This is a fictional scenario that attempts to demonstrate the flexibility, versatility and efficiency of QQdata.quality™.

Normally, we recommend for this case that all 3 sub-rules are included in the same alerting and resolution logic: filling in the missing information in the source ERP by the responsible initiator, the alerting being done consolidated at the level of each customer identified with problems, regardless how many sub-rules have identified problems at that customer, if the resolution method is the same for all sub-rules. This alert may include automatic recommendations for completion defined in each sub-rule.

Once a record has been identified as problematic, regardless of the typology and defined correction mechanism, email alerts are sent to the responsible persons, each email alert having a specific message depending on the rule and sub-rules of the related problematic assessment, and on the possible correction proposals, if they have been defined in sub-rules.

All customers who have had problems identified are put in a Holded state, while customers who have been successfully validated are sent forward in the continuation of the import flow. But before they are sent onward, because they might be customers who have previously had a DQ issue, they are also checked to see if any of those just passed DQ validation are somehow on Hold from a previous assessment process, in which case they are also flagged and “released” from Holded status.

Each problem customer has included in the same specific alert all the validation sub-rules they have violated, if they have resolution paths, email messages and specific destinations consistently grouped together.

Below, you can see a simplified process diagram that explains this example process.

Customers for which alerts have been issued require intervention in ERP1.Comecial and will subsequently be re-imported from there, given the new, newly modified customer time-stamps in the nomenclature.

If it is confirmed that the new versions of information have passed DQ reverification, the client in question is removed from the Still Problem Clients report (Holded) and will be available in history (with Released status). Otherwise the client is put back on Holded and a new alert is issued, on the channel and with the associated message with the validation sub-rule that was not met.

There remains separate in fact a whole history of the problems identified and then solved, including the originator of the data, the person responsible for correction, the time to resolution and the person who solved it (for management, motivation, training and possible subsequent reorganization).

Data Quality Customer Inspection Interface

Provides a filterable table with information about customers (recent & relevant) that have not passed the defined Data Quality checks and are still in Holded status.

In the example below, we have presented a more generic dataset where:

  • DQId identifies the rule that triggered the problem identification,
  • the following 3 fields ensure the uniqueness (granularity of analysis) of the client, in our case,
  • then we see the time of problem identification;
    then we have all the columns on which rule specific DQ checks have been done with the values found on the row (client) in question;
    then the Holded state signaling,
  • then the column for which the DQ problem was identified, the original value and the recommended value.
Interface Manual Correction Enrichment (example for sales invoices & related notes)

Allows you to correct and/or add additional information to the sales invoice lines and/or the related notes, in particular with regard to the association of accounting dimensions (Account/Branch/Department/etc.).

It saves who made the correction and when for possible later analysis. Until a new processing iteration is launched post Manual Correction, corrections can be repeatedly overwritten. The last saved correction remains valid.

The Manual Correction Enrichment interface is usually offered to the central or local DQ. team, it is usually not necessary to correct the information in ERP1.Commercial, but only to “enrich” with additional properties the information sent to ERP2.Financial-Accounting.

The Inphinity Form General extension with a normalized view is used for maximum flexibility in the structure of the DQ rules for MC.

The email alert mechanism can be active or inactive.

The DQ.MC system also includes the definition of formulas for the recommendation of correction values, with the fallback response option “Unable to Recommend. Waiting for Human Input” in case a recommended value cannot be calculated automatically.

QQdq.mc interface provides :

  • filterable table with information about recent transactions to which Enrichment interventions are recommended by Manual Corrections as defined in the DQ.MC sub-rules (not included in the picture below);
  • Inphinity Form General correction object, with simultaneous editing option by different DQ Team members. At top right, you can visulize the number and identity of those editing. The row taken for editing by one user is locked for editing for other users. See in the picture below!

Butoanele din debutul obiectului permit :

  1. resetarea valorilor la cele de dinante de orice corectie nesalvata (clear);
  2. salvarea valorilor editate aferente inregsitrarilor vizibile/selectate (save);
    (se pot face si corectari + salvari individuale sau pe tipologi de date asemenatoare)
  3. exportarea valorilor in Excel pentru reimport ulterior sau pentru alte utilizari (export).

The following columns and functionalities can be identified above:

  • identification columns (KeyField1 & KeyField2 in our case);
  • the fields used in the evaluation with the corresponding values on each identified problem row (RecommendationFieldName);
  • recommended content in the RecommendedFieldValue column;
  • edit column with:
    • Switch between recommended value takeover or manual editing via the small square button with the fx logo or a pencil. The switching logic is highlighted as follows:
        • the recommendation fetch mode, which is the default, is active when the fx icon is visible;
        • and edit mode is active when the pencil icon is visible.

for the fx situations you can see that the value is still identical to the recommended one, while on the rows with the pencil icon, the values entered have already been altered.

  • column with delete row(s) buttons (probably unnecessary);
  • the column for completing observations (which, at the bottom of the object, allows you to fill in an observation once and populate, by pressing the button below it, all the currently visible (selected) rows with the statement filled in the input object at the bottom of the column;
  • columns with details of who and when (manually or using the recommended default value) last corrected the record in question.
Data Quality Dashboard Inspection Interface

Provides an operational summary, related to all entities currently in Holded state (regardless of DQ rule, source table or specific details).

Also included are details of the person responsible for the correction (unless it’s autocorrect, which basically only transiently places records in Holded status) and the age since the record was placed in Holded status.

Interfața de inspectare Data Quality Review

Provides a broader, historical perspective on all entities that have so far passed through any of the states:

  1. Holded 
  2. Released
  3. Passed

(regardless of DQ rule, source table or specific details).

Known historical details of the rule that triggered the alarm, recommended values, correction methodology (alert/auto/manual), final corrected value (if correction has already been made), and remarks in the manual correction mechanism are also included if needed.

Also included are details of the person responsible for the correction, the duration of the correction, and the person who actually made the correction to the record in question, (in ERP1.Comecial, or in QQdq.mc, or the fact that it was an auto-correction).

For specific and specialized solutions from QQinfo, click here: QQsolutions.
In order to be in touch with the latest news in the field, unique solutions explained, but also with our personal perspectives regarding the world of management, data and analytics, click here: QQblog !
For information about Qlik™, click here: qlik.com.