In some of the last projects, we faced a special situation: the need to consolidate information located on several thousand servers located in multiple locations. Beyond the huge number of servers (and implied databases, tables and fields, etc.) the challenges also consisted in identifying the meaning, logics, quality, age, relevance and redundancy of the data in all these containers. This is how QQdata.inventory™ was born.
The QQdata.inventory™ engine allows, once configured, addresses and credentials to connect to all of these databases:
- connecting to all data sources;
- saving the login result;
- scanning related metadata;
- identifying the age of the data contained;
- if necessary, copying all data in compressed format (this is where backup options are born!);
- analyzing data structures and identifying potential logics for connecting tables to each other;
- construirea unui catalog de date.
- monitoring the connectivity status of each server.
Once we have all this information loaded into the Qlik™ space, we can fully exploit, first of all, the Qosmic interface, applied even to the metadata extracted from the database, or even to the metadata (logs) of the scanning processes.
1. Dashboard of sub-process activations
Any process of reading and processing data that takes at least 10 minutes, multiplied by several thousand times, generates at least significant processing time, as such, it is extremely important to monitor and manage the running of all sub-processes in batches and on stages.
For this purpose, we have designed and created a series of monitoring and control interfaces, which allow the launching and stopping of processing activities in the most versatile way, which allows rapid adaptation to various situations that may be encountered.
The matrix approach, systematic, allows a modularization of the processing.
2. Analysis of the freshness of the information in the Server, Databases and Tables
Allows the identification of frozen Servers, Databases or Tables (with freshness exceeding a number of days specified in a variable in the interface or with the maximum year of data in the current year / year month).
Of course, the definition of the thresholds for separating fresh information from old information can be reconfigured depending on the purpose of the information aging analysis project. And the data, maximum of hierarchically lower data structures (cells, columns), determines, relatively automatically, the seniority classification for higher data hierarchies (tables, data schemas, databases, servers).
Even though the option to identify future-dated information in datasets is quite small, it can exist, (sometimes even makes sense). As such, these situations are also explicitly highlighted at any level of granularity.
It should be noted that the algorithms include, in a step immediately following the connection and retrieval of data, an automatic identification of all the columns that contain time-type information (data), and which are then used to identify the recency/freshness of the respective entity.
And variations on other evaluation and counting granularity:
or
In addition, a number of capabilities less common in common BI projects can be exploited, related to the visual highlighting of relationships between entities.
3. Charts and Networking
When we have hierarchies or processes in analysis, the dynamic visualization of hierarchical relationships and sequential interdependence between various entities becomes essential.
Here are some examples of applicability:
- Data Hierarchy (Servers/ DB/ Schemas/ Tables/ Fields/ Values);
- Execution steps of operations with included flow sequences, such as tracking courier deliveries;
- Data Lineage on tables and/or fields.
Both NetworkDiagram and OrgChart views can be used.
And if you purchase the license for the Inphinity Visualization Pack, you can add the extremely powerful and versatile Inphinity Flow visualization.
Additionally, textually or visually represented measures can be dynamically switched with the Qosmic.interface™ functionality between preconfigured measures.
Note: Views have technological limitations regarding the maximum number of entities viewed simultaneously!
Example 1: Visualization as a NetworkDiagram for the subordinate units of an entity (here OJP Galati), where the size of the symbols expresses the total number of hierarchical subordinates of the respective subunit, and the thickness of the connecting lines the number of subordinates of the direct subordinate. On Hoover (aka Mouse-Over) additional information about the selected entity appears.
Example 2: Additional information can be added to all elements on the diagram, as in the example below, at the risk of cluttering the diagram too much.
Example 3: Visualization as OrgChart for subunits under the Galati county entity.
Parts of the hierarchy can be collapsed/expanded.
Example 4: The visualization as Flow for the sub-units subordinated to OJP Galati + its superior (Galati county).
Attention! Flow views are ONLY available with Inphinity Suite commercial licensing !
Here you can define more complex symbol association logic, as well as page layout rules based on various criteria, including matrix layouts for optimal use of space.
Example of repositioning entities for more efficient use of space, depending on the setting controlled by the upper right slider of each example.
4. Comparisons of data structures
In order to identify the redundancy of data in various data structures, potentially fully or partially overlapping, we have imagined and realized an algorithm and an interface that allows understanding the levels of overlap between complex data structures.
The approach started from:
- QQconfig.tracker™, used as an auxiliary in other projects and solutions;
- QQvalidator™, used in the comparative processing of 2 data sets;
- and was also inspired by current solutions for monitoring changes in tabular structures.
For specific and specialized solutions from QQinfo, click here: QQsolutions.
If you are interested in QQdata.inventory™ product, or you need more information, please fill in the form here !
In order to be in touch with the latest news in the field, unique solutions explained, but also with our personal perspectives regarding the world of management, data and analytics, click here: QQblog !