v1.6.0 - ADQ Release Notes

Status: GA

ADQ v1.6.0 focuses on performance improvements for rule executions and contains several UI enhancements to improve the user experience throughout the application.

1 New Features

1.1 DQ Workflow Optimisation

The DQ monitoring solution has been refactored to improve performance and offer more flexibility to users. The data extraction phase has been optimised to use the new ADQ toolsets, users can now enable or disable primary key validations, and users have the option to choose whether to upload DQ results data to the Data Issues (DQC) database and/or the DQ results (SSDQ) database.

For Workflow Tests users can enable/disable the check for duplicate primary keys in the rule input and choose to allow/disallow records in the rule result file (the file output by the rule containing the DQ breaks) with the same primary key. They also have the option to choose whether to upload the rule result file to Data Issues for remediation and whether to upload results to the DQ results database to be surfaced in the Insights Hub. When choosing what results to push to the results database the user can decide whether to upload only the metrics, e.g. failed count value and checked count value for calculating a DQ score, or to upload the DQ breaks as well.

For Input Data Extractions users can disable the check for duplicate primary keys in the rule input.

1.2 Introduction of ADQ Toolsets

The DQM toolsets have been extended to have a group of ADQ toolsets to simplify the complexity of the data extraction layer when running DQ rules.

1.3 Custom Data Preparation Layer

Users can create customised data preparation layers on data source views by building FlowDesigner projects to transform the data source view into the desired format. These are known as Composite Data Source Views. The output of these composite data source views can be used as input to rules.

This is particularly useful when a data preparation steps must take place for multiple rules. Rather than building out this logic each time at the beginning of every rule, the logic can be performed after the data is extracted from source and before the data is fed into the DQ rule.

1.4 Database Metadata Performance Improvements

When creating a database data source view via the SQL builder, the performance of the database table names, column names and primary key information retrieval has been improved.

Users can now also specify a table name to retrieve it’s containing columns without needing to wait for all table names to be listed first.

1.5 Apache Kafka Integration

Entity updates, e.g. rule modifications, new profile jobs setup, can now follow an event-driven approach rather than periodic polling. Since ADQ only updates when an actual change is detected, it avoids unnecessary frontend refreshes. This improves performance by reducing the number of API calls, leading to a more responsive UI.

1.6 Bulk Enable Rules

In My Rules library, the bulk actions have been extended so that rules can now be enabled in bulk to speed up the rule implementation process.

1.7 Extended Rule Clone Action

When cloning a custom rule, all rule attributes can be modified during the clone action and SQL rules can now be cloned.

1.8 Outlier Detection Improvements

• Results refinement: Additional filtering of outliers detected has been applied to reduce the number of false positives returned and the default threshold/confidence level is set to 0.99.

• Filter fixes: The outlier counts when viewing results are also now updated to reflect the filters applied.

• Performance: Improved performance of outlier detection results loading.

1.9 Search Capability for Data Previews

Enabled search capability for multiple data preview popup grids to include Data Source View previews, Data Source View extractions and SQL rule test outputs.

1.10 User Interface Enhancements

Several UI enhancements were made throughout ADQ:

• Back button consistent across execution layouts: Input Data Extractions, Workflow Tests, Rule Execution Group Executions and Data Source View Extractions.

• Reusable Rules library updates: wrap function for columns, search bars added for each column, and the DQ dimension is shown as a colour-coded pill.

• The data source view name is displayed at the top of the data preview pop-up grid.

• Multi-column sorting available in Data Issues.

• In My Rules library, rules can be filtered by Data Source View and Type.

• Scroll position is retained in My Rules after viewing/editing a rule.

• Input Data Extractions and Workflow Tests are accessible via the left hand side main menu.

• Search fields added to rule execution group creation wizard and to the column selection boxes when creating views and defining rule inputs.

• Removed the auto-closing of the rule promotion successful notification.

• Aligned the view layout of data source connections with other layouts.

• Multiple columns can be selected at once when adding/removing columns to display in My Rules and Data Issues.

• When uploading/downloading FlowDesigner rule projects, the Description field is now renamed to Change Comments to users to document any changes made to the rule logic inside the FlowDesigner project.

• New components added for configuring rule metadata attributes such as the rule threshold and rule weight.

• The allowlist expiry date can be configured via a calendar component as opposed to the user needing to specify the expiry date via the free text Comment field.

• Toggle added to the Data Issues by DQ Rule Screen to hide/display rules which have produced zero data issues, i.e. no offending records.

• Data issues assignee displayed using a coloured avatar.

• The break field and primary key icons in Data Issues are shown at the beginning of the column header so the user can easily identify their break field and primary key columns.

• Helper text added alongside the schema field when creating a data source connection.

1.11 Table Names Stored for SQL Query Data Source Views

SQL query database data source views can optionally have a table name specified to be used only for reporting purposes.

1.12 Limited Rules in Advanced Insights

When drilling down from a data source in Advanced Insights to the rule level, rules are only shown for that data source if it has a rule input field with both a primary key and a break field defined for that data source.

1.12 Automated Client Upgrade Improvements

The following updates were made to the automated SSDQ client upgrade:

• Enable data source sets in correct solutions for all client upgrades not just from SSDQ v2023.02.08.

• Reduce constraints of data source sets upgrade.

• Transfer client level data source sets to new client.

• Allow for old rule project input file names to be taken from metrics producer solution(s) rather than projects.

• Remote File Stores Upgrade.

• Transfer values of existing client level properties to new client.

• Convert rule attribute default_assigned_to to data_issues_assignee in insights-filters.

• Map reusable rule item names to the new values and capitalise the rule dimensions.

• Creates rule execution groups.

• Search field added to the rule execution group wizard.

1.12 Advanced Insights Drilldown Permissions

Added permissions to check if the user has access to drilldown between the Advanced Insights statistics and Data Issues to see the offending data records.

1.13 Improved Rule Execution Results Display

Improved the display of rule execution group execution results by having a dedicated side panel menu item for Executions and displaying the execution status of individual rules within an execution group. Users can also view the rule result and count CSV file outputs of individual rules.

1.14 Reduced Data Source View Edit Restrictions

Reduced the restrictions of editing data source views by allowing columns to be removed from the views if they are not in use.

1.15 Allow Empty Row Stripe and Row Limit For Data Source View Extractions

When performing a data source view preview or extraction, the row stripe and row limit values can be left empty allowing the user to extract all data rows from their data source. The default option now is to extract all data as opposed to a sample.

1.16 Rule Status Logic Updated

Within DQ Metrics of the Insights Hub the logic used to calculate rule status has been updated. The new method is:

• If the DQ score is greater than or equal to the rule threshold, the rule status is PASS .

• If the DQ Score is less than the rule threshold, the rule status is FAIL .

The rule threshold is a value between 0 and 100 specifying the percentage of records which must pass the DQ rule check in order for the rule to flagged as passed. It is defined for each DQ rule via the Rule Creation Wizard.

The DQ score for each rule is calculated by number of failed records ÷ number of checked records multiplied by 100.

There is also a new column passed_count_percent which can be displayed in DQ Metrics if desired.

1.17 Save & Minimise Data Previews

Implemented a new Windows feature which allows users to minimise pop-up grids containing data source view extractions, SQL rule test results, and rule result outputs, and reopen them at a later stage.

1.18 Data Source Connection Usage Feature

The Usage feature has been extended to allow users to review all data source views, DQ rules and profiling jobs which use data from a particular data source.

1.19 Changed Authentication Redirect URI

Changed the authentication redirect URI from /authentication-token to /authenticationtoken-adq. The external security configuration in Data Quality Manager and ADQ Server's configuration file authentication.properties must be updated to reflect this change.

1.20 Change History Tables Housekeeping

DQM housekeeping solutions have been added to remove entries older than 365 days from DQM database tables custom_tables_history and data_quality_solutions_history.

1.21 Support for HTTP

Added property apache-tomcat.https.is-enabled to the configuration file apachetomcat.properties to allow use of either http or https.

1.22 Vaadin Upgrade

Vaadin has been upgraded to v24.5.4.

2 Defects Fixed

2.1 Quoted characters caused record remediation failures

When remediating records in Data Issues, the record update failed if there were quoted characters in the data.

2.2 Carriage returns not displayed on Remediate Data Issue screen

Carriage returns were displaying as spaces on the Remediate Data Issue screen. These are now displayed as a new line character.

2.3 Outlier detection counts not updating when filters applied

When threshold filters were applied when viewing the outlier detection results, the graphs and count badges were not updated to reflect the filtered number of outliers.

2.4 Possible to add more than one rule input for a reusable rule

When editing a reusable rule, it was possible to add more than one rule input field. This should not be allowed as reusable rules can currently only accept one input.

2.5 Error when running an execution group with no enabled rules

It was possible to schedule and manually trigger an execution group with no enabled rules which results in an unintuitive error message being produced. The option to run an execution group with no enabled rules has now been removed.

2.6 Spaces in Excel worksheet names results in an error when running profiles and rules

Spaces are now allowed in Excel worksheet names.

2.7 DuckDB lib files being stored in the Windows Temp directory

The DQ workflow has been updated to remove any DuckDB lib files from the C:/Windows/Temp directory to avoid the disk space filling up. These files were being written by DuckDB during the DQ workflow.

3 Known Defects

3.1 Database data source view exports convert decimals past a certain length to scientific notation

When retrieving the rule input from a database data source view created using the SQL builder or the SQL query functionality, if the data column type is a float where the decimal numbers are past a certain length (greater than 10,000,000 or less than 0.001) the JDBC connector exporting the data converts this decimal number into scientific notation.

Workaround

Use a SQL query rule to perform the check rather than a custom FlowDesigner project rule. SQL query rules perform the rule logic directly on the source database so do not need a data source view defined. The SQL query should include a CAST in the query to ensure the data is not output in scientific notation, e.g. CAST(account_balance AS DECIMAL(20, 2)). Note you will need to know the number of digits after the decimal point.

3.2 Search attributes page displayed when cloning reusable rule

When cloning a reusable rule, the Search Attributes page is displayed as part of the rule clone wizard. This page should be hidden from the user. If the user enters any configuration in the Search field it will not be honoured by the rule.

Open

Workaround

Ignore the Search Attributes page when cloning a reusable rule.

3.3 Blank column remains in profile configuration

If a profiling configuration is initially created with a subset of the profiling options, i.e. not rule suggestion, outlier detection and profiling, a blank column is added to the Columns field. If the user then modifies this profiling configuration to enable another one of the previously disabled profiling options (rule suggestion, outlier detection or profiling) and they do not untick the selection of the empty column in the Columns field, then the data extraction will fail causing the newly enabled profiling option to fail.

See below the blank column selected at the top of the column list dropdown.

Open

4 Known Limitations

4.1 Rule Dictionary has a maximum limit of 45 columns

A rule dictionary of greater than 45 columns can be uploaded to DQM, but the workflow will fail if it exceeds 45 columns.

4.2 Rule Results files are limited to 30 columns

The rule is able to export more than 30 columns, but the workflow will fail if the results exceed 30 columns. This includes the row id, rule id and data owner columns. The rule input does not have a limit of columns.

4.3 FlowDesigner string length limitation

FlowDesigner has a maximum capacity of 32,000 bytes per string length for each row.

4.4 Disallowed rule result file column names

The following columns (case insensitive) are not permitted to be included in the rule result file produced by the rule as these column names are reserved for components of the DQ workflow:

1. autonumber

2. state

3. table

4.5 Rule result cannot have more than six primary keys

Rule result file can have up to six columns as a compound key for it to be successfully processed.

4.6 Solidatus integration only updates rules using SQL builder views without quoted identifiers

The Solidatus integration solution can currently only update rules whose input is from a SQL builder view and where this view is extracted without quoted identifiers, i.e. the tick box on the View Creation wizard ‘Do use quoted identifiers during extraction' is unticked.

5 Notices

5.1 Optimised DQ Workflow Limitations

1. Record level auto-assignment

The workflow will not automatically assign data issues/broken records to users at the row level. This means if the user populates the data_owner field in the FlowDesigner rule project or SQL rule with the username or user group name of the individual or team who is responsible for remediating these records, the workflow will not assign the data issues to these users. The records will be automatically assigned to the user or user group specified as the Data Issues Assignee for that DQ rule.

To have the data issues records assigned to different users or groups, the user will need to do this manually or via a bulk assignment script in Data Issues.

The auto-assignment of records to the Default Assignee only occurs for new records detected by the DQ rule.

For example, if Record_1 and Record_2 were detected by the DQ rule Rule_1 on Monday at 09:00 where the data_owner was set to User_1 and User_2 respectively for each record, the DQ workflow would automatically assign these records to Rule_1’s default Data Issues Assignee, User_3. This means in Data Issues Record_1 and Record_2 will both be assigned to User_3.

Then in Data Issues User_4 assigns Record_1 to User_1 and Record_2 to User_2. If Record_1 and Record_2 are detected as data issues by DQ Rule_1 again by the next scheduled run of the execution group at Tuesday at 09:00, and Record_3 has been detected as a new rule break. Record_1 and Record_2 will remain assigned to User_1 and User_2 respectively, and Record_3 will be assigned to User_3.

2. Specifying the winning record in the FlowDesigner rule project

When building DQ rule logic in FlowDesigner users were able to specify which record in the rule result file was the winning record. One purpose of this feature was for matching or deduplication rules where the user may have wanted to suggest which of the candidate match records was the “best” record or match. The best match would then be the winning record in Data Issues to assist the Data Steward during remediation. Another was for data enrichment, where users could extend their FlowDesigner rule logic to cleanse and transform the records which failed the DQ check so they conform to the expected format or value. The corrected or enriched records would be the winning records in Data Issues where the Data Steward can download the already remediated records and make the relevant updates in the source system.

3. join-and-filter and custom-solutions custom tables

The join-and-filter and custom-solutions custom tables have been superseded by composite data source views meaning any data transformations defined here must be migrated to composite data source view FlowDesigner projects. Alternatively, if all input data for the join-and-filter and custom-solutions transformations is extracted from the one database, they can be migrated to SQL query views.

4. Data source change affecting allowlist records

There existed a property ssdq.data-source-change-affect-allowlist in the ssdq-settings property set which dictated whether records should be unallowlisted, i.e. moved back to the main dataset/list of records for remediation, if the source data changed for the offending record/DQ break.

In ADQ v1.6.0, allowlisted records are only unallowlisted if the allowlist expiry date has been met or if the user manually updates the Remediation Category of the record to any value except ALLLOWLIST.

5. Record type column in Data Issues/DQC

The Record Type column in Data Issues/DQC has now been removed. This column held one of the following values: OLD, OLD-CHANGED or NEW. It informed the user if the record was an old broken record, i.e. identified as a DQ break by a previous execution of the rule, an old broken record where the source data has changed since the previous execution of the rule, or a new broken record, i.e. a record which was not identified as a DQ break by the previous execution of the rule.

Users can now obtain break age metrics from Advanced Insights which informs the user of how many days it took to remediate the broken record (break age) and how times that record failed the DQ check following remediation (reoccurrence time).

6. AWS and Azure Data Extraction

AWS s3 bucket and Azure data lake data extractions are no longer supported. Any data extractions defined in the custom tables aws-files-extract and azure-ADLS2-files-extractwill not be executed meaning any rules using these sources will not run.

7. DQ results database tables aggregated_results_last_execution_horizontal and data_scoring_rules_last_execution_horizontal not populated

The following database tables in the DQ results database are no longer populated:

• aggregated_results_last_execution_horizontal

• data_scoring_rules_last_execution_horizontal

The data stored previously stored in aggregated_results_last_execution_horizontal can be obtained from the aggregated_results table.

5.2 Deprecating DQC Collector

The DQC collector can be uninstalled following a successful upgrade to ADQ v1.6.0. Before uninstalling the collector, any previous versions of the SSDQ client (older than version 2025.07.03) must be archived and deleted from the DQM SYSTEM client. Alternatively, all old SSDQ clients must have the DQC collector trigger disabled from solution dqc-results-scanner.

5.3 Migrating SQL Join & Filters to Composite Views

As mentioned above, any data extractions defined in the join-and-filter and custom-solutions custom tables must be migrated to composite data source views or SQL query views.

5.4 Collections created outside of DQ workflow

The collections per rule to house the DQ breaks in the Data Issues database (DQC) get created outside of the DQ workflow.

• For SQL rules, the collection is created when the SQL rule is created.

• For reusable rules, the collection is created when the reusable rule is added to My Rules library.

• For custom rules, the collection is created when the user uploads the FlowDesigner rule project via ADQ. However, if the FlowDesigner rule project is deployed directly from FlowDesigner to DQM (deploy to server option) then the rule’s collection will need manually created. To do this, click on Create Collection in the action options for the rule.

Open

Collections can also be created in bulk by selecting the relevant rules in My Rules library, selecting the Bulk action button in the bottom right hand corner and choosing Create Collections.

Open

5.5 PingFederate and Kerberos Authentication Behaviour

Scenario Overview

In environments making use of ADQ with PingFederate and Kerberos for authentication, users may encounter a specific behaviour:

Regardless of the username entered, a user can log in to ADQ if their machine user is a member of the appropriate groups. This means that authentication is based on the machine user rather than the individual’s entered username.

Technical Details

• PingFederate Token Handling: This behaviour is partly due to how PingFederate processes and honours authentication tokens.

• Auth Model Configuration: Our current authentication model requests a login for each session. However, PingFederate’s token handling mechanism bypasses this by validating the machine user’s group membership.

Implications for Users

• Group Membership: Ensure that only authorized machine users are members of the relevant groups, as their membership will govern access to ADQ.

• Username Entry: The entered username during login will not affect the authentication outcome if the machine user is appropriately grouped.

Recommendations

• Review and manage group memberships to maintain secure access controls.

• Inform users about this behaviour to set correct expectations during the login process.