v1.8.1 - ADQ Release Notes

Status: GA

ADQ v1.8.1 focuses on improving user experience. More control, better flow - drafts, flexible settings, and smarter time orchestration.

1 New Features

1.1 Data Quality Issues UI improvements

Multiple enhancements have been made to the Data Issues screen in ADQ, these include:

1. Option to remove resolved issues from the total issue count on the DQ Issues

2. Ability to change the default value for the records filter

3. Ability to change the default value of the “Data Issues by Record” toggle.

4. A new permission to give visibility of the “Show rules with zero issues” toggle to non-administrator users.

5. All Break Field column names displayed

1.2 DQ Issues Bulk Remediation enhancements

The following enhancements have been made to Bulk Remediation:

• Can set value of one field to another

• Can save script without exiting

• Can set value to null

1.3 Execution Orchestration Controls

Changes have been made to orchestration controls, allowing for smoother scheduling and execution of execution groups

• The ability to set one or more execution groups to run when another group finishes, with the option to set them to run when the first group has completed or only when it is successful.

• The ability to prevent concurrent runs - If 2 groups attempt to run at the same time, one will enter a “Queued” state until the first group has finished

1.4 Trigger custom solutions from the ADQ UI

To democratize the use of Datactics products, custom solutions can now be triggered from ADQ, removing the need for DQM access to run them. This allows non-admin users to now trigger custom solutions. Users must have the Custom Solution role attached to their user to access this tab.

These solutions can be run at anytime, and can also be set up on a schedule.

1.5 Manage incomplete configurations

Users now have the ability to save incomplete configurations as drafts, this functionality is available when:

1. Creating Connections

2. Creating Views

3. Creating Profiles

4. Creating Rules

These wizards are also now non-sequential, so users can complete pages without having to complete the previous pages first (this also applies to the Access Control screen).

1.6 Dialog Views added

“Viewing” certain things within ADQ now opens a sidebar dialog box, these are present in:

• Connections

• Views

• Profiling

• My Rules

1.7 New date based reusable rules added

Rules regarding dates have been added to the reusable rules library and suggested by Rule Suggestion.

1.8 UI & Usability Improvements

Multiple changes have been made to the UI of ADQ and the DQM executions screen, allowing more configurability and easier navigation. These include:

• Ability to change the system defaults for the rule execution toggles in ADQ.
  This will allow certain operations, such as the rule input duplicate primary key check, to be disabled by default

• Redesign of the DQM Executions screens, improving interactivity.
  New features include a search functionality on execution item graphs and the ability to display DQM executions in a tree grid

• New error message alerting users if they attempt to promote rules to an environment that doesn’t have the connection set up yet.

• “Delete Collection” option added to the “My Rules” screen in ADQ.
  This is only useable if the user has permissions to modify the rule.

• “Data Source Group” shown by default in Connections

• Object type column when exploring/can specify table types on explore in Connections.

• Profiles can now be cancelled from the Data Profiles page.

• Drilldowns available in Data Profiles using the magnifying glass button.

• Split panel for filters in My Rule page.

• Ability to select multiple Rule IDs in My Rules page.

• Can cancel Worker executions in My Rules Page.

• Can select a Workflow Test row limit.

• New Execution Group properties layout in My Rules.
  
  

1.8 Distributed Architecture Improvements

DQM Workers now replace DQM Slaves, stabilising the syncing of Master/Worker DQMs.

This allows support for XiPD, and allows all executions to be viewed in the Master DQM.

1.9 Can add encrypted properties to JDBC URLs of Connections

This functionality allows you to specify encrypted properties in ADQ when creating the connection e.g. a key or bearer token. You can then add them into the URL in the same wizard

1.10 Accessibility improvements

ADQ has underwent some aesthetic updates in the aim of better accessibility.

2 Defects Fixed

2.1 Large numbers in rule descriptions are being displayed in scientific notation

2.2 Reusable rule 'column_'A'cannot_be_an_element_of_set''X''' cannot be applied

2.3 Query to delete temp-solutions non functional with postgres

2.4 When editing a view if required db object is not present view becomes inaccessible

2.5 Connection explore layout empty when no tables present

2.6 ADQ unable to load rule suggestion results if additional info field contains regex

2.7 Rule parameters are not promoted

2.8 Rule promotion corrupts data source view column ordinal order

2.9 Results upload solution fails when no rules set to upload

2.10 script-rule-preparation creates SQL script files incorrectly for long queries

2.11 Do not attempt to authenticate LDAP user password in DQC on login to ADQ

2.12 New rules added to execution group not inheriting overall configuration

2.13 No error message if a user attempts to create a connection with the same name as an existing one that they do not have access to

3 Known Defects

3.1 Database data source view exports convert decimals past a certain length to scientific notation

When retrieving the rule input from a database data source view created using the SQL builder or the SQL query functionality, if the data column type is a float where the decimal numbers are past a certain length (greater than 10,000,000 or less than 0.001) the JDBC connector exporting the data converts this decimal number into scientific notation.

Workaround

Use a SQL query rule to perform the check rather than a custom FlowDesigner project rule. SQL query rules perform the rule logic directly on the source database so do not need a data source view defined. The SQL query should include a CAST in the query to ensure the data is not output in scientific notation, e.g. CAST(account_balance AS DECIMAL(20, 2)). Note you will need to know the number of digits after the decimal point.

3.2 Scheduled rule group execution fails when no rules are enabled

If there is a rule execution group which is scheduled to run but there are no enabled rules in the group, the execution will fail.

Workaround

None.

3.3 Solutions showing as running when they have completed

DQM may sometimes show solutions as “Running” when they have completed.

4 Known Limitations

4.1 Automated upgrade limitations for updating user roles and permissions

For the new User Management area in ADQ, the automated upgrade will not fully upgrade all user management and access control entities as it is impossible to do a 1:1 mapping of all permissions and roles as they existed in previous releases to the formats required for ADQ.

Please review the manual steps outlined in the User Management Upgrade section of the ADQ v1.8.1 Deployment Guide to ensure all users, roles, permissions and data access is configured correctly following the upgrade.

4.2 Optimised DQ Workflow Limitations from ADQ v1.6.0

1. Record level auto-assignment

The workflow will not automatically assign data issues/broken records to users at the row level. This means if the user populates the data_owner field in the FlowDesigner rule project or SQL rule with the username or user group name of the individual or team who is responsible for remediating these records, the workflow will not assign the data issues to these users. The records will be automatically assigned to the user or user group specified as the Data Issues Assignee for that DQ rule.

To have the data issues records assigned to different users or groups, the user will need to do this manually or via a bulk assignment script in Data Issues.

The auto-assignment of records to the Default Assignee only occurs for new records detected by the DQ rule.

For example, if Record_1 and Record_2 were detected by the DQ rule Rule_1 on Monday at 09:00 where the data_owner was set to User_1 and User_2 respectively for each record, the DQ workflow would automatically assign these records to Rule_1’s default Data Issues Assignee, User_3. This means in Data Issues Record_1 and Record_2 will both be assigned to User_3.

Then in Data Issues User_4 assigns Record_1 to User_1 and Record_2 to User_2. If Record_1 and Record_2 are detected as data issues by DQ Rule_1 again by the next scheduled run of the execution group at Tuesday at 09:00, and Record_3 has been detected as a new rule break. Record_1 and Record_2 will remain assigned to User_1 and User_2 respectively, and Record_3 will be assigned to User_3.

2. Specifying the winning record in the FlowDesigner rule project

When building DQ rule logic in FlowDesigner users were able to specify which record in the rule result file was the winning record. One purpose of this feature was for matching or deduplication rules where the user may have wanted to suggest which of the candidate match records was the “best” record or match. The best match would then be the winning record in Data Issues to assist the Data Steward during remediation. Another was for data enrichment, where users could extend their FlowDesigner rule logic to cleanse and transform the records which failed the DQ check so they conform to the expected format or value. The corrected or enriched records would be the winning records in Data Issues where the Data Steward can download the already remediated records and make the relevant updates in the source system.

3. join-and-filter and custom-solutions custom tables

The join-and-filter and custom-solutions custom tables have been superseded by composite data source views meaning any data transformations defined here must be migrated to composite data source view FlowDesigner projects. Alternatively, if all input data for the join-and-filter and custom-solutions transformations is extracted from the one database, they can be migrated to SQL query views.

4. Data source change affecting allowlist records

There existed a property ssdq.data-source-change-affect-allowlist in the ssdq-settings property set which dictated whether records should be unallowlisted, i.e. moved back to the main dataset/list of records for remediation, if the source data changed for the offending record/DQ break.

In ADQ v1.6.0, allowlisted records are only unallowlisted if the allowlist expiry date has been met or if the user manually updates the Remediation Category of the record to any value except ALLOWLIST.

5. Record type column in Data Issues/DQC

The Record Type column in Data Issues/DQC has now been removed. This column held one of the following values: OLD, OLD-CHANGED or NEW. It informed the user if the record was an old broken record, i.e. identified as a DQ break by a previous execution of the rule, an old broken record where the source data has changed since the previous execution of the rule, or a new broken record, i.e. a record which was not identified as a DQ break by the previous execution of the rule.

Users can now obtain break age metrics from Advanced Insights which informs the user of how many days it took to remediate the broken record (break age) and how times that record failed the DQ check following remediation (reoccurrence time).

6. AWS and Azure Data Extraction

AWS s3 bucket and Azure data lake data extractions are no longer supported. Any data extractions defined in the custom tables aws-files-extract and azure-ADLS2-files-extractwill not be executed meaning any rules using these sources will not run.

7. DQ results database tables aggregated_results_last_execution_horizontal and data_scoring_rules_last_execution_horizontal not populated

The database table data_scoring_rules_last_execution_horizontal in the DQ results database is no longer populated.

4.3 Rule Dictionary has a maximum limit of 45 columns

A rule dictionary of greater than 45 columns can be uploaded to DQM, but the workflow will fail if it exceeds 45 columns.

4.4 Rule Results files are limited to 30 columns

The rule is able to export more than 30 columns, but the workflow will fail if the results exceed 30 columns. This includes the row id, rule id and data owner columns. The rule input does not have a limit of columns.

4.5 FlowDesigner string length limitation

FlowDesigner has a maximum capacity of 32,000 bytes per string length for each row.

4.6 Disallowed rule result file column names

The following columns (case insensitive) are not permitted to be included in the rule result file produced by the rule as these column names are reserved for components of the DQ workflow:

1. autonumber

2. state

3. table

4.7 Rule result cannot have more than six primary keys

Rule result file can have up to six columns as a compound key for it to be successfully processed.

4.8 Solidatus integration only updates rules using SQL builder views without quoted identifiers

The Solidatus integration solution can currently only update rules whose input is from a SQL builder view and where this view is extracted without quoted identifiers, i.e. the tick box on the View Creation wizard ‘Do use quoted identifiers during extraction' is unticked.

5 Notices

5.1 Deprecating DQC Collector

The DQC collector can be uninstalled following a successful upgrade to ADQ v1.6.0. Before uninstalling the collector, any previous versions of the SSDQ client (older than version 2025.07.03) must be archived and deleted from the DQM SYSTEM client. Alternatively, all old SSDQ clients must have the DQC collector trigger disabled from solution dqc-results-scanner.

5.2 PingFederate and Kerberos Authentication Behaviour

Scenario Overview

In environments making use of ADQ with PingFederate and Kerberos for authentication, users may encounter a specific behaviour:

Regardless of the username entered, a user can log in to ADQ if their machine user is a member of the appropriate groups. This means that authentication is based on the machine user rather than the individual’s entered username.

Technical Details

• PingFederate Token Handling: This behaviour is partly due to how PingFederate processes and honours authentication tokens.

• Auth Model Configuration: Our current authentication model requests a login for each session. However, PingFederate’s token handling mechanism bypasses this by validating the machine user’s group membership.

Implications for Users

• Group Membership: Ensure that only authorized machine users are members of the relevant groups, as their membership will govern access to ADQ.

• Username Entry: The entered username during login will not affect the authentication outcome if the machine user is appropriately grouped.

Recommendations

• Review and manage group memberships to maintain secure access controls.

• Inform users about this behaviour to set correct expectations during the login process.