v1.5.0 - ADQ Release Notes
Status: GA
ADQ v1.5 makes the ADQ platform more user-friendly, efficient, and powerful, enabling users to manage data quality with greater ease and confidence.
1 New Features
1.1 Richer Profiling Metrics
New profiling metrics, including data, frequency and patterns, have been introduced providing deeper insights into data characteristics for better analysis.
1.2 Enhanced Visualisation for Outlier Detection Results
Improved graphical representation of outlier detection results in the profiling section making it easier to identify and interpret data anomalies.
1.3 Data Source View Preview
A data preview function has been introduced to assist in selecting columns and defining SQL queries for data source views to help users make more informed decisions when configuring views. The preview function helps users validate that the data being extracted is correct.
1.4 Extended Reusable Rules Functionality
Added parameterised reusable rules, with suggested values from ML profiling, and the ability to manually add all reusable rules to the user’s library streamlining rule creation and allowing for greater customisation of reusable data quality rules.
1.5 Rule Authoring Improvements
Several updates have been made to simplify and enhance rule creation and management.
Primary key values are now prepopulated from the data source view primary key.
Tooltip for rule threshold has been updated.
The "Sources" attribute in My Rules is now "Source Count" to clarify that this metric shows how many sources are input to the data quality rule.
1.6 GenAI for SQL Rule Generation
Introduced LLM models to automatically generate SQL code for a SQL script rule from free text instructions provided by the user in a chat box layout. This accelerates rule creation and makes it accessible even for non-technical users.
1.7 Bulk Issue Remediation
Added the ability to create, save and execute bulk remediation scripts to resolve multiple data issues at once speeding up the remediation process, and improving efficiency.
1.8 Bulk Issue Assignment
Users can now assign multiple data quality issues to stewards in bulk to streamline issue management and accelerate resolution.
1.9 Data Stewards View Restriction
Data stewards now only see issues assigned to them or their user group in attempt to honour data access controls, and improve focus and task management by limiting the view to relevant issues.
1.10 Data Issues Export
Enabled the ability to download offending records from Data Issues to be ingested downstream, or to export remediated records for update at source.
1.11 Enhanced Insights Dashboard Configuration
Users can now customise Insights dashboards via a dashboard edit layout as opposed to through a wizard. New charts, graphs, filters and visuals have been added allowing for more tailored dashboarding to report on key data quality and remediation metrics.
1.12 Advanced Insights Drill-Down for All Source Types
Drill-down capabilities are now available across all data source types, allowing users to analyse data issues down to specific records. This provides deeper visibility into data quality issues, enabling more effective remediation.
1.13 UI Standardization and Improvements
Continued enhancements to UI consistency, including standardised buttons, terminology, and user-friendly tooltips.
1.14 Extended Validation Solutions
Validation checks have been extended to ensure key columns in custom tables are populated correctly after an upgrade to ensure a smoother transition during the upgrade process.
1.15 Data Source Connection Setup Improvements
Default values and dropdowns have been added for database connection configuration to simplify the setup process, reducing configuration errors and saving time.
1.16 New ADQ Installation and Upgrade Command
Introduced an interactive command-line tool to guide users through ADQ server installation and upgrade configurations to streamline the deployment process, ensuring all necessary settings are correctly configured.
1.17 DQM Deployment Enhancements
Improved the DQM installation and upgrade processes with better error handling during upgrades and performance enhancements when uploading clients and starting the service. This enhances reliability and reduces downtime during system updates.
1.18 Extended SSDQ Automated Upgrade
The upgrade process now includes automatic updates for rule input file names and corresponding projects to significantly reduce manual effort during upgrades.
1.19 Snowflake Driver Included Out-of-the-Box
The Snowflake driver is now bundled with the DQM toolsets, eliminating the need for manual installation. This facilitates quicker setup and integration with Snowflake data sources.
1.20 Improved User Login Experience
The login redirect for LDAP-authenticated users has been removed which provides a faster and more seamless login experience.
1.21 Support for Postgres Databases
ADQ can now be installed using Postgres databases for DQM, DQC, and SSDQ results expanding deployment options and compatibility with existing infrastructure.
1.22 ISO27001 Certification Achieved
Accredited ISO27001 certification to formalise data security practices enhancing trust and compliance with industry standards.
2 Defects Fixed
2.1 Cannot create a SQL rule
SQL rules could not be created due to the Data Issue Assignee not being selectable from the dropdown. The dropdown now correctly lists users and data source access groups based on the sources used by the rule.
2.2 No housekeeping being performed on workflow test solutions
Temporary solutions cloned for rule workflow tests were accumulating and not being deleted, leading to slow service start-up and client upload times. The workflow test housekeeping now deletes these temporary solutions older than a set number of days.
2.3 Ampersand (&) in the reusable project name leads to loading failures in ADQ
Projects with an ampersand in their name failed to load and caused other reusable projects to not display. Addressed the issue in project loading to correctly handle names with ampersands.
2.4 Break age table not being populated when rule IDs contained hyphens
The break age table was not being populated as expected if the rule ID contained a hyphen. The DQ workflow has been updated to handle hyphens in rule ID and ensure proper population of the break age table.
2.5 Incorrect batch execution status when a SQL rule fails
Execution group execution was marked as successful even when a SQL rule failed. The status now correctly reflects the failure of any SQL rule.
2.6 Error file all-failed-rules.txt displayed in ADQ is empty when SQL rule fails
Corrected issue where the all-failed-rules.txt file was empty when a SQL rule failed to ensure it lists the failed rules upon failure.
2.7 DQC Upload fails when a hyphen (-) is present in the rule description
Updated the process to handle hyphens in rule descriptions to avoid DQC upload failure.
2.8 Notify the user they can only upload CSV files as a CSV source
No error message was shown when a non-CSV file was uploaded as a CSV source, causing user confusion. Added a notification to inform users that only files with a .csv extension are allowed.
2.9 Error message cut off on Executions screen
Error messages were cut off on the Executions screen, preventing users from seeing the full message. Updated the display to ensure error messages wrap correctly and are fully visible.
2.10 Profiling data preview not showing first header
The first header name in the profiling data preview was not displayed correctly. Fixed the issue to ensure the first header name is displayed as expected.
2.11 Custom solution data extraction failure
Changes in custom table structures were not addressed, causing data extraction failures for custom solutions. Updated the structure of custom tables to match the new requirements, ensuring successful data extraction.
2.12 create_csv_iterator for t46 has incorrect node configuration for quote character and separator
Incorrect configuration for quote characters and separators in create_csv_iterator was causing issues with join and filter functionality. Configurations have been corrected to use comma-separated and double-quoted settings.
2.13 No error message returned if reusable rule project iteration file specified incorrectly/not found
No error message was provided when the iteration file for a reusable rule was specified incorrectly. Error messages are now returned to the user in such cases, allowing for easier troubleshooting.
2.14 Rule Suggestion rules fail to run SSDQ process when the break field is included as a primary key field
SSDQ workflow failed when the Break Field was a Primary Key or part of a Primary Key. Updated the process to handle such scenarios correctly.
2.15 Incorrect password used for item in create-if-service-exists in solution dqc-results-uploader-preparation
Incorrect password was used, causing failures in the create-if-service-exists item. The correct superuser password is now used, preventing such failures.
2.16 Rule Suggestion pandas version not supporting .append
The pandas version did not support the .append method, causing issues in Rule Suggestion therefore adjusted the Rule Suggestion process to work with the updated pandas version.
2.17 Ensure all validation error files are returned to the user in ADQ
Several validation error files were not returned to users, hindering debugging. All relevant validation error files are now returned to the user for better debugging support.
2.18 Rule promotion process does not update data-source-names custom table
The custom table data-source-names was not updated during rule promotion, potentially hiding data sources and promoted rules. The rule-promotion process now correctly updates the data-source-names table.
2.19 Validation failing when at least one rule has the incorrect rule export structure
The validation failed when any rule had an incorrect rule export structure, stopping the process. The validation now notes the issue and continues processing other rules.
2.20 pi.server properties require update in advanced insights
Incorrect URL configuration in advanced insights due to pi.server properties including the port. Removed the unnecessary port from URLs in the solution.
2.21 Make project node statistics retry count configurable
Retry attempts for project node statistics were not configurable. The retry count is now configurable to provide more flexibility and reduce the chance of failure during execution.
2.22 More than 1000 characters in the rule fields columns causes a failure in filter-file-on-rule-id solution
The filter-file-on-rule-id solution failed when rule fields contained more than 1000 characters. Updated the solution to handle up to 8000 characters, preventing the failure.
2.23 Inaccurate data issues metrics displayed
The number of DQ issues was incorrectly displayed as triple in Data Issues. The display now shows the correct number of DQ issues.
2.24 Empty collection causes DQ workflow failure
SSDQ workflow failed when attempting to export a dataset from a collection with no datasets. Implemented a check to delete empty collections before running the process.
2.25 dqc-results-swap solution fixes
The dqc-results-swap solution had issues with string truncation and missing functionality. Increased allowed length for the comment field and enabled use-file-database option.
2.26 Iterator Rules Preparation removes \ from \d+, breaking RegEx
The iterator rules preparation solution incorrectly removed \ from \d+, breaking the regex. Updated the solution to preserve backslashes in regex patterns.
2.27 Enhance validation solutions in case of rule count file not being produced
Validation failed if the rule count file was not produced. Improved validation to handle cases where the rule count file is missing.
2.28 rollback-check solution treats excluded rules as successful rules
Excluded rules were treated as successful during the rollback check. Updated the process to correctly identify and handle excluded rules.
2.29 validate_count_values project producing incorrect results
Incorrect results were returned due to an improperly scoped regex. Updated the regex in t7 of project validate_count_values to match only the beginning and end of the string: ^${dqm.csv.iteration.field.0}$
2.30 Carriage return character in rule results file from SQL rules causes DQ workflow failures
SQL rules failed when carriage return characters were present in the results data, causing formatting issues. Ensured that carriage return characters do not cause formatting issues, allowing the process to complete successfully.
2.31 Not possible to use a CSV source when creating an SQL rule
A CSV source could be selected when creating an SQL rule, leading to failures as SQL rules cannot run on CSV sources therefore removed the option to select a CSV source in the SQL rule creation wizard.
2.32 Encrypted properties displayed
Encrypted values are not displayed in item arguments or artifacts.
2.33 Missing rapiddat.rls file error when opening FlowDesigner project
When opening a project in FlowDesigner that was originally built with a FlowDesigner version prior to v11.0.0 and using a transformation library in it’s own project directory, i.e. not the default library packaged with FlowDesigner, an error messaging stating that rapiddat.rls is missing appears. The rapiddat.rls file is now copied across to the transform library used by the project to avoid this error.
3 Known Defects
3.1 Database data source view exports convert decimals past a certain length to scientific notation
When retrieving the rule input from a database data source view created using the SQL builder or the SQL query functionality, if the data column type is a float where the decimal numbers are past a certain length (greater than 10,000,000 or less than 0.001) the JDBC connector exporting the data converts this decimal number into scientific notation.
Workaround
Use a SQL query rule to perform the check rather than a custom FlowDesigner project rule. SQL query rules perform the rule logic directly on the source database so do not need a data source view defined. The SQL query should include a CAST in the query to ensure the data is not output in scientific notation, e.g. CAST(account_balance AS DECIMAL(20, 2)). Note you will need to know the number of digits after the decimal point.
3.2 Validation solutions do not correctly validate do-ignore-failures
The validation solutions do not correctly validate if the column DO_IGNORE_FAILURES in the custom table database-sources is set to true or false. The user can input any value (except null) in this column and the validation solution will deem it is correct.
Workaround
None.
3.3 Cannot extract columns which contain commas in the column header name
Source data columns which contain commas in the column header name cannot be processed by the DQ workflow.
These types of columns are not permitted to be selected when creating a SQL builder view and will cause the data extraction for SQL query views to fail if an alias is not used.
Workaround
For database sources create a SQL query view using an alias for any columns containing a comma in the name for the data extraction.
For CSV or Excel sources, the column header name must be renamed.
Conditions for aliases and new column names are listed below:
Can only contain letters, digits, underscores and hyphens,
Must start with a letter or underscore,
Must be no more than 256 characters long
3.4 Solidatus integration solution failure when updating model
The Solidatus integration solution fails when attempting to update the Solidatus model via REST API due to the substitution property for the API token, ${solidatus.api-token}, not expanding.
Workaround
Option 1:
In SYSTEM level property set
solidatus, rename propertyapi-tokentoapi-token-encrypted.Create a new plaintext property called
api-tokenand populate the value with the API token.Update property set.
Option 2:
In the DQM run.properties file, add property
dqm.do-allow-insecure-encrypted-property-usewith value set totrue, e.g.dqm.do-allow-insecure-encrypted-property-use=true
3.5 Profiling maximum row count for flat files is one less than expected
When creating a profiling configuration in ADQ there is the option to specify a maximum row count to limit the number of data rows extracted for the profiling. This row count includes the headers in the file meaning if the user sets the maximum row count to 5000, 4999 rows of data will be extracted.
Workaround
Set the maximum row count to 1 row higher than the desired row count.
3.6 Failing rule count query causes rule count file to be flagged as invalid
For SQL script rules, if the rule count query fails, the error message reported back to the user states that the rule count file output by the query is invalid. This occurs for both the Workflow Test and when the rule is ran as part of a Rule Execution Group.
Example error reported:
Workaround
None.
3.7 DQC setup script fails with some relative paths
setup.sh fails if invoked via a relative path from either the setup or setup/bin subdirectories of the distribution directory.
Work Around
Invoke other than via a relative path from either the setup or setup/bin subdirectories of the distribution directory. For example, invoke via a relative path from the DQC distribution directory.
3.8 Automated upgrade overwrites SSDQ client version
The automated upgrade will overwrite the version of the SSDQ client stored in the meta substitution property. This will mean the ADQ version will not show correctly on the About screen.
Workaround
Update the value of the version property in the meta set to 2025.03.05 after running the automated upgrade.
3.9 Automated upgrade can leave project items in metrics-producer-custom which have no corresponding rows in the rule dictionary
If a rule is present and enabled in the target client for the automated upgrade but not present in the source client, following the automated upgrade, the project item will remain in metrics-producer-custom but the row associated with the rule will not be present in the rule dictionary. These project items will fail when any execution groups are run.
Workaround
Remove the offending project items from metrics-producer-custom.
3.10 A single failure in the results data validation preparation stops execution
If the validation-prep solution fails for a single rule, the rule execution group execution is stopped for all rules and the execution status set as Failed.
Workaround
None.
4 Known Limitations
4.1 Rule Dictionary has a maximum limit of 45 columns
A rule dictionary of greater than 45 columns can be uploaded to DQM, but the workflow will fail if it exceeds 45 columns.
4.2 Rule Results files are limited to 30 columns
The rule is able to export more than 30 columns, but the workflow will fail if the results exceed 30 columns. This includes the row id, rule id and data owner columns. The rule input does not have a limit of columns.
4.3 FlowDesigner string length limitation
FlowDesigner has a maximum capacity of 32,000 bytes per string length for each row.
4.4 Disallowed rule result file column names
The following columns (case insensitive) are not permitted to be included in the rule result file produced by the rule as these column names are reserved for components of the DQ workflow:
autonumberstatetable
4.5 Rule result cannot have more than six primary keys
Rule result file can have up to six columns as a compound key for it to be successfully processed.
4.6 Solidatus integration only updates rules using SQL builder views without quoted identifiers
The Solidatus integration solution can currently only update rules whose input is from a SQL builder view and where this view is extracted without quoted identifiers, i.e. the tick box on the View Creation wizard ‘Do use quoted identifiers during extraction' is unticked.
5 Notices
5.1 PingFederate and Kerberos Authentication Behaviour
Scenario Overview
In environments making use of ADQ with PingFederate and Kerberos for authentication, users may encounter a specific behaviour:
Regardless of the username entered, a user can log in to ADQ if their machine user is a member of the appropriate groups. This means that authentication is based on the machine user rather than the individual’s entered username.
Technical Details
PingFederate Token Handling: This behaviour is partly due to how PingFederate processes and honours authentication tokens.
Auth Model Configuration: Our current authentication model requests a login for each session. However, PingFederate’s token handling mechanism bypasses this by validating the machine user’s group membership.
Implications for Users
Group Membership: Ensure that only authorized machine users are members of the relevant groups, as their membership will govern access to ADQ.
Username Entry: The entered username during login will not affect the authentication outcome if the machine user is appropriately grouped.
Recommendations
Review and manage group memberships to maintain secure access controls.
Inform users about this behaviour to set correct expectations during the login process.