v1.4.2 - ADQ Release Notes
ADQ v1.4.2 introduces two new key features to improve and simplify the steps required when upgrading the SSDQ client to improve the overall ADQ upgrade experience. This version also includes several bug fixes to enhance robustness of the data extraction layer and handles DQ rule failures more elegantly.
1 New Features
1.1 Automated SSDQ Client Upgrade
An upgrade client feature has been provided with ADQ to automate the upgrade of custom tables, FlowDesigner project inputs, and data source property sets within the SSDQ client to ensure it is compatible with ADQ. This saves a lot of manual effort involved when upgrading from an SSDQ to ADQ implementation, and reduces the likelihood of human error throughout this process.
1.2 Additional Validation Solutions
To further assist with the SSDQ to ADQ upgrade process, new validation solutions have been added to the SSDQ client to ensure the structure and content of the custom tables in this client are in the expected format by ADQ.
2 Defects Fixed
2.1 Rules Incorrectly Flagged as Failed
Rules which contain a hyphen in the rule ID are no longer incorrectly marked as a failed rule in the rule_status table.
2.2 No records in scope for the DQ Rule
Rule count files with only column headers are now accepted by the DQ workflow.
2.3 Disabled Reusable Rules
Disabled parameterized reusable rules (not suggested by the rule suggestion profile) no longer run in the DQ workflow, preventing potential process failures.
2.4 Batch Process Stability
If a producer solution fails, i.e. only a subset of DQ rules fail, it no longer causes the entire batch process to fail.
2.5 Service Reliability
Fixed issues causing occasional failures when starting the ADQ service.
2.6 Data Extraction Robustness
Data extraction now handles multiple database data source view rule inputs without failing, and now correctly handles scenarios where the rule ID is not the first column in the rule dictionary.
2.7 Column Name Visibility
Long column names in Data Source Views are no longer truncated, making them fully readable.
2.8 Accurate Record Counts
Data Issues statistics now shows the correct number of resolved, unresolved and pending-resolved records.
2.9 Log File Management
Rule suggestion log files are no longer deleted automatically.
2.10 Rollback Fix
The database driver has been specified for the aggregated results imports in the rollback postprocess solution.
3 Known Defects
3.1 Spaces forbidden in rule execution group name
Rule execution group names must not contain spaces. Names with spaces will cause the manual trigger of the rule execution group to fail.
Workaround
Do not use spaces in rule execution group names.
3.2 Rule results produced by SQL rules must not contain carriage returns or line feeds
The upload of data issues fails if there are carriage returns or line feeds within data columns in the rule results files produced by SQL rules.
Workaround
Use a custom FlowDesigner project rule to perform the logic instead.
3.3 Database data source view exports convert decimals past a certain length to scientific notation
When retrieving the rule input from a database data source view created using the SQL builder or the SQL query functionality, if the data column type is a float where the decimal numbers are past a certain length (greater than 10,000,000 or less than 0.001) the JDBC connector exporting the data converts this decimal number into scientific notation.
Workaround
Use a SQL query rule to perform the check rather than a custom FlowDesigner project rule. SQL query rules perform the rule logic directly on the source database so do not need a data source view defined. The SQL query should include a CAST in the query to ensure the data is not output in scientific notation, e.g. CAST(account_balance AS DECIMAL(20, 2)). Note you will need to know the number of digits after the decimal point.
3.4 Advanced Insights does not load for SQL rules
The drill down from data source to record level is not available for SQL based rules in Advanced Insights.
Workaround
None
3.5 Outlier Detection Profiling Option Always Available
When setting the property outlier-detection-enabled to false in substitution property set ssdq-settings, the outlier detection option is still available when configuring a profile.
Workaround
Disable the outlier-detection-controller solution item in SSDQ solution adq-profile-controller by setting the Activation Mode of the outlier-detection-controller item to Never.
3.6 Data Issues unavailable due to DQC database column data types no longer supported
For DQC instances initially created before December 2021, following an upgrade the Data Issues can fail to load due to the data types of columns comment, record_columns_metadata and auto_assignment_specification in the data_set_collections table, and the comment column in the data_sets table in the DQC database no longer being supported by Microsoft SQL Server.
Workaround
This involves altering tables within the DQC database so it is advised to have a backup of this database before making the fix.
Run the following commands one by one:
ALTER TABLE data_set_collections ALTER COLUMN comment nvarchar (MAX) NOT NULL
ALTER TABLE data_set_collections ALTER COLUMN record_columns_metadata nvarchar(MAX) NOT NULL
ALTER TABLE data_set_collections ALTER COLUMN auto_assignment_specification nvarchar(MAX)
If there are any errors reported, this will be an existing constraint on the data_set_collections table stopping the column’s data type from being changed. If so, run:
ALTER TABLE data_set_collections DROP CONSTRAINT (name of constraint reported in error)
Then repeat for the data_sets table, so run:
ALTER TABLE data_sets ALTER COLUMN comment nvarchar (MAX) NOT NULL
Likewise if an error is reported run:
ALTER TABLE data_sets DROP CONSTRAINT (name of constraint given in error)
Example SQL
ALTER TABLE data_set_collections DROP CONSTRAINT DF__data_set___comme__33D4B598
ALTER TABLE data_set_collections ALTER COLUMN comment nvarchar (MAX) NOT NULL
ALTER TABLE data_set_collections ALTER COLUMN record_columns_metadata nvarchar(MAX) NOT NULL
ALTER TABLE data_set_collections ALTER COLUMN auto_assignment_specification nvarchar(MAX)
ALTER TABLE data_sets DROP CONSTRAINT DF__data_sets__comme__37A5467C
ALTER TABLE data_sets ALTER COLUMN comment nvarchar (MAX) NOT NULL SQL3.7 Validation solutions do not correctly validate do-ignore-failures
The validation solutions do not correctly validate if the column DO_IGNORE_FAILURES in the custom table database-sources is set to true or false. The user can input any value (except null) in this column and the validation solution will deem it is correct.
Workaround
None.
3.8 Results table data_entry_definition populated incorrectly when column names contains hyphens
If a rule uses columns whose names contain hyphens, then the column names are split into parts and the different parts are used to populate the data_entry_server, data_entry_database, data_entry_schema, data_entry_table and data_entry_column columns in the data_entry_definition table. This also means that the primary key and break field columns are not correctly marked which will result in the break age table being empty.
Fix
To correct the problem in the SSDQ workflow, update the data_entry_definition_dqc_id project to use the attached project zip -
If any incorrect rows are present in data_entry_definition, then the below solutions can be used to correct these. To do this:
Upload the attached solution xmls -
Run the solution correct-data-entry-definition to correct entries in data_entry_definition.
Run the solution update-days-to-consider to populate the break age table.
3.9 Test connection solution fails for Oracle data source
When creating a database connection to Oracle, the test connection solution fails meaning the connection cannot be created.
Fix
In the SSDQ client in the DQM, rename existing solution database-source-test-connection to database-source-test-connection-OLD
Upload attached solution XML - .
3.10 Cannot create SQL builder views if there are commas within database column headers
When creating a SQL builder view, if any of the columns in the database tables have commas within their names, the solution which retrieves the list of table and column names, and primary and foreign key information fails.
Workaround
Create a SQL query view instead using an alias for any columns containing a comma in the name.
4 Known Limitations
4.1 Rule Dictionary has a maximum limit of 45 columns
A rule dictionary of greater than 45 columns can be uploaded to DQM, but the workflow will fail if it exceeds 45 columns.
4.2 Rule Results files are limited to 30 columns
The rule is able to export more than 30 columns, but the workflow will fail if the results exceed 30 columns. This includes the row id, rule id and data owner columns. The rule input does not have a limit of columns.
4.3 FlowDesigner string length limitation
FlowDesigner has a maximum capacity of 32,000 bytes per string length for each row.
4.4 Rule result cannot contain a column called autonumber
A column called autonumber (case insensitive) cannot be present in rule result. This column name is reserved for workflow projects.
4.5 Rule result cannot have more than six primary keys
Rule result file can have up to six columns as a compound key for it to be successfully processed.
5 Notices
5.1 PingFederate and Kerberos Authentication Behaviour
Scenario Overview
In environments making use of ADQ with PingFederate and Kerberos for authentication, users may encounter a specific behaviour:
Regardless of the username entered, a user can log in to ADQ if their machine user is a member of the appropriate groups. This means that authentication is based on the machine user rather than the individual’s entered username.
Technical Details
PingFederate Token Handling: This behaviour is partly due to how PingFederate processes and honours authentication tokens.
Auth Model Configuration: Our current authentication model requests a login for each session. However, PingFederate’s token handling mechanism bypasses this by validating the machine user’s group membership.
Implications for Users
Group Membership: Ensure that only authorized machine users are members of the relevant groups, as their membership will govern access to ADQ.
Username Entry: The entered username during login will not affect the authentication outcome if the machine user is appropriately grouped.
Recommendations
Review and manage group memberships to maintain secure access controls.
Inform users about this behaviour to set correct expectations during the login process.